如何使用IronWebScraper与ChatGPT Codex

Curtis Chau
Curtis Chau
更新: 2026年10月19日

一个定价团队每晚记录在线目录的价格,并与前一晚的价格进行比较。 目录涵盖五个页面,每个页面都链接到其他页面。 当一个页面在爬取过程中失败时,爬取仍然完成,并仍然写入一个文件。然后比较将缺失页面上的每个产品列为已停产,并且团队对从未更改过的价格采取行动。

ChatGPT Codex是OpenAI的编码代理。 在一个存储库中,它读取文件,编辑它们,运行如dotnet run的命令,并报告它所改变的,从终端、IDE扩展、ChatGPT桌面应用程序或云端。 IronWebScraper在.NET中提供爬取功能,作为一个项目扩展的基类。 基类排队请求,根据主机调整它们的节奏,重试失败并将爬取的行写入JSON Lines文件。

本指南给Codex一个在localhost上提供服务的测试目录,一个规则文件和两个晚上的价格。 每一个要求计数的提示也要求记录它的文件,因此每个Codex报告的计数都由数据支持。 在写入爬虫之前,一个探测器衡量IronWebScraper如何处理失败的页面、robots.txt和该站点上的输出文件。

快速入门
NuGet使用NuGet安装

PM > Install-Package IronWebScraper

通过在 NuGet 包管理器控制台中运行上述命令安装 IronWebScraper,或在 NuGet 包管理器中搜索该包。

在一个包含Data和TestData文件夹的.NET 9控制台项目中启动Codex,测试目录运行在端口5461。

$ironwebscraper Add IronWebScraper 2026.10.1 to this console app and read the first catalogue page.

The test catalogue runs at http://localhost:5461/catalog/page-1.html.
Program.cs crawls that page alone and writes out/quickstart.csv, one row per product.
The CSV has these columns:

- sku, from the data-sku attribute
- name
- price, from the data-price attribute
- shown, the price text the page displays

The price monitor reads IRONWEBSCRAPER_LICENSE_KEY in code and assigns it to License.LicenseKey.
Keep the value out of output, files and logs.
Don't run anything that lists environment variables.

After three failed builds or runs, stop and paste the last error.
Otherwise, count the CSV rows and tell me the number.
Text


什么是ChatGPT Codex,它如何融入.NET网络爬取工作中?

Codex在其被授予的权限范围内行动。 Codex CLI文档涵盖安装和终端使用,开源存储库发布其版本。

Codex发明的一个成员使建立失败。 一个编译、运行并无声跳过页面的爬虫通过了构建,本指南解决了第二个故障。

IronWebScraper提供.NET Standard 2.0和2.1程序集并解析服务器返回的HTML。 本指南中的任何爬取都不会达到第三方网站。

您如何编写IronWebScraper提示并为其设置Codex?

快速入门提示命名起始页面、字段和输出文件。快照和服务提示参考Data/README.md中的站点规则。

Codex 从工作目录到存储库根目录的每个文件夹中扫描 .agents/skills。 将IronWebScraper的skill.md保存为.agents/skills/IronWebScraper/SKILL.md允许$ironwebscraper提及可以解析。 该技能的示例程序针对IronWebScraper 2026.8.1,而本指南安装2026.10.1。

Codex在运行开始时读取AGENTS.md一次。 它从存储库根目录走到工作文件夹,较深的文件会覆盖较浅的文件。 运行期间写入的文件在下次运行时生效。 /init命令生成初稿。

Create AGENTS.md in the repository root with a Price watch heading.
Give it these rules, one per line:

- IronWebScraper 2026.10.1 does all fetching and parsing.
- No HttpClient code, HtmlAgilityPack or AngleSharp.
- Data/README.md describes the site, the snapshot and the change report. It overrides anything else.
- Crawl localhost only: the test site on port 5461, or a listener a test starts.
- Every scraper adds the patterns */cart/* and */account/* to BannedUrls.
- RateLimitPerHost stays at one second or more.
- Init adds an HttpIdentity whose UserAgent is the one Data/README.md names.
- Data/ and TestData/ are read-only.
- Check IronWebScraper members in the ironwebscraper skill and the installed IronWebScraper.xml.
- When a probe against the test site disagrees with either of them, the probe wins.
- Any count you report comes from a file written that turn, and the reply names it.
- The key stays in IRONWEBSCRAPER_LICENSE_KEY.

Show me the file when it's done.
Text

规则在Init中添加一个HttpIdentity,使每个请求都命名爬虫。 集线器的爬取默认设置显示IronWebScraper在没有默认值的情况下发送的内容。

您如何添加IronWebScraper并应用密钥?

Show me the IronWebScraper PackageReference in PriceWatch.csproj and the version NuGet restored.
Paste all of what dotnet list package --include-transitive prints after that.
Leave every package reference as it is.
Text

IronWebScraper 2026.10.1依赖IronSoftware.Common 2026.4.2和IronSoftware.System.Drawing 2026.8.2。测试运行中的包列表显示了30个传递性包,其中包括SixLabors.ImageSharp 3.1.12、Grpc.Net.Client 2.62.0和Polly 8.6.6。在固定版本之前查看IronWebScraper变更日志。

IronWebScraper忽略IRONWEBSCRAPER_LICENSE_KEY,所以在第一次爬取之前,将密钥保存在该名称下的代码会分配License.LicenseKey。

Move the key line into Startup.ApplyIronWebScraperKey and call it at the top of Program.cs.
After the assignment, the method prints License.IsLicensed.

Next, find out what IronWebScraper does without a key.
Launch three child copies of the program, each with --licence-check N, set up like this:

1. The first child runs with IRONWEBSCRAPER_LICENSE_KEY removed from its environment.
2. The second keeps the variable, but --no-startup stops it from calling Startup.
3. The third keeps the variable and calls Startup.

Each child crawls catalogue page 1 into out/licence-check-N.csv.
Log every child in out/licence-check.csv with these columns:

- run and IsLicensed
- exception type, with no message
- rows written, and whether the child's CSV exists

Tell me which of the three wrote rows.
Text

《许可证密钥页面》列出了提供密钥的其他方法。

IronWebScraper对处理失败的页面、robots.txt和输出文件做了什么?

购物网站指南写入使用Scrape的产品,并从其类别解析器中跟随下一页面链接。 IronWebScraper 2026.10.1的四种行为决定每晚的价格文件是否可信,探测器对测试目录中的每一种进行测量。

  • 在目录页面3每次都响应503的情况下,IronWebScraper在默认HttpRetryAttempts下请求它九次,然后无错误地丢弃它。 输出持有47个产品中的37个。
  • ObeyRobotsDotTxt默认开启,但没有请求/robots.txt到达测试站点,/cart/下的链接被获取,robots.txt不允许。 IronWebScraper只在默认端口寻找robots.txt。 */cart/*在BannedUrls中排除了那些链接。
  • Scrape附加到其文件。两次运行进入同一个文件,留下47个产品的94行。
  • ScrapeUnique失去了它写入的第一条记录。 三个新运行,每个有47个产品,写了46行,LK-1001每次都缺失。
Probe how IronWebScraper treats the test site, in a console app at scratch/crawl-probe/.
Add a Compile Remove item so PriceWatch.csproj skips scratch/**.
Apply the key as Startup does.
Each case gets its own scraper, output folder and try block.

Measure these cases from Parse calls, FailedUrls and the files written:

- queue every nav.pager link from every page, then compare Parse calls with pages
- Scrape the same 10 rows into one file in two separate runs, and count lines after each
- ScrapeUnique 10 different rows into a new file, and count the lines
- request /probe/once-503.html and /probe/always-503.html, and note what reaches Parse
- leave ObeyRobotsDotTxt on and request /probe/blocked/page.html, which robots.txt disallows
- read LK-1045 on catalogue page 5: the price text and the data-price attribute

Then have the explorer agent read IronWebScraper.xml and the ironwebscraper skill, editing nothing.
It reports what both say about Scrape, ScrapeUnique, HttpRetryAttempts and ObeyRobotsDotTxt.

Write every finding into out/crawl-probe.csv, with a quote from the run, the XML or the skill.
Flag each row where a source and the probe disagree, and list those rows for me.
Text

如何构建快照爬虫?

Data/README.md定义一个完整的快照为每个页面从1到M到达Parse且没有请求失败的情况。 价格来自data-price属性,因为显示的文本LK-1045读作$1,249.00而属性保存1249.00。

Have the worker agent write Services/CatalogueCrawler.cs.
It builds a snapshot as Data/README.md describes.

PriceWatch takes two commands:

- --snapshot --date YYYY-MM-DD crawls every catalogue page into out/snapshots/<date>.csv
- --compare OLD NEW writes out/changes-NEW.csv from those two snapshot files

Use what the probe found when the crawler writes and checks its output:

- every run starts a new output file, so no rows carry over from an earlier run
- the snapshot is refused when FailedUrls is above zero
- it is also refused when a page from 1 to M never reached Parse
- a refusal prints FailedUrls and the page numbers that reached Parse
- the price column comes from data-price
- the was column comes from span.price-was, without the $ sign or commas

Run --snapshot --date 2026-10-15.
Tell me the row count and the number of pages, both read from the snapshot file.
Text

如何检查Codex报告的价格变化?

The store changed its prices overnight.
Run --snapshot --date 2026-10-16, then --compare 2026-10-15 2026-10-16.

Check the report with code kept apart from the crawler.
That code reads both snapshots and out/changes-2026-10-16.csv.
It writes out/changes-check.csv with these columns:

- sku
- the change the snapshots show
- the change the report lists
- match, yes or no

Give me the count of each change type in the report, plus any row that doesn't match.
Text

输出

下面的报告是爬虫为测试运行的第二个晚上写入的文件,未更改。

变更报告CSV列出三个降价,三个涨价,两个新产品,一个移除产品和一次库存变化。

测试运行的变更报告,out/changes-2026-10-16.csv,显示为纯文本。

当一个目录页面失败时会发生什么?

Catalogue page 3 now answers 503 to every request.
Show me that no snapshot gets saved in that state.

Run --snapshot --date 2026-10-17.
out/snapshots/2026-10-16.csv must stay identical, byte for byte.
Write out/failure-check.csv, holding these values:

- the exit code and the message printed
- FailedUrls and the pages that reached Parse, copied from that message
- whether out/snapshots/2026-10-17.csv exists
- the SHA-256 of the 2026-10-16 snapshot before and after the run

Say whether the snapshot was refused and whether the two hashes are equal.
Text

如何从截图中检查价格?

Codex 接受图像作为上下文。 通过-i标志(长表--image)传递给codex exec或codex exec resume,或将其粘贴到交互式编辑器中。

测试目录页面1,Trail Lantern价格为$24.99,Ridge Tent 2P价格为$289.00,Summit Pack 45L价格为$179.95。

测试运行中附加的截图,测试站点的目录页面1,在第一个晚上提供。

A customer sent the screenshot I attached with -i.
They say the Ridge Tent 2P price on it is wrong.
Leave the code alone and go through this in order:

1. Read the name, price and stock of the first three products in the image.
2. Compare them with out/snapshots/2026-10-15.csv and out/snapshots/2026-10-16.csv.
3. Save the comparison as out/screenshot-check.csv.

Then quote the heading shown in the image.
Tell me which day's catalogue it shows and what the tent costs now.
Text

如何调试错误发出的价格警报?

Work out why the 2026-10-16 price alerts went out wrong.
These files describe what happened:

- Data/alert-complaint.md lists four problems.
- Output/price-alerts-2026-10-16.csv is the file that was sent.
- Logs/price-check-2026-10-16.log reports success.
- Logs/store-access-2026-10-16.log is the store's record of the price check's requests.

Start with the explorer agent.
It reads Jobs/LegacyPriceCheck.cs, the two logs and the alert file.
Don't edit any code.
A script under scratch/ may read the files if that helps.

Write out/legacy-diagnosis.md, and include:

- the code line that causes each problem
- why the job log still said it succeeded
- the smallest fix for each problem, backed by a row of out/crawl-probe.csv

Then tell me each problem and its cause, one line per problem.
Text

在允许任何编辑前查看诊断结果。 旧作业在Start返回后记录其成功行,并且从未读取FailedUrls。

如何将爬虫变成经过测试的服务?

Ask the worker agent for a PriceWatchService class with these properties:

- A Microsoft.Extensions.DependencyInjection container can resolve it.
- A WatchOptions record configures it with the date, base URL, output folder and rate limit.
- It keeps every rule from Data/README.md and every check CatalogueCrawler makes.
- PriceWatchService.cs stays below 150 lines.
- Every public member carries an XML doc comment.
- Nothing calls Jobs/LegacyPriceCheck.cs.

Then take the 2026-10-16 snapshot again through the service, into out/final/2026-10-16.csv.
Compare it with out/snapshots/2026-10-16.csv and list every difference for me.
Text

IronWebScraper只解析它自己获取的响应。 Response无公开方式加载保存的HTML,传递给Request的本地文件地址无Parse调用且无错误。 在Codex沙箱测试中,Windows PowerShell中的HttpListener构造函数抛出PlatformNotSupportedException。 绑定到回送的TcpListener工作,所以测试提示从其中提供保存的页面。

Add an xUnit project at tests/PriceWatch.Tests, in one solution with the main project.
Keep tests/** out of PriceWatch.csproj.
The tests apply the key through Startup.ApplyIronWebScraperKey.

IronWebScraper only parses pages it fetches.
So the tests serve TestData/ from a small TcpListener server on a free localhost port.
None of them calls port 5461.

Together, the tests cover these cases:

- catalog-page-1.html yields 10 products, the first being LK-1001
- LK-1045 on catalog-page-5.html has the price 1249.00
- a crawl where one page answers 503 and the rest answer 200 writes no snapshot
- comparing two small snapshot files finds each change type once

TestData/ holds only pages 1 and 5 of the catalogue.
The 503 case can serve a small made-up catalogue, so the 503 is the only failure.

Use dotnet test --logger trx --results-directory out/tests/.
Read the pass and fail counts from the trx file and tell me both.
Text

最后的提示检查IronWebScraper.xml中的成员与探测和测试结果,因为XML描述ScrapeUnique为忽略重复项并未提及丢失的第一条记录。

$ironwebscraper Find the IronWebScraper.xml members that pace, retry, filter or save a crawl.
For each member, give me these two things:

- the XML's own description of it
- whether out/crawl-probe.csv or a test result under out/tests/ covers it

Then write a line that compiles for each member, in a project at scratch/api-probe/.
Build it with dotnet build scratch/api-probe, and don't run it.
Text

如何在另一个目录上重用这些提示?

提示中命名了这个目录的选择器、产品代码和测试页面。 要在供应商价格表、市场类别或内部产品提要上重用它们,请替换提示中的那些名称,以及Data/README.md中的起始页面、选择器和快照列。 对于适用于每个助手的设置,请参考IronWebScraper AI编码助手指南。

在实际运行中这些提示生成了什么?

上面的提示依次发送到Codex中,在一个会话中,针对localhost上的测试目录。在比较提示之前,网站切换到第二晚的价格。 旧作业、日志及警报在调试提示之前被添加。 单独的脚本读取每个输出文件和测试站点的请求日志。

  • 一开始,存储库有一个没有包的控制台项目、目录规则和两个保存的目录页面。
  • 在变量清除且启动调用跳过的情况下,爬取引发了许可异常并未写入产品行。
  • 在探测中,ScrapeUnique写了10行中的9行,LK-1001缺失。
  • 第一个快照持有五页中的47个产品。
  • 两个快照中的每个字段都与目录匹配。
  • 变更报告列出3次降价、3次涨价、2个新产品、1次移除和1次库存变更。
  • 在页面3失败的情况下,爬虫以代码2退出,并且没有保存快照。
  • 来自IronWebScraper的目录请求至少相隔1,037毫秒。
  • 运行中的93个请求中没有进入购物车或帐户页面。
  • Codex将截图与第一个晚上的快照匹配。
  • Codex将帐篷的当前价格给出为$259.00。
  • 诊断将四个投诉中的每一个与其在旧作业中的行联系起来。
  • Codex在每一个命名探险者或工作代理的四个提示中启动单独的代理。
  • 服务逐字节重建了第二个快照。
  • 503测试提供了两页的假目录,其第二页回答503。
  • 所有四个测试在Codex最终运行和我们的运行中通过。
  • 有些步骤与页面中的运行不一样。 第一次全面测试运行失败了四个测试中的一个,Codex修正了该测试的CSV解析。 当一个测试命令停顿时,Codex强制停止所有在设定时间后启动的dotnet进程,无论哪个程序启动了它们。 NuGet包来自在运行前恢复的缓存,因为沙箱无法访问nuget.org。

Google Antigravity指南还通过浏览器检查爬取输出与一个人看到的页面。 Claude Code指南还衡量IronWebScraper如何在博客存档爬取中下降失败的页面。 共享的运行记录列出此运行的Codex版本、型号、测试机器和时间。


故障排除


有问题吗?

如果您有任何问题,请联系support@ironsoftware.com

Curtis Chau 拥有卡尔顿大学的计算机科学学士学位,专注于前端开发,精通 Node.js、TypeScript、JavaScript 和 React。他热衷于打造直观且美观的用户界面,喜欢使用现代框架并创建结构良好、视觉吸引力强的手册。

阅读更多

准备开始了吗?

NuGet 下载量 150,061版本:2026.11刚刚发布