如何使用IronWebScraper與ChatGPT Codex

Curtis Chau
Curtis Chau
已更新: 2026年10月19日

一個定價團隊每晚記錄線上目錄的價格並與前一晚進行比較。 目錄涵蓋五個頁面,每個頁面相互連結。 當其中一個頁面在爬取過程中失敗時,爬取仍會完成,並且仍會生成檔案。接下來的比較將缺少頁面的每個產品列為已終止,並讓團隊對從未改變的價格採取行動。

ChatGPT Codex是由OpenAI開發的編程代理。 在儲存庫中讀取檔案、編輯它們,執行諸如dotnet run的命令,並從終端機、IDE 擴展、ChatGPT 桌面應用或雲端報告所更改的內容。 IronWebScraper 提供 .NET 中的爬蟲,作為一個專案的基類進行擴展。 基類將請求排成佇列,為每個主機節流,重試失敗,並將爬取的行寫入 JSON Lines 檔案。

本指南為 Codex 提供了一個在本地主機上提供的測試目錄、一個規則檔和兩晚的價格。 每個請求數量的提示同時請求一個記錄它的檔案,因此Codex報告的每個數量都有數據支持。 在寫爬蟲之前,探測器測量了 IronWebScraper 如何處理失敗頁面、robots.txt 和其在該網站上的輸出文件。

快速開始
NuGet使用NuGet安裝

PM > Install-Package IronWebScraper

在 NuGet 包管理器控制台中運行上述命令安裝 IronWebScraper,或在 NuGet 包管理器中搜索該包。

在包含Data和TestData資料夾的.NET 9控制台專案中啟動Codex,測試目錄運行在埠5461上。

$ironwebscraper Add IronWebScraper 2026.10.1 to this console app and read the first catalogue page.

The test catalogue runs at http://localhost:5461/catalog/page-1.html.
Program.cs crawls that page alone and writes out/quickstart.csv, one row per product.
The CSV has these columns:

- sku, from the data-sku attribute
- name
- price, from the data-price attribute
- shown, the price text the page displays

The price monitor reads IRONWEBSCRAPER_LICENSE_KEY in code and assigns it to License.LicenseKey.
Keep the value out of output, files and logs.
Don't run anything that lists environment variables.

After three failed builds or runs, stop and paste the last error.
Otherwise, count the CSV rows and tell me the number.
Text


什麼是ChatGPT Codex,它如何融入 .NET 網路爬蟲工作?

Codex 在被授予的許可範圍內行事。 Codex CLI 文件涵蓋了安裝和終端使用,而開源儲存庫發布了其版本。

Codex 發明的成員會使構建失敗。 一個編譯、運行且靜默跳過頁面的爬蟲通過了構建,但本指南將解決第二個失敗。

IronWebScraper 提供 .NET Standard 2.0 和 2.1 的組件,並解析伺服器返回的HTML。 本指南中的爬蟲不會訪問第三方網站。

如何寫 IronWebScraper 的提示並為其設置 Codex?

快速入門提示命名起始頁、字段和輸出文件。快照和服務提示則參考網站規則的Data/README.md。

Codex在工作的目錄中查找.agents/skills每個文件夾直到儲存庫根目錄上。 將 IronWebScraper 的 skill.md 保存為.agents/skills/IronWebScraper/SKILL.md允許$ironwebscraper引用解析。 技能的示例程序針對 IronWebScraper 2026.8.1,而本指安裝2026.10.1。

Codex在運行開始時會讀取一次AGENTS.md。 它從儲存庫根部走到工作資料夾,並且較深的文件會覆蓋較淺的。 運行期間寫入的檔案在下次運行中生效。 /init命令生成首份草稿。

Create AGENTS.md in the repository root with a Price watch heading.
Give it these rules, one per line:

- IronWebScraper 2026.10.1 does all fetching and parsing.
- No HttpClient code, HtmlAgilityPack or AngleSharp.
- Data/README.md describes the site, the snapshot and the change report. It overrides anything else.
- Crawl localhost only: the test site on port 5461, or a listener a test starts.
- Every scraper adds the patterns */cart/* and */account/* to BannedUrls.
- RateLimitPerHost stays at one second or more.
- Init adds an HttpIdentity whose UserAgent is the one Data/README.md names.
- Data/ and TestData/ are read-only.
- Check IronWebScraper members in the ironwebscraper skill and the installed IronWebScraper.xml.
- When a probe against the test site disagrees with either of them, the probe wins.
- Any count you report comes from a file written that turn, and the reply names it.
- The key stays in IRONWEBSCRAPER_LICENSE_KEY.

Show me the file when it's done.
Text

規則在Init中添加HttpIdentity,因此每個請求都會命名爬蟲。 集線器的爬行預設顯示沒有爬蟲時IronWebScraper發送的內容。

如何添加IronWebScraper並應用密鑰?

Show me the IronWebScraper PackageReference in PriceWatch.csproj and the version NuGet restored.
Paste all of what dotnet list package --include-transitive prints after that.
Leave every package reference as it is.
Text

IronWebScraper 2026.10.1依賴IronSoftware.Common 2026.4.2和IronSoftware.System.Drawing 2026.8.2。測試運行中的包列表顯示了30個透過依賴的包,其中包括SixLabors.ImageSharp 3.1.12,Grpc.Net.Client 2.62.0和Polly 8.6.6。在鎖定版本前查看IronWebScraper更新日誌。

IronWebScraper 忽略IRONWEBSCRAPER_LICENSE_KEY,因此以該名稱保存密鑰的代碼在第一次爬取前分配License.LicenseKey。

Move the key line into Startup.ApplyIronWebScraperKey and call it at the top of Program.cs.
After the assignment, the method prints License.IsLicensed.

Next, find out what IronWebScraper does without a key.
Launch three child copies of the program, each with --licence-check N, set up like this:

1. The first child runs with IRONWEBSCRAPER_LICENSE_KEY removed from its environment.
2. The second keeps the variable, but --no-startup stops it from calling Startup.
3. The third keeps the variable and calls Startup.

Each child crawls catalogue page 1 into out/licence-check-N.csv.
Log every child in out/licence-check.csv with these columns:

- run and IsLicensed
- exception type, with no message
- rows written, and whether the child's CSV exists

Tell me which of the three wrote rows.
Text

授權金鑰頁面列出了提供金鑰的其他方式。

IronWebScraper對失敗頁面、robots.txt和輸出文件做了什麼?

購物網站指南使用Scrape編寫產品,並從其類別分析器中跟隨下一頁鏈接。 IronWebScraper 2026.10.1的四種行為決定了夜間價格檔案是否可靠,並且探測器測量了它們對測試目錄的影響。

  • 目錄頁 3 每次回答 503 時,IronWebScraper 以默認HttpRetryAttempts請求了九次,然後在無錯誤的情況下放棄它。 輸出包括了47個產品中的37個。
  • ObeyRobotsDotTxt預設打開,但沒有針對 /robots.txt 的請求到達測試站點,而robots.txt禁止的/cart/下的鏈接卻已提取。 IronWebScraper僅在默認埠上尋找robots.txt。 BannedUrls中的模式*/cart/*排除了這些鏈接。
  • Scrape追加到它的檔案。同一檔案的兩次運行留下了47個產品的94行。
  • ScrapeUnique會遺失寫入的第一條記錄。 47個產品的三次全新運行每次寫了46行,且每次都缺少LK-1001。
Probe how IronWebScraper treats the test site, in a console app at scratch/crawl-probe/.
Add a Compile Remove item so PriceWatch.csproj skips scratch/**.
Apply the key as Startup does.
Each case gets its own scraper, output folder and try block.

Measure these cases from Parse calls, FailedUrls and the files written:

- queue every nav.pager link from every page, then compare Parse calls with pages
- Scrape the same 10 rows into one file in two separate runs, and count lines after each
- ScrapeUnique 10 different rows into a new file, and count the lines
- request /probe/once-503.html and /probe/always-503.html, and note what reaches Parse
- leave ObeyRobotsDotTxt on and request /probe/blocked/page.html, which robots.txt disallows
- read LK-1045 on catalogue page 5: the price text and the data-price attribute

Then have the explorer agent read IronWebScraper.xml and the ironwebscraper skill, editing nothing.
It reports what both say about Scrape, ScrapeUnique, HttpRetryAttempts and ObeyRobotsDotTxt.

Write every finding into out/crawl-probe.csv, with a quote from the run, the XML or the skill.
Flag each row where a source and the probe disagree, and list those rows for me.
Text

如何構建快照爬蟲?

Data/README.md 定義了一個完整的快照,其中每個頁面從1到M到達Parse,並且沒有請求失敗。 價格來自 data-price 屬性,因為 LK-1045 顯示的文本為 $1,249.00,而屬性持有 1249.00。

Have the worker agent write Services/CatalogueCrawler.cs.
It builds a snapshot as Data/README.md describes.

PriceWatch takes two commands:

- --snapshot --date YYYY-MM-DD crawls every catalogue page into out/snapshots/<date>.csv
- --compare OLD NEW writes out/changes-NEW.csv from those two snapshot files

Use what the probe found when the crawler writes and checks its output:

- every run starts a new output file, so no rows carry over from an earlier run
- the snapshot is refused when FailedUrls is above zero
- it is also refused when a page from 1 to M never reached Parse
- a refusal prints FailedUrls and the page numbers that reached Parse
- the price column comes from data-price
- the was column comes from span.price-was, without the $ sign or commas

Run --snapshot --date 2026-10-15.
Tell me the row count and the number of pages, both read from the snapshot file.
Text

如何檢查Codex報告的價格變化?

The store changed its prices overnight.
Run --snapshot --date 2026-10-16, then --compare 2026-10-15 2026-10-16.

Check the report with code kept apart from the crawler.
That code reads both snapshots and out/changes-2026-10-16.csv.
It writes out/changes-check.csv with these columns:

- sku
- the change the snapshots show
- the change the report lists
- match, yes or no

Give me the count of each change type in the report, plus any row that doesn't match.
Text

輸出

下面的報告是爬蟲為測試運行第二晚寫入的文件,未改動過。

變更報告 CSV 列出三次降價、三次漲價、兩個新產品、一個移除的產品和一次庫存變更

來自測試運行的變更報告,out/changes-2026-10-16.csv,顯示為純文本。

目錄頁面失敗時發生什麼?

Catalogue page 3 now answers 503 to every request.
Show me that no snapshot gets saved in that state.

Run --snapshot --date 2026-10-17.
out/snapshots/2026-10-16.csv must stay identical, byte for byte.
Write out/failure-check.csv, holding these values:

- the exit code and the message printed
- FailedUrls and the pages that reached Parse, copied from that message
- whether out/snapshots/2026-10-17.csv exists
- the SHA-256 of the 2026-10-16 snapshot before and after the run

Say whether the snapshot was refused and whether the two hashes are equal.
Text

如何從截圖中檢查價格?

Codex接受圖像作為上下文。 用-i標誌(長格式--image)傳遞到codex exec或codex exec resume,或將其粘貼到互動式作曲器中。

測試目錄第1頁Trail Lantern價格為$24.99,Ridge Tent 2P價格為$289.00和Summit Pack 45L價格為$179.95

測試運行中附上的截圖,測試網站的目錄第1頁,作為第一晚服務。

A customer sent the screenshot I attached with -i.
They say the Ridge Tent 2P price on it is wrong.
Leave the code alone and go through this in order:

1. Read the name, price and stock of the first three products in the image.
2. Compare them with out/snapshots/2026-10-15.csv and out/snapshots/2026-10-16.csv.
3. Save the comparison as out/screenshot-check.csv.

Then quote the heading shown in the image.
Tell me which day's catalogue it shows and what the tent costs now.
Text

如何調試錯誤發出的價格提醒?

Work out why the 2026-10-16 price alerts went out wrong.
These files describe what happened:

- Data/alert-complaint.md lists four problems.
- Output/price-alerts-2026-10-16.csv is the file that was sent.
- Logs/price-check-2026-10-16.log reports success.
- Logs/store-access-2026-10-16.log is the store's record of the price check's requests.

Start with the explorer agent.
It reads Jobs/LegacyPriceCheck.cs, the two logs and the alert file.
Don't edit any code.
A script under scratch/ may read the files if that helps.

Write out/legacy-diagnosis.md, and include:

- the code line that causes each problem
- why the job log still said it succeeded
- the smallest fix for each problem, backed by a row of out/crawl-probe.csv

Then tell me each problem and its cause, one line per problem.
Text

在允許任何編輯之前檢查診斷。 傳統任務在Start返回後記錄其成功行,而從未讀取FailedUrls。

如何將爬蟲轉變為受測試的服務?

Ask the worker agent for a PriceWatchService class with these properties:

- A Microsoft.Extensions.DependencyInjection container can resolve it.
- A WatchOptions record configures it with the date, base URL, output folder and rate limit.
- It keeps every rule from Data/README.md and every check CatalogueCrawler makes.
- PriceWatchService.cs stays below 150 lines.
- Every public member carries an XML doc comment.
- Nothing calls Jobs/LegacyPriceCheck.cs.

Then take the 2026-10-16 snapshot again through the service, into out/final/2026-10-16.csv.
Compare it with out/snapshots/2026-10-16.csv and list every difference for me.
Text

IronWebScraper僅解析其自身獲取的響應。 Response沒有公共的方法加載已保存的HTML,且傳遞至Request的本地檔案地址未產生Parse調用且無錯誤。 在Codex沙箱測試中,Windows PowerShell中的HttpListener構造函數引發 PlatformNotSupportedException。 綁定至回環的TcpListener正常工作,因此測試提示從中提供了保存頁。

Add an xUnit project at tests/PriceWatch.Tests, in one solution with the main project.
Keep tests/** out of PriceWatch.csproj.
The tests apply the key through Startup.ApplyIronWebScraperKey.

IronWebScraper only parses pages it fetches.
So the tests serve TestData/ from a small TcpListener server on a free localhost port.
None of them calls port 5461.

Together, the tests cover these cases:

- catalog-page-1.html yields 10 products, the first being LK-1001
- LK-1045 on catalog-page-5.html has the price 1249.00
- a crawl where one page answers 503 and the rest answer 200 writes no snapshot
- comparing two small snapshot files finds each change type once

TestData/ holds only pages 1 and 5 of the catalogue.
The 503 case can serve a small made-up catalogue, so the 503 is the only failure.

Use dotnet test --logger trx --results-directory out/tests/.
Read the pass and fail counts from the trx file and tell me both.
Text

最後的提示檢查IronWebScraper.xml中的成員與探測和測試結果,因為XML描述ScrapeUnique忽略重複且未提及丟失的首條記錄。

$ironwebscraper Find the IronWebScraper.xml members that pace, retry, filter or save a crawl.
For each member, give me these two things:

- the XML's own description of it
- whether out/crawl-probe.csv or a test result under out/tests/ covers it

Then write a line that compiles for each member, in a project at scratch/api-probe/.
Build it with dotnet build scratch/api-probe, and don't run it.
Text

如何在其他目錄上重用這些提示?

提示命名了此目錄的選擇器、產品代碼和測試頁面。 要在供應商價格表、市場類別或內部產品饋送上重用它們,請替換提示中的這些名稱,以及Data/README.md中的起始頁、選擇器和快照列。 若要設定適用於每個助理,請參閱IronWebScraper AI 程式助理指南。

在實際運行中,這些提示產生了什麼結果?

上述提示按順序進入Codex,在一節中,針對localhost的測試目錄。網站在比較提示之前切換到第二晚的價格。 傳統任務、其日誌和警報在調試提示之前被添加。 分離的腳本讀取每個輸出檔案和測試網站的請求日誌。

  • 起初,儲存庫包含了一個無包的控制台項目、目錄規則和兩個保存的目錄頁面。
  • 在變數清除時,跳過Startup調用,爬蟲引發授權異常,未寫入任何產品行。
  • 在探測中,ScrapeUnique為10個獨特行寫入了9行,並且缺少LK-1001。
  • 第一個快照包含來自五頁的所有47個產品。
  • 兩個快照中的每個字段都與目錄匹配。
  • 變更報告列出了三次降價、三次漲價、兩個新產品、一個移除和一次庫存變更。
  • 第 3 頁失敗,爬蟲以代碼 2 退出並未保存快照。
  • 來自IronWebScraper的目錄請求間隔至少為1,037毫秒。
  • 運行中的93個請求中沒有一個發往購物車或帳戶頁面。
  • Codex將截圖與第一晚的快照匹配。
  • Codex給出帳篷的當前價格為$259.00。
  • 診斷將四個投訴中的每一個都與傳統任務中的行聯繫起來。
  • Codex在名為探險者或工作代理的四個提示中單獨啟動了每個代理。
  • 該服務逐字節重建了第二個快照。
  • 503測試提供了兩頁的虛構目錄,第二頁回答503。
  • Codex的最終運行和我們的四個測試全部通過。
  • 一些步驟與頁面上運行的不同。 首次完整測試運行中有四個測試中的一個失敗,Codex糾正了該測試的CSV解析。 當一個測試命令停滯時,Codex強行停止了在設置時間後啟動的所有dotnet進程,無論是哪個程序啟動它們。 NuGet 包來自在運行之前恢復的緩存,因為沙箱無法訪問 nuget.org。

Google Antigravity指南還通過瀏覽器將爬行輸出與人們看到的頁面進行對比。 Claude Code指南也測量了IronWebScraper如何在博客存檔爬取中丟棄失敗頁面。 共享的運行記錄列出了Codex版本、模型、測試機器和本次運行的計時。


疑難排解


問題?

如果您有任何疑問,請聯繫support@ironsoftware.com

Curtis Chau 擁有卡爾頓大學計算機科學學士學位,專注於前端開發,擅長於 Node.js、TypeScript、JavaScript 和 React。Curtis 熱衷於創建直觀且美觀的用戶界面,喜歡使用現代框架並打造結構良好、視覺吸引人的手冊。

閱讀更多

準備好開始了嗎?

NuGet 下載次數 150,061版本:2026.11剛剛發布