How to Use IronWebScraper with ChatGPT Codex
A pricing team records the prices of an online catalogue every night and compares them with the night before. The catalogue spans five pages, and each page links to the others. When one page fails during the crawl, the crawl still finishes and still writes a file. The comparison then lists every product on the missing page as discontinued, and the team acts on prices that never changed.
ChatGPT Codex is OpenAI's coding agent. Inside a repository it reads files, edits them, runs commands such as dotnet run and reports what it changed, from a terminal, an IDE extension, the ChatGPT desktop app or the cloud. IronWebScraper provides crawling in .NET as a base class that a project extends. The base class queues requests, throttles them per host, retries failures and writes scraped rows to JSON Lines files.
This guide gives Codex a test catalogue served on localhost, a rules file and two nights of prices. Every prompt that asks for a count also asks for a file that records it, so each count Codex reports is backed by data. Before the crawler is written, a probe measures how IronWebScraper treats failed pages, robots.txt and its own output files on that site.
Quickstart
Install with NuGetStart Codex in a .NET 9 console project that contains the Data and TestData folders, with the test catalogue running on port 5461.
$ironwebscraper Add IronWebScraper 2026.10.1 to this console app and read the first catalogue page.
The test catalogue runs at http://localhost:5461/catalog/page-1.html.
Program.cs crawls that page alone and writes out/quickstart.csv, one row per product.
The CSV has these columns:
- sku, from the data-sku attribute
- name
- price, from the data-price attribute
- shown, the price text the page displays
The price monitor reads IRONWEBSCRAPER_LICENSE_KEY in code and assigns it to License.LicenseKey.
Keep the value out of output, files and logs.
Don't run anything that lists environment variables.
After three failed builds or runs, stop and paste the last error.
Otherwise, count the CSV rows and tell me the number.Minimal Workflow (5 steps)
- Download the C# web scraping library from NuGet
- Save IronWebScraper's skill as .agents/skills/ironwebscraper/SKILL.md so the
$ironwebscrapermention resolves - Commit an AGENTS.md that bans the cart and account paths and keeps one second between requests
- Probe failed pages and output files against a locally served copy of the store before writing the crawler
- Refuse any snapshot with a failed request or a missing page, and check the price report with separate code
What is ChatGPT Codex, and how does it fit into .NET web scraping work?
Codex acts within the permissions it is granted. The Codex CLI documentation covers installation and terminal use, and the open-source repository publishes its releases.
A member that Codex invents fails the build. A crawler that compiles, runs and silently skips a page passes the build, and this guide addresses that second failure.
IronWebScraper ships .NET Standard 2.0 and 2.1 assemblies and parses the HTML a server returns. No crawl in this guide reaches a third-party site.
How do you write an IronWebScraper prompt and set up Codex for it?
The quickstart prompt names the start page, the fields and the output file. The snapshot and service prompts refer to Data/README.md for the site's rules.
Codex scans .agents/skills in every folder from the working directory up to the repository root. Saving IronWebScraper's skill.md as .agents/skills/ironwebscraper/SKILL.md allows the $ironwebscraper mention to resolve. The skill's sample program targets IronWebScraper 2026.8.1, while this guide installs 2026.10.1.
Codex reads AGENTS.md once, when a run starts. It walks from the repository root down to the working folder, and a deeper file overrides a shallower one. A file written during a run takes effect in the next run. The /init command generates a first draft.
Create AGENTS.md in the repository root with a Price watch heading.
Give it these rules, one per line:
- IronWebScraper 2026.10.1 does all fetching and parsing.
- No HttpClient code, HtmlAgilityPack or AngleSharp.
- Data/README.md describes the site, the snapshot and the change report. It overrides anything else.
- Crawl localhost only: the test site on port 5461, or a listener a test starts.
- Every scraper adds the patterns */cart/* and */account/* to BannedUrls.
- RateLimitPerHost stays at one second or more.
- Init adds an HttpIdentity whose UserAgent is the one Data/README.md names.
- Data/ and TestData/ are read-only.
- Check IronWebScraper members in the ironwebscraper skill and the installed IronWebScraper.xml.
- When a probe against the test site disagrees with either of them, the probe wins.
- Any count you report comes from a file written that turn, and the reply names it.
- The key stays in IRONWEBSCRAPER_LICENSE_KEY.
Show me the file when it's done.The rules add an HttpIdentity in Init so that every request names the crawler. The hub's crawl defaults show what IronWebScraper sends without one.
How do you add IronWebScraper and apply the key?
Show me the IronWebScraper PackageReference in PriceWatch.csproj and the version NuGet restored.
Paste all of what dotnet list package --include-transitive prints after that.
Leave every package reference as it is.IronWebScraper 2026.10.1 depends on IronSoftware.Common 2026.4.2 and IronSoftware.System.Drawing 2026.8.2. The package list in the test run showed 30 transitive packages, among them SixLabors.ImageSharp 3.1.12, Grpc.Net.Client 2.62.0 and Polly 8.6.6. Review the IronWebScraper changelog before pinning a release.
IronWebScraper ignores IRONWEBSCRAPER_LICENSE_KEY, so code that keeps the key under that name assigns License.LicenseKey before the first crawl.
Move the key line into Startup.ApplyIronWebScraperKey and call it at the top of Program.cs.
After the assignment, the method prints License.IsLicensed.
Next, find out what IronWebScraper does without a key.
Launch three child copies of the program, each with --licence-check N, set up like this:
1. The first child runs with IRONWEBSCRAPER_LICENSE_KEY removed from its environment.
2. The second keeps the variable, but --no-startup stops it from calling Startup.
3. The third keeps the variable and calls Startup.
Each child crawls catalogue page 1 into out/licence-check-N.csv.
Log every child in out/licence-check.csv with these columns:
- run and IsLicensed
- exception type, with no message
- rows written, and whether the child's CSV exists
Tell me which of the three wrote rows.The license keys page lists the other ways to supply a key.
What does IronWebScraper do with failed pages, robots.txt and output files?
The shopping website guide writes products with Scrape and follows the next-page link from its category parser. Four behaviors of IronWebScraper 2026.10.1 decide whether a nightly price file can be trusted, and the probes measured each one against the test catalogue.
- With catalogue page 3 answering 503 every time, IronWebScraper requested it nine times under the default
HttpRetryAttemptsand then dropped it without an error. The output held 37 of 47 products. ObeyRobotsDotTxtis on by default, yet no request for /robots.txt reached the test site, and links under /cart/, which robots.txt disallows, were fetched. IronWebScraper looks for robots.txt only on the default port. The pattern*/cart/*inBannedUrlsexcluded those links.Scrapeappends to its file. Two runs into the same file left 94 lines for 47 products.ScrapeUniqueloses the first record it writes. Three fresh runs of 47 products each wrote 46 lines, and LK-1001 was missing every time.
Probe how IronWebScraper treats the test site, in a console app at scratch/crawl-probe/.
Add a Compile Remove item so PriceWatch.csproj skips scratch/**.
Apply the key as Startup does.
Each case gets its own scraper, output folder and try block.
Measure these cases from Parse calls, FailedUrls and the files written:
- queue every nav.pager link from every page, then compare Parse calls with pages
- Scrape the same 10 rows into one file in two separate runs, and count lines after each
- ScrapeUnique 10 different rows into a new file, and count the lines
- request /probe/once-503.html and /probe/always-503.html, and note what reaches Parse
- leave ObeyRobotsDotTxt on and request /probe/blocked/page.html, which robots.txt disallows
- read LK-1045 on catalogue page 5: the price text and the data-price attribute
Then have the explorer agent read IronWebScraper.xml and the ironwebscraper skill, editing nothing.
It reports what both say about Scrape, ScrapeUnique, HttpRetryAttempts and ObeyRobotsDotTxt.
Write every finding into out/crawl-probe.csv, with a quote from the run, the XML or the skill.
Flag each row where a source and the probe disagree, and list those rows for me.How do you build the snapshot crawler?
Data/README.md defines a complete snapshot as one in which every page from 1 to M reached Parse and no request failed. The price comes from the data-price attribute, because the shown text of LK-1045 reads $1,249.00 while the attribute holds 1249.00.
Have the worker agent write Services/CatalogueCrawler.cs.
It builds a snapshot as Data/README.md describes.
PriceWatch takes two commands:
- --snapshot --date YYYY-MM-DD crawls every catalogue page into out/snapshots/<date>.csv
- --compare OLD NEW writes out/changes-NEW.csv from those two snapshot files
Use what the probe found when the crawler writes and checks its output:
- every run starts a new output file, so no rows carry over from an earlier run
- the snapshot is refused when FailedUrls is above zero
- it is also refused when a page from 1 to M never reached Parse
- a refusal prints FailedUrls and the page numbers that reached Parse
- the price column comes from data-price
- the was column comes from span.price-was, without the $ sign or commas
Run --snapshot --date 2026-10-15.
Tell me the row count and the number of pages, both read from the snapshot file.How do you check the price changes Codex reports?
The store changed its prices overnight.
Run --snapshot --date 2026-10-16, then --compare 2026-10-15 2026-10-16.
Check the report with code kept apart from the crawler.
That code reads both snapshots and out/changes-2026-10-16.csv.
It writes out/changes-check.csv with these columns:
- sku
- the change the snapshots show
- the change the report lists
- match, yes or no
Give me the count of each change type in the report, plus any row that doesn't match.Output
The report below is the file the crawler wrote for the second night in the test run, unchanged.
The change report from the test run, out/changes-2026-10-16.csv, shown as plain text.
What happens when a catalogue page fails?
Catalogue page 3 now answers 503 to every request.
Show me that no snapshot gets saved in that state.
Run --snapshot --date 2026-10-17.
out/snapshots/2026-10-16.csv must stay identical, byte for byte.
Write out/failure-check.csv, holding these values:
- the exit code and the message printed
- FailedUrls and the pages that reached Parse, copied from that message
- whether out/snapshots/2026-10-17.csv exists
- the SHA-256 of the 2026-10-16 snapshot before and after the run
Say whether the snapshot was refused and whether the two hashes are equal.How do you check a price from a screenshot?
Codex accepts images as context. Pass one with the -i flag (long form --image) to codex exec or codex exec resume, or paste it into the interactive composer.
The screenshot attached in the test run, catalogue page 1 of the test site as served on the first night.
A customer sent the screenshot I attached with -i.
They say the Ridge Tent 2P price on it is wrong.
Leave the code alone and go through this in order:
1. Read the name, price and stock of the first three products in the image.
2. Compare them with out/snapshots/2026-10-15.csv and out/snapshots/2026-10-16.csv.
3. Save the comparison as out/screenshot-check.csv.
Then quote the heading shown in the image.
Tell me which day's catalogue it shows and what the tent costs now.How do you debug a price alert that went out wrong?
Work out why the 2026-10-16 price alerts went out wrong.
These files describe what happened:
- Data/alert-complaint.md lists four problems.
- Output/price-alerts-2026-10-16.csv is the file that was sent.
- Logs/price-check-2026-10-16.log reports success.
- Logs/store-access-2026-10-16.log is the store's record of the price check's requests.
Start with the explorer agent.
It reads Jobs/LegacyPriceCheck.cs, the two logs and the alert file.
Don't edit any code.
A script under scratch/ may read the files if that helps.
Write out/legacy-diagnosis.md, and include:
- the code line that causes each problem
- why the job log still said it succeeded
- the smallest fix for each problem, backed by a row of out/crawl-probe.csv
Then tell me each problem and its cause, one line per problem.Review the diagnosis before allowing any edit. The legacy job logged its success line after Start returned and never read FailedUrls.
How do you turn the crawler into a tested service?
Ask the worker agent for a PriceWatchService class with these properties:
- A Microsoft.Extensions.DependencyInjection container can resolve it.
- A WatchOptions record configures it with the date, base URL, output folder and rate limit.
- It keeps every rule from Data/README.md and every check CatalogueCrawler makes.
- PriceWatchService.cs stays below 150 lines.
- Every public member carries an XML doc comment.
- Nothing calls Jobs/LegacyPriceCheck.cs.
Then take the 2026-10-16 snapshot again through the service, into out/final/2026-10-16.csv.
Compare it with out/snapshots/2026-10-16.csv and list every difference for me.IronWebScraper parses only the responses it fetches itself. Response has no public way to load saved HTML, and a local file address passed to Request produced no Parse call and no error. In a Codex sandbox test, the HttpListener constructor in Windows PowerShell threw a PlatformNotSupportedException. A TcpListener bound to loopback worked, so the test prompt serves the saved pages from one.
Add an xUnit project at tests/PriceWatch.Tests, in one solution with the main project.
Keep tests/** out of PriceWatch.csproj.
The tests apply the key through Startup.ApplyIronWebScraperKey.
IronWebScraper only parses pages it fetches.
So the tests serve TestData/ from a small TcpListener server on a free localhost port.
None of them calls port 5461.
Together, the tests cover these cases:
- catalog-page-1.html yields 10 products, the first being LK-1001
- LK-1045 on catalog-page-5.html has the price 1249.00
- a crawl where one page answers 503 and the rest answer 200 writes no snapshot
- comparing two small snapshot files finds each change type once
TestData/ holds only pages 1 and 5 of the catalogue.
The 503 case can serve a small made-up catalogue, so the 503 is the only failure.
Use dotnet test --logger trx --results-directory out/tests/.
Read the pass and fail counts from the trx file and tell me both.The last prompt checks the members in IronWebScraper.xml against the probe and test results, because the XML describes ScrapeUnique as ignoring duplicates and says nothing of the lost first record.
$ironwebscraper Find the IronWebScraper.xml members that pace, retry, filter or save a crawl.
For each member, give me these two things:
- the XML's own description of it
- whether out/crawl-probe.csv or a test result under out/tests/ covers it
Then write a line that compiles for each member, in a project at scratch/api-probe/.
Build it with dotnet build scratch/api-probe, and don't run it.How do you reuse these prompts on another catalogue?
The prompts name this catalogue's selectors, product codes and test pages. To reuse them on a supplier price list, a marketplace category or an internal product feed, replace those names in the prompts, along with the start page, the selectors and the snapshot columns in Data/README.md. For setup that applies to every assistant, refer to the IronWebScraper AI coding assistants guide.
What did the prompts produce in a real run?
The prompts above went to Codex in order, in one session, against the test catalogue on localhost. The site switched to the second night's prices before the comparison prompt. The legacy job, its logs and its alerts were added before the debugging prompt. Separate scripts read every output file and the test site's request log.
- At the start, the repository held a console project with no packages, the catalogue rules and two saved catalogue pages.
- With the variable cleared, and with the Startup call skipped, the crawl threw a licensing exception and wrote no product rows.
- In the probe, ScrapeUnique wrote 9 lines for 10 distinct rows, and LK-1001 was missing.
- The first snapshot held all 47 products from five pages.
- Every field in both snapshots matched the catalogue.
- The change report listed 3 price drops, 3 price rises, 2 new products, 1 removal and 1 stock change.
- With page 3 failing, the crawler exited with code 2 and saved no snapshot.
- Catalogue requests from IronWebScraper were at least 1,037 milliseconds apart.
- None of the 93 requests in the run went to a cart or account page.
- Codex matched the screenshot to the first night's snapshot.
- Codex gave the tent's current price as $259.00.
- The diagnosis tied each of the four complaints to its lines in the legacy job.
- Codex started a separate agent in each of the four prompts that named the explorer or worker agent.
- The service rebuilt the second snapshot byte for byte.
- The 503 test served a made-up catalogue of two pages, and its second page answered 503.
- All four tests passed in the final run by Codex and in ours.
- Some steps ran differently from the page. The first full test run failed one of four tests, and Codex corrected that test's CSV parsing. While a test command stalled, Codex force-stopped all dotnet processes started after a set time, whichever program had started them. NuGet packages came from a cache restored before the run, because the sandbox had no access to nuget.org.
The Google Antigravity guide also checks crawl output against the page a person sees, through a browser. The Claude Code guide also measures how IronWebScraper drops a failing page, in a blog archive crawl. The shared run record lists the Codex version, model, test machine and timing for this run.
Troubleshooting
- Scrape a Shopping Website if the product selectors or the pagination need a reference
- Advanced Webscraping Features if requests need throttling, identities or a resumable crawl
- Applying License Keys if every crawl stops with a licensing exception before any request is sent
Questions?
If you have any questions, reach out to support@ironsoftware.com
Curtis Chau holds a Bachelor’s degree in Computer Science (Carleton University) and specializes in front-end development with expertise in Node.js, TypeScript, JavaScript, and React. Passionate about crafting intuitive and aesthetically pleasing user interfaces, Curtis enjoys working with modern frameworks and creating well-structured, visually appealing manuals.
