How to Use IronWebScraper with Claude Code
A scraper is easy to write and easy to write badly. The bad version works on the page in front of you, ignores the host it is pointed at, and stops being correct the first time a layout changes. A marketing team that plans to move its company blog to a new platform needs an inventory of every published post first, with the address, title, author, publication date, categories, tags, word count and images of each. The archive covers five years and two site themes, and one post is linked from no archive page. Anthropic built Claude Code as an agent for programming, available from the command line, inside VS Code or JetBrains, as a desktop application, and in the browser. In all of them it reads code, edits files and runs commands, which suits crawler work, where each change to a selector is followed by another run and another look at the output.
IronWebScraper is built around one base class. You extend WebScraper, override Init to request a starting URL and set the crawl rules, and override Parse to pull values out of each response. The queue and the throttling are already in place, so the agent writes only the parts that depend on the site. The blog tutorial shows the pattern on a blog. This guide explains how to configure Claude Code to build the content inventory with four tested prompts, and records the IronWebScraper behaviors that each prompt depends on.
Anthropic documents Claude Fable 5.1 and Claude Fable 5 as the most capable models in Claude Code, suited to tasks larger than a single sitting. The /model command changes the model without ending the session.
Name the fields, the output file and the crawl limits in each prompt, and leave the rest to the agent.
Quickstart
Install with NuGetOpen a terminal in the project folder, run claude, and enter /model opusplan before the first prompt. The test served a copy of the company blog on port 5437 of the local machine, and every prompt names that address.
Add the IronWebScraper 2026.10.1 package to BlogInventory.csproj.
Then make Program.cs a smoke test for the company blog at http://localhost:5437/blog/.
The inventory tool puts the value of IRONWEBSCRAPER_LICENSE_KEY in License.LicenseKey first.
Keep that key private with these rules:
- The key never goes to the console, a log or a file.
- Don't run env, printenv, Get-ChildItem Env: or any other command that lists the environment.
- When you check the variable, answer yes or no to whether it has a value.
After setting the key, Program.cs does these things in order:
1. Prints License.IsLicensed.
2. Runs a WebScraper subclass that requests http://localhost:5437/blog/ and no other page.
3. Prints how many post cards that page lists, and the title of the first card.
Run it once and show me the console output.Minimal Workflow (5 steps)
- Download the C# web scraping library from NuGet
- Open Claude Code from the project folder and set the model with /model
- Record IronWebScraper's measured crawl rules in CLAUDE.md so that every session loads them
- Point the crawler at a local copy of the site before any live host
- Check the inventory against each post and the request log before a migration starts
What does Claude Code do in a .NET crawler project?
The agent writes the WebScraper subclass and runs it against the test blog. The developer decides which site is crawled, how fast, and which pages are excluded.
The project rules in the next sections forbid a hand-written request loop and name the package and its version, so the agent does not pick a package by name alone.
How do you set up Claude Code?
Install Claude Code and sign in by following Anthropic's quickstart. Setup is complete when claude starts in a terminal and responds.
How do you set up a project Claude Code can work in?
Give the agent a real target first. The test fixture is a .NET 9 console application with no package reference, and the blog runs as a local site with 26 posts. Posts from 2023 onward use the current theme, and the nine older posts use the previous one. The archive links one post through a redirect, one post answers with a server error, one older post is sent in the windows-1252 encoding, and robots.txt disallows a drafts folder that the second archive page links.
CLAUDE.md is loaded at the start of each session, so rules written there apply to every later prompt. The second prompt creates the file and a read-only reviewer for the crawler code.
Create CLAUDE.md for this repo, with a Crawling section that later sessions follow:
- IronWebScraper fetches and parses every page. No HttpClient, HtmlAgilityPack or AngleSharp.
- Stay on the IronWebScraper version that BlogInventory.csproj has now.
- The crawler only ever requests http://localhost:5437.
- Check each IronWebScraper member against docs/IronWebScraper.xml before using it.
- Crawl results go under out/.
Copy IronWebScraper.xml into docs/ now.
Your NuGet global packages folder has it in ironwebscraper/2026.10.1/lib/netstandard2.1.
Next, create a subagent at .claude/agents/crawl-reviewer.md.
Give it only the Read, Grep and Glob tools.
It reviews Inventory/ and answers four questions:
- Which IronWebScraper members does the code call, and which are missing from the XML?
- Does anything fetch a page without going through IronWebScraper?
- Which robots.txt rules in CLAUDE.md does the crawler skip?
- Which output calls does CLAUDE.md warn against?
I'm adding IronWebScraper's measured crawl rules to CLAUDE.md before the first review.
Tell me the path you copied the XML from.For guides beyond the member list, IronWebScraper lists its documentation for agents in llms.txt.
Which IronWebScraper crawl rules belong in CLAUDE.md?
Each line comes from a probe that crawled the local blog with and without a license key and compared the site's request log with what Parse received. Paste the block under the Crawling section.
These crawl rules were measured on IronWebScraper 2026.10.1:
- IronWebScraper never reads IRONWEBSCRAPER_LICENSE_KEY on its own.
- Assign License.LicenseKey from that variable before Start.
- With no valid key, Start throws LicensingException before the first request.
- LicensingException is not accessible, so catching it by name fails with CS0122.
- Compare the exception's type name as a string instead.
- Requests carry no User-Agent header until an HttpIdentity sets one.
- IronWebScraper reads robots.txt from port 80 of the host, whatever port the start URL uses.
- On localhost:5437 no rule applies, and the failed lookup delays the first request by 4.5 s.
- Set ObeyRobotsDotTxt to false.
- In Init, read http://localhost:5437/robots.txt with FetchUrlContents.
- Turn each Disallow path into a BannedUrls glob, and Crawl-delay into RateLimitPerHost.
- RateLimitPerHost spaces the starts of requests to one host. Its default is 50 ms.
- OpenConnectionLimitPerHost defaults to 2, so a slow page overlaps the next request.
- A second Request for a URL already seen is dropped. A redirect and its target are two URLs.
- Two Parse calls can then share one Response.FinalUrl, so deduplicate on FinalUrl.
- By default a URL that returns 500 gets 9 requests, because HttpRetryAttempts is 8.
- Parse never sees that page, and FailedUrls counts it.
- Call Start() with no CrawlId. After a finished crawl, Start(CrawlId) parses nothing.
- Scrape appends to its file on every run, so delete the old file first.
- ScrapeUnique drops the first record written to a new file. Don't use it for the inventory.
- A windows-1252 page is decoded as UTF-8, though Response.CharSet says windows-1252.
- Call Encoding.RegisterProvider(CodePagesEncodingProvider.Instance) before Start to fix that.How does IronWebScraper apply robots.txt to a local copy of a site?
ObeyRobotsDotTxt is on by default, but IronWebScraper requested robots.txt from http://localhost/robots.txt, on port 80, whatever port the start URL named. On the test blog it never asked for robots.txt and fetched the disallowed draft. With nothing listening on port 80, the first request also started 4.5 seconds after Start, and setting ObeyRobotsDotTxt to false brought that down to 0.5 seconds.
A second probe served the blog's rules on port 80 as well. With no user agent set, Request threw an ArgumentException with the message "Not a valid user-agent string". With an HttpIdentity that set one, the draft stayed unrequested and requests kept the one-second crawl delay. A BannedUrls glob such as */blog/drafts/* kept the draft out in the probe.
Which pages reach Parse, and how many times is each one requested?
With default settings, the probe crawl sent 43 requests to the test blog. A URL that the code requested twice was fetched once. A redirect and its target counted as two URLs, so the redirected post was fetched twice and reached Parse twice with the same FinalUrl.
The post that answered 500 received 9 requests with the default HttpRetryAttempts of 8, and Parse never ran for it. A crawler that must list its failures therefore compares the links it queued with the rows it wrote.
RateLimitPerHost spaces the start of each request to a host. At 1,000 milliseconds the gaps measured 1.03 to 1.06 seconds, but the post that took 2.5 seconds to answer overlapped two later requests, because OpenConnectionLimitPerHost defaults to 2. IronWebScraper also sent the requests out of page order, so a crawler that needs ordered output sorts its rows before writing them.
What does IronWebScraper write to the output files?
Scrape appends to its JSON Lines file, and a second run into the same working folder doubled the probe's file from 27 lines to 54. ScrapeUnique drops the first record written to a new file, and the first post written was missing from the unique file in all 14 probe crawls.
The windows-1252 post lost its accents. Response.CharSet reported windows-1252, yet the text was decoded as UTF-8, and the author's name arrived with replacement characters in place of the accented letters. Posts by the same author sent as UTF-8 decoded correctly. Registering CodePagesEncodingProvider with Encoding.RegisterProvider before Start fixed the decoding.
The windows-1252 post looks like this in a browser. The inventory has to keep the accented letters in the author name.
The windows-1252 post on the local test blog, as Chrome displayed it. The site and its posts were built for the test.
How do you prompt for IronWebScraper code that compiles?
Request the crawler and its output rules in one prompt, and allow plan mode to settle the design before any file changes. Switch to plan mode with Shift+Tab before sending the prompt. With opusplan selected, Opus writes the plan and Sonnet carries it out once the plan is approved.
Build the inventory crawler in Inventory/BlogInventoryScraper.cs, and have Program.cs run it.
It starts at http://localhost:5437/blog/ and crawls by these rules:
- Follow the Older posts links and every category page.
- One post is on no archive page, but its category page links to it.
- Apply the site's robots.txt the way CLAUDE.md describes.
- Send LarkspurInventory/1.0 as the user agent.
- Give up on a failing page after two retries.
- Posts from 2021 and 2022 use the old theme, so read both layouts.
Write out/content-inventory.csv with one row per post, in this shape:
- The columns are Url, Title, Author, Published, Layout, Categories, Tags, Words and Images.
- Url is where the post was served from, after any redirect.
- Published is yyyy-MM-dd, and Layout is current or old.
- Categories and Tags are joined with semicolons. Old-theme posts have no tags.
- Words is the number of whitespace-separated words in the body.
- Images counts the img tags in the body.
- Rows go newest first, and each run replaces the file.
Put every post URL that answered with an error in out/failed-pages.txt, one URL per line.
Run it, then tell me the row count and which pages failed.
After that, have the crawl-reviewer subagent review Inventory/ and give me its findings.Two parts of the prompt protect the inventory.
- One row per served address: the inventory feeds a migration, and a post listed under two addresses would be moved twice.
- Failures in a separate file: without it, a post missing from the inventory cannot be told apart from a post that was never published.
What happens when the blog changes its post template?
The test switched the local blog to a new post template between the third and fourth prompts. Archive pages and older posts kept their markup, while every post from 2023 onward moved to new class names. A path that starts with @ attaches that file to the request.
The blog moved its posts to a new template this morning.
Check the crawler against it with these steps:
1. Copy @out/content-inventory.csv to before-template.csv in the repo root, and keep it unchanged.
2. Run the crawler again and compare the new inventory with the copy.
3. Fix the crawler so it reads the old theme, the current theme and the new template.
4. Repeat steps 2 and 3 until every row matches before-template.csv, at most three times.
Posts in the new template still get Layout current.
Tell me which fields the new template broke, what you changed, and anything that still differs.What should you verify before trusting generated crawler code?
Include the following checks in the review.
- The request log: count the requests per page, confirm that nothing under a disallowed path was fetched, and measure the gaps between requests.
- The build:
dotnet buildrejects a member that does not exist, such as a corrected spelling ofResponse.RequestlUrl. - A sample of rows: open several posts from each theme and compare their fields with the inventory before the migration starts.
For a complete IronWebScraper task with fixed pass criteria that can be rerun, see the AI coding assistants guide.
What did the prompts produce in a real run?
Claude Code ran the four prompts in one session, in order, against the local blog. The rules block went into CLAUDE.md before the third prompt. No code was edited by hand.
- Before the run, the repository held a one-line Program.cs and no package reference.
- The Quickstart program printed True for IsLicensed, counted 10 post cards and named the first post correctly.
- CLAUDE.md received the five Crawling rules, and the subagent file listed only the Read, Grep and Glob tools.
- While planning, Claude Code sent 239 PowerShell requests to the blog, and 6 of them went to the drafts folder that robots.txt disallows.
- The crawler set ObeyRobotsDotTxt to false, read robots.txt itself and converted its rules into BannedUrls and RateLimitPerHost settings.
- Each complete crawl sent 38 requests, all with the LarkspurInventory/1.0 user agent and none to the drafts folder.
- In the third prompt's crawl, the median gap between request starts was 1.05 seconds, and the shorter gaps followed only robots.txt or a redirect.
- The failing post received 3 requests in each complete crawl and was the only entry in failed-pages.txt.
- After the third prompt, the inventory held 25 rows, and every field matched the blog's own data in our check.
- The redirected post appeared once, under its final address, and the post missing from the archive pages was present.
- The windows-1252 post kept its accented author name, and no row contained a replacement character.
- Against the new template, the third prompt's crawler wrote 8 rows, still listed only the failing post and exited with code 0.
- After one fix in the fourth prompt, the new inventory was byte-identical to the copy taken before the template change.
- Fresh clones of the final code produced the same 25 rows on the current theme and on the new template.
- Without the key, the program reported that no valid license was found and sent no request to the blog.
- The license key appeared in no transcript, session file, log or repository file from this run.
- Some steps ran differently from the page. The session ran headless, so a typed reply approved the plan. The run's guard, which kept the agent inside the repository, blocked the plan file in the user profile. Two faults in the guard also blocked two planning subagents, and the main session did their work itself. A process outside the run stopped the first crawl after 22 requests, and the agent started it again.
The last eight rows of the inventory hold the posts in the old theme.
The header and the last eight rows of content-inventory.csv from the test run, drawn as text from the file. The line numbers are those of the file.
The Cursor guide also has the agent plan the work before it writes code, as plan mode does here. The ChatGPT Codex guide keeps its crawl rules in AGENTS.md, which Codex reads before any work, as this guide does with CLAUDE.md. The Claude Code version, models, timings and test machine for this run are in the shared run record.
Troubleshooting
- Applying License Keys if
Startthrows LicensingException - Thread Safety for the settings that control how many requests run at once
- IronWebScraper Security CVE for the security position on the library itself
Questions?
If you have any questions, reach out to support@ironsoftware.com
Curtis Chau holds a Bachelor’s degree in Computer Science (Carleton University) and specializes in front-end development with expertise in Node.js, TypeScript, JavaScript, and React. Passionate about crafting intuitive and aesthetically pleasing user interfaces, Curtis enjoys working with modern frameworks and creating well-structured, visually appealing manuals.
