Using IronWebScraper with AI Coding Assistants
IronWebScraper is Iron Software's web scraping library for .NET. It crawls websites, follows links between pages and extracts structured data from HTML in C#, and a single package covers every platform that runs .NET Standard 2.0.
A coding assistant can produce a .NET web scraper from a single request. Whether that code respects the site it targets, and whether it still works on the second page, is what requires checking. An assistant working from memory can write a raw HttpClient loop, add its own retry logic, and send requests as fast as the connection allows. IronWebScraper already manages the request queue, the throttling and the identity, so the code an assistant needs to write is one class with two overrides. With the documentation available, the assistant in this guide's test run wrote one WebScraper subclass and no HttpClient call.
The setup process is the same for every assistant. Add the package, assign the key from the environment, provide the assistant with IronWebScraper's machine-readable documentation, and compare the crawler's output and its requests with the site before marking the task complete. This applies to the basic scraping tutorial, blog scraping, shopping site scraping, and the advanced settings.
How to Use IronWebScraper with an AI Coding Assistant
- Download the C# web scraping library from NuGet
- Assign
IronWebScraper.License.LicenseKeyfromIRONWEBSCRAPER_LICENSE_KEYat startup - Give the assistant IronWebScraper's llms.txt index before it writes code
- Record the crawl rules in the instruction file your assistant loads
- Set the user agent, rate limit and retry count before the first request
- Compare the output file and the site's request log with the pages
Can't My AI Assistant Write a Scraper on Its Own?
It can write one. The difficulties appear outside the program, in what the target site receives and in what the output file holds.
- A scraper that runs can still overload a site:
RateLimitPerHostsets the minimum delay between requests to one host, andOpenConnectionLimitPerHostlimits parallel connections. - HTML selection does not need regular expressions:
ResponseprovidesCss,XPath,QuerySelectorandGetElementById. - Name the package exactly: the package and the namespace are both
IronWebScraper. Attackers register package names that an assistant can produce by mistake, a practice called slopsquatting. - Licensing terms are defined in advance: the licensing page lists the number of developers each tier allows.
Step 1: Install IronWebScraper
Add the IronWebScraper NuGet package, version 2026.10.1, to this project and restore it.
Tell me which IronWebScraper version the project file lists now.From a terminal, pin the release explicitly.
dotnet add package IronWebScraper --version 2026.10.1The package contains no browser engine and no native binaries. Its only dependencies are IronSoftware.Common and IronSoftware.System.Drawing, and the second brings in SixLabors.ImageSharp 3.1.12. For that release, dotnet list package --vulnerable --include-transitive reported five advisories, three of them rated high.
Step 2: Apply Your License Key
Without a key, Start throws IronSoftware.Exceptions.LicensingException before the crawler sends a single request, and nothing reaches the output folder. The message mentions seven days of free development use, yet the exception appeared on the first call in the probe for this article. A mistyped key causes a "not recognized" message, which did not repeat the key. A free trial key is available below.
Start using IronWebScraper in your project today with a free trial.
Assign the license key once at application startup, before the first crawl. Retrieve it from an environment variable so it is not stored in the repository.
IronWebScraper.License.LicenseKey = Environment.GetEnvironmentVariable("IRONWEBSCRAPER_LICENSE_KEY");IronWebScraper.License.IsLicensed returns true after a valid key is assigned, and IronWebScraper.License.IsValidLicense tests a key string without applying it. Without the assignment above, IronWebScraper ignores IRONWEBSCRAPER_LICENSE_KEY. It does read an environment variable named IronWebScraper.LicenseKey, IronWebScraper_LicenseKey or IronSuite.LicenseKey with no code at all.
Step 3: Point Your Assistant at IronWebScraper's LLM Documentation
- llms.txt lists 17 links, each with a one-line summary.
- llms-full.txt summarizes 13 documentation pages in a single file.
- llms-full.txt describes llms-full-catalog.txt as an index that also covers blog articles, but on October 17, 2026, the file was identical to llms.txt.
- The API Reference lists the signature of each member.
- Tools that support skill files can use skill.md, which covers setup, licensing, recipes and common failures.
- The AI Product Fact Sheet outlines supported platforms and primary use cases.
Provide the index at the start of every session, along with a clear instruction.
Read https://ironsoftware.com/webscraper/csharp/llms.txt before you touch the scraper code again.
Keep to what the pages it links describe.
An IronWebScraper crawl is built like this:
- A scraper class extends WebScraper and overrides Init and Parse.
- Init sets the crawl options and queues the first URL with Request.
- Parse gets a Response, selects nodes with Css or XPath, and queues more pages.
- Scrape appends one JSON line per record to a file under WorkingDirectory.
- Start runs the crawl and returns once the queue is empty.
Response.RequestlUrl has that extra l in the real API, so don't correct it.
For a member those pages don't mention, follow these steps:
1. Look it up at https://ironsoftware.com/webscraper/csharp/object-reference/api/.
2. Compile once to show it exists.
3. If you're still unsure, tell me and don't guess.
Then list every IronWebScraper call here that none of those pages covers.The property spelled RequestlUrl is the real member, and the assembly has no RequestUrl. An assistant that corrects the spelling produces code that does not compile.
Step 4: Add IronWebScraper Context to Your Project
Instructions entered in chat are lost after the session ends. Store the crawl rules in the project file your team uses. Claude Code reads CLAUDE.md, ChatGPT Codex reads AGENTS.md, GitHub Copilot reads .github/copilot-instructions.md, Cursor reads .mdc files in .cursor/rules/ or AGENTS.md, and Google Antigravity reads AGENTS.md or GEMINI.md. A single block of text is compatible with all, though Cursor rule files also include MDC frontmatter. Begin with the following.
Crawlers in this project are written with IronWebScraper and follow these rules:
- The NuGet package and the namespace share one name, `IronWebScraper`.
- The documentation index is https://ironsoftware.com/webscraper/csharp/llms.txt.
- If the tool supports skill files, use https://ironsoftware.com/webscraper/csharp/skill.md too.
- Startup code assigns `IronWebScraper.License.LicenseKey` before any crawl runs.
- Its value is looked up by name in IRONWEBSCRAPER_LICENSE_KEY, and nothing prints the environment.
- The key stays out of console output, log files and commits.
- A scraper is a subclass of `WebScraper` with `Init` and `Parse` overridden.
- No raw `HttpClient` loop fetches pages beside it.
- Pick content out of the `Response` with `Css` or `XPath`, never with regular expressions.
- Add an `HttpIdentity` whose `UserAgent` names the crawler. Without one, no user agent is sent.
- Set `RateLimitPerHost`, `OpenConnectionLimitPerHost` and `HttpRetryAttempts` in `Init`.
- Never lower those limits to speed up a crawl.
- `ObeyRobotsDotTxt` reads robots.txt from the default port only.
- For a site on another port, add its disallowed paths to `BannedUrls`.
- A page that keeps failing never reaches `Parse`. Override `Log` to record which URL failed.
- Each crawl adds lines to the file `Scrape` wrote last time. Delete it before a new crawl.
- `ScrapeUnique` drops the first record it writes to a file.
- Write records with `Scrape` and remove duplicates by id.
- Develop against a local copy of the site before pointing at a live host.The skill file is outdated in three areas. Its single-file example pins IronWebScraper 2026.8.1, it presents ObeyRobotsDotTxt as honoring robots.txt paths and rates, and it describes ScrapeUnique as a de-duplicated append. The probes for 2026.10.1 contradicted the last two.
Step 5: One-Shot Setup Prompt
Steps 1 to 4 can be sent together as one message at the start of a project or session.
Set this project up for IronWebScraper crawls with these rules:
- Keep the IronWebScraper package pinned at 2026.10.1.
- The app fills IronWebScraper.License.LicenseKey before the first crawl.
- It looks up IRONWEBSCRAPER_LICENSE_KEY itself and never enumerates the environment.
- No console line, log or committed file ever contains the key.
- Start from https://ironsoftware.com/webscraper/csharp/llms.txt and keep the code within it.
- Look up anything those pages miss in the API reference that llms.txt links.
- A tool that reads skill files also loads the agent skill listed there.
- Every scraper sets a user agent, a rate limit and a retry count before its first request.
- Nobody removes those settings to make a crawl finish sooner.
- Test selectors on a local copy of a site first.
Save these rules in the file your later sessions load.
Tell me that file's name and list any other change you made.Step 6: Prompt Patterns That Produce Working Code
Name the start page, the pages to follow, the fields and the output file.
Vague Write a scraper for an events site with IronWebScraper.
Better Extend
WebScraper. InInit, add anHttpIdentitywith a named user agent, set a one-secondRateLimitPerHost, and request the first listing page. InParse, follow the Next link, request each event page with a second parse method, and write one record per event withScrapeto events.jsonl.
State the IronWebScraper release and the target framework.
Better Use IronWebScraper 2026.10.1 on .NET 9, and assign the key from IRONWEBSCRAPER_LICENSE_KEY before
Start.
Request complete files. A scraper is a subclass, so a partial snippet can omit the class declaration or the using IronWebScraper; directive.
Better Return the whole Program.cs and the scraper class, with their using directives and the license line, ready to build as is.
Step 7: Verify What You Get Back
Each of the following mistakes is easy to miss in a review of generated code.
- A request loop beside the library: a page fetched with
HttpClientbypasses every throttling setting, so the generated code should contain no such call. - A rate limit removed to make a test finish: check for the setting in
Init, before the firstRequest. - A selector that starts with an attribute:
Css("[data-event-id]")threw XPathException in 2026.10.1, whileCss("article[data-event-id]")returned the element. Put a tag name before each attribute test.
Let the Agent Verify Its Own Work
Rebuild in Release from clean and remove the previous crawl's output files.
Keep events.run1.json, then run the crawl again.
Show me what that run proves:
- Every IronWebScraper class and member used, with its documentation page.
- Whether IronWebScraper.License.IsLicensed came back true.
- The user agent, rate limit, connection limit and retry count set in Init.
- How many records events.json has, plus every event link without one and the reason.
Then tell me whether events.json holds every event the crawl was meant to collect.Crawl defaults with IronWebScraper
The probes for this article crawled a local site on port 5467 with each default left in place.
| Setting | Default in 2026.10.1 | Effect measured on the test site |
|---|---|---|
ObeyRobotsDotTxt | true | No robots.txt request reached the site, and a disallowed page was fetched |
RateLimitPerHost | 50 milliseconds | The 42 requests of a full crawl arrived within 2.55 seconds |
OpenConnectionLimitPerHost | 2 | All 42 requests arrived over one connection |
HttpRetryAttempts | 8 | Nine requests for each page that failed |
User agent (Identities empty) | none | Every request carried an empty User-Agent header |
WorkingDirectory | Scrape | Output files appeared in a Scrape folder under the current directory |
IronWebScraper builds the robots.txt address from the scheme and the host name and leaves out the port. For the test site, the crawler tried port 80 on both loopback addresses, waited 4.1 seconds for those connections to fail, and then crawled with no rules. Its log still reported that robots.txt would be obeyed for the host. A site on the default port is not affected. For a site on any other port, add the disallowed paths to BannedUrls, which kept the members-only page out of the crawl in the probe.
Failed and skipped pages with IronWebScraper
IronWebScraper handles a page that answers 404 and a page that answers 500 the same way.
- With
HttpRetryAttemptsset to 2, each failing page was requested three times, and the log line still reported "after 2 attempts". - A setting of 0 produced one request for each failing page.
Parsewas never called for a page that failed, so the scraper never sees its status code.- The URL appears only in the critical log line "Url failed permanently", and
FailedUrlscounts it. An override ofLogcan record which pages failed. - An exception thrown inside a parse method has the same effect as a failed request. The page was requested nine times, logged as failed, and the crawl continued.
IronWebScraper 2026.10.1 also skips some pages requested in Init without recording a failure. In the probes for these guides, a crawler that requested three venue pages in Init fetched all three in 0 of 8 runs. A crawler that queued 10 pages there requested between 2 and 9 of them in 8 runs. The skipped pages never reached the site, FailedUrls stayed at 0, and the log still reported "Crawl Complete". Requesting the first page in Init and queuing the rest from a parse method fetched every venue page in each of the 8 runs.
Output files with IronWebScraper
Scrape serializes any object to one line of JSON and appends it to a JSON Lines file. A relative file name resolves against WorkingDirectory, and a call without a file name writes to one named after the type, such as IronWebScraper.ScrapedData.jsonl.
- The file is UTF-8 without a byte order mark, with CRLF line endings. Accented titles were written as the characters themselves.
- A
DateTimeOffsetvalue kept its offset, and a decimal price of 12.50 kept both digits. Scrapeappends on every run. A second crawl into the same folder doubles the file unless the program deletes it first.ScrapeUniqueskips a record whose whole JSON line was already written, so the same event with a changed price is written again.ScrapeUniquealso drops the first record it writes to each file in each scraper instance. Writing A, B, C and A to a new file left B and C. A single call never created the file, and in a full crawl the first event parsed was missing from the output.
A Result You Can Reproduce
The test consists of one prompt, and you write none of the code. Serve the test site, open a new session in an empty folder, paste the prompt, and compare the outcome with the pass criteria.
IRONWEBSCRAPER_LICENSE_KEY, and the test site on port 5467. gen_site.py writes its pages and serve.py serves them on the loopback addresses and logs each request.The site lists 23 distinct event links on three pages. Twenty lead to event pages, robots.txt disallows one, one answers 404 and one answers 500. One event appears both in a featured box and in the list.
The prompt
Make a .NET 9 console program that exports a local test site's event listings to JSON.
If this folder already holds project files, say so and stop.
Don't ask me anything.
Write all output here.
Add the NuGet package IronWebScraper 2026.10.1.
Go through https://ironsoftware.com/webscraper/csharp/llms.txt, then its agent skill.
Also read its license keys page and the tutorial /tutorials/webscraping-in-c-sharp/.
Call only members documented there or in the API reference.
The events app's first action sets IronWebScraper.License.LicenseKey.
It reads IRONWEBSCRAPER_LICENSE_KEY by name for that value.
Never let the key reach the console, files, shell echoes or environment variable listings.
If IronWebScraper.License.IsLicensed is false, the app exits with a message.
The site's listings start at http://localhost:5467/events/.
Follow the Next link through every listing page and open each event link.
The IronWebScraper crawler keeps to these limits:
- It requests only http://localhost:5467/ addresses.
- It requests no path that http://localhost:5467/robots.txt disallows.
- It sends the user agent RiversideEventsBot/1.0 on every request.
- It waits at least one second between requests, over one connection at a time.
- It retries a failed page at most twice.
Write a record per loaded event page with IronWebScraper's Scrape or ScrapeUnique.
events.json holds those records as a UTF-8 JSON array sorted by start time.
Each record has these fields:
- id, from the data-event-id attribute.
- title, with its accented letters intact.
- start, the time element's datetime value, unchanged.
- venue.
- price as a number, null when the page says Free.
- tags as an array.
- status in lower case, "scheduled" when the page shows none.
- url, the page's full address.
Also write these files:
- links.json lists every distinct a.event-link target on the listing pages.
- skipped.json gives each failed or disallowed link and its reason.
- requests.log has every URL the crawler requests, with a timestamp.
Each link in links.json ends as a record, a failed request, or a path robots.txt disallows.
Next, do a Release build and run it twice.
Keep the first run's file as events.run1.json and both runs' SHA-256 values.
The second run's events.json must match events.run1.json byte for byte.
Last, add a check mode that sends no requests, and run it to write readback.md.
It reads each actual value from disk.
readback.md gives expected, actual and match for these checks:
- Records plus skipped links equal the distinct event links found.
- No two records share an id.
- Records are in start order, and every start keeps its UTC offset.
- requests.log has no disallowed path, no other host, and no gap under one second.
- The two SHA-256 values are equal.
readback.md names any request you sent the site outside the crawl, and why.
When a check fails, find the cause, fix it and rerun.
Some causes are inside IronWebScraper, with no workaround.
For those, readback.md names the member, what you tried and the unmatched check.
Then finish the rest.
Tell me the Release build result, matching checks out of the total, and your assumptions.Output
The app's unedited readback.md lists its five checks, and events.json and skipped.json are available as the app wrote them. The prompt was not edited after the run. The figures below come from an independent reading of those files and of the test site's log.
Lines 104 to 129 of events.json as the app wrote it, drawn as plain text.
- A Release build for net9.0 completed with no warnings and no errors.
- events.json held 20 records, one for each event page, sorted by start time.
- Every field of every record matched its event page. The accented titles and the UTC offset of each start time were preserved.
- The crawler requested robots.txt itself as its first request and rejected the disallowed paths in an override of AcceptUrl.
- No crawl that the app sent requested a disallowed path.
- Before the app had its robots.txt rules, the session fetched the members-only page three times by hand.
- Each of the two Release crawls sent 30 requests with the user agent RiversideEventsBot/1.0.
- The shortest gap between requests in those crawls was 1.028 seconds.
- Each failing page received three requests, the first attempt and two retries.
- The app's notes recorded that ScrapeUnique lost the first record, and the final code wrote every record with Scrape.
- events.json was byte for byte the same after the app's two runs and after two runs of a separate rebuild.
- The app's readback matched 5 of 5 checks.
- The independent reading matched all 37 of its checks.
- readback.md listed 165 requests sent outside the two Release crawls, and the test site's log recorded 195.
- No file in the project folder, its Git history or the Claude Code session files contained the license key.
- The setup prompt wrote its rules to a new CLAUDE.md of 112 lines.
- Some steps ran differently from the page. The Step 1, 3, 5 and 7 prompts ran after the acceptance prompt in the same session. The test setup blocked seven valid commands, and the session completed each task another way.
What passes
- A Release build completes without any manual edits.
- events.json holds 20 records sorted by start time, with every field matching its event page when you read the pages yourself.
- skipped.json names the members-only page and the two links that failed.
- The site's own request log shows no disallowed path, no gap under one second between crawler requests, and at most three requests for a failing page.
- A second run produces an identical events.json.
- A search of the project folder finds no file that contains the license key.
The run record
- All runs used one Windows 11 x64 desktop with a Core i9-14900KF and 64 GB of memory.
- This guide's project targets net9.0 and pins
IronWebScraper2026.10.1, and every crawl targeted a test site served on the same machine. - A guide's duration is the sum of the logged times for the prompts on its page, and this guide's row counts only the acceptance prompt.
- Last tested October 17, 2026, for every guide in the table.
| Guide | Assistant and version | Model | Duration (minutes) |
|---|---|---|---|
| This guide | Claude Code 2.1.277, one session, permission prompts off | Claude Opus 5 | 23.7 |
| Claude Code | Claude Code 2.1.277, prompt 3 in plan mode | Sonnet 5 with Opus 5 planning | 31.7 |
| ChatGPT Codex | Codex CLI 0.160.0 | GPT-6 Luna | 27.8 |
| GitHub Copilot | Copilot CLI 1.0.95 | GPT-6 Luna, picked by Auto | 18.9 |
| Cursor | Cursor CLI 2026.10.16, headless | Composer 2.5 | 17.0 |
| Google Antigravity | Antigravity CLI 1.4.0, permission prompts skipped | Gemini 4 Argon (Medium) | 109.9 |
Agent-Specific Guides
Each guide below applies these steps to one tool and one scraping task.
- Use IronWebScraper with Claude Code for turning a company blog archive into a content inventory
- Use IronWebScraper with ChatGPT Codex for price monitoring across a paginated product catalogue
- Use IronWebScraper with GitHub Copilot for a cinema showtimes aggregator
- Use IronWebScraper with Cursor for moving a supplier directory crawler off HttpClient and HtmlAgilityPack
- Use IronWebScraper with Google Antigravity for a job board crawler that resumes after a stop
Troubleshooting
- IronWebScraper License Keys if
Startthrows a licensing exception - IronWebScraper and JavaScript if a page looks empty because its content arrives after load
- Thread Safety if parallel parsing produces results that do not add up
Questions?
If you have any questions, reach out to support@ironsoftware.com
Curtis Chau holds a Bachelor’s degree in Computer Science (Carleton University) and specializes in front-end development with expertise in Node.js, TypeScript, JavaScript, and React. Passionate about crafting intuitive and aesthetically pleasing user interfaces, Curtis enjoys working with modern frameworks and creating well-structured, visually appealing manuals.
