How to Use IronWebScraper with Cursor
When a C# crawler moves from HttpClient and HtmlAgilityPack to IronWebScraper, a port that queues every listing page in Init loses records. IronWebScraper 2026.10.1 can end such a crawl before it has requested every queued page. In our tests, four listing pages queued in Init produced three, three and two fetched pages in three runs, and the output lost 10, 10 and 24 of 46 suppliers without an error.
Cursor is an AI code editor built on the VS Code codebase, with an agent that edits files and runs commands. This guide uses two of its features, the documentation links the agent reads and the project rules committed in .cursor/rules, to migrate a purchasing team's supplier-directory crawler. The crawler copies 46 suppliers and their members-only contact details from a trade directory into a CSV file, and the guide moves it to IronWebScraper without changing a row of that file or exceeding the directory's limit of 4 requests per second.
The guide follows the throttling and identity features of the advanced web scraping tutorial. Every crawl ran against a local copy of the directory served on localhost. Read a public site's terms and obtain permission before writing a crawler for it.
Quickstart
Install with NuGetWith the HarborDirectory repository open, paste this prompt into a new Agent chat.
@src/HarborDirectory/ @README.md
Add a pages command that checks the supplier listing with IronWebScraper.
Invoke it as `dotnet run --project src/HarborDirectory -- pages`.
Stay on IronWebScraper 2026.10.1.
Start with https://ironsoftware.com/webscraper/csharp/llms.txt.
After that, read the advanced tutorial linked from there.
The pages command walks the listing as follows:
- It starts at /directory?page=1 on the test site and follows each Next page link.
- For every listing page, it prints the page number and the number of suppliers on that page.
The directory crawler already loads its key from IRONWEBSCRAPER_LICENSE_KEY in Program.cs.
Don't write the key to any output or file, and don't dump environment variables.
When it builds, run the pages command and show me what it printed.Do not paste the license key into a chat, because pasted content can appear in a source file, a commit or a build log. IronWebScraper 2026.10.1 ignores an environment variable named IRONWEBSCRAPER_LICENSE_KEY, so the repository's Program.cs copies that variable into License.LicenseKey. The license key guide lists the places where the key can be stored.
Minimal Workflow (5 steps)
- Download the C# web scraping library from NuGet
- Add a project rule under .cursor/rules that contains IronWebScraper's documentation link
- Record the site's request limit and the Init queuing rule in that rule
- Reference the crawler code and the expected output in each request
- Compare every run with the site's request log, then confirm each member name in the API reference
Which Cursor model should you use for IronWebScraper work?
Composer is Cursor's own agentic model. The model picker also offers SpaceXAI's Grok models, which Cursor co-developed, and models from OpenAI, Anthropic, Google and other labs. The test run below used Composer 2.5.
How do you give Cursor the IronWebScraper documentation?
Include IronWebScraper's llms.txt link in the request, or store it in a rule that the whole team shares. As of October 2026, Cursor's @ menu attaches files, folders, terminals, chats, Git diffs and the browser, so IronWebScraper's documentation reaches the agent as a link.
The advanced tutorial sets RateLimitPerHost in its examples, but it states no default values, and no page says whether two scrapers share a rate limit.
@src/HarborDirectory/Legacy/PoliteHttp.cs
Answer these five questions about IronWebScraper without editing any files:
1. Which members replace the 500 ms wait that PoliteHttp keeps between requests?
2. What delay and connection limit per host does a new WebScraper use?
3. When a scraper holds several HttpIdentity objects, which one sends a given request?
4. Can an HttpIdentity send a fixed Authorization header, as PoliteHttp does?
5. Do two scrapers running at once against one host share a rate limit?
Answer only using https://ironsoftware.com/webscraper/csharp/llms.txt, plus the pages linked there.
Cite the source page for each answer.
If the docs don't cover a question, write "not documented" and don't guess.How do you write project rules for IronWebScraper crawler code?
A project rule is an .mdc file stored in .cursor/rules and versioned with the repository. Cursor ignores a plain .md file in that folder, because a file without frontmatter gives it no way to decide when to load the rule.
The rule below lists its file patterns in globs, separated by commas as Cursor's rules page specifies, and sets alwaysApply to false. Cursor then attaches it when a file under those paths is in context. Save it as .cursor/rules/harbor-crawlers.mdc.
---
description: IronWebScraper conventions for the Harbor directory crawlers
globs: src/HarborDirectory/**/*.cs, tests/HarborDirectory.Tests/**/*.cs
alwaysApply: false
---
Every scraper in this repository keeps to these rules:
- Fetch and parse pages with IronWebScraper 2026.10.1 only.
- Don't add HttpClient calls or another HTML parser.
- Check IronWebScraper members against https://ironsoftware.com/webscraper/csharp/llms.txt.
- The site at http://localhost:5493 allows 4 requests per second across all scrapers combined.
- Each scraper sets RateLimitPerHost and OpenConnectionLimitPerHost in Init.
- Queue only the first listing page in Init, then follow the Next page link from Parse.
- A login set with NetworkUsername sends each member page twice, once without it and once with it.
- A request that keeps failing never reaches Parse, so check how many records a run wrote.
- When the docs don't list a member, tell me which pages you checked.The rules on Init, the login and failing requests record IronWebScraper behavior that the sections below measure.
How do HttpClient and HtmlAgilityPack calls map to IronWebScraper?
The old crawler fetched each page itself and waited between requests. IronWebScraper queues the requests, spaces them per host and calls a parse method for each successful response.
| HttpClient and HtmlAgilityPack | IronWebScraper |
|---|---|
A loop over page numbers with GetAsync | Request for page 1 in Init, then the Next page link queued from Parse |
Task.Delay between requests | RateLimitPerHost, with OpenConnectionLimitPerHost for parallel requests |
DefaultRequestHeaders.UserAgent | UserAgent on an HttpIdentity added to Identities |
DefaultRequestHeaders.Authorization with Basic credentials | NetworkUsername and NetworkPassword on an identity, or HttpIdentity.DefaultHttpRequestHeaders |
| A retry loop that waits for Retry-After on 429 and 503 | HttpRetryAttempts |
SelectNodes and SelectSingleNode with XPath | Css or XPath on the Response, which return HtmlNode arrays |
File.WriteAllTextAsync for the CSV | Scrape, which appends JSON lines under WorkingDirectory, or the program's own writer |
Why does a ported crawler miss listing pages?
The GitHub Copilot guide measures how IronWebScraper drops requests made in Init. In the probes for this guide, FailedUrls stayed at 0 for the dropped pages. The overload of Request that takes a list of URLs fetched two, four and three of the four listing pages in three runs.
One request in Init and every further request queued from a parse method fetched all 46 suppliers in 8 of 8 runs, which is why the rule limits Init to page 1 of the listing.
How does IronWebScraper space its requests to one host?
A new WebScraper used a RateLimitPerHost of 50 ms, an OpenConnectionLimitPerHost of 2, a MaxHttpConnectionLimit of 10, 8 for HttpRetryAttempts and a 20-second HttpTimeOut. With those values the crawler sent the 50 directory requests between 50 and 88 ms apart.
Against a test site that refused requests arriving too quickly with 429 and a Retry-After header, the defaults failed. Each refused URL was requested 9 times at the same spacing, the Retry-After header made no difference, and 13 URLs were abandoned. The old crawler honored Retry-After, so a port that relies on retries alone loses records. A RateLimitPerHost of 500 ms produced gaps of 533 to 563 ms, which keeps one scraper within the directory's limit.
How do you plan and run the migration?
Press Shift+Tab in the chat input until Plan Mode is selected, then send the request below. Cursor writes the plan as an editable Markdown file.
@src/HarborDirectory/Legacy/ @TestData/expected/suppliers.csv
Plan the move of DirectoryCrawler from HttpClient and HtmlAgilityPack to IronWebScraper.
Leave CertificateScraper out of this change.
Don't write code until I approve the plan.
The plan should answer these questions:
- What replaces the wait, the retry, the user agent and the login in PoliteHttp?
- How does the new scraper reach all four listing pages?
- How does it stay within the request limit in the harbor-crawlers rule?
- How does the suppliers command keep its output file and its column order?
List your assumptions at the end of the plan.Correct any errors in the plan. Then build it from the plan, or switch back to Agent and send this request.
@harbor-crawlers
The plan is approved. Build it in this order:
1. Run the current suppliers command.
2. Count the rows of out/suppliers.csv that match TestData/expected/suppliers.csv.
3. Replace DirectoryCrawler and PoliteHttp with one IronWebScraper class named SupplierScraper.
4. Remove HtmlAgilityPack from the project file.
5. Build, run the suppliers command again and count the matching rows.
Then tell me both match counts, how long each run took and how many requests the new run sent.The @harbor-crawlers mention attaches the rule by name.
Which Cursor feature fits each change?
Select a feature based on the scope of the change. Tab completes an edit that the developer has started, and inline edit (Ctrl+K) changes a selection that one sentence can describe. Agent covers changes across several files or work that requires a crawl, and Plan Mode is worthwhile when an incorrect change costs more to undo than a plan costs to write. Every request in this guide went to Agent or Plan Mode.
Which identity sends each request?
Each identity in Identities carries its own user agent, cookies and login. With no identity, IronWebScraper sent no User-Agent header at all. With several identities, it picked one at random for every request and for every retry. A scraper that held a member identity and a public identity needed 140 requests for 46 contact pages, 94 of them refused with 401.
A login set through NetworkUsername and NetworkPassword costs a second request on every protected page. Each member page went out first without credentials, received a 401, and the credentialed request followed within 1.4 ms, outside the RateLimitPerHost spacing. Passing the member identity to Request removes the random choice but not the second request.
The tutorial lists HttpRequestHeaders for custom headers, yet no header placed there reached the site. The probe added Authorization and a custom X-Crawler header, and neither arrived. A header added to the static HttpIdentity.DefaultHttpRequestHeaders before the identity was created was sent on every request and removed every 401. That dictionary applies to every identity the process creates afterwards, public pages included.
How do you debug the scraper that already uses IronWebScraper?
The certificate scraper already used IronWebScraper before the migration. It held a public identity and a member identity, queued the three register pages in Init and set no rate limit.
@src/HarborDirectory/Scrapers/CertificateScraper.cs
The certificates command should write 32 certificates, but it writes far fewer.
The number it writes also varies between runs.
Find the cause before you change anything:
1. Run the command twice and note both counts.
2. Raise the scraper's LoggingLevel and read what happened to the missing certificates.
3. Explain each cause you find.
Then fix CertificateScraper.
Run the command three times and give me the count from each run.Output
The fixed scraper writes the register as one JSON line per certificate, in the order the pages are parsed.
The first 10 of the 32 lines in out/certificates.jsonl as the test run left it after the crawl-all prompt, rendered as plain text from the file.
What happens when two scrapers crawl the same host?
Each WebScraper instance throttles its own requests. Two scrapers that each set RateLimitPerHost to 500 ms and used one connection kept two requests in flight and sent up to 6 requests in one second to the same host. Their combined rate therefore has to be planned in the code.
@src/HarborDirectory/
Add a crawl-all command that runs SupplierScraper and CertificateScraper at the same time.
The command is done when it meets these conditions:
- It writes the same suppliers.csv and certificates.jsonl as the two separate commands.
- Neither scraper gets a 429.
- It prints each scraper's request count and the total run time.
Run crawl-all, compare both files with TestData/expected and tell me what matched.Output
The chart counts the requests the local site received in each second of the crawl-all run.
Requests per second during the crawl-all run of the test, drawn from the local site's request log. 209 requests in 100 seconds.
How do you write regression tests for the scrapers?
A Response built with its public constructor threw an ArgumentNullException from Css, and the ChatGPT Codex guide explains why IronWebScraper parses only pages that it fetched itself. The tests therefore serve the saved pages over HTTP.
@src/HarborDirectory/ @TestData/pages/
Create an xUnit project at tests/HarborDirectory.Tests and add it to HarborDirectory.slnx.
The tests must not call the Harbor site on port 5493.
Serve the files in TestData/pages from a local listener inside the tests instead.
Write tests for these cases:
- SupplierScraper reads every field from the saved supplier page and contact page.
- CertificateScraper reads every field from the saved certificate page.
- A contact page that answers 401 leaves Email and Phone empty and keeps the supplier.
- A listing page with no Next page link ends the crawl.
Finish by running dotnet test, then report the passing and failing test counts.What did the prompts produce in a real run?
Fresh Release builds of each commit that changed code repeated every crawl below against a new copy of the local site. A script read the site's request log and compared every output file with TestData/expected.
- Before the move, the HttpClient and HtmlAgilityPack crawler wrote all 46 suppliers in 96 requests and 49.2 seconds.
- After the move, SupplierScraper wrote the same 46 rows, with the same header and row order.
- The new crawl sent 142 requests in 57.9 seconds, and the contact pages answered 46 of them with 401.
- Cursor reported 96 requests for the new run, a count of successful requests only.
- SupplierScraper queued only the first listing page in Init and followed the Next page link from Parse.
- Cursor answered the five documentation questions without changing a file and found no page that states the default limits or answers the two-scraper question.
- For the Authorization header, Cursor found no documented example and described HttpRequestHeaders as sent with every request, which the probe showed to be incorrect.
- The plan claimed that pinning the member identity on contact requests avoids the double fetch, which 46 responses of 401 disproved.
- Before its fix, the certificates command wrote 1 and 0 certificates in Cursor's runs and 6 of 32 in an independent run.
- Cursor traced the failures to the random identity, the member identity in the random pool and the missing rate limit.
- The fixed CertificateScraper still requested all three register pages from Init, against the project rule.
- After the fix, the command wrote 32 of 32 certificates in three independent runs, each with 67 requests and no 429.
- The first crawl-all build received no 429, and no second held more than 4 requests.
- Two independent crawl-all runs wrote all 46 suppliers and all 32 certificates in 105.2 and 104.8 seconds.
- Running the two commands one after the other took 80.8 seconds.
- The xUnit suite passed 4 of 4 tests, and the site on port 5493 logged no request while they ran.
- Some steps ran differently from the page. All seven prompts went through the Cursor CLI in one headless session. Plan Mode persisted after prompt 03, so prompts 05 to 07 each needed the follow-up message "Build the plan." In prompt 04, Cursor also changed the pages scraper and the README, which the prompt did not request.
The GitHub Copilot guide limits its IronWebScraper conventions to the crawler and test paths in a path-specific instructions file, as the rule file does here. The Google Antigravity guide works from the same advanced tutorial and measures what IronWebScraper's web cache and saved crawl state keep. The run record lists the assistant version, model, duration and test machine for each guide.
Troubleshooting
- Advanced Webscraping Features when a crawl needs throttling or a login
- Thread Safety when one crawl runs several requests at once
- Applying License Keys when Start throws a licensing exception
- IronWebScraper & JavaScript when a page is full in the browser and empty to the scraper
Questions?
If you have any questions, reach out to support@ironsoftware.com
Curtis Chau holds a Bachelor’s degree in Computer Science (Carleton University) and specializes in front-end development with expertise in Node.js, TypeScript, JavaScript, and React. Passionate about crafting intuitive and aesthetically pleasing user interfaces, Curtis enjoys working with modern frameworks and creating well-structured, visually appealing manuals.
