How to Use IronWebScraper with Google Antigravity
A recruitment analytics team collects every listing from a regional job board each morning. A crawl of a few hundred pages can stop halfway when a server restarts, a laptop sleeps or someone presses Ctrl+C. When the crawler starts again, it has to finish the remaining pages without downloading the finished ones twice and without writing a job twice. A crawl can also end without an error and still record a salary that the page no longer shows. Neither fault appears in a console log.
Google Antigravity can check the crawl against the page the way a job seeker sees it. Its /browser command launches a browser subagent that opens each page and captures a screenshot. IronWebScraper keeps a web cache and can restart a crawl under a crawl id, the features covered in its Advanced Webscraping Features tutorial. This guide builds a job board crawler with both tools and tests it against a local copy of a board whose 60 listings are known in advance.
This guide is intended for developers using AI agents to write .NET crawlers. Every crawl in it targets a test board served on localhost, and it does not address crawling sites that you do not operate.
Quickstart
Install with NuGetWith the HarborJobs.Crawler project open in Antigravity, describe the crawl.
/plan Build a crawler for Harbor Jobs, the local test board at http://localhost:5481.
This repo is a .NET 9 console app named HarborJobs.Crawler.
Install IronWebScraper 2026.10.1 and subclass WebScraper.
The crawl runs in this order:
1. Start at /jobs/page-1.html and follow the rel="next" links through page 6.
2. Open every job linked from a listing page.
3. Write one line per job to output/jobs.jsonl.
A line holds the job id, title, company, location, salary, posted date and a closed flag.
The crawler also follows these rules:
- It requests nothing outside localhost:5481, and at most one page every 250 ms.
- It obeys robots.txt, and its user agent is HarborJobsCrawler/1.0.
- IronWebScraper's working directory is output/.
- Pages go into IronWebScraper's web cache, so a restart doesn't download them again.
- The crawl gets an id, so a stopped run can start again under that id.
- A --stop-after N option stops the crawl after N jobs, for testing restarts.
Program.cs reads IRONWEBSCRAPER_LICENSE_KEY and sets IronWebScraper.License.LicenseKey once.
The key stays out of output, logs and files. Don't list environment variables either.
The Implementation Plan should answer two questions:
- Where on disk do the cache and the saved crawl state go?
- What happens to queued requests when a crawl stops?
Once the plan is done, build the crawler and run one full crawl.
Tell me how many lines output/jobs.jsonl ends up with.IronWebScraper does not read IRONWEBSCRAPER_LICENSE_KEY by itself, so the prompt has Program.cs assign License.LicenseKey from it. The license key guide describes the other ways to apply the key.
Minimal Workflow (5 steps)
- Download the C# web scraping library from NuGet
- Start the agent on a console project that builds, with a test site on localhost
- Give the crawl a working directory, a web cache and a way to stop after N jobs
- Have /browser compare the job pages with the output file
- Stop and restart the crawl, then confirm every member in the API reference
What is Google Antigravity, and how is it different from a chat assistant?
According to Google's overview, the Antigravity App is a standalone desktop application for launching and inspecting teams of AI agents. Its agents return Artifacts, such as an Implementation Plan, a Walkthrough and screenshots, for review at key milestones.
The AI coding assistants guide explains why a task that adds the package names IronWebScraper exactly. It also lists IronWebScraper's documentation files for agents and includes a single-prompt acceptance test.
What should you check in the plan, the code changes and the Walkthrough?
The plan should place the web cache and the saved crawl state under the working directory, and it should say that a stop leaves the queued requests listed in the saved state.
In the code changes, look for reflection. The test run stopped a crawl and later read cached pages by calling non-public IronWebScraper members through reflection, with 13 uses of BindingFlags.NonPublic after the first prompt and 9 at the end. Those members are absent from the API reference and carry obfuscated names, so ask for the public API and confirm each member in the API reference.
The Walkthrough should hold a screenshot of each job page that the browser opened. The Antigravity App stores each screenshot as an image artifact that accepts comments. In the CLI, a prompt that names a folder, such as artifacts/shots, keeps the screenshots in the repository.
How do you check the records against the board in a browser?
/browser Check the crawl against the board as a reader sees it.
Delete output/, cache included, and run the crawler once with no options. Then, in the browser:
1. Open http://localhost:5481/jobs/page-1.html and go through all six listing pages.
2. Count the job cards on each page.
3. Open J-1004, J-1012, J-1025, J-1047 and J-1058, and save a screenshot of each in artifacts/shots.
4. Note every field as the page displays it.
Put two tables in the Walkthrough and in artifacts/browser-check.md:
- job cards on the board against lines in output/jobs.jsonl
- one row per field of those five jobs, with the page value, the record value and a match column
If something differs, trace it in the crawler code before editing.
Then fix it, delete output/, crawl again and add both tables a second time.
Tell me what differed and what you changed.The board holds 60 jobs on six listing pages. Seven listings were edited after posting, and each keeps its old salary in a hidden element ahead of the salary the page displays. J-1004, J-1012 and J-1047 are among them. A crawler that reads the first dd.salary element records the old figure, while the browser shows the new one.
J-1004 as the /browser subagent captured it in the test run.
Output
The file below is the jobs.jsonl that the crawl wrote after the browser comparison, unchanged.
60 job records from the test board, one JSON object per line, with the displayed salary for J-1004.
What do the web cache and the saved state keep?
EnableWebCache writes one .webcache file per URL in the WebCache folder of the working directory. For the 66 pages of the test board, the probe measured 66 files and 32,341 bytes. A rerun made no request to the board and finished in 4.6 seconds instead of 9.4. The CretaedFromWebCache flag, spelled this way in the API, was true on every response.
The cache does not detect changes on the site. After the board added two jobs and changed a salary, a cached rerun made no request and wrote the old listings. EnableWebCache with a TimeSpan limits this, and with a 15-second expiry a rerun 20 seconds later fetched all 66 pages again.
Start with a crawl id writes two files to the SavedState folder, one listing every queued request and one listing the requests that received a response. Both files existed after a crawl that ran for 6 seconds, although the XML documentation describes a save every five minutes. A finished crawl id started again made one request and wrote nothing, so a daily job that reuses one id collects no new jobs.
ScrapeUnique never wrote the first record that a process passed to it, so 6 of 6 probe crawls of 60 jobs wrote 59 lines.
How do you set up a project Antigravity can work in?
What IronWebScraper version does HarborJobs.Crawler.csproj pin? Quote the PackageReference.
Quote the lines that apply the license key as well.
Judge everything from the code, without starting the crawler.
The key handling should meet these rules:
- The crawler's own code takes the key only from IRONWEBSCRAPER_LICENSE_KEY.
- No appsettings file or launch profile contains it.
- Without that variable, the crawler prints one line naming it and exits with code 2.
- A licensing failure is reported by exception type name, never by its message.
If the missing-variable line isn't there, add it to Program.cs and touch nothing else.
For every other rule that fails, tell me the file and line.Without a license key, IronWebScraper 2026.10.1 throws a LicensingException from Start before Init runs. The exception type is not public, so catch (LicensingException) fails to compile with CS0122 and a handler has to match the type by name.
ObeyRobotsDotTxt reads robots.txt from the default port only, so the test run's agent added the board's disallowed path to BannedUrls.
How do you save the crawl rules for later tasks?
When the browser comparison passes, record the session's decisions as rules. According to Antigravity's rules documentation, the CLI reads AGENTS.md, GEMINI.md and .agents/rules from the repository root and from subdirectories. The /learn command condenses corrections from the session into persistent rules.
/learn Write down what this session settled, in AGENTS.md at the repo root.
Use the heading "Crawling Harbor Jobs" and cover these points:
- the board URL, the 250 ms limit, robots.txt and the user agent
- the folders for the cache and the crawl state, and when to delete them
- which selector gives the salary the page displays
- that the key comes only from IRONWEBSCRAPER_LICENSE_KEY and is never printed or stored
- that no command lists environment variables
- that a crawl is finished once the browser comparison matches
Show me AGENTS.md when it's written.What happens when a crawl stops halfway?
Stop does not end a crawl cleanly in IronWebScraper 2026.10.1. Its XML summary calls the stop graceful and says the crawl can restart under its crawl id with no loss of data. After Stop, called from a parse method or from another thread, Start and StartAsync never returned in 3 of 3 probes, which were ended after 60 to 150 seconds.
A restart under the same crawl id did not finish the crawl either. In 10 probe restarts after a stop at 20 jobs, Start returned normally each time, and the file held 19 to 57 of the 60 jobs. In the 9 restarts whose saved state was inspected afterwards, 1 to 19 queued requests still had no response.
/plan A crawl that stops halfway must finish on the next start, with every job exactly once.
Write the Implementation Plan first, and wait until I approve it before you edit any file.
After I approve, measure a restart before you fix anything:
1. Delete output/.
2. Run the crawler with --stop-after 20.
3. Run it again with no options.
4. Count the lines in output/jobs.jsonl, the repeated ids and the ids missing from J-1001 to J-1060.
5. Have the crawler log the pages each run took from the board and from the cache.
6. Write those counts to output/run-counts.txt.
Repeat steps 1 to 6 three times. Treat a run still going after two minutes as hung, and kill it.
Then fix every cause you found, keep the web cache, and do the three restarts again.
Add what the restarts showed to AGENTS.md.
Then tell me:
- the counts for each run, before and after the fix
- what output/SavedState contained after each stop, or that the fix no longer uses itThis example uses only IronWebScraper's public API and compiles with IronWebScraper 2026.10.1. In three stop and restart cycles against the test board, it requested 22 pages up to the stop and the remaining 44 on restart, and its file held each of the 60 jobs once. After the third cycle, the output folder held jobs.jsonl and 66 cached pages in WebCache, and the example deletes neither.
using System.Text.Json;
using IronWebScraper;
var key = Environment.GetEnvironmentVariable("IRONWEBSCRAPER_LICENSE_KEY");
if (string.IsNullOrWhiteSpace(key))
{
Console.WriteLine("Missing required environment variable: IRONWEBSCRAPER_LICENSE_KEY");
return 2;
}
License.LicenseKey = key;
int stopAfter = args.Length == 2 && args[0] == "--stop-after" ? int.Parse(args[1]) : 0;
new HarborJobsScraper(stopAfter).Start();
return 0;
public sealed class HarborJobsScraper(int stopAfter) : WebScraper
{
private const string Board = "http://localhost:5481";
private readonly object _fileLock = new();
private readonly HashSet<string> _written = new();
private int _jobsThisRun;
public override void Init()
{
WorkingDirectory = "output";
RateLimitPerHost = TimeSpan.FromMilliseconds(250);
ObeyRobotsDotTxt = true;
AllowedDomains.Add("localhost");
Identities.Add(new HttpIdentity { UserAgent = "HarborJobsCrawler/1.0" });
// Pages fetched before a stop come back from output/WebCache on the next start.
EnableWebCache(TimeSpan.FromHours(12));
string jobsFile = Path.Combine(WorkingDirectory, "jobs.jsonl");
if (File.Exists(jobsFile))
{
foreach (var line in File.ReadLines(jobsFile))
{
_written.Add(JsonDocument.Parse(line).RootElement.GetProperty("id").GetString()!);
}
}
Request(Board + "/jobs/page-1.html", Parse);
}
public override void Parse(Response response)
{
foreach (var link in response.Css("article.job-card h2 a"))
{
Request(response.ToAbsoluteUrl(link.Attributes["href"]), ParseJob);
}
foreach (var next in response.Css("a[rel=next]"))
{
Request(response.ToAbsoluteUrl(next.Attributes["href"]), Parse);
}
}
public void ParseJob(Response response)
{
// The page carries an old salary in a hidden element ahead of the one it displays.
var salary = response.Css("dd.salary").Last(n => n.GetAttribute("style") is not { } s || !s.Contains("display:none"));
var job = new
{
id = response.Css("dd.job-id")[0].TextContentClean,
title = response.Css("h1.job-title")[0].TextContentClean,
company = response.Css("dd.company")[0].TextContentClean,
location = response.Css("dd.location")[0].TextContentClean,
salary = salary.TextContentClean,
postedDate = response.Css("time")[0].GetAttribute("datetime"),
closed = response.CssExists("p.closed"),
};
lock (_fileLock)
{
// ScrapeUnique drops the first record of each process, so the crawler writes the file itself.
if (_written.Add(job.id))
{
File.AppendAllText(Path.Combine(WorkingDirectory, "jobs.jsonl"), JsonSerializer.Serialize(job) + "\n");
}
if (stopAfter > 0 && ++_jobsThisRun >= stopAfter)
{
// Start never returns after Stop() in 2026.10.1, so a test stop ends the process.
Environment.Exit(0);
}
}
}
}When should you dispatch more than one agent?
Harbor Jobs adds jobs every morning, and the crawler is scheduled once a day.
Divide the next part between two agents.
Agent A builds an xUnit project, HarborJobs.Crawler.Tests:
- Save the J-1004, J-1012, J-1021, J-1047 and J-1061 pages from today's board into TestData.
- Test the record each saved page produces, salary and closed flag included.
- The tests read only those files and send no request.
- Keep the test project out of the crawler's compile items.
- Run it with dotnet test HarborJobs.Crawler.Tests.
If the tests fail, stop there and tell me why.
Otherwise Agent B takes these steps:
1. Run the crawler with today's command and the existing output/ folder, as tomorrow's run would.
2. Use /browser to open page 1 of the board and each job it links to.
3. Check that every one of those jobs has a line in jobs.jsonl with the same values.
If the board and the file disagree, explain why before changing anything.
A restart on the same day still reads the cache, and the fix has to keep that.
Save screenshots in artifacts/shots-day2.
Write how many page 1 jobs matched before and after to artifacts/day2-check.md.
Tell me what each agent found.Before this prompt, the test board moved to the next morning's listings, with two new jobs at the top of page 1, one changed salary and one closed job.
What did the prompts produce in a real run?
All six prompts ran unattended in one main Antigravity CLI conversation, with permission prompts skipped. Every figure below comes from the timing log and from independent checks of the run's commits, request log and reruns.
- Before the first prompt, the repository held a console project that only printed its name.
- While planning, the agent fetched the job pages with its own PowerShell requests and found the hidden salaries.
- The first full crawl wrote 60 lines, and all 60 matched the board.
- The first test of the stop option never exited and was ended by the operator after 23 minutes.
- In the first prompt, 64 of 273 intervals between crawler requests were under 240 ms, the shortest 8 ms.
- From the second prompt on, no crawler interval fell below 292 ms.
- The browser comparison counted 60 job cards against 60 lines and found no field difference, so its fix step had nothing to change.
- The browser subagent's Chrome requested 125 pages under /apply/, which robots.txt disallows, and the crawler requested none.
- Without the key variable, the final crawler printed one line naming it and exited with code 2.
- With an invalid key, it ended with an unhandled LicensingException, a failure that the setup prompt reported and left in place.
- The restart fix dropped the crawl id and kept its own queue of pending pages.
- In three independent stop and restart cycles, the final crawler wrote all 60 jobs once, with 23 requests before each stop and 44 after.
- On the next day's board, the first prompt's crawler made no request and missed both new jobs.
- That crawler's file matched 7 of the 10 jobs on the new page 1.
- After Agent B's fix, the final crawler matched all 60 jobs on that board.
- Agent A's five tests passed without contacting the board.
- Some steps ran differently from the page. The first prompt's 57.2 minutes included the wait for the hung stop test. Every prompt from the second on ended with an agy status of ERROR, for a malformed function call, after its full reply. Agent A ran as a separate subagent, and the main conversation ran Agent B's steps with a browser subagent. The /browser subagent used a headless Chrome that the run started, since the CLI looks for Chrome's DevTools port file in the user's profile. The final crawler cleared its cache after a complete crawl, so a rerun after completion requested every page again.
ChatGPT Codex reads its standing rules from AGENTS.md, as this guide does, and hands named parts of a task to explorer and worker agents. GitHub Copilot also tests a crawler against saved pages without a request to the site. Antigravity's version, the model, the run time and the test machine are recorded in the shared run record.
Troubleshooting
- Advanced Webscraping Features for the web cache, crawl ids and throttling
- Applying License Keys when
Startthrows before the first request - IronWebScraper & JavaScript when the browser shows a value that the crawler never receives
- Thread Safety for how IronWebScraper manages its crawl threads
Questions?
If you have any questions, reach out to support@ironsoftware.com
Curtis Chau holds a Bachelor’s degree in Computer Science (Carleton University) and specializes in front-end development with expertise in Node.js, TypeScript, JavaScript, and React. Passionate about crafting intuitive and aesthetically pleasing user interfaces, Curtis enjoys working with modern frameworks and creating well-structured, visually appealing manuals.
