How to Use IronWebScraper with GitHub Copilot
A cinema showtimes aggregator collects the screenings of several venues into one file that a listings app reads every night. Each venue publishes its program on its own pages, and the same film can play at three venues at once. A screening that the crawl misses does not appear in the app at all.
GitHub Copilot works with the repository through chat and agent mode in Visual Studio and VS Code, and through Copilot CLI in a terminal. Its cloud agent on GitHub turns an assigned issue into a pull request. IronWebScraper crawls websites from .NET through a subclass of WebScraper that overrides Init and Parse. Instruction files committed to the repository carry the crawler's rules to each of those surfaces.
This guide covers directing Copilot to build and maintain the crawler. The movie website guide describes the listing page, detail page and MetaData pattern that the crawler follows. Every crawl described here targets a test site served on the local machine, never a cinema's live website.
Install with NuGetOpen a terminal at the repository root, start Copilot CLI and submit the prompt. The @ mentions attach the named files in Copilot CLI, and Copilot Chat in Visual Studio takes # references instead.
Build the showtimes crawler in @src/Showtimes.Scraper/ShowtimeScraper.cs.
It collects every screening from the cinema test site at http://localhost:5447/.
Directory.Packages.props already pins IronWebScraper 2026.10.1.
The crawler should do this:
- Start at the site's start page and follow each venue's pages, page 2 included.
- Write one line per screening to out/showtimes.jsonl, using the Screening class.
- Fill the venue-page fields as @docs/output-format.md describes.
- Leave the film-page fields empty for now.
Then do these steps in order:
1. Add a crawl command to @src/Showtimes.Cli/Program.cs that takes --site <url> and --out <path>.
2. Run dotnet build.
3. In the repo root, run crawl --site http://localhost:5447/ --out out/showtimes.jsonl.
Tell me how many screenings each venue has.The prompt names no selector, because docs/output-format.md lists the source element of every field.
Set IRONWEBSCRAPER_LICENSE_KEY in the environment where Copilot runs, and have the program copy it into License.LicenseKey. The hub guide names the variables that IronWebScraper reads by itself, and the license key guide lists the other supported locations.
Minimal Workflow (5 steps)
- Download the C# web scraping library from NuGet
- Write the crawl rules into .github/copilot-instructions.md plus one path-specific instructions file
- Request each film page once and copy its details onto every screening of that film
- Request one start page from Init and queue every other page from a parse method
- Assign the next change to the cloud agent as an issue and verify the pull request against the IronWebScraper API reference
How do you set IronWebScraper conventions for Copilot?
Place the project-wide rules in .github/copilot-instructions.md. GitHub's support reference lists that file for Copilot Chat in Visual Studio, VS Code and the JetBrains IDEs, and for the cloud agent and Copilot code review. Copilot CLI reads it from the same location.
This repo holds the showtimes crawler, which gathers cinema screenings into one file.
Follow these rules in every change:
- Use IronWebScraper for all crawling and HTML parsing, never HttpClient.
- Don't add AngleSharp, HtmlAgilityPack or another scraping package.
- Change the IronWebScraper version in Directory.Packages.props only in an upgrade task.
- The showtimes crawler and its tests read IRONWEBSCRAPER_LICENSE_KEY into License.LicenseKey.
- IronWebScraper won't read that variable name on its own, so set the key before Start.
- Keep the key out of all output, logs included.
- Don't list environment variables.
- Crawl only the site a task names, which is http://localhost:5447/ in development.
- Tests crawl TestData/site through a local HTTP listener and never touch that address.
- Keep RateLimitPerHost at one second.
- Write records with WebScraper.Scrape, not with a JSON writer of your own.
- Treat docs/output-format.md as the contract for out/showtimes.jsonl.
- If the contract looks wrong, tell me and leave the file alone.
- Read the IronWebScraper docs for agents at https://ironsoftware.com/webscraper/csharp/llms.txtThe rate limit rule sets the interval between requests in every crawl. At the default RateLimitPerHost of 50 milliseconds, one probe crawl reached the test site with requests 0.048 to 0.064 seconds apart. At one second, the gaps measured 1.02 to 1.10 seconds.
How do you give the crawler its own rules?
A file such as .github/instructions/scraper.instructions.md holds rules for the parts of the tree that its applyTo patterns name, here the crawler and its tests. According to GitHub's documentation, Copilot CLI includes the file only when it works on a file that matches one of those patterns.
---
applyTo: "src/**/*,tests/**/*"
---
The crawler and its tests rely on IronWebScraper 2026.10.1, which behaves like this:
- A crawl fetches each URL once per parse method.
- A second Request for that URL with the same parse method is dropped, even with new MetaData.
- MetaData passes the parse method the same object instance, not a copy.
- When a parse method throws, the page is fetched and parsed again, up to 8 more times.
- Each retry repeats every Scrape call made before the throw.
- A URL that answers 500 is requested 9 times, and its parse method is never called.
- Css throws an XPathException on a selector with two > combinators, such as div > p > a.
- TextContentClean returns entities such as é unchanged.
Follow these rules in this code:
- Request each film page once and copy its details onto every screening of that film.
- Decode text from TextContentClean with WebUtility.HtmlDecode.Each behavior listed in that file comes from a probe against the test site.
What does IronWebScraper do with a film page that three venues link to?
The venue pages link every film to one film page, so the three venues that show Harbour Lights link to the same URL.
Fill in the film-page fields on every line the crawler writes.
They are RuntimeMinutes, Genres and Rating, and @docs/output-format.md gives the source of each.
Follow this guide for the detail requests:
https://ironsoftware.com/webscraper/csharp/how-to/scraping-from-an-online-movie-website/
Two things about the test site matter here:
- Harbour Lights plays at all three venues, and all 6 of its screenings need the runtime.
- The Night Train to Arles page answers 500 on purpose.
Keep the Night Train to Arles screening, with its film fields empty.
Then run the crawl again with --out out/details/showtimes.jsonl.
Tell me how many lines got a runtime, and which film's lines didn't.IronWebScraper fetches a URL once per parse method for the whole crawl. In the probe, three venues requested the Harbour Lights page with the same parse method and different MetaData, and the site received one request. The parse method saw the first venue's MetaData only, so the other two venues' screenings never received details. The same URL requested with a different parse method was fetched again, and so was the page with a different query string.
The movie guide's detail parser fails on the guide's own markup. Its genre selector div > p > a and its actor selector div > p:nth-child(2) > a both threw an XPathException, because IronWebScraper rejects a selector with two child combinators. Selectors with one combinator, such as div.mvici-left > p, worked.
IronWebScraper retries a page whose parse method throws, as it retries a page that fails to load. In the probe, the parse method was called 9 times for one page, and a Scrape call made before the throw wrote 9 lines.
Output
In a browser, the Riverside venue page shows its name with an accented letter. The page source writes that letter as a named HTML entity, and TextContentClean returned the entity unchanged.
Page 1 of the Riverside venue on the local test site, captured in Microsoft Edge. All films and venues on the site are fictional.
The Harbour Lights and Night Train to Arles lines of out/details/showtimes.jsonl as the second prompt wrote them, with their line numbers in the file.
How do you test the crawler without the live site?
Use Shift+Tab to cycle Copilot CLI to plan mode, then submit this prompt.
Plan a new xUnit project, tests/Showtimes.Tests, listed in Showtimes.slnx.
Show me the plan, and don't write code until I say go.
The tests' local listener must answer 500 for /film/night-train-to-arles-20790/, like the test site.
Each test runs ShowtimeScraper and checks one of these:
- Northgate Picturehouse has 10 screenings, and the other two venues have 7 each.
- The Riverside venue name has no HTML entity left in it.
- All 6 Harbour Lights screenings have a RuntimeMinutes of 112.
- The Night Train to Arles screening is still there, with a null RuntimeMinutes.
After my go, write the tests and rerun dotnet test until every one passes.
Then give me the number that pass.The robots.txt in TestData/site has no effect on a test crawl. IronWebScraper requests robots.txt without the port of the start URL, so a listener on a random port never serves those rules. With a listener on port 80, a probe crawl read them and skipped the disallowed /booking/ page. The Claude Code guide covers this behavior in full.
Why does a venue go missing from some runs?
In this scenario, a teammate's tools/VenueCheck requests the three venue pages before the nightly crawl and reports any venue that does not answer.
tools/VenueCheck reports a venue as DOWN on some runs, though the site is up.
It checks the three venue pages listed in tools/VenueCheck/venues.json.
Find out why and fix it.
Before you fix anything, do these steps in order:
1. Run it five times against http://localhost:5447/ and note which venues came back DOWN.
2. Turn on LogLevel.All and find out what IronWebScraper does with the Request calls in Init.
3. Tell me the cause.
Keep the fix within these limits:
- Change files in tools/VenueCheck only.
- Add no sleep, and keep a single call to Start.
Then run it five more times and show me the results.IronWebScraper does not keep every request made in Init. With three venue URLs requested from Init, the crawl fetched all three in 1 of 26 probe runs and one or two in the other 25. The log listed three requests and reported the crawl complete after two, and the skipped URLs never reached the server. Requests made from a parse method were never lost. Requesting the first venue from Init and the other two from its parse method returned all three venues in 8 of 8 runs.
What makes an issue the cloud agent can act on?
According to GitHub, an issue assigned to Copilot passes the agent its title, its description, the comments present at assignment and any instructions given at that point. Comments added afterward do not reach the agent, so the issue body must hold the whole request.
Title: A second crawl must not double out/showtimes.jsonl
Files: src/Showtimes.Scraper and tests/Showtimes.Tests
Versions: net9.0 with IronWebScraper 2026.10.1
Background: IronWebScraper's Scrape appends to the file it writes and never empties it.
Without a cleanup step, a second crawl to the same --out path leaves 48 lines instead of 24.
Expected: after any crawl, the output file holds exactly one line per screening.
Constraint: the crawl command keeps its options.
Acceptance: close the issue once these are true:
- Two crawls in a row leave the same 24 lines, in any order.
- A new test crawls twice into one temp folder and counts the lines.
- dotnet test passes.
Summary: say where the old output gets cleared, and why there.Why do the cloud agent's tests fail on an IronWebScraper project?
The tests that crawl need the license key in the cloud agent's environment. GitHub's documentation describes that environment as Ubuntu Linux by default, and a copilot-setup-steps job in .github/workflows/copilot-setup-steps.yml can prepare it. The job runs in GitHub Actions before Copilot starts working, and the workflow does not trigger unless the file is on the default branch. When a setup step fails, Copilot skips the remaining setup steps and begins working in its environment as it stands.
Create the cloud agent's setup workflow in .github/workflows/copilot-setup-steps.yml.
Call its job copilot-setup-steps, because GitHub skips a job with any other name.
The job needs these settings:
- The runner is ubuntu-latest.
- The workflow token has contents: read and nothing else.
- workflow_dispatch triggers it, and so does a pull request that edits the file.
Its steps go in this order:
1. Check out the code.
2. Install .NET 9.
3. Run dotnet restore, then dotnet build -c Release.
4. Look for the license key.
The key is IRONWEBSCRAPER_LICENSE_KEY, saved as an Agents secret.
The job gets it as an environment variable already, so skip the secrets context.
When the key is missing, the last step behaves like this:
- On pull_request or workflow_dispatch, it logs a warning and passes.
- On any other event, it fails with an error naming the secret.
Show me the finished YAML, and don't try to run the workflow.Store the key as an Agents secret: according to GitHub, the cloud agent has no access to Actions, Codespaces or Dependabot secrets. In the repository's Settings, open Secrets and variables in the Security section, select Agents and add IRONWEBSCRAPER_LICENSE_KEY on the Secrets tab. GitHub states that these secrets are "exposed to Copilot as environment variables" and can be used by scripts and tools that Copilot runs, "including by your copilot-setup-steps.yml workflow." The documentation does not say that a workflow step reads one without an env mapping, which the prompt above assumes.
What should you check before merging?
Run the review as a separate copilot -p command, outside the session that wrote the code.
/review the changes on this branch compared with main.
Check the code against these three files:
- .github/copilot-instructions.md
- .github/instructions/scraper.instructions.md
- IronWebScraper.xml in the IronWebScraper package of the NuGet cache
Also look for two patterns that neither instructions file covers:
- an Init method that calls Request more than once
- a crawl that starts with an old output file still in place
List every mismatch you find, or say that nothing came up.
Don't change any file.What did the prompts produce in a real run?
The seven prompts ran unattended in Copilot CLI against a showtimes repository and the local test site built for this guide. Every figure below comes from the output files, the site's request log and the repository, not from Copilot's replies.
- Before the run, the CLI printed a usage line and exited with code 1.
- After the first prompt, out/showtimes.jsonl held 24 screenings, 10 at Northgate Picturehouse and 7 at each of the other two venues.
- In the second prompt's crawl, each film page received one request, except Night Train to Arles, which received 9 and answered 500 to each.
- The file from the second prompt held 24 lines, and 23 of them had a runtime.
- The Night Train to Arles screening remained, with an empty runtime, genre list and rating.
- All 6 Harbour Lights screenings, at three venues, carried the 112-minute runtime.
- In plan mode, Copilot wrote a plan, changed no file and waited for the approval message.
- The 4 tests from the third prompt passed without a single request to the test site.
- Before the fix, VenueCheck reported one or two venues as down in 4 of 5 check runs.
- The fix left one Request call in Init and requested each next venue from the parse method.
- All three venues answered in 10 of 10 runs of the fixed check.
- With the first venue answering 500, the fixed VenueCheck reported all three venues as down in 3 of 3 runs.
- The other two venue pages received no request in those runs.
- The issue session added the new test and left the crawler unchanged, because its Init had deleted old output since the first prompt.
- All 5 tests in the final repository passed with the key set and failed without it.
- The review session reported no mismatch and changed no file. Neither pattern it searched for remained after the earlier prompts, so this run did not test whether the review finds them.
- Some steps ran differently from the page. The issue body went to a new Copilot CLI session, because the test repository has no GitHub remote. The setup workflow was written but not run, for the same reason.
Cursor also scopes committed rules to the paths they name, and Claude Code also holds a plan for approval before it edits the project. The shared run record lists the Copilot CLI release, the model Auto selected, the duration and the test machine.
Troubleshooting
- How to Scrape an Online Movie Website for the listing and detail page pattern with
MetaData - Applying License Keys for a crawl that throws a licensing exception at
Start - Thread Safety for the connection and rate limits of a crawl
Questions?
If you have any questions, reach out to support@ironsoftware.com
Curtis Chau holds a Bachelor’s degree in Computer Science (Carleton University) and specializes in front-end development with expertise in Node.js, TypeScript, JavaScript, and React. Passionate about crafting intuitive and aesthetically pleasing user interfaces, Curtis enjoys working with modern frameworks and creating well-structured, visually appealing manuals.
