--- name: ironwebscraper description: > Crawl websites and extract structured data in C#/.NET using IronWebScraper (the `IronWebScraper` NuGet package). Use when the task involves writing a web crawler or scraper, subclassing `WebScraper`, parsing HTML with CSS selectors (`response.Css`) or XPath (`response.XPath`), following links across pages, exporting scraped data as `ScrapedData`/JSON/JSONL, downloading files or images from a site, managing crawl concurrency and politeness (rate limiting, per-host throttling, `robots.txt`), rotating proxies/user agents/cookies/logins via `HttpIdentity`, or resuming/auto-saving a long crawl job — or whenever a project already references `IronWebScraper`, `WebScraper`, `Response`, `ScrapedData`, or `HttpIdentity`. --- # IronWebScraper (C# / .NET) IronWebScraper is a crawler **framework**, not a one-shot fetch-and-parse call: you subclass `WebScraper`, describe where to start and how to parse each page, and the base class drives the crawl — queuing requests, managing threads, throttling per host, retrying failures, and optionally saving/resuming the whole job. It is a pure managed library (no embedded browser, no native binaries) that parses static HTML/DOM with CSS selectors, XPath, or jQuery-style selectors. ## Scope of this skill | | | |---|---| | Package | `IronWebScraper` | | Versions | 2024.x – 2026.x | | Namespaces | `IronWebScraper`, `IronWebScraper.Urls` | | Runtimes | .NET Framework 4.6.2+, .NET Core 3.1+, .NET Standard 2.0/2.1, .NET 5–10 | **Not a headless browser.** The shipped assembly has no reference to Chromium, Selenium, or any browser automation library — it downloads and parses raw HTML/DOM. If a target page only renders its content via client-side JavaScript, IronWebScraper will not see that content; say so rather than assuming `Css`/`XPath` will find it (a browser-driven tool, e.g. IronPDF's Chromium renderer for a snapshot, or a real browser-automation library, is needed instead). ## Install No platform variants — one package, pure managed code, pulls in `IronSoftware.Common` and `IronSoftware.System.Drawing` (used for `DownloadImage`'s resizing) automatically. ```bash dotnet add package IronWebScraper ``` ## Licensing — do this first, every time ```csharp IronWebScraper.License.LicenseKey = Environment.GetEnvironmentVariable("IRONWEBSCRAPER_LICENSE_KEY"); if (!IronWebScraper.License.IsLicensed) Console.Error.WriteLine("IronWebScraper is unlicensed — production use will be refused."); ``` - Read the key from the environment (`IRONWEBSCRAPER_LICENSE_KEY`) or user secrets. **Never** inline a key in source, commit one, or echo one to the terminal. `License.LicenseKey` also reads from `Web.config`/`App.config` (`` in `appSettings`) or `appsettings.json` (key `"IronWebScraper.LicenseKey"`) on .NET Core, if set there instead. - **Treat a missing key as a blocker, not a warning.** Verified directly: an unlicensed `Start()` throws `IronSoftware.Exceptions.LicensingException: Production License Required` with the message *"Development use: Free for 7 days / Production use: Requires a license"* — after that grace window every run fails outright rather than degrading quietly. If `IsLicensed` is false, say so and ask for a key instead of promising a working crawl. Trial keys: . - `IronWebScraper.License.IsValidLicense(key)` checks a key without applying it. - `License.DisableAppAnalytics()` turns off anonymous usage analytics ("limitations apply"). ## Running a one-off crawl from the terminal Requires only the .NET SDK. Unlike IronPDF/IronOCR, IronWebScraper needs no native binaries and no runtime code generation, so a **.NET 10 file-based app works with no AOT opt-out** — verified by running it as `dotnet run task.cs` and confirming it built and executed straight through to a runtime licensing check, with no `PlatformNotSupportedException`: ```bash cat > /tmp/task.cs <<'EOF' #:package IronWebScraper@2026.8.1 using IronWebScraper; IronWebScraper.License.LicenseKey = Environment.GetEnvironmentVariable("IRONWEBSCRAPER_LICENSE_KEY"); new QuoteScraper().Start(); class QuoteScraper : WebScraper { public override void Init() => Request("https://example.com/", Parse); public override void Parse(Response response) { foreach (var h1 in response.Css("h1")) Scrape(new ScrapedData() { { "Heading", h1.TextContentClean } }); } } EOF dotnet run /tmp/task.cs ``` On older SDKs, use a scratch project instead: ```bash dotnet new console -o /tmp/ironwebscraper-scratch && cd /tmp/ironwebscraper-scratch dotnet add package IronWebScraper # write Program.cs / scraper classes, then: dotnet run ``` When building a feature rather than asking for a one-off crawl, write the scraper class into the application project instead of a scratch project — crawl jobs are often long-running and belong in a worker/console host, not inline request-handler code. ## Recipes Every member below is verified against the shipped assembly's XML docs and, where noted, an actual run. `WebScraper` is `abstract`: create a subclass, override `Init()` and `Parse()`, then instantiate and call `Start()`. ### Define a scraper ```csharp using IronWebScraper; public class BlogScraper : WebScraper { public override void Init() { this.LoggingLevel = WebScraper.LogLevel.All; // None/Critical/Http/Decision/ScrapedData/All this.WorkingDirectory = @"C:\Scrapes\Blog\"; // scraped data + saved state land here this.Request("https://example.com/blog/", Parse); // at least one start URL, required } public override void Parse(Response response) { foreach (var titleLink in response.Css("h2.entry-title a")) { string title = titleLink.TextContentClean; // whitespace-normalised text Scrape(new ScrapedData() { { "Title", title } }); // appended to a .jsonl file } if (response.CssExists("a.next-page")) this.Request(response.Css("a.next-page")[0].Attributes["href"], Parse); } } new BlogScraper().Start(); // Start(string crawlId = "") — see "Resuming and reliability" ``` `Parse` may be one method or several (e.g. `ParseListing`, `ParseDetail`) — pass whichever one handles a given page shape as the callback to `Request`. ### Selecting data: CSS, XPath, DOM ```csharp HtmlNode[] links = response.Css("div.card > a[href]"); // jQuery-style CSS selectors bool hasCard = response.CssExists("div.card"); HtmlNode[] byXPath = response.XPath("//div[@class='card']/a"); HtmlNode byId = response.GetElementById("main"); HtmlNode[] byTag = response.GetElementsByTagName("article"); HtmlNode first = response.QuerySelector("h1"); HtmlNode[] all = response.QuerySelectorAll("p"); HtmlNode node = links[0]; string text = node.TextContentClean; // cleaned, whitespace-collapsed text string raw = node.TextContent; // uncleaned string html = node.InnerHtml; // also OuterHtml string href = node.Attributes["href"]; // Dictionary; or node.GetAttribute("href") HtmlNode[] kids = node.Css("span"); // Css/CssExists/XPath/QuerySelector* also work per-node HtmlNode root = response.Document; // the whole page as one HtmlNode ``` Whole-page data: `response.Html` (raw HTML string), `response.TextContent`, `response.StatusCode`, `response.FinalUrl` (after redirects), `response.MimeType`, `response.WasSuccessful`. ### Following links and shaping the crawl ```csharp this.Request("https://example.com/next", Parse); // enqueue another page this.Request(new[] { url1, url2 }, Parse); // enqueue a batch this.Request(detailUrl, ParseDetail, new MetaData() { { "page", 2 } }); // carry state forward this.PostRequest(loginUrl, ParseAfterLogin, new Dictionary { { "user", "a" }, { "pass", "b" } }); // form POST (logins) // In a later Parse, read what was attached: int page = response.MetaData.Get("page"); ``` Restrict what the crawler is allowed to touch — set these in `Init()`: ```csharp this.AllowedDomains.Add("example.com"); // glob wildcards or regex strings this.BannedUrls.Add(new[] { "*/logout/*", "*.pdf" }); this.AllowedUrls.Add(@"^https://example\.com/blog/.*$"); ``` `AllowedUrls`/`BannedUrls`/`AllowedDomains`/`BannedDomains` are each an `IronWebScraper.Urls.UrlMatchPatternCollection` (empty = no restriction; if non-empty, a URL must match at least one `Allowed*` pattern and none of the `Banned*` patterns). Override `AcceptUrl(string url)` for logic beyond pattern matching. ### Concurrency and politeness ```csharp this.MaxHttpConnectionLimit = 80; // total concurrent requests this.OpenConnectionLimitPerHost = 25; // concurrent requests per host/IP this.RateLimitPerHost = TimeSpan.FromMilliseconds(200); // minimum polite delay per host/IP this.ThrottleMode = WebScraper.Throttle.ByDomainHostName; // or ByIpAddress this.ObeyRobotsDotTxt = true; // honour /robots.txt paths + rates this.HttpRetryAttempts = 3; // retries before giving up on a URL this.HttpTimeOut = TimeSpan.FromSeconds(30); this.SetSiteSpecificCrawlRateLimit("slow-host.example.com", TimeSpan.FromSeconds(1)); ``` `ObeyRobotsDotTxtForHost(string host)` is virtual — override it for per-host exceptions to the default `ObeyRobotsDotTxt` policy. ### Identities: proxies, user agents, cookies, logins ```csharp this.Identities.Add(new HttpIdentity { UserAgent = CommonUserAgents.ChromeDesktopUserAgents[0], Proxy = "http://user:pass@proxy.example.com:8080", UseCookies = true, // persists cookies per identity ("cookie jar") HttpRequestHeaders = { { "Accept-Language", "en-US" } }, }); this.Identities.Add(new HttpIdentity { NetworkUsername = "svc-account", // HTTP/NTLM/Kerberos auth, not a login form NetworkPassword = "secret", NetworkDomain = "CORP", }); ``` `WebScraper.ChooseIdentityForRequest(Request request)` picks a random `Identity` per request by default — override it for sticky-session or round-robin logic. `CommonUserAgents` also exposes `FireFoxDesktopUserAgents`, `SafariDesktopUserAgents`, `IE11DesktopUserAgents`, `IPhoneUserAgents`, `IPadUserAgents`, `WindowsTabletUserAgents`, and the aggregate `DesktopUserAgents` / `MobileUserAgents` / `All`. For a site that requires a login form (not HTTP auth), `PostRequest` the login form's fields first, with `UseCookies = true` on the identity so the session cookie carries into subsequent `Request` calls. ### Downloading files and images ```csharp string savedPath = this.DownloadFile("https://example.com/report.pdf", @"C:\out\report.pdf", overWrite: false, identity: null); this.DownloadFileUnique(url, @"C:\out\", identity: null); // auto-names to avoid collisions this.DownloadImage(url, @"C:\out\photo.jpg", maxWidth: 800, maxHeight: 600, overWrite: true, identity: null); // resizes on save ``` ### Exporting scraped data ```csharp Scrape(new ScrapedData() { { "Title", title }, { "Price", price } }); // → jsonl named after the type Scrape(myPocoInstance, "products.jsonl"); // any object, explicit file ScrapeUnique(new ScrapedData() { { "Sku", sku } }, "products.jsonl"); // de-duplicated append // Reading results back (e.g. in a follow-up process, or to check progress): foreach (ScrapedData row in UnScrape("products.jsonl", ignoreErrors: true)) Console.WriteLine(row.Get("Title")); foreach (Product p in UnScrape("products.jsonl", ignoreErrors: true)) Console.WriteLine(p.Sku); string json = someScrapedData.ToJson(); ScrapedData parsed = ScrapedData.FromJson(json); ``` `ScrapedData` and `MetaData` are both `Dictionary` subclasses — the collection initializer (`{ { "key", value } }`) and indexer (`data["key"] = value`) both work, and `Get("key")` casts a value back to its type (throws `KeyNotFoundException` if absent). Files are written under `WorkingDirectory`; prefer a typed POCO over `ScrapedData` once the shape of scraped rows stabilizes. ### Resuming and reliability ```csharp job.Start("nightly-catalog-crawl"); // Start(crawlId = "") — non-empty id makes it resumable await job.StartAsync("nightly-catalog-crawl"); job.Stop(); // graceful stop; Start(sameId) picks up where it left off job.EnableWebCache(); // cache http responses to disk during development job.EnableWebCache(TimeSpan.FromHours(2)); // ...for a bounded duration ``` Passing a non-empty `crawlId` to `Start`/`StartAsync` auto-saves crawl state every five minutes so a crash, redeploy, or power loss loses only a few minutes of progress, not the whole job — worth doing for anything that runs more than a couple of minutes. `EnableWebCache()` replays previously-fetched pages from disk instead of re-requesting them, so `Parse` logic can be changed and re-run without re-hitting the target site. In `Parse`, call `Retry(response)` to re-fetch a page that came back as a CAPTCHA or error screen rather than treating it as data. ## Deployment No native binaries or browser engine ship with this package, so deployment is largely just "can this process reach the internet and write to disk": | Environment | What to do | |---|---| | Docker / Linux / macOS / Windows | No special native deps — any base image with the .NET runtime works. | | Long-running crawls | Prefer a console app or worker service over a request/response web action — `Start()` blocks until the crawl finishes (or `StartAsync()` for async hosts), and jobs commonly run minutes to days. | | AWS Lambda / short-lived functions | A poor fit for anything beyond a tiny crawl: Lambda's execution time limit will cut off a `Start()` call mid-job. If used, give it a resumable `crawlId` and re-invoke, or move the crawl to a long-running compute target (ECS/Fargate, a VM, a worker service) instead. | | Storage | `WorkingDirectory` (scraped `.jsonl` files + `SavedState`) and the web cache (`WebCache` folder) need a writable, persistent path — don't point them at ephemeral container storage if the crawl must resume after a restart. | | Outbound network | The process needs outbound HTTP(S) to every crawled host, and to the target's `robots.txt` if `ObeyRobotsDotTxt` is set; egress-restricted networks need an allowlist or proxy. | ## When something fails | Symptom | Cause and fix | |---|---| | `LicensingException: Production License Required` | No valid licence, or the 7-day development grace period has elapsed. Set `IronWebScraper.License.LicenseKey` before `Start()`. | | `Parse` never runs on page two | `Init()` only calls `Request` once and `Parse` doesn't call `Request` again for the "next" link — following pages is the developer's own `Parse` logic, not automatic. | | A link is silently never fetched | It didn't match `AllowedUrls`/`AllowedDomains`, or matched a `BannedUrls`/`BannedDomains` pattern, or `AcceptUrl` rejected it. | | `Css`/`XPath`/`CssExists` find nothing that's visible in a browser | The content is rendered by client-side JavaScript; this library parses the HTML as returned by the server, with no browser engine — it won't see script-injected DOM. | | Getting blocked / lots of failed requests | Too aggressive for the target: raise `RateLimitPerHost`, lower `OpenConnectionLimitPerHost`/`MaxHttpConnectionLimit`, add more `Identities` (user agents/proxies), or check `ObeyRobotsDotTxt` isn't being bypassed against the site's wishes. | | Duplicate rows in scraped output | Use `ScrapeUnique` instead of `Scrape`, or de-duplicate by a key when reading back with `UnScrape`. | | Crawl loses hours of progress after a crash | Pass a stable, non-empty `crawlId` to `Start`/`StartAsync` so state auto-saves every 5 minutes and the same call resumes it. | | Re-running during development re-downloads everything | Call `EnableWebCache()` in `Init()` so previously-fetched pages replay from disk. | | Login-gated pages return the login page | The identity used for the follow-up `Request` didn't carry the session cookie — reuse the same `HttpIdentity` (with `UseCookies = true`) for the login `PostRequest` and the subsequent requests. | ## Rules - **Never invent a member.** Confirm against the XML documentation that ships in the package before writing code: `grep -o 'name="[MPF]:IronWebScraper\.[^"]*RateLimit[^"]*"' ~/.nuget/packages/ironwebscraper//lib/netstandard2.1/IronWebScraper.xml` That file is the authoritative surface for the installed version. - Keep licence keys out of source and out of terminal output. - **Scrape politely and lawfully.** Leave `ObeyRobotsDotTxt` on unless there's a specific, justified reason to turn it off; keep `RateLimitPerHost`/`OpenConnectionLimitPerHost` at levels that don't hammer a shared server; set a real `UserAgent` rather than spoofing one to evade a block; and check the target site's terms of service before scraping it — this library gives you the throttling knobs, but using them politely is a judgment call it can't make for you. - This is a static-HTML/DOM parser, not a browser: don't tell a user `Css`/`XPath` will pick up JavaScript-rendered content, and say so if a target needs a real browser instead. - `ScrapedData`/`MetaData` are dictionaries — prefer a typed POCO for `Scrape`/`UnScrape` once a scraped shape stabilizes, rather than stringly-typed keys throughout. - Give any crawl expected to run more than a couple of minutes a stable `crawlId` for resumability, and use `EnableWebCache()` while iterating on `Parse` logic. - Official docs and full API reference: . Support: support@ironsoftware.com.