Static helper class which lists common web-browser user-agent strings.
The HtmlNode class represents a single DOM element in a HTML or XML document.
Extension methods for finding elements within with IEnumerable<HtmlNode>
A class defining the browsing 'identity' to be used to fetch a given Url. Contains Proxy, UserAgent and Http Header information.
A static class providing configurationand licensing of IronWebScraper
A public static class used to apply license keys to IronWebScraper.
A flexible dictionary of object values which can be used to attach your own additional data or objects to any Request. Meta can contain objects of any Type including instances of classes, List and Dictionaries. This meta-data can then be accessed while Paring the Response and even passed forwards to the next Request.
Metadata send might include pagination page numbers, referrer Urls, User Ids etc.E.g:
Request["page-number"] = 2;
int pageNumber = Response.Request.Meta.Get<int>("page-number");
Represents a http request to be made by IronWebScraper
Represents a http response made by IronWebScraper
A flexible dictionary of object values used to conveniently store scraped data of any Type in a key-value dictionary which can be saved as JSON using the Yield method. ScrapedData can hold data objects of any Type, including Classes.
var Data = new ScrapedData();
Data['title'] = "Page Title";
Data['date'] = DateTime.Now;
An easy to use base class which developers can extend to rapidly build custom web-scraping applications.
Level of WebScraper logging to the Console. Because this Enum is a Flag type options can be combined using a pipe: e.g. LogLevel.Critical | LogLevel.ScrapedData
Throttle remote clients by their host name or by their public IP address.