Class Request
Represents a http request to be made by IronWebScraper
Inheritance
Namespace: IronWebScraper
Assembly: IronWebScraper.dll
Syntax
public class Request : Object
When a scraper queues a URL to fetch, it travels as a Request. A developer rarely builds one directly; calling WebScraper.Request(url, parse, ...) from Init or Parse creates the Request and adds it to the crawl queue, where the base class fetches it and hands the result to the parse method named in the call.
A Request records what to fetch and how to handle it. Url is the target, Action is the Action<Response> parse callback to run when the page arrives, and Method reports whether the fetch is a GET or POST. Identity pins a specific HttpIdentity to this request, overriding the crawler's normal rotation, and MetaData attaches a bag of your own objects that the matching Response carries through, which is how per-request context (a category, a parent record, a page number) reaches the parser.
The Method reflects whether the fetch is a GET, queued through Request, or a POST, queued through the related WebScraper.PostRequest overloads that send form data, and Action holds the same parse callback the response will run. Retries reports how many times the request has been re-attempted after a failure, and the static Request.Hash produces a stable key for a request, useful for de-duplicating URLs across a large crawl. Set any MetaData at queue time, since the values are captured once the request enters the crawl queue and read back only when its response is parsed.
public override void Init()
{
var meta = new MetaData();
meta["category"] = "books";
Request("https://example.com/books", Parse, metaData: meta);
}The web scraping tutorial queues requests from a parser, and the blog web scraper tutorial carries context through MetaData.
Constructors
Request()
Declaration
public Request()
Fields
Action
The Action<Response> which will be used to parse the response. See IronWebScraper.Parse.
Declaration
public Action<Response> Action
Field Value
| Type | Description |
|---|---|
| System.Action<Response> |
Identity
An HttpIdentity comprising Proxy, Headers, UserAgnet etc.. which will be used to fetch the Request.
Declaration
public HttpIdentity Identity
Field Value
| Type | Description |
|---|---|
| HttpIdentity |
MetaData
Allows additional meta-data of any Type to be attached to a request, and retrieved when the subsequent Response is parsed.
Declaration
public MetaData MetaData
Field Value
| Type | Description |
|---|---|
| MetaData |
Properties
Method
Http Request method.
Declaration
public string Method { get; }
Property Value
| Type | Description |
|---|---|
| System.String | "GET" or "POST" |
Retries
Number of attempts which have been made to fetch content from this Request so far.
Declaration
public int Retries { get; }
Property Value
| Type | Description |
|---|---|
| System.Int32 |
Url
Absolute URL to be scraped.
Declaration
public string Url { get; set; }
Property Value
| Type | Description |
|---|---|
| System.String |
Methods
Hash(Request)
Hashing function for uniquely identifying web requests.
Declaration
public static string Hash(Request request)
Parameters
| Type | Name | Description |
|---|---|---|
| Request | request | A Request object |
Returns
| Type | Description |
|---|---|
| System.String |