Search Results for

    Show / Hide Table of Contents

    Class Request

    Represents a http request to be made by IronWebScraper

    Inheritance
    System.Object
    Request
    Namespace: IronWebScraper
    Assembly: IronWebScraper.dll
    Syntax
    public class Request : Object

    When a scraper queues a URL to fetch, it travels as a Request. A developer rarely builds one directly; calling WebScraper.Request(url, parse, ...) from Init or Parse creates the Request and adds it to the crawl queue, where the base class fetches it and hands the result to the parse method named in the call.

    A Request records what to fetch and how to handle it. Url is the target, Action is the Action<Response> parse callback to run when the page arrives, and Method reports whether the fetch is a GET or POST. Identity pins a specific HttpIdentity to this request, overriding the crawler's normal rotation, and MetaData attaches a bag of your own objects that the matching Response carries through, which is how per-request context (a category, a parent record, a page number) reaches the parser.

    The Method reflects whether the fetch is a GET, queued through Request, or a POST, queued through the related WebScraper.PostRequest overloads that send form data, and Action holds the same parse callback the response will run. Retries reports how many times the request has been re-attempted after a failure, and the static Request.Hash produces a stable key for a request, useful for de-duplicating URLs across a large crawl. Set any MetaData at queue time, since the values are captured once the request enters the crawl queue and read back only when its response is parsed.

    public override void Init()
    {
        var meta = new MetaData();
        meta["category"] = "books";
        Request("https://example.com/books", Parse, metaData: meta);
    }

    The web scraping tutorial queues requests from a parser, and the blog web scraper tutorial carries context through MetaData.

    Constructors

    Request()

    Declaration
    public Request()

    Fields

    Action

    The Action<Response> which will be used to parse the response. See IronWebScraper.Parse.

    Declaration
    public Action<Response> Action
    Field Value
    Type Description
    System.Action<Response>

    Identity

    An HttpIdentity comprising Proxy, Headers, UserAgnet etc.. which will be used to fetch the Request.

    Declaration
    public HttpIdentity Identity
    Field Value
    Type Description
    HttpIdentity

    MetaData

    Allows additional meta-data of any Type to be attached to a request, and retrieved when the subsequent Response is parsed.

    Declaration
    public MetaData MetaData
    Field Value
    Type Description
    MetaData

    Properties

    Method

    Http Request method.

    Declaration
    public string Method { get; }
    Property Value
    Type Description
    System.String

    "GET" or "POST"

    Retries

    Number of attempts which have been made to fetch content from this Request so far.

    Declaration
    public int Retries { get; }
    Property Value
    Type Description
    System.Int32

    Url

    Absolute URL to be scraped.

    Declaration
    public string Url { get; set; }
    Property Value
    Type Description
    System.String

    Methods

    Hash(Request)

    Hashing function for uniquely identifying web requests.

    Declaration
    public static string Hash(Request request)
    Parameters
    Type Name Description
    Request request

    A Request object

    Returns
    Type Description
    System.String
    ☀
    ☾
    Downloads
    • Download with Nuget
    • Start for Free
    In This Article
    Back to top
    Install with Nuget