Search Results for

    Show / Hide Table of Contents

    Class MetaData

    A flexible dictionary of object values which can be used to attach your own additional data or objects to any Request. Meta can contain objects of any Type including instances of classes, List and Dictionaries. This meta-data can then be accessed while Paring the Response and even passed forwards to the next Request.

    Metadata send might include pagination page numbers, referrer Urls, User Ids etc.

    E.g:
    Request["page-number"] = 2;
    int pageNumber = Response.Request.Meta.Get<int>("page-number");
    Inheritance
    System.Object
    MetaData
    Namespace: IronWebScraper
    Assembly: IronWebScraper.dll
    Syntax
    public class MetaData : Dictionary<string, object>

    MetaData is the per-request bag a scraper uses to carry its own objects from a Request through to the matching Response. It extends Dictionary<string, object>, so any value can be stored under a string key and read back when the page is parsed, which is how context such as a category, a parent record, or a page number reaches the parser.

    Store values with the dictionary indexer and read them with the typed Get<T>(key) helper, which casts the stored object for you. Attach a MetaData to a WebScraper.Request call; the resulting Response.MetaData exposes the same instance during parsing. Because the bag holds object values, a whole record or domain model can ride along rather than just strings, which keeps a multi-step crawl stateful without external storage, and the typed Get<T> removes the cast at the read site.

    var meta = new MetaData();
    meta["pageType"] = "listing";
    Request(url, Parse, meta);
    
    // later, in Parse:
    string type = response.MetaData.Get<string>("pageType");

    The blog web scraper tutorial threads context through MetaData, and the advanced tutorial uses it across multi-step crawls.

    Constructors

    MetaData()

    Declaration
    public MetaData()

    Methods

    Get<T>(String)

    Returns the specified meta object cast as the appropriate type.

    E.g:
    Get<int>("index")
    Declaration
    public T Get<T>(string key)
    Parameters
    Type Name Description
    System.String key

    Key

    Returns
    Type Description
    T
    Type Parameters
    Name Description
    T

    Type

    Exceptions
    Type Condition
    System.Collections.Generic.KeyNotFoundException
    ☀
    ☾
    Downloads
    • Download with Nuget
    • Start for Free
    In This Article
    Back to top
    Install with Nuget