Search Results for

    Show / Hide Table of Contents

    Class ScrapedData

    A flexible dictionary of object values used to conveniently store scraped data of any Type in a key-value dictionary which can be saved as JSON using the Yield method. ScrapedData can hold data objects of any Type, including Classes.

    E.g:
    var Data = new ScrapedData();
    Data['title'] = "Page Title";
    Data['date'] = DateTime.Now;
    Inheritance
    System.Object
    ScrapedData
    Namespace: IronWebScraper
    Assembly: IronWebScraper.dll
    Syntax
    public class ScrapedData : Dictionary<string, object>

    ScrapedData is the key-value record a scraper fills with extracted fields and serializes as JSON. It extends Dictionary<string, object>, so each field is stored under a string key and the whole record converts to and from JSON for saving a crawl's results.

    Set fields with the dictionary indexer, read a typed value with Get<T>(key), write the record to a JSON string with ToJson(), and rebuild one from JSON with the static ScrapedData.FromJson(json). In a WebScraper, passing an object to Scrape persists it, and UnScrape<T>() reads stored results back, so ScrapedData is the natural shape for both ends of that round trip. Because values are stored as object, a single record can mix strings, numbers, and nested data, and the JSON round trip through ToJson and FromJson makes a crawl's output easy to persist to disk and reload in a later run.

    var data = new ScrapedData();
    data["title"] = node.TextContentClean;
    data["price"] = node.GetAttribute("data-price");
    Scrape(data);

    The web scraping tutorial saves results as ScrapedData, and the C# web scraper example shows the JSON output.

    Constructors

    ScrapedData()

    Declaration
    public ScrapedData()

    Methods

    FromJson(String)

    Deserializes a ScrapedData object from JSON created using the toJson() method.

    Declaration
    public static ScrapedData FromJson(string json)
    Parameters
    Type Name Description
    System.String json
    Returns
    Type Description
    ScrapedData

    Get<T>(String)

    Returns the specified ScrapedData item cast as the appropriate type.

    E.g:
    Get<string>("url");
    Declaration
    public T Get<T>(string key)
    Parameters
    Type Name Description
    System.String key

    Key

    Returns
    Type Description
    T
    Type Parameters
    Name Description
    T

    Type

    Exceptions
    Type Condition
    System.Collections.Generic.KeyNotFoundException

    ToJson()

    Converts the ScrapedData to a JSON string.

    Declaration
    public string ToJson()
    Returns
    Type Description
    System.String
    ☀
    ☾
    Downloads
    • Download with Nuget
    • Start for Free
    In This Article
    Back to top
    Install with Nuget