Search Results for

    Show / Hide Table of Contents

    Class Response

    Represents a http response made by IronWebScraper

    Inheritance
    System.Object
    Response
    Namespace: IronWebScraper
    Assembly: IronWebScraper.dll
    Syntax
    public class Response : Object

    Response carries the result of a fetched URL, including its content, status, and parsed DOM, into a scraper's Parse method. The WebScraper base hands one Response to Parse for every page it downloads, so this is the object a developer reads from to extract data and decide which links to follow next.

    Content arrives in several forms. Html and TextContent give the raw markup and stripped text, BinaryContent gives the bytes for non-HTML resources, and Document exposes the parsed root as an HtmlNode. For targeted extraction, query helpers run selectors directly against the response: Css and QuerySelectorAll return matching HtmlNode arrays, GetElementById and QuerySelector return a single node, and XPath runs an XPath expression. CssExists tests for a match without returning nodes.

    When the page structure is unknown, Document gives the parsed root node for manual walking, and CssExists tests for a selector match before extraction is attempted. Request context travels with the response. StatusCode and WasSuccessful report the outcome, FinalUrl reflects redirects, and Request links back to the originating Request. The MetaData carried from that request rides along on the response, so context queued with the fetch is available while parsing. The ToAbsoluteUrl helpers turn the relative links found on a page into absolute URLs ready to pass to a follow-up Request. Read the values inside Parse; the response is a snapshot of one fetch.

    public override void Parse(Response response)
    {
        if (!response.WasSuccessful) return;
        foreach (HtmlNode link in response.Css("a.product"))
            Request(response.ToAbsoluteUrl(link.GetAttribute("href")), ParseProduct);
    }

    The web scraping tutorial reads responses end to end, and the shopping-site how-to shows selector queries against real pages.

    Constructors

    Response()

    Declaration
    public Response()

    Fields

    CharSet

    Name of the charset used to encode the web resource. When reading TextConent, the result has already been decoded to C# default encoding (utf-8) on your behalf.

    Declaration
    public string CharSet
    Field Value
    Type Description
    System.String

    CretaedFromWebCache

    True if the Response was created from a cached version of the Url rather than from live data. See WebScraper.EnableWebCache() method, which is can be called from within the WebScraper.Init() method.

    Declaration
    public bool CretaedFromWebCache
    Field Value
    Type Description
    System.Boolean

    FinalUrl

    The Url from which the Response was returned. This may differ from RequestlUrl if these was an http redirect.

    Declaration
    public string FinalUrl
    Field Value
    Type Description
    System.String

    MimeType

    The MIME type of the Response content. E.g: "text/html"

    Declaration
    public string MimeType
    Field Value
    Type Description
    System.String

    Request

    The Request from which the Response was created.

    Declaration
    public Request Request
    Field Value
    Type Description
    Request

    StatusCode

    The http status code given by the url which as 200 or 404. 0 is returned in case the server is uncontactable.

    Declaration
    public int StatusCode
    Field Value
    Type Description
    System.Int32

    Properties

    BinaryContent

    The content downloaded from the Url as raw binary data

    Declaration
    public byte[] BinaryContent { get; }
    Property Value
    Type Description
    System.Byte[]

    Document

    Returns an HtmlNode for the entire Response Document. (HTML/XML)

    Declaration
    public HtmlNode Document { get; }
    Property Value
    Type Description
    HtmlNode

    Html

    Synonym of content which checks for an Html content

    Declaration
    public string Html { get; }
    Property Value
    Type Description
    System.String

    The HTML.

    Exceptions
    Type Condition
    System.FormatException

    Document does not have an HTML mime type or contain a <body> or <html> tag

    MetaData

    Returns additional meta-data which was attached to the Request.

    Declaration
    public MetaData MetaData { get; }
    Property Value
    Type Description
    MetaData

    RequestlUrl

    The Url from which the Response was requested. This may differ from FinalUrl if these was an http redirect.

    Declaration
    public string RequestlUrl { get; }
    Property Value
    Type Description
    System.String

    TextContent

    The content downloaded from the Url encoded as a String.

    Declaration
    public string TextContent { get; }
    Property Value
    Type Description
    System.String

    WasSuccessful

    Gets a value indicating whether this HTTP request yielded a successful response code.

    Declaration
    public bool WasSuccessful { get; }
    Property Value
    Type Description
    System.Boolean

    true if the response was successful; otherwise, false.

    Methods

    Css(String)

    Uses CSS selectors to find all matching nodes within the Response Document. This works in the same way as $('.ClassName') in jQuery or querySelectorAll() in JavaScript.

    Declaration
    public HtmlNode[] Css(string selector)
    Parameters
    Type Name Description
    System.String selector
    Returns
    Type Description
    HtmlNode[]

    CssExists(String)

    Uses CSS selectors to find if there are any matching nodes within the Response Document. This works in the same way as
    $('.ClassName').length > 0 in jQuery or querySelectorAll().length > 0 in JavaScript.

    Declaration
    public bool CssExists(string selector)
    Parameters
    Type Name Description
    System.String selector
    Returns
    Type Description
    System.Boolean

    GetElementById(String)

    Synonym of JavaScript's getElementById function. Searches inside the response as an HTML or XML Document by ID attribute.

    Declaration
    public HtmlNode GetElementById(string id)
    Parameters
    Type Name Description
    System.String id
    Returns
    Type Description
    HtmlNode

    GetElementsByTagName(String)

    Synonym of JavaScript's getElementById function. Searches inside the response as an HTML or XML Document by tag-name such as "a" or "img".

    Declaration
    public HtmlNode[] GetElementsByTagName(string tagName)
    Parameters
    Type Name Description
    System.String tagName
    Returns
    Type Description
    HtmlNode[]

    QuerySelector(String)

    Synonym of JavaScript's querySelector function. Searches inside the Response document as an HTML or XML Document using CSS selectors.

    Declaration
    public HtmlNode QuerySelector(string selector)
    Parameters
    Type Name Description
    System.String selector
    Returns
    Type Description
    HtmlNode

    QuerySelectorAll(String)

    Synonym of JavaScript's querySelectorAll function. Searches inside the response document as an HTML or XML Document using CSS selectors.

    Declaration
    public HtmlNode[] QuerySelectorAll(string selector)
    Parameters
    Type Name Description
    System.String selector
    Returns
    Type Description
    HtmlNode[]

    ToAbsoluteUrl(IEnumerable<String>)

    Makes any relative url strings absolute relative to this Response Document.

    Declaration
    public IEnumerable<string> ToAbsoluteUrl(IEnumerable<string> urls)
    Parameters
    Type Name Description
    System.Collections.Generic.IEnumerable<System.String> urls
    Returns
    Type Description
    System.Collections.Generic.IEnumerable<System.String>

    ToAbsoluteUrl(String)

    Makes a relative url strings absolute relative to this Response Document.

    Declaration
    public string ToAbsoluteUrl(string url)
    Parameters
    Type Name Description
    System.String url
    Returns
    Type Description
    System.String

    XPath(String)

    Searches inside the response as an HTML/XML Document using an XPath expression.

    Declaration
    public HtmlNode[] XPath(string expression)
    Parameters
    Type Name Description
    System.String expression
    Returns
    Type Description
    HtmlNode[]
    ☀
    ☾
    Downloads
    • Download with Nuget
    • Start for Free
    In This Article
    Back to top
    Install with Nuget