Class Response
Represents a http response made by IronWebScraper
Inheritance
System.Object
Response
Assembly: IronWebScraper.dll
Syntax
public class Response : Object
Response carries the result of a fetched URL, including its content, status, and parsed DOM, into a scraper's Parse method. The WebScraper base hands one Response to Parse for every page it downloads, so this is the object a developer reads from to extract data and decide which links to follow next.
Content arrives in several forms. Html and TextContent give the raw markup and stripped text, BinaryContent gives the bytes for non-HTML resources, and Document exposes the parsed root as an HtmlNode. For targeted extraction, query helpers run selectors directly against the response: Css and QuerySelectorAll return matching HtmlNode arrays, GetElementById and QuerySelector return a single node, and XPath runs an XPath expression. CssExists tests for a match without returning nodes.
When the page structure is unknown, Document gives the parsed root node for manual walking, and CssExists tests for a selector match before extraction is attempted. Request context travels with the response. StatusCode and WasSuccessful report the outcome, FinalUrl reflects redirects, and Request links back to the originating Request. The MetaData carried from that request rides along on the response, so context queued with the fetch is available while parsing. The ToAbsoluteUrl helpers turn the relative links found on a page into absolute URLs ready to pass to a follow-up Request. Read the values inside Parse; the response is a snapshot of one fetch.
public override void Parse(Response response)
{
if (!response.WasSuccessful) return;
foreach (HtmlNode link in response.Css("a.product"))
Request(response.ToAbsoluteUrl(link.GetAttribute("href")), ParseProduct);
}
The web scraping tutorial reads responses end to end, and the shopping-site how-to shows selector queries against real pages.
Constructors
Response()
Declaration
Fields
CharSet
Name of the charset used to encode the web resource.
When reading TextConent, the result has already been decoded
to C# default encoding (utf-8) on your behalf.
Declaration
Field Value
| Type |
Description |
| System.String |
|
CretaedFromWebCache
True if the Response was created from a cached version of the Url
rather than from live data. See WebScraper.EnableWebCache() method,
which is can be called from within the WebScraper.Init() method.
Declaration
public bool CretaedFromWebCache
Field Value
| Type |
Description |
| System.Boolean |
|
FinalUrl
The Url from which the Response was returned. This may differ
from RequestlUrl if these was an http redirect.
Declaration
Field Value
| Type |
Description |
| System.String |
|
MimeType
The MIME type of the Response content. E.g: "text/html"
Declaration
Field Value
| Type |
Description |
| System.String |
|
Request
The Request from which the Response was created.
Declaration
Field Value
StatusCode
The http status code given by the url which as 200 or 404. 0 is
returned in case the server is uncontactable.
Declaration
Field Value
| Type |
Description |
| System.Int32 |
|
Properties
BinaryContent
The content downloaded from the Url as raw binary data
Declaration
public byte[] BinaryContent { get; }
Property Value
| Type |
Description |
| System.Byte[] |
|
Document
Returns an HtmlNode for the entire Response Document. (HTML/XML)
Declaration
public HtmlNode Document { get; }
Property Value
Html
Synonym of content which checks for an Html content
Declaration
public string Html { get; }
Property Value
| Type |
Description |
| System.String |
The HTML.
|
Exceptions
| Type |
Condition |
| System.FormatException |
Document does not have an HTML
mime type or contain a <body> or <html> tag
|
Returns additional meta-data which was attached to the Request.
Declaration
public MetaData MetaData { get; }
Property Value
RequestlUrl
The Url from which the Response was requested. This may differ
from FinalUrl if these was an http redirect.
Declaration
public string RequestlUrl { get; }
Property Value
| Type |
Description |
| System.String |
|
TextContent
The content downloaded from the Url encoded as a String.
Declaration
public string TextContent { get; }
Property Value
| Type |
Description |
| System.String |
|
WasSuccessful
Gets a value indicating whether this HTTP request yielded a
successful response code.
Declaration
public bool WasSuccessful { get; }
Property Value
| Type |
Description |
| System.Boolean |
true if the response was successful; otherwise, false.
|
Methods
Css(String)
Uses CSS selectors to find all matching nodes within the Response
Document. This works in the same way as $('.ClassName') in jQuery
or querySelectorAll() in JavaScript.
Declaration
public HtmlNode[] Css(string selector)
Parameters
| Type |
Name |
Description |
| System.String |
selector |
|
Returns
CssExists(String)
Uses CSS selectors to find if there are any matching nodes within
the Response Document. This works in the same way as
$('.ClassName').length > 0 in jQuery or
querySelectorAll().length > 0 in JavaScript.
Declaration
public bool CssExists(string selector)
Parameters
| Type |
Name |
Description |
| System.String |
selector |
|
Returns
| Type |
Description |
| System.Boolean |
|
GetElementById(String)
Synonym of JavaScript's getElementById function. Searches
inside the response as an HTML or XML Document by ID attribute.
Declaration
public HtmlNode GetElementById(string id)
Parameters
| Type |
Name |
Description |
| System.String |
id |
|
Returns
GetElementsByTagName(String)
Synonym of JavaScript's getElementById function. Searches
inside the response as an HTML or XML Document by tag-name
such as "a" or "img".
Declaration
public HtmlNode[] GetElementsByTagName(string tagName)
Parameters
| Type |
Name |
Description |
| System.String |
tagName |
|
Returns
QuerySelector(String)
Synonym of JavaScript's querySelector function. Searches inside
the Response document as an HTML or XML Document using CSS selectors.
Declaration
public HtmlNode QuerySelector(string selector)
Parameters
| Type |
Name |
Description |
| System.String |
selector |
|
Returns
QuerySelectorAll(String)
Synonym of JavaScript's querySelectorAll function. Searches
inside the response document as an HTML or XML Document using
CSS selectors.
Declaration
public HtmlNode[] QuerySelectorAll(string selector)
Parameters
| Type |
Name |
Description |
| System.String |
selector |
|
Returns
ToAbsoluteUrl(IEnumerable<String>)
Makes any relative url strings absolute relative to this
Response Document.
Declaration
public IEnumerable<string> ToAbsoluteUrl(IEnumerable<string> urls)
Parameters
| Type |
Name |
Description |
| System.Collections.Generic.IEnumerable<System.String> |
urls |
|
Returns
| Type |
Description |
| System.Collections.Generic.IEnumerable<System.String> |
|
ToAbsoluteUrl(String)
Makes a relative url strings absolute relative to this
Response Document.
Declaration
public string ToAbsoluteUrl(string url)
Parameters
| Type |
Name |
Description |
| System.String |
url |
|
Returns
| Type |
Description |
| System.String |
|
XPath(String)
Searches inside the response as an HTML/XML Document using an
XPath expression.
Declaration
public HtmlNode[] XPath(string expression)
Parameters
| Type |
Name |
Description |
| System.String |
expression |
|
Returns