Search Results for

    Show / Hide Table of Contents

    Class HttpIdentity

    A class defining the browsing 'identity' to be used to fetch a given Url. Contains Proxy, UserAgent and Http Header information.

    Inheritance
    System.Object
    HttpIdentity
    Namespace: IronWebScraper
    Assembly: IronWebScraper.dll
    Syntax
    public class HttpIdentity : Object

    The proxy, user agent, and credentials a scraper sends requests with are bundled into an HttpIdentity. Presenting a varied, realistic identity keeps a crawler from looking like a single robotic client, which is what most rate-limiting and blocking is tuned to catch.

    Build one with new HttpIdentity() and set the fields that matter: Proxy routes the request through a proxy server, UserAgent sets the browser string (often taken from CommonUserAgents), and NetworkUsername, NetworkPassword, and NetworkDomain supply credentials for an authenticated proxy or site. UseCookies turns on the per-identity Cookies container so a session persists across requests, and HttpRequestHeaders adds custom headers.

    With UseCookies enabled, the per-identity Cookies container persists session state across requests, so a login performed under one identity carries forward on its later fetches, and NetworkLoginCredential exposes the assembled credential. The static DefaultHttpRequestHeaders seeds the headers shared by new identities.

    Identities are used through the WebScraper.Identities list. Add several and the crawler rotates among them, calling ChooseIdentityForRequest to pick one per fetch, which spreads load across proxies and user agents. A single request can also override the rotation by passing a specific HttpIdentity to Request. ProxyFailureCount tracks how often an identity's proxy has failed so unreliable ones can be retired. Spreading work across several identities is the core technique for crawling at volume without tripping the per-client rate limits and blocks that target a single repeated address.

    Identities.Add(new HttpIdentity
    {
        Proxy = "http://10.0.0.1:8080",
        UserAgent = CommonUserAgents.ChromeDesktopUserAgents[0],
        UseCookies = true
    });

    The advanced web scraping tutorial sets up identity rotation, and the movie-site how-to uses identities against a real site.

    Constructors

    HttpIdentity()

    Declaration
    public HttpIdentity()

    Fields

    DefaultHttpRequestHeaders

    Default HTTP request headers to be assigned to every new identity.

    Declaration
    public static Dictionary<string, string> DefaultHttpRequestHeaders
    Field Value
    Type Description
    System.Collections.Generic.Dictionary<System.String, System.String>

    HttpRequestHeaders

    The default headers which will be sent with all http Requests for this identity.

    Declaration
    public Dictionary<string, string> HttpRequestHeaders
    Field Value
    Type Description
    System.Collections.Generic.Dictionary<System.String, System.String>

    NetworkDomain

    The network domain to be used for user authentication. Supports Windows, NTLM , Keroberos, Linux, BSD and Mac OS X networks.

    See also NetworkUsername, NetworkPassword

    Declaration
    public string NetworkDomain
    Field Value
    Type Description
    System.String

    NetworkPassword

    The network/http password to be used for user authentication. Supports Http , Windows networks, NTLM , Keroberos , Linux networks, BSD networks and Mac OS X networks.

    See also NetworkUsername, NetworkDomain

    Declaration
    public string NetworkPassword
    Field Value
    Type Description
    System.String

    NetworkUsername

    The network/http username to be used for user authentication. Supports Http , Windows networks, NTLM , Keroberos , Linux networks, BSD networks and Mac OS X networks.

    See also NetworkPassword, NetworkDomain

    Declaration
    public string NetworkUsername
    Field Value
    Type Description
    System.String

    Proxy

    A proxy string such as "102.192.92.192:8088" or "username:password@102.192.92.192"

    Declaration
    public string Proxy
    Field Value
    Type Description
    System.String

    UseCookies

    Stores a "Cookie Jar" which persists http cookies for this browser identity.

    Declaration
    public bool UseCookies
    Field Value
    Type Description
    System.Boolean

    UserAgent

    A default User-Agent header string. E.g: "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36".

    Find examples online at: http://www.useragentstring.com/pages/useragentstring.php

    Declaration
    public string UserAgent
    Field Value
    Type Description
    System.String

    Properties

    Cookies

    A "Cookie Jar" which persists http cookies for this browser identity.

    Declaration
    public CookieContainer Cookies { get; }
    Property Value
    Type Description
    System.Net.CookieContainer

    NetworkLoginCredential

    Returns the System.Net.NetworkCredential which will be used on for networks and http authentication (Supports: Windows / NTLM / Keroberos / Linux / BSD / Mac)

    Declaration
    public NetworkCredential NetworkLoginCredential { get; }
    Property Value
    Type Description
    System.Net.NetworkCredential

    ProxyFailureCount

    Gets the number of times this proxy has failed.

    Declaration
    public int ProxyFailureCount { get; }
    Property Value
    Type Description
    System.Int32

    The proxy failure count.

    ☀
    ☾
    Downloads
    • Download with Nuget
    • Start for Free
    In This Article
    Back to top
    Install with Nuget