Class HttpIdentity
A class defining the browsing 'identity' to be used to fetch a given Url. Contains Proxy, UserAgent and Http Header information.
Inheritance
Namespace: IronWebScraper
Assembly: IronWebScraper.dll
Syntax
public class HttpIdentity : Object
The proxy, user agent, and credentials a scraper sends requests with are bundled into an HttpIdentity. Presenting a varied, realistic identity keeps a crawler from looking like a single robotic client, which is what most rate-limiting and blocking is tuned to catch.
Build one with new HttpIdentity() and set the fields that matter: Proxy routes the request through a proxy server, UserAgent sets the browser string (often taken from CommonUserAgents), and NetworkUsername, NetworkPassword, and NetworkDomain supply credentials for an authenticated proxy or site. UseCookies turns on the per-identity Cookies container so a session persists across requests, and HttpRequestHeaders adds custom headers.
With UseCookies enabled, the per-identity Cookies container persists session state across requests, so a login performed under one identity carries forward on its later fetches, and NetworkLoginCredential exposes the assembled credential. The static DefaultHttpRequestHeaders seeds the headers shared by new identities.
Identities are used through the WebScraper.Identities list. Add several and the crawler rotates among them, calling ChooseIdentityForRequest to pick one per fetch, which spreads load across proxies and user agents. A single request can also override the rotation by passing a specific HttpIdentity to Request. ProxyFailureCount tracks how often an identity's proxy has failed so unreliable ones can be retired. Spreading work across several identities is the core technique for crawling at volume without tripping the per-client rate limits and blocks that target a single repeated address.
Identities.Add(new HttpIdentity
{
Proxy = "http://10.0.0.1:8080",
UserAgent = CommonUserAgents.ChromeDesktopUserAgents[0],
UseCookies = true
});The advanced web scraping tutorial sets up identity rotation, and the movie-site how-to uses identities against a real site.
Constructors
HttpIdentity()
Declaration
public HttpIdentity()
Fields
DefaultHttpRequestHeaders
Default HTTP request headers to be assigned to every new identity.
Declaration
public static Dictionary<string, string> DefaultHttpRequestHeaders
Field Value
| Type | Description |
|---|---|
| System.Collections.Generic.Dictionary<System.String, System.String> |
HttpRequestHeaders
The default headers which will be sent with all http Requests for this identity.
Declaration
public Dictionary<string, string> HttpRequestHeaders
Field Value
| Type | Description |
|---|---|
| System.Collections.Generic.Dictionary<System.String, System.String> |
NetworkDomain
The network domain to be used for user authentication. Supports Windows, NTLM , Keroberos, Linux, BSD and Mac OS X networks.
See also NetworkUsername, NetworkPassword
Declaration
public string NetworkDomain
Field Value
| Type | Description |
|---|---|
| System.String |
NetworkPassword
The network/http password to be used for user authentication. Supports Http , Windows networks, NTLM , Keroberos , Linux networks, BSD networks and Mac OS X networks.
See also NetworkUsername, NetworkDomain
Declaration
public string NetworkPassword
Field Value
| Type | Description |
|---|---|
| System.String |
NetworkUsername
The network/http username to be used for user authentication. Supports Http , Windows networks, NTLM , Keroberos , Linux networks, BSD networks and Mac OS X networks.
See also NetworkPassword, NetworkDomain
Declaration
public string NetworkUsername
Field Value
| Type | Description |
|---|---|
| System.String |
Proxy
A proxy string such as "102.192.92.192:8088" or "username:password@102.192.92.192"
Declaration
public string Proxy
Field Value
| Type | Description |
|---|---|
| System.String |
UseCookies
Stores a "Cookie Jar" which persists http cookies for this browser identity.
Declaration
public bool UseCookies
Field Value
| Type | Description |
|---|---|
| System.Boolean |
UserAgent
A default User-Agent header string. E.g: "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36".
Find examples online at: http://www.useragentstring.com/pages/useragentstring.php
Declaration
public string UserAgent
Field Value
| Type | Description |
|---|---|
| System.String |
Properties
Cookies
A "Cookie Jar" which persists http cookies for this browser identity.
Declaration
public CookieContainer Cookies { get; }
Property Value
| Type | Description |
|---|---|
| System.Net.CookieContainer |
NetworkLoginCredential
Returns the System.Net.NetworkCredential which will be used on for networks and http authentication (Supports: Windows / NTLM / Keroberos / Linux / BSD / Mac)
Declaration
public NetworkCredential NetworkLoginCredential { get; }
Property Value
| Type | Description |
|---|---|
| System.Net.NetworkCredential |
ProxyFailureCount
Gets the number of times this proxy has failed.
Declaration
public int ProxyFailureCount { get; }
Property Value
| Type | Description |
|---|---|
| System.Int32 | The proxy failure count. |