Advanced Webscraping Features in C#

Updated: May 9, 2026

HttpIdentity Feature

Some website systems require the user to be logged in to view the content; in this case, we can use an HttpIdentity. Here is how you can set it up:

// Create a new instance of HttpIdentity
HttpIdentity id = new HttpIdentity();

// Set the network username and password for authentication
id.NetworkUsername = "username";
id.NetworkPassword = "pwd";

// Add the identity to the collection of identities
Identities.Add(id);

// Create a new instance of HttpIdentity
HttpIdentity id = new HttpIdentity();

// Set the network username and password for authentication
id.NetworkUsername = "username";
id.NetworkPassword = "pwd";

// Add the identity to the collection of identities
Identities.Add(id);

' Create a new instance of HttpIdentity
Dim id As New HttpIdentity()

' Set the network username and password for authentication
id.NetworkUsername = "username"
id.NetworkPassword = "pwd"

' Add the identity to the collection of identities
Identities.Add(id)

' Create a new instance of HttpIdentity
Dim id As New HttpIdentity()

' Set the network username and password for authentication
id.NetworkUsername = "username"
id.NetworkPassword = "pwd"

' Add the identity to the collection of identities
Identities.Add(id)

One of the most impressive and powerful features in IronWebScraper is the ability to use thousands of unique user credentials and/or browser engines to spoof or scrape websites using multiple login sessions.

public override void Init()
{
    // Set the license key for IronWebScraper
    License.LicenseKey = "LicenseKey";

    // Set the logging level to capture all logs
    this.LoggingLevel = WebScraper.LogLevel.All;

    // Assign the working directory for the output files
    this.WorkingDirectory = AppSetting.GetAppRoot() + @"\ShoppingSiteSample\Output\";

    // Define an array of proxies
    var proxies = "IP-Proxy1:8080,IP-Proxy2:8081".Split(',');

    // Iterate over common Chrome desktop user agents
    foreach (var UA in IronWebScraper.CommonUserAgents.ChromeDesktopUserAgents)
    {
        // Iterate over the proxies
        foreach (var proxy in proxies)
        {
            // Add a new HTTP identity with specific user agent and proxy
            Identities.Add(new HttpIdentity()
            {
                UserAgent = UA,
                UseCookies = true,
                Proxy = proxy
            });
        }
    }
    
    // Make an initial request to the website with a parse method
    this.Request("http://www.Website.com", Parse);
}

public override void Init()
{
    // Set the license key for IronWebScraper
    License.LicenseKey = "LicenseKey";

    // Set the logging level to capture all logs
    this.LoggingLevel = WebScraper.LogLevel.All;

    // Assign the working directory for the output files
    this.WorkingDirectory = AppSetting.GetAppRoot() + @"\ShoppingSiteSample\Output\";

    // Define an array of proxies
    var proxies = "IP-Proxy1:8080,IP-Proxy2:8081".Split(',');

    // Iterate over common Chrome desktop user agents
    foreach (var UA in IronWebScraper.CommonUserAgents.ChromeDesktopUserAgents)
    {
        // Iterate over the proxies
        foreach (var proxy in proxies)
        {
            // Add a new HTTP identity with specific user agent and proxy
            Identities.Add(new HttpIdentity()
            {
                UserAgent = UA,
                UseCookies = true,
                Proxy = proxy
            });
        }
    }
    
    // Make an initial request to the website with a parse method
    this.Request("http://www.Website.com", Parse);
}

Public Overrides Sub Init()
    ' Set the license key for IronWebScraper
    License.LicenseKey = "LicenseKey"

    ' Set the logging level to capture all logs
    Me.LoggingLevel = WebScraper.LogLevel.All

    ' Assign the working directory for the output files
    Me.WorkingDirectory = AppSetting.GetAppRoot() & "\ShoppingSiteSample\Output\"

    ' Define an array of proxies
    Dim proxies = "IP-Proxy1:8080,IP-Proxy2:8081".Split(","c)

    ' Iterate over common Chrome desktop user agents
    For Each UA In IronWebScraper.CommonUserAgents.ChromeDesktopUserAgents
        ' Iterate over the proxies
        For Each proxy In proxies
            ' Add a new HTTP identity with specific user agent and proxy
            Identities.Add(New HttpIdentity() With {
                .UserAgent = UA,
                .UseCookies = True,
                .Proxy = proxy
            })
        Next
    Next

    ' Make an initial request to the website with a parse method
    Me.Request("http://www.Website.com", AddressOf Parse)
End Sub

Public Overrides Sub Init()
    ' Set the license key for IronWebScraper
    License.LicenseKey = "LicenseKey"

    ' Set the logging level to capture all logs
    Me.LoggingLevel = WebScraper.LogLevel.All

    ' Assign the working directory for the output files
    Me.WorkingDirectory = AppSetting.GetAppRoot() & "\ShoppingSiteSample\Output\"

    ' Define an array of proxies
    Dim proxies = "IP-Proxy1:8080,IP-Proxy2:8081".Split(","c)

    ' Iterate over common Chrome desktop user agents
    For Each UA In IronWebScraper.CommonUserAgents.ChromeDesktopUserAgents
        ' Iterate over the proxies
        For Each proxy In proxies
            ' Add a new HTTP identity with specific user agent and proxy
            Identities.Add(New HttpIdentity() With {
                .UserAgent = UA,
                .UseCookies = True,
                .Proxy = proxy
            })
        Next
    Next

    ' Make an initial request to the website with a parse method
    Me.Request("http://www.Website.com", AddressOf Parse)
End Sub

You have multiple properties to give you different behaviors, preventing websites from blocking you.

Some of these properties include:

NetworkDomain: The network domain to be used for user authentication. Supports Windows, NTLM, Kerberos, Linux, BSD, and Mac OS X networks. Must be used with NetworkUsername and NetworkPassword.
NetworkUsername: The network/http username to be used for user authentication. Supports HTTP, Windows networks, NTLM, Kerberos, Linux networks, BSD networks, and Mac OS.
NetworkPassword: The network/http password to be used for user authentication. Supports HTTP, Windows networks, NTLM, Kerberos, Linux networks, BSD networks, and Mac OS.
Proxy: To set proxy settings.
UserAgent: To set a browser engine (e.g., Chrome desktop, Chrome mobile, Chrome tablet, IE, and Firefox, etc.).
HttpRequestHeaders: For custom header values that will be used with this identity, it accepts a dictionary object Dictionary<string, string>.
UseCookies: Enable/disable using cookies.

IronWebScraper runs the scraper using random identities. If we need to specify the use of a specific identity to parse a page, we can do so:

public override void Init()
{
    // Set the license key for IronWebScraper
    License.LicenseKey = "LicenseKey";

    // Set the logging level to capture all logs
    this.LoggingLevel = WebScraper.LogLevel.All;

    // Assign the working directory for the output files
    this.WorkingDirectory = AppSetting.GetAppRoot() + @"\ShoppingSiteSample\Output\";

    // Create a new instance of HttpIdentity
    HttpIdentity identity = new HttpIdentity();
    
    // Set the network username and password for authentication
    identity.NetworkUsername = "username";
    identity.NetworkPassword = "pwd";
    
    // Add the identity to the collection of identities
    Identities.Add(identity);
    
    // Make a request to the website with the specified identity
    this.Request("http://www.Website.com", Parse, identity);
}

public override void Init()
{
    // Set the license key for IronWebScraper
    License.LicenseKey = "LicenseKey";

    // Set the logging level to capture all logs
    this.LoggingLevel = WebScraper.LogLevel.All;

    // Assign the working directory for the output files
    this.WorkingDirectory = AppSetting.GetAppRoot() + @"\ShoppingSiteSample\Output\";

    // Create a new instance of HttpIdentity
    HttpIdentity identity = new HttpIdentity();
    
    // Set the network username and password for authentication
    identity.NetworkUsername = "username";
    identity.NetworkPassword = "pwd";
    
    // Add the identity to the collection of identities
    Identities.Add(identity);
    
    // Make a request to the website with the specified identity
    this.Request("http://www.Website.com", Parse, identity);
}

Public Overrides Sub Init()
    ' Set the license key for IronWebScraper
    License.LicenseKey = "LicenseKey"

    ' Set the logging level to capture all logs
    Me.LoggingLevel = WebScraper.LogLevel.All

    ' Assign the working directory for the output files
    Me.WorkingDirectory = AppSetting.GetAppRoot() & "\ShoppingSiteSample\Output\"

    ' Create a new instance of HttpIdentity
    Dim identity As New HttpIdentity()
    
    ' Set the network username and password for authentication
    identity.NetworkUsername = "username"
    identity.NetworkPassword = "pwd"
    
    ' Add the identity to the collection of identities
    Identities.Add(identity)
    
    ' Make a request to the website with the specified identity
    Me.Request("http://www.Website.com", AddressOf Parse, identity)
End Sub

Public Overrides Sub Init()
    ' Set the license key for IronWebScraper
    License.LicenseKey = "LicenseKey"

    ' Set the logging level to capture all logs
    Me.LoggingLevel = WebScraper.LogLevel.All

    ' Assign the working directory for the output files
    Me.WorkingDirectory = AppSetting.GetAppRoot() & "\ShoppingSiteSample\Output\"

    ' Create a new instance of HttpIdentity
    Dim identity As New HttpIdentity()
    
    ' Set the network username and password for authentication
    identity.NetworkUsername = "username"
    identity.NetworkPassword = "pwd"
    
    ' Add the identity to the collection of identities
    Identities.Add(identity)
    
    ' Make a request to the website with the specified identity
    Me.Request("http://www.Website.com", AddressOf Parse, identity)
End Sub

Enable the Web Cache Feature

This feature is used to cache requested pages. It is often used in the development and testing phases, enabling developers to cache required pages for reuse after updating code. This enables you to execute your code on cached pages after restarting your web scraper without needing to connect to the live website every time (action-replay).

You can use it in the Init() method:

// Enable web cache without an expiration time
EnableWebCache();

// OR enable web cache with a specified expiration time
EnableWebCache(new TimeSpan(1, 30, 30));

// Enable web cache without an expiration time
EnableWebCache();

// OR enable web cache with a specified expiration time
EnableWebCache(new TimeSpan(1, 30, 30));

' Enable web cache without an expiration time
EnableWebCache()

' OR enable web cache with a specified expiration time
EnableWebCache(New TimeSpan(1, 30, 30))

' Enable web cache without an expiration time
EnableWebCache()

' OR enable web cache with a specified expiration time
EnableWebCache(New TimeSpan(1, 30, 30))

It will save your cached data to the WebCache folder under the working directory folder.

public override void Init()
{
    // Set the license key for IronWebScraper
    License.LicenseKey = "LicenseKey";

    // Set the logging level to capture all logs
    this.LoggingLevel = WebScraper.LogLevel.All;

    // Assign the working directory for the output files
    this.WorkingDirectory = AppSetting.GetAppRoot() + @"\ShoppingSiteSample\Output\";

    // Enable web cache with a specific expiration time of 1 hour, 30 minutes, and 30 seconds
    EnableWebCache(new TimeSpan(1, 30, 30));
    
    // Make an initial request to the website with a parse method
    this.Request("http://www.Website.com", Parse);
}

public override void Init()
{
    // Set the license key for IronWebScraper
    License.LicenseKey = "LicenseKey";

    // Set the logging level to capture all logs
    this.LoggingLevel = WebScraper.LogLevel.All;

    // Assign the working directory for the output files
    this.WorkingDirectory = AppSetting.GetAppRoot() + @"\ShoppingSiteSample\Output\";

    // Enable web cache with a specific expiration time of 1 hour, 30 minutes, and 30 seconds
    EnableWebCache(new TimeSpan(1, 30, 30));
    
    // Make an initial request to the website with a parse method
    this.Request("http://www.Website.com", Parse);
}

Public Overrides Sub Init()
    ' Set the license key for IronWebScraper
    License.LicenseKey = "LicenseKey"

    ' Set the logging level to capture all logs
    Me.LoggingLevel = WebScraper.LogLevel.All

    ' Assign the working directory for the output files
    Me.WorkingDirectory = AppSetting.GetAppRoot() & "\ShoppingSiteSample\Output\"

    ' Enable web cache with a specific expiration time of 1 hour, 30 minutes, and 30 seconds
    EnableWebCache(New TimeSpan(1, 30, 30))

    ' Make an initial request to the website with a parse method
    Me.Request("http://www.Website.com", Parse)
End Sub

Public Overrides Sub Init()
    ' Set the license key for IronWebScraper
    License.LicenseKey = "LicenseKey"

    ' Set the logging level to capture all logs
    Me.LoggingLevel = WebScraper.LogLevel.All

    ' Assign the working directory for the output files
    Me.WorkingDirectory = AppSetting.GetAppRoot() & "\ShoppingSiteSample\Output\"

    ' Enable web cache with a specific expiration time of 1 hour, 30 minutes, and 30 seconds
    EnableWebCache(New TimeSpan(1, 30, 30))

    ' Make an initial request to the website with a parse method
    Me.Request("http://www.Website.com", Parse)
End Sub

IronWebScraper also has features to enable your engine to continue scraping after restarting the code by setting the engine start process name using Start(CrawlID).

static void Main(string[] args)
{
    // Create an object from the Scraper class
    EngineScraper scrape = new EngineScraper();
    
    // Start the scraping process with the specified crawl ID
    scrape.Start("enginestate");
}

static void Main(string[] args)
{
    // Create an object from the Scraper class
    EngineScraper scrape = new EngineScraper();
    
    // Start the scraping process with the specified crawl ID
    scrape.Start("enginestate");
}

Sub Main(ByVal args As String())
    ' Create an object from the Scraper class
    Dim scrape As New EngineScraper()

    ' Start the scraping process with the specified crawl ID
    scrape.Start("enginestate")
End Sub

Sub Main(ByVal args As String())
    ' Create an object from the Scraper class
    Dim scrape As New EngineScraper()

    ' Start the scraping process with the specified crawl ID
    scrape.Start("enginestate")
End Sub

The execution request and response will be saved in the SavedState folder inside the working directory.

Throttling

We can control the minimum and maximum connection numbers and connection speed per domain.

public override void Init()
{
    // Set the license key for IronWebScraper
    License.LicenseKey = "LicenseKey";

    // Set the logging level to capture all logs
    this.LoggingLevel = WebScraper.LogLevel.All;

    // Assign the working directory for the output files
    this.WorkingDirectory = AppSetting.GetAppRoot() + @"\ShoppingSiteSample\Output\";

    // Set the total number of allowed open HTTP requests (threads)
    this.MaxHttpConnectionLimit = 80;
    
    // Set minimum polite delay (pause) between requests to a given domain or IP address
    this.RateLimitPerHost = TimeSpan.FromMilliseconds(50);
    
    // Set the allowed number of concurrent HTTP requests (threads) per hostname or IP address
    this.OpenConnectionLimitPerHost = 25;
    
    // Do not obey the robots.txt files
    this.ObeyRobotsDotTxt = false;
    
    // Makes the WebScraper intelligently throttle requests not only by hostname, but also by host servers' IP addresses
    this.ThrottleMode = Throttle.ByDomainHostName;
    
    // Make an initial request to the website with a parse method
    this.Request("https://www.Website.com", Parse);
}

public override void Init()
{
    // Set the license key for IronWebScraper
    License.LicenseKey = "LicenseKey";

    // Set the logging level to capture all logs
    this.LoggingLevel = WebScraper.LogLevel.All;

    // Assign the working directory for the output files
    this.WorkingDirectory = AppSetting.GetAppRoot() + @"\ShoppingSiteSample\Output\";

    // Set the total number of allowed open HTTP requests (threads)
    this.MaxHttpConnectionLimit = 80;
    
    // Set minimum polite delay (pause) between requests to a given domain or IP address
    this.RateLimitPerHost = TimeSpan.FromMilliseconds(50);
    
    // Set the allowed number of concurrent HTTP requests (threads) per hostname or IP address
    this.OpenConnectionLimitPerHost = 25;
    
    // Do not obey the robots.txt files
    this.ObeyRobotsDotTxt = false;
    
    // Makes the WebScraper intelligently throttle requests not only by hostname, but also by host servers' IP addresses
    this.ThrottleMode = Throttle.ByDomainHostName;
    
    // Make an initial request to the website with a parse method
    this.Request("https://www.Website.com", Parse);
}

Public Overrides Sub Init()
    ' Set the license key for IronWebScraper
    License.LicenseKey = "LicenseKey"

    ' Set the logging level to capture all logs
    Me.LoggingLevel = WebScraper.LogLevel.All

    ' Assign the working directory for the output files
    Me.WorkingDirectory = AppSetting.GetAppRoot() & "\ShoppingSiteSample\Output\"

    ' Set the total number of allowed open HTTP requests (threads)
    Me.MaxHttpConnectionLimit = 80

    ' Set minimum polite delay (pause) between requests to a given domain or IP address
    Me.RateLimitPerHost = TimeSpan.FromMilliseconds(50)

    ' Set the allowed number of concurrent HTTP requests (threads) per hostname or IP address
    Me.OpenConnectionLimitPerHost = 25

    ' Do not obey the robots.txt files
    Me.ObeyRobotsDotTxt = False

    ' Makes the WebScraper intelligently throttle requests not only by hostname, but also by host servers' IP addresses
    Me.ThrottleMode = Throttle.ByDomainHostName

    ' Make an initial request to the website with a parse method
    Me.Request("https://www.Website.com", AddressOf Parse)
End Sub

Public Overrides Sub Init()
    ' Set the license key for IronWebScraper
    License.LicenseKey = "LicenseKey"

    ' Set the logging level to capture all logs
    Me.LoggingLevel = WebScraper.LogLevel.All

    ' Assign the working directory for the output files
    Me.WorkingDirectory = AppSetting.GetAppRoot() & "\ShoppingSiteSample\Output\"

    ' Set the total number of allowed open HTTP requests (threads)
    Me.MaxHttpConnectionLimit = 80

    ' Set minimum polite delay (pause) between requests to a given domain or IP address
    Me.RateLimitPerHost = TimeSpan.FromMilliseconds(50)

    ' Set the allowed number of concurrent HTTP requests (threads) per hostname or IP address
    Me.OpenConnectionLimitPerHost = 25

    ' Do not obey the robots.txt files
    Me.ObeyRobotsDotTxt = False

    ' Makes the WebScraper intelligently throttle requests not only by hostname, but also by host servers' IP addresses
    Me.ThrottleMode = Throttle.ByDomainHostName

    ' Make an initial request to the website with a parse method
    Me.Request("https://www.Website.com", AddressOf Parse)
End Sub

Throttling properties

MaxHttpConnectionLimit Total number of allowed open HTTP requests (threads)
RateLimitPerHost Minimum polite delay or pause (in milliseconds) between requests to a given domain or IP address
OpenConnectionLimitPerHost Allowed number of concurrent HTTP requests (threads) per hostname
ThrottleMode Makes the WebScraper intelligently throttle requests not only by hostname, but also by host servers' IP addresses. This is polite in case multiple scraped domains are hosted on the same machine.

Get started with IronWebScraper

Start using IronWebScraper in your project today with a free trial.

First Step:

Frequently Asked Questions

What is the purpose of using the HttpIdentity feature in IronWebScraper?

The HttpIdentity feature in IronWebScraper allows users to impersonate different user credentials and browser engines for web scraping, helping to bypass restrictions on content access by simulating multiple login sessions.

How can you enable web caching in IronWebScraper?

Web caching can be enabled in IronWebScraper using the 'EnableWebCache()' method. This allows developers to cache requested pages, which can be reused, especially during development and testing phases, without needing to connect to the website repeatedly.

What is the advantage of enabling throttling in IronWebScraper?

Enabling throttling in IronWebScraper helps manage the number and speed of connections to domains, reducing the likelihood of being blocked for making too many requests in a short period. It ensures a polite and efficient scraping process by distributing requests over time.

Can IronWebScraper continue scraping if the process is restarted?

Yes, IronWebScraper can continue scraping processes after they are restarted by saving the current state using the 'Start(CrawlID)' command, which allows resuming from where it left off.

What is the use of setting 'UseCookies' property in HttpIdentity?

The 'UseCookies' property in HttpIdentity allows IronWebScraper to manage cookies during web scraping sessions, enabling it to maintain state between requests and potentially bypass certain restrictions set by websites based on session data.

What functionalities do the new HttpIdentity objects provide in web scraping?

HttpIdentity objects in IronWebScraper allow you to control user agents, proxies, and authentication credentials, enabling sophisticated options for mimicking user behavior and accessing web content tailored to different scenarios.

How does the 'RateLimitPerHost' property help in web scraping?

The 'RateLimitPerHost' property ensures a minimum delay between outgoing requests to the same domain, which helps prevent the scraper from overwhelming the server, thereby reducing the risk of IP bans.

What are the benefits of using multiple HttpIdentity instances in a single scraping task?

Using multiple HttpIdentity instances allows IronWebScraper to cycle through different identities for requests, spreading the load across various credentials and browser engines, which can help avoid detection and enhance the success rate of scraping tasks.

Why is it important to set the 'WorkingDirectory' in IronWebScraper?

Setting the 'WorkingDirectory' in IronWebScraper specifies where output files, cached data, and state information will be stored. It's crucial for organizing data and ensuring the scraper accesses and saves resources correctly throughout its operations.

Can IronWebScraper work without connecting to a live website constantly?

Yes, IronWebScraper can work by utilizing its web cache feature, which allows it to perform actions on cached pages without repeatedly connecting to the live website, thus facilitating faster testing and development cycles.

Curtis Chau

Chat with engineering team now

Technical Writer

Curtis Chau holds a Bachelor’s degree in Computer Science (Carleton University) and specializes in front-end development with expertise in Node.js, TypeScript, JavaScript, and React. Passionate about crafting intuitive and aesthetically pleasing user interfaces, Curtis enjoys working with modern frameworks and creating well-structured, visually appealing manuals.

...