Advanced Webscraping Features in C#

Q: 如何在需要登錄的網站上使用C#進行使用者身份驗證？

您可以在IronWebScraper中利用 HttpIdentity 功能設定 NetworkDomain 、 NetworkUsername 和 NetworkPassword 等屬性來驗證使用者身份。

Q: 網頁抓取的高級限制選項有哪些？

IronWebScraper提供 ThrottleMode 設定，以主機名稱和IP地址為基準來智能地管理請求限制，確保與共享主機環境的尊重互動。

Q: 如何在IronWebScraper中使用代理？

要使用代理，請定義一個代理陣列並與IronWebScraper中的 HttpIdentity 實例相關聯，以便請求穿過不同的IP地址進行匿名和存取控制。

Q: IronWebScraper如何處理請求延遲以防止伺服器過載？

IronWebScraper中的 RateLimitPerHost 設定指定了對特定域名或IP地址請求之間的最小延遲，幫助透過適度分散請求來防止伺服器過載。

Q: 可以在中斷之後重新開始網頁抓取嗎？

可以，IronWebScraper能夠使用 Start(CrawlID) 方法在中斷後恢復抓取，該方法可以儲存執行狀態並從上次儲存點繼續。

Q: 如何控制網路爬蟲中的並發HTTP連接數量？

在IronWebScraper中，您可以設置 MaxHttpConnectionLimit 屬性來控制允許的開放HTTP請求總數，有助於管理伺服器負載和資源。

Q: 有什麼選項可以用於記錄網頁抓取活動？

IronWebScraper允許您使用 LoggingLevel 屬性設置日誌等級，以便在抓取操作中提供全面的日誌記錄供詳細分析和故障排除。

Curtis Chau

Updated: 2026年6月29日

HttpIdentity 功能

某些網站系統需要使用者登入才能查看內容；在這種情況下，我們可以使用HttpIdentity。以下是設置方法：

// Create a new instance of HttpIdentity
HttpIdentity id = new HttpIdentity();

// Set the network username and password for authentication
id.NetworkUsername = "username";
id.NetworkPassword = "pwd";

// Add the identity to the collection of identities
Identities.Add(id);

// Create a new instance of HttpIdentity
HttpIdentity id = new HttpIdentity();

// Set the network username and password for authentication
id.NetworkUsername = "username";
id.NetworkPassword = "pwd";

// Add the identity to the collection of identities
Identities.Add(id);

' Create a new instance of HttpIdentity
Dim id As New HttpIdentity()

' Set the network username and password for authentication
id.NetworkUsername = "username"
id.NetworkPassword = "pwd"

' Add the identity to the collection of identities
Identities.Add(id)

' Create a new instance of HttpIdentity
Dim id As New HttpIdentity()

' Set the network username and password for authentication
id.NetworkUsername = "username"
id.NetworkPassword = "pwd"

' Add the identity to the collection of identities
Identities.Add(id)

IronWebScraper最令人印象深刻和強大的功能之一是能夠使用成千上萬個獨特的使用者憑證和/或瀏覽器引擎，以多個登入會話偽裝或抓取網站。

public override void Init()
{
    // Set the license key for IronWebScraper
    License.LicenseKey = "LicenseKey";

    // Set the logging level to capture all logs
    this.LoggingLevel = WebScraper.LogLevel.All;

    // Assign the working directory for the output files
    this.WorkingDirectory = AppSetting.GetAppRoot() + @"\ShoppingSiteSample\Output\";

    // Define an array of proxies
    var proxies = "IP-Proxy1:8080,IP-Proxy2:8081".Split(',');

    // Iterate over common Chrome desktop user agents
    foreach (var UA in IronWebScraper.CommonUserAgents.ChromeDesktopUserAgents)
    {
        // Iterate over the proxies
        foreach (var proxy in proxies)
        {
            // Add a new HTTP identity with specific user agent and proxy
            Identities.Add(new HttpIdentity()
            {
                UserAgent = UA,
                UseCookies = true,
                Proxy = proxy
            });
        }
    }
    
    // Make an initial request to the website with a parse method
    this.Request("http://www.Website.com", Parse);
}

public override void Init()
{
    // Set the license key for IronWebScraper
    License.LicenseKey = "LicenseKey";

    // Set the logging level to capture all logs
    this.LoggingLevel = WebScraper.LogLevel.All;

    // Assign the working directory for the output files
    this.WorkingDirectory = AppSetting.GetAppRoot() + @"\ShoppingSiteSample\Output\";

    // Define an array of proxies
    var proxies = "IP-Proxy1:8080,IP-Proxy2:8081".Split(',');

    // Iterate over common Chrome desktop user agents
    foreach (var UA in IronWebScraper.CommonUserAgents.ChromeDesktopUserAgents)
    {
        // Iterate over the proxies
        foreach (var proxy in proxies)
        {
            // Add a new HTTP identity with specific user agent and proxy
            Identities.Add(new HttpIdentity()
            {
                UserAgent = UA,
                UseCookies = true,
                Proxy = proxy
            });
        }
    }
    
    // Make an initial request to the website with a parse method
    this.Request("http://www.Website.com", Parse);
}

Public Overrides Sub Init()
    ' Set the license key for IronWebScraper
    License.LicenseKey = "LicenseKey"

    ' Set the logging level to capture all logs
    Me.LoggingLevel = WebScraper.LogLevel.All

    ' Assign the working directory for the output files
    Me.WorkingDirectory = AppSetting.GetAppRoot() & "\ShoppingSiteSample\Output\"

    ' Define an array of proxies
    Dim proxies = "IP-Proxy1:8080,IP-Proxy2:8081".Split(","c)

    ' Iterate over common Chrome desktop user agents
    For Each UA In IronWebScraper.CommonUserAgents.ChromeDesktopUserAgents
        ' Iterate over the proxies
        For Each proxy In proxies
            ' Add a new HTTP identity with specific user agent and proxy
            Identities.Add(New HttpIdentity() With {
                .UserAgent = UA,
                .UseCookies = True,
                .Proxy = proxy
            })
        Next
    Next

    ' Make an initial request to the website with a parse method
    Me.Request("http://www.Website.com", AddressOf Parse)
End Sub

Public Overrides Sub Init()
    ' Set the license key for IronWebScraper
    License.LicenseKey = "LicenseKey"

    ' Set the logging level to capture all logs
    Me.LoggingLevel = WebScraper.LogLevel.All

    ' Assign the working directory for the output files
    Me.WorkingDirectory = AppSetting.GetAppRoot() & "\ShoppingSiteSample\Output\"

    ' Define an array of proxies
    Dim proxies = "IP-Proxy1:8080,IP-Proxy2:8081".Split(","c)

    ' Iterate over common Chrome desktop user agents
    For Each UA In IronWebScraper.CommonUserAgents.ChromeDesktopUserAgents
        ' Iterate over the proxies
        For Each proxy In proxies
            ' Add a new HTTP identity with specific user agent and proxy
            Identities.Add(New HttpIdentity() With {
                .UserAgent = UA,
                .UseCookies = True,
                .Proxy = proxy
            })
        Next
    Next

    ' Make an initial request to the website with a parse method
    Me.Request("http://www.Website.com", AddressOf Parse)
End Sub

您有多個屬性可以提供不同的行為，防止網站封鎖您。

其中一些屬性包括：

NetworkDomain：用於使用者驗證的網路域。支援Windows、NTLM、Kerberos、Linux、BSD和Mac OS X網路。必須與NetworkPassword一起使用。
NetworkUsername：用於使用者驗證的網路/HTTP使用者名。支援HTTP、Windows網路、NTLM、Kerberos、Linux網路、BSD網路和Mac OS。
NetworkPassword：用於使用者驗證的網路/HTTP密碼。支援HTTP、Windows網路、NTLM、Kerberos、Linux網路、BSD網路和Mac OS。
Proxy：設置代理設定。
UserAgent：設置瀏覽器引擎（如Chrome桌面版、Chrome移動版、Chrome平板版、IE和Firefox等）。
Dictionary<string, string>。
UseCookies：啟用/禁用使用Cookie。

IronWebScraper使用隨機身份運行爬蟲。如果我們需要指定使用特定身份來解析頁面，我們可以這樣做：

public override void Init()
{
    // Set the license key for IronWebScraper
    License.LicenseKey = "LicenseKey";

    // Set the logging level to capture all logs
    this.LoggingLevel = WebScraper.LogLevel.All;

    // Assign the working directory for the output files
    this.WorkingDirectory = AppSetting.GetAppRoot() + @"\ShoppingSiteSample\Output\";

    // Create a new instance of HttpIdentity
    HttpIdentity identity = new HttpIdentity();
    
    // Set the network username and password for authentication
    identity.NetworkUsername = "username";
    identity.NetworkPassword = "pwd";
    
    // Add the identity to the collection of identities
    Identities.Add(identity);
    
    // Make a request to the website with the specified identity
    this.Request("http://www.Website.com", Parse, identity);
}

public override void Init()
{
    // Set the license key for IronWebScraper
    License.LicenseKey = "LicenseKey";

    // Set the logging level to capture all logs
    this.LoggingLevel = WebScraper.LogLevel.All;

    // Assign the working directory for the output files
    this.WorkingDirectory = AppSetting.GetAppRoot() + @"\ShoppingSiteSample\Output\";

    // Create a new instance of HttpIdentity
    HttpIdentity identity = new HttpIdentity();
    
    // Set the network username and password for authentication
    identity.NetworkUsername = "username";
    identity.NetworkPassword = "pwd";
    
    // Add the identity to the collection of identities
    Identities.Add(identity);
    
    // Make a request to the website with the specified identity
    this.Request("http://www.Website.com", Parse, identity);
}

Public Overrides Sub Init()
    ' Set the license key for IronWebScraper
    License.LicenseKey = "LicenseKey"

    ' Set the logging level to capture all logs
    Me.LoggingLevel = WebScraper.LogLevel.All

    ' Assign the working directory for the output files
    Me.WorkingDirectory = AppSetting.GetAppRoot() & "\ShoppingSiteSample\Output\"

    ' Create a new instance of HttpIdentity
    Dim identity As New HttpIdentity()
    
    ' Set the network username and password for authentication
    identity.NetworkUsername = "username"
    identity.NetworkPassword = "pwd"
    
    ' Add the identity to the collection of identities
    Identities.Add(identity)
    
    ' Make a request to the website with the specified identity
    Me.Request("http://www.Website.com", AddressOf Parse, identity)
End Sub

Public Overrides Sub Init()
    ' Set the license key for IronWebScraper
    License.LicenseKey = "LicenseKey"

    ' Set the logging level to capture all logs
    Me.LoggingLevel = WebScraper.LogLevel.All

    ' Assign the working directory for the output files
    Me.WorkingDirectory = AppSetting.GetAppRoot() & "\ShoppingSiteSample\Output\"

    ' Create a new instance of HttpIdentity
    Dim identity As New HttpIdentity()
    
    ' Set the network username and password for authentication
    identity.NetworkUsername = "username"
    identity.NetworkPassword = "pwd"
    
    ' Add the identity to the collection of identities
    Identities.Add(identity)
    
    ' Make a request to the website with the specified identity
    Me.Request("http://www.Website.com", AddressOf Parse, identity)
End Sub

啟用網頁快取功能

此功能用於快取請求的頁面。它通常用於開發和測試階段，讓開發人員能夠快取所需頁面以便在更新程式碼後重複使用。這使您可以在重新啟動您的網路爬蟲後，對快取頁面執行程式碼而無需每次都連接到即時網站（行動重播）。

您可以在Init()方法中使用它：

// Enable web cache without an expiration time
EnableWebCache();

// OR enable web cache with a specified expiration time
EnableWebCache(new TimeSpan(1, 30, 30));

// Enable web cache without an expiration time
EnableWebCache();

// OR enable web cache with a specified expiration time
EnableWebCache(new TimeSpan(1, 30, 30));

' Enable web cache without an expiration time
EnableWebCache()

' OR enable web cache with a specified expiration time
EnableWebCache(New TimeSpan(1, 30, 30))

' Enable web cache without an expiration time
EnableWebCache()

' OR enable web cache with a specified expiration time
EnableWebCache(New TimeSpan(1, 30, 30))

它會將您的快取資料保存到工作目錄下的WebCache資料夾。

public override void Init()
{
    // Set the license key for IronWebScraper
    License.LicenseKey = "LicenseKey";

    // Set the logging level to capture all logs
    this.LoggingLevel = WebScraper.LogLevel.All;

    // Assign the working directory for the output files
    this.WorkingDirectory = AppSetting.GetAppRoot() + @"\ShoppingSiteSample\Output\";

    // Enable web cache with a specific expiration time of 1 hour, 30 minutes, and 30 seconds
    EnableWebCache(new TimeSpan(1, 30, 30));
    
    // Make an initial request to the website with a parse method
    this.Request("http://www.Website.com", Parse);
}

public override void Init()
{
    // Set the license key for IronWebScraper
    License.LicenseKey = "LicenseKey";

    // Set the logging level to capture all logs
    this.LoggingLevel = WebScraper.LogLevel.All;

    // Assign the working directory for the output files
    this.WorkingDirectory = AppSetting.GetAppRoot() + @"\ShoppingSiteSample\Output\";

    // Enable web cache with a specific expiration time of 1 hour, 30 minutes, and 30 seconds
    EnableWebCache(new TimeSpan(1, 30, 30));
    
    // Make an initial request to the website with a parse method
    this.Request("http://www.Website.com", Parse);
}

Public Overrides Sub Init()
    ' Set the license key for IronWebScraper
    License.LicenseKey = "LicenseKey"

    ' Set the logging level to capture all logs
    Me.LoggingLevel = WebScraper.LogLevel.All

    ' Assign the working directory for the output files
    Me.WorkingDirectory = AppSetting.GetAppRoot() & "\ShoppingSiteSample\Output\"

    ' Enable web cache with a specific expiration time of 1 hour, 30 minutes, and 30 seconds
    EnableWebCache(New TimeSpan(1, 30, 30))

    ' Make an initial request to the website with a parse method
    Me.Request("http://www.Website.com", Parse)
End Sub

Public Overrides Sub Init()
    ' Set the license key for IronWebScraper
    License.LicenseKey = "LicenseKey"

    ' Set the logging level to capture all logs
    Me.LoggingLevel = WebScraper.LogLevel.All

    ' Assign the working directory for the output files
    Me.WorkingDirectory = AppSetting.GetAppRoot() & "\ShoppingSiteSample\Output\"

    ' Enable web cache with a specific expiration time of 1 hour, 30 minutes, and 30 seconds
    EnableWebCache(New TimeSpan(1, 30, 30))

    ' Make an initial request to the website with a parse method
    Me.Request("http://www.Website.com", Parse)
End Sub

IronWebScraper還具有功能，允許您的引擎在重新啟動程式碼後繼續抓取，通過使用Start(CrawlID)設定引擎啟動過程名稱。

static void Main(string[] args)
{
    // Create an object from the Scraper class
    EngineScraper scrape = new EngineScraper();
    
    // Start the scraping process with the specified crawl ID
    scrape.Start("enginestate");
}

static void Main(string[] args)
{
    // Create an object from the Scraper class
    EngineScraper scrape = new EngineScraper();
    
    // Start the scraping process with the specified crawl ID
    scrape.Start("enginestate");
}

Sub Main(ByVal args As String())
    ' Create an object from the Scraper class
    Dim scrape As New EngineScraper()

    ' Start the scraping process with the specified crawl ID
    scrape.Start("enginestate")
End Sub

Sub Main(ByVal args As String())
    ' Create an object from the Scraper class
    Dim scrape As New EngineScraper()

    ' Start the scraping process with the specified crawl ID
    scrape.Start("enginestate")
End Sub

執行請求和回應將保存於工作目錄內的SavedState資料夾。

節流

我們可以控制每個域的最小和最大連接數及連接速度。

public override void Init()
{
    // Set the license key for IronWebScraper
    License.LicenseKey = "LicenseKey";

    // Set the logging level to capture all logs
    this.LoggingLevel = WebScraper.LogLevel.All;

    // Assign the working directory for the output files
    this.WorkingDirectory = AppSetting.GetAppRoot() + @"\ShoppingSiteSample\Output\";

    // Set the total number of allowed open HTTP requests (threads)
    this.MaxHttpConnectionLimit = 80;
    
    // Set minimum polite delay (pause) between requests to a given domain or IP address
    this.RateLimitPerHost = TimeSpan.FromMilliseconds(50);
    
    // Set the allowed number of concurrent HTTP requests (threads) per hostname or IP address
    this.OpenConnectionLimitPerHost = 25;
    
    // Do not obey the robots.txt files
    this.ObeyRobotsDotTxt = false;
    
    // Makes the WebScraper intelligently throttle requests not only by hostname, but also by host servers' IP addresses
    this.ThrottleMode = Throttle.ByDomainHostName;
    
    // Make an initial request to the website with a parse method
    this.Request("https://www.Website.com", Parse);
}

public override void Init()
{
    // Set the license key for IronWebScraper
    License.LicenseKey = "LicenseKey";

    // Set the logging level to capture all logs
    this.LoggingLevel = WebScraper.LogLevel.All;

    // Assign the working directory for the output files
    this.WorkingDirectory = AppSetting.GetAppRoot() + @"\ShoppingSiteSample\Output\";

    // Set the total number of allowed open HTTP requests (threads)
    this.MaxHttpConnectionLimit = 80;
    
    // Set minimum polite delay (pause) between requests to a given domain or IP address
    this.RateLimitPerHost = TimeSpan.FromMilliseconds(50);
    
    // Set the allowed number of concurrent HTTP requests (threads) per hostname or IP address
    this.OpenConnectionLimitPerHost = 25;
    
    // Do not obey the robots.txt files
    this.ObeyRobotsDotTxt = false;
    
    // Makes the WebScraper intelligently throttle requests not only by hostname, but also by host servers' IP addresses
    this.ThrottleMode = Throttle.ByDomainHostName;
    
    // Make an initial request to the website with a parse method
    this.Request("https://www.Website.com", Parse);
}

Public Overrides Sub Init()
    ' Set the license key for IronWebScraper
    License.LicenseKey = "LicenseKey"

    ' Set the logging level to capture all logs
    Me.LoggingLevel = WebScraper.LogLevel.All

    ' Assign the working directory for the output files
    Me.WorkingDirectory = AppSetting.GetAppRoot() & "\ShoppingSiteSample\Output\"

    ' Set the total number of allowed open HTTP requests (threads)
    Me.MaxHttpConnectionLimit = 80

    ' Set minimum polite delay (pause) between requests to a given domain or IP address
    Me.RateLimitPerHost = TimeSpan.FromMilliseconds(50)

    ' Set the allowed number of concurrent HTTP requests (threads) per hostname or IP address
    Me.OpenConnectionLimitPerHost = 25

    ' Do not obey the robots.txt files
    Me.ObeyRobotsDotTxt = False

    ' Makes the WebScraper intelligently throttle requests not only by hostname, but also by host servers' IP addresses
    Me.ThrottleMode = Throttle.ByDomainHostName

    ' Make an initial request to the website with a parse method
    Me.Request("https://www.Website.com", AddressOf Parse)
End Sub

Public Overrides Sub Init()
    ' Set the license key for IronWebScraper
    License.LicenseKey = "LicenseKey"

    ' Set the logging level to capture all logs
    Me.LoggingLevel = WebScraper.LogLevel.All

    ' Assign the working directory for the output files
    Me.WorkingDirectory = AppSetting.GetAppRoot() & "\ShoppingSiteSample\Output\"

    ' Set the total number of allowed open HTTP requests (threads)
    Me.MaxHttpConnectionLimit = 80

    ' Set minimum polite delay (pause) between requests to a given domain or IP address
    Me.RateLimitPerHost = TimeSpan.FromMilliseconds(50)

    ' Set the allowed number of concurrent HTTP requests (threads) per hostname or IP address
    Me.OpenConnectionLimitPerHost = 25

    ' Do not obey the robots.txt files
    Me.ObeyRobotsDotTxt = False

    ' Makes the WebScraper intelligently throttle requests not only by hostname, but also by host servers' IP addresses
    Me.ThrottleMode = Throttle.ByDomainHostName

    ' Make an initial request to the website with a parse method
    Me.Request("https://www.Website.com", AddressOf Parse)
End Sub

節流屬性

MaxHttpConnectionLimit 允許打開HTTP請求的總數（執行緒）
RateLimitPerHost 對特定域或IP地址之間的請求的最小禮貌延遲或暫停時間（以毫秒為單位）
OpenConnectionLimitPerHost 每個主機名允許的同時HTTP請求數（執行緒）
ThrottleMode 使WebScraper能夠智能地對請求進行節流，不僅按主機名，還按主機伺服器的IP地址。當多個抓取的域託管在同一台機器上時，這是禮貌的。

開始使用IronWebScraper

Start using IronWebScraper in your project today with a free trial.

第一步：

常見問題

如何在需要登錄的網站上使用C#進行使用者身份驗證？

您可以在IronWebScraper中利用HttpIdentity功能設定NetworkDomain、NetworkUsername和NetworkPassword等屬性來驗證使用者身份。

使用網頁快取開發有何好處？

網頁快取功能允許您快取請求的頁面以供重用，這有助於節省時間和資源，尤其是在開發和測試階段，避免重複連接到實時網站。

如何在網頁抓取中管理多個登錄會話？

IronWebScraper允許使用數千個獨特的使用者憑據和瀏覽器引擎來模擬多個登錄會話，這有助於防止網站偵測和阻止爬蟲。

網頁抓取的高級限制選項有哪些？

IronWebScraper提供ThrottleMode設定，以主機名稱和IP地址為基準來智能地管理請求限制，確保與共享主機環境的尊重互動。

如何在IronWebScraper中使用代理？

要使用代理，請定義一個代理陣列並與IronWebScraper中的HttpIdentity實例相關聯，以便請求穿過不同的IP地址進行匿名和存取控制。

IronWebScraper如何處理請求延遲以防止伺服器過載？

IronWebScraper中的RateLimitPerHost設定指定了對特定域名或IP地址請求之間的最小延遲，幫助透過適度分散請求來防止伺服器過載。

可以在中斷之後重新開始網頁抓取嗎？

可以，IronWebScraper能夠使用Start(CrawlID)方法在中斷後恢復抓取，該方法可以儲存執行狀態並從上次儲存點繼續。

如何控制網路爬蟲中的並發HTTP連接數量？

在IronWebScraper中，您可以設置MaxHttpConnectionLimit屬性來控制允許的開放HTTP請求總數，有助於管理伺服器負載和資源。

有什麼選項可以用於記錄網頁抓取活動？

IronWebScraper允許您使用LoggingLevel屬性設置日誌等級，以便在抓取操作中提供全面的日誌記錄供詳細分析和故障排除。

Can IronWebScraper work without connecting to a live website constantly?

Yes, IronWebScraper can work by utilizing its web cache feature, which allows it to perform actions on cached pages without repeatedly connecting to the live website, thus facilitating faster testing and development cycles.