1. IronWebScraper
  2. 教程
  3. 高階網頁抓取功能

Advanced Webscraping Features in C

Darrius Serrant
Darrius Serrant
已更新:
This article was translated from English: Does it need improvement?
Translated
View the article in English

HttpIdentity 功能

某些網站系統需要使用者登入才能查看內容； 在這種情況下，我們可以使用HttpIdentity。 以下是設置方法：

// Create a new instance of HttpIdentity
HttpIdentity id = new HttpIdentity();

// Set the network username and password for authentication
id.NetworkUsername = "username";
id.NetworkPassword = "pwd";

// Add the identity to the collection of identities
Identities.Add(id);
// Create a new instance of HttpIdentity
HttpIdentity id = new HttpIdentity();

// Set the network username and password for authentication
id.NetworkUsername = "username";
id.NetworkPassword = "pwd";

// Add the identity to the collection of identities
Identities.Add(id);
' Create a new instance of HttpIdentity
Dim id As New HttpIdentity()

' Set the network username and password for authentication
id.NetworkUsername = "username"
id.NetworkPassword = "pwd"

' Add the identity to the collection of identities
Identities.Add(id)
$vbLabelText   $csharpLabel

IronWebScraper最令人印象深刻和強大的功能之一是能夠使用成千上萬個獨特的使用者憑證和/或瀏覽器引擎，以多個登入會話偽裝或抓取網站。

public override void Init()
{
    // Set the license key for IronWebScraper
    License.LicenseKey = "LicenseKey";

    // Set the logging level to capture all logs
    this.LoggingLevel = WebScraper.LogLevel.All;

    // Assign the working directory for the output files
    this.WorkingDirectory = AppSetting.GetAppRoot() + @"\ShoppingSiteSample\Output\";

    // Define an array of proxies
    var proxies = "IP-Proxy1:8080,IP-Proxy2:8081".Split(',');

    // Iterate over common Chrome desktop user agents
    foreach (var UA in IronWebScraper.CommonUserAgents.ChromeDesktopUserAgents)
    {
        // Iterate over the proxies
        foreach (var proxy in proxies)
        {
            // Add a new HTTP identity with specific user agent and proxy
            Identities.Add(new HttpIdentity()
            {
                UserAgent = UA,
                UseCookies = true,
                Proxy = proxy
            });
        }
    }

    // Make an initial request to the website with a parse method
    this.Request("http://www.Website.com", Parse);
}
public override void Init()
{
    // Set the license key for IronWebScraper
    License.LicenseKey = "LicenseKey";

    // Set the logging level to capture all logs
    this.LoggingLevel = WebScraper.LogLevel.All;

    // Assign the working directory for the output files
    this.WorkingDirectory = AppSetting.GetAppRoot() + @"\ShoppingSiteSample\Output\";

    // Define an array of proxies
    var proxies = "IP-Proxy1:8080,IP-Proxy2:8081".Split(',');

    // Iterate over common Chrome desktop user agents
    foreach (var UA in IronWebScraper.CommonUserAgents.ChromeDesktopUserAgents)
    {
        // Iterate over the proxies
        foreach (var proxy in proxies)
        {
            // Add a new HTTP identity with specific user agent and proxy
            Identities.Add(new HttpIdentity()
            {
                UserAgent = UA,
                UseCookies = true,
                Proxy = proxy
            });
        }
    }

    // Make an initial request to the website with a parse method
    this.Request("http://www.Website.com", Parse);
}
Public Overrides Sub Init()
	' Set the license key for IronWebScraper
	License.LicenseKey = "LicenseKey"

	' Set the logging level to capture all logs
	Me.LoggingLevel = WebScraper.LogLevel.All

	' Assign the working directory for the output files
	Me.WorkingDirectory = AppSetting.GetAppRoot() & "\ShoppingSiteSample\Output\"

	' Define an array of proxies
	Dim proxies = "IP-Proxy1:8080,IP-Proxy2:8081".Split(","c)

	' Iterate over common Chrome desktop user agents
	For Each UA In IronWebScraper.CommonUserAgents.ChromeDesktopUserAgents
		' Iterate over the proxies
		For Each proxy In proxies
			' Add a new HTTP identity with specific user agent and proxy
			Identities.Add(New HttpIdentity() With {
				.UserAgent = UA,
				.UseCookies = True,
				.Proxy = proxy
			})
		Next proxy
	Next UA

	' Make an initial request to the website with a parse method
	Me.Request("http://www.Website.com", Parse)
End Sub
$vbLabelText   $csharpLabel

您有多個屬性可以提供不同的行為，防止網站封鎖您。

其中一些屬性包括：

  • NetworkDomain：用於使用者驗證的網路域。 支援Windows、NTLM、Kerberos、Linux、BSD和Mac OS X網路。 必須與NetworkPassword一起使用。
  • NetworkUsername：用於使用者驗證的網路/HTTP使用者名。 支援HTTP、Windows網路、NTLM、Kerberos、Linux網路、BSD網路和Mac OS。
  • NetworkPassword：用於使用者驗證的網路/HTTP密碼。 支援HTTP、Windows網路、NTLM、Kerberos、Linux網路、BSD網路和Mac OS。
  • Proxy：設置代理設定。
  • UserAgent：設置瀏覽器引擎（如Chrome桌面版、Chrome移動版、Chrome平板版、IE和Firefox等）。
  • Dictionary<string, string>。
  • UseCookies：啟用/禁用使用Cookie。

IronWebScraper使用隨機身份運行爬蟲。 如果我們需要指定使用特定身份來解析頁面，我們可以這樣做：

public override void Init()
{
    // Set the license key for IronWebScraper
    License.LicenseKey = "LicenseKey";

    // Set the logging level to capture all logs
    this.LoggingLevel = WebScraper.LogLevel.All;

    // Assign the working directory for the output files
    this.WorkingDirectory = AppSetting.GetAppRoot() + @"\ShoppingSiteSample\Output\";

    // Create a new instance of HttpIdentity
    HttpIdentity identity = new HttpIdentity();

    // Set the network username and password for authentication
    identity.NetworkUsername = "username";
    identity.NetworkPassword = "pwd";

    // Add the identity to the collection of identities
    Identities.Add(identity);

    // Make a request to the website with the specified identity
    this.Request("http://www.Website.com", Parse, identity);
}
public override void Init()
{
    // Set the license key for IronWebScraper
    License.LicenseKey = "LicenseKey";

    // Set the logging level to capture all logs
    this.LoggingLevel = WebScraper.LogLevel.All;

    // Assign the working directory for the output files
    this.WorkingDirectory = AppSetting.GetAppRoot() + @"\ShoppingSiteSample\Output\";

    // Create a new instance of HttpIdentity
    HttpIdentity identity = new HttpIdentity();

    // Set the network username and password for authentication
    identity.NetworkUsername = "username";
    identity.NetworkPassword = "pwd";

    // Add the identity to the collection of identities
    Identities.Add(identity);

    // Make a request to the website with the specified identity
    this.Request("http://www.Website.com", Parse, identity);
}
Public Overrides Sub Init()
	' Set the license key for IronWebScraper
	License.LicenseKey = "LicenseKey"

	' Set the logging level to capture all logs
	Me.LoggingLevel = WebScraper.LogLevel.All

	' Assign the working directory for the output files
	Me.WorkingDirectory = AppSetting.GetAppRoot() & "\ShoppingSiteSample\Output\"

	' Create a new instance of HttpIdentity
	Dim identity As New HttpIdentity()

	' Set the network username and password for authentication
	identity.NetworkUsername = "username"
	identity.NetworkPassword = "pwd"

	' Add the identity to the collection of identities
	Identities.Add(identity)

	' Make a request to the website with the specified identity
	Me.Request("http://www.Website.com", Parse, identity)
End Sub
$vbLabelText   $csharpLabel

啟用網頁快取功能

此功能用於快取請求的頁面。 它通常用於開發和測試階段，讓開發人員能夠快取所需頁面以便在更新程式碼後重複使用。 這使您可以在重新啟動您的網路爬蟲後，對快取頁面執行程式碼而無需每次都連接到即時網站（行動重播）。

您可以在Init()方法中使用它：

// Enable web cache without an expiration time
EnableWebCache();

// OR enable web cache with a specified expiration time
EnableWebCache(new TimeSpan(1, 30, 30));
// Enable web cache without an expiration time
EnableWebCache();

// OR enable web cache with a specified expiration time
EnableWebCache(new TimeSpan(1, 30, 30));
' Enable web cache without an expiration time
EnableWebCache()

' OR enable web cache with a specified expiration time
EnableWebCache(New TimeSpan(1, 30, 30))
$vbLabelText   $csharpLabel

它會將您的快取資料保存到工作目錄下的WebCache資料夾。

public override void Init()
{
    // Set the license key for IronWebScraper
    License.LicenseKey = "LicenseKey";

    // Set the logging level to capture all logs
    this.LoggingLevel = WebScraper.LogLevel.All;

    // Assign the working directory for the output files
    this.WorkingDirectory = AppSetting.GetAppRoot() + @"\ShoppingSiteSample\Output\";

    // Enable web cache with a specific expiration time of 1 hour, 30 minutes, and 30 seconds
    EnableWebCache(new TimeSpan(1, 30, 30));

    // Make an initial request to the website with a parse method
    this.Request("http://www.Website.com", Parse);
}
public override void Init()
{
    // Set the license key for IronWebScraper
    License.LicenseKey = "LicenseKey";

    // Set the logging level to capture all logs
    this.LoggingLevel = WebScraper.LogLevel.All;

    // Assign the working directory for the output files
    this.WorkingDirectory = AppSetting.GetAppRoot() + @"\ShoppingSiteSample\Output\";

    // Enable web cache with a specific expiration time of 1 hour, 30 minutes, and 30 seconds
    EnableWebCache(new TimeSpan(1, 30, 30));

    // Make an initial request to the website with a parse method
    this.Request("http://www.Website.com", Parse);
}
Public Overrides Sub Init()
	' Set the license key for IronWebScraper
	License.LicenseKey = "LicenseKey"

	' Set the logging level to capture all logs
	Me.LoggingLevel = WebScraper.LogLevel.All

	' Assign the working directory for the output files
	Me.WorkingDirectory = AppSetting.GetAppRoot() & "\ShoppingSiteSample\Output\"

	' Enable web cache with a specific expiration time of 1 hour, 30 minutes, and 30 seconds
	EnableWebCache(New TimeSpan(1, 30, 30))

	' Make an initial request to the website with a parse method
	Me.Request("http://www.Website.com", Parse)
End Sub
$vbLabelText   $csharpLabel

IronWebScraper還具有功能，允許您的引擎在重新啟動程式碼後繼續抓取，通過使用Start(CrawlID)設定引擎啟動過程名稱。

static void Main(string[] args)
{
    // Create an object from the Scraper class
    EngineScraper scrape = new EngineScraper();

    // Start the scraping process with the specified crawl ID
    scrape.Start("enginestate");
}
static void Main(string[] args)
{
    // Create an object from the Scraper class
    EngineScraper scrape = new EngineScraper();

    // Start the scraping process with the specified crawl ID
    scrape.Start("enginestate");
}
Shared Sub Main(ByVal args() As String)
	' Create an object from the Scraper class
	Dim scrape As New EngineScraper()

	' Start the scraping process with the specified crawl ID
	scrape.Start("enginestate")
End Sub
$vbLabelText   $csharpLabel

執行請求和回應將保存於工作目錄內的SavedState資料夾。

節流

我們可以控制每個域的最小和最大連接數及連接速度。

public override void Init()
{
    // Set the license key for IronWebScraper
    License.LicenseKey = "LicenseKey";

    // Set the logging level to capture all logs
    this.LoggingLevel = WebScraper.LogLevel.All;

    // Assign the working directory for the output files
    this.WorkingDirectory = AppSetting.GetAppRoot() + @"\ShoppingSiteSample\Output\";

    // Set the total number of allowed open HTTP requests (threads)
    this.MaxHttpConnectionLimit = 80;

    // Set minimum polite delay (pause) between requests to a given domain or IP address
    this.RateLimitPerHost = TimeSpan.FromMilliseconds(50);

    // Set the allowed number of concurrent HTTP requests (threads) per hostname or IP address
    this.OpenConnectionLimitPerHost = 25;

    // Do not obey the robots.txt files
    this.ObeyRobotsDotTxt = false;

    // Makes the WebScraper intelligently throttle requests not only by hostname, but also by host servers' IP addresses
    this.ThrottleMode = Throttle.ByDomainHostName;

    // Make an initial request to the website with a parse method
    this.Request("https://www.Website.com", Parse);
}
public override void Init()
{
    // Set the license key for IronWebScraper
    License.LicenseKey = "LicenseKey";

    // Set the logging level to capture all logs
    this.LoggingLevel = WebScraper.LogLevel.All;

    // Assign the working directory for the output files
    this.WorkingDirectory = AppSetting.GetAppRoot() + @"\ShoppingSiteSample\Output\";

    // Set the total number of allowed open HTTP requests (threads)
    this.MaxHttpConnectionLimit = 80;

    // Set minimum polite delay (pause) between requests to a given domain or IP address
    this.RateLimitPerHost = TimeSpan.FromMilliseconds(50);

    // Set the allowed number of concurrent HTTP requests (threads) per hostname or IP address
    this.OpenConnectionLimitPerHost = 25;

    // Do not obey the robots.txt files
    this.ObeyRobotsDotTxt = false;

    // Makes the WebScraper intelligently throttle requests not only by hostname, but also by host servers' IP addresses
    this.ThrottleMode = Throttle.ByDomainHostName;

    // Make an initial request to the website with a parse method
    this.Request("https://www.Website.com", Parse);
}
Public Overrides Sub Init()
	' Set the license key for IronWebScraper
	License.LicenseKey = "LicenseKey"

	' Set the logging level to capture all logs
	Me.LoggingLevel = WebScraper.LogLevel.All

	' Assign the working directory for the output files
	Me.WorkingDirectory = AppSetting.GetAppRoot() & "\ShoppingSiteSample\Output\"

	' Set the total number of allowed open HTTP requests (threads)
	Me.MaxHttpConnectionLimit = 80

	' Set minimum polite delay (pause) between requests to a given domain or IP address
	Me.RateLimitPerHost = TimeSpan.FromMilliseconds(50)

	' Set the allowed number of concurrent HTTP requests (threads) per hostname or IP address
	Me.OpenConnectionLimitPerHost = 25

	' Do not obey the robots.txt files
	Me.ObeyRobotsDotTxt = False

	' Makes the WebScraper intelligently throttle requests not only by hostname, but also by host servers' IP addresses
	Me.ThrottleMode = Throttle.ByDomainHostName

	' Make an initial request to the website with a parse method
	Me.Request("https://www.Website.com", Parse)
End Sub
$vbLabelText   $csharpLabel

節流屬性

  • MaxHttpConnectionLimit 允許打開HTTP請求的總數（執行緒）
  • RateLimitPerHost 對特定域或IP地址之間的請求的最小禮貌延遲或暫停時間（以毫秒為單位）
  • OpenConnectionLimitPerHost 每個主機名允許的同時HTTP請求數（執行緒）
  • ThrottleMode 使WebScraper能夠智能地對請求進行節流，不僅按主機名，還按主機伺服器的IP地址。 當多個抓取的域託管在同一台機器上時，這是禮貌的。

開始使用IronWebScraper

今天就使用IronWebScraper開始專案，免費試用。

第一步：
green arrow pointer

常見問題

如何在需要登錄的網站上使用C#進行使用者身份驗證？

您可以在IronWebScraper中利用HttpIdentity功能設定NetworkDomain、NetworkUsername和NetworkPassword等屬性來驗證使用者身份。

使用網頁快取開發有何好處？

網頁快取功能允許您快取請求的頁面以供重用，這有助於節省時間和資源，尤其是在開發和測試階段，避免重複連接到實時網站。

如何在網頁抓取中管理多個登錄會話？

IronWebScraper允許使用數千個獨特的使用者憑據和瀏覽器引擎來模擬多個登錄會話，這有助於防止網站偵測和阻止爬蟲。

網頁抓取的高級限制選項有哪些？

IronWebScraper提供ThrottleMode設定，以主機名稱和IP地址為基準來智能地管理請求限制，確保與共享主機環境的尊重互動。

如何在IronWebScraper中使用代理？

要使用代理，請定義一個代理陣列並與IronWebScraper中的HttpIdentity實例相關聯，以便請求穿過不同的IP地址進行匿名和存取控制。

IronWebScraper如何處理請求延遲以防止伺服器過載？

IronWebScraper中的RateLimitPerHost設定指定了對特定域名或IP地址請求之間的最小延遲，幫助透過適度分散請求來防止伺服器過載。

可以在中斷之後重新開始網頁抓取嗎？

可以，IronWebScraper能夠使用Start(CrawlID)方法在中斷後恢復抓取，該方法可以儲存執行狀態並從上次儲存點繼續。

如何控制網路爬蟲中的並發HTTP連接數量？

在IronWebScraper中，您可以設置MaxHttpConnectionLimit屬性來控制允許的開放HTTP請求總數，有助於管理伺服器負載和資源。

有什麼選項可以用於記錄網頁抓取活動？

IronWebScraper允許您使用LoggingLevel屬性設置日誌等級，以便在抓取操作中提供全面的日誌記錄供詳細分析和故障排除。

Darrius Serrant
Darrius Serrant
  立即與工程團隊聊天
全端軟體工程師（WebOps）

Darrius Serrant擁有邁阿密大學的電腦科學學士學位，並在Iron Software擔任全端WebOps行銷工程師。從小就對程式設計有興趣，他認為計算既神秘又易於理解，成為創意和問題解決的完美媒介。

在Iron Software，Darrius喜歡創造新事物並簡化複雜的概念，使其更易於理解。作為我們的常駐開發人員之一，他還志願教學，將他的專業知識傳授給下一代。

對Darrius來說，他的工作是有意義的，因為它有價值且對社會有真正的影響。

準備好開始了嗎？
Nuget 下載 140,761 | 版本： 2026.7 剛剛發布
Still Scrolling Icon

還在捲動嗎？

想快速獲得證明嗎？ PM > Install-Package IronWebScraper
執行一個範例 看您的目標網站變成結構化資料。