Webscraping in C#

Curtis Chau

Updated: 2026年6月29日

什么是IronWebScraper？

IronWebScraper是一个适用于C#和.NET编程平台的类库和框架，允许开发人员以编程方式读取网站并提取其内容。这非常适合对现有网站或内部网进行反向工程，并将其转换为数据库或JSON数据。它也适用于从互联网上下载大量文档。

在许多方面，IronWebScraper类似于Python的Scrapy库，但利用了C#的优势，特别是其在网络抓取过程中能够逐步执行代码并进行调试的能力。

安装

您的第一步是安装IronWebScraper，您可以从NuGet获取，也可以从我们的网站下载DLL。

您需要的所有类都可以在IronWebScraper命名空间中找到。

PM > Install-Package IronWebScraper

Install-Package IronWebScraper

常见用例

将网站迁移到数据库

IronWebScraper提供了工具和方法，使您能够将网站重新构建为结构化数据库。当将来自旧网站和内部网的内容迁移到您的新C#应用程序时，这项技术非常有用。

网站迁移

能够在C#中轻松提取部分或完整网站的内容减少了网站和内部网资源迁移或升级的时间和成本。这可以比直接的SQL转换效率高得多，因为它将数据平整为每个网页上可见的内容，而不需要理解以前的SQL数据结构，也不需要构建复杂的SQL查询。

填充搜索索引

IronWebScraper可以指向您自己的网站或内联网以读取结构化数据，读取每个页面，并提取正确的数据，以便您组织内的搜索引擎可以准确填充。

IronWebScraper是为您的搜索索引抓取内容的理想工具。像IronSearch这样的搜索应用程序可以从IronWebScraper读取结构化内容以构建强大的企业搜索系统。

使用Iron Webscraper

要学习如何使用IronWebScraper，最好查看示例。这个基本示例创建了一个类来从网站博客中抓取标题。

using IronWebScraper;

namespace WebScrapingProject
{
    class MainClass
    {
        public static void Main(string [] args)
        {
            var scraper = new BlogScraper();
            scraper.Start();
        }
    }

    class BlogScraper : WebScraper
    {
        // Initialize scraper settings and make the first request
        public override void Init()
        {
            // Set logging level to show all log messages
            this.LoggingLevel = WebScraper.LogLevel.All;
            
            // Request the initial page to start scraping
            this.Request("https://ironpdf.com/blog/", Parse);
        }

        // Method to handle parsing of the page response
        public override void Parse(Response response)
        {
            // Loop through each blog post title link found by CSS selector
            foreach (var title_link in response.Css("h2.entry-title a"))
            {
                // Clean and extract the title text
                string strTitle = title_link.TextContentClean;
                
                // Store the extracted title for later use
                Scrape(new ScrapedData() { { "Title", strTitle } });
            }

            // Check if there is a link to the previous post page and if exists, follow it
            if (response.CssExists("div.prev-post > a[href]"))
            {
                // Get the URL for the next page
                var next_page = response.Css("div.prev-post > a[href]")[0].Attributes["href"];
                
                // Request the next page to continue scraping
                this.Request(next_page, Parse);
            }
        }
    }
}

using IronWebScraper;

namespace WebScrapingProject
{
    class MainClass
    {
        public static void Main(string [] args)
        {
            var scraper = new BlogScraper();
            scraper.Start();
        }
    }

    class BlogScraper : WebScraper
    {
        // Initialize scraper settings and make the first request
        public override void Init()
        {
            // Set logging level to show all log messages
            this.LoggingLevel = WebScraper.LogLevel.All;
            
            // Request the initial page to start scraping
            this.Request("https://ironpdf.com/blog/", Parse);
        }

        // Method to handle parsing of the page response
        public override void Parse(Response response)
        {
            // Loop through each blog post title link found by CSS selector
            foreach (var title_link in response.Css("h2.entry-title a"))
            {
                // Clean and extract the title text
                string strTitle = title_link.TextContentClean;
                
                // Store the extracted title for later use
                Scrape(new ScrapedData() { { "Title", strTitle } });
            }

            // Check if there is a link to the previous post page and if exists, follow it
            if (response.CssExists("div.prev-post > a[href]"))
            {
                // Get the URL for the next page
                var next_page = response.Css("div.prev-post > a[href]")[0].Attributes["href"];
                
                // Request the next page to continue scraping
                this.Request(next_page, Parse);
            }
        }
    }
}

Imports IronWebScraper

Namespace WebScrapingProject
    Class MainClass
        Public Shared Sub Main(ByVal args As String())
            Dim scraper As New BlogScraper()
            scraper.Start()
        End Sub
    End Class

    Class BlogScraper
        Inherits WebScraper

        ' Initialize scraper settings and make the first request
        Public Overrides Sub Init()
            ' Set logging level to show all log messages
            Me.LoggingLevel = WebScraper.LogLevel.All

            ' Request the initial page to start scraping
            Me.Request("https://ironpdf.com/blog/", AddressOf Parse)
        End Sub

        ' Method to handle parsing of the page response
        Public Overrides Sub Parse(ByVal response As Response)
            ' Loop through each blog post title link found by CSS selector
            For Each title_link In response.Css("h2.entry-title a")
                ' Clean and extract the title text
                Dim strTitle As String = title_link.TextContentClean

                ' Store the extracted title for later use
                Scrape(New ScrapedData() From {{"Title", strTitle}})
            Next

            ' Check if there is a link to the previous post page and if exists, follow it
            If response.CssExists("div.prev-post > a[href]") Then
                ' Get the URL for the next page
                Dim next_page = response.Css("div.prev-post > a[href]")(0).Attributes("href")

                ' Request the next page to continue scraping
                Me.Request(next_page, AddressOf Parse)
            End If
        End Sub
    End Class
End Namespace

Imports IronWebScraper

Namespace WebScrapingProject
    Class MainClass
        Public Shared Sub Main(ByVal args As String())
            Dim scraper As New BlogScraper()
            scraper.Start()
        End Sub
    End Class

    Class BlogScraper
        Inherits WebScraper

        ' Initialize scraper settings and make the first request
        Public Overrides Sub Init()
            ' Set logging level to show all log messages
            Me.LoggingLevel = WebScraper.LogLevel.All

            ' Request the initial page to start scraping
            Me.Request("https://ironpdf.com/blog/", AddressOf Parse)
        End Sub

        ' Method to handle parsing of the page response
        Public Overrides Sub Parse(ByVal response As Response)
            ' Loop through each blog post title link found by CSS selector
            For Each title_link In response.Css("h2.entry-title a")
                ' Clean and extract the title text
                Dim strTitle As String = title_link.TextContentClean

                ' Store the extracted title for later use
                Scrape(New ScrapedData() From {{"Title", strTitle}})
            Next

            ' Check if there is a link to the previous post page and if exists, follow it
            If response.CssExists("div.prev-post > a[href]") Then
                ' Get the URL for the next page
                Dim next_page = response.Css("div.prev-post > a[href]")(0).Attributes("href")

                ' Request the next page to continue scraping
                Me.Request(next_page, AddressOf Parse)
            End If
        End Sub
    End Class
End Namespace

要抓取特定网站，我们需要创建自己的类来读取该网站。这个类将继承自Web Scraper。我们将为这个类添加一些方法，包括Init，我们可以在其中设置初始设置并启动第一个请求，然后这将引发一个连锁反应，整个网站将被抓取。

我们还必须至少添加一个Parse方法。 Parse方法读取从互联网上下载的网页，并使用类似jQuery的CSS选择器选择内容并提取相关的文本和/或图像以供使用。

在Parse方法中，我们也可以指定哪些超链接希望爬虫继续跟踪，哪些将被忽略。

我们可以使用Scrape方法提取任何数据并将其转储到方便的JSON格式文件中以备后用。