Scrape a Shopping Website in C
This article was translated from English: Does it need improvement?
Translated
View the article in English
WebScraper çerçevesini kullanarak C# ile alışveriş web sitelerinden ürün kategorilerini ve öğeleri nasıl kazıyacağınızı ve HTML öğelerinden yapılandırılmış verileri özel modellere nasıl aktaracağınızı öğrenin. Bu kapsamlı kılavuz, IronWebScraper kütüphanesini kullanarak sağlam bir e-ticaret kazıyıcı oluşturma sürecini adım adım anlatır.
@@--AH2BG--@@Hızlı Başlangıç: C#'ta Alışveriş Web Sitesini Kazıyın@@--AH2EG--@
-
IronWebScraper aşağıdaki NuGet Paket Yöneticisi ile yükleyin
PM > Install-Package IronWebScraper -
Bu kod parçacığını kopyalayın ve çalıştırın.
using IronWebScraper; public class QuickShoppingScraper : WebScraper { public override void Init() { // Apply your license key License.LicenseKey = "YOUR-LICENSE-KEY"; // Set the starting URL this.Request("https://shopping-site.com", Parse); } public override void Parse(Response response) { // Extract product data foreach (var product in response.Css(".product-item")) { var item = new { Name = product.Css(".product-name").First().InnerText, Price = product.Css(".price").First().InnerText, Image = product.Css("img").First().Attributes["src"] }; Scrape(item, "products.jsonl"); } } } // Run the scraper var scraper = new QuickShoppingScraper(); scraper.Start(); -
Canlı ortamınızda test için dağıtım yapın
Ücretsiz deneme ile bugün projenizde IronWebScraper kullanmaya başlayın
- "ShoppingSiteSample" adlı yeni bir Konsol Uygulaması projesi oluşturun
WebScraper'den miras alan "ShoppingScraper" adlı bir sınıf ekleyinCategoryveProductverileri için modeller oluşturun- Kazıma için başlangıç URL'sini ve
Parse()yöntemini ayarlamak üzereInit()'yi geçersiz kılın - Kategorileri ve ürünleri JSONL dosyalarına çıkarmak için kazıyıcıyı çalıştırın
Alışveriş Sitesinin HTML Yapısını Nasıl Analiz Ederim?
İçerik yapısını analiz etmek için bir alışveriş sitesi seçin. HTML yapısını anlamak başarılı bir web kazıma için çok önemlidir. Herhangi bir kod yazmadan önce, tarayıcı geliştirici araçlarını kullanarak hedef web sitesinin yapısını analiz etmek için zaman harcayın.
Resimde gösterildiği gibi, sol kenar çubuğu sitenin ürün kategorileri için bağlantılar içeriyor. İlk adım, sitenin HTML'sini araştırmak ve kazıma yaklaşımını planlamaktır. Bu analiz aşaması, etkili bir kazıma stratejisi oluşturmak için çok önemlidir.
HTML Yapısını Anlamak Neden Önemlidir?
Moda sitesi kategorilerinin alt kategorileri vardır (Erkek, Kadın, Çocuk). Bu hiyerarşik yapının anlaşılması, uygun veri modellerinin ve kazıma mantığının tasarlanmasına yardımcı olur. gelişmiş web kazıma özellikleri ile çalışırken, doğru HTML analizi daha da kritik hale gelir.
<li class="menu-item" data-id="">
<a href="https://domain.com/fashion-by-/" class="main-category">
<i class="cat-icon osh-font-fashion"></i>
<span class="nav-subTxt">FASHION </span>
<i class="osh-font-light-arrow-left"></i><i class="osh-font-light-arrow-right"></i>
</a>
<div class="navLayerWrapper" style="width: 633px; display: none;">
<div class="submenu">
<div class="column">
<div class="categories">
<a class="category" href="https://domain.com/fashion-by-/?sort=newest&dir=desc&viewType=gridView3">New Arrivals !</a>
</div>
<div class="categories">
<a class="category" href="https://domain.com/men-fashion/">Men</a>
<a class="subcategory" href="https://domain.com/mens-shoes/">Shoes</a>
<a class="subcategory" href="https://domain.com/mens-clothing/">Clothing</a>
<a class="subcategory" href="https://domain.com/mens-accessories/">Accessories</a>
</div>
<div class="categories">
<a class="category" href="https://domain.com/women-fashion/">Women</a>
<a class="subcategory" href="https://domain.com/womens-shoes/">Shoes</a>
<a class="subcategory" href="https://domain.com/womens-clothing/">Clothing</a>
<a class="subcategory" href="https://domain.com/womens-accessories/">Accessories</a>
</div>
<div class="categories">
<a class="category" href="https://domain.com/girls-boys-fashion/">Kids</a>
<a class="subcategory" href="https://domain.com/boys-fashion/">Boys</a>
<a class="subcategory" href="https://domain.com/girls/">Girls</a>
</div>
<div class="categories">
<a class="category" href="https://domain.com/maternity-clothes/">Maternity Clothes</a>
</div>
</div>
<div class="column">
<div class="categories">
<span class="category defaultCursor">Men Best Sellers</span>
<a class="subcategory" href="https://domain.com/mens-casual-shoes/">Casual Shoes</a>
<a class="subcategory" href="https://domain.com/mens-sneakers/">Sneakers</a>
<a class="subcategory" href="https://domain.com/mens-t-shirts/">T-shirts</a>
<a class="subcategory" href="https://domain.com/mens-polos/">Polos</a>
</div>
<div class="categories">
<span class="category defaultCursor">Women Best Sellers</span>
<a class="subcategory" href="https://domain.com/womens-sandals/">Sandals</a>
<a class="subcategory" href="https://domain.com/womens-sneakers/">Sneakers</a>
<a class="subcategory" href="https://domain.com/women-dresses/">Dresses</a>
<a class="subcategory" href="https://domain.com/women-tops/">Tops</a>
</div>
<div class="categories">
<a class="category" href="https://domain.com/womens-curvy-clothing/">Women's Curvy Clothing</a>
</div>
<div class="categories">
<a class="category" href="https://domain.com/fashion-bundles/v/">Fashion Bundles</a>
</div>
<div class="categories">
<a class="category" href="https://domain.com/hijab-fashion/">Hijab Fashion</a>
</div>
</div>
<div class="column">
<div class="categories">
<a class="category" href="https://domain.com/brands/fashion-by-/">SEE ALL BRANDS</a>
<a class="subcategory" href="https://domain.com/adidas/">Adidas</a>
<a class="subcategory" href="https://domain.com/converse/">Converse</a>
<a class="subcategory" href="https://domain.com/ravin/">Ravin</a>
<a class="subcategory" href="https://domain.com/dejavu/">Dejavu</a>
<a class="subcategory" href="https://domain.com/agu/">Agu</a>
<a class="subcategory" href="https://domain.com/activ/">Activ</a>
<a class="subcategory" href="https://domain.com/oxford--bellini--tie-house--milano/">Tie House</a>
<a class="subcategory" href="https://domain.com/shoe-room/">Shoe Room</a>
<a class="subcategory" href="https://domain.com/town-team/">Town Team</a>
</div>
</div>
</div>
</div>
</li><li class="menu-item" data-id="">
<a href="https://domain.com/fashion-by-/" class="main-category">
<i class="cat-icon osh-font-fashion"></i>
<span class="nav-subTxt">FASHION </span>
<i class="osh-font-light-arrow-left"></i><i class="osh-font-light-arrow-right"></i>
</a>
<div class="navLayerWrapper" style="width: 633px; display: none;">
<div class="submenu">
<div class="column">
<div class="categories">
<a class="category" href="https://domain.com/fashion-by-/?sort=newest&dir=desc&viewType=gridView3">New Arrivals !</a>
</div>
<div class="categories">
<a class="category" href="https://domain.com/men-fashion/">Men</a>
<a class="subcategory" href="https://domain.com/mens-shoes/">Shoes</a>
<a class="subcategory" href="https://domain.com/mens-clothing/">Clothing</a>
<a class="subcategory" href="https://domain.com/mens-accessories/">Accessories</a>
</div>
<div class="categories">
<a class="category" href="https://domain.com/women-fashion/">Women</a>
<a class="subcategory" href="https://domain.com/womens-shoes/">Shoes</a>
<a class="subcategory" href="https://domain.com/womens-clothing/">Clothing</a>
<a class="subcategory" href="https://domain.com/womens-accessories/">Accessories</a>
</div>
<div class="categories">
<a class="category" href="https://domain.com/girls-boys-fashion/">Kids</a>
<a class="subcategory" href="https://domain.com/boys-fashion/">Boys</a>
<a class="subcategory" href="https://domain.com/girls/">Girls</a>
</div>
<div class="categories">
<a class="category" href="https://domain.com/maternity-clothes/">Maternity Clothes</a>
</div>
</div>
<div class="column">
<div class="categories">
<span class="category defaultCursor">Men Best Sellers</span>
<a class="subcategory" href="https://domain.com/mens-casual-shoes/">Casual Shoes</a>
<a class="subcategory" href="https://domain.com/mens-sneakers/">Sneakers</a>
<a class="subcategory" href="https://domain.com/mens-t-shirts/">T-shirts</a>
<a class="subcategory" href="https://domain.com/mens-polos/">Polos</a>
</div>
<div class="categories">
<span class="category defaultCursor">Women Best Sellers</span>
<a class="subcategory" href="https://domain.com/womens-sandals/">Sandals</a>
<a class="subcategory" href="https://domain.com/womens-sneakers/">Sneakers</a>
<a class="subcategory" href="https://domain.com/women-dresses/">Dresses</a>
<a class="subcategory" href="https://domain.com/women-tops/">Tops</a>
</div>
<div class="categories">
<a class="category" href="https://domain.com/womens-curvy-clothing/">Women's Curvy Clothing</a>
</div>
<div class="categories">
<a class="category" href="https://domain.com/fashion-bundles/v/">Fashion Bundles</a>
</div>
<div class="categories">
<a class="category" href="https://domain.com/hijab-fashion/">Hijab Fashion</a>
</div>
</div>
<div class="column">
<div class="categories">
<a class="category" href="https://domain.com/brands/fashion-by-/">SEE ALL BRANDS</a>
<a class="subcategory" href="https://domain.com/adidas/">Adidas</a>
<a class="subcategory" href="https://domain.com/converse/">Converse</a>
<a class="subcategory" href="https://domain.com/ravin/">Ravin</a>
<a class="subcategory" href="https://domain.com/dejavu/">Dejavu</a>
<a class="subcategory" href="https://domain.com/agu/">Agu</a>
<a class="subcategory" href="https://domain.com/activ/">Activ</a>
<a class="subcategory" href="https://domain.com/oxford--bellini--tie-house--milano/">Tie House</a>
<a class="subcategory" href="https://domain.com/shoe-room/">Shoe Room</a>
<a class="subcategory" href="https://domain.com/town-team/">Town Team</a>
</div>
</div>
</div>
</div>
</li>
HTML
Web Kazıma Projesini Nasıl Kurarım?
C# web kazıma için en iyi uygulamaları takip eden bir proje oluşturun.
"ShoppingSiteSample" adında yeni bir Konsol Uygulaması oluşturun veya örnek için yeni bir klasör ekleyin
- "
ShoppingScraper" adlı yeni bir sınıf ekleyin Site kategorilerini ve alt kategorilerini kazıyarak başlayın - NuGet Paket Yöneticisi veya Paket Yöneticisi Konsolu aracılığıyla
IronWebScraper'yi yükleyin:
Install-Package IronWebScraperInstall-Package IronWebScraper
$vbLabelText
$csharpLabel
Kategoriler için Hangi Veri Modelini Kullanmalıyım?
Keşfedilen hiyerarşik yapıyı düzgün bir şekilde temsil eden bir Kategoriler Modeli oluşturun:
public class Category
{
/// <summary>
/// Gets or sets the name.
/// </summary>
/// <value>
/// The name.
/// </value>
public string Name { get; set; }
/// <summary>
/// Gets or sets the URL.
/// </summary>
/// <value>
/// The URL.
/// </value>
public string URL { get; set; }
/// <summary>
/// Gets or sets the subcategories.
/// </summary>
/// <value>
/// The subcategories.
/// </value>
public List<Category> SubCategories { get; set; }
// Additional properties for enhanced data collection
public int ProductCount { get; set; }
public DateTime LastScraped { get; set; }
public string CategoryType { get; set; }
}public class Category
{
/// <summary>
/// Gets or sets the name.
/// </summary>
/// <value>
/// The name.
/// </value>
public string Name { get; set; }
/// <summary>
/// Gets or sets the URL.
/// </summary>
/// <value>
/// The URL.
/// </value>
public string URL { get; set; }
/// <summary>
/// Gets or sets the subcategories.
/// </summary>
/// <value>
/// The subcategories.
/// </value>
public List<Category> SubCategories { get; set; }
// Additional properties for enhanced data collection
public int ProductCount { get; set; }
public DateTime LastScraped { get; set; }
public string CategoryType { get; set; }
}Public Class Category
''' <summary>
''' Gets or sets the name.
''' </summary>
''' <value>
''' The name.
''' </value>
Public Property Name As String
''' <summary>
''' Gets or sets the URL.
''' </summary>
''' <value>
''' The URL.
''' </value>
Public Property URL As String
''' <summary>
''' Gets or sets the subcategories.
''' </summary>
''' <value>
''' The subcategories.
''' </value>
Public Property SubCategories As List(Of Category)
' Additional properties for enhanced data collection
Public Property ProductCount As Integer
Public Property LastScraped As DateTime
Public Property CategoryType As String
End Class
$vbLabelText
$csharpLabel
Temel Kazıyıcı Mantığını Nasıl Oluşturabilirim?
Kazıyıcı mantığını oluşturun ve kazıyıcıyı çalıştırmadan önce lisans anahtarınızı uygulamayı unutmayın:
public class ShoppingScraper : WebScraper
{
/// <summary>
/// Initialize the web scraper, setting the start URLs and allowed/banned domains or URL patterns.
/// </summary>
public override void Init()
{
// Apply your license key - get one from https://ironsoftware.com/csharp/webscraper/licensing/
License.LicenseKey = "LicenseKey";
this.LoggingLevel = WebScraper.LogLevel.All;
this.WorkingDirectory = AppSetting.GetAppRoot() + @"\ShoppingSiteSample\Output\";
// Configure request settings for better performance
this.Request("www.webSite.com", Parse);
}
/// <summary>
/// Parses the HTML document of the response to scrap the necessary data.
/// </summary>
/// <param name="response">The HTTP Response object to parse.</param>
public override void Parse(Response response)
{
var categoryList = new List<Category>();
// Iterate through each link in the menu and extract the category data.
foreach (var Links in response.Css("#menuFixed > ul > li > a"))
{
var cat = new Category
{
URL = Links.Attributes["href"],
Name = Links.InnerText,
LastScraped = DateTime.Now
};
categoryList.Add(cat);
}
// Save the scraped data into a JSONL file.
Scrape(categoryList, "Shopping.jsonl");
}
}public class ShoppingScraper : WebScraper
{
/// <summary>
/// Initialize the web scraper, setting the start URLs and allowed/banned domains or URL patterns.
/// </summary>
public override void Init()
{
// Apply your license key - get one from https://ironsoftware.com/csharp/webscraper/licensing/
License.LicenseKey = "LicenseKey";
this.LoggingLevel = WebScraper.LogLevel.All;
this.WorkingDirectory = AppSetting.GetAppRoot() + @"\ShoppingSiteSample\Output\";
// Configure request settings for better performance
this.Request("www.webSite.com", Parse);
}
/// <summary>
/// Parses the HTML document of the response to scrap the necessary data.
/// </summary>
/// <param name="response">The HTTP Response object to parse.</param>
public override void Parse(Response response)
{
var categoryList = new List<Category>();
// Iterate through each link in the menu and extract the category data.
foreach (var Links in response.Css("#menuFixed > ul > li > a"))
{
var cat = new Category
{
URL = Links.Attributes["href"],
Name = Links.InnerText,
LastScraped = DateTime.Now
};
categoryList.Add(cat);
}
// Save the scraped data into a JSONL file.
Scrape(categoryList, "Shopping.jsonl");
}
}Imports System
Imports System.Collections.Generic
Public Class ShoppingScraper
Inherits WebScraper
''' <summary>
''' Initialize the web scraper, setting the start URLs and allowed/banned domains or URL patterns.
''' </summary>
Public Overrides Sub Init()
' Apply your license key - get one from https://ironsoftware.com/csharp/webscraper/licensing/
License.LicenseKey = "LicenseKey"
Me.LoggingLevel = WebScraper.LogLevel.All
Me.WorkingDirectory = AppSetting.GetAppRoot() & "\ShoppingSiteSample\Output\"
' Configure request settings for better performance
Me.Request("www.webSite.com", AddressOf Parse)
End Sub
''' <summary>
''' Parses the HTML document of the response to scrap the necessary data.
''' </summary>
''' <param name="response">The HTTP Response object to parse.</param>
Public Overrides Sub Parse(response As Response)
Dim categoryList As New List(Of Category)()
' Iterate through each link in the menu and extract the category data.
For Each Links In response.Css("#menuFixed > ul > li > a")
Dim cat As New Category With {
.URL = Links.Attributes("href"),
.Name = Links.InnerText,
.LastScraped = DateTime.Now
}
categoryList.Add(cat)
Next
' Save the scraped data into a JSONL file.
Scrape(categoryList, "Shopping.jsonl")
End Sub
End Class
$vbLabelText
$csharpLabel
Menüde Hangi Öğeleri Hedefliyorum?
Menüdeki bağlantıların kazınması hassas CSS seçicileri gerektirir. API Referansi mevcut seçici yöntemleri hakkında ayrıntılı bilgi sağlar:
Hem Ana Kategorileri Hem de Alt Kategorileri Nasıl Kazırım?
Ana kategorileri ve tüm alt bağlantıları kazımak için kodu güncelleyin. Bu yaklaşım, navigasyon yapısının tam olarak yakalanmasını sağlar:
public override void Parse(Response response)
{
// List of Category Links (Root)
var categoryList = new List<Category>();
// Traverse each 'li' under the fixed menu
foreach (var li in response.Css("#menuFixed > ul > li"))
{
// List of Main Links
foreach (var Links in li.Css("a"))
{
var cat = new Category
{
URL = Links.Attributes["href"],
Name = Links.InnerText,
SubCategories = new List<Category>(),
LastScraped = DateTime.Now
};
// List of Subcategories Links
foreach (var subCategory in li.Css("a[class=subcategory]"))
{
var subcat = new Category
{
URL = subCategory.Attributes["href"],
Name = subCategory.InnerText,
CategoryType = "Subcategory"
};
// Check if subcategory link already exists
if (cat.SubCategories.Find(c => c.Name == subcat.Name && c.URL == subcat.URL) == null)
{
// Add sublinks
cat.SubCategories.Add(subcat);
}
}
// Update product count based on subcategories
cat.ProductCount = cat.SubCategories.Count;
// Add Main Category to the list
categoryList.Add(cat);
}
}
// Save the scraped data into a JSONL file.
Scrape(categoryList, "Shopping.jsonl");
}public override void Parse(Response response)
{
// List of Category Links (Root)
var categoryList = new List<Category>();
// Traverse each 'li' under the fixed menu
foreach (var li in response.Css("#menuFixed > ul > li"))
{
// List of Main Links
foreach (var Links in li.Css("a"))
{
var cat = new Category
{
URL = Links.Attributes["href"],
Name = Links.InnerText,
SubCategories = new List<Category>(),
LastScraped = DateTime.Now
};
// List of Subcategories Links
foreach (var subCategory in li.Css("a[class=subcategory]"))
{
var subcat = new Category
{
URL = subCategory.Attributes["href"],
Name = subCategory.InnerText,
CategoryType = "Subcategory"
};
// Check if subcategory link already exists
if (cat.SubCategories.Find(c => c.Name == subcat.Name && c.URL == subcat.URL) == null)
{
// Add sublinks
cat.SubCategories.Add(subcat);
}
}
// Update product count based on subcategories
cat.ProductCount = cat.SubCategories.Count;
// Add Main Category to the list
categoryList.Add(cat);
}
}
// Save the scraped data into a JSONL file.
Scrape(categoryList, "Shopping.jsonl");
}Option Strict On
Public Overrides Sub Parse(response As Response)
' List of Category Links (Root)
Dim categoryList As New List(Of Category)()
' Traverse each 'li' under the fixed menu
For Each li In response.Css("#menuFixed > ul > li")
' List of Main Links
For Each Links In li.Css("a")
Dim cat As New Category With {
.URL = Links.Attributes("href"),
.Name = Links.InnerText,
.SubCategories = New List(Of Category)(),
.LastScraped = DateTime.Now
}
' List of Subcategories Links
For Each subCategory In li.Css("a[class=subcategory]")
Dim subcat As New Category With {
.URL = subCategory.Attributes("href"),
.Name = subCategory.InnerText,
.CategoryType = "Subcategory"
}
' Check if subcategory link already exists
If cat.SubCategories.Find(Function(c) c.Name = subcat.Name AndAlso c.URL = subcat.URL) Is Nothing Then
' Add sublinks
cat.SubCategories.Add(subcat)
End If
Next
' Update product count based on subcategories
cat.ProductCount = cat.SubCategories.Count
' Add Main Category to the list
categoryList.Add(cat)
Next
Next
' Save the scraped data into a JSONL file.
Scrape(categoryList, "Shopping.jsonl")
End Sub
$vbLabelText
$csharpLabel
Kategori Sayfalarından Ürün Bilgilerini Nasıl Çıkarırım?
Tüm site kategorilerine bağlantılar mevcut olduğundan, her kategorideki ürünleri kazımaya başlayın. Ürün sayfalarıyla uğraşırken, optimum performans için thread safety önemli hale gelir. Herhangi bir kategoriye gidin ve içeriği inceleyin:
Ürün HTML Yapısı Nasıl Görünüyor?
Ürün organizasyonunu anlamak için HTML yapısını inceleyin:
<section class="products">
<div class="sku -gallery -validate-size " data-sku="AG249FA0T2PSGNAFAMZ" ft-product-sizes="41,42,43,44,45" ft-product-color="Multicolour">
<a class="link" href="http://www.WebSite.com/agu-bundle-of-2-sneakers-black-navy-blue-653884.html">
<div class="image-wrapper default-state">
<img class="lazy image -loaded" alt="Bundle Of 2 Sneakers - Black & Navy Blue" data-image-vertical="1" width="210" height="262" src="https://static.WebSite.com/p/agu-6208-488356-1-catalog_grid_3.jpg" data-sku="AG249FA0T2PSGNAFAMZ" data-src="https://static.WebSite.com/p/agu-6208-488356-1-catalog_grid_3.jpg" data-placeholder="placeholder_m_1.jpg">
<noscript><img src="https://static.WebSite.com/p/agu-6208-488356-1-catalog_grid_3.jpg" width="210" height="262" class="image" /></noscript>
</div>
<h2 class="title"></h2>
<span class="brand ">Agu </span>
<span class="name" dir="ltr">Bundle Of 2 Sneakers - Black & Navy Blue</span>
</h2>
<div class="price-container clearfix">
<span class="price-box">
<span class="price">
<span data-currency-iso="EGP">EGP</span>
<span dir="ltr" data-price="299">299</span>
</span>
<span class="price -old -no-special"></span>
</span>
</div>
<div class="rating-stars">
<div class="stars-container">
<div class="stars" style="width: 62%"></div>
</div>
<div class="total-ratings">(30)</div>
</div>
<span class="shop-first-logo-container">
<img src="http://www.WebSite.com/images/local/logos/shop_first/ShoppingSite/logo_normal.png" data-src="http://www.WebSite.com/images/local/logos/shop_first/ShoppingSite/logo_normal.png" class="lazy shop-first-logo-img -mbxs -loaded">
</span>
<span class="osh-icon -ShoppingSite-local shop_local--logo -block -mbs -mts"></span>
<div class="list -sizes" data-selected-sku="">
<span class="js-link sku-size" data-href="http://www.WebSite.com/agu-bundle-of-2-sneakers-black-navy-blue-653884.html?size=41">41</span>
<span class="js-link sku-size" data-href="http://www.WebSite.com/agu-bundle-of-2-sneakers-black-navy-blue-653884.html?size=42">42</span>
<span class="js-link sku-size" data-href="http://www.WebSite.com/agu-bundle-of-2-sneakers-black-navy-blue-653884.html?size=43">43</span>
<span class="js-link sku-size" data-href="http://www.WebSite.com/agu-bundle-of-2-sneakers-black-navy-blue-653884.html?size=44">44</span>
<span class="js-link sku-size" data-href="http://www.WebSite.com/agu-bundle-of-2-sneakers-black-navy-blue-653884.html?size=45">45</span>
</div>
</a>
</div>
<div class="sku -gallery -validate-size " data-sku="LE047FA01SRK4NAFAMZ" ft-product-sizes="110,115,120,125,130,135" ft-product-color="Black">
<a class="link" href="http://www.WebSite.com/leather-shop-genuine-leather-belt-black-712030.html">
<div class="image-wrapper default-state">
<img class="lazy image -loaded" alt="Genuine Leather Belt - Black" data-image-vertical="1" width="210" height="262" src="https://static.WebSite.com/p/leather-shop-1831-030217-1-catalog_grid_3.jpg" data-sku="LE047FA01SRK4NAFAMZ" data-src="https://static.WebSite.com/p/leather-shop-1831-030217-1-catalog_grid_3.jpg" data-placeholder="placeholder_m_1.jpg">
<noscript><img src="https://static.WebSite.com/p/leather-shop-1831-030217-1-catalog_grid_3.jpg" width="210" height="262" class="image" /></noscript>
</div>
<h2 class="title"><span class="brand ">Leather Shop </span> <span class="name" dir="ltr">Genuine Leather Belt - Black</span></h2>
<div class="price-container clearfix">
<span class="sale-flag-percent">-29%</span>
<span class="price-box">
<span class="price"><span data-currency-iso="EGP">EGP</span> <span dir="ltr" data-price="96">96</span> </span>
<span class="price -old"><span data-currency-iso="EGP">EGP</span> <span dir="ltr" data-price="135">135</span> </span>
</span>
</div>
<div class="rating-stars">
<div class="stars-container">
<div class="stars" style="width: 100%"></div>
</div>
<div class="total-ratings">(1)</div>
</div>
<span class="osh-icon -ShoppingSite-local shop_local--logo -block -mbs -mts"></span>
<div class="list -sizes" data-selected-sku="">
<span class="js-link sku-size" data-href="http://www.WebSite.com/leather-shop-genuine-leather-belt-black-712030.html?size=110">110</span>
<span class="js-link sku-size"data-href="http://www.WebSite.com/leather-shop-genuine-leather-belt-black-712030.html?size=115">115</span>
<span class="js-link sku-size"data-href="http://www.WebSite.com/leather-shop-genuine-leather-belt-black-712030.html?size=120">120</span>
<span class="js-link sku-size"data-href="http://www.WebSite.com/leather-shop-genuine-leather-belt-black-712030.html?size=125">125</span>
<span class="js-link sku-size"data-href="http://www.WebSite.com/leather-shop-genuine-leather-belt-black-712030.html?size=130">130</span>
<span class="js-link sku-size"data-href="http://www.WebSite.com/leather-shop-genuine-leather-belt-black-712030.html?size=135">135</span>
</div>
</a>
</div>
</section><section class="products">
<div class="sku -gallery -validate-size " data-sku="AG249FA0T2PSGNAFAMZ" ft-product-sizes="41,42,43,44,45" ft-product-color="Multicolour">
<a class="link" href="http://www.WebSite.com/agu-bundle-of-2-sneakers-black-navy-blue-653884.html">
<div class="image-wrapper default-state">
<img class="lazy image -loaded" alt="Bundle Of 2 Sneakers - Black & Navy Blue" data-image-vertical="1" width="210" height="262" src="https://static.WebSite.com/p/agu-6208-488356-1-catalog_grid_3.jpg" data-sku="AG249FA0T2PSGNAFAMZ" data-src="https://static.WebSite.com/p/agu-6208-488356-1-catalog_grid_3.jpg" data-placeholder="placeholder_m_1.jpg">
<noscript><img src="https://static.WebSite.com/p/agu-6208-488356-1-catalog_grid_3.jpg" width="210" height="262" class="image" /></noscript>
</div>
<h2 class="title"></h2>
<span class="brand ">Agu </span>
<span class="name" dir="ltr">Bundle Of 2 Sneakers - Black & Navy Blue</span>
</h2>
<div class="price-container clearfix">
<span class="price-box">
<span class="price">
<span data-currency-iso="EGP">EGP</span>
<span dir="ltr" data-price="299">299</span>
</span>
<span class="price -old -no-special"></span>
</span>
</div>
<div class="rating-stars">
<div class="stars-container">
<div class="stars" style="width: 62%"></div>
</div>
<div class="total-ratings">(30)</div>
</div>
<span class="shop-first-logo-container">
<img src="http://www.WebSite.com/images/local/logos/shop_first/ShoppingSite/logo_normal.png" data-src="http://www.WebSite.com/images/local/logos/shop_first/ShoppingSite/logo_normal.png" class="lazy shop-first-logo-img -mbxs -loaded">
</span>
<span class="osh-icon -ShoppingSite-local shop_local--logo -block -mbs -mts"></span>
<div class="list -sizes" data-selected-sku="">
<span class="js-link sku-size" data-href="http://www.WebSite.com/agu-bundle-of-2-sneakers-black-navy-blue-653884.html?size=41">41</span>
<span class="js-link sku-size" data-href="http://www.WebSite.com/agu-bundle-of-2-sneakers-black-navy-blue-653884.html?size=42">42</span>
<span class="js-link sku-size" data-href="http://www.WebSite.com/agu-bundle-of-2-sneakers-black-navy-blue-653884.html?size=43">43</span>
<span class="js-link sku-size" data-href="http://www.WebSite.com/agu-bundle-of-2-sneakers-black-navy-blue-653884.html?size=44">44</span>
<span class="js-link sku-size" data-href="http://www.WebSite.com/agu-bundle-of-2-sneakers-black-navy-blue-653884.html?size=45">45</span>
</div>
</a>
</div>
<div class="sku -gallery -validate-size " data-sku="LE047FA01SRK4NAFAMZ" ft-product-sizes="110,115,120,125,130,135" ft-product-color="Black">
<a class="link" href="http://www.WebSite.com/leather-shop-genuine-leather-belt-black-712030.html">
<div class="image-wrapper default-state">
<img class="lazy image -loaded" alt="Genuine Leather Belt - Black" data-image-vertical="1" width="210" height="262" src="https://static.WebSite.com/p/leather-shop-1831-030217-1-catalog_grid_3.jpg" data-sku="LE047FA01SRK4NAFAMZ" data-src="https://static.WebSite.com/p/leather-shop-1831-030217-1-catalog_grid_3.jpg" data-placeholder="placeholder_m_1.jpg">
<noscript><img src="https://static.WebSite.com/p/leather-shop-1831-030217-1-catalog_grid_3.jpg" width="210" height="262" class="image" /></noscript>
</div>
<h2 class="title"><span class="brand ">Leather Shop </span> <span class="name" dir="ltr">Genuine Leather Belt - Black</span></h2>
<div class="price-container clearfix">
<span class="sale-flag-percent">-29%</span>
<span class="price-box">
<span class="price"><span data-currency-iso="EGP">EGP</span> <span dir="ltr" data-price="96">96</span> </span>
<span class="price -old"><span data-currency-iso="EGP">EGP</span> <span dir="ltr" data-price="135">135</span> </span>
</span>
</div>
<div class="rating-stars">
<div class="stars-container">
<div class="stars" style="width: 100%"></div>
</div>
<div class="total-ratings">(1)</div>
</div>
<span class="osh-icon -ShoppingSite-local shop_local--logo -block -mbs -mts"></span>
<div class="list -sizes" data-selected-sku="">
<span class="js-link sku-size" data-href="http://www.WebSite.com/leather-shop-genuine-leather-belt-black-712030.html?size=110">110</span>
<span class="js-link sku-size"data-href="http://www.WebSite.com/leather-shop-genuine-leather-belt-black-712030.html?size=115">115</span>
<span class="js-link sku-size"data-href="http://www.WebSite.com/leather-shop-genuine-leather-belt-black-712030.html?size=120">120</span>
<span class="js-link sku-size"data-href="http://www.WebSite.com/leather-shop-genuine-leather-belt-black-712030.html?size=125">125</span>
<span class="js-link sku-size"data-href="http://www.WebSite.com/leather-shop-genuine-leather-belt-black-712030.html?size=130">130</span>
<span class="js-link sku-size"data-href="http://www.WebSite.com/leather-shop-genuine-leather-belt-black-712030.html?size=135">135</span>
</div>
</a>
</div>
</section>
HTML
Hangi Ürün Modelini Oluşturmalıyım?
Bu içerik için bir ürün modeli oluşturun. alışveriş sitesi kazıma ile çalışırken, ilgili tüm ürün ayrıntılarını yakalayın:
public class Product
{
/// <summary>
/// Gets or sets the name.
/// </summary>
/// <value>
/// The name.
/// </value>
public string Name { get; set; }
/// <summary>
/// Gets or sets the price.
/// </summary>
/// <value>
/// The price.
/// </value>
public string Price { get; set; }
/// <summary>
/// Gets or sets the image.
/// </summary>
/// <value>
/// The image.
/// </value>
public string Image { get; set; }
// Additional properties for comprehensive data collection
public string Brand { get; set; }
public string OldPrice { get; set; }
public string Discount { get; set; }
public float Rating { get; set; }
public int ReviewCount { get; set; }
public List<string> AvailableSizes { get; set; }
public string ProductUrl { get; set; }
public string SKU { get; set; }
public DateTime ScrapedDate { get; set; }
}public class Product
{
/// <summary>
/// Gets or sets the name.
/// </summary>
/// <value>
/// The name.
/// </value>
public string Name { get; set; }
/// <summary>
/// Gets or sets the price.
/// </summary>
/// <value>
/// The price.
/// </value>
public string Price { get; set; }
/// <summary>
/// Gets or sets the image.
/// </summary>
/// <value>
/// The image.
/// </value>
public string Image { get; set; }
// Additional properties for comprehensive data collection
public string Brand { get; set; }
public string OldPrice { get; set; }
public string Discount { get; set; }
public float Rating { get; set; }
public int ReviewCount { get; set; }
public List<string> AvailableSizes { get; set; }
public string ProductUrl { get; set; }
public string SKU { get; set; }
public DateTime ScrapedDate { get; set; }
}Public Class Product
''' <summary>
''' Gets or sets the name.
''' </summary>
''' <value>
''' The name.
''' </value>
Public Property Name As String
''' <summary>
''' Gets or sets the price.
''' </summary>
''' <value>
''' The price.
''' </value>
Public Property Price As String
''' <summary>
''' Gets or sets the image.
''' </summary>
''' <value>
''' The image.
''' </value>
Public Property Image As String
' Additional properties for comprehensive data collection
Public Property Brand As String
Public Property OldPrice As String
Public Property Discount As String
Public Property Rating As Single
Public Property ReviewCount As Integer
Public Property AvailableSizes As List(Of String)
Public Property ProductUrl As String
Public Property SKU As String
Public Property ScrapedDate As DateTime
End Class
$vbLabelText
$csharpLabel
Ürün Kazıma İşlevselliğini Nasıl Ekleyebilirim?
Kategori sayfalarını kazımak için hata işleme ve veri doğrulama özelliklerine sahip yeni bir kazıma yöntemi ekleyin:
public void ParseCategory(Response response)
{
// List of Products
var productList = new List<Product>();
// Iterate through product links in the product section
foreach (var Links in response.Css("section.products > div > a"))
{
try
{
var product = new Product
{
Name = Links.Css("h2.title > span.name").First().InnerText,
Brand = Links.Css("h2.title > span.brand").FirstOrDefault()?.InnerText ?? "Unknown",
Price = Links.Css("div.price-container > span.price-box > span.price > span[data-price]").First().InnerText,
Image = Links.Css("div.image-wrapper.default-state > img").First().Attributes["src"],
ProductUrl = Links.Attributes["href"],
SKU = Links.ParentNode.Attributes["data-sku"],
ScrapedDate = DateTime.Now
};
// Extract old price if available
var oldPriceElement = Links.Css("span.price.-old > span[data-price]").FirstOrDefault();
if (oldPriceElement != null)
{
product.OldPrice = oldPriceElement.InnerText;
}
// Extract discount percentage
var discountElement = Links.Css("span.sale-flag-percent").FirstOrDefault();
if (discountElement != null)
{
product.Discount = discountElement.InnerText;
}
// Extract rating information
var ratingWidth = Links.Css("div.stars").FirstOrDefault()?.Attributes["style"];
if (!string.IsNullOrEmpty(ratingWidth))
{
var width = System.Text.RegularExpressions.Regex.Match(ratingWidth, @"(\d+)%").Groups[1].Value;
if (int.TryParse(width, out int ratingPercent))
{
product.Rating = ratingPercent / 20.0f; // Convert percentage to 5-star scale
}
}
// Extract review count
var reviewText = Links.Css("div.total-ratings").FirstOrDefault()?.InnerText;
if (!string.IsNullOrEmpty(reviewText))
{
var reviewCount = System.Text.RegularExpressions.Regex.Match(reviewText, @"\d+").Value;
if (int.TryParse(reviewCount, out int count))
{
product.ReviewCount = count;
}
}
// Extract available sizes
product.AvailableSizes = Links.Css("div.list.-sizes > span.sku-size")
.Select(s => s.InnerText)
.ToList();
productList.Add(product);
}
catch (Exception ex)
{
// Log error and continue with next product
Console.WriteLine($"Error parsing product: {ex.Message}");
}
}
// Save the scraped product data into a JSONL file.
Scrape(productList, "Products.jsonl");
// Handle pagination if needed
var nextPageLink = response.Css("a.pagination-next").FirstOrDefault();
if (nextPageLink != null)
{
var nextPageUrl = nextPageLink.Attributes["href"];
this.Request(nextPageUrl, ParseCategory);
}
}public void ParseCategory(Response response)
{
// List of Products
var productList = new List<Product>();
// Iterate through product links in the product section
foreach (var Links in response.Css("section.products > div > a"))
{
try
{
var product = new Product
{
Name = Links.Css("h2.title > span.name").First().InnerText,
Brand = Links.Css("h2.title > span.brand").FirstOrDefault()?.InnerText ?? "Unknown",
Price = Links.Css("div.price-container > span.price-box > span.price > span[data-price]").First().InnerText,
Image = Links.Css("div.image-wrapper.default-state > img").First().Attributes["src"],
ProductUrl = Links.Attributes["href"],
SKU = Links.ParentNode.Attributes["data-sku"],
ScrapedDate = DateTime.Now
};
// Extract old price if available
var oldPriceElement = Links.Css("span.price.-old > span[data-price]").FirstOrDefault();
if (oldPriceElement != null)
{
product.OldPrice = oldPriceElement.InnerText;
}
// Extract discount percentage
var discountElement = Links.Css("span.sale-flag-percent").FirstOrDefault();
if (discountElement != null)
{
product.Discount = discountElement.InnerText;
}
// Extract rating information
var ratingWidth = Links.Css("div.stars").FirstOrDefault()?.Attributes["style"];
if (!string.IsNullOrEmpty(ratingWidth))
{
var width = System.Text.RegularExpressions.Regex.Match(ratingWidth, @"(\d+)%").Groups[1].Value;
if (int.TryParse(width, out int ratingPercent))
{
product.Rating = ratingPercent / 20.0f; // Convert percentage to 5-star scale
}
}
// Extract review count
var reviewText = Links.Css("div.total-ratings").FirstOrDefault()?.InnerText;
if (!string.IsNullOrEmpty(reviewText))
{
var reviewCount = System.Text.RegularExpressions.Regex.Match(reviewText, @"\d+").Value;
if (int.TryParse(reviewCount, out int count))
{
product.ReviewCount = count;
}
}
// Extract available sizes
product.AvailableSizes = Links.Css("div.list.-sizes > span.sku-size")
.Select(s => s.InnerText)
.ToList();
productList.Add(product);
}
catch (Exception ex)
{
// Log error and continue with next product
Console.WriteLine($"Error parsing product: {ex.Message}");
}
}
// Save the scraped product data into a JSONL file.
Scrape(productList, "Products.jsonl");
// Handle pagination if needed
var nextPageLink = response.Css("a.pagination-next").FirstOrDefault();
if (nextPageLink != null)
{
var nextPageUrl = nextPageLink.Attributes["href"];
this.Request(nextPageUrl, ParseCategory);
}
}Public Sub ParseCategory(response As Response)
' List of Products
Dim productList As New List(Of Product)()
' Iterate through product links in the product section
For Each Links In response.Css("section.products > div > a")
Try
Dim product As New Product With {
.Name = Links.Css("h2.title > span.name").First().InnerText,
.Brand = If(Links.Css("h2.title > span.brand").FirstOrDefault()?.InnerText, "Unknown"),
.Price = Links.Css("div.price-container > span.price-box > span.price > span[data-price]").First().InnerText,
.Image = Links.Css("div.image-wrapper.default-state > img").First().Attributes("src"),
.ProductUrl = Links.Attributes("href"),
.SKU = Links.ParentNode.Attributes("data-sku"),
.ScrapedDate = DateTime.Now
}
' Extract old price if available
Dim oldPriceElement = Links.Css("span.price.-old > span[data-price]").FirstOrDefault()
If oldPriceElement IsNot Nothing Then
product.OldPrice = oldPriceElement.InnerText
End If
' Extract discount percentage
Dim discountElement = Links.Css("span.sale-flag-percent").FirstOrDefault()
If discountElement IsNot Nothing Then
product.Discount = discountElement.InnerText
End If
' Extract rating information
Dim ratingWidth = Links.Css("div.stars").FirstOrDefault()?.Attributes("style")
If Not String.IsNullOrEmpty(ratingWidth) Then
Dim width = System.Text.RegularExpressions.Regex.Match(ratingWidth, "(\d+)%").Groups(1).Value
Dim ratingPercent As Integer
If Integer.TryParse(width, ratingPercent) Then
product.Rating = ratingPercent / 20.0F ' Convert percentage to 5-star scale
End If
End If
' Extract review count
Dim reviewText = Links.Css("div.total-ratings").FirstOrDefault()?.InnerText
If Not String.IsNullOrEmpty(reviewText) Then
Dim reviewCount = System.Text.RegularExpressions.Regex.Match(reviewText, "\d+").Value
Dim count As Integer
If Integer.TryParse(reviewCount, count) Then
product.ReviewCount = count
End If
End If
' Extract available sizes
product.AvailableSizes = Links.Css("div.list.-sizes > span.sku-size") _
.Select(Function(s) s.InnerText) _
.ToList()
productList.Add(product)
Catch ex As Exception
' Log error and continue with next product
Console.WriteLine($"Error parsing product: {ex.Message}")
End Try
Next
' Save the scraped product data into a JSONL file.
Scrape(productList, "Products.jsonl")
' Handle pagination if needed
Dim nextPageLink = response.Css("a.pagination-next").FirstOrDefault()
If nextPageLink IsNot Nothing Then
Dim nextPageUrl = nextPageLink.Attributes("href")
Me.Request(nextPageUrl, AddressOf ParseCategory)
End If
End Sub
$vbLabelText
$csharpLabel
Alışveriş sitelerinin kazınmasına yönelik bu kapsamlı yaklaşım, hataları incelikle ele alırken ilgili tüm ürün bilgilerinin yakalanmasını sağlar. Daha gelişmiş senaryolar için, IronWebScraper'da bulunan gelişmiş web kazıma özelliklerini inceleyin.
Sıkça Sorulan Sorular
Alışveriş web sitelerinden ürün verilerini C# ile nasıl çıkarabilirim?
IronWebScraper, alışveriş web sitelerinden ürün verilerini çıkarmayı CSS seçicileri kullanarak kolaylaştırır. Bir WebScraper sınıfı oluşturabilir, Parse yöntemini geçersiz kılabilir ve response.Css()'i kullanarak ürün adları, fiyatları ve görüntüleri gibi belirli HTML öğelerini hedefleyebilirsiniz. Çıkartılan veriler JSON ve JSONL dosyaları dahil çeşitli formatlarda kaydedilebilir.
Bir alışveriş web sitesi kazıyıcısı oluşturmak için temel adımlar nelerdir?
IronWebScraper ile bir alışveriş web sitesi kazıyıcısı oluşturmak için: 1) Konsol Uygulama projesi oluşturun, 2) WebScraper'dan türeyen bir sınıf ekleyin, 3) Kategoriler ve ürünler için veri modelleri oluşturun, 4) Başlangıç URL'nizi ayarlamak için Init() yöntemi geçersiz kılın, 5) CSS seçicileri kullanarak verileri çıkarmak için Parse() yöntemi geçersiz kılın ve 6) Verileri tercih ettiğiniz formata kaydetmek için kazıyıcıyı çalıştırın.
E-ticaret sitelerini kazırken hiyerarşik kategori yapılarını nasıl ele alabilirim?
IronWebScraper, E-ticaret sitelerini kazırken hiyerarşik yapıların üstesinden gelmek için uygun veri modelleri oluşturarak üst-alt ilişkileri (Moda > Erkek > Ayakkabı gibi) yansıtabilir. CSS seçicileri kullanarak iç içe geçmiş HTML öğeleri arasında gezinebilir ve IronWebScraper'ın gelişmiş özellikleriyle çalışırken özellikle yararlı olan kategori ağaç yapınızı programatik olarak oluşturabilirsiniz.
Bir alışveriş sitesinin HTML yapısını kazımadan önce analiz etmenin en iyi yolu nedir?
Bir alışveriş sitesini kazımadan önce IronWebScraper kullanarak, tarayıcı geliştirici araçlarını kullanarak HTML yapısını inceleyin. CSS sınıflarında ve öğe hiyerarşilerinde tutarlı kalıplar arayın. Bu analiz, IronWebScraper Parse() yönteminizde doğru CSS seçicileri kullanarak ürün bilgilerini, kategorileri ve diğer veri öğelerini doğru bir şekilde hedeflemenize yardımcı olur.
Aynı sayfadan hem ürün listeleri hem de kategori gezinmesi çıkarabilir miyim?
Evet, IronWebScraper bir sayfadan birden fazla veri türünü çıkarmayı mümkün kılar. Parse() yönteminizde, kategori bağlantılarını ('.category-item' gibi) ve ürün listelerini ('.product-item' gibi) hedeflemek için farklı CSS seçiciler kullanabilir, ardından bunları ayrı çıktı dosyalarına veya veri yapılarına kaydedebilirsiniz.
Kazınmış ürün verilerini bir dosyaya nasıl kaydederim?
IronWebScraper, verileri otomatik olarak kaydeden yerleşik bir Scrape() yöntemi sağlar. Sadece veri nesnenizi ve dosya adınızı Scrape(item, "products.jsonl")'a geçirin. Kütüphane, kazınmış e-ticaret verilerinizi daha fazla işlemek için JSON, JSONL ve CSV gibi çeşitli çıktı formatlarını destekler.
Başlamaya Hazır mısınız?
Nuget İndirmeler 137,906 | Sürüm: 2026.6 just released
Hâlâ Kaydırıyor Musunuz?
Hızlıca kanıt ister misiniz? PM > Install-Package IronWebScraper
örnek çalıştır hedef sitenizi yapılandırılmış verilere dönüştürün.
Ücretsiz Başlayın
Ücretsiz Başlayın
Ücretsiz Canlı Demo Rezervasyonu
Kredi kartı veya hesap oluşturma gerektirmez
