# Scrape a Shopping Website in C#
Learn how to scrape product categories and items from shopping websites using C# with the `WebScraper` framework, extracting structured data from HTML elements into custom models. This comprehensive guide walks you through building a robust e-commerce scraper using the [`IronWebScraper` library](https://ironsoftware.com/csharp/webscraper/).
*as-heading:2(Quickstart: Scrape Shopping Website in C#)*
```csharp
:title=QuickstartShoppingScraper.cs
using IronWebScraper;
public class QuickShoppingScraper : WebScraper
{
public override void Init()
{
// Apply your license key
License.LicenseKey = "YOUR-LICENSE-KEY";
// Set the starting URL
this.Request("https://shopping-site.com", Parse);
}
public override void Parse(Response response)
{
// Extract product data
foreach (var product in response.Css(".product-item"))
{
var item = new
{
Name = product.Css(".product-name").First().InnerText,
Price = product.Css(".price").First().InnerText,
Image = product.Css("img").First().Attributes["src"]
};
Scrape(item, "products.jsonl");
}
}
}
// Run the scraper
var scraper = new QuickShoppingScraper();
scraper.Start();
```
1. Create a new Console App project named "ShoppingSiteSample"
2. Add a class named "`ShoppingScraper`" that inherits from `WebScraper`
3. Create models for `Category` and `Product` data
4. Override `Init()` to set start URL and `Parse()` method for scraping
5. Run the scraper to extract categories and products to JSONL files
## How Do I Analyze the Shopping Site's HTML Structure?
Select a shopping site to analyze its content structure. Understanding the HTML structure is crucial for successful web scraping. Before writing any code, spend time analyzing the target website's structure using browser developer tools.
<p><a rel="nofollow" href="/img/tutorials/webscraping-in-c-sharp/shoppingSite.jpg" target="_blank"><img src="/img/tutorials/webscraping-in-c-sharp/shoppingSite.jpg" alt="Jumia e-commerce homepage with Ramadan promotional banner and navigation menu" class="img-responsive add-shadow img-margin" /></a></p>
As shown in the image, the left sidebar contains links for the site's product categories. The first step is investigating the site's HTML and planning the scraping approach. This analysis phase is essential for building an effective scraping strategy.
<p><a rel="nofollow" href="/img/tutorials/webscraping-in-c-sharp/shoppingSiteLeftBar.jpg" target="_blank"><img src="/img/tutorials/webscraping-in-c-sharp/shoppingSiteLeftBar.jpg" alt="E-commerce website navigation menu showing product categories, subcategories, and brand sections" class="img-responsive add-shadow img-margin" /></a></p>
### Why Does Understanding the HTML Structure Matter?
The fashion site categories have subcategories (Men, Women, Kids). Understanding this hierarchical structure helps design appropriate data models and scraping logic. When working with [advanced web scraping features](https://ironsoftware.com/csharp/webscraper/tutorials/webscraping-in-c-sharp-advanced/), proper HTML analysis becomes even more critical.
```html
<li class="menu-item" data-id="">
<a href="https://domain.com/fashion-by-/" class="main-category">
<i class="cat-icon osh-font-fashion"></i>
<span class="nav-subTxt">FASHION </span>
<i class="osh-font-light-arrow-left"></i><i class="osh-font-light-arrow-right"></i>
</a>
<div class="navLayerWrapper" style="width: 633px; display: none;">
<div class="submenu">
<div class="column">
<div class="categories">
<a class="category" href="https://domain.com/fashion-by-/?sort=newest&dir=desc&viewType=gridView3">New Arrivals !</a>
</div>
<div class="categories">
<a class="category" href="https://domain.com/men-fashion/">Men</a>
<a class="subcategory" href="https://domain.com/mens-shoes/">Shoes</a>
<a class="subcategory" href="https://domain.com/mens-clothing/">Clothing</a>
<a class="subcategory" href="https://domain.com/mens-accessories/">Accessories</a>
</div>
<div class="categories">
<a class="category" href="https://domain.com/women-fashion/">Women</a>
<a class="subcategory" href="https://domain.com/womens-shoes/">Shoes</a>
<a class="subcategory" href="https://domain.com/womens-clothing/">Clothing</a>
<a class="subcategory" href="https://domain.com/womens-accessories/">Accessories</a>
</div>
<div class="categories">
<a class="category" href="https://domain.com/girls-boys-fashion/">Kids</a>
<a class="subcategory" href="https://domain.com/boys-fashion/">Boys</a>
<a class="subcategory" href="https://domain.com/girls/">Girls</a>
</div>
<div class="categories">
<a class="category" href="https://domain.com/maternity-clothes/">Maternity Clothes</a>
</div>
</div>
<div class="column">
<div class="categories">
<span class="category defaultCursor">Men Best Sellers</span>
<a class="subcategory" href="https://domain.com/mens-casual-shoes/">Casual Shoes</a>
<a class="subcategory" href="https://domain.com/mens-sneakers/">Sneakers</a>
<a class="subcategory" href="https://domain.com/mens-t-shirts/">T-shirts</a>
<a class="subcategory" href="https://domain.com/mens-polos/">Polos</a>
</div>
<div class="categories">
<span class="category defaultCursor">Women Best Sellers</span>
<a class="subcategory" href="https://domain.com/womens-sandals/">Sandals</a>
<a class="subcategory" href="https://domain.com/womens-sneakers/">Sneakers</a>
<a class="subcategory" href="https://domain.com/women-dresses/">Dresses</a>
<a class="subcategory" href="https://domain.com/women-tops/">Tops</a>
</div>
<div class="categories">
<a class="category" href="https://domain.com/womens-curvy-clothing/">Women's Curvy Clothing</a>
</div>
<div class="categories">
<a class="category" href="https://domain.com/fashion-bundles/v/">Fashion Bundles</a>
</div>
<div class="categories">
<a class="category" href="https://domain.com/hijab-fashion/">Hijab Fashion</a>
</div>
</div>
<div class="column">
<div class="categories">
<a class="category" href="https://domain.com/brands/fashion-by-/">SEE ALL BRANDS</a>
<a class="subcategory" href="https://domain.com/adidas/">Adidas</a>
<a class="subcategory" href="https://domain.com/converse/">Converse</a>
<a class="subcategory" href="https://domain.com/ravin/">Ravin</a>
<a class="subcategory" href="https://domain.com/dejavu/">Dejavu</a>
<a class="subcategory" href="https://domain.com/agu/">Agu</a>
<a class="subcategory" href="https://domain.com/activ/">Activ</a>
<a class="subcategory" href="https://domain.com/oxford--bellini--tie-house--milano/">Tie House</a>
<a class="subcategory" href="https://domain.com/shoe-room/">Shoe Room</a>
<a class="subcategory" href="https://domain.com/town-team/">Town Team</a>
</div>
</div>
</div>
</div>
</li>
```
## How Do I Set Up the Web Scraping Project?
Set up a project following best practices for [C# web scraping](https://ironsoftware.com/csharp/webscraper/tutorials/webscraping-in-c-sharp/).
1. Create a new Console App or add a new folder for the sample named "ShoppingSiteSample"
2. Add a new class named "`ShoppingScraper`"
3. Start by scraping site categories and their subcategories
4. Install `IronWebScraper` via NuGet Package Manager or Package Manager Console:
```shell
:ProductInstall
```
### What Data Model Should I Use for Categories?
Create a Categories Model that properly represents the hierarchical structure discovered:
```csharp
public class Category
{
/// <summary>
/// Gets or sets the name.
/// </summary>
/// <value>
/// The name.
/// </value>
public string Name { get; set; }
/// <summary>
/// Gets or sets the URL.
/// </summary>
/// <value>
/// The URL.
/// </value>
public string URL { get; set; }
/// <summary>
/// Gets or sets the subcategories.
/// </summary>
/// <value>
/// The subcategories.
/// </value>
public List<Category> SubCategories { get; set; }
// Additional properties for enhanced data collection
public int ProductCount { get; set; }
public DateTime LastScraped { get; set; }
public string CategoryType { get; set; }
}
```
## How Do I Build the Basic Scraper Logic?
Build the scraper logic, remembering to [apply your license key](https://ironsoftware.com/csharp/webscraper/get-started/license-keys/) before running the scraper:
```csharp
public class ShoppingScraper : WebScraper
{
/// <summary>
/// Initialize the web scraper, setting the start URLs and allowed/banned domains or URL patterns.
/// </summary>
public override void Init()
{
// Apply your license key - get one from https://ironsoftware.com/csharp/webscraper/licensing/
License.LicenseKey = "LicenseKey";
this.LoggingLevel = WebScraper.LogLevel.All;
this.WorkingDirectory = AppSetting.GetAppRoot() + @"\ShoppingSiteSample\Output\";
// Configure request settings for better performance
this.Request("www.webSite.com", Parse);
}
/// <summary>
/// Parses the HTML document of the response to scrap the necessary data.
/// </summary>
/// <param name="response">The HTTP Response object to parse.</param>
public override void Parse(Response response)
{
var categoryList = new List<Category>();
// Iterate through each link in the menu and extract the category data.
foreach (var Links in response.Css("#menuFixed > ul > li > a"))
{
var cat = new Category
{
URL = Links.Attributes["href"],
Name = Links.InnerText,
LastScraped = DateTime.Now
};
categoryList.Add(cat);
}
// Save the scraped data into a JSONL file.
Scrape(categoryList, "Shopping.jsonl");
}
}
```
### What Elements Am I Targeting in the Menu?
Scraping links from the menu requires precise CSS selectors. The [API reference](https://ironsoftware.com/csharp/webscraper/object-reference/api/) provides detailed information about available selector methods:
<p><a rel="nofollow" href="/img/tutorials/webscraping-in-c-sharp/ShoppingSiteScrapeMenu.jpg" target="_blank"><img src="/img/tutorials/webscraping-in-c-sharp/ShoppingSiteScrapeMenu.jpg" alt="JSON file in Notepad showing e-commerce category structure with nested subcategories and URLs" class="img-responsive add-shadow img-margin" /></a></p>
## How Do I Scrape Both Main Categories and Subcategories?
Update the code to scrape main categories and all sub-links. This approach ensures complete navigation structure capture:
```csharp
public override void Parse(Response response)
{
// List of Category Links (Root)
var categoryList = new List<Category>();
// Traverse each 'li' under the fixed menu
foreach (var li in response.Css("#menuFixed > ul > li"))
{
// List of Main Links
foreach (var Links in li.Css("a"))
{
var cat = new Category
{
URL = Links.Attributes["href"],
Name = Links.InnerText,
SubCategories = new List<Category>(),
LastScraped = DateTime.Now
};
// List of Subcategories Links
foreach (var subCategory in li.Css("a[class=subcategory]"))
{
var subcat = new Category
{
URL = subCategory.Attributes["href"],
Name = subCategory.InnerText,
CategoryType = "Subcategory"
};
// Check if subcategory link already exists
if (cat.SubCategories.Find(c => c.Name == subcat.Name && c.URL == subcat.URL) == null)
{
// Add sublinks
cat.SubCategories.Add(subcat);
}
}
// Update product count based on subcategories
cat.ProductCount = cat.SubCategories.Count;
// Add Main Category to the list
categoryList.Add(cat);
}
}
// Save the scraped data into a JSONL file.
Scrape(categoryList, "Shopping.jsonl");
}
```
## How Do I Extract Product Information from Category Pages?
With links to all site categories available, start scraping products within each category. When dealing with product pages, [thread safety](https://ironsoftware.com/csharp/webscraper/troubleshooting/thread-safety/) becomes important for optimal performance. Navigate to any category and examine the content:
<p><a rel="nofollow" href="/img/tutorials/webscraping-in-c-sharp/ProductSubCategoryList.jpg" target="_blank"><img src="/img/tutorials/webscraping-in-c-sharp/ProductSubCategoryList.jpg" alt="E-commerce product listing page showing shoes and accessories with prices, ratings, and filtering controls" class="img-responsive add-shadow img-margin" /></a></p>
### What Does the Product HTML Structure Look Like?
Examine the HTML structure to understand product organization:
```html
<section class="products">
<div class="sku -gallery -validate-size " data-sku="AG249FA0T2PSGNAFAMZ" ft-product-sizes="41,42,43,44,45" ft-product-color="Multicolour">
<a class="link" href="http://www.WebSite.com/agu-bundle-of-2-sneakers-black-navy-blue-653884.html">
<div class="image-wrapper default-state">
<img class="lazy image -loaded" alt="Bundle Of 2 Sneakers - Black & Navy Blue" data-image-vertical="1" width="210" height="262" src="https://static.WebSite.com/p/agu-6208-488356-1-catalog_grid_3.jpg" data-sku="AG249FA0T2PSGNAFAMZ" data-src="https://static.WebSite.com/p/agu-6208-488356-1-catalog_grid_3.jpg" data-placeholder="placeholder_m_1.jpg">
<noscript><img src="https://static.WebSite.com/p/agu-6208-488356-1-catalog_grid_3.jpg" width="210" height="262" class="image" /></noscript>
</div>
<h2 class="title">
<span class="brand ">Agu </span>
<span class="name" dir="ltr">Bundle Of 2 Sneakers - Black & Navy Blue</span>
</h2>
<div class="price-container clearfix">
<span class="price-box">
<span class="price">
<span data-currency-iso="EGP">EGP</span>
<span dir="ltr" data-price="299">299</span>
</span>
<span class="price -old -no-special"></span>
</span>
</div>
<div class="rating-stars">
<div class="stars-container">
<div class="stars" style="width: 62%"></div>
</div>
<div class="total-ratings">(30)</div>
</div>
<span class="shop-first-logo-container">
<img src="http://www.WebSite.com/images/local/logos/shop_first/ShoppingSite/logo_normal.png" data-src="http://www.WebSite.com/images/local/logos/shop_first/ShoppingSite/logo_normal.png" class="lazy shop-first-logo-img -mbxs -loaded">
</span>
<span class="osh-icon -ShoppingSite-local shop_local--logo -block -mbs -mts"></span>
<div class="list -sizes" data-selected-sku="">
<span class="js-link sku-size" data-href="http://www.WebSite.com/agu-bundle-of-2-sneakers-black-navy-blue-653884.html?size=41">41</span>
<span class="js-link sku-size" data-href="http://www.WebSite.com/agu-bundle-of-2-sneakers-black-navy-blue-653884.html?size=42">42</span>
<span class="js-link sku-size" data-href="http://www.WebSite.com/agu-bundle-of-2-sneakers-black-navy-blue-653884.html?size=43">43</span>
<span class="js-link sku-size" data-href="http://www.WebSite.com/agu-bundle-of-2-sneakers-black-navy-blue-653884.html?size=44">44</span>
<span class="js-link sku-size" data-href="http://www.WebSite.com/agu-bundle-of-2-sneakers-black-navy-blue-653884.html?size=45">45</span>
</div>
</a>
</div>
<div class="sku -gallery -validate-size " data-sku="LE047FA01SRK4NAFAMZ" ft-product-sizes="110,115,120,125,130,135" ft-product-color="Black">
<a class="link" href="http://www.WebSite.com/leather-shop-genuine-leather-belt-black-712030.html">
<div class="image-wrapper default-state">
<img class="lazy image -loaded" alt="Genuine Leather Belt - Black" data-image-vertical="1" width="210" height="262" src="https://static.WebSite.com/p/leather-shop-1831-030217-1-catalog_grid_3.jpg" data-sku="LE047FA01SRK4NAFAMZ" data-src="https://static.WebSite.com/p/leather-shop-1831-030217-1-catalog_grid_3.jpg" data-placeholder="placeholder_m_1.jpg">
<noscript><img src="https://static.WebSite.com/p/leather-shop-1831-030217-1-catalog_grid_3.jpg" width="210" height="262" class="image" /></noscript>
</div>
<h2 class="title"><span class="brand ">Leather Shop </span> <span class="name" dir="ltr">Genuine Leather Belt - Black</span></h2>
<div class="price-container clearfix">
<span class="sale-flag-percent">-29%</span>
<span class="price-box">
<span class="price"><span data-currency-iso="EGP">EGP</span> <span dir="ltr" data-price="96">96</span> </span>
<span class="price -old"><span data-currency-iso="EGP">EGP</span> <span dir="ltr" data-price="135">135</span> </span>
</span>
</div>
<div class="rating-stars">
<div class="stars-container">
<div class="stars" style="width: 100%"></div>
</div>
<div class="total-ratings">(1)</div>
</div>
<span class="osh-icon -ShoppingSite-local shop_local--logo -block -mbs -mts"></span>
<div class="list -sizes" data-selected-sku="">
<span class="js-link sku-size" data-href="http://www.WebSite.com/leather-shop-genuine-leather-belt-black-712030.html?size=110">110</span>
<span class="js-link sku-size"data-href="http://www.WebSite.com/leather-shop-genuine-leather-belt-black-712030.html?size=115">115</span>
<span class="js-link sku-size"data-href="http://www.WebSite.com/leather-shop-genuine-leather-belt-black-712030.html?size=120">120</span>
<span class="js-link sku-size"data-href="http://www.WebSite.com/leather-shop-genuine-leather-belt-black-712030.html?size=125">125</span>
<span class="js-link sku-size"data-href="http://www.WebSite.com/leather-shop-genuine-leather-belt-black-712030.html?size=130">130</span>
<span class="js-link sku-size"data-href="http://www.WebSite.com/leather-shop-genuine-leather-belt-black-712030.html?size=135">135</span>
</div>
</a>
</div>
</section>
```
### Which Product Model Should I Create?
Build a product model for this content. When working with [shopping website scraping](https://ironsoftware.com/csharp/webscraper/how-to/scraping-from-a-shopping-website/), capture all relevant product details:
```csharp
public class Product
{
/// <summary>
/// Gets or sets the name.
/// </summary>
/// <value>
/// The name.
/// </value>
public string Name { get; set; }
/// <summary>
/// Gets or sets the price.
/// </summary>
/// <value>
/// The price.
/// </value>
public string Price { get; set; }
/// <summary>
/// Gets or sets the image.
/// </summary>
/// <value>
/// The image.
/// </value>
public string Image { get; set; }
// Additional properties for comprehensive data collection
public string Brand { get; set; }
public string OldPrice { get; set; }
public string Discount { get; set; }
public float Rating { get; set; }
public int ReviewCount { get; set; }
public List<string> AvailableSizes { get; set; }
public string ProductUrl { get; set; }
public string SKU { get; set; }
public DateTime ScrapedDate { get; set; }
}
```
### How Do I Add Product Scraping Functionality?
To scrape category pages, add a new scrape method with error handling and data validation:
```csharp
public void ParseCategory(Response response)
{
// List of Products
var productList = new List<Product>();
// Iterate through product links in the product section
foreach (var Links in response.Css("section.products > div > a"))
{
try
{
var product = new Product
{
Name = Links.Css("h2.title > span.name").First().InnerText,
Brand = Links.Css("h2.title > span.brand").FirstOrDefault()?.InnerText ?? "Unknown",
Price = Links.Css("div.price-container > span.price-box > span.price > span[data-price]").First().InnerText,
Image = Links.Css("div.image-wrapper.default-state > img").First().Attributes["src"],
ProductUrl = Links.Attributes["href"],
SKU = Links.ParentNode.Attributes["data-sku"],
ScrapedDate = DateTime.Now
};
// Extract old price if available
var oldPriceElement = Links.Css("span.price.-old > span[data-price]").FirstOrDefault();
if (oldPriceElement != null)
{
product.OldPrice = oldPriceElement.InnerText;
}
// Extract discount percentage
var discountElement = Links.Css("span.sale-flag-percent").FirstOrDefault();
if (discountElement != null)
{
product.Discount = discountElement.InnerText;
}
// Extract rating information
var ratingWidth = Links.Css("div.stars").FirstOrDefault()?.Attributes["style"];
if (!string.IsNullOrEmpty(ratingWidth))
{
var width = System.Text.RegularExpressions.Regex.Match(ratingWidth, @"(\d+)%").Groups[1].Value;
if (int.TryParse(width, out int ratingPercent))
{
product.Rating = ratingPercent / 20.0f; // Convert percentage to 5-star scale
}
}
// Extract review count
var reviewText = Links.Css("div.total-ratings").FirstOrDefault()?.InnerText;
if (!string.IsNullOrEmpty(reviewText))
{
var reviewCount = System.Text.RegularExpressions.Regex.Match(reviewText, @"\d+").Value;
if (int.TryParse(reviewCount, out int count))
{
product.ReviewCount = count;
}
}
// Extract available sizes
product.AvailableSizes = Links.Css("div.list.-sizes > span.sku-size")
.Select(s => s.InnerText)
.ToList();
productList.Add(product);
}
catch (Exception ex)
{
// Log error and continue with next product
Console.WriteLine($"Error parsing product: {ex.Message}");
}
}
// Save the scraped product data into a JSONL file.
Scrape(productList, "Products.jsonl");
// Handle pagination if needed
var nextPageLink = response.Css("a.pagination-next").FirstOrDefault();
if (nextPageLink != null)
{
var nextPageUrl = nextPageLink.Attributes["href"];
this.Request(nextPageUrl, ParseCategory);
}
}
```
This comprehensive approach to scraping shopping websites ensures capture of all relevant product information while handling errors gracefully. For more advanced scenarios, explore the [advanced web scraping features](https://ironsoftware.com/csharp/webscraper/tutorials/webscraping-in-c-sharp-advanced/) available in `IronWebScraper`.
Learn how to scrape product categories and items from shopping websites using C# with the WebScraper framework, extracting structured data from HTML elements into custom models. This comprehensive guide walks you through building a robust e-commerce scraper using the IronWebScraper library.
Quickstart: Scrape Shopping Website in C#
1Install IronWebScraper with NuGet Package Manager
PM > Install-Package IronWebScraper
Install-Package IronWebScraper
2Copy and run this code snippet.
using IronWebScraper;public class QuickShoppingScraper : WebScraper{ public override voidInit() { // Apply your license keyLicense.LicenseKey = "YOUR-LICENSE-KEY"; // Set the starting URL this.Request("https://shopping-site.com", Parse); } public override voidParse(Response response) { // Extract product data foreach (var product in response.Css(".product-item")) { var item = new {Name = product.Css(".product-name").First().InnerText,Price = product.Css(".price").First().InnerText,Image = product.Css("img").First().Attributes["src"] };Scrape(item, "products.jsonl"); } }}// Run the scrapervar scraper = new QuickShoppingScraper();scraper.Start();
using IronWebScraper;
public class QuickShoppingScraper : WebScraper
{
public override void Init()
{
// Apply your license key
License.LicenseKey = "YOUR-LICENSE-KEY";
// Set the starting URL
this.Request("https://shopping-site.com", Parse);
}
public override void Parse(Response response)
{
// Extract product data
foreach (var product in response.Css(".product-item"))
{
var item = new
{
Name = product.Css(".product-name").First().InnerText,
Price = product.Css(".price").First().InnerText,
Image = product.Css("img").First().Attributes["src"]
};
Scrape(item, "products.jsonl");
}
}
}
// Run the scraper
var scraper = new QuickShoppingScraper();
scraper.Start();
C#
3Deploy to test on your live environment
Start using IronWebScraper in your project today with a free trial
Create a new Console App project named "ShoppingSiteSample"
Add a class named "ShoppingScraper" that inherits from WebScraper
Create models for Category and Product data
Override Init() to set start URL and Parse() method for scraping
Run the scraper to extract categories and products to JSONL files
How Do I Analyze the Shopping Site's HTML Structure?
Select a shopping site to analyze its content structure. Understanding the HTML structure is crucial for successful web scraping. Before writing any code, spend time analyzing the target website's structure using browser developer tools.
As shown in the image, the left sidebar contains links for the site's product categories. The first step is investigating the site's HTML and planning the scraping approach. This analysis phase is essential for building an effective scraping strategy.
Why Does Understanding the HTML Structure Matter?
The fashion site categories have subcategories (Men, Women, Kids). Understanding this hierarchical structure helps design appropriate data models and scraping logic. When working with advanced web scraping features, proper HTML analysis becomes even more critical.
Set up a project following best practices for C# web scraping.
Create a new Console App or add a new folder for the sample named "ShoppingSiteSample"
Add a new class named "ShoppingScraper"
Start by scraping site categories and their subcategories
Install IronWebScraper via NuGet Package Manager or Package Manager Console:
PM > Install-Package IronWebScraper
Install-Package IronWebScraper
What Data Model Should I Use for Categories?
Create a Categories Model that properly represents the hierarchical structure discovered:
public class Category{ /// <summary> /// Gets or sets the name. /// </summary> /// <value> /// The name. /// </value> public stringName { get; set; } /// <summary> /// Gets or sets the URL. /// </summary> /// <value> /// The URL. /// </value> public stringURL { get; set; } /// <summary> /// Gets or sets the subcategories. /// </summary> /// <value> /// The subcategories. /// </value> public List<Category> SubCategories { get; set; } // Additional properties for enhanced data collection public intProductCount { get; set; } public DateTimeLastScraped { get; set; } public stringCategoryType { get; set; }}
public class Category
{
/// <summary>
/// Gets or sets the name.
/// </summary>
/// <value>
/// The name.
/// </value>
public string Name { get; set; }
/// <summary>
/// Gets or sets the URL.
/// </summary>
/// <value>
/// The URL.
/// </value>
public string URL { get; set; }
/// <summary>
/// Gets or sets the subcategories.
/// </summary>
/// <value>
/// The subcategories.
/// </value>
public List<Category> SubCategories { get; set; }
// Additional properties for enhanced data collection
public int ProductCount { get; set; }
public DateTime LastScraped { get; set; }
public string CategoryType { get; set; }
}
Public Class Category ''' <summary> ''' Gets or sets the name. ''' </summary> ''' <value> ''' The name. ''' </value> Public Property NameAsString ''' <summary> ''' Gets or sets the URL. ''' </summary> ''' <value> ''' The URL. ''' </value> Public Property URLAsString ''' <summary> ''' Gets or sets the subcategories. ''' </summary> ''' <value> ''' The subcategories. ''' </value> Public Property SubCategoriesAsList(OfCategory) ' Additional properties for enhanced data collection Public Property ProductCountAsInteger Public Property LastScrapedAsDateTime Public Property CategoryTypeAsStringEnd Class
Public Class Category
''' <summary>
''' Gets or sets the name.
''' </summary>
''' <value>
''' The name.
''' </value>
Public Property Name As String
''' <summary>
''' Gets or sets the URL.
''' </summary>
''' <value>
''' The URL.
''' </value>
Public Property URL As String
''' <summary>
''' Gets or sets the subcategories.
''' </summary>
''' <value>
''' The subcategories.
''' </value>
Public Property SubCategories As List(Of Category)
' Additional properties for enhanced data collection
Public Property ProductCount As Integer
Public Property LastScraped As DateTime
Public Property CategoryType As String
End Class
How Do I Build the Basic Scraper Logic?
Build the scraper logic, remembering to apply your license key before running the scraper:
public class ShoppingScraper : WebScraper{ /// <summary> /// Initialize the web scraper, setting the start URLs and allowed/banned domains or URL patterns. /// </summary> public override voidInit() { // Apply your license key - get one from https://ironsoftware.com/csharp/webscraper/licensing/License.LicenseKey = "LicenseKey"; this.LoggingLevel = WebScraper.LogLevel.All; this.WorkingDirectory = AppSetting.GetAppRoot() + @"\ShoppingSiteSample\Output\"; // Configure request settings for better performance this.Request("www.webSite.com", Parse); } /// <summary> /// Parses the HTML document of the response to scrap the necessary data. /// </summary> /// <param name="response">The HTTP Response object to parse.</param> public override voidParse(Response response) { var categoryList = new List<Category>(); // Iterate through each link in the menu and extract the category data. foreach (varLinksin response.Css("#menuFixed > ul > li > a")) { var cat = new Category {URL = Links.Attributes["href"],Name = Links.InnerText,LastScraped = DateTime.Now }; categoryList.Add(cat); } // Save the scraped data into a JSONL file.Scrape(categoryList, "Shopping.jsonl"); }}
public class ShoppingScraper : WebScraper
{
/// <summary>
/// Initialize the web scraper, setting the start URLs and allowed/banned domains or URL patterns.
/// </summary>
public override void Init()
{
// Apply your license key - get one from https://ironsoftware.com/csharp/webscraper/licensing/
License.LicenseKey = "LicenseKey";
this.LoggingLevel = WebScraper.LogLevel.All;
this.WorkingDirectory = AppSetting.GetAppRoot() + @"\ShoppingSiteSample\Output\";
// Configure request settings for better performance
this.Request("www.webSite.com", Parse);
}
/// <summary>
/// Parses the HTML document of the response to scrap the necessary data.
/// </summary>
/// <param name="response">The HTTP Response object to parse.</param>
public override void Parse(Response response)
{
var categoryList = new List<Category>();
// Iterate through each link in the menu and extract the category data.
foreach (var Links in response.Css("#menuFixed > ul > li > a"))
{
var cat = new Category
{
URL = Links.Attributes["href"],
Name = Links.InnerText,
LastScraped = DateTime.Now
};
categoryList.Add(cat);
}
// Save the scraped data into a JSONL file.
Scrape(categoryList, "Shopping.jsonl");
}
}
Public Class ShoppingScraperInheritsWebScraper ''' <summary> ''' Initialize the web scraper, setting the start URLs and allowed/banned domains or URL patterns. ''' </summary> PublicOverrides Sub Init() ' Apply your license key - get one from https://ironsoftware.com/csharp/webscraper/licensing/License.LicenseKey = "LicenseKey"Me.LoggingLevel = WebScraper.LogLevel.AllMe.WorkingDirectory = AppSetting.GetAppRoot() & "\ShoppingSiteSample\Output\" ' Configure request settings for better performanceMe.Request("www.webSite.com", AddressOfParse) End Sub ''' <summary> ''' Parses the HTML document of the response to scrap the necessary data. ''' </summary> ''' <param name="response">The HTTP Response object to parse.</param> PublicOverrides Sub Parse(responseAsResponse) Dim categoryList As New List(OfCategory)() ' Iterate through each link in the menu and extract the category data. For EachLinksInresponse.Css("#menuFixed > ul > li > a") Dim cat As New CategoryWith { .URL = Links.Attributes("href"), .Name = Links.InnerText, .LastScraped = DateTime.Now } categoryList.Add(cat) Next ' Save the scraped data into a JSONL file.Scrape(categoryList, "Shopping.jsonl") End SubEnd Class
Public Class ShoppingScraper
Inherits WebScraper
''' <summary>
''' Initialize the web scraper, setting the start URLs and allowed/banned domains or URL patterns.
''' </summary>
Public Overrides Sub Init()
' Apply your license key - get one from https://ironsoftware.com/csharp/webscraper/licensing/
License.LicenseKey = "LicenseKey"
Me.LoggingLevel = WebScraper.LogLevel.All
Me.WorkingDirectory = AppSetting.GetAppRoot() & "\ShoppingSiteSample\Output\"
' Configure request settings for better performance
Me.Request("www.webSite.com", AddressOf Parse)
End Sub
''' <summary>
''' Parses the HTML document of the response to scrap the necessary data.
''' </summary>
''' <param name="response">The HTTP Response object to parse.</param>
Public Overrides Sub Parse(response As Response)
Dim categoryList As New List(Of Category)()
' Iterate through each link in the menu and extract the category data.
For Each Links In response.Css("#menuFixed > ul > li > a")
Dim cat As New Category With {
.URL = Links.Attributes("href"),
.Name = Links.InnerText,
.LastScraped = DateTime.Now
}
categoryList.Add(cat)
Next
' Save the scraped data into a JSONL file.
Scrape(categoryList, "Shopping.jsonl")
End Sub
End Class
What Elements Am I Targeting in the Menu?
Scraping links from the menu requires precise CSS selectors. The API reference provides detailed information about available selector methods:
How Do I Scrape Both Main Categories and Subcategories?
Update the code to scrape main categories and all sub-links. This approach ensures complete navigation structure capture:
public override voidParse(Response response){ // List of Category Links (Root) var categoryList = new List<Category>(); // Traverse each 'li' under the fixed menu foreach (var li in response.Css("#menuFixed > ul > li")) { // List of Main Links foreach (varLinksin li.Css("a")) { var cat = new Category {URL = Links.Attributes["href"],Name = Links.InnerText,SubCategories = new List<Category>(),LastScraped = DateTime.Now }; // List of Subcategories Links foreach (var subCategory in li.Css("a[class=subcategory]")) { var subcat = new Category {URL = subCategory.Attributes["href"],Name = subCategory.InnerText,CategoryType = "Subcategory" }; // Check if subcategory link already exists if (cat.SubCategories.Find(c => c.Name == subcat.Name && c.URL == subcat.URL) == null) { // Add sublinks cat.SubCategories.Add(subcat); } } // Update product count based on subcategories cat.ProductCount = cat.SubCategories.Count; // Add Main Category to the list categoryList.Add(cat); } } // Save the scraped data into a JSONL file.Scrape(categoryList, "Shopping.jsonl");}
public override void Parse(Response response)
{
// List of Category Links (Root)
var categoryList = new List<Category>();
// Traverse each 'li' under the fixed menu
foreach (var li in response.Css("#menuFixed > ul > li"))
{
// List of Main Links
foreach (var Links in li.Css("a"))
{
var cat = new Category
{
URL = Links.Attributes["href"],
Name = Links.InnerText,
SubCategories = new List<Category>(),
LastScraped = DateTime.Now
};
// List of Subcategories Links
foreach (var subCategory in li.Css("a[class=subcategory]"))
{
var subcat = new Category
{
URL = subCategory.Attributes["href"],
Name = subCategory.InnerText,
CategoryType = "Subcategory"
};
// Check if subcategory link already exists
if (cat.SubCategories.Find(c => c.Name == subcat.Name && c.URL == subcat.URL) == null)
{
// Add sublinks
cat.SubCategories.Add(subcat);
}
}
// Update product count based on subcategories
cat.ProductCount = cat.SubCategories.Count;
// Add Main Category to the list
categoryList.Add(cat);
}
}
// Save the scraped data into a JSONL file.
Scrape(categoryList, "Shopping.jsonl");
}
PublicOverrides Sub Parse(responseAsResponse) ' List of Category Links (Root) Dim categoryList = New List(OfCategory)() ' Traverse each 'li' under the fixed menu For Each li Inresponse.Css("#menuFixed > ul > li") ' List of Main Links For EachLinksIn li.Css("a") Dim cat = New CategoryWith { .URL = Links.Attributes("href"), .Name = Links.InnerText, .SubCategories = New List(OfCategory)(), .LastScraped = DateTime.Now } ' List of Subcategories Links For Each subCategory In li.Css("a[class=subcategory]") Dim subcat = New CategoryWith { .URL = subCategory.Attributes("href"), .Name = subCategory.InnerText, .CategoryType = "Subcategory" } ' Check if subcategory link already exists If cat.SubCategories.Find(Function(c) c.Name = subcat.NameAndAlso c.URL = subcat.URL) Is Nothing Then ' Add sublinks cat.SubCategories.Add(subcat) End If Next ' Update product count based on subcategories cat.ProductCount = cat.SubCategories.Count ' Add Main Category to the list categoryList.Add(cat) Next Next ' Save the scraped data into a JSONL file.Scrape(categoryList, "Shopping.jsonl")End Sub
Public Overrides Sub Parse(response As Response)
' List of Category Links (Root)
Dim categoryList = New List(Of Category)()
' Traverse each 'li' under the fixed menu
For Each li In response.Css("#menuFixed > ul > li")
' List of Main Links
For Each Links In li.Css("a")
Dim cat = New Category With {
.URL = Links.Attributes("href"),
.Name = Links.InnerText,
.SubCategories = New List(Of Category)(),
.LastScraped = DateTime.Now
}
' List of Subcategories Links
For Each subCategory In li.Css("a[class=subcategory]")
Dim subcat = New Category With {
.URL = subCategory.Attributes("href"),
.Name = subCategory.InnerText,
.CategoryType = "Subcategory"
}
' Check if subcategory link already exists
If cat.SubCategories.Find(Function(c) c.Name = subcat.Name AndAlso c.URL = subcat.URL) Is Nothing Then
' Add sublinks
cat.SubCategories.Add(subcat)
End If
Next
' Update product count based on subcategories
cat.ProductCount = cat.SubCategories.Count
' Add Main Category to the list
categoryList.Add(cat)
Next
Next
' Save the scraped data into a JSONL file.
Scrape(categoryList, "Shopping.jsonl")
End Sub
How Do I Extract Product Information from Category Pages?
With links to all site categories available, start scraping products within each category. When dealing with product pages, thread safety becomes important for optimal performance. Navigate to any category and examine the content:
What Does the Product HTML Structure Look Like?
Examine the HTML structure to understand product organization:
Build a product model for this content. When working with shopping website scraping, capture all relevant product details:
public class Product{ /// <summary> /// Gets or sets the name. /// </summary> /// <value> /// The name. /// </value> public stringName { get; set; } /// <summary> /// Gets or sets the price. /// </summary> /// <value> /// The price. /// </value> public stringPrice { get; set; } /// <summary> /// Gets or sets the image. /// </summary> /// <value> /// The image. /// </value> public stringImage { get; set; } // Additional properties for comprehensive data collection public stringBrand { get; set; } public stringOldPrice { get; set; } public stringDiscount { get; set; } public floatRating { get; set; } public intReviewCount { get; set; } public List<string> AvailableSizes { get; set; } public stringProductUrl { get; set; } public stringSKU { get; set; } public DateTimeScrapedDate { get; set; }}
public class Product
{
/// <summary>
/// Gets or sets the name.
/// </summary>
/// <value>
/// The name.
/// </value>
public string Name { get; set; }
/// <summary>
/// Gets or sets the price.
/// </summary>
/// <value>
/// The price.
/// </value>
public string Price { get; set; }
/// <summary>
/// Gets or sets the image.
/// </summary>
/// <value>
/// The image.
/// </value>
public string Image { get; set; }
// Additional properties for comprehensive data collection
public string Brand { get; set; }
public string OldPrice { get; set; }
public string Discount { get; set; }
public float Rating { get; set; }
public int ReviewCount { get; set; }
public List<string> AvailableSizes { get; set; }
public string ProductUrl { get; set; }
public string SKU { get; set; }
public DateTime ScrapedDate { get; set; }
}
Public Class Product ''' <summary> ''' Gets or sets the name. ''' </summary> ''' <value> ''' The name. ''' </value> Public Property NameAsString ''' <summary> ''' Gets or sets the price. ''' </summary> ''' <value> ''' The price. ''' </value> Public Property PriceAsString ''' <summary> ''' Gets or sets the image. ''' </summary> ''' <value> ''' The image. ''' </value> Public Property ImageAsString ' Additional properties for comprehensive data collection Public Property BrandAsString Public Property OldPriceAsString Public Property DiscountAsString Public Property RatingAsSingle Public Property ReviewCountAsInteger Public Property AvailableSizesAsList(OfString) Public Property ProductUrlAsString Public Property SKUAsString Public Property ScrapedDateAsDateTimeEnd Class
Public Class Product
''' <summary>
''' Gets or sets the name.
''' </summary>
''' <value>
''' The name.
''' </value>
Public Property Name As String
''' <summary>
''' Gets or sets the price.
''' </summary>
''' <value>
''' The price.
''' </value>
Public Property Price As String
''' <summary>
''' Gets or sets the image.
''' </summary>
''' <value>
''' The image.
''' </value>
Public Property Image As String
' Additional properties for comprehensive data collection
Public Property Brand As String
Public Property OldPrice As String
Public Property Discount As String
Public Property Rating As Single
Public Property ReviewCount As Integer
Public Property AvailableSizes As List(Of String)
Public Property ProductUrl As String
Public Property SKU As String
Public Property ScrapedDate As DateTime
End Class
How Do I Add Product Scraping Functionality?
To scrape category pages, add a new scrape method with error handling and data validation:
public voidParseCategory(Response response){ // List of Products var productList = new List<Product>(); // Iterate through product links in the product section foreach (varLinksin response.Css("section.products > div > a")) { try { var product = new Product {Name = Links.Css("h2.title > span.name").First().InnerText,Brand = Links.Css("h2.title > span.brand").FirstOrDefault()?.InnerText ?? "Unknown",Price = Links.Css("div.price-container > span.price-box > span.price > span[data-price]").First().InnerText,Image = Links.Css("div.image-wrapper.default-state > img").First().Attributes["src"],ProductUrl = Links.Attributes["href"],SKU = Links.ParentNode.Attributes["data-sku"],ScrapedDate = DateTime.Now }; // Extract old price if available var oldPriceElement = Links.Css("span.price.-old > span[data-price]").FirstOrDefault(); if (oldPriceElement != null) { product.OldPrice = oldPriceElement.InnerText; } // Extract discount percentage var discountElement = Links.Css("span.sale-flag-percent").FirstOrDefault(); if (discountElement != null) { product.Discount = discountElement.InnerText; } // Extract rating information var ratingWidth = Links.Css("div.stars").FirstOrDefault()?.Attributes["style"]; if (!string.IsNullOrEmpty(ratingWidth)) { var width = System.Text.RegularExpressions.Regex.Match(ratingWidth, @"(\d+)%").Groups[1].Value; if (int.TryParse(width, out int ratingPercent)) { product.Rating = ratingPercent / 20.0f; // Convert percentage to 5-star scale } } // Extract review count var reviewText = Links.Css("div.total-ratings").FirstOrDefault()?.InnerText; if (!string.IsNullOrEmpty(reviewText)) { var reviewCount = System.Text.RegularExpressions.Regex.Match(reviewText, @"\d+").Value; if (int.TryParse(reviewCount, out int count)) { product.ReviewCount = count; } } // Extract available sizes product.AvailableSizes = Links.Css("div.list.-sizes > span.sku-size") .Select(s => s.InnerText) .ToList(); productList.Add(product); } catch (Exception ex) { // Log error and continue with next productConsole.WriteLine($"Error parsing product: {ex.Message}"); } } // Save the scraped product data into a JSONL file.Scrape(productList, "Products.jsonl"); // Handle pagination if needed var nextPageLink = response.Css("a.pagination-next").FirstOrDefault(); if (nextPageLink != null) { var nextPageUrl = nextPageLink.Attributes["href"]; this.Request(nextPageUrl, ParseCategory); }}
public void ParseCategory(Response response)
{
// List of Products
var productList = new List<Product>();
// Iterate through product links in the product section
foreach (var Links in response.Css("section.products > div > a"))
{
try
{
var product = new Product
{
Name = Links.Css("h2.title > span.name").First().InnerText,
Brand = Links.Css("h2.title > span.brand").FirstOrDefault()?.InnerText ?? "Unknown",
Price = Links.Css("div.price-container > span.price-box > span.price > span[data-price]").First().InnerText,
Image = Links.Css("div.image-wrapper.default-state > img").First().Attributes["src"],
ProductUrl = Links.Attributes["href"],
SKU = Links.ParentNode.Attributes["data-sku"],
ScrapedDate = DateTime.Now
};
// Extract old price if available
var oldPriceElement = Links.Css("span.price.-old > span[data-price]").FirstOrDefault();
if (oldPriceElement != null)
{
product.OldPrice = oldPriceElement.InnerText;
}
// Extract discount percentage
var discountElement = Links.Css("span.sale-flag-percent").FirstOrDefault();
if (discountElement != null)
{
product.Discount = discountElement.InnerText;
}
// Extract rating information
var ratingWidth = Links.Css("div.stars").FirstOrDefault()?.Attributes["style"];
if (!string.IsNullOrEmpty(ratingWidth))
{
var width = System.Text.RegularExpressions.Regex.Match(ratingWidth, @"(\d+)%").Groups[1].Value;
if (int.TryParse(width, out int ratingPercent))
{
product.Rating = ratingPercent / 20.0f; // Convert percentage to 5-star scale
}
}
// Extract review count
var reviewText = Links.Css("div.total-ratings").FirstOrDefault()?.InnerText;
if (!string.IsNullOrEmpty(reviewText))
{
var reviewCount = System.Text.RegularExpressions.Regex.Match(reviewText, @"\d+").Value;
if (int.TryParse(reviewCount, out int count))
{
product.ReviewCount = count;
}
}
// Extract available sizes
product.AvailableSizes = Links.Css("div.list.-sizes > span.sku-size")
.Select(s => s.InnerText)
.ToList();
productList.Add(product);
}
catch (Exception ex)
{
// Log error and continue with next product
Console.WriteLine($"Error parsing product: {ex.Message}");
}
}
// Save the scraped product data into a JSONL file.
Scrape(productList, "Products.jsonl");
// Handle pagination if needed
var nextPageLink = response.Css("a.pagination-next").FirstOrDefault();
if (nextPageLink != null)
{
var nextPageUrl = nextPageLink.Attributes["href"];
this.Request(nextPageUrl, ParseCategory);
}
}
Public Sub ParseCategory(responseAsResponse) ' List of Products Dim productList = New List(OfProduct)() ' Iterate through product links in the product section For EachLinksInresponse.Css("section.products > div > a")Try Dim product = New ProductWith { .Name = Links.Css("h2.title > span.name").First().InnerText, .Brand = If(Links.Css("h2.title > span.brand").FirstOrDefault()?.InnerText, "Unknown"), .Price = Links.Css("div.price-container > span.price-box > span.price > span[data-price]").First().InnerText, .Image = Links.Css("div.image-wrapper.default-state > img").First().Attributes("src"), .ProductUrl = Links.Attributes("href"), .SKU = Links.ParentNode.Attributes("data-sku"), .ScrapedDate = DateTime.Now } ' Extract old price if available Dim oldPriceElement = Links.Css("span.price.-old > span[data-price]").FirstOrDefault() If oldPriceElement IsNot Nothing Then product.OldPrice = oldPriceElement.InnerText End If ' Extract discount percentage Dim discountElement = Links.Css("span.sale-flag-percent").FirstOrDefault() If discountElement IsNot Nothing Then product.Discount = discountElement.InnerText End If ' Extract rating information Dim ratingWidth = Links.Css("div.stars").FirstOrDefault()?.Attributes("style") IfNotString.IsNullOrEmpty(ratingWidth) Then Dim width = System.Text.RegularExpressions.Regex.Match(ratingWidth, "(\d+)%").Groups(1).Value Dim ratingPercent AsInteger IfInteger.TryParse(width, ratingPercent) Then product.Rating = ratingPercent / 20.0F ' Convert percentage to 5-star scale End If End If ' Extract review count Dim reviewText = Links.Css("div.total-ratings").FirstOrDefault()?.InnerText IfNotString.IsNullOrEmpty(reviewText) Then Dim reviewCount = System.Text.RegularExpressions.Regex.Match(reviewText, "\d+").Value Dim count AsInteger IfInteger.TryParse(reviewCount, count) Then product.ReviewCount = count End If End If ' Extract available sizes product.AvailableSizes = Links.Css("div.list.-sizes > span.sku-size").Select(Function(s) s.InnerText).ToList() productList.Add(product)Catch ex AsException ' Log error and continue with next productConsole.WriteLine($"Error parsing product: {ex.Message}")EndTry Next ' Save the scraped product data into a JSONL file.Scrape(productList, "Products.jsonl") ' Handle pagination if needed Dim nextPageLink = response.Css("a.pagination-next").FirstOrDefault() If nextPageLink IsNot Nothing Then Dim nextPageUrl = nextPageLink.Attributes("href")Me.Request(nextPageUrl, AddressOfParseCategory) End IfEnd Sub
Public Sub ParseCategory(response As Response)
' List of Products
Dim productList = New List(Of Product)()
' Iterate through product links in the product section
For Each Links In response.Css("section.products > div > a")
Try
Dim product = New Product With {
.Name = Links.Css("h2.title > span.name").First().InnerText,
.Brand = If(Links.Css("h2.title > span.brand").FirstOrDefault()?.InnerText, "Unknown"),
.Price = Links.Css("div.price-container > span.price-box > span.price > span[data-price]").First().InnerText,
.Image = Links.Css("div.image-wrapper.default-state > img").First().Attributes("src"),
.ProductUrl = Links.Attributes("href"),
.SKU = Links.ParentNode.Attributes("data-sku"),
.ScrapedDate = DateTime.Now
}
' Extract old price if available
Dim oldPriceElement = Links.Css("span.price.-old > span[data-price]").FirstOrDefault()
If oldPriceElement IsNot Nothing Then
product.OldPrice = oldPriceElement.InnerText
End If
' Extract discount percentage
Dim discountElement = Links.Css("span.sale-flag-percent").FirstOrDefault()
If discountElement IsNot Nothing Then
product.Discount = discountElement.InnerText
End If
' Extract rating information
Dim ratingWidth = Links.Css("div.stars").FirstOrDefault()?.Attributes("style")
If Not String.IsNullOrEmpty(ratingWidth) Then
Dim width = System.Text.RegularExpressions.Regex.Match(ratingWidth, "(\d+)%").Groups(1).Value
Dim ratingPercent As Integer
If Integer.TryParse(width, ratingPercent) Then
product.Rating = ratingPercent / 20.0F ' Convert percentage to 5-star scale
End If
End If
' Extract review count
Dim reviewText = Links.Css("div.total-ratings").FirstOrDefault()?.InnerText
If Not String.IsNullOrEmpty(reviewText) Then
Dim reviewCount = System.Text.RegularExpressions.Regex.Match(reviewText, "\d+").Value
Dim count As Integer
If Integer.TryParse(reviewCount, count) Then
product.ReviewCount = count
End If
End If
' Extract available sizes
product.AvailableSizes = Links.Css("div.list.-sizes > span.sku-size").Select(Function(s) s.InnerText).ToList()
productList.Add(product)
Catch ex As Exception
' Log error and continue with next product
Console.WriteLine($"Error parsing product: {ex.Message}")
End Try
Next
' Save the scraped product data into a JSONL file.
Scrape(productList, "Products.jsonl")
' Handle pagination if needed
Dim nextPageLink = response.Css("a.pagination-next").FirstOrDefault()
If nextPageLink IsNot Nothing Then
Dim nextPageUrl = nextPageLink.Attributes("href")
Me.Request(nextPageUrl, AddressOf ParseCategory)
End If
End Sub
This comprehensive approach to scraping shopping websites ensures capture of all relevant product information while handling errors gracefully. For more advanced scenarios, explore the advanced web scraping features available in IronWebScraper.
Frequently Asked Questions
How can I start web scraping on a shopping website using C#?
To start web scraping on a shopping website using C#, you need to create a new Console App project, add a class that inherits from `WebScraper`, and configure the scraper to set a start URL and parse methods using the `IronWebScraper` library.
What are the main steps to set up a C# web scraping project?
The main steps to set up a C# web scraping project include creating a Console App, installing the `IronWebScraper` via NuGet, setting up data models for categories and products, and overriding the `Init()` and `Parse()` methods within your scraper class.
Why is understanding the HTML structure important for web scraping?
Understanding the HTML structure is crucial for web scraping because it allows you to design appropriate data models and logic to effectively extract the necessary elements such as product names, prices, and images from the web pages.
How do I extract product data from a shopping site using `IronWebScraper`?
You can extract product data using `IronWebScraper` by iterating through HTML elements with CSS selectors in the `Parse()` method, organizing them into custom data models, and saving the scraped data into a JSONL file.
What data models should I use for scraping e-commerce sites?
For scraping e-commerce sites, you should use data models like `Category` and `Product` to represent hierarchical structures and individual product details, capturing fields such as name, price, URL, image, and more.
Can `IronWebScraper` handle pagination while scraping?
Yes, `IronWebScraper` can handle pagination by requesting the next page URL found in the pagination links and continuing to scrape across pages until all necessary data is collected.
How does `IronWebScraper` improve the efficiency of a web scraping project?
`IronWebScraper` enhances web scraping efficiency through features like concurrent requests, CSS selectors for targeted scraping, and harvesting data into structured formats, making the scraping process faster and more organized.
How do I ensure thread safety when scraping product pages?
To ensure thread safety when scraping product pages, you should manage concurrent requests and resource sharing carefully, using the advanced features of `IronWebScraper` designed to handle multi-threaded operations efficiently.
What are common elements targeted when scraping e-commerce categories?
Common elements targeted when scraping e-commerce categories include category names, URLs, subcategory links, and sometimes metadata like product counts or last updated timestamps.
How do I scrape subcategories along with main categories?
You can scrape subcategories along with main categories by iterating through each main category's HTML structure, identifying subcategory links, and extracting their respective names and URLs using `IronWebScraper`.
Curtis Chau holds a Bachelor’s degree in Computer Science (Carleton University) and specializes in front-end development with expertise in Node.js, TypeScript, JavaScript, and React. Passionate about crafting intuitive and aesthetically pleasing user interfaces, Curtis enjoys working with modern frameworks and creating well-structured, visually appealing manuals.