Class ScrapedData
A flexible dictionary of object values used to conveniently store scraped data of any Type in a key-value dictionary which can be saved as JSON using the Yield method. ScrapedData can hold data objects of any Type, including Classes.
E.g:var Data = new ScrapedData();
Data['title'] = "Page Title";
Data['date'] = DateTime.Now;Inheritance
Namespace: IronWebScraper
Assembly: IronWebScraper.dll
Syntax
public class ScrapedData : Dictionary<string, object>
ScrapedData is the key-value record a scraper fills with extracted fields and serializes as JSON. It extends Dictionary<string, object>, so each field is stored under a string key and the whole record converts to and from JSON for saving a crawl's results.
Set fields with the dictionary indexer, read a typed value with Get<T>(key), write the record to a JSON string with ToJson(), and rebuild one from JSON with the static ScrapedData.FromJson(json). In a WebScraper, passing an object to Scrape persists it, and UnScrape<T>() reads stored results back, so ScrapedData is the natural shape for both ends of that round trip. Because values are stored as object, a single record can mix strings, numbers, and nested data, and the JSON round trip through ToJson and FromJson makes a crawl's output easy to persist to disk and reload in a later run.
var data = new ScrapedData();
data["title"] = node.TextContentClean;
data["price"] = node.GetAttribute("data-price");
Scrape(data);The web scraping tutorial saves results as ScrapedData, and the C# web scraper example shows the JSON output.
Constructors
ScrapedData()
Declaration
public ScrapedData()
Methods
FromJson(String)
Deserializes a ScrapedData object from JSON created using the toJson() method.
Declaration
public static ScrapedData FromJson(string json)
Parameters
| Type | Name | Description |
|---|---|---|
| System.String | json |
Returns
| Type | Description |
|---|---|
| ScrapedData |
Get<T>(String)
Returns the specified ScrapedData item cast as the appropriate type.
E.g:Get<string>("url");Declaration
public T Get<T>(string key)
Parameters
| Type | Name | Description |
|---|---|---|
| System.String | key | Key |
Returns
| Type | Description |
|---|---|
| T |
Type Parameters
| Name | Description |
|---|---|
| T | Type |
Exceptions
| Type | Condition |
|---|---|
| System.Collections.Generic.KeyNotFoundException |
ToJson()
Converts the ScrapedData to a JSON string.
Declaration
public string ToJson()
Returns
| Type | Description |
|---|---|
| System.String |