Click or drag to resize

WebScraper Methods

The WebScraper type exposes the following members.

Methods
  NameDescription
Public methodAcceptUrl
Decides if the WebScraper will accept a given url. My be overridden to apply custom middleware logic.
Public methodChooseIdentityForRequest
Picks a random identity from WebScraper.Identities for each request. Add Identities with proxy IP addresses, userAgents, headers, cookies, username and password in your Init Method and add them to the WebScraper.Identities List;

Override this method to create your own logic for non-random selection of a HttpIdentity for each request.

Public methodDownloadFile(String, String, Boolean, HttpIdentity)
Requests a file to be downloaded from the given Url to the local file-system. Often used for scraping documents, assets and images.

Normally called with an Parse Method of IronWebScraper.WebScraper

Public methodDownloadFile(Uri, String, Boolean, HttpIdentity)
Requests a file to be downloaded from the given Url to the local file-system. Often used for scraping documents, assets and images.

Normally called with an Parse Method of IronWebScraper.WebScraper

Public methodDownloadFileUnique
Much like DownloadFile except if the file has already been downloaded or exists locally, it will not be re-downloaded.

Requests a file to be downloaded from the given Url to the local file-system. Often used for scraping documents, assets and images.

Normally called with an Parse Method of IronWebScraper.WebScraper

Public methodDownloadImage(String, String, Int32, Int32, Boolean, HttpIdentity)
Requests a file to be downloaded from the given Url to the local file-system. Often used for scraping documents, assets and images.

Normally called with an Parse Method of IronWebScraper.WebScraper

Public methodDownloadImage(Uri, String, Int32, Int32, Boolean, HttpIdentity)
Requests a file to be downloaded from the given Url to the local file-system. Often used for scraping documents, assets and images.

Normally called with an Parse Method of IronWebScraper.WebScraper

Public methodEnableWebCache
Caches web http responses for reuse. This allows WebScraper classes to be modified and restarted without re-downloading previously scraped urls.
Public methodEnableWebCache(TimeSpan)
Caches web http responses for reuse. This allows WebScraper classes to be modified and restarted without re-downloading previously scrape urls.
Public methodEquals (Inherited from Object.)
Public methodStatic memberFetchUrlContents
A handy shortcut method that fetches the text content from any Url (synchronously).
Public methodFetchUrlContentsBinary
A handy shortcut method that fetches the text content from any Url (synchronously) as a binary data in a byye array (byte[])
Public methodGetHashCode (Inherited from Object.)
Public methodGetType (Inherited from Object.)
Public methodInit
Override this method initialize your web-scraper. Important tasks will be to Request at least one start url... and set allowed/banned domain or url patterns.
Public methodLog
Logs the specified message to the console. Logs can be Enabled using the EnableLogging. This function has been exposed and is over-ridable to allow for easy Email and Slack notification integration.
Public methodObeyRobotsDotTxtForHost
Causes the WebScraper to always obey /robots.txt directives including path restrictions and crawl rates on a domain by domain basis. May be overridden for advanced control.
Public methodParse
Override this method to create the default Response handler for your web scraper. If you have multiple page types, you can add additional similar methods.
Public methodParseWebscraperDownload
Internal method to parse binary files downloaded by a webScraper.
Public methodParseWebscraperDownloadImage
Internal method to parse images downloaded by a webScraper.
Public methodPostRequest(String, ActionResponse, DictionaryString, String)
Request adds a new request to the scrape-job queue using the POST http method.
Public methodPostRequest(Uri, ActionResponse, DictionaryString, String)
Request adds a new request to the scrape-job queue using the POST http method.
Public methodPostRequest(String, ActionResponse, DictionaryString, String, MetaData)
Request adds a new request to the scrape-job queue using the POST http method.
Public methodPostRequest(Uri, ActionResponse, DictionaryString, String, MetaData)
Request adds a new request to the scrape-job queue using the POST http method.
Public methodPostRequest(String, ActionResponse, DictionaryString, String, HttpIdentity, MetaData)
Request adds a new request to the scrape-job queue using the POST http method.
Public methodPostRequest(Uri, ActionResponse, DictionaryString, String, HttpIdentity, MetaData)
Request adds a new request to the scrape-job queue using the POST http method.
Public methodRequest(IEnumerableString, ActionResponse)
A key method called from with the Init and Parse Methods. Request adds new requests to the scrape-job queue, and decides which method (e.g. Parse) will be used to parse the Response object.
Public methodRequest(String, ActionResponse)
A key method called from with the Init and Parse Methods. Request adds a new request to the scrape-job queue, and decides which method (e.g. Parse) will be used to parse the Response object.
Public methodRequest(Uri, ActionResponse)
A key method called from with the Init and Parse Methods. Request adds a new request to the scrape-job queue, and decides which method (e.g. Parse) will be used to parse the Response object.
Public methodRequest(String, ActionResponse, MetaData)
A key method called from with the Init and Parse Methods. Request adds a new request to the scrape-job queue, and decides which method (e.g. Parse) will be used to parse the Response object.
Public methodRequest(Uri, ActionResponse, MetaData)
A key method called from with the Init and Parse Methods. Request adds a new request to the scrape-job queue, and decides which method (e.g. Parse) will be used to parse the Response object.
Public methodRequest(String, ActionResponse, HttpIdentity, MetaData)
A key method called from with the Init and Parse Methods. Request adds a new request to the scrape-job queue, and decides which method (e.g. Parse) will be used to parse the Response object.
Public methodRequest(Uri, ActionResponse, HttpIdentity, MetaData)
A key method called from with the Init and Parse Methods. Request adds a new request to the scrape-job queue, and decides which method (e.g. Parse) will be used to parse the Response object.
Public methodRetry
Retries a Response.

Usually called in a Parse method, this method is useful if a Captcha or error screen was encountered during Html parsing.

Public methodScrape
Appends any scraped data to a file in the JsonLines format. (1 json object per line). Will save any .Net object of any kind. This method is typically used with IronWebScraper.ScrapedData or developer defined classes for scraped data items. The default filename will follow the pattern "NameSpace.TypeName.jsonl". E.g: IronWebScraper.ScrapedData.jsonl
Public methodScrapeUnique
Appends scraped data to a file in the JsonLines format. (1 json object per line). Automatically ignores duplicates. Will save any .Net object of any kind. This method is typically used with IronWebScraper.ScrapedData or developer defined classes for scraped data items. The default filename will follow the pattern "WorkingDirecory/NameSpace.TypeName.jsonl". E.g: Scrape/IronWebScraper.ScrapedData.jsonl
Public methodSetSiteSpecificCrawlRateLimit
Set a throttle limit for a specific domain
Public methodStart
Starts the WebScraper.

Set CrawlId to make this crawl resumable. Will also resume a previous scrawl with the same CrawlId if it exists.

Giving a CrawlId also causes the WebScraper to auto-save its state every 5 minutes in case of a crash, system failure or power outage. This feature is particularly useful for long running web-scraping tasks, allowing hours, days or even weeks of work to be recovered effortlessly.

Public methodStartAsync
Starts the WebScraper Asynchronously. Set CrawlId to make this crawl resumable. Will resume a previous scrawl with the same CrawlId if it exists.
Public methodStop
Stops this WebScraper instance graceful. The WebScraper may be restated later with no loss of data by calling Start(CrawlId) or StartAsync(CrawlId)
Public methodToString (Inherited from Object.)
Public methodUnScrape(Boolean)
Retrieves IronWebScraper.ScrapedData objects which were saved using the WebScraper.Scrape method.
Public methodUnScrape(String, Boolean)
Retrieves IronWebScraper.ScrapedData objects which were saved using the WebScraper.Scrape method.
Public methodUnScrapeT(Boolean)
Retrieves native C# objects which were saved using the WebScraper.Scrape method in the JsonLines format.
Public methodUnScrapeT(String, Boolean)
Retrieves native C# objects which were saved using the WebScraper.Scrape method in the JsonLines format.
Top
See Also