The WebScraper type exposes the following members.
Gets the number of failed http requests which have exceeded their total maximum number of retries.
The number of times WebScraper will retry a failed URL (normally with a new identity) before considering it non-scrapable.
Gets or the time after-which a HTTP request will be considered failed or lost. (non-contactable or Dns unavailable)
Gets or sets the total number of allowed open HTTP requests (threads)
Gets or sets the allowed number of concurrent HTTP requests (threads) per hostname or IP address. This helps protect hosts against too many requests.
Gets or sets minimum polite delay (pause) between request to a given domain or IP address.
Gets the number of successful http downloads using the DownloadFile and DownloadImage methods..
Gets the number of successful http requests.
Makes the WebSraper intelligently throttle requests not only by hostname, but also by host servers' IP addresses. This is polite in-case multiple scraped domains are hosted on the same machine.