Click or drag to resize

WebScraper Properties

The WebScraper type exposes the following members.

Public propertyFailedUrls
Gets the number of failed http requests which have exceeded their total maximum number of retries.
Public propertyHttpRetryAttempts
The number of times WebScraper will retry a failed URL (normally with a new identity) before considering it non-scrapable.
Public propertyHttpTimeOut
Gets or the time after-which a HTTP request will be considered failed or lost. (non-contactable or Dns unavailable)
Public propertyMaxHttpConnectionLimit
Gets or sets the total number of allowed open HTTP requests (threads)
Public propertyOpenConnectionLimitPerHost
Gets or sets the allowed number of concurrent HTTP requests (threads) per hostname or IP address. This helps protect hosts against too many requests.
Public propertyRateLimitPerHost
Gets or sets minimum polite delay (pause) between request to a given domain or IP address.
Public propertySuccessfulFileDownloadCount
Gets the number of successful http downloads using the DownloadFile and DownloadImage methods..
Public propertySuccessfulfulRequestCount
Gets the number of successful http requests.
Public propertyThrottleMode
Makes the WebSraper intelligently throttle requests not only by hostname, but also by host servers' IP addresses. This is polite in-case multiple scraped domains are hosted on the same machine.
See Also