Click or drag to resize

WebScraperObeyRobotsDotTxt Field

Causes the WebScraper to always obey /robots.txt directives including url and path restrictions and crawl rates.

Namespace:  IronWebScraper
Assembly:  IronWebScraper (in IronWebScraper.dll) Version: 4.0.4.25470 (4.0.4.3)
Syntax
public bool ObeyRobotsDotTxt

Field Value

Type: Boolean
See Also