Click or drag to resize

WebScraperObeyRobotsDotTxtForHost Method

Causes the WebScraper to always obey /robots.txt directives including path restrictions and crawl rates on a domain by domain basis. May be overridden for advanced control.

Namespace:  IronWebScraper
Assembly:  IronWebScraper (in IronWebScraper.dll) Version: 4.0.4.25470 (4.0.4.3)
Syntax
public virtual bool ObeyRobotsDotTxtForHost(
	string Host
)

Parameters

Host
Type: SystemString

Return Value

Type: Boolean
See Also