Class UrlMatchPatternCollection
A class used for control URL, Domain and File type access using positive and negative wild-card(*?) and Regex rules.
Inheritance
Namespace: IronWebScraper.Urls
Assembly: IronWebScraper.dll
Syntax
public class UrlMatchPatternCollection : List<Regex>
The wildcard and regex rules that decide which URLs a crawler may follow live in a UrlMatchPatternCollection. A WebScraper exposes four of them, AllowedDomains, AllowedUrls, BannedDomains, and BannedUrls, and a request is followed only when it passes the allow rules and matches none of the ban rules.
The collection extends List<Regex>, so rules can be added as full regular expressions, but the convenient form is the wildcard string. Add("*.example.com/*") accepts a single pattern using * and ? wildcards, and the params overload, Add("*/cart*", "*/login*"), adds several at once. Patterns added to a Banned collection exclude matching URLs even when an Allowed rule would otherwise admit them. Keeping the allow and ban rules explicit is what stops a crawler from wandering off the target site, the most common cause of a runaway scrape, and rule order does not matter because a ban always takes precedence over an allow.
AllowedDomains.Add("*.example.com");
BannedUrls.Add("*/cart*", "*/login*");The advanced web scraping tutorial scopes a crawl with these collections, and the shopping-site how-to limits which pages are visited.
Constructors
UrlMatchPatternCollection()
Declaration
public UrlMatchPatternCollection()
Methods
Add(String)
Adds a match pattern using strings with wild-card.
? matches exactly 1 character
* matches exactly zero to any number of characters
Declaration
public void Add(string wildCardPattern)
Parameters
| Type | Name | Description |
|---|---|---|
| System.String | wildCardPattern | The wild-card pattern. |
Add(String[])
Adds multiple match patterns using strings with wild-card.
? matches exactly 1 character
* matches exactly zero to any number of characters
Declaration
public void Add(params string[] wildCardPatterns)
Parameters
| Type | Name | Description |
|---|---|---|
| System.String[] | wildCardPatterns | The wild-card patterns. |