Search Results for

    Show / Hide Table of Contents

    Class UrlMatchPatternCollection

    A class used for control URL, Domain and File type access using positive and negative wild-card(*?) and Regex rules.

    Inheritance
    System.Object
    UrlMatchPatternCollection
    Namespace: IronWebScraper.Urls
    Assembly: IronWebScraper.dll
    Syntax
    public class UrlMatchPatternCollection : List<Regex>

    The wildcard and regex rules that decide which URLs a crawler may follow live in a UrlMatchPatternCollection. A WebScraper exposes four of them, AllowedDomains, AllowedUrls, BannedDomains, and BannedUrls, and a request is followed only when it passes the allow rules and matches none of the ban rules.

    The collection extends List<Regex>, so rules can be added as full regular expressions, but the convenient form is the wildcard string. Add("*.example.com/*") accepts a single pattern using * and ? wildcards, and the params overload, Add("*/cart*", "*/login*"), adds several at once. Patterns added to a Banned collection exclude matching URLs even when an Allowed rule would otherwise admit them. Keeping the allow and ban rules explicit is what stops a crawler from wandering off the target site, the most common cause of a runaway scrape, and rule order does not matter because a ban always takes precedence over an allow.

    AllowedDomains.Add("*.example.com");
    BannedUrls.Add("*/cart*", "*/login*");

    The advanced web scraping tutorial scopes a crawl with these collections, and the shopping-site how-to limits which pages are visited.

    Constructors

    UrlMatchPatternCollection()

    Declaration
    public UrlMatchPatternCollection()

    Methods

    Add(String)

    Adds a match pattern using strings with wild-card.

    ? matches exactly 1 character

    * matches exactly zero to any number of characters

    Declaration
    public void Add(string wildCardPattern)
    Parameters
    Type Name Description
    System.String wildCardPattern

    The wild-card pattern.

    Add(String[])

    Adds multiple match patterns using strings with wild-card.

    ? matches exactly 1 character

    * matches exactly zero to any number of characters

    Declaration
    public void Add(params string[] wildCardPatterns)
    Parameters
    Type Name Description
    System.String[] wildCardPatterns

    The wild-card patterns.

    ☀
    ☾
    Downloads
    • Download with Nuget
    • Start for Free
    In This Article
    Back to top
    Install with Nuget