feat: globs & regexps for SitemapRequestList

[barjin]

Adds globs, regexps and exclude options to SitemapRequestList to make it list only URLs that match those patterns.

While this looked like a single-responsibility violation to me at first, these changes are imo quite justifiable, as the possible use cases (wanting to crawl only a part of a website while utilizing the official sitemap) seem to be very sane.

[janbuchar]

Bruh, extract those massive boolean expressions into variables please.

[barjin]

I first thought of a function that would return these arrow functions, closuring a URL (so we could only write this.urlPatternObjects.some(matchUrl(url)) - imo beautifully readable). Both of the arrow functions currently are the same thing. I just didn't want to define this matchUrl function in the middle of isUrlMatchingPatterns (as it's gonna be called many many many times) - and making it a private method in SitemapRequestList didn't really feel right either.

Idk, what do you think?

[janbuchar]

LGTM, except for that one comment

[janbuchar]

I was thinking about something like this:

Suggested change

	private isUrlMatchingPatterns(url: string): boolean {
	return (
	!this.urlExcludePatternObjects.some((patternObject) => {
	const { regexp, glob } = patternObject;
	return (regexp && url.match(regexp)) || (glob && minimatch(url, glob, { nocase: true }));
	}) &&
	(this.urlPatternObjects.length === 0 ||
	this.urlPatternObjects.some((patternObject) => {
	const { regexp, glob } = patternObject;
	return (regexp && url.match(regexp)) || (glob && minimatch(url, glob, { nocase: true }));
	}))
	);
	}
	private isUrlMatchingPatterns(url: string): boolean {
	const matchesSomeExcludePattern = this.urlExcludePatternObjects.some((patternObject) => {
	const { regexp, glob } = patternObject;
	return (regexp && url.match(regexp)) || (glob && minimatch(url, glob, { nocase: true }));
	})
	const matchesSomeIncludePattern = this.urlPatternObjects.some((patternObject) => {
	const { regexp, glob } = patternObject;
	return (regexp && url.match(regexp)) || (glob && minimatch(url, glob, { nocase: true }));
	})
	return (
	!matchesSomeExcludePattern &&
	(this.urlPatternObjects.length === 0 || matchesSomeIncludePattern)
	);
	}

[barjin]

Oh, right, I guess I went a bit overboard... but I'd argue that my solution makes it ever so slightly even DRYier.

Tomayto, tomahto, I think both of the solutions solve the same issue of readability in the condition, albeit each one a bit differently.

[janbuchar]

Yeah, what you did is cool. Feel free to merge.