Spider v1.6.1

Performance Tuned

Speed of crawler cranked up a notch and now the fastest open-source spider crawler available. View the benchmarks in the CI action for results. If you know of any alternative crawlers feel free to open an issue so we can add the benchmark comparisons.

What's Changed

• test(bench): add self task execution of bench by @j-mendez in #40
• perf(links): filter dup links after async batch
• chore(delay): fix crawl delay thread groups

Full Changelog: v1.6.0...v1.6.1

Perf increased after commit 053eea4.

Benchmarks against crolly and node-crawler (cases are about identical in implementation ) .