Skip to content

spidy Web Crawler Release 1.2

Compare
Choose a tag to compare
@rivermont rivermont released this 07 Sep 23:08
· 173 commits to master since this release

Added domain restrictions. Crawling can now be limited to a certain domain, such as wsj.com, https://www.wsj.com, or https://www.wsj.com/article. Can be set when entering configuration settings or in the config files.
Also more bugfixes and MIME types because those are cool.