spidy Web Crawler Release 1.2
Added domain restrictions. Crawling can now be limited to a certain domain, such as wsj.com
, https://www.wsj.com
, or https://www.wsj.com/article
. Can be set when entering configuration settings or in the config files.
Also more bugfixes and MIME types because those are cool.