Spider v1.9.0

What's Changed

• feat(crawl): add subdomain and tld crawling by @j-mendez in #59

You can now gather all the content for all of your pages in one go between tlds and subdomains.

-- Example

extern crate spider;

use spider::website::Website;

fn main() {
  let mut website: Website = Website::new("https://rsseau.fr");
  website.configuration.subdomains = true;
  website.configuration.tld = true;
  website.crawl();

  for page in website.get_pages() {
    println!("- {}", page.get_url());
  }
}

Full Changelog: v1.8.0...v1.9.0