A scalable, open-source webcrawler that writes website data to file while crawling each new webpage
$ git clone https://github.com/Boomslet/Web_Crawler
$ python setup.py install
%Run controller.py
>>> crawl('https://github.com/')
Successfully crawled https://github.com/
Successfully crawled https://github.com/#start-of-content
Successfully crawled https://github.com/features
Successfully crawled https://github.com/business
Successfully crawled https://github.com/pricing
Successfully crawled https://github.com/dashboard