Webcrawler for incoming links

Requirements

NPM, Node, MySQL

Install

Clone this repo
Run npm install
Create mysql table and modify it in ./db.js
Run node crawler.js --install
Add a first website (e.g. http://news.ycombinator.com) with empty status to Table sites

Run the crawler

node crawler.js

What it can not do

crawl pdfs
crawl flash