Version 2 of the crawler for the Decentralized Search project https://rorur.com

Build each of the top level files and run them as systemd daemons ( see example.service file).

Create file /etc/dse/dse.conf with the following information:

directory where data structures will be located

available disk space in GB ( at the moment, we do only namespace crawl so this number must be amortized: multiply your actual disk space by 10^4)

cpu usage ( target cpu usage for single processor)

bandwidth in MB/s ( target BW usage)