The automation of the process of crawling through the deep web and searching for key terms or domain names would be a valuable tool for any tester, researcher, or IT security team. This framework would need to be simple to use and able to continuously run. Additionally, results would need to meaningful and easy for the user to understand.
To solve this issue, I’ve developed a deep web crawling tool called Deepminer, that allows easy searching to help users identify any results. Deepminer pulls Onion sites from well-known resources and searches using any extracted HTML. Deepminer runs continuously and indefinitely to ensure that the database is updated. Site names, directories, HTML, and connected sites are saved to a SQLite database file where users can analyze the results.
Deepminer is written using Python 3 and tested in Ubuntu 20.04. Searches are conducted using SQLite FTS5 Extension, for both full text and regex searches. For ease of searching, users can use SQLite DB Browser for a graphical interface.
git
Screen
On a fresh install of Ubuntu 20.04 run the following commands
-
sudo apt install git -
sudo apt install python3-pip -
pip3 install pysocks -
sudo apt install tor -
sudo apt install screen -
git clone https://github.com/Conso1eCowb0y/Deepminer -
cd Deepminer -
sudo python3 deepminer.py -
To install SQLite DB Browser, run the following command:
sudo apt install sqlitebrowser
- Multi-threading
- Kivy based search GUI