This Python project can be used as a corresponding web crawler from a specific URL. This browser monitors its features while navigating on the given URL and keeps detailed logs for each URL visited.
- Searches a large number of HTML from a URL.
- Finds links in the explored HTML content and adds URLs to visit them.
- You can set the maximum depth level.
- Keeps a list of visited URLs and does not revisit the same URL.
- There are appropriate error message and exception handling elements for error handling.
- Uses color logging.
-
Clone the project:
git clone https://github.com/0MeMo07/Web-Crawler.git
-
Go to the project directory:
cd Web-Crawler
-
Install required dependencies:
pip install -r requirements.txt
-
Run the crawler Python file::
python crawler.py