Refactoring & Improvements

[MikeMeliz]

Description

I went through the code to improve the maintainability, and to make it easier for new contributors. The commits are broken down to explain each change, as follows:

• a873088[Maintainability]: Fixed several typos and grammar errors around the files, which was producing warnings in IDEs.
• e4e1f57[Maintainability]: Changed the names of functions and variables into something readable.
• 1e20434[Fix]: Since we're on Python3 it's not necessary to have the names of modules inside the init file.
• b59a593[Fix]: Fixed tests to not fail with https websites.
• 4b95c19[EasinessToUse]: Made the name now() of files shorter for easier navigation.
• 1109342[Maintainability]: Introduced an /output folder (with a .gitkeep inside) to keep the project clean from output files, and have an easier navigation.
• d3070dc[Fix]: When crawling and extract, if the same filename exists (or index.htm if it can't fetch a filename), it would write above it.
• ddca979[Fix]: Changed now() of links.txt to match the rest of the results.
• 0ddfb9f[Performance]: Removed the 1 sec pause between requests, which greatly improved the performance.
• e474760[Fix]: Small fix on the output of extractor.py to make it easier to find the file.
• b689557[Performance]: External links, telephones and mails were not getting appending on the file.
• 6a1a9da[Fix]: Logs didn't work, or was a bit similar to links.txt. Added time, and module name.
• fd66e20[Maintainability]: Added some comments for future reference.
• e960721[Maintainability]: Updated README.md.
• f7bdb9d[Fix]: Fix Log function, and include 40x errors also.
• 63c1e3c[Maintainability]: Reduced the size of .gitignore but removing unnecessary exclusions.

Motivation and Context

I was planning for way too long to go over the code and start making it easier to new contributors to understand how it works.
Also during several tests the script was failing, which drives users away of it, so several fixes were needed.
It should be now on an okay-ish stage to start including improvements (eg. parallel scanning, IP rotations, etc)

How Has This Been Tested?

python torcrawl.py -w -u google.com -c -d 1 -p 1 -l -v -e
python torcrawl.py -w -u google.com -c -d 1 -p 1 -l -v
python torcrawl.py -w -u google.com -c -d 3 -l -v

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.

[sonarcloud]

Quality Gate passed

Issues
0 New issues
2 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud