GitHub - liangw89/WhoWas

Branches Tags

Name		Name	Last commit message	Last commit date
Latest commit History 38 Commits
data		data
README		README
api.py		api.py
blacklist		blacklist
clus_sub.py		clus_sub.py
clus_top.py		clus_top.py
common.py		common.py
config.py		config.py
feature.py		feature.py
logger.py		logger.py
robotexclusionrulesparser.py		robotexclusionrulesparser.py
scanner.py		scanner.py
test_input.csv		test_input.csv

Repository files navigation

dependency: 
sudo pip install requests==2.3.0
sudo pip install logutils
sudo pip install pymysql
sudo pip install python-hashes

1. configure.py: set basic parameters in the file
2. scanner.py: prob+webpage feteching
3. feature.py: feature extractor
4. clus_top.py: top-level clusters
5. clus_sub.py: get the final clustering results.