##Web Mining
-
scrapy
Scrapy is a fast high-level screen scraping and web crawling framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.
Project Source: https://github.com/scrapy/scrapy
Project Homepage: http://scrapy.org/ -
Pattern
Web mining module for Python, with tools for scraping, natural language processing, machine learning, network analysis and visualization.
Project Source: https://github.com/clips/pattern
Project Homepage: http://www.clips.ua.ac.be/pages/pattern -
portia
Portia is a tool for visually scraping web sites without any programming knowledge.
Project Source: https://github.com/scrapinghub/portia -
python-goose
Html Content / Article Extractor, web scrapping lib in Python.
Project Source: https://github.com/grangier/python-goose -
newspaper
News extraction, article extraction and content curation in python.
Project Source: https://github.com/codelucas/newspaper
Project Homepage: http://newspaper.readthedocs.org/en/latest/ -
gensim
Gensim is a Python library for topic modelling, document indexing and similarity retrieval with large corpora.
Project Source: https://github.com/piskvorky/gensim
Project Homepage: http://radimrehurek.com/gensim/ -
distribute_crawler
A distributed web crawler.
Project Source: https://github.com/gnemoug/distribute_crawler -
pyspider
A spider system in python.
Project Source: https://github.com/binux/pyspider -
tagger
A Python module for extracting relevant tags from text documents.
Project Source: https://github.com/apresta/tagger -
cola
A distributed crawling framework.
Project Source: https://github.com/chineking/cola