This is a web scraper for MyAnimeList score progressions on the Wayback Machine internet archive.
The data gathered from this repo is used in another project MALTrendsWeb
- Make sure you have a version of
python3
andvirtualenv
installed.
Seehttps://virtualenv.pypa.io/en/latest/installation/
for installation instructions.
- Clone this repo to your local machine using
https://github.com/danielzhaotongliu/MALTrends
- Navigate to the MALTrends root directory, then run the following commands to setup virtualenv and install libraries
$ virtualenv -p python3 venv/
$ source venv/bin/activate
$ pip install -r requirements.txt
$ source venv/bin/activate
- Before running the spiders you need the MyAnimeList anime ID for the anime that you would like to scrape for
For example: the anime ID for Sword Art Online is 11757
- Run the following command in the MALTrends root directory
$ scrapy crawl score_spider -a id=mal_id -o snapshots_mal_id.jl
$ python plot.py