This application is a Django project which is a ad scraper.
- Clone the repository.
git clone https://github.com/Dylan-YD/scraper.git - Install Python on your machine.
- Navigate to the project folder.
cd scraper - Create virtual environment.
python -m venv {environment name} - Activate environment.
cd {enviroment name}/Scripts activate or source bin/activate - Install requirements.
pip install -r requirements.txt - Create .env file
- Put your secret key, allowed host url, aws information in .env file like .env.example
- Change baseURL variable in
\crawler\static\settings.jsif you want run project on custom url - Run the server.
python manage.py runserver
- Go to the home page.
http://127.0.0.1:8000/home - Enter the query of the ads you want to scrape.
- Click the button to scrape the website.
- The progress will be shown on the page.
- After the scraping is done, the result will be shown on the page
http://127.0.0.1:8000/crawler/ads.
scraper
├── crawler
│ ├── admin.py
│ ├── apps.py
│ ├── __init__.py
│ ├── migrations
│ ├── models.py
│ ├── static
│ ├── templates
│ ├── tests.py
│ ├── urls.py
│ └── views.py
├── db.sqlite3
├── manage.py
├── README.md
├── requirements.txt
└── scraper
├── asgi.py
├── __init__.py
├── settings.py
├── urls.py
└── wsgi.py