-
This code can run only on linux. Use WSL if you on window.
-
Using python3.10
-
Fastest way to load environment variables from .env file:
export $(grep -v '^#' .env | xargs)
- Import URLs to Redis:
python3 scripts/redis_init_urls.py
- Start the monitor:
celery -A property_crawler flower
- Start the worker:
celery -A property_crawler worker -l info -Q crawl_list,crawl_item,io --purge -c 1
- Run main script to start pipeline:
python3 -m property_crawler.main --mode full
If you need to stop waiting tasks, use the following command:
celery -A property_crawler purge -Q crawl_list,crawl_item,io