Celery doesn't auto reload in development #3898
Conversation
|
Worth adding a link to the Celery issue you mentioned on Slack. |
|
Yeah, it seems that native autoreload in Celery was dropped in 3.1. |
bin/docker-entrypoint
Outdated
| @@ -14,10 +14,14 @@ scheduler() { | |||
| WORKERS_COUNT=${WORKERS_COUNT:-1} | |||
| QUEUES=${QUEUES:-celery} | |||
| SCHEDULE_DB=${SCHEDULE_DB:-celerybeat-schedule} | |||
| if [[ -n "$AUTO_RELOAD" ]]; then | |||
| WATCHER="watchmedo auto-restart --directory=./ --pattern=*.py --recursive --" | |||
There was a problem hiding this comment.
Maybe only watch the code library (redash/) so it doesn't reload on tests?
There was a problem hiding this comment.
Maybe we should add a new entrypoint for dev-celery to avoid any confusion or issues that might result here?
Of course :) It does. |
|
Test summary
Run details
View run in Cypress Dashboard ➡️ This comment has been generated by cypress-bot as a result of your project's GitHub integration settings. You can manage this integration in your project's settings in the Cypress Dashboard |
bin/docker-entrypoint
Outdated
| @@ -41,6 +51,7 @@ help() { | |||
| echo "server -- start Redash server (with gunicorn)" | |||
| echo "worker -- start Celery worker" | |||
| echo "scheduler -- start Celery worker with a beat (scheduler) process" | |||
| echo "dev_scheduler -- start Celery worker with a beat (scheduler) process which picks up code changes and reloads" | |||
There was a problem hiding this comment.
Let's name this dev_worker as it better describes what it does.
bin/docker-entrypoint
Outdated
| @@ -20,6 +20,16 @@ scheduler() { | |||
| exec /usr/local/bin/celery worker --app=redash.worker --beat -s$SCHEDULE_DB -c$WORKERS_COUNT -Q$QUEUES -linfo --max-tasks-per-child=10 -Ofair | |||
| } | |||
|
|
|||
| dev_scheduler() { | |||
| WORKERS_COUNT=${WORKERS_COUNT:-1} | |||
| QUEUES=${QUEUES:-celery} | |||
There was a problem hiding this comment.
As we have a dedicated dev worker entrypoint, we can have defaults that are inline with it:
WORKERS_COUNT = 2
QUEUES = "queries,scheduled_queries,celery,schemas"
Restarting Celery workers manually is no longer a thing after getredash/redash#3898
Restarting Celery workers manually is no longer a thing after getredash/redash#3898
* pick up *.py file changes and restart scheduler * only watch /redash in order to avoid reloading on other file changes (e.g. tests) * add dev_scheduler entrypoint * use exec * Update bin/docker-entrypoint * rename dev_scheduler to dev_worker * use same defaults as worker
Issue Summary
Using the standard Docker development setup, it seems that code changes are not picked up by
redash-worker.Steps to Reproduce
docker-compose upAny other info e.g. Why do you consider this to be a bug? What did you expect to happen instead?
Technical details: