-
Notifications
You must be signed in to change notification settings - Fork 445
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat!: don't run Celery workers in dev mode #1041
Conversation
0122e69
to
4334c84
Compare
4334c84
to
bf0460b
Compare
bf0460b
to
5c3b229
Compare
@@ -0,0 +1,2 @@ | |||
- 💥[Feature] Use Docker compose profiles to control services. (by @arbrandes) --> | |||
- [Fix] Don't start Celery workers in dev mode, as they're never used. (by @arbrandes) --> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We would need some sort of guidelines to understand how the workers can be setup in dev mode, especially now that we are using docker compose profiles.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just a simple tutor dev dc start lms-worker
would start the service, for instance, but the workers wouldn't be actually usable without modifying settings.py. Is this what you mean we should document?
Of course, the other question would be: why would anybody want to do this in a dev environment?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, the command and setting changes, yes. I get the point that most devs might not be using this, we can have the doc in a followup. It is not a blocker.
@@ -171,6 +171,8 @@ services: | |||
{%- endfor %} | |||
depends_on: | |||
- lms | |||
profiles: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wasn't familiar with profiles, to be honest. I like the idea of not running workers in dev, but I'd like to avoid making a breaking change to the filter API, and also changes across many python functions.
Instead, could we simply move the lms-worker and cms-worker declarations from docker-compose.yml to docker-compose.prod.yml?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think it's going to be that simple. There are other places in the code that assume those service definitions exist. For instance, https://github.com/overhangio/tutor/blob/master/tutor/commands/compose.py#L289. Plus, I suspect profiles might be useful down the road, woudn't you say so?
If you're dead set against it, though, there is another way: we can just set the worker services in the dev compose override file to a profile like "donotstart". We get the same thing this PR provides, but without actually supporting different profiles.
5c3b229
to
41c9dc5
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The changes look good to me, though I will keep this open until Régis is back.
41c9dc5
to
35eb685
Compare
Haha, I have a competing proposal (although I let it languish, because I was never able to confirm or deny that My thinking was that it would be good to run these tasks asynchronously, because it would expose developers to the same race conditions that inevitably always happen with Celery tasks in production. But I can see the other side of the argument: async tasks are harder to debug and the workers use resources. Either way, I think we can all agree that we should either use these workers, or turn them off. So I see four paths forward:
My vote is (1), but I would support any of them. |
|
Tutor's importing * from devstack.py[1] for the development settings, and that means that we aren't using Celery workers at all in dev mode (see [2]). This makes it so they don't start in dev mode, thus saving everyboding a significant chunk of RAM. [1] https://github.com/overhangio/tutor/blob/master/tutor/templates/apps/openedx/settings/lms/development.py#L3 [2] https://github.com/openedx/edx-platform/blob/master/lms/envs/devstack.py#L35 To do so, we rely on Docker compose profiles[3]. [3] https://docs.docker.com/compose/profiles/ BREAKING CHANGE: the `COMPOSE_PROJECT_STARTED` hook signature had to be changed to accomodate profile selection.
35eb685
to
e3919ee
Compare
@arbrandes Hi, what's the plan for this PR? Thanks |
@DawoudSheraz, good question. I still used it locally all the time. I figure I could implement the alternative I describe above that has a smaller footprint (setting a fake docker profile for the workers in the dev override docker-compose), which addresses Regis' concerns and gets us the same immediate effect of not having workers in dev. (But we lose the option of using profiles for other things in the future.) I'll issue a separate PR so we can compare. |
In development, edx-platform runs with CELERY_ALWAYS_EAGER=True, which means that lms-worker and cms-worker never catch celery tasks! This change was heavily inspired by: #1041
In development, edx-platform runs with CELERY_ALWAYS_EAGER=True, which means that lms-worker and cms-worker never catch celery tasks! This change was heavily inspired by: #1041
Hey Adolfo, your PR completely fell off my radar, I'm sorry about that. A recent issue (#1126) has drawn my attention on celery again. I found a not-so-hackish way of disabling lms-worker and cms-worker in development, without using profiles. You can check out my PR here: #1128 While I do agree that profiles are super interesting, and we should probably use them in the future instead of multiple docker-compose files, I think it's a much bigger change that needs extensive testing. Getting rid of celery workers in dev is an obvious improvement for everyone, and I don't want to wait longer to merge it. |
In development, edx-platform runs with CELERY_ALWAYS_EAGER=True, which means that lms-worker and cms-worker never catch celery tasks! This change was heavily inspired by: #1041
Closing in favour of #1128. |
Tutor's importing * from devstack.py[1] for the development settings, and that means that we aren't using Celery workers at all in dev mode (see [2]). This removes them from the dev compose file, thus saving everyboding a significant chunk of RAM.
[1] https://github.com/overhangio/tutor/blob/master/tutor/templates/apps/openedx/settings/lms/development.py#L3
[2] https://github.com/openedx/edx-platform/blob/master/lms/envs/devstack.py#L35
To do so, we rely on Docker compose profiles[3].
[3] https://docs.docker.com/compose/profiles/
BREAKING CHANGE: the
COMPOSE_PROJECT_STARTED
hook signature had to be changed to accomodate profile selection.