Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Make Celery workers and queues configurable #780

Merged
merged 4 commits into from
Jan 16, 2024

Conversation

webjunkie
Copy link
Contributor

@webjunkie webjunkie commented Dec 6, 2023

Description

We want to run separate Celery workers that consume a different set of queues each.
See RFC https://github.com/PostHog/product-internal/pull/537

Type of change

  • New feature (non-breaking change which adds functionality)

Changes

  • add possibility to configure several workers, then use range in the template
    • please advise if this is the way to go or not at all
  • we can pass the config options as environment variables, which override Celery default settings
  • by default, nothing should change right now
    • but allows that we can put a config into dev environment and then see how it works

Example config for dev later:

  workers:
    - name: priority
      queues:
        - email
    - name: default
      queues:
        - celery # default queue for Celery
        - insight_refresh
        - insight_export
    - name: gevent
      pool: gevent
      concurrency: 50
      queues:
        - analytics_queries

How has this been tested?

  • how can I properly test this?

Checklist

  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have added tests that prove my fix is effective or that my feature works

@webjunkie webjunkie marked this pull request as ready for review December 14, 2023 08:07
@webjunkie webjunkie requested a review from frankh January 8, 2024 10:44
@webjunkie
Copy link
Contributor Author

@frankh Can you take a look here?

@webjunkie webjunkie merged commit c96fcbb into main Jan 16, 2024
48 checks passed
@webjunkie webjunkie deleted the feature/celery-multi-queues branch January 16, 2024 11:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants