Skip to content
This repository has been archived by the owner on Sep 16, 2022. It is now read-only.

Introduce Celery worker for processing background tasks #385

Merged
merged 3 commits into from
Aug 28, 2019

Conversation

rptrchv
Copy link
Contributor

@rptrchv rptrchv commented Aug 27, 2019

Closes #359

Everything's ready for both:

  • one time async tasks execution
  • cron-like periodic tasks run

I've set max task execution time to 60s in order to avoid worker hangs. If we'll have more long-running tasks - we can increase this value.
For now we have 1 worker instance with the number of parallel processes equals to CPU cores, this also can be adjusted in the future.

@vpetersson
Copy link
Contributor

This also need to be ported to the corresponding Helm files.

@rptrchv
Copy link
Contributor Author

rptrchv commented Aug 27, 2019

This also need to be ported to the corresponding Helm files.

@a-martynovich can you do this please?

@a-martynovich
Copy link
Contributor

@rptrchv I can try, but it will be blind coding. To check this stuff locally I need to set up local k8s cluster, DNS, SSL certs, etc. it’s possible but will take lots of time.
I don’t like the idea of testing stuff on the prod but that’s the last resort.

@vpetersson
Copy link
Contributor

Fair - i can take care of this part as long as you guys are OK with the code base.

@a-martynovich
Copy link
Contributor

@vpetersson I believe the Docker part of this PR isn't going to change anymore so you can push your helm code to this branch or you could create your own and we could merge it to this one.

@vpetersson
Copy link
Contributor

@vpetersson I believe the Docker part of this PR isn't going to change anymore so you can push your helm code to this branch or you could create your own and we could merge it to this one.

I pushed it to this branch as i was unable to push it directly this branch (since it is a separate cloned repo).

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Introduce Celery worker for processing background tasks
3 participants