Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The scheduler may publish task when acquiring the lock failure #238

Closed
Junzki opened this issue Apr 11, 2023 · 1 comment
Closed

The scheduler may publish task when acquiring the lock failure #238

Junzki opened this issue Apr 11, 2023 · 1 comment

Comments

@Junzki
Copy link

Junzki commented Apr 11, 2023

Same as described in #208
The redbeat acquires the lock here:
https://github.com/sibson/redbeat/blob/main/redbeat/schedulers.py#L515

from the source code above, the scheduler connects to the beat_init signal. It will acquire lock on start.

However, according to celery's implementation:
https://github.com/celery/celery/blob/main/celery/utils/dispatch/signal.py#L275

When acquire lock failed, it raises an exception. The celery signal dispatcher will ignore this exception (just print out to the log/stderr),
and the scheduler will continue to work.

In the situation which have multiple scheduler working in one cluster, there will be more than one scheduler sending tasks at same time.

@nicklyra
Copy link

We have seen this behavior as well, when at one point we had a misconfiguration in some parameters that caused Redbeat to always throw an exception during startup; in that case, we ended up with every instance in the cluster thinking it needed to publish tasks. Non-hilarity ensued.

@sibson sibson closed this as completed in ca59dc7 Apr 29, 2023
sibson added a commit that referenced this issue Apr 29, 2023
fix #238, #208, failure to acquire lock on startup
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants