Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Several fixes to reduce cases of stuck jobs #281

Merged
merged 5 commits into from
Sep 11, 2014
Merged

Several fixes to reduce cases of stuck jobs #281

merged 5 commits into from
Sep 11, 2014

Conversation

arikfr
Copy link
Member

@arikfr arikfr commented Sep 11, 2014

Due to the locking mechanism we have, that prevents same query from being executed more than once at a time, some queries might get into a "limbo" state and stop being executed until the lock is manually removed.

This pull request adds some fixes to reduce (eliminate?) those cases:

  1. Set expiry time to locks (default: 12 hours).
  2. If lock found is of a "ready" job (finished, revoked or failed), ignore it and execute a new job.
  3. Cleanup job that removes locks of non existing tasks (might happen after cold restart of the worker).

ready - revoked, finished or failed.
Enumerates all locks and removes those of non existing jobs. Useful
for case the worker is being cold restarted, and jobs are finished
properly.
arikfr added a commit that referenced this pull request Sep 11, 2014
Several fixes to reduce cases of stuck jobs
@arikfr arikfr merged commit a0c81f8 into master Sep 11, 2014
washort pushed a commit to washort/redash that referenced this pull request Jan 4, 2018
washort pushed a commit to washort/redash that referenced this pull request Jan 9, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant