Stop already rescheduled but somehow running allocs #8886
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
notnoop
pushed a commit
that referenced
this pull request
Sep 16, 2020
Stop already rescheduled but somehow running allocs
teutat3s
pushed a commit
to teutat3s/nomad
that referenced
this pull request
Oct 27, 2020
…ation Stop already rescheduled but somehow running allocs
teutat3s
pushed a commit
to teutat3s/nomad
that referenced
this pull request
Dec 9, 2020
…ation Stop already rescheduled but somehow running allocs
teutat3s
pushed a commit
to teutat3s/nomad
that referenced
this pull request
Jan 16, 2021
…ation Stop already rescheduled but somehow running allocs
teutat3s
pushed a commit
to teutat3s/nomad
that referenced
this pull request
Jan 17, 2021
…ation Stop already rescheduled but somehow running allocs
|
I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a band-aid fix for a case where an alloc has been rescheduled but somehow is left in a running state. In such case, currently, the alloc is left running uninterrupted as it is removed from consideration by the scheduler. The alloc will remain running uninterrupted even after a new job version is pushed, resulting into a mixed fleet. Currently, operators need to manually force stop these "leaked" allocations.
This PR fixes the issue by reconsidering an alloc that has been rescheduled but is still running for scheduling purposes. The check in reconciler_util.go meant that once an allocation is rescheduled, forever it will never be examined again as it's removed from the untainted allocations.
But how did it get here
It is very unclear how an alloc can get into this state. In all of my testing so far, only failed allocs can be rescheduled, and once they are rescheduled, and
alloc.DesiredStatusis set tostop. So theoretically, we should never see a running allocation withNextAllocation != "".@cgbaker observed this issue in #5921 (comment). We've also had bugs in the past where finished allocations get to re-run again upon a client restart, e.g. #6354, #5945.
So while we need to keep digging into understanding the underlying cause, I propose this "band-aid" to at least recover smoothly from the bad state.