-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
reconciler: 2 phase reconnects and tests #12333
reconciler: 2 phase reconnects and tests #12333
Conversation
3249c78
to
371e2f9
Compare
371e2f9
to
479d73e
Compare
479d73e
to
a92d25c
Compare
a92d25c
to
08f4037
Compare
08f4037
to
a4cbd80
Compare
a4cbd80
to
277c36e
Compare
…ct eval trigger by.
…ng and 2 phase reconnects.
…ch for unknown status.
277c36e
to
568df7c
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Overall this is looking good @DerekStrickland. I've left some suggestions on the tests, as making those legible will be important for maintenance. And then you've got a couple of open questions that we should resolve before it's ready to merge.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
* structs: Add alloc.Expired & alloc.Reconnected functions. Add Reconnect eval trigger by. * node_endpoint: Emit new eval for reconnecting unknown allocs. * filterByTainted: handle 2 phase commit filtering rules. * reconciler: Append AllocState on disconnect. Logic updates from testing and 2 phase reconnects. * allocs: Set reconnect timestamp. Destroy if not DesiredStatusRun. Watch for unknown status.
* structs: Add alloc.Expired & alloc.Reconnected functions. Add Reconnect eval trigger by. * node_endpoint: Emit new eval for reconnecting unknown allocs. * filterByTainted: handle 2 phase commit filtering rules. * reconciler: Append AllocState on disconnect. Logic updates from testing and 2 phase reconnects. * allocs: Set reconnect timestamp. Destroy if not DesiredStatusRun. Watch for unknown status.
I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions. |
Not ready for review
This PR changes the generic scheduler reconcile logic to support zero-downtime reconnects.
The initial implementation for disconnected clients stopped replacement allocations before the reconnecting node synced its alloc state with the server. If the alloc failed while the node was disconnected, this could lead to a scenario where the replacement was stopped, but the reconnecting alloc was not healthy, and the cluster ends up underprovisioned.
This PR changes the logic so that allocs are identified as reconnecting after the reconnecting client sends its alloc updates to the server. To make this work, I made the following changes.
Node.UpdateAlloc
EvalTriggerReconnect
.Expired
function toAllocation
that usesAllocStates
to determine if an unknown allocation has expired.AllocState
entry when creating the unknown alloc update.Reconnected
function toAllocation
that usesAllocStates
plusTaskStates
to determine if an allocation has reconnected and whether it expired before reconnecting.filterByTainted
function to:disconnecting
,lost
, orignore
based on client status and expirationstate.
reconnecting
,lost
, orignore
based on their alloc and task states.filterByResheduable
function to also be able to handledisconnecting
allocations.ignore
set now returned byfilterByTainted
intoDesiredChanges
allocWatcher
to also trigger when a previous allocation transitions to unknown