reconciler: 2 phase reconnects and tests

[DerekStrickland]

Not ready for review

This PR changes the generic scheduler reconcile logic to support zero-downtime reconnects.

The initial implementation for disconnected clients stopped replacement allocations before the reconnecting node synced its alloc state with the server. If the alloc failed while the node was disconnected, this could lead to a scenario where the replacement was stopped, but the reconnecting alloc was not healthy, and the cluster ends up underprovisioned.

This PR changes the logic so that allocs are identified as reconnecting after the reconnecting client sends its alloc updates to the server. To make this work, I made the following changes.

• Node.UpdateAlloc
  • Looks up the server-side alloc state immediately instead of after checking if the incoming client alloc is terminal.
  • Previously, the incoming alloc update would be skipped if it was in a terminal state on the client.
  • Now it will only be skipped if it is in a terminal state at the client and not unknown at the server.
  • Any incoming alloc updates that are unknown at the sever will now trigger an eval with EvalTriggerReconnect.
• I added anExpiredfunction to Allocation that uses AllocStates to determine if an unknown allocation has expired.
• I updated the reconciler to append an AllocState entry when creating the unknown alloc update.
• I added aReconnectedfunction to Allocation that uses AllocStates plus TaskStates to determine if an allocation has reconnected and whether it expired before reconnecting.
• I updated the filterByTainted function to:
  • Filter allocs on disconnected nodes into disconnecting, lost, or ignore based on client status and expiration
    state.
  • Filter allocs on reconnecting nodes into reconnecting, lost, or ignore based on their alloc and task states.
• I updated the filterByResheduable function to also be able to handle disconnecting allocations.
• I updated the reconciler to:
  • Incorporate the ignore set now returned by filterByTainted into DesiredChanges
  • Consider job version/create_index in multiple areas.
• I updated the allocWatcher to also trigger when a previous allocation transitions to unknown

[tgross]

Overall this is looking good @DerekStrickland. I've left some suggestions on the tests, as making those legible will be important for maintenance. And then you've got a couple of open questions that we should resolve before it's ready to merge.

[tgross]

LGTM!

[github-actions]

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.