Skip to content
This repository has been archived by the owner on Nov 1, 2023. It is now read-only.

Add NoNewWork node command to tell free nodes to stop asking for new work #866

Conversation

bmc-msft
Copy link
Contributor

@bmc-msft bmc-msft commented May 7, 2021

Currently, there are two states that are non-obvious as to what happens next.

  1. A free node is "out of date". Right now, it will stay on the old version until it tries to launch a task, at which point the node will check "can I run this work". When the node asks "can I run this work", the node is rejected and then moves into the reimage/delete state.
  2. A free node is part of a shrinking scaleset. Right now, it will wait until a task launch similar to the "out of date" node to start the shrinking.

This PR adds sending "no_new_work" command to Nodes which basically allows us to tell free nodes to stop.

NOTE: The node command infra in the supervisor will fail badly if it receives an unsupported command. As such, this includes "minimum supported version" support that gate sending the message such that we don't accidentally break nodes that are actively doing work.

@bmc-msft bmc-msft marked this pull request as ready for review May 7, 2021 16:32
@bmc-msft bmc-msft requested review from chkeita and ranweiler May 7, 2021 16:32
@bmc-msft
Copy link
Contributor Author

bmc-msft commented May 7, 2021

Note, as is this currently checks is the minimum version 2.16.1, which is the next possible release this could go into.

If we make a 2.16.1 that doesn't include this value, we should bump it.

@bmc-msft
Copy link
Contributor Author

bmc-msft commented May 7, 2021

Manual testing done:

  • Verified that the no_new_work messages don't get sent to older nodes.
  • Verified that nodes already running work do not stop because of these messages.
  • Verified that nodes not running work stop as expected.

@ranweiler ranweiler changed the title Add NewNewWork node command to tell free nodes to stop asking for new work Add NoNewWork node command to tell free nodes to stop asking for new work May 7, 2021
@bmc-msft bmc-msft requested a review from ranweiler May 7, 2021 17:29
@bmc-msft bmc-msft merged commit 221a331 into microsoft:main May 7, 2021
@bmc-msft bmc-msft deleted the enable-telling-free-nodes-to-stop-asking-for-work branch May 7, 2021 17:55
@bmc-msft bmc-msft linked an issue May 17, 2021 that may be closed by this pull request
@ghost ghost locked as resolved and limited conversation to collaborators Jun 6, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

VMSS Shutdown command in CLI not working
4 participants