Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

horizontal-cluster: add custom drain message when draining nodes for scale-in #627

Closed
jrasell opened this issue Apr 28, 2023 · 0 comments
Closed

Comments

@jrasell
Copy link
Member

jrasell commented Apr 28, 2023

When draining a node, the Nomad API and CLI supports specifying custom metadata and a message which is persisted on the node object stored within state. The Nomad Autoscaler should take advantage of this, by adding custom data when triggering node drains due to scale-in cluster activities. This will allow operators to track certain autoscaler operations via the Nomad API for administrative purposes and allow searching for failed scaling due to the current inability of the autoscaler to share state across instances of itself.

The exact content of the metadata should be discussed before moving forward. If any community members reading this have thoughts, please feel free to share them as a comment.

Nomad CLI docs: https://developer.hashicorp.com/nomad/docs/commands/node/drain
Nomad API docs: https://developer.hashicorp.com/nomad/api-docs/nodes#drain-node

Example of draining with message and metadata:

$ nomad node drain -enable -m "drain due to autoscaling" -meta 'user=jrasell' 4bd8
Are you sure you want to enable drain mode for node "4bd861a2-707a-32d6-89ba-ce07ea8bb05a"? [y/N] y
2023-04-28T08:38:57+01:00: Ctrl-C to stop monitoring: will not cancel the node drain
2023-04-28T08:38:57+01:00: Node "4bd861a2-707a-32d6-89ba-ce07ea8bb05a" drain strategy set
2023-04-28T08:38:57+01:00: All allocations on node "4bd861a2-707a-32d6-89ba-ce07ea8bb05a" have stopped
2023-04-28T08:38:57+01:00: Drain complete for node 4bd861a2-707a-32d6-89ba-ce07ea8bb05a
$ curl -s localhost:4646/v1/node/4bd861a2-707a-32d6-89ba-ce07ea8bb05a | jq '.LastDrain'
{
  "AccessorID": "",
  "Meta": {
    "user": "jrasell",
    "message": "drain due to autoscaling"
  },
  "StartedAt": "2023-04-28T08:38:57+01:00",
  "Status": "complete",
  "UpdatedAt": "2023-04-28T08:38:57+01:00"
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants