Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

scaleutils: add nodeselector package to select nodes suitable for termination #435

Merged
merged 7 commits into from
Mar 31, 2021

Conversation

jrasell
Copy link
Member

@jrasell jrasell commented Mar 26, 2021

closes #417

This work allows operators to specify the strategy which is used to select nodes for termination during a scale in action. Up until now, the Nomad Autoscaler would simply pick nodes from the top of the list as returned from the Nomad node list API endpoint. This method is still included and is now known as newest_create_index.

In order to explicitly configure the selector behaviour, a new policy target configuration option node_selector_strategy has been made available. This config option is parsed by the scaleutils helper and therefore is available to all cluster target plugins which utilise it.

In the absence of an operator configured option, the value defaults to least_busy. This strategy calculates the percentage allocation of each node in the pool, selecting those which have the lowest value and therefore are allocated the least amount of resources. The resources are CPU and memory.

The second selector strategy available is empty. This strategy will only select nodes for termination that do not have non-terminal allocations. This is ideal for worker pools running batch workloads which should not be interrupted or migrated.

Admin notes:

  • Documentation on the Nomad site will be required as a follow up step.
  • To those internally testing this, I have some new TF I have been working on that will allow easier testing of different architectures such as this example. I will share this privately for now as it is very alpha.
  • This PR only implements a single strategy whereas the original issue states several. I will raise follow up issues to individually track additional selector strategies. This is ideal candidates for first-issues and will be marked as such.

Copy link
Contributor

@cgbaker cgbaker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

one array bounds question and a few comments 👍

sdk/helper/scaleutils/nodeselector/defualt.go Outdated Show resolved Hide resolved
sdk/helper/scaleutils/nodeselector/defualt_test.go Outdated Show resolved Hide resolved
sdk/helper/scaleutils/nodeselector/defualt_test.go Outdated Show resolved Hide resolved
sdk/helper/scaleutils/nodeselector/empty_nodes.go Outdated Show resolved Hide resolved
sdk/helper/scaleutils/nodeselector/empty_nodes.go Outdated Show resolved Hide resolved
},
{
inputCfg: map[string]string{
"node_selector_strategy": "aliens",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👽 👽 👽

@jrasell jrasell marked this pull request as draft March 26, 2021 17:10
@jrasell jrasell marked this pull request as ready for review March 29, 2021 12:49
@jrasell jrasell requested a review from cgbaker March 29, 2021 12:49
@jrasell jrasell force-pushed the f-gh-417 branch 3 times, most recently from 316c940 to 8395e9a Compare March 29, 2021 13:08
Copy link
Contributor

@lgfa29 lgfa29 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor comments, but it seems to be working 🎉

sdk/helper/scaleutils/nodeselector/least_busy.go Outdated Show resolved Hide resolved
sdk/helper/scaleutils/nodeselector/least_busy.go Outdated Show resolved Hide resolved
@jrasell jrasell merged commit 87ddd32 into main Mar 31, 2021
@jrasell jrasell deleted the f-gh-417 branch March 31, 2021 06:40
jrasell added a commit that referenced this pull request Mar 31, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

feature: allow user to specify the strategy used for node identification during scale-in
3 participants