Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

agent: support scaling cooldowns to aid oscillation migration. #117

Merged
merged 6 commits into from
May 11, 2020

Conversation

jrasell
Copy link
Member

@jrasell jrasell commented May 5, 2020

Cooldown is now supported within a scaling policy as a time
duration. If the scaling policy omits the cooldown field, a
default value will be used. The default value can be controlled
by the CLI and has a default value of 5mins.

In the event the agent successfully registers a scaling action,
the policy manager will be instructed to place the policy eval
handler in cooldown by updating the ticker to the cooldown time
duration. The ticker is reset to its eval time duration on the
first tick out of cooldown.

Target plugins can optionally return a meta entry to detail the
last scaling event from their prespective. This allows the
autoscaler to enforce cooldowns, even when scaling actions have
been triggered manually by operators. The flow is similar to that
mentioned above, the only difference being that the ticker update
is based on the calculated remaning cooldown time from the last
event timestamp.

closes #12

@jrasell jrasell added this to the 0.0.2 milestone May 5, 2020
@jrasell jrasell requested review from lgfa29, jazzyfresh and cgbaker May 5, 2020 14:26
@jrasell jrasell self-assigned this May 5, 2020
Copy link
Contributor

@lgfa29 lgfa29 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few comments, but nothing major. The most important one is the mutex for the ticker, I think we can have a race condition without it (but that was already true before this PR).

policy/manager.go Outdated Show resolved Hide resolved
policy/handler.go Outdated Show resolved Hide resolved
policy/handler.go Outdated Show resolved Hide resolved
policy/handler_test.go Outdated Show resolved Hide resolved
agent/config/config.go Outdated Show resolved Hide resolved
agent/config/config_test.go Outdated Show resolved Hide resolved
Cooldown is now supported within a scaling policy as a time
duration. If the scaling policy omits the cooldown field, a
default value will be used. The default value can be controlled
by the CLI and has a default value of 5mins.

In the event the agent successfully registers a scaling action,
the policy manager will be instructed to place the policy eval
handler in cooldown. The cooldown is enforced by performing a
timed block using time.Timer.

Target plugins can optionally return a meta entry to detail the
last scaling event from their prespective. This allows the
autoscaler to enforce cooldowns, even when scaling actions have
been triggered manually by operators. The flow is similar to that
mentioned above, the only difference being that the block period
is based on the calculated remaning cooldown time from the last
event timestamp.
Copy link
Contributor

@lgfa29 lgfa29 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Much clear solution now. There's no state to keep track of anymore since the Go routine will block during cooldown, so it reduces complexity and mental loading when reasoning about the code.

I made a few suggestions but nothing major.

Good stuff 👍

policy/handler.go Outdated Show resolved Hide resolved
policy/handler.go Outdated Show resolved Hide resolved
policy/handler.go Outdated Show resolved Hide resolved
policy/manager.go Outdated Show resolved Hide resolved
policy/manager.go Show resolved Hide resolved
policy/handler.go Outdated Show resolved Hide resolved
agent/config/config_test.go Outdated Show resolved Hide resolved
policy/handler.go Outdated Show resolved Hide resolved
policy/handler.go Outdated Show resolved Hide resolved
jrasell and others added 2 commits May 7, 2020 16:41
Copy link
Contributor

@cgbaker cgbaker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some minor comments, no big changes.

policy/handler.go Outdated Show resolved Hide resolved
agent/config/config.go Outdated Show resolved Hide resolved
plugins/builtin/target/nomad/plugin/state.go Outdated Show resolved Hide resolved
plugins/builtin/target/nomad/plugin/state.go Show resolved Hide resolved
policy/handler.go Outdated Show resolved Hide resolved
policy/handler.go Show resolved Hide resolved
policy/handler.go Show resolved Hide resolved
policy/handler.go Show resolved Hide resolved
agent/config/config.go Show resolved Hide resolved
policy/handler.go Show resolved Hide resolved
@jrasell jrasell merged commit a071c81 into master May 11, 2020
@jrasell jrasell deleted the f-gh-12 branch May 11, 2020 11:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

agent: add cooldown period
4 participants