Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

state: reflect key Nomad data within internal store #30

Merged
merged 42 commits into from
Apr 29, 2020

Conversation

jrasell
Copy link
Member

@jrasell jrasell commented Mar 20, 2020

This PR moves the policy-storage package to use blocking queries on the scaling policy list endpoint. It also adds processing for job scaling state, which is used to identify jobs that are stopped, and in the future will help provide functionality such as cooldowns.

closes #38

@jrasell jrasell self-assigned this Mar 20, 2020
@jrasell jrasell changed the title policy-storage: using blocking queries to track scaling policy updates state: reflect key Nomad data within internal store Mar 23, 2020
@jrasell
Copy link
Member Author

jrasell commented Mar 23, 2020

PR should have commits squashed before merging!

@jrasell jrasell marked this pull request as ready for review April 7, 2020 11:42
Copy link
Contributor

@cgbaker cgbaker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good stuff 👍. i left some comments, then some issues which i'm not sure about (e.g., whether we stop tracking deleted policies).

helper/blocking/blocking.go Outdated Show resolved Hide resolved
state/policy/watcher.go Outdated Show resolved Hide resolved
state/policy/watcher.go Outdated Show resolved Hide resolved
state/policy/state.go Outdated Show resolved Hide resolved
state/policy/watcher.go Outdated Show resolved Hide resolved
state/scale.go Outdated Show resolved Hide resolved
state/status/watcher.go Outdated Show resolved Hide resolved
helper/blocking/blocking.go Outdated Show resolved Hide resolved
state/policy.go Outdated Show resolved Hide resolved
state/policy.go Outdated Show resolved Hide resolved
state/scale.go Outdated Show resolved Hide resolved
Copy link
Contributor

@lgfa29 lgfa29 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great stuff!

Copy link
Contributor

@lgfa29 lgfa29 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On working on #63 I noticed that policies are not being removed when the job is purged (nomad stop --purge). This should probably be handled in the update function.

@jrasell jrasell marked this pull request as draft April 22, 2020 06:49
@jrasell jrasell marked this pull request as ready for review April 22, 2020 10:25
@jrasell jrasell requested review from lgfa29 and cgbaker April 22, 2020 10:25
@jrasell jrasell marked this pull request as draft April 22, 2020 15:46
@lgfa29 lgfa29 marked this pull request as ready for review April 28, 2020 19:55
@lgfa29
Copy link
Contributor

lgfa29 commented Apr 28, 2020

This PR has been updated with the new control loop redesign. The main changes include:

  1. Use of blocking queries and channels to communicate state changes from Nomad to the Autoscaler
  2. New concepts:
    1. Policy Source: provide channels and methods for a caller to receive updates about the list of policies registered or for a specific policy.
    2. Nomad Policy Source: implementation of a Policy Source to read policies from a Nomad cluster. It uses blocking queries to avoid unnecessary API calls.
    3. Policy Manager: receives updates from a Policy Source and creates/removes Policy Handlers as necessary. It blocks until the list of policies changes, at which point the list of handlers is reconciled.
    4. Policy Handler: monitors a specific policy for updates and submit them back to the agent for evaluation when required. It blocks until the policy being monitored is updated.
  3. Update the Target plugin interface to replace Count with Status. Status returns Count in addition to a flag call Ready to indicate if the target is ready to receive scaling actions and Meta tags that will be used in future work.
  4. Update the Nomad target plugin to store job status in a local state store and use blocking queries to update it when something changes.

@jrasell
Copy link
Member Author

jrasell commented Apr 29, 2020

commits should be squashed when merging.

Copy link
Contributor

@cgbaker cgbaker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

small changes before merge, no need for re-review

agent/agent.go Outdated Show resolved Hide resolved
policy/manager.go Outdated Show resolved Hide resolved
policy/handler.go Outdated Show resolved Hide resolved
policy/nomad/validate.go Outdated Show resolved Hide resolved
policy/nomad/validate.go Outdated Show resolved Hide resolved
policy/nomad/validate.go Outdated Show resolved Hide resolved
policy/nomad/validate.go Outdated Show resolved Hide resolved
policy/nomad/validate.go Outdated Show resolved Hide resolved
policy/handler.go Outdated Show resolved Hide resolved
policy/nomad/source.go Show resolved Hide resolved
@lgfa29 lgfa29 merged commit 9526956 into master Apr 29, 2020
@lgfa29 lgfa29 deleted the f-blocking-query-track-policies branch April 29, 2020 17:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

state: use blocking queries to mirror critical Nomad state internally
3 participants