-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
keep placed canaries aligned in raft store #6975
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change | ||
---|---|---|---|---|
|
@@ -529,10 +529,6 @@ func (s *GenericScheduler) computePlacements(destructive, place []placementResul | |||
// If we are placing a canary and we found a match, add the canary | ||||
// to the deployment state object and mark it as a canary. | ||||
if missing.Canary() && s.deployment != nil { | ||||
if state, ok := s.deployment.TaskGroups[tg.Name]; ok { | ||||
state.PlacedCanaries = append(state.PlacedCanaries, alloc.ID) | ||||
} | ||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm unclear why an inplace update is inappropriate or the source of a bug here. It's hard to follow this flow, but Line 170 in f5441e6
This seems like an intentional decision to make deployments safe for inplace updates, and it appears If that's not the case I'd love to see a little more cleanup or commenting in the code to help future developers. Maybe an explicit comment on the There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. So there is definitely some misdirection going on with all these
The scheduler deployment is actually not set by the reconciler, the reconciler copies the scheduler deployment to its own https://github.com/hashicorp/nomad/blob/master/scheduler/generic_sched.go#L355-L357 Currently the reconciler will only return a So the following logic is mutating state that never gets written to the raft log because only changes on if state, ok := s.deployment.TaskGroups[tg.Name]; ok {
state.PlacedCanaries = append(state.PlacedCanaries, alloc.ID)
} |
||||
|
||||
alloc.DeploymentStatus = &structs.AllocDeploymentStatus{ | ||||
Canary: true, | ||||
} | ||||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actually, do we need to update
UpdateDeploymentPromotion
so that canaries that get promoted are removed from PlacedCanaries?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think so, since
UpdateDeploymentPromotion
deals with promoting the deployment/canaries I think that the values for Placed shouldn't ever be decremented. If you want to see the deployment status of a previous deployment the value for Placed should be the total that were placedThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Umm.. I'm a bit concerned about the reconciler logic. Currently,
handleGroupCanaries
returnscanaries
, by checkingPlacedCanaries
exclusively. If a canary got promoted, how do we ensure that scheduler no longer treats it as one.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe (and can followup / confirm) Canary promotion only happens when a deployment is promoted. If there is an old deployment with non-promoted canaries, or the deployment is failed, it will use PlacedCanaries that haven't been promoted to be stopped.
And then after the deployment is promoted, the promotion eval includes the promoted deployment canaries as canaries, but then further down
canaryState
will be falseWhich is used in computeStop to stop the previous version non canary allocs. I think after all of this, the deploy is marked as successful, and even though placed canaries stays populated, we check deployment status and no new canaries are ever placed until a new deployment