Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Canary AutoPromote fails when non-canary taskgroups exist #8139

Closed
djenriquez opened this issue Jun 9, 2020 · 1 comment · Fixed by #11013
Closed

Canary AutoPromote fails when non-canary taskgroups exist #8139

djenriquez opened this issue Jun 9, 2020 · 1 comment · Fixed by #11013

Comments

@djenriquez
Copy link

djenriquez commented Jun 9, 2020

Nomad version

Output from nomad version

nomad version
Nomad v0.11.2 (807cfebe90d56f9e5beec3e72936ebe86acc8ce3)

Operating system and Environment details

Amazon Linux 2

Issue

When canary deploys are mixed with non-canary deploys, auto-promote defaults to false.

This should consider the deployment process for each taskgroup and realize that, if each taskgroup were its own job, each would autopromote successfully.

The deployer should only set autopromote for the job to false IF there exists a taskgroup where canary > 0 AND autopromote == false. Otherwise, if all taskgroups where canary > 0 && autopromote == true, the entire job should autopromote.

Screen Shot 2020-06-09 at 12 01 35 PM

In the above screenshot, taskgroups 1,2, and 5 have canary = 1 and autopromote = true. Taskgroups 3 and 4 are configured for rolling deploys. All taskgroups succeed their deployment, but the job still requires a manual promotion.

I believe this is because the rolling deploys need to be configured with autopromote = false, otherwise we get a warning message:

rpc error: 1 error occurred: * Task group <REDACTED> validation failed: 1 error occurred: * 1 error occurred: * Auto Promote requires a Canary count greater than zero

but since this is a rolling deploy taskgroup, we cannot set Canary > 0.

Reproduction steps

1 Job, 2 Taskgroups

  1. Create a job with 2 taskgroups
  2. Define one taskgroup as a canary deploy /w autopromotion
  3. Define the other taskgroup as a rolling deploy
  4. Make a change to the job config to trigger a deployment

2 Jobs, 1 Taskgroup (separated)

  1. Separate the job into two with one taskgroup each
  2. Perform a deploy on each job and watch each job deploy to completion

In summary, if each taskgroup were separated into separate jobs and can complete a deployment on its own, then a job that is a collection of those same taskgroups should also behave the same way.

@github-actions
Copy link

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Oct 17, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants