-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[question] [enhancement] Canary tagging in Nomad 0.6.0 #2920
Comments
we're just adding tag with v:${VERSION} to distinguish between them |
We were already doing adding a If you deploy a new nomad job version using with a canary update strategy but using with the same service artefact (let's say you just changed job parameters, like resources, ports, etc) a version tag will be ambiguous, and yet these changes can dictate the success of failure of the deployment. I agree that is this a stretched corner case, but it's just for sake of example. Having a well known tag, like |
According to https://www.nomadproject.io/docs/operating-a-job/update-strategies/blue-green-and-canary-deployments.html the Being able to point your staging frontend to a DNS name like One idea would be to have a |
For what it's worth, even having a rudimentary ability to add a Looking at Nomad code it also seems fairly straight forward to add the extra tag during the deployment start and remove it upon promote. |
Just a thought; could canary deployments during upgrade get a feature of connection draining to services rather than the host. What this could do is have a safe manner of upgrade which is transparent to the user, and possibly help the user to achieve the right thing. |
Hi, @dagar looking how straight forward to add and remove the canary can we add this feature in the 0.8 release ? I can raise an pull request if it ok to add this feature . |
Just to update folks, we will be tackling this in the near term. It will hopefully be in a 0.8.X release and at latest in 0.9 (worst case). |
While you're thinking about this, please also consider changing the alloc index. If I now add one canary to my job, it gets index 0. So there's 2 instances of index 0. This seriously screws up metrics. Another thought: since we're not replacing exiting instances, but adding new ones, the overall load on the job gets less (as there's more instances). If you are trying to do performance tuning, this is quite annoying. |
I've confirmed this is still an open issue for my hashi-ui.nomad job that "Requires Promotion" Running
Using
where my http://consul.localhost/ui/dc1/services/hashi-ui has all three allocation tagged with both traefik.frontend.rule. IMHO only the canary running service should have the canary tag and nothing else. Now http://canary-hashiui.localhost/ works because it is the final route rule defined as
|
I would have reopened #3340 but it was closed since it was marked as a duplicate of this ticket. maybe it would have to have canary_name as well as canary_tag |
nvm ... ignore my comments, I figured it out.
|
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Canaries are now a first class thing in the new
update
stanza and while within nomad is it possible to check what allocations are in which version, this does not propagate to the service registration in consul.How can the canaries be filtered apart under the consul catalog? I was hopping for them to have a "canary" tag or something. But I could not find anything in the service catalog that allows me to filter out the canary.
If there is a recommended way of doing this, maybe it could mentioned in the docs, if there is not here is a suggestion:
Maybe nomad could add a pre-defined "canary" tag for the services in the canary state, and remove it after promotion. Or that tag could be configured with a new
canary_tag
keyword in theupdate
stanza.This would help tremendously marking the canary instances in the gateways in order to adjust it's weight in the pool or only allow access from a non-public endpoint
The text was updated successfully, but these errors were encountered: