-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Template: Allow rolling restart for restart change_mode. #2202
Comments
@dadgar I've already try to use splay but with multiple instances, all the instances of the task will restart at the same time. Here is my test Job:
In this case, when the value change in consul , Nomad will wait for a random time (but less than 2 minutes) and restart both instances at the time. |
@cyrilgdn The issue is about rolling restart. This means restart one-by-one not random. |
@akaspin Yes I know (given that I created the issue :)), that's what I'd like to do. I only tried the splay option for an alternative way to avoid downtime. |
Ok. Finally I implement solution (https://hub.docker.com/r/akaspin/docker-backstab/). This designed for docker. Design:
For now this implementation tested under CoreOS cluster with Nomad where I'm running Mesos. |
@cyrilgdn You didn't miss something I did 😄 The splay wasn't being applied correctly. Fixed and will be in 0.5.3 |
@dadgar Thanks for fixing the splay option! But for me the question of this issue remains valid. In my case, if I have 2 instances of the same task (count=2), I can set a big splay hopping both will not restart at the same time but this is not guaranteed. |
@cyrilgdn Unfortunately we will not be doing coordinated restarts because of template changes. If this is a requirement for you I would suggest building some external tooling to manage the restarts. |
@dadgar Thanks for your answer! Not be doing, like never? Maybe I missed or misunderstood something (as we started to use Nomad few weeks ago and just tested template stenza). |
@dadgar Wouldn't it be possible to reuse the the It's probably possible to do something with external tooling, but
All in all, it seems that this defeats the purpose of having the |
In its current form we can not support that type of behavior. The plugin is executing locally on every client and there is no coordination. I have referenced a GitHub issue on consul-template that could solve this. If consul-template supported locking then we could limit parallelism. |
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Nomad version
Nomad v0.5.2
Issue
We started using template stenza with consul.
It works well but the restart change_mode causes (a big) downtime even with multiple instances of the task.
Especially with the docker driver which removes the docker image and download it again during the restart (seems to be linked to #1530).
It would be great if the restarts can be done (optionally) more smoothly, ideally based on update strategy.
What do you think? Did I missed something ?
Thank you!
The text was updated successfully, but these errors were encountered: