-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add ability to restart all running tasks/allocs of a job #698
Comments
Its not a bad idea! In the mean time if you just want to restart the job you can stop and then run it again�. |
I think it will be good feature. Now i can stop and then run job, but it won`t be graceful. |
+1 @supernomad did you get a chance to look into it? |
+1 for this feature |
This is much needed in order to effectively reload configurations without having downtimes. As mentioned above, blue/green doesn't really scale well when you have too many tasks and it is sort of unpredictable since it depends on the specific app being deployed playing well with multiple versions of it running at the same time. |
I'd very much like to see this, for a slightly different use case: I have something running as a |
+1 |
Why not, guys please add it, should be trivial. |
👍 on this feature as well :) |
👍 For us, too. |
We would be happy to see this feature as well. Sometimes... services just need a manual restart. :( Would be nice if it was possible to restart individual tasks or task groups. |
Having rolling "restart" option is a very valid case for tasks/jobs. |
What i've done as a hack is to have a
that each got a so i can do manual rolling restart of any nomad task by simply changing or creating one of those consul keys in my cluster programatically... at my own pace to do a controlled restart too :) writing to consul KV |
@jippi that's super smart! Thanks, I guess I'll use that for the time being. :) But that level of control is something that would be great to see in Nomad's native API. P.S.: That reminds me about my hack/workaround to secure any resource in Nginx (e.g., Nomad API) using Consul ACL tokens with |
Would be useful for us too. |
This would also be useful for the new deployment stuff. The ability to re-trigger a deployment would be great. |
Throwing in my +1 but also my non-consul based brute force way:
It requires the Again, kinda brute force and not as elegant as @jippi's, but it works if I need to get something done quickly. |
Really useful feature! Please do it :D |
I have do some verification follow @jippi suggestion, and data = "{{ key apps/app1/app1/${NOMAD_ALLOC_INDEX} }}" in template stanza, but job start always pending, seems env just get by https://www.nomadproject.io/docs/job-specification/template.html#inline-template {{ env "ENV_VAR" }}, i want to know how to integrate env variable into key string, does anybody have the same question? |
This is standard golang template:
|
I suggest you use |
As a workaround I've been using Nomad's meta stanza to control restarts. Meta keys are populated as environment variables to tasks, so whenever For example to restart all tasks in all task groups you could run this:
This follows |
I have made a first pass at implementing this, you can find my changes here. Basically, I've added a
When the If there is agreement that this implementation is going down the right path, I will go the the trouble of writing tests and making sure it works for system scheduler, parameterized jobs, etc. |
Why not implement this without the need for a plan? Basically, As a sysop, I sometimes need to force a restart of a job, but I don't have the plan (and don't want to go through |
Agreeing with @jovandeginste here. A restart shouldn't need a job definition in my option, since the job definition is already known inside Nomad. |
I do see the case to re-submit an existing job with a plan that may or may not have changed but always wanting to force a restart (of the whole job) while submitting. So both are interesting options. |
looking forward to this as well |
same here! |
I am also surprised this feature doesn't exist. In Docker Swarm for example |
+1 for this feature. We just moved to nomad 1.x and are trying to move to the built in Nomad UI (from HashiUI - https://github.com/jippi/hashi-ui), and having the ability to restart a job from here would be great. Sometimes we have application instances that go unhealthy from a system perspective but are still running fine in docker. In this case we don't want to force restart them as depending on the reason they are unhealthy they may not be able to safely restart. Restarting the whole job via a rolling restart is a great way to fix this state, but there is no way to do it for us other than building a new container version and promoting a new job over the existing job (even if the bits being deployed are identical). HashiUI can restart via rolling restart or a stop/start. Nomad UI and CLI should support doing this as well. |
is this added in nomad UI. Or still in phase.? |
+1 to this feature, really need to shut down hashi-ui and use only nomad native, but can't due to unvailabilty of rolling restart |
yeah @tgross there is situation where container dependent on consul key-value and if we update key value in consul then after restart our service it will populate new values in out container so we really think this need to be allocated in nomad UI and get rid of hashiui . don't need to maintain two UI for nomad |
are we supposed to think this is on our roadmap |
+1 to this feature |
The way hashi-ui implements this is by injecting a label into the job, which messes with |
Yes they are adding meta param like date time stamp
…On Wed, 1 Sep 2021, 13:19 Jose Diaz-Gonzalez, ***@***.***> wrote:
The way hashi-ui implements this is by injecting a label into the job,
which messes with nomad job plan as the same job will result in a change
as the local job won't have the injected label.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#698 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ADU5G3LCV2KYJY6AXQFFCJTT7XLGZANCNFSM4BZJUGLQ>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
|
An easy CLI subcommand / HTTP API call that function would be very handy. |
I ended up getting what I wanted (a rolling restart of an existing application) using the following python snippet and the Nomad HTTP API
|
Unless I'm overlooking a possible drawback, the command suggested by @mxab looks good to me. nomad job status <job-name> | awk '{if (/run(.*)running/) {system("nomad alloc restart " $1)}}'
nomad job status <job-name> | awk '/run(.*)running/{print $1}' | xargs -t -n 1 nomad alloc restart |
As I understand "nomad alloc restart" doesn't re-download artifacts and docker images ? I need to restart a job with an actual docker image. |
Doing some issue cleanup and realizing there's a whole lot of different feature requests being discussed in this issue over the years, many of which landed long ago. I'm going to re-title this issue to narrow the scope to the remaining request. |
With the command above I can not restart the task which failed:
It is not clear how to restart failed task? |
Please open a new issue for that. This issue is many years old and closed :) |
So I would love the ability to restart tasks, at the very least restart an entire job, but preferably single allocations. This is very useful for when a particular allocation or job happens to get in a bad state.
I am thinking something like
nomad restart <job>
ornomad alloc-restart <alloc-id>
.One of my specific use cases, is I have a cluster of rabbitmq nodes, and at some point one of the nodes gets partitioned from the rest of the cluster. I would like to restart that specific node (allocation in nomad parlance), or be able to preform a rolling restart to the entire cluster (job in nomad parlance).
Does this sound useful?
The text was updated successfully, but these errors were encountered: