Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ability to restart all running tasks/allocs of a job #698

Closed
supernomad opened this issue Jan 22, 2016 · 64 comments · Fixed by #16278
Closed

Add ability to restart all running tasks/allocs of a job #698

supernomad opened this issue Jan 22, 2016 · 64 comments · Fixed by #16278
Labels
stage/accepted Confirmed, and intend to work on. No timeline committment though. theme/api HTTP API and SDK issues type/enhancement

Comments

@supernomad
Copy link

So I would love the ability to restart tasks, at the very least restart an entire job, but preferably single allocations. This is very useful for when a particular allocation or job happens to get in a bad state.

I am thinking something like nomad restart <job> or nomad alloc-restart <alloc-id>.

One of my specific use cases, is I have a cluster of rabbitmq nodes, and at some point one of the nodes gets partitioned from the rest of the cluster. I would like to restart that specific node (allocation in nomad parlance), or be able to preform a rolling restart to the entire cluster (job in nomad parlance).

Does this sound useful?

@dadgar dadgar added stage/thinking theme/api HTTP API and SDK issues labels Jan 22, 2016
@dadgar
Copy link
Contributor

dadgar commented Jan 22, 2016

Its not a bad idea! In the mean time if you just want to restart the job you can stop and then run it again�.

@mkabischev
Copy link

I think it will be good feature. Now i can stop and then run job, but it won`t be graceful.

@gpaggi
Copy link

gpaggi commented Apr 19, 2016

+1
Another use case: most of our services read their configuration either from static files or consul and when there are any changes in the properties the services need to be rolling-restarted.
Stopping and starting the job would cause a service interruption and a blue/green deployment for a configuration change is a bit over kill.

@supernomad did you get a chance to look into it?

@jtuthehien
Copy link

+1 for this feature

@c4milo
Copy link
Contributor

c4milo commented Jun 14, 2016

This is much needed in order to effectively reload configurations without having downtimes. As mentioned above, blue/green doesn't really scale well when you have too many tasks and it is sort of unpredictable since it depends on the specific app being deployed playing well with multiple versions of it running at the same time.

@liclac
Copy link

liclac commented Jul 14, 2016

I'd very much like to see this, for a slightly different use case:

I have something running as a system job (in this case, a wrapper script that essentially does docker pull ... && docker run ..., it needs to mount a host directory to work, this is a workaround for #150). To roll out an update, I currently need to change a dummy environment variable, or Nomad won't know anything changed.

@mohitarora
Copy link

+1

@dennybaa
Copy link

Why not, guys please add it, should be trivial.

@jippi
Copy link
Contributor

jippi commented Sep 27, 2016

👍 on this feature as well :)

@xyzjace
Copy link

xyzjace commented Jan 16, 2017

👍 For us, too.

@ashald
Copy link

ashald commented Jan 26, 2017

We would be happy to see this feature as well. Sometimes... services just need a manual restart. :( Would be nice if it was possible to restart individual tasks or task groups.

@rokka-n
Copy link

rokka-n commented Jan 26, 2017

Having rolling "restart" option is a very valid case for tasks/jobs.

@jippi
Copy link
Contributor

jippi commented Jan 26, 2017

What i've done as a hack is to have a key_or_default inline template{} stanza in the task stanza for each of these keys, simply writing them to some random temp file

  • apps/${NOMAD_JOB_NAME}
  • apps/${NOMAD_JOB_NAME}/${NOMAD_TASK_NAME}
  • apps/${NOMAD_JOB_NAME}/${NOMAD_TASK_NAME}/${NOMAD_ALLOC_INDEX}
  • apps/${NOMAD_ALLOC_NAME}

that each got a change_type = restart or signal with the appropriate change_signal value

so i can do manual rolling restart of any nomad task by simply changing or creating one of those consul keys in my cluster programatically... at my own pace to do a controlled restart too :)

writing to consul KV /apps/${NOMAD_JOB_NAME} will restart all tasks in the job
writing to consul KV /apps/${NOMAD_JOB_NAME}/${NOMAD_TASK_NAME} will restart all tasks within a job
writing to consul KV /apps/${NOMAD_JOB_NAME}/${NOMAD_TASK_NAME}/${NOMAD_ALLOC_INDEX} will restart one specific task index within the job

@ashald
Copy link

ashald commented Jan 26, 2017

@jippi that's super smart! Thanks, I guess I'll use that for the time being. :)

But that level of control is something that would be great to see in Nomad's native API.

P.S.: That reminds me about my hack/workaround to secure any resource in Nginx (e.g., Nomad API) using Consul ACL tokens with auth_request to some read-only api endpoints. :D

@pznamensky
Copy link

Would be useful for us too.

@dansteen
Copy link

dansteen commented Sep 6, 2017

This would also be useful for the new deployment stuff. The ability to re-trigger a deployment would be great.

@JewelPengin
Copy link

JewelPengin commented Sep 6, 2017

Throwing in my +1 but also my non-consul based brute force way:

export NOMAD_ADDR=http://[server-ip]:[admin-port]

curl $NOMAD_ADDR/v1/job/:jobId | jq '.TaskGroups[0].Count = 0 | {"Job": .}' | curl -X POST -d @- $NOMAD_ADDR/v1/job/:jobId

sleep 5

curl $NOMAD_ADDR/v1/job/:jobId | jq '.TaskGroups[0].Count = 1 | {"Job": .}' | curl -X POST -d @- $NOMAD_ADDR/v1/job/:jobId

It requires the jq binary to be installed (which I would highly recommend anyway), but it will first grab the job, modify the task group count to 0, post it back to update, then all over again back to 1 (or whatever number is needed).

Again, kinda brute force and not as elegant as @jippi's, but it works if I need to get something done quickly.

@danielwpz
Copy link

Really useful feature! Please do it :D

@sullivanchan
Copy link

I have do some verification follow @jippi suggestion, and data = "{{ key apps/app1/app1/${NOMAD_ALLOC_INDEX} }}" in template stanza, but job start always pending, seems env just get by https://www.nomadproject.io/docs/job-specification/template.html#inline-template {{ env "ENV_VAR" }}, i want to know how to integrate env variable into key string, does anybody have the same question?

@mildred
Copy link
Contributor

mildred commented Sep 19, 2017

This is standard golang template:

          {{keyOrDefault (printf "apps/app1/app1/%s" (env "NOMAD_ALLOC_INDEX")) ""}}

@mildred
Copy link
Contributor

mildred commented Sep 19, 2017

I suggest you use keyOrDefault instead of just key which will prevent your service to start unless the key exists in consul.

@thevilledev
Copy link
Contributor

As a workaround I've been using Nomad's meta stanza to control restarts. Meta keys are populated as environment variables to tasks, so whenever meta block is changed all related tasks (or task groups) are restarted. Meta blocks can be defined on the top-level of the job, per task-group or per task.

For example to restart all tasks in all task groups you could run this:

$ nomad inspect some-job | \
jq --arg d "$(date)" '.Job.Meta={restarted_at: $d}' | \
curl -X POST -d @- nomad.service.consul:4646/v1/jobs

This follows update stanza as well.

@maihde
Copy link
Contributor

maihde commented Mar 2, 2018

I have made a first pass at implementing this, you can find my changes here.

Basically, I've added a -restart flag to nomad run. For example:

nomad run -restart myjob.nomad

When the -restart flag is applied it triggers an update, the same as if you would have changed the meta block, so you get the benefits of canaries and rolling restarts without having to actually change the job file.

If there is agreement that this implementation is going down the right path, I will go the the trouble of writing tests and making sure it works for system scheduler, parameterized jobs, etc.

@jovandeginste
Copy link

Why not implement this without the need for a plan? Basically, nomad restart myjobname (which should use the current plan)

As a sysop, I sometimes need to force a restart of a job, but I don't have the plan (and don't want to go through nomad inspect | parse)

@rkettelerij
Copy link
Contributor

Agreeing with @jovandeginste here. A restart shouldn't need a job definition in my option, since the job definition is already known inside Nomad.

@jovandeginste
Copy link

I do see the case to re-submit an existing job with a plan that may or may not have changed but always wanting to force a restart (of the whole job) while submitting. So both are interesting options.

@Oloremo
Copy link
Contributor

Oloremo commented Dec 1, 2020

looking forward to this as well

@datadexer
Copy link

same here!

@OmarQunsul
Copy link

OmarQunsul commented Jan 7, 2021

I am also surprised this feature doesn't exist. In Docker Swarm for example docker service update --force SERVICE_NAME.
I was expecting something under the job command nomad job restart, that restarts each alloc without downtime on the whole job

@tgross tgross added this to Needs Roadmapping in Nomad - Community Issues Triage Feb 12, 2021
@jpasichnyk
Copy link

+1 for this feature. We just moved to nomad 1.x and are trying to move to the built in Nomad UI (from HashiUI - https://github.com/jippi/hashi-ui), and having the ability to restart a job from here would be great. Sometimes we have application instances that go unhealthy from a system perspective but are still running fine in docker. In this case we don't want to force restart them as depending on the reason they are unhealthy they may not be able to safely restart. Restarting the whole job via a rolling restart is a great way to fix this state, but there is no way to do it for us other than building a new container version and promoting a new job over the existing job (even if the bits being deployed are identical). HashiUI can restart via rolling restart or a stop/start. Nomad UI and CLI should support doing this as well.

@tgross tgross removed this from Needs Roadmapping in Nomad - Community Issues Triage Mar 3, 2021
@thatsk
Copy link

thatsk commented Apr 27, 2021

is this added in nomad UI. Or still in phase.?

@stupidlamo
Copy link

+1 to this feature, really need to shut down hashi-ui and use only nomad native, but can't due to unvailabilty of rolling restart

@kunalsingthakur
Copy link

yeah @tgross there is situation where container dependent on consul key-value and if we update key value in consul then after restart our service it will populate new values in out container so we really think this need to be allocated in nomad UI and get rid of hashiui . don't need to maintain two UI for nomad

@kunalsingthakur
Copy link

are we supposed to think this is on our roadmap

@dg-eparizzi
Copy link

+1 to this feature

@josegonzalez
Copy link
Contributor

The way hashi-ui implements this is by injecting a label into the job, which messes with nomad job plan as the same job will result in a change as the local job won't have the injected label.

@thatsk
Copy link

thatsk commented Sep 1, 2021 via email

@sbrl
Copy link

sbrl commented Sep 3, 2021

The way hashi-ui implements this is by injecting a label into the job, which messes with nomad job plan as the same job will result in a change as the local job won't have the injected label.

An easy CLI subcommand / HTTP API call that function would be very handy.

@victusfate
Copy link

I ended up getting what I wanted (a rolling restart of an existing application) using the following python snippet and the Nomad HTTP API

    get_job_url = NOMAD_URL + os.path.join('/v1/job',job_id)
    get_job_response = requests.get(get_job_url)
    job = get_job_response.json()
    if 'Meta' not in job or job['Meta'] is None:
      job['Meta'] = {}
    job['Meta']['Restart'] = str(time.time())
    job = { 'Job': job, 'PreserveCounts': True }
    
    # now post it back
    post_url = NOMAD_URL + os.path.join('/v1/jobs')
    post_job_response = requests.post(post_url,json=job)
    print('restart job response',post_job_response.json())

@maxadamo
Copy link

maxadamo commented Feb 7, 2022

Unless I'm overlooking a possible drawback, the command suggested by @mxab looks good to me.
You can use any variation of the command and add it onto your shell aliases:

nomad job status <job-name> | awk '{if (/run(.*)running/) {system("nomad alloc restart " $1)}}'
nomad job status <job-name> | awk '/run(.*)running/{print $1}' | xargs -t -n 1 nomad alloc restart

@Laboltus
Copy link

Unless I'm overlooking a possible drawback, the command suggested by @mxab looks good to me. You can use any variation of the command and add it onto your shell aliases:

nomad job status <job-name> | awk '{if (/run(.*)running/) {system("nomad alloc restart " $1)}}'
nomad job status <job-name> | awk '/run(.*)running/{print $1}' | xargs -t -n 1 nomad alloc restart

As I understand "nomad alloc restart" doesn't re-download artifacts and docker images ? I need to restart a job with an actual docker image.

@tgross
Copy link
Member

tgross commented Aug 23, 2022

Doing some issue cleanup and realizing there's a whole lot of different feature requests being discussed in this issue over the years, many of which landed long ago. I'm going to re-title this issue to narrow the scope to the remaining request.

@tgross tgross changed the title Add ability to restart running tasks/jobs Add ability to restart all running tasks/allocs of a job Aug 23, 2022
@tgross tgross added this to Needs Triage in Nomad - Community Issues Triage via automation Aug 23, 2022
@tgross tgross moved this from Needs Triage to Needs Roadmapping in Nomad - Community Issues Triage Aug 23, 2022
Nomad - Community Issues Triage automation moved this from Needs Roadmapping to Done Mar 23, 2023
@EugenKon
Copy link

With the command above I can not restart the task which failed:

$ nomad job restart -task nginx-task portal
==> 2024-02-23T11:13:56-05:00: Restarting 1 allocation
    2024-02-23T11:13:56-05:00: Restarting task "nginx-task" in allocation "27caddf2" for group "services"
==> 2024-02-23T11:13:56-05:00: Job restart finished with errors

1 error occurred while restarting job:
* Error restarting allocation "27caddf2": Failed to restart task "nginx-task": Unexpected response code: 500 (Task not running)


$ nomad alloc restart 27caddf2
Failed to restart allocation:

Unexpected response code: 500 (restart of an alloc that should not run)

It is not clear how to restart failed task?

@jippi
Copy link
Contributor

jippi commented Feb 23, 2024

Please open a new issue for that. This issue is many years old and closed :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
stage/accepted Confirmed, and intend to work on. No timeline committment though. theme/api HTTP API and SDK issues type/enhancement
Projects
Development

Successfully merging a pull request may close this issue.