Add optional shutdown delay to tasks #3043
Conversation
Fixes #2441 Defaults to 0 (no delay) for backward compat and because this feature should be opt-in.
client/task_runner_test.go
Outdated
| } | ||
|
|
||
| // No shutdown escape hatch for this delay, so don't set it too high | ||
| task.ShutdownDelay = 5 * time.Second |
There was a problem hiding this comment.
Way too long :) Try to avoid sleeping for any longer than a second.
There was a problem hiding this comment.
If we're unable to check the Consul ops before the delay is up we'll get a false failure. Mind if I make it 2s to give Travis plenty of time? Since this test runs in parallel with the rest of client/ -- which takes ~50s total -- it should slow down the overall wallclock time.
nomad/structs/diff_test.go
Outdated
| @@ -1981,8 +1981,8 @@ func TestTaskGroupDiff(t *testing.T) { | |||
| { | |||
| Type: DiffTypeEdited, | |||
| Name: "Driver", | |||
| Old: "docker", | |||
| New: "exec", | |||
| Old: "1s", | |||
There was a problem hiding this comment.
Not the intended change I don't think?
| @@ -71,6 +71,12 @@ job "docs" { | |||
| [Consul][] for service discovery. Nomad automatically registers when a task | |||
| is started and de-registers it when the task dies. | |||
|
|
|||
| - `shutdown_delay` `(string: "0s")` - Specifies the duration to wait when | |||
| killing a task between removing it from Consul and sending it a shutdown | |||
client/task_runner.go
Outdated
| @@ -1171,6 +1171,12 @@ func (r *TaskRunner) run() { | |||
| interpTask := interpolateServices(r.envBuilder.Build(), r.task) | |||
| r.consul.RemoveTask(r.alloc.ID, interpTask) | |||
|
|
|||
| // Delay actually killing the task if configured. See #244 | |||
| if r.task.ShutdownDelay > 0 { | |||
| r.logger.Printf("[INFO] client: delaying shutdown of alloc %q task %q for %q", r.alloc.ID, r.task.Name, r.task.ShutdownDelay) | |||
There was a problem hiding this comment.
I just worry people are going to forget they set a shutdown delay and wonder why Nomad seems to hang since there's no indication outside of looking at the job file that this task is blocked on a shutdown delay...
I'll make it DEBUG though... users stopping/updating jobs should be aware of the job's settings and behavior.
|
I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions. |
Fixes #2441
Defaults to 0 (no delay) for backward compat and because this feature
should be opt-in.