nomad job stop sends interrupt signal for docker driver?

[jf]

Nomad version

Nomad v0.13.0-dev (fb170f37a05d712e3046d604c362804c7934cfc9+CHANGES)

Operating system and Environment details

Ubuntu 20.04.1 LTS

Issue

Reproduction steps

Not too sure why nobody has noticed, but it seems like nomad job stop sends a docker task the interrupt signal, as opposed to the expected TERM as per https://www.nomadproject.io/docs/job-specification/task#kill_signal

To confirm this, I basically added debug logging to

nomad/drivers/docker/handle.go

Line 127 in 9438544

func (h *taskHandle) Kill(killTimeout time.Duration, signal os.Signal) error {

My debug output for the signal parameter gives interrupt.

My first clue was from my alloc logs, where I see 'Sent interrupt'. The line from nomad job stop is called out below

Time                  Type        Description
2020-09-19T10:50:04Z  Killing     Sent interrupt. Waiting 30s before force killing
2020-09-19T10:50:04Z  Killed      Task successfully killed
2020-09-19T10:50:04Z  Terminated  Exit Code: 1, Exit Message: "Docker container exited with non-zero exit code: 1"
2020-09-19T10:50:02Z  Killing     Sent interrupt. Waiting 30s before force killing                 --> did `nomad job stop` here
2020-09-19T10:45:51Z  Started     Task started by client
2020-09-19T10:45:37Z  Driver      Downloading image
2020-09-19T10:45:37Z  Task Setup  Building Task Directory
2020-09-19T10:45:34Z  Received    Task received by client

Job file (if appropriate)

any standard job file will do; you can use the example file from nomad job init too

[shoenig]

Thanks for reporting, and the PR, @jf!

Let me see if I can reproduce this going back in Nomad versions - pretty sure the Docker driver was intended to use SIGTERM to behave more like the real docker stop command, but now I'm wondering if that was actually true.

[jf]

Sure. I just had the impression (as I believe most folks would as well) from the docs (https://www.nomadproject.io/docs/job-specification/task#kill_signal) that my tasks would be getting TERM instead of KILL

[shoenig]

I was able to confirm this hasn't worked as intended since at least Nomad v0.8.x.

Given that fixing this is a change in behavior I think we should add a section in the version upgrade guide + call it out as a backwards incompatibility in the changelog. Also add some test(s). I'll go ahead and merge #8933 and then take care of the rest.

[jf]

Great, thank you @shoenig ! Would you mind tagging me when you do the other stuff? Especially for the tests, so I can learn and then hopefully take care of that for any future PRs. Thank you!

[DWSR]

Does this affect stopping allocations individually as well?

[jf]

Does this affect stopping allocations individually as well?

I believe so. I dont see why stopping allocations would end up using another code path. Can you try it out and see if you encounter this as well?

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.