systemd unit file kills executors when nomad agent is stopped

[ramSeraph]

Nomad version

Nomad v0.8.3 (c85483d)

Operating system and Environment details

uname -a

Linux <hostname> 4.4.0-97-generic #120-Ubuntu SMP Tue Sep 19 17:28:18 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

systemd --version

systemd 229
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN

Issue

restarting nomad agent on the host restarts the jobs on the host when nomad is running under systemd

Reproduction steps

systemctl stop nomad stops the running executors on the host

then if i do a systemctl start nomad, the reattachment attempt kills the existing containers as the corresponding executors are dead

[ramSeraph]

adding KillMode=process to the systemd unit file below fixes the problem

https://github.com/hashicorp/nomad/blob/master/dist/systemd/nomad.service

documentation reference: https://www.freedesktop.org/software/systemd/man/systemd.kill.html#KillMode=

[preetapan]

@ramSeraph Could you submit a PR for changing that sample config file? - this default makes sense

[insanejudge]

I considered doing the same (we're running Killmode=process) but this should have been addressed by #3572 , which places executors in their own process group.

edit: on a second read, i might be wrong about that.

[ramSeraph]

@insanejudge thanks for the PR. I was offline and didn't see the response

[onlyjob]

FYI, #4305 (comment)

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.