Restarting nomad can lead to orphaned containers

[kelseyhightower]

After restarting nomad I get the following errors in the logs and nomad fails to start.

Sep 29 14:40:18 nomad0.c.hashiconf.internal systemd[1]: Stopped Nomad.
Sep 29 14:40:18 nomad0.c.hashiconf.internal systemd[1]: Started Nomad.
Sep 29 14:40:18 nomad0.c.hashiconf.internal nomad[2972]: ==> Starting Nomad agent...
Sep 29 14:40:18 nomad0.c.hashiconf.internal nomad[2972]: ==> Error starting agent: client setup failed: failed to restore state: 1 error(s) occurred:
Sep 29 14:40:18 nomad0.c.hashiconf.internal nomad[2972]: * 2 error(s) occurred:
Sep 29 14:40:18 nomad0.c.hashiconf.internal nomad[2972]: * Failed to find container 54b0a899668b8e009fa902cd0474cfdaff8e1a19faad5b1b7f0149b0a692159d: <nil>
Sep 29 14:40:18 nomad0.c.hashiconf.internal nomad[2972]: * Failed to find container b2029c3e2fc6426889d5f64116511ca937aa5bf07079c357ddc1d03193cf2a7d: <nil>
Sep 29 14:40:18 nomad0.c.hashiconf.internal systemd[1]: nomad.service: Main process exited, code=exited, status=1/FAILURE
Sep 29 14:40:18 nomad0.c.hashiconf.internal systemd[1]: nomad.service: Unit entered failed state.
Sep 29 14:40:18 nomad0.c.hashiconf.internal systemd[1]: nomad.service: Failed with result 'exit-code'.

[LordFPL]

For information i ask for a similar pb... and it can be worse :
https://groups.google.com/forum/#!topic/nomad-tool/Hm7BhF0iCIE

[ghost]

+1 This prevents the service from restarting with any options using the original data-dir.

[dadgar]

This should be fixed now.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.