Nomad does not handle container names conflict

[sugdyzhekov]

Nomad version

Nomad v0.6.3

Operating system and Environment details

CentOS Linux release 7.4.1708 (Core)

Docker:
Version: 17.09.0-ce
API version: 1.32 (minimum version 1.12)
Go version: go1.8.3
Git commit: afdb6d4
Built: Tue Sep 26 22:42:49 2017
OS/Arch: linux/amd64
Experimental: false

Issue

Nomad does not handle container names conflict and marks tasks as dead instead of migrating to another client.

1. Docker service was stopped due system updates.
2. Nomad tries restart task
3. Docker service starts, all exist containers are stopped
4. Nomad starts new containers
5. Docker returns error
6. Nomad marks allocation as failed and job as dead

Nomad Client logs

[ERR] driver.docker: Failed to create container: API error (400): {"message":"Conflict. The container name \"/docker-mirror-07cd78c9-b583-c8ff-4d3c-a6979b5c767a\" is already in use by container \"201e48a90828849d8c8dfd02e71365838303086a69047d062090d31bba021ec4\". You have to remove (or rename) that container to be able to reuse that name."}
[WARN] client: failed to start task "docker-mirror" for alloc "07cd78c9-b583-c8ff-4d3c-a6979b5c767a": Failed to create container: API error (400): {"message":"Conflict. The container name \"/docker-mirror-07cd78c9-b583-c8ff-4d3c-a6979b5c767a\" is already in use by container \"201e48a90828849d8c8dfd02e71365838303086a69047d062090d31bba021ec4\". You have to remove (or rename) that container to be able to reuse that name."}
[INFO] client: Not restarting task: docker-mirror for alloc: 07cd78c9-b583-c8ff-4d3c-a6979b5c767a
[INFO] client: marking allocation 07cd78c9-b583-c8ff-4d3c-a6979b5c767a for GC

Event log of allocation:

Recent Events:
Time                   Type            Description
10/19/17 04:22:32 MSK  Not Restarting  Error was unrecoverable
10/19/17 04:22:32 MSK  Driver Failure  failed to start task "docker-mirror" for alloc "07cd78c9-b583-c8ff-4d3c-a6979b5c767a": Failed to create container: API error (400): {"message":"Conflict. The container name \"/docker-mirror-07cd78c9-b583-c8ff-4d3c-a6979b5c767a\" is already in use by container \"201e48a90828849d8c8dfd02e71365838303086a69047d062090d31bba021ec4\". You have to remove (or rename) that container to be able to reuse that name."}
10/19/17 04:22:02 MSK  Restarting      Task restarting in 29.375756557s
10/19/17 04:22:02 MSK  Terminated      Exit Code: 2, Exit Message: "Docker container exited with non-zero exit code: 2"

Expected result

Nomad handles the issue and migrates task with allocation directory to another cluster node.

[sugdyzhekov]

Something similar is described in #629

[sugdyzhekov]

I suppose that docker driver should detect:

• unavailability of docker service
• issue with a container start

.. and marks node as "problem" with the following migration of all tasks to another node.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.