docker containers are not removed when status is dead

[g0t4]

Nomad version

v0.3.1

Operating system and Environment details

centos 7 3.10.0-327.10.1.el7.x86_64
Docker version 1.10.3, build 20f81dd

Issue

When I call nomad stop on a batch job, containers associated with dead allocations are not cleaned up. However, if the allocation is still running, the container is cleaned up.

Is it possible Kill is only called on running allocations and thus cleanup only happens for running containers?

Reproduction steps

1. Create a container that sleeps for 30 seconds (or long enough you can call nomad stop before it finishes)
2. Create job1 that runs this container as a batch type job.
   • Run job1 and let it complete, status should be dead.
   • take note of the node the container ran on
   • run nomad stop job1
   • inspect docker on the node that the container ran on, ssh into the node and run docker ps -a, the container will be there with a status of exited
3. Create job2 based on job1
   • Run job2 long enough for the container to start executing.
   • take note of the node the container is running on
   • before the job completes, call nomad stop job2
   • inspect docker on the node that the container ran on, ssh into the node and run docker ps -a, there won't be any containers, in this case the container is cleaned up.

[diptanu]

@g0t4 So Nomad does garbage collection of allocations some time after a job completes. The idea behind keeping the allocations after they have completed for some time is to allow users to inspect the allocation once they have finished in case they want to debug something.

I think the time is roughly four hours now. We are going to do more aggressive garbage collection on the client side when there is pressure for resources.

cc/ @dadgar Can you confirm?

[g0t4]

Ok yeah, I was wondering if it's GCd, can we add a setting to nuke them on
stop?

Can I run GC myself and force nomad to clean them up?

On Thursday, March 24, 2016, Diptanu Choudhury notifications@github.com
wrote:

@g0t4 https://github.com/g0t4 So Nomad does garbage collection of
allocations some time after a job completes. The idea behind keeping the
allocations after they have completed for some time is to allow users to
inspect the allocation once they have finished in case they want to debug
something.

I think the time is roughly four hours now. We are going to do more
aggressive garbage collection on the client side when there is pressure for
resources.

cc/ @dadgar https://github.com/dadgar Can you confirm?

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#973 (comment)

[dvusboy]

Running /v1/system/gc doesn't clean up exited containers. It looks like RemoveContainer() is invoked only if the task is killed, which I assume is aimed for services(?). For batch jobs, I think the exited containers just hang around, indefinitely. I'm looking forward to the fix in 0.3.2. Would I need to manually clean up the leftovers?

[g0t4]

I setup a cron job for now

On Wednesday, April 6, 2016, K.C. Wong notifications@github.com wrote:

Running /v1/system/gc doesn't clean up exited containers. It looks like
RemoveContainer() is invoked only if the task is killed
https://github.com/hashicorp/nomad/blob/master/client/driver/docker.go#L829,
which I assume is aimed for services(?). For batch jobs, I think the exited
containers just hang around, indefinitely. I'm looking forward to the fix
in 0.3.2. Would I need to manually clean up the leftovers?

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#973 (comment)

[diptanu]

This is going to be fixed for 0.3.2, some tests are failing so I haven't merged the branch with the fix. Few more days!

[g0t4]

Thanks for quickly getting to this

On Thu, Apr 7, 2016 at 12:44 AM, Diptanu Choudhury <notifications@github.com

wrote:

This is going to be fixed for 0.3.2, some tests are failing so I haven't
merged the branch with the fix. Few more days!

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#973 (comment)

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.