Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Clean up dead containers after ECS_ENGINE_TASK_CLEANUP_WAIT_DURATION #1684

Closed
yumex93 opened this issue Nov 9, 2018 · 3 comments
Closed

Clean up dead containers after ECS_ENGINE_TASK_CLEANUP_WAIT_DURATION #1684

yumex93 opened this issue Nov 9, 2018 · 3 comments
Labels
kind/feature request scope/ECS Agent

Comments

@yumex93
Copy link
Contributor

yumex93 commented Nov 9, 2018

Summary

Based on what I got from customers, so far after ECS_ENGINE_TASK_CLEANUP_WAIT_DURATION, agent cleans up only the stopped tasks and docker images that are not being used by any tasks on your container instances. They also want agent to clean up containers in 'dead' status.

Currently, customers' workaround is to run this command "#docker rm $(docker ps --all -q -f status=dead)" to manually remove the dead containers.
@fjromerom
Copy link

Also, include 'Created' status to cover those cases where Docker times out to create a container and it remains in 'Created' status, Example:

[...] created docker container for task: cron -> , took 4m0.000109703s
[...] Could not transition to created; timed out after waiting 4m0s
[...] error creating container [...]; marking its desired status as STOPPED: Could not transition to created; timed out after waiting 4m0s

This container will remain in 'Created' status.

sparrc added a commit to sparrc/amazon-ecs-agent that referenced this issue May 2, 2019
@sparrc
image manager: cleanup 'dead' and 'created' containers
f559483 
also cleanup of 'dangling' images that have no tags or names associated
with them (ie, they show as <none> in 'docker images')

see aws#1684
sparrc added a commit to sparrc/amazon-ecs-agent that referenced this issue May 2, 2019
@sparrc
image manager: cleanup 'dead' and 'created' containers
6ae8f4e 
also cleanup of 'dangling' images that have no tags or names associated
with them (ie, they show as <none> in 'docker images')

see aws#1684
@sparrc sparrc mentioned this issue May 2, 2019
Merged
sparrc added a commit to sparrc/amazon-ecs-agent that referenced this issue May 2, 2019
@sparrc
image manager: cleanup 'dead' and 'created' containers
be2d686 
also cleanup of 'dangling' images that have no tags or names associated
with them (ie, they show as <none> in 'docker images')

closes aws#1684

unit tests
@sparrc
Copy link
Contributor

sparrc commented May 2, 2019

While working on the solution to this I also noticed that we had a problem with cleaning up "dangling" docker images (docker images -f dangling=true). This was due to us trying to inspect images based on the name rather than the ID (which in the case of dangling images is <none>).

I have included a fix for this as well as part of #2015

sparrc added a commit to sparrc/amazon-ecs-agent that referenced this issue May 2, 2019
@sparrc
image manager: cleanup 'dead' and 'created' containers
0da8124 
also cleanup of 'dangling' images that have no tags or names associated
with them (ie, they show as <none> in 'docker images')

closes aws#1684

unit tests
sparrc added a commit to sparrc/amazon-ecs-agent that referenced this issue May 2, 2019
@sparrc
image manager: cleanup 'dead' and 'created' containers
99ceb3b 
also cleanup of 'dangling' images that have no tags or names associated
with them (ie, they show as <none> in 'docker images')

closes aws#1684

unit tests
sparrc added a commit to sparrc/amazon-ecs-agent that referenced this issue May 2, 2019
@sparrc
image manager: cleanup 'dead' and 'created' containers
ca2b7bb 
also cleanup of 'dangling' images that have no tags or names associated
with them (ie, they show as <none> in 'docker images')

closes aws#1684

unit tests
sparrc added a commit to sparrc/amazon-ecs-agent that referenced this issue May 3, 2019
@sparrc
image manager: cleanup 'dead' and 'created' containers
8372fef 
also cleanup of 'dangling' images that have no tags or names associated
with them (ie, they show as <none> in 'docker images')

closes aws#1684

unit tests

dont touch dangling images -- for now
sparrc added a commit to sparrc/amazon-ecs-agent that referenced this issue May 3, 2019
@sparrc
image manager: cleanup 'dead' and 'created' containers
06c59c3 
also cleanup of 'dangling' images that have no tags or names associated
with them (ie, they show as <none> in 'docker images')

closes aws#1684

unit tests

dont touch dangling images -- for now

skip containers that don't have a finished time
sparrc added a commit to sparrc/amazon-ecs-agent that referenced this issue May 3, 2019
@sparrc
image manager: cleanup 'dead' and 'created' containers
55c24df 
also cleanup of 'dangling' images that have no tags or names associated
with them (ie, they show as <none> in 'docker images')

closes aws#1684

unit tests

dont touch dangling images -- for now

skip containers that don't have a finished time
sparrc added a commit that referenced this issue May 3, 2019
@sparrc
image manager: cleanup 'dead' and 'created' containers (#2015)
9fb3226 
also cleanup of 'dangling' images that have no tags or names associated
with them (ie, they show as <none> in 'docker images')

closes #1684

unit tests

dont touch dangling images -- for now

skip containers that don't have a finished time
@sparrc
Copy link
Contributor

sparrc commented May 8, 2019

closed by #2015

@sparrc sparrc closed this as completed May 8, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/feature request scope/ECS Agent
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@sparrc @fjromerom @yumex93