ECS-Agent image deletion stops if non-existant image encountered

[JimEarly]

Summary

The ECS-Agent image deletion / clean-up stops with an error if it encounters a reference to an image that was removed manually via Docker. It processes no other candidate images following this error, nor does it ignore the absent image during future clean-up cycles.

Description

The following command was issued on an EC2 container instance in response to low disk space:
docker rmi $(docker images -q)
This was intended to remove all images not currently associated with running or stopped containers.

However, the ECS-Agent now repeatedly looks for these absent images and will error out after not being able to locate the first one. Ex:

2019-02-19T20:08:34Z [INFO] Found 52 eligible images for deletion
2019-02-19T20:08:34Z [INFO] Removing Image: mycompany/goms-rest-staging:latest
2019-02-19T20:08:34Z [ERROR] Error removing Image mycompany/goms-rest-staging:latest - Error: No such image: mycompany/goms-rest-staging:latest

At this point, the clean-up stops, and no further image deletion is attempted. The result is a steadily increasing number of images that are not used by running containers.

Expected Behavior

Ideally, the ECS-Agent should recognize that the image is no longer available and continue to the next candidate image. In addition, it should remove the image from its tracking list such that it is not referenced again as a "candidate".

Another acceptable alternative would be to reset the agent's image tracking list such that it only tracked images related to managed containers on start-up

Observed Behavior

Environment Details

Supporting Log Snippets

[shubham2892]

Thanks for reporting. I have a PR #1897 in place for the fix.

[shubham2892]

The PR is merged and should be out in Agent version 1.26.0. Please reopen if you see the issue again after the release.