StartContainer edge case

[petderek]

Summary

We've observed that occasionally, go-dockerclient will return an EOF instead of a complete response from Docker. Our StartContainer logic does not account for this, and the task gets marked as 'failed to start' when in reality the containers may be running. Agent should be modified to check for this failure state.

Expected Behavior

Either:

1. Containers do not start, task gets marked as failed.

or

2. Containers start, agent checks if they actually started, task marks as success.

Observed Behavior

• Docker API returns invalid response
• Docker daemon starts container
• Agent marks task as failed to start
• Container runs anyway, agent does not track or stop it

Supporting Log Snippets

Logs will contain messaging similar to:

error transitioning container [<container_name>] to [RUNNING]: Post http://unix.sock/v1.19/containers/<container_id>/start EOF 

[mfortin]

Faced this bug again today. Is there an ETA ?

[yhlee-aws]

Latest agent release (v1.24.0) has replaced go-dockerclient with Docker's official SDK. @mfortin is upgrading to the latest release an option for you?

[mfortin]

@yunhee-l It is. I will be testing it out starting Monday. Thanks!

[mfortin]

@yunhee-l I did. Now this is new:

CannotStartContainerError: error during connect: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.19/containers/eaeca3d1c68c067c0963a2a48785a65b1b5961acd9c8c6f72c7e316afef11442/start: EOF

[mfortin]

@yunhee-l I have grabbed the logs using ecs-logs-collector if you want to have a look.

[adnxn]

I have grabbed the logs using ecs-logs-collector if you want to have a look.

@mfortin: mind sending the logs to adnkha at amazon dot com?

[adnxn]

closing, released with version v1.32.1