Support docker engine 25.x #4075

sparrc · 2024-01-19T22:11:13Z

Summary

Since the release of docker engine 25.x (Jan 19 2024), the docker engine no longer supports API versions less than 1.24. See deprecated API versions here: https://docs.docker.com/engine/api/#deprecated-api-versions

This issue would show up in the agent logs like:

Error response from daemon: client version 1.21 is too old. Minimum supported API version is 1.24, please upgrade your client to a newer version

This PR allows the agent to dynamically determine the minimum/default API version that it should use based on what versions the docker engine on the instance supports.
This PR also updates the logic around advertising capabilities to the ECS backend. We will now advertise all known API versions that we are compatible with, rather than using the response from the docker API. This is for the sake of task placement requiredAttributes, which are static strings like docker-remote-api.1.19. Since some tasks will have these attributes added to them, we need to advertise that we support this remote API version even though technically it is deprecated and we no longer support it. Although we don't support using the specific API version, we still should be able to support all functionality from that version in the minimum docker API version.
When the ECS backend specifies a docker API version in it's container payload (via the DockerConfig.Version field on the container struct) we might use a different API version to actually create and start the container if the specified version is less than the minimum API version. For example, if the ECS backend specifies using docker API version 1.19 when we are using Docker engine 25, we will create/start that container using API 1.24.

fixes aws/containers-roadmap#2267
fixes #4074

NOTE: Our minimum docker engine version supported is moving from 1.9.0 to 1.12.0, which maps to api version 1.24. Docker API versions prior to 1.24 are deprecated as of Jan 2024. See https://docs.docker.com/engine/api/#deprecated-api-versions.

~~Docker version 1.12 was released in 2016 and is not supported in Amazon Linux.~~

Details

After this change, we will check all versions of the Docker API on startup, and then set agent's minimum/default version to whatever is the minimum supported. So when starting up on Docker 25.x we see log messages like this appear:

deprecated api versions fail (info-level messages)

level=info time=2024-01-25T23:25:00Z msg="Unable to get Docker client for version 1.17: Error response from daemon: client version 1.17 is too old. Minimum supported API version is 1.24, please upgrade your client to a newer version" module=sdkclientfactory.go
level=info time=2024-01-25T23:25:00Z msg="Unable to get Docker client for version 1.18: Error response from daemon: client version 1.18 is too old. Minimum supported API version is 1.24, please upgrade your client to a newer version" module=sdkclientfactory.go
level=info time=2024-01-25T23:25:00Z msg="Unable to get Docker client for version 1.19: Error response from daemon: client version 1.19 is too old. Minimum supported API version is 1.24, please upgrade your client to a newer version" module=sdkclientfactory.go
level=info time=2024-01-25T23:25:00Z msg="Unable to get Docker client for version 1.20: Error response from daemon: client version 1.20 is too old. Minimum supported API version is 1.24, please upgrade your client to a newer version" module=sdkclientfactory.go
level=info time=2024-01-25T23:25:00Z msg="Unable to get Docker client for version 1.21: Error response from daemon: client version 1.21 is too old. Minimum supported API version is 1.24, please upgrade your client to a newer version" module=sdkclientfactory.go
level=info time=2024-01-25T23:25:00Z msg="Unable to get Docker client for version 1.22: Error response from daemon: client version 1.22 is too old. Minimum supported API version is 1.24, please upgrade your client to a newer version" module=sdkclientfactory.go
level=info time=2024-01-25T23:25:00Z msg="Unable to get Docker client for version 1.23: Error response from daemon: client version 1.23 is too old. Minimum supported API version is 1.24, please upgrade your client to a newer version" module=sdkclientfactory.go

minimum api version is bumped from 1.21 to 1.24:

level=info time=2024-01-25T23:25:00Z msg="Setting minimum docker API version" newMinAPIVersion=1.24 previousMinAPIVersion=1.21

while 1.17-1.23 are not directly supported, they are supported in "extended" versions in order to advertise to the ECS backend that we support features added in these versions (such as awslogs added in API version 1.19):

level=debug time=2024-01-25T23:25:00Z msg="Extended supported versions" supportedVersions=[1.17 1.18 1.19 1.20 1.21 1.22 1.23 1.24 1.25 1.26 1.27 1.28 1.29 1.30 1.31 1.32 1.33 1.34 1.35 1.36 1.37 1.38 1.39 1.40 1.41 1.42 1.43 1.44]

Testing

unit and functional tests

Description for the changelog

Bug: Fix ECS agent docker API failures on docker engine 25.x

Licensing

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

agent/dockerclient/sdkclientfactory/versionsupport_windows.go

agent/dockerclient/logging_drivers.go

agent/app/agent.go

agent/app/agent_capability.go

yinyic · 2024-01-26T21:13:37Z

agent/app/agent.go

@@ -1097,26 +1097,24 @@ func (agent *ecsAgent) startACSSession(
 	return exitcodes.ExitSuccess
 }

-// validateRequiredVersion validates docker version.
+// verifyRequiredDockerVersion validates docker version.
 // Minimum docker version supported is 1.9.0, maps to api version 1.21


Nit - should probably update this

This minimum version has actually not changed on Linux, so we should be good to leave this here

- adds support for docker engine 25+, which has deprecated docker API versions less than 1.24. - dynamically determines which docker versions are available and what to use to call docker API, as well as advertising correct version support capabilities to the ECS backend

raphael5178 · 2024-01-30T22:25:06Z

Any prediction on when a new version with this fix will be released?

iforapsy · 2024-01-31T00:06:14Z

@raphael5178 Please see comment 1906827813.

sparrc · 2024-02-07T19:48:03Z

@raphael5178 sorry missed updating this PR, this has now been released in agent version 1.81.0

raphael5178 · 2024-02-09T13:58:33Z

@sparrc no problem my friend, thanks :)

19h · 2024-06-05T13:18:08Z

This "fix" broke the ECS agent for the entire cluster when using Flatcar Container Linux running Docker 24 on all stable, beta and alpha update trains; we have since downgraded the agent to an explicit 1.80.0 version until further notice.

Can you please lmk if there's a workaround to the following issue arising when running any ECS agent version gte 1.81.0:

level=info time=2024-06-05T13:16:22Z msg="Unable to get Docker client for version 1.44: Error response from daemon: client version 1.44 is too new. Maximum supported API version is 1.43" module=sdkclientfactory.go

I can replicate this on all build trains for flatcar; I don't think the container is respecting any API-version related env vars as we tried 1) configuring them on a systemd level, 2) on a docker level, 3) on the ecs.config level; neither worked and left the cluster in an unoperable state.

Please advise asap.

sparrc requested a review from a team as a code owner January 19, 2024 22:11

sparrc force-pushed the update-docker-api-version branch 12 times, most recently from 8b2bb71 to 9076d31 Compare January 20, 2024 00:08

sparrc changed the title ~~[wip] Bump min/default docker API version to 1.24~~ Support docker engine 25.x Jan 20, 2024

sparrc added the bot/test label Jan 20, 2024

amazon-ecs-bot removed the bot/test label Jan 20, 2024

sparrc added the bot/test label Jan 20, 2024

amazon-ecs-bot removed the bot/test label Jan 20, 2024

Yiyuanzzz previously approved these changes Jan 20, 2024

View reviewed changes

sparrc added the bot/test label Jan 20, 2024

amazon-ecs-bot removed the bot/test label Jan 20, 2024

shanet mentioned this pull request Jan 20, 2024

Incompatible with Docker v25 (API 1.44) #4074

Closed

amogh09 reviewed Jan 22, 2024

View reviewed changes

fierlion reviewed Jan 22, 2024

View reviewed changes

agent/app/agent_capability.go Outdated Show resolved Hide resolved

sparrc dismissed Yiyuanzzz’s stale review via 3f8c8c1 January 22, 2024 19:40

sparrc force-pushed the update-docker-api-version branch 5 times, most recently from 712745e to 2e6eb57 Compare January 22, 2024 20:24

amazon-ecs-bot removed the bot/test label Jan 25, 2024

sparrc force-pushed the update-docker-api-version branch from 763e5ab to 8a8fb1f Compare January 26, 2024 00:49

sparrc added the bot/test label Jan 26, 2024

amazon-ecs-bot removed the bot/test label Jan 26, 2024

sparrc added the bot/test label Jan 26, 2024

amazon-ecs-bot removed the bot/test label Jan 26, 2024

sparrc force-pushed the update-docker-api-version branch 3 times, most recently from c36132a to 62708fd Compare January 26, 2024 06:18

sparrc added the bot/test label Jan 26, 2024

amazon-ecs-bot removed the bot/test label Jan 26, 2024

yinyic previously approved these changes Jan 26, 2024

View reviewed changes

mye956 previously approved these changes Jan 26, 2024

View reviewed changes

sparrc dismissed stale reviews from mye956 and yinyic via b2c24a0 January 26, 2024 23:23

sparrc force-pushed the update-docker-api-version branch from 62708fd to b2c24a0 Compare January 26, 2024 23:23

mye956 approved these changes Jan 26, 2024

View reviewed changes

sparrc added the bot/test label Jan 26, 2024

amazon-ecs-bot removed the bot/test label Jan 26, 2024

yinyic approved these changes Jan 27, 2024

View reviewed changes

sparrc merged commit 31fb7f5 into aws:dev Jan 28, 2024
38 checks passed

sparrc deleted the update-docker-api-version branch January 28, 2024 07:09

chienhanlin mentioned this pull request Jan 31, 2024

Release 1.81.0 #4083

Merged

sparrc mentioned this pull request Mar 14, 2024

ecs-init support for old docker engines (pre docker 17.x) and future docker engines (when api 1.25 is deprecated). #4080

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support docker engine 25.x #4075

Support docker engine 25.x #4075

sparrc commented Jan 19, 2024 •

edited

Loading

yinyic Jan 26, 2024

sparrc Jan 26, 2024

raphael5178 commented Jan 30, 2024

iforapsy commented Jan 31, 2024

sparrc commented Feb 7, 2024

raphael5178 commented Feb 9, 2024

19h commented Jun 5, 2024

Support docker engine 25.x #4075

Support docker engine 25.x #4075

Conversation

sparrc commented Jan 19, 2024 • edited Loading

Summary

Details

Testing

Description for the changelog

Licensing

yinyic Jan 26, 2024

Choose a reason for hiding this comment

sparrc Jan 26, 2024

Choose a reason for hiding this comment

raphael5178 commented Jan 30, 2024

iforapsy commented Jan 31, 2024

sparrc commented Feb 7, 2024

raphael5178 commented Feb 9, 2024

19h commented Jun 5, 2024

sparrc commented Jan 19, 2024 •

edited

Loading