ecs-init support for old docker engines (pre docker 17.x) and future docker engines (when api 1.25 is deprecated). #4080

sparrc · 2024-01-29T16:57:06Z

Summary

note: this is an ecs-init followup to a similar change already made to the agent codebase: #4075

Recently, docker engine has deprecated docker API versions 1.17-1.23: https://docs.docker.com/engine/deprecated/#deprecate-legacy-api-versions

In the above announcement, they state that the minimum supported API version (currently 1.24) would continue to be raised in future versions:

Support for API versions lower than 1.24 will be permanently removed in Docker Engine v26, and the minimum supported API version will be incrementally raised in releases following that.

This will soon be a problem for ecs-init, because currently ecs-init hardcodes it's API version to 1.25 and does not have any ability to detect or switch to supported versions.

Implementation details

This PR changes ecs-init to keep a list of known API versions rather than a single API version. When ecs-init starts up, it will retry creating the docker client until it finds an API version that works.

As part of this PR, we are both supporting future versions of docker and very old versions of docker which did not have support for API version 1.25.

For example on docker 1.10 ecs-init previously failed, but will now startup with messages like this (note that it fails on versions 1.25-1.44, then succeeds on 1.21):

Jan 29 17:34:46 ip-10-0-7-14 amazon-ecs-init[1281]: 2024-01-29T17:34:46Z [INFO] start
Jan 29 17:34:46 ip-10-0-7-14 amazon-ecs-init[1281]: 2024-01-29T17:34:46Z [INFO] Could not create docker client with API version 1.25: API error (400): client is newer than server (client API version: 1.25, server API version: 1.22)
Jan 29 17:34:46 ip-10-0-7-14 amazon-ecs-init[1281]: 2024-01-29T17:34:46Z [INFO] Could not create docker client with API version 1.26: API error (400): client is newer than server (client API version: 1.26, server API version: 1.22)
Jan 29 17:34:46 ip-10-0-7-14 amazon-ecs-init[1281]: 2024-01-29T17:34:46Z [INFO] Could not create docker client with API version 1.27: API error (400): client is newer than server (client API version: 1.27, server API version: 1.22)
Jan 29 17:34:46 ip-10-0-7-14 amazon-ecs-init[1281]: 2024-01-29T17:34:46Z [INFO] Could not create docker client with API version 1.28: API error (400): client is newer than server (client API version: 1.28, server API version: 1.22)
Jan 29 17:34:46 ip-10-0-7-14 amazon-ecs-init[1281]: 2024-01-29T17:34:46Z [INFO] Could not create docker client with API version 1.29: API error (400): client is newer than server (client API version: 1.29, server API version: 1.22)
Jan 29 17:34:46 ip-10-0-7-14 amazon-ecs-init[1281]: 2024-01-29T17:34:46Z [INFO] Could not create docker client with API version 1.30: API error (400): client is newer than server (client API version: 1.30, server API version: 1.22)
Jan 29 17:34:46 ip-10-0-7-14 amazon-ecs-init[1281]: 2024-01-29T17:34:46Z [INFO] Could not create docker client with API version 1.31: API error (400): client is newer than server (client API version: 1.31, server API version: 1.22)
Jan 29 17:34:46 ip-10-0-7-14 amazon-ecs-init[1281]: 2024-01-29T17:34:46Z [INFO] Could not create docker client with API version 1.32: API error (400): client is newer than server (client API version: 1.32, server API version: 1.22)
Jan 29 17:34:46 ip-10-0-7-14 amazon-ecs-init[1281]: 2024-01-29T17:34:46Z [INFO] Could not create docker client with API version 1.33: API error (400): client is newer than server (client API version: 1.33, server API version: 1.22)
Jan 29 17:34:46 ip-10-0-7-14 amazon-ecs-init[1281]: 2024-01-29T17:34:46Z [INFO] Could not create docker client with API version 1.34: API error (400): client is newer than server (client API version: 1.34, server API version: 1.22)
Jan 29 17:34:46 ip-10-0-7-14 amazon-ecs-init[1281]: 2024-01-29T17:34:46Z [INFO] Could not create docker client with API version 1.35: API error (400): client is newer than server (client API version: 1.35, server API version: 1.22)
Jan 29 17:34:46 ip-10-0-7-14 amazon-ecs-init[1281]: 2024-01-29T17:34:46Z [INFO] Could not create docker client with API version 1.36: API error (400): client is newer than server (client API version: 1.36, server API version: 1.22)
Jan 29 17:34:46 ip-10-0-7-14 amazon-ecs-init[1281]: 2024-01-29T17:34:46Z [INFO] Could not create docker client with API version 1.37: API error (400): client is newer than server (client API version: 1.37, server API version: 1.22)
Jan 29 17:34:46 ip-10-0-7-14 amazon-ecs-init[1281]: 2024-01-29T17:34:46Z [INFO] Could not create docker client with API version 1.38: API error (400): client is newer than server (client API version: 1.38, server API version: 1.22)
Jan 29 17:34:47 ip-10-0-7-14 amazon-ecs-init[1281]: 2024-01-29T17:34:46Z [INFO] Could not create docker client with API version 1.39: API error (400): client is newer than server (client API version: 1.39, server API version: 1.22)
Jan 29 17:34:47 ip-10-0-7-14 amazon-ecs-init[1281]: 2024-01-29T17:34:47Z [INFO] Could not create docker client with API version 1.40: API error (400): client is newer than server (client API version: 1.40, server API version: 1.22)
Jan 29 17:34:47 ip-10-0-7-14 amazon-ecs-init[1281]: 2024-01-29T17:34:47Z [INFO] Could not create docker client with API version 1.41: API error (400): client is newer than server (client API version: 1.41, server API version: 1.22)
Jan 29 17:34:47 ip-10-0-7-14 amazon-ecs-init[1281]: 2024-01-29T17:34:47Z [INFO] Could not create docker client with API version 1.42: API error (400): client is newer than server (client API version: 1.42, server API version: 1.22)
Jan 29 17:34:47 ip-10-0-7-14 amazon-ecs-init[1281]: 2024-01-29T17:34:47Z [INFO] Could not create docker client with API version 1.43: API error (400): client is newer than server (client API version: 1.43, server API version: 1.22)
Jan 29 17:34:47 ip-10-0-7-14 amazon-ecs-init[1281]: 2024-01-29T17:34:47Z [INFO] Could not create docker client with API version 1.44: API error (400): client is newer than server (client API version: 1.44, server API version: 1.22)
Jan 29 17:34:47 ip-10-0-7-14 amazon-ecs-init[1281]: 2024-01-29T17:34:47Z [INFO] Successfully created docker client with API version 1.21
Jan 29 17:34:47 ip-10-0-7-14 amazon-ecs-init[1281]: 2024-01-29T17:34:47Z [INFO] Container name: /ecs-agent
Jan 29 17:34:47 ip-10-0-7-14 amazon-ecs-init[1281]: 2024-01-29T17:34:47Z [INFO] Removing existing agent container ID: 257af52c52388e77eb75eb840df5f8264b52c8c561b61b50bca6b4075e8bbe12
Jan 29 17:34:47 ip-10-0-7-14 amazon-ecs-init[1281]: 2024-01-29T17:34:47Z [INFO] Starting Amazon Elastic Container Service Agent

Testing

New tests cover the changes: yes

new unit tests
functional testing suites
manually tested on ubuntu 16.04 using docker engine version 1.10

Description for the changelog

Enhancement: ecs-init support for old docker engines (pre docker 17.x) and future docker engines (when api 1.25 is deprecated).

Licensing

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

sparrc · 2024-01-29T21:43:13Z

ecs-init/docker/dependencies.go

@@ -157,6 +241,9 @@ func isNetworkError(err error) bool {
 func isRetryablePingError(err error) bool {
 	godockerError, ok := err.(*godocker.Error)
 	if ok {
+		if godockerError.Status == http.StatusBadRequest {


when provided with an unsupported docker api version, docker returns "Bad Request" (status code 400), so don't get into a retry loop if the API version is not supported (client errors in general probably shouldn't be retried at all, but just sticking with 400 for now to avoid changing behavior more than necessary).

sparrc · 2024-01-29T21:44:07Z

ecs-init/docker/docker_test.go

@@ -1,6 +1,3 @@
-//go:build test


these tests were not running because of these flags, so some had to be fixed as they havent been passing for a long time (or possibly not ever)

ecs-init/docker/dependencies_test.go

ecs-init/docker/dependencies.go

amogh09 · 2024-01-30T22:48:20Z

ecs-init/docker/dependencies.go

+func preferredAPIVersions() []string {
+	return []string{
+		"1.25",
+		"1.26",
+		"1.27",
+		"1.28",
+		"1.29",
+		"1.30",
+		"1.31",
+		"1.32",
+		"1.33",
+		"1.34",
+		"1.35",
+		"1.36",
+		"1.37",
+		"1.38",
+		"1.39",
+		"1.40",
+		"1.41",
+		"1.42",
+		"1.43",
+		"1.44",
+	}


Do we have a plan to maintain this list and "backup versions" list going forwards?

No plan in particular, this more just matches what we're doing in ecs-agent:

amazon-ecs-agent/agent/dockerclient/versions.go

Line 43 in 61c8a8c

func GetKnownAPIVersions() []DockerVersion {

So deprecation of older API versions would be the forcing function for us to update this list?

yes I guess deprecation of older versions, or potentially if a new ECS feature comes along that is only supported in a newer version of the docker API (though I think that's fairly unlikely at this point).

ecs-init/docker/dependencies_test.go

ecs-init/docker/docker_test.go

amogh09 · 2024-01-30T23:16:49Z

ecs-init/docker/dependencies.go

+func backupAPIVersions() []string {
+	return []string{
+		"1.21",
+		"1.22",
+		"1.23",
+		"1.24",
+	}
+}


IIUC before this change ecs-init was not compatible with environments with Docker engines that do not support API versions >= 1.25. But this PR makes ecs-init compatible with such environments. So, does that mean the older API versions were artificially made incompatible with ecs-init?

I understand that Task ENI is not supported on API versions < 1.25, but ecs-init doesn't actually make any Docker API calls that are not supported on API versions < 1.25?

ecs-init doesn't actually make any Docker API calls that are not supported on API versions < 1.25?

Not by default, but I will check on the various flags that ecs-init can add during startup to see if there are any that are not supported before docker 1.25.

…docker engines (when api 1.25 is deprecated).

sparrc requested a review from a team as a code owner January 29, 2024 16:57

sparrc force-pushed the dynamic-docker-api-ecs-init branch 12 times, most recently from 58be102 to d5398da Compare January 29, 2024 21:37

sparrc changed the title ~~[wip] ecs-init docker api version~~ ecs-init: support multiple docker API versions Jan 29, 2024

sparrc commented Jan 29, 2024

View reviewed changes

ecs-init/docker/dependencies_test.go Show resolved Hide resolved

sparrc added the bot/test label Jan 29, 2024

amazon-ecs-bot removed the bot/test label Jan 29, 2024

sparrc changed the title ~~ecs-init: support multiple docker API versions~~ ecs-init support for old docker engines (pre docker 17.x) and future docker engines (when api 1.25 is deprecated). Jan 30, 2024

amogh09 reviewed Jan 30, 2024

View reviewed changes

sparrc force-pushed the dynamic-docker-api-ecs-init branch from d5398da to 0061700 Compare February 26, 2024 22:51

sparrc added the bot/test label Feb 27, 2024

amazon-ecs-bot removed the bot/test label Feb 27, 2024

ecs-init support for old docker engines (pre docker 17.x) and future …

c60a86c

…docker engines (when api 1.25 is deprecated).

sparrc force-pushed the dynamic-docker-api-ecs-init branch from 0061700 to c60a86c Compare March 13, 2024 22:35

sparrc added the bot/test label Mar 13, 2024

amazon-ecs-bot removed the bot/test label Mar 13, 2024

sparrc added the bot/test label Mar 13, 2024

amazon-ecs-bot removed the bot/test label Mar 13, 2024

amogh09 approved these changes Mar 14, 2024

View reviewed changes

danehlim approved these changes Mar 15, 2024

View reviewed changes

sparrc merged commit f4f2411 into aws:dev Mar 15, 2024
40 checks passed

sparrc deleted the dynamic-docker-api-ecs-init branch March 15, 2024 18:21

mye956 mentioned this pull request Mar 19, 2024

Release 1.82.1 #4118

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ecs-init support for old docker engines (pre docker 17.x) and future docker engines (when api 1.25 is deprecated). #4080

ecs-init support for old docker engines (pre docker 17.x) and future docker engines (when api 1.25 is deprecated). #4080

sparrc commented Jan 29, 2024 •

edited

Loading

sparrc Jan 29, 2024 •

edited

Loading

sparrc Jan 29, 2024

amogh09 Jan 30, 2024

sparrc Jan 30, 2024

amogh09 Jan 31, 2024

sparrc Jan 31, 2024

amogh09 Jan 30, 2024

sparrc Jan 31, 2024

ecs-init support for old docker engines (pre docker 17.x) and future docker engines (when api 1.25 is deprecated). #4080

ecs-init support for old docker engines (pre docker 17.x) and future docker engines (when api 1.25 is deprecated). #4080

Conversation

sparrc commented Jan 29, 2024 • edited Loading

Summary

Implementation details

Testing

Description for the changelog

Licensing

sparrc Jan 29, 2024 • edited Loading

Choose a reason for hiding this comment

sparrc Jan 29, 2024

Choose a reason for hiding this comment

amogh09 Jan 30, 2024

Choose a reason for hiding this comment

sparrc Jan 30, 2024

Choose a reason for hiding this comment

amogh09 Jan 31, 2024

Choose a reason for hiding this comment

sparrc Jan 31, 2024

Choose a reason for hiding this comment

amogh09 Jan 30, 2024

Choose a reason for hiding this comment

sparrc Jan 31, 2024

Choose a reason for hiding this comment

sparrc commented Jan 29, 2024 •

edited

Loading

sparrc Jan 29, 2024 •

edited

Loading