CLI versions 1.20.9 and 2.2.24 shipped a breaking change in a dot release

[dylanlingelbach]

Confirm by changing [ ] to [x] below to ensure that it's a bug:

• I've gone though the User Guide and the API reference
• I've searched for previous similar issues and didn't find any solution

Describe the bug
Versions 1.20.9 and 2.2.24 shipped a breaking change to EKS support by changing the supported k8s API value from v1alpha1 to v1beta1. This breaks any existing kubeconfigs that have v1alpha1 specified in their kubeconfig.

The PR that changed this behavior clearly specified this is a breaking change, however the changed shipped in a point release, with no clear communication about it being a breaking change.

The error message you see when attempting to use a kubeconfig with an incorrect version is:

unable to use kubectl with the EKS cluster (check 'kubectl version'): Unable to connect to the server: getting credentials: exec plugin is configured to use API version client.authentication.k8s.io/v1alpha1, plugin returned version client.authentication.k8s.io/v1beta1

SDK version number
1.20.9
2.2.24

Platform/OS/Hardware/Device
What are you running the cli on?

Linux/Mac OS X

To Reproduce (observed behavior)

1. Have a kubeconfig that uses eks get-token for authentication
2. Upgrade CLI to 1.20.9 or 2.2.24
3. Run a kubectl command

Expected behavior
Point releases of CLIs to not include breaking changes

Logs/output

unable to use kubectl with the EKS cluster (check 'kubectl version'): Unable to connect to the server: getting credentials: exec plugin is configured to use API version client.authentication.k8s.io/v1alpha1, plugin returned version client.authentication.k8s.io/v1beta1

Additional context
Related to aws-samples/eks-workshop#1232

[kdaily]

Hi @dylanlingelbach,

Thanks for opening this issue. This change breaks kubectl when upgrading the AWS CLI but not running aws eks update-kubeconfig, and this was unintentional.

We recommend downgrading the AWS CLI to 1.20.8 or 2.2.23. We are going to roll back this change (pull request #6309).

We would not recommend mitigating by updating the kubeconfig or re-running the aws eks update-kubeconfig with the affected versions of the AWS CLI until we release the versions 1.20.12 and 2.2.25. We will immediately release AWS CLI v2. We are aiming to release the AWS CLI v1 on Monday, August 2, 2021.

[gabegorelick]

We would not recommend mitigating by updating the kubeconfig or re-running the aws eks update-kubeconfig with the affected versions of the AWS CLI until we release the versions 1.20.12 and 2.2.25.

What do you recommend for people that have already worked around this issue by updating their kubeconfigs to v1beta1? Will there be a way to force aws eks get-token to return v1beta1?

[kyleknap]

@gabegorelick

What do you recommend for people that have already worked around this issue by updating their kubeconfigs to v1beta1?

I’d say pin to the current version of the AWS CLI you are on if you do not want to update your kubeconfig again. Otherwise, when 1.20.12 and 2.2.25 are released, make sure to rerun the update-kubeconfig command after you upgrade the AWS CLI.

Will there be a way to force aws eks get-token to return v1beta1?

Not in the 1.20.12 and 2.2.25 releases. These releases are stopgaps to prevent essentially every user who has preexisting kubeconfig file that upgrades the AWS CLI from having to troubleshoot this issue and figure out that they need to either rerun their update-kubeconfig command or manually update their kubeconfig file. We are sorry that this does cause churn for people who have already handled this update, but we suspect that a majority of users have not ran into this issue yet since this is a recent change.

As a fast follow up to these releases, we are considering adding options to opt into a particular API version (e.g. by introducing --api-version flags to both the update-kubeconfig and get-token commands) so users can 1) Lock to an API version without having to worry about it changing from under them. 2) Unblock users who want to use the v1beta1 API version.

[kyleknap]

Version 2.2.25 is now available, which fixes this issue by reverting the upgrade to the v1beta1 API version.

[joshsleeper]

hey @kyleknap I always feel guilty bumping stuff, but is there any progress or tracking issue on those --api-version opts you said would be considered?
we were in the apparently-too-quick-to-update camp across nearly 100 repos and we've been pinned to 2.2.24 since 😆

[bigspawn]

Hey, don't forget to rollback aws-iam-authenticator because it also has an impact on it. I use v0.5.3

[stevehipwell]

I've opened #6916 to request adding the --api-version flag to aws eks get-token.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.