Kubectl in ubuntu20-20220503.1 breaks EKS integration

[shallax]

Description

When trying to use kubectl to communicate with AWS EKS, it reports the following:

error: exec plugin: invalid apiVersion "client.authentication.k8s.io/v1alpha1"

I can't follow the Kubectl version numbers. The previous image used Kubectl v1.23.6, this new one uses Kubectl 4.5.4?

Issue is similar to this: https://www.reddit.com/r/k8s/comments/ui46zf/deploying_on_eks_cia_travis_ci_fails/

Virtual environments affected

• Ubuntu 18.04
• Ubuntu 20.04
• Ubuntu 22.04
• macOS 10.15
• macOS 11
• macOS 12
• Windows Server 2019
• Windows Server 2022

Image version and build link

No public example

Is it regression?

No response

Expected behavior

Kubectl authentication should work.

Actual behavior

Kubectl authentication does not work.

Repro steps

Try using kubectl to communicate with an EKS cluster in any way.

[al-cheb]

We have updated kubectl version to 1.24.0. Looks like it is a known issue - kubernetes/kubectl#1210

https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.24.md#deprecation

The client.authentication.k8s.io/v1alpha1 ExecCredential has been removed. If you are using a client-go credential plugin that relies on the v1alpha1 API please contact the distributor of your plugin for instructions on how to migrate to the v1 API. (kubernetes/kubernetes#108616, @margocrawf)

As a workaround you can rollback to 1.23.6:

sudo curl -o /usr/local/bin/kubectl -LO "https://dl.k8s.io/release/v1.23.6/bin/linux/amd64/kubectl"
sudo chmod +x /usr/local/bin/kubectl

or use task:

       - uses: azure/setup-kubectl@v2.0
          with:
            version: 'v1.23.6'
          id: install

[shallax]

Thanks for the quick response. I went with the former set of sudo commands in ADO pipelines to fudge the version in the image and it worked.

Suspect this is going to cause a storm, so is there any chance of downgrading the version of Kubectl in the image or is there a way to explicitly request the previous tag of ubuntu-latest as part of the vm-image directive?

[bendotexe]

Great to see this being sorted so quickly! Is there an Azure DevOps Pipelines equivalent to that task?

[shallax]

A task would be nice. However, I've done the following for now:

/pipelines/jobs/downgrade_kubectl.yaml:

steps:
- script: |
    which kubectl
    kubectl version || true
    sudo curl -o /usr/local/bin/kubectl -LO "https://dl.k8s.io/release/v1.23.6/bin/linux/amd64/kubectl"
    sudo chmod +x /usr/local/bin/kubectl
    kubectl version || true
    exit 0
  displayName: Downgrade kubectl

then in each of your jobs (joy!):

- job: build_and_publish
  displayName: Build and publish
  dependsOn: ${{parameters.dependsOn}}
  steps:
  - template: /pipelines/jobs/downgrade_kubectl.yaml
....

[bendotexe]

An ADO supported root level spec would be best yeah but in the meantime that's going to unblock us by the looks of things so thanks!

[ahilmathew]

As a workaround you can rollback to 1.23.6:

sudo curl -o /usr/local/bin/kubectl -LO "https://dl.k8s.io/release/v1.23.6/bin/linux/amd64/kubectl"
sudo chmod +x /usr/local/bin/kubectl

Thanks for the workaround. Had pipeline failures out of nowhere.

[checor]

I can confirm that this also happens to Ubuntu 18.04.
I will wait some days and if there are still problems, I will apply the workaround. Kudos!

[joakimvivas]

Hi! I can confirm that in GitHub actions it is not working, we have the same issue with the "ubuntu-20.04" or the latest version. Many thanks!

[Voxelot]

Now that aws-cli has resolved the issue on their end, it seems like the problem is that the Github hosted runners are running on stale images.

[oxeye-schmil]

I've encountered the same issue with azure/setup-helm@v1
Is there a workaround for like the the one for azure/setup-kubectl@v2.0

[jake8787]

Seems like facing it for Windows19 here 20220503 however on paper it says Kubectl 1.23.6
I am using Azure and Packer to build an image. Is there a way to resolve it or new image 20220511 has everything sorted already?

[al-cheb]

Close as external - aws/aws-cli#6920