How to run with CredentialProvider in my EKS cluster?

[czhczh0123]

Environment:

• AWS Region: cn-northwest-1
• Instance Type(s): t3.medium
• EKS Platform version (use aws eks describe-cluster --name <name> --query cluster.platformVersion): eks.6
• Kubernetes version (use aws eks describe-cluster --name <name> --query cluster.version): 1.23
• AMI Version: ami-030d7615436dd9131
• Kernel (e.g. uname -a): Linux ip-10-0-0-166.cn-northwest-1.compute.internal 5.4.231-137.341.amzn2.x86_64#1 SMP Tue Feb 14 21:50:55 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
• Release information (run cat /etc/eks/release on a node):

BUILD_TIME="Sat Mar  4 23:57:55 UTC 2023"
BUILD_KERNEL="5.4.231-137.341.amzn2.x86_64"
ARCH="x86_64"

What happened:
tried to use CredentialProviderConfig in EKS 1.23 cluster in CN region. The result is I can pull image from my ecr repo, but found the following message in kubelet logs.

Apr 22 15:22:39 ip-10-0-0-89.cn-northwest-1.compute.internal kubelet[12449]: I0422 15:22:39.623119   12449 provider.go:102] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 22 15:22:39 ip-10-0-0-89.cn-northwest-1.compute.internal kubelet[12449]: E0422 15:22:39.749586   12449 plugin.go:233] Failed getting credential from external registry credential provider: error execing credential provider plugin ecr-credential-provider for image 3228xxxxxxxx.dkr.ecr.cn-northwest-1.amazonaws.com.cn/publicnginx: exit status 1

What you expected to happen:
We expected kubelet to run with "ecr-credential-provider" CredentialProvider.

Environment:
EKS cluster version 1.23
node image: ami-030d7615436dd9131

[root@ip-10-0-0-166 ~]# cat /etc/eks/ecr-credential-provider/ecr-credential-provider-config
apiVersion: kubelet.config.k8s.io/v1alpha1
kind: CredentialProviderConfig
providers:
  - name: ecr-credential-provider
    matchImages:
      - "*.dkr.ecr.*.amazonaws.com"
      - "*.dkr.ecr.*.amazonaws.cn"
      - "*.dkr.ecr-fips.*.amazonaws.com"
      - "*.dkr.ecr.us-iso-east-1.c2s.ic.gov"
      - "*.dkr.ecr.us-isob-east-1.sc2s.sgov.gov"
    defaultCacheDuration: "12h"
    apiVersion: credentialprovider.kubelet.k8s.io/v1alpha1
    args:
      - get-credentials

This is the default configuration file. Because my EKS cluster was launched in China region, so I added the following provider.
- "*.dkr.ecr.*.amazonaws.com.cn"

When I ran a new pod with the container image from my ECR repo, the pod could be launched without any issue.
But checking the kubelet log, I could find the logs above.

Actually, if I did not change the default config file, I will not see the second message related to my ecr repo credentials.

Anything else we need to know?:
I thought it might related to api version, and then I switch to EKS 1.26 cluster.
The apiversion in the default config file is kubelet.config.k8s.io/v1beta1 and credentialprovider.kubelet.k8s.io/v1beta1, and when I tested with a pod with ECR image, the result is identical.
node image: ami-028547407941241ec

[maximmold]

I'm seeing the same thing. Everything seems to be pulling fine despite the errors.

[cartermckinnon]

The kubelet had an in-tree ECR credential provider that was removed in 1.27. On previous Kubernetes versions, issues with the external credential provider do not result in failures, because the in-tree provider will be used as a fallback (but you'll see these warnings in the logs).

We've fixed a configuration issue with the external credential provider that will be included in the next AMI release (#1269). This will make the warnings go away.

[cartermckinnon]

However, before 1.27 launches; we do need to ensure that the external credential provider properly matches ECR patterns for China regions; thanks for pointing our that issue @czhczh0123 .

[czhczh0123]

Hi, @cartermckinnon ,

Thank you for your response. Sorry I didn't reply in time.

But I am still confused of this message.
Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider

With in-tree ECR credential provider, kubelet will not auto-refresh the ECR token, right?
Because I can only view the GetAuthorizationToken API once via CloudTrail when I launched a pod with my ECR image.
After 12 hours, I did not see another GetAuthorizationToken API unless I pull another image for my repo.

Besides, I would like to ask whether kubelet will help refresh the token when we enable external credential provider in 1.27.

Thanks a lot.

[cartermckinnon]

Kubelet will only refresh credentials if it needs to -- i.e. you actually pull an image that matches the credential provider's patterns. If you're not pulling images, it's not going to refresh the tokens.