By any chance, did you forget to make k8s.gcr.io public?

[redbitshift]

/kind bug

What happened?
Tried to deploy latest EBS CSI driver, and my cluster could not pull images from k8s.gcr.io:

Failed to pull image "k8s.gcr.io/provider-aws/aws-ebs-csi-driver:latest": rpc error: code = Unknown desc = Error response from daemon: manifest for k8s.gcr.io/provider-aws/aws-ebs-csi-driver:latest not found: manifest unknown: Failed to fetch "latest" from request "/v2/provider-aws/aws-ebs-csi-driver/manifests/latest".

What you expected to happen?
I expected it to work.

How to reproduce it (as minimally and precisely as possible)?
I simply followed instructions on https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html , the step that causes the problem is:

kubectl apply -k "github.com/kubernetes-sigs/aws-ebs-csi-driver/deploy/kubernetes/overlays/stable/?ref=master"

Anything else we need to know?:
I noticed that a recent PR (merge pull request 642) replaced amazon/aws-ebs-csi-driver:latest to k8s.gcr.io/provider-aws/aws-ebs-csi-driver:latest, and k8s.gcr.io is the registry I seem to be unable to access.

Environment

• Kubernetes version (use kubectl version):
  Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:50:19Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"windows/amd64"}
  Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.15-eks-ad4801", GitCommit:"ad4801fd44fe0f125c8d13f1b1d4827e8884476d", GitTreeState:"clean", BuildDate:"2020-10-20T23:27:12Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

• Driver version: At the time of writing, the repo is at f541380 (Merge pull request 644)

[redbitshift]

Update. After cloning the repo locally and rolling back to e181446, the driver installed fine.

[michalpacholczyk]

Installation link does not work for me either, it cannot pull from k8s.gcr.io/provider-aws/aws-ebs-csi-driver:latest . According to read.me shouldn't it be amazon/aws-ebs-csi-driver:latest

[focaaby]

Got the same issue, and can not pull the image from k8s.gcr.io/provider-aws/aws-ebs-csi-driver:latest.

However, I check the change log v0.8.0[1]. This seems that the image did not push to k8s.gcr.io.

Images will be published to k8s.gcr.io in addition to ECR, GitHub, and Docker Hub.

[1] https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/CHANGELOG-0.x.md

[wongma7]

I did not push a latest tag, let me do that now. The workaround is to use the v0.8.0 tag which should be public

[wongma7]

Sorry for the disruption, I pushed latest tags. They are public from my end, i.e. I don't need to authenticate to gcr to pull them.

~ $ docker pull k8s.gcr.io/provider-aws/aws-ebs-csi-driver:latest
latest: Pulling from provider-aws/aws-ebs-csi-driver
Digest: sha256:a8ed0a90b49c16301f3ca934b75db63b8d5c8d68c377a865bce15c9af4a12ff5
Status: Downloaded newer image for k8s.gcr.io/provider-aws/aws-ebs-csi-driver:latest
k8s.gcr.io/provider-aws/aws-ebs-csi-driver:latest
~ $ docker pull k8s.gcr.io/provider-aws/aws-ebs-csi-driver:v0.8.0
v0.8.0: Pulling from provider-aws/aws-ebs-csi-driver
Digest: sha256:a8ed0a90b49c16301f3ca934b75db63b8d5c8d68c377a865bce15c9af4a12ff5
Status: Image is up to date for k8s.gcr.io/provider-aws/aws-ebs-csi-driver:v0.8.0
k8s.gcr.io/provider-aws/aws-ebs-csi-driver:v0.8.0

There are other issues like the stable kustomize overlay still trying to find/replace amazon/aws-ebs-csi-driver:latest to amazon/aws-ebs-csi-driver:v0.7.1, instead of k8s.gcr.io/provider-aws/aws-ebs-csi-driver:latest to k8s.gcr.io/provider-aws/aws-ebs-csi-driver:v0.8.0 , patch for taht is coming.

[wongma7]

The overlays have been updated and the stable overlay correctly point to v0.8.0 tag now from my side

[jaronoff97]

Seeing this happen again for v0.8.1, I assume the same change must be made?

[wongma7]

Yes, kubernetes/k8s.io#1509 has'nt merged yet.

Sorry, the release process is still new and a little backwards since maintainer has to make the tag first, then the image gets built, then must be promoted with a PR like above. Normally it should only take a few hours but the PR hasn't been reviewed yet.

For docker hub it isn't a problem since the github action immediately pushes the image after the tag is made.

In the future I will refrain from updating helm charts/kustomize templates until after the image is actually available. The trade off will be that the tag/release won't match the helm chart/kustomize template, i.e. if I clone v0.8.1 of this repo the template is still going to refer to v0.8.0, but I think that's not a huge deal

[wongma7]

It seems available now on my end. Can continue discussion in #685