Push Permission Error on latest

[zkanda]

Actual behavior
Permission error on pushing to google container registry.

Step #0: error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "eu.gcr.io/<project-id>/services/svc:master-xxxx": creating push check transport for eu.gcr.io failed: GET https://eu.gcr.io/v2/token?scope=repository%3A<project-id>%2Fservices%2Fsvc%3Apush%2Cpull&service=eu.gcr.io: UNAUTHORIZED: Not Authorized.
Finished Step #0
ERROR
ERROR: build step 0 "gcr.io/kaniko-project/executor:latest" failed: step exited with non-zero status: 1

Expected behavior
It should successfully push, I reverted to 1.6.0 and it works great.

To Reproduce
Make a google build with this yaml:

steps:
- name: 'gcr.io/kaniko-project/executor:latest'
  args:
  - --destination=eu.gcr.io/$PROJECT_ID/services/svc:$BRANCH_NAME-$SHORT_SHA
  - --dockerfile=svc/Dockerfile
  - --cache=true
  - --cache-ttl=168h
  - --cache-repo=eu.gcr.io/$PROJECT_ID/cache
  - --snapshotMode=redo
  - --use-new-run

Additional Information

• Dockerfile
  Any dockerfile will reproduce the error.
• Kaniko Image (fully qualified with digest)
  gcr.io/kaniko-project/executor:latest
  digest: 8504bde9a9a8c9c4e9a4fe659703d265697a36ff13607b7669a4caa4407baa52

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile

[yuiseki]

Same here.

My environment is:

• Google Cloud Build
• Google Container Registry

I've confirmed when I specify gcr.io/kaniko-project/executor:v1.6.0, it works expectedly.

[jonbuffington]

My environment is also:

• Google Cloud Build
• Google Container Registry

If I specify gcr.io/kaniko-project/executor:edge, it also works.

[rodosskiand]

+1

• Google Container Registry
• debug image

[chandanpasunoori]

Same here.

My environment is:

Google Cloud Build
Google Container Registry

I've confirmed when I specify gcr.io/kaniko-project/executor:v1.6.0, it works expectedly.

[nielsbox]

Same

My environment also is:

• Google Cloud Build
• Google Container Registry

[nirav-chotai]

gcr.io/kaniko-project/executor:debug

This is also broken, cannot push the images to GCR

[fr0stylo]

Same here:
Environnment:

• GKE
• GCR
• debug 1.7 image

Downgrading to 1.6 works

[slahs3r]

Same here.

My environment is:

• Google Cloud Build
• Google Container Registry

This workaround works for me:

gcloud config set builds/kaniko_image gcr.io/kaniko-project/executor:v1.6.0

[chrisapplegate]

Getting the same issue. My environment is:

• Google Cloud Build
• Google Container Registry

Changing the container reference in my cloudbuild.yaml to gcr.io/kaniko-project/executor:v1.6.0 solves the issue, I have found

[Retsuki]

Could this fix be the cause?
#1471

[plechi]

In reference to the comment by @jonbuffington
#1786 (comment)

My environment is also:

• Google Cloud Build
• Google Container Registry

If I specify gcr.io/kaniko-project/executor:edge, it also works.

Keep in mind that this image is from May 7, 2020. I think it's better to use the v1.6.0 tag until this is fixed.

[stijntratsaertit]

Same issue, Google Cloud Build & Google Container Registry here

[jonbuffington]

In reference to the comment by @jonbuffington #1786 (comment)

My environment is also:

• Google Cloud Build
• Google Container Registry

If I specify gcr.io/kaniko-project/executor:edge, it also works.

Keep in mind that this image is from May 7, 2020. I think it's better to use the v1.6.0 tag until this is fixed.

RIght. Thanks!

[amichal]

Diffing v1.6.0 and v1.7.0 i noticed a change to conditionally loading a credential helper here v1.6.0...v1.7.0#diff-4c345264209a6b2e4584f42b5fc96ef58595aef6dbb3b342e02aa723ce7323d7R68-R81 I know zero about this codebase so probably not reading the code right but seems maybe related

[robertlindner]

Same:

• Push to GCR check fails
• gitlab.com runner (on gke)
• GOOGLE_APPLICATION_CREDENTIALS pointing to service account key file for gcr auth
• I have additional credentials for the gitlab registry in /kaniko/.docker/config.json

[dakl]

Same problem. Pinning 1.6.0 ( - name: "gcr.io/kaniko-project/executor:v1.6.0" in cloudbuild.yaml) works fine.

[jdziek]

Same problem here. Exactly when i started learning CI/CD with kaniko so naturally i assumed that the error was on my part. Anyone has any alternative to using gcr.io/kaniko-project/executor:debug with GCP? Im very new to that.

EDIT: For anyone using debug version use for now "gcr.io/kaniko-project/executor:v1.6.0-debug" instead of "gcr.io/kaniko-project/executor:debug". At leat that did the trick for me.
EDIT2: At least partially. There are still some issues with it but definetly better than not passing at all.

[krishnarajan-acumenTec]

myself also tried unable to push image to gcr

error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "gcr.io/[MASKED]/[MASKED]:2.0.10": creating push check transport for gcr.io failed: GET https://gcr.io/v2/token?scope=repository%3A[MASKED]%2Flb4b-mi%2F[MASKED]%3Apush%2Cpull&service=gcr.io: UNAUTHORIZED: Not Authorized.

[ferrastas]

I managed to fix it by explicitly calling the docker-credential-gcr helper before the build.

docker-credential-gcr config --token-source=env
docker-credential-gcr configure-docker --registries=[your-registry-path]

Where registry-path is the root path for the GCP Container Registry like gcr.io/$PROJECT_ID

[imjasonh]

I did some digging in #702 (comment) (which probably should have been in this issue 🤦‍♂️), but it sounds like the docker-credential-gcr config --token-source=env was removed from deploy/Dockerfile between v1.6.0 and v1.7.0, which might be the cause.

[Paraplegix]

Same error using Gitlab CI with gitlab runner (docker executor)
I tryed including ferratas fix (explicitly calling docker-credential-gcr) but it's still refusing to run.
I saw to variation of the error during testing, they are similar but different (I am not sure what the difference is) :

UNAUTHORIZED: Not Authorized.

and

UNAUTHORIZED: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication

I also tryed regenerating the service acount key (the json file), giving our service account Editor role just in case but it didn't help
We are reverting back to v1.6.0-debug allow us to have build working

[dinvlad]

Yea this is not about permissions, unfortunately - it's just that Kaniko no longer generates the authentication token from env credentials, so it's a 401 error not 403..

[dogomedia-github]

+1 Same as everyone else

My environment is:

Google Cloud Build
Google Container Registry

Downgrading to 1.6.0 fixes the error, but this is not a viable work around. Updating hundreds of projects to use 1.6.0 is hardly a solution. Why was this change made without proper QA testing to make sure it doesn't break anything?

[joeholley]

+1, although I'm using Google Artifact Registry as my target from Cloud Build, not Container Registry.

[briandealwis]

We’ve rolled back the Kaniko images to point to v1.6.0.

[dinvlad]

@briandealwis thanks! Btw are there any regression/integration tests that could prevent this kind of issue from cropping up again? It would be really helpful to test Kaniko in all of the supported environments, if you have time to implement that :-)

[zkanda]

The latest version now works for me for a few days. Closing this.

[gsabbih6]

It does not work, was this resolved. ! have same issue, tried every build of kaniko

[inimaz]

Same issue here, we are using the debug tag.