Error on pushing image to GCR

[vasyan1992]

Actual behavior
Getting an error on kaniko with GCR after latest update (version 28a6bdc-debug
1.7.0-debug)

Expected behavior
image a0c96b4-debug is OK (ver 1.6.0-debug)

To Reproduce
printf $REGISTRY_ACCESSKEY > ${CI_PROJECT_DIR}/gcr.json
/kaniko/executor --context $CI_PROJECT_DIR --dockerfile ./Dockerfile --destination ${IMAGE}

Additional Information

• Error:
  error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "eu.gcr.io/******/****************:": creating push check transport for eu.gcr.io failed: GET https://eu.gcr.io/v2/token?scope=repository%3A***********%2F*****%3Apush%2Cpull&service=eu.gcr.io: UNAUTHORIZED: Not Authorized.
• Kaniko Image 28a6bdc-debug
  1.7.0-debug

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile

[amity0]

Also getting this error

[SirPhemmiey]

Having this issue also!!!

[lienerieksta]

Using Workload Identity to push images to GCR is producing the same error as OP posted.

[rushilsrivastava]

Seeing this error as well.

[nielsbox]

Dupe of #1786

[amichal]

I posed on the dupe but I'm wondering if its somehow here cf4822c

[Keramblock]

Same, downgraded to 1.6.0 - fixed issue for me =(

[eremeevfd]

We are facing this issue as well

[dhpollack]

Same here. Using v1.6.0 instead of latest solved the problem for me

[dannyvanl]

Same issue for us this morning. Rolling back to 1.6.0 solved it for now.

[Foxinou35]

Same issue here :/ Workaround worked thanks
PS : lost several hours checking and checking the roles on my service account ^^

[gmega]

Same here, looks like this could be a regression of #1242.

[tejal29]

Thanks fix is the way! #1794

[Billcountry]

Having this issue too. Had to tie down the version v1.6.0

[Tarang]

is this live yet?

[Richard87]

Sam error happend to me as well, but downgrading to v.1.6.0 did not solve the error :/

[nicolas-goudry]

Same issue here.
A workaround to use 1.7.0 with GCR using a service account is to use the .docker/config.json method providing following credentials:

$ mkdir -p /kaniko/.docker
$ echo "{\"auths\":{\"gcr.io\":{\"auth\":\"$(echo -n _json_key:$(cat service-account-key.json) | base64 | tr -d "\n")\"}}}" > /kaniko/.docker/config.json

[jmuleiro]

Same issue here, using Kaniko for test environments, all build pipelines were broken. Temporarily fixed with rollback to version 1.6.0.

[mattstrain]

Same issue with image builds on gitlab runner in GKE using workload identity auth.

[corveen]

I am using gcloud to build dockers.

gcloud config set builds/use_kaniko True

Is there a workaround or way to downgrade to 1.6.0?
Because it seems that gcloud always uses gcr.io/kaniko-project/executor:latest

[mareksuscak]

@corveen and anyone who's struggling with this regression that's been introduced and left unsolved 3 days ago (yes, unbelievable) along with another regression in gcloud CLI preventing AppEngine deploys that has been introduced on the same day in SDK 361.0.0 and has not been addressed to this day either, here's what we did:

1. Add cloudbuild.yml (see below) to the root of your project

   # cloudbuild.yml
   
   steps:
     # TODO: revert to :latest tag once the following issue is closed:
     # https://github.com/GoogleContainerTools/kaniko/issues/1786
     - name: 'gcr.io/kaniko-project/executor:v1.6.0'
       id: 'Build and push container image'
       args:
         [
           '--destination=${_IMAGE}',
           '--cache=true',
           '--cache-ttl=6h'
         ]
   
   timeout: 1800s
   

2. Rewrite your deployment script as follows:

   IMAGE=us.gcr.io/your-project-id/your-image:your-tag
   - gcloud builds submit --tag $IMAGE
   + gcloud builds submit --substitutions _IMAGE=$IMAGE

3. Pin gcloud SDK to the version 360.0.0 for the time being

This is the equivalent of using gcloud builds submit --tag and gives you full control over the version of Kaniko and other parameters.

Relevant documentation that helped us solve this issue:

• https://cloud.google.com/build/docs/configuring-builds/substitute-variable-values

[stefpb]

Keeping version 1.6 is now the answer? Why no one revert this 1.7 version?

[imjasonh]

Keeping version 1.6 is now the answer? Why no one revert this 1.7 version?

The current :latest used by gcloud points to 1.6.0, since there was a bug in 1.7.0.

[kaaboaye]

Current latest works for me. I think it was fixed.

…

On 21 Dec 2021, at 20:48, Jason Hall ***@***.***> wrote:  Keeping version 1.6 is now the answer? Why no one revert this 1.7 version? The current :latest used by gcloud points to 1.6.0, since there was a bug in 1.7.0. — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you are subscribed to this thread.

[rwong2888]

@stefpb , @imjasonh

I was able to use 1.7.0 by mounting a docker config using the SA creds

https://issueexplorer.com/issue/GoogleContainerTools/kaniko/1791#10012106

https://webcache.googleusercontent.com/search?q=cache:jx9iGg75zJcJ:https://issueexplorer.com/issue/GoogleContainerTools/kaniko/1791+&cd=1&hl=en&ct=clnk&gl=us

nicolas-goudry wrote this answer on 2021-10-21
Same issue here.
A workaround to use 1.7.0 with GCR using a service account is to use the .docker/config.json method providing following credentials:
$ mkdir -p /kaniko/.docker
$ echo "{"auths":{"gcr.io":{"auth":"$(echo -n _json_key:$(cat service-account-key.json) | base64 | tr -d "\n")"}}}" > /kaniko/.docker/config.json

@Keramblock latest got reverted to 1.6.0 due to issues

[hudac]

Also tested v1.8.0 and v1.8.1, same issue. Reverted back to v1.6.0, now it works :/

[kdaghay]

Still an issue when using 1.9.1. Used @nicolas-goudry's suggestion which works.

#1791 (comment)

Same issue here. A workaround to use 1.7.0 with GCR using a service account is to use the .docker/config.json method providing following credentials:

$ mkdir -p /kaniko/.docker
$ echo "{\"auths\":{\"gcr.io\":{\"auth\":\"$(echo -n _json_key:$(cat service-account-key.json) | base64 | tr -d "\n")\"}}}" > /kaniko/.docker/config.json

[aaron-prindle]

Can anyone here comment on if this issue is still occuring for them w/ the latest version of Kaniko v1.12.1 and any specific repro steps that can be done to replicate this? I have tested uploading images to GCR w/ Kaniko (using gcr.io domain) using both a json service account key w/ GCR push perms as well as by setting up Workload Identity w/ a service account w/ GCR push perms and Kaniko is successfully uploading images in both cases. Happy to dig into this more if someone else here can provide additional repro details. Thanks

[hudac]

I can't reproduce this issue anymore. We are using v1.9.2 (and testing v1.12), no issue so far.

In our case, this was a configuration issue. We are using GCR auth via workload identity. Till v1.6 we had to provide a Docker config file (/kaniko/.docker/config.json), like this one:

{
  "credHelpers": {
    "eu.gcr.io": "gcr",
    "gcr.io": "gcr"
  }
}

Since at least v1.8 (maybe even v1.7) you don't need, actually you cannot have this Docker config file anymore. If this file is present, Kaniko v1.8 or newer fails with an auth error as reported in this issue. And vice versa, v1.6 does not work without the config file above.

If you are using the newest version of Kaniko with GCR and workload identities, omit the Docker config file. Keeping an empty one {} also works.

[aaron-prindle]

Thanks @hudac, closing this issue as I believe this is resolved @ HEAD