Error checking push permissions 0.17.0 onwards

[mmmdreg]

We use a private artifactory registry which works completely fine for all tags from 0.10.0 to 0.16.0, using gitlab with a command like:

image:
  name: kaniko-project/executor:debug-v0.16.0
  entrypoint: [“/busybox/sh”, “-c”]
script:
  - export PATH=$PATH:/kaniko/
  - echo $authfile > /kaniko/.docker/config.json
  - /kaniko/executor —context $project_dir —dockerfile $dockerfile —destination $repo:$tag —skip-tls-verify —skip-tls-verify-pull —cache=true

From v0.17.0 to v0.19.0, this yields “error checking push permissions...unsupported status code 401”

I do not see anything in the change log that seems relevant to this but did see the note under 0.18.0 that v0.17.0 introduced many regressions.

[danielfbm]

+1 experiencing similar issue using artifactory-jcr https://github.com/jfrog/charts/tree/master/stable/artifactory-jcr

[tejal29]

Sorry for jumping late.
can you provide more detailed logs please?

[mmmdreg]

Tried with the latest debug-v0.22.0 with verbosity increased and both with and without cache. I’m in a corporate network so I can’t paste logs but here is the gist of it:

DEBU[0000] Copying file /..../Dockerfile to /kaniko/Dockerfile
TRAC[0000] Adding /var/run to initialWhitelist
error checking push permissions — make sure you entered the correct tag name... checking push permissions for “myregistry/myImage:myTag”: creating push check transport for myregistry failed: GET https://myregistry/artifactory/api/docker/docker-dev/v2/token?scope=repository%3myimage%3Apush%2Cpull&service=myregistry: unsupported status code 401; body:...
<title>401 Authorization Required</title>...
<address>Apache/2.2.15 (Red Hat) server at myregistry Port 443</address>...

There are no other lines.

Note that this still works with 0.16.0 and below. My best guess is something changed in how go-containerregistry performs auth.

Edit:

• i also tried username/password instead of auth as hinted by RFE: Add to documentation how to push to docker registry V2 #887 with the same result
• I had multiple registries in my config.json so also tried using only one, in case it was related to Kaniko supporting multiple auths #687 or Using multiple Registries bug #918

[johnmanko]

Update: Seems to have resolved itself.

---

I'm running into this same error. My setup is GitLab & GitLab Runner in docker. The private registry is Nexus. Nexus is accessible and I'm able to request tokens outside of my build job.

build-docker:
  stage: build
  tags:
    - docker
  image:
    name: gcr.io/kaniko-project/executor:debug-v0.22.0
    entrypoint: [""]
  script:
    - ls $CI_PROJECT_DIR
    - ls $CI_PROJECT_DIR/target
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASS\"}}}" > /kaniko/.docker/config.json
    - /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/docker/Dockerfile --destination $CI_REGISTRY/products/lc-service:$CI_COMMIT_TAG
  only:
    - tags

GitLab Runner Output:

Running with gitlab-runner 12.5.0 (577f813d)
  on AGO GitLab Runner sPyvnUq7
Using Docker executor with image gcr.io/kaniko-project/executor:debug-v0.22.0 ...
Pulling docker image gcr.io/kaniko-project/executor:debug-v0.22.0 ...
Using docker image sha256:c9f1d49fc7f7635598e32ffe15f888be61fb0554dc906d1b1448b5b5436fd1a3 for gcr.io/kaniko-project/executor:debug-v0.22.0 ...
Running on runner-sPyvnUq7-project-42-concurrent-0 via d4e0e91dc5ad...
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /builds/products/lc-service/.git/
From http://gitlab-local:8778/products/lc-service
 * [new ref]         refs/pipelines/1823 -> refs/pipelines/1823
 t [tag update]      1.0.0-SNAPSHOT      -> 1.0.0-SNAPSHOT
Checking out a8dbfe83 as 1.0.0-SNAPSHOT...
Removing .m2/repository/
Removing target/

Skipping Git submodules setup
Checking cache for default-1...
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted. 
Successfully extracted cache
$ echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASS\"}}}" > /kaniko/.docker/config.json
$ /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/docker/Dockerfile --destination $CI_REGISTRY/products/lc-service:$CI_COMMIT_TAG
error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "gitlab-local:18081/products/lc-service:1.0.0-SNAPSHOT": creating push check transport for gitlab-local:18081 failed: GET http://gitlab-local:18081/v2/token?scope=repository%3Aproducts%2Flc-service%3Apush%2Cpull&service=http%3A%2F%2Fgitlab-local%3A18081%2Fv2%2Ftoken: unsupported status code 401
ERROR: Job failed: exit code 1

Trying to run with debug-v0.16.0 is the same result:

error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "gitlab-local:18081/products/lc-service:1.0.0-SNAPSHOT": creating push check transport for gitlab-local:18081 failed: unsupported status code 401

[MrAmbiG]

same issue here, gitlab-runner + kaniko debug 0.16.0, private registry is harbor.

[HaveFun83]

same here gitlab and kaniko v0.23.0

[mmmdreg]

Closing as this is fixed for me using 1.2.0.