Using multiple Registries bug

[ArnaudWald]

I am trying to build on top of a public DockerHub image, and push the result into my private registry.

My Dockerfile:

FROM python:3
RUN cat /kaniko/.docker/config.json

The command I run

docker run -ti --rm -v "$(pwd)"/build-context:/workspace -v "$(pwd)"/docker:/kaniko/.docker:ro gcr.io/kaniko-project/executor:latest --dockerfile=Dockerfile --context=dir:///workspace --destination=my-private-registry/test-kaniko/test-kani:latest

Result
The initial permission checking succeeds, the build is successful, but I cannot push, I get error pushing image: failed to push to destination my-private-registry/testkaniko/test-kani:latest: unsupported status code 401; body: Unauthorized request

Full log

INFO[0000] Resolved base name python:3 to python:3
INFO[0000] Resolved base name python:3 to python:3
INFO[0000] Retrieving image manifest python:3
INFO[0002] Image python:3 not found in cache
INFO[0002] Retrieving image manifest python:3
INFO[0003] Built cross stage deps: map[]
INFO[0003] Retrieving image manifest python:3
INFO[0004] Image python:3 not found in cache
INFO[0004] Retrieving image manifest python:3
INFO[0004] Unpacking rootfs as cmd RUN cat /kaniko/.docker/config.json requires it.
INFO[0024] Taking snapshot of full filesystem...
INFO[0029] RUN cat /kaniko/.docker/config.json
INFO[0029] cmd: /bin/sh
INFO[0029] args: [-c cat /kaniko/.docker/config.json]
{
  "auths": {
    "my-private-registry": {
      "auth": "XXX"
    }
  }
}
INFO[0029] Taking snapshot of full filesystem...
INFO[0030] No files were changed, appending empty layer to config. No layer added to image.
error pushing image: failed to push to destination my-private-registry/test-kaniko/test-kani:latest: unsupported status code 401; body: Unauthorized request

My interpretation
It seems for the debug that kaniko is trying to get the image from the public docker hub. However, there is no credentials, so it fall back to an anonymous identity to pull the base image. And then it does not switch back when trying to push after the build succeeds.

Workaround
A workaround I did is to stay on my private registry from start to finish.

• Pull the base image docker pull python:3
• Tag it accordingly docker tag python:3 my-private-registry/test-kaniko/python:3
• Push it to your private registry docker push my-private-registry/test-kaniko/python:3
• Change the Dockerfile FROM my-private-registry/test-kaniko/python:3

---

Note: I though about adding two auths in the config.json for each registry, like this:

{
  "auths": {
    "my-private-registry": {
      "auth": "XXX"
    },
    "https://index.docker.io/v1/": {
      "auth": "YYY"
    }
  }
}

However, this does not fix the issue

---

Therefore, my question is : is there a better way to use different registries for running my Kaniko pipelines ? Or is this an unexpected behavior ?

[cvgw]

I'm pretty sure pulling a base image from a public registry and pushing to a private registry works as thats what I do daily for testing and such. There might be some edge case here, but it might be related to auth on your private registry rather than combination of public and private.

If you don't mind me asking, what private registry are you using? (Service provider or project name would be great)

[ArnaudWald]

I'm using the Scaleway Registry.

[tejal29]

@ArnaudWald Does docker push to your private registry work?

[oleksdovz]

Have same error with latest Kaniko images. Checked via k8s and local docker. Docker registry has cert and I'm able to push new image from docker but with Kaniko it doesn't work

cat ~/config.json
{
"auths": {
"test.docker.test": {
"auth": "XXXXXXXX"
}
}
}

Run docker:

sudo docker run -ti --rm -v pwd:/workspace -v /home/ubuntu/config.json:/kaniko/.docker/config.json:ro gcr.io/kaniko-project/executor:latest --dockerfile=Dockerfile --destination='test.docker.test/test/test:1'

End error message at end:

Installing collected packages: pydantic, starlette, fastapi, uvloop, httptools, websockets, click, h11, uvicorn, MarkupSafe, jinja2, urllib3, elasticsearch
Successfully installed MarkupSafe-1.1.1 click-7.1.1 elasticsearch-7.6.0 fastapi-0.53.2 h11-0.9.0 httptools-0.1.1 jinja2-2.11.1 pydantic-1.4 starlette-0.13.2 urllib3-1.25.8 uvicorn-0.11.3 uvloop-0.14.0 websockets-8.1
INFO[0038] Taking snapshot of full filesystem...
INFO[0039] Resolving paths
INFO[0043] CMD ["python", "docker_app.py"]
error pushing image: failed to push to destination test.docker.test/test/test:1: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:test/test Type:repository] map[Action:push Class: Name:test/test Type:repository]]

[oleksdovz]

Sorry the problem was in docker-compose file for registry, in ENV params.

version: '3'
services:
registry:
restart: always
image: registry:latest
container_name: registry
ports:
- "5000:5000"
environment:
REGISTRY_AUTH: htpasswd
REGISTRY_AUTH_HTPASSWD_REALM: Registry Realm
REGISTRY_AUTH_HTPASSWD_PATH: /auth/registry.password
REGISTRY_STORAGE_DELETE_ENABLED: 'true'
REGISTRY_LOG_LEVEL: warn
REGISTRY_HTTP_HOST: https://XXX.XXX.XXX:443
REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY: /data
volumes:
- ./auth:/auth
- /registry:/data
nginx:
image: nginx:alpine
container_name: nginx
restart: unless-stopped
tty: true
ports:
- "80:80"
volumes:
- ./configs/nginx/:/etc/nginx/conf.d/

Env params was incorrect - REGISTRY_HTTP_HOST: https://XXX.XXX.XXX:443, just delete and everything works

[caiofbpa]

I'm having a similar issue, but my image is FROM a private ECR and gets pushed to a private GitLab Container Registry. Apparently depending on where I run the CI job, the FROM layer works, or it doesn't.

INFO[0000] Resolved base name XXX.dkr.ecr.us-east-1.amazonaws.com/org/image:tag to XXX.dkr.ecr.us-east-1.amazonaws.com/org/image:tag 
INFO[0000] Using dockerignore file: /builds/org/project/.dockerignore 
INFO[0000] Resolved base name XXX.dkr.ecr.us-east-1.amazonaws.com/org/image:tag to XXX.dkr.ecr.us-east-1.amazonaws.com/org/image:tag 
INFO[0000] Retrieving image manifest XXX.dkr.ecr.us-east-1.amazonaws.com/org/image:tag 
ERRO[0041] Error while retrieving image from cache: XXX.dkr.ecr.us-east-1.amazonaws.com/org/image:tag unsupported status code 401; body: Not Authorized 
INFO[0041] Retrieving image manifest XXX.dkr.ecr.us-east-1.amazonaws.com/org/image:tag error building image: unsupported status code 401; body: Not Authorized

The configuration is the same in both cases, just the runner that changes:

{
  "credHelpers": {
    "XXX.dkr.ecr.us-east-1.amazonaws.com": "ecr-login"
  },
  "auths": {
    "registry.gitlab.com": {
      "username": "username",
      "password": "password"
    }
  }
}

[dinvlad]

Same issue here. We're using GCR for --cache-repo and ECR for --destination. Specifying either

{
  "auths": {
    "XXX.dkr.ecr.us-east-1.amazonaws.com": {
      "auth": "YYY"
    }
  }
}

or

{
  "credsStore": "gcr"
}

works, but when we supply both:

{
  "credsStore": "gcr",
  "auths": {
    "XXX.dkr.ecr.us-east-1.amazonaws.com": {
      "auth": "YYY"
    }
  }
}

then we are getting the above error (POST https://XXX.dkr.ecr.us-east-1.amazonaws.com/v2/zzz/blobs/uploads/: unsupported status code 401; body: Not Authorized)

[LeslieK]

Here is a sequence diagram of authn : a package used by Kaniko.
https://github.com/google/go-containerregistry/tree/master/pkg/authn#authn

After the 401 is received from the Registry, does kaniko send out the next request: GetAuthConfig()?

[insider89]

Does anyone found workaround?

[tejal29]

@insider89 release v1.11.0 should have fixed this issue.
I merged a PR #1439 to fix issue where credential initializer was only called for the first destination.

Are you still seeing this issue?

[insider89]

@tejal29 Looks like fixed. I am using following version executor:debug-v1.2.0.

I am using gitlab CI to run kaniko. Before run kaniko I log in to both ecr and gcr, and then mount .docker/config.json to kaniko container. As cache I am using gcr, final image successfully push to both ecr and gcr.

docker run --rm \
		-v $PWD/.kaniko:/kaniko/.docker/ \
		gcr.io/kaniko-project/executor:debug-v1.2.0 \
		--cache=true \
		--cache-dir=/workspace/.kaniko/cache \
		--cache-repo ${DOCKER_REGISTRY_CI} \
		--destination $IMAGE_MAIN_TAG \
		--context dir:///workspace/ \
		--build-arg SOURCE_VERSION=${CI_COMMIT_SHORT_SHA} \
		--dockerfile $$DOCKERFILE \
		--snapshotMode=redo \
		$KANIKO_EXTRA_ARGS \
		; \

[DavidAntliff]

In case this helps anyone, I wasn't able to get any combination of credsStore and auths to work together in config.json so I ended up doing this: #687 (comment)

This creates two auths entries, similar to:

{
  "auths": {
    "$CI_REGISTRY": {
      "username": "$CI_REGISTRY_USER",
      "password": "$CI_REGISTRY_PASSWORD"
    },
    "$AWS_ECR_URL": {
      "auth": "ecr_auth"
    }
  }
}

This worked fine for me and was the only way I found to have simultaneous ECR and GitLab authentication in a single kaniko 1.6.0 invocation. I'm using this in a GitLab-CI pipeline that pulls down GitLab registry images and pushes them to ECR.

[chopeen]

The same trick does not work with Docker. When I add "auth": "ecr_auth" to ~/.docker/config.json, I am getting the following error:

Error loading config file: [..]/.docker/config.json: illegal base64 data at input byte 3

I guess that means that the contents of auth (i.e. ecr_auth in this case) must be a base64-encoded value.

[franchb]

Still the same issue in 2024, auth must be a base64-encoded value.