settings.container-registry.credentials not working as expected

[BenB196]

Hi All,

I'm hoping someone might be able to help. I'm trying to implement: #1618 via a custom container image. I'm using a self-hosted GitLab instance container registry, and am using AWS EKS managed node group with the Bottlerocket AMI (1.23). I'm seeing the newly created Bottlerocket node contact the container registry, but I'm only seeing 401s on GitLab end. I've validated that the username/password work directly, so not sure where I have a configuration issue. The docs seem a bit spare in this area, so wondering if anyone can spot an error/issue in my configuration.

Userdata addition:

[[settings.container-registry.credentials]]
registry = "gitlab.domain.com:5050/bottlerocket-os-ephemeral-disk"
username = "bottlerocket-image"
password = "<snipped>"
[settings.bootstrap-containers.setup-ephemeral-disks]
source = "gitlab.domain:5050/bottlerocket-os-ephemeral-disk:latest-xfs"
mode = "always"
essential = true

[arnaldo2792]

Hello @BenB196 could you please provide the logs of the failed bootstrap containers? You can use the configurations that you have, but instead of essential = true make them essential = false so that you can extract the relevant logs from either the host or admin host containers via /.bottlerocket/support/bottlerocket-logs.tar.gz.

Is your custom gitlab endpoint behind https? If so, did you updated the CA store through the API?

[etungsten]

Hi @BenB196, container-registry.credentials[].registry should be the registry host for your container registry and not the image repository URL. In this case, the registry host would be "gitlab.domain.com:5050".

The settings are mapped to containerd's configuration for registry credentials as shown in https://github.com/containerd/containerd/blob/release/1.6/docs/cri/registry.md#configure-registry-credentials
Specially this part pertains to how container-registry.credentials[].registry should be set:

The registry host has to be a domain name or IP. Port number is also needed if the default HTTPS or HTTP port is not used.

Please try that out and let us know how it goes.

[BenB196]

Thanks, that did the trick. I guess I assumed that the registry source worked similar to an image pull secret in that you could specify a specific repo. Good to know that in this case, it's just expecting the top-level connection.