Is --config ignored when using the k8s driver for buildx

[goffinf]

re: comment: #2044 (comment)

I noted the above comment under another issue and wanted to check whether using --config with the buildx k8s driver is NOT supported right now (and if not, if it is likely to be in the near term future) ?

This is what I have observed in trying to get this to work.

I have been struggling trying to get buildx to allow me to use a private registry. Per the documentation I added --config with buildx create ..., like this:

docker buildx create --name buildx-test --config config.toml --driver kubernetes --driver-opt replicas=1,namespace=buildx --use

where the config.toml file exists in the directory where I execute the above command and contains:

debug = true
root = "/var/lib/buildkit"
insecure-entitlements = [ "network.host", "security.insecure" ]
[registry."host.k3d.internal:5000"]
http = true
insecure = true

When I request a build and use --load or --push I get the following output, which shows that the build worked successfully but buildx attempts to connect to the private registry with HTTPS rather than HTTP:

docker buildx build --platform "linux/amd64" -t "goffinf/buildx-test:1.0.0" -t "goffinf/buildx-test:latest" --build-arg MAINTAINER=goffinf@gmail.com --load .

[+] Building 4.2s (7/7) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                                        0.6s
 => => transferring dockerfile: 32B                                                                                                                                                                         0.0s
 => [internal] load .dockerignore                                                                                                                                                                           0.8s
 => => transferring context: 2B                                                                                                                                                                             0.0s
 => [internal] load metadata for docker.io/library/alpine:latest                                                                                                                                            1.2s
 => [1/2] FROM docker.io/library/alpine:latest@sha256:69e70a79f2d41ab5d637de98c1e0b055206ba40a8145e7bddb55ccc04e13cf8f                                                                                      0.5s
 => => resolve docker.io/library/alpine:latest@sha256:69e70a79f2d41ab5d637de98c1e0b055206ba40a8145e7bddb55ccc04e13cf8f                                                                                      0.4s
 => CACHED [2/2] RUN mkdir /buildx &&     echo "${MESSAGE}" >> /buildx/README.md;                                                                                                                           0.0s
 => exporting to oci image format                                                                                                                                                                           1.3s
 => => exporting layers                                                                                                                                                                                     0.0s
 => => exporting manifest sha256:a8cd3a8da4fea6fa7b4ae350dd402f8274e2de864f9bd175b109d1727316aa44                                                                                                           0.3s
 => => exporting config sha256:0a31360539ed3c0e8a5c5b443d2b3fd476cc32b8967460cd38d54105888e4f3c                                                                                                             0.4s
 => => sending tarball                                                                                                                                                                                      0.4s
 => importing to docker                                                                                                                                                                                     0.1s
error: failed to do request: Head https://host.k3d.internal:5000/v2/goffinf/buildx-test/manifests/1.0.0: http: server gave HTTP response to HTTPS client

If I check the buildx container, there is definitely no sign of the toml, and the /etc/buildkit directory doesn't exist (assuming thats where it would be copied).

I can confirm that the buildx container can resolve the registry by exec'ing into the container and calling the registry API:

kubectl exec -it -n buildx buildx-test0-87c44c79c-xw4xr -- /bin/sh

/ # wget http://host.k3d.internal:5000/v2/_catalog
Connecting to host.k3d.internal:5000 (172.19.0.1:5000)
saving to '_catalog'
_catalog             100% |*****************************************************************************************************************************************************************|   280  0:00:00 ETA
'_catalog' saved

/ # cat _catalog
{"repositories":["alpine","busybox","docker","docker/compose","goffinf/azure-agent","goffinf/azure-agent-standalone","goffinf/build-test","goffinf/k8s-tools","goffinf/rancher-tools","goffinf/rvault","goffinf/terraform","kir4h/rvault","nginx","nginxdemos/hello","ubuntu","vault"]}

Kind Regards

Fraser Goffin

[tonistiigi]

This is not buildkit issue. Open in build repo