buildctl ignores buildkitd.toml

[Raboo]

Hi

So I'm trying to build a image in a k8s hosted buildkit, cache and push it to a local unsecured http registry resolvable via registry.
I have a buildkitd.toml like this:

debug = false
[worker.containerd]
  namespace = "buildkit"
[registry."registry:5000"]
  http = true
  insecure = true

And when I run

buildctl --addr kube-pod://buildkitd-0?namespace=buildkit build --frontend dockerfile.v0 \
--export-cache type=registry,mode=max,ref=registry:5000/hello-kubernetes:cache \
--import-cache type=registry,ref=registry:5000/hello-kubernetes:cache \
--output type=image,name=registry:5000/hello-kubernetes:latest,push=true \
--local context=./app --local dockerfile=.

It fails

------
 > exporting to image:
------
error: failed to solve: rpc error: code = Unknown desc = failed to do request: Head https://registry:5000/v2/hello-kubernetes/blobs/sha256:8c8ce9df421ec8f20ef35a84aa9b858a9c01486b1a7fd867c9850c4e8960eb7d: dial tcp 10.43.70.12:5000: connect: connection refused

So the problem here is that it is using https.
I don't understand. I clearly have a configuration that says it's http (not https) and insecure. So why bother having a configuration file if buildctl just tells buildkitd to ignore the configuration? How do I make buildctl use the registry that is provided in buildkitd.toml?

[tonistiigi]

buildkitd.toml is daemon configuration file. It needs to be present on the node that is performing the build (or --config with buildx that will copy it to the buildkit container for you).

[Raboo]

Ok, but that still doesn't answer why there is a registry. section in buildkitd.toml if buildctl cannot use the config when building on that host with the daemon and worker that consumes the buildkitd.toml.

[registry."registry:5000"]
  http = true
  insecure = true

To simplify my question.

How do I tell buildctl to use the registry defined in buildkitd.toml when building against a host that has the buildkitd.toml?

[tonistiigi]

Like I said before. Registry configuration is based on the buildkitd daemon, not buildctl or any other client making the call to the daemon. It is loaded when buildkitd starts and then all requests by any client to that daemon follow that registry configuration.

[Raboo]

First I must apologize. But there seem to have been a problem with my registry that caused the error connection refused. That was fixed but I am still unable to get "any client (buildctl in my case)" to use the registry configuration specified in buildkitd (/etc/buildkit/buildkitd.toml).

• buildkitd has a registry configuration via buildkitd.toml ex. [registry."registry:5000"]
• buildctl dispatches a build to buildkitd.
• buildctl has a ref=registry:5000/xyz or a name=registry:5000/xyz that matches the registry in the buildkitd configuration.
• buildkitd does not use the information from it's registry configuration but instead it uses the configuration that comes from the client builtctl.

So is this still a question or a bug? I'm super confused.

[Raboo]

It seems to be a bug in rootless mode as this works when I run in priveleged mode.

[tonistiigi]

can you write down the steps that work and ones that do not. I wonder if the config file is not just readable for the rootless user. Might need to chown it to user 1000, FYI @AkihiroSuda

[Raboo]

Something like this

buildkit.yaml (working)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: buildkit
  namespace: buildkit
  labels:
    app: buildkit
    rootless: "false"
    runtime: docker
    worker: runc
  annotations:
    buildkit.mobyproject.org/builder: 0.1.3
spec:
  replicas: 1
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  selector:
    matchLabels:
      app: buildkit
      rootless: "false"
      runtime: docker
      worker: runc
  template:
    metadata:
      labels:
        app: buildkit
        rootless: "false"
        runtime: docker
        worker: runc
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app: buildkit
                rootless: "false"
                runtime: docker
                worker: runc
            topologyKey: kubernetes.io/hostname
      imagePullSecrets:
      - name: dockerhub
      containers:
        - name: buildkitd
          image: moby/buildkit:v0.8.2
          resources:
            limits:
              cpu: "2"
            requests:
              cpu: 100m
          volumeMounts:
          - mountPath: /etc/buildkit/
            name: buildkitd-config
          - mountPath: /run/docker.sock
            name: docker-sock
          readinessProbe:
            exec:
              command:
                - buildctl
                - debug
                - workers
            initialDelaySeconds: 5
            failureThreshold: 3
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          livenessProbe:
            exec:
              command:
                - buildctl
                - debug
                - workers
            initialDelaySeconds: 5
            periodSeconds: 30
          securityContext:
            privileged: true
      volumes:
      - configMap:
          defaultMode: 420
          name: buildkit
        name: buildkitd-config
      - hostPath:
          path: /var/run/docker.sock
          type: Socket
        name: docker-sock
      restartPolicy: Always
      terminationGracePeriodSeconds: 30

buildkit-rootless.yaml (not working)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: buildkit
  namespace: buildkit
  labels:
    app: buildkit
    rootless: "true"
    runtime: docker
    worker: runc
  annotations:
    buildkit.mobyproject.org/builder: 0.1.3
spec:
  replicas: 1
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  selector:
    matchLabels:
      app: buildkit
      rootless: "true"
      runtime: docker
      worker: runc
  template:
    metadata:
      labels:
        app: buildkit
        rootless: "true"
        runtime: docker
        worker: runc
      annotations:
        container.apparmor.security.beta.kubernetes.io/buildkitd: unconfined
        container.seccomp.security.alpha.kubernetes.io/buildkitd: unconfined
    spec:
      imagePullSecrets:
      - name: dockerhub
      containers:
        - name: buildkitd
          image: moby/buildkit:v0.8.2-rootless
          args:
            - --oci-worker-no-process-sandbox
            - --rootless
          resources:
            limits:
              cpu: "2"
            requests:
              cpu: 100m
          volumeMounts:
          - mountPath: /etc/buildkit/
            name: buildkitd-config
          readinessProbe:
            exec:
              command:
                - buildctl
                - debug
                - workers
            initialDelaySeconds: 5
            failureThreshold: 3
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          livenessProbe:
            exec:
              command:
                - buildctl
                - debug
                - workers
            initialDelaySeconds: 5
            periodSeconds: 30
          securityContext:
            runAsUser: 1000
            runAsGroup: 1000
      volumes:
      - configMap:
          defaultMode: 420
          name: buildkit
        name: buildkitd-config
      restartPolicy: Always
      terminationGracePeriodSeconds: 30

buildkit-cfg.yaml (buildkitd.toml for buildkitd)

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: buildkit
  name: buildkit
data:
  buildkitd.toml: |
    debug = false
    [worker.containerd]
      namespace = "buildkit"
    [registry."registry:5000"]
      http = true
      insecure = true

I can also add that when running rootless I am still able to exec into the container and read /etc/buildkit/buildkitd.toml with out any permission problems.

[AkihiroSuda]

The config path is ~/.config/buildkit/config.toml for rootless

[Raboo]

Does the pod spec.containers.volumeMounts.mountPath support tilde(~)? Not sure it does..
What if I add --config /etc/buildkit/buildkitd.toml to the arguments list in rootless mode, will it respect the config argument?

[tonistiigi]

@AkihiroSuda I guess we should add rootless=true driver opt to container driver as well. It's hardcoded to https://github.com/docker/buildx/blob/master/driver/docker-container/driver.go#L328 atm. Also, looks like k8s driver doesn't support --config.

[tonistiigi]

Ah, mixed up the repository (thought this was buildx). I guess we can close this then. @Raboo the user in the rootless image is called "user".

[Raboo]

@tonistiigi So I should put the config in /home/user/.config/buildkit/config.toml for rootless mode?

This should probably be documented.

[Raboo]

putting the configuration in /home/user/.config/buildkit/config.toml still doesn't work.

[treo]

If you are building on top of the job.rootless.yaml template then you can add --config /etc/buildkit/buildkitd.toml in https://github.com/moby/buildkit/blob/master/examples/kubernetes/job.rootless.yaml#L31-L32

That worked for me.

[LinuxSuRen]

If you are building on top of the job.rootless.yaml template then you can add --config /etc/buildkit/buildkitd.toml in https://github.com/moby/buildkit/blob/master/examples/kubernetes/job.rootless.yaml#L31-L32

That worked for me.

Works well in the Argo workflows. Now I can use buildkit to pull and push images to an insecure image registry.

See also https://github.com/devops-ws/argo-workflows-guide

[Raboo]

closing, the acceptable workaround is to use the config argument and then volume mount the file to the same path.

args:
  - --config=/home/user/.config/buildkit/buildkitd.toml

or whatever path you want to use.

[puckettgw]

I'm running into this issue even with the config available and explicitly defined:
I set up the builder using the CLI:

docker --tlscacert=/etc/ssl/certs/ca-certificates.crt buildx create --bootstrap --name=kube --driver=kubernetes --driver-opt=namespace=something --driver-opt=image=registry.somewhere/buildkit:2.0 --buildkitd-flags="--config=/etc/buildkit/buildkitd.toml"

The /etc/buildkit/buildkit.toml looks like this:

debug = true
trace = true

insecure-entitlements = ["network.host", "security.insecure"]

[registry. "registry.somewhere"]
http = true
insecure = true
ca=["/usr/local/share/ca-certificates/ca.crt"]

[registry. "registry.somewhere:443"]
http = true
insecure = true
ca=["/usr/local/share/ca-certificates/ca.crt"]

The CA certificate is baked into the image defined in the driver-opt at that specified location. I tried updating the local trust store via update-ca-certificates and validated that the cert was there, but that didn't work. So that is why I directly pointed to the CA location like this.

The buildkit container shows the process running with the config file:

kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
PID   USER     TIME  COMMAND
    1 root      0:19 buildkitd --config=/etc/buildkit/buildkitd.toml 

However, when I attempt to build and push an image via this command:
docker --tlscacert=/etc/ssl/certs/ca-certificates.crt buildx build . --builder=kube -t registry.somewhere/someimage:sometag --push

I receive the following error message:

ERROR: failed to solve: failed to push registry.somewhere/someimage:4be66a33c47f0af8a6fa02e5913a734d8a6028eb: failed to authorize: failed to fetch anonymous token: Get "https://registry.somewhere/v2/token?scope=%2A%3A%3A&scope=repository%3A***%2Fsomeimage%3Apull%2Cpush&service=container_registry": tls: failed to verify certificate: x509: certificate signed by unknown authority

When I look in the logs for the pod, I notice the following:

194 0.19.1-1 /usr/libexec/docker/cli-plugins/docker-buildx --tlscacert=/etc/ssl/certs/ca-certificates.crt buildx build . --builder=kube -t registry.somewhere/george/someimage:4be66a33c47f0af8a6fa02e5913a734d8a6028eb --push
github.com/moby/buildkit/session/auth/authprovider.(*authProvider).FetchToken
	/build/src/vendor/github.com/moby/buildkit/session/auth/authprovider/authprovider.go:143
github.com/moby/buildkit/session/auth._Auth_FetchToken_Handler.func1
	/build/src/vendor/github.com/moby/buildkit/session/auth/auth_grpc.pb.go:166
github.com/moby/buildkit/session/auth._Auth_FetchToken_Handler
	/build/src/vendor/github.com/moby/buildkit/session/auth/auth_grpc.pb.go:168
google.golang.org/grpc.(*Server).processUnaryRPC
	/build/src/vendor/google.golang.org/grpc/server.go:1394
google.golang.org/grpc.(*Server).handleStream
	/build/src/vendor/google.golang.org/grpc/server.go:1805
google.golang.org/grpc.(*Server).serveStreams.func2.1
	/build/src/vendor/google.golang.org/grpc/server.go:1029
runtime.goexit
	/usr/local/go/src/runtime/asm_arm64.s:1222

1 v0.17.3 buildkitd --config=/etc/buildkit/buildkitd.toml --allow-insecure-entitlement=network.host
main.unaryInterceptor
	/src/cmd/buildkitd/main.go:717
google.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1
	/src/vendor/google.golang.org/grpc/server.go:1202
github.com/moby/buildkit/api/services/control._Control_Solve_Handler
	/src/api/services/control/control_grpc.pb.go:289
google.golang.org/grpc.(*Server).processUnaryRPC
	/src/vendor/google.golang.org/grpc/server.go:1394
google.golang.org/grpc.(*Server).handleStream
	/src/vendor/google.golang.org/grpc/server.go:1805
google.golang.org/grpc.(*Server).serveStreams.func2.1
	/src/vendor/google.golang.org/grpc/server.go:1029
runtime.goexit
	/usr/local/go/src/runtime/asm_arm64.s:1222

I am using the buildkit 0.17.3 image. I noticed the first block in the trace shows a 0.19.1-1 image -- I have not specified that image anywhere. Why is it being pulled? I have a feeling that whatever is causing this image to be pulled and used is causing the issues with CA trust.