buildx new builder do not use proxy settings of default docker daemon.

[kunalkushwaha]

Bug: while trying to build multiarchitecture image using buildx, proxy settings of docker daemon are ignored, resulting unable to download the images.

This is only reproducible in Linux, Docker desktop for Mac works fine.

Steps to be followed

ref: same steps from : docker-for-arm-on-linux

$ docker buildx create --name mybuilder
$ docker buildx use mybuilder
$ docker buildx inspect --bootstrap
$ docker buildx build --platform linux/arm,linux/arm64,linux/amd64 -t timtsai2018/hello .

error:

failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to build LLB: failed to load cache key: failed to do request: Head https://registry-1.doc
ker.io/v2/library/alpine/manifests/latest: dial tcp 52.87.94.70:443: connect: connection refused    

Software versions:

$ docker buildx version
github.com/docker/buildx v0.2.2-10-g3f18b65-tp-docker 3f18b659a09804c738226dbf6bacbcae54afd7c6

$ docker version
Client: Docker Engine - Community
 Version:           19.03.1
 API version:       1.40
 Go version:        go1.12.5
 Git commit:        74b1e89
 Built:             Thu Jul 25 21:21:05 2019
 OS/Arch:           linux/amd64
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          19.03.1
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.5
  Git commit:       74b1e89
  Built:            Thu Jul 25 21:19:41 2019
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          1.2.6
  GitCommit:        894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc:
  Version:          1.0.0-rc6+dev
  GitCommit:        2b18fe1d885ee5083ef9f0838fee39b62d653e30
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

[tonistiigi]

Only builders backed by docker driver use the docker daemon config. For the container driver you need to either use --buildkitd-flags on create or make a buildkitd toml config and use --config.

[fpgajoe]

@tonistiigi
Would you help me?

I'm behind proxy.
I've already set the proxy setting on ~/.docker/config.json and /etc/systemd/docker/docker.service.d/<xxx.conf>
Of course, I've already set "experimental": "enabled" in ~/.docker/config.json

And I set DNS settings on /etc/docker/daemon.json

As a result, I'm able to use normally docker behind proxy.
But when I try to use buildx, I stacked:(

Here is my try.

# docker buildx create --use --name test --buildkitd-flags '--allow-insecure-entitlement security.insecure --allow-insecure-entitlement network.host'
test
# docker buildx inspect --bootstrap
[+] Building 8.9s (1/1) FINISHED                                                                     
 => [internal] booting buildkit                                                                 8.9s
 => => pulling image moby/buildkit:buildx-stable-1                                              8.2s
 => => creating container buildx_buildkit_test0                                                 0.7s
Name:   test
Driver: docker-container

Nodes:
Name:      test0
Endpoint:  unix:///var/run/docker.sock
Status:    running
Flags:     --allow-insecure-entitlement security.insecure --allow-insecure-entitlement network.host
Platforms: linux/amd64, linux/arm64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6

# docker buildx build --allow security.insecure,network.host -f <dockerfile> -t <tag> <place>

WARN[0000] No output specified for docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load 
[+] Building 60.1s (5/13)                                                                            
 => [internal] load .dockerignore                                                               0.0s
 => => transferring context: 2B                                                                 0.0s
 => [internal] load build definition from Dockerfile.arm32v7.debug                              0.0s
 => => transferring dockerfile: 665B                                                            0.0s
 => ERROR [internal] load metadata for docker.io/library/ubuntu:xenial                         30.0s
 => ERROR [base 1/2] FROM docker.io/library/ubuntu:xenial                                      30.0s
 => => resolve docker.io/library/ubuntu:xenial                                                 30.0s
 => [internal] load build context                                                               0.0s
------
 > [internal] load metadata for docker.io/library/ubuntu:xenial:
------
------
 > [base 1/2] FROM docker.io/library/ubuntu:xenial:
------
failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to build LLB: failed to load cache key: failed to do request: Head https://registry-1.docker.io/v2/library/ubuntu/manifests/xenial: dial tcp 34.228.211.243:443: i/o timeout

Would you have any idea for this problem?

[tonistiigi]

Container driver does not use the settings from docker daemon config. There isn't currently a way to set proxy settings in buildkit config but it could be added. Or we could just add it to driver-opt in the container level.

An ugly workaround currently is to create a new buildkit image based on moby/buildkit with extra proxy settings in the ENV. And then use buildx create --driver-opt image=yourimage

edit: for clarity. The config currently allows setting mirrors to registries but doesn't allow setting up global HTTP_PROXY variables. If you only need mirrors you can use --config pointing to config file like https://github.com/moby/buildkit/blob/master/docs/buildkitd.toml.md

[sirlatrom]

You can use --driver-opt env.ENV_NAME=ENV_VALUE after #170 was merged.

[spejic88]

@sirlatrom when I use
docker buildx create --driver-opt env.http_proxy=$http_proxy --driver-opt env.https_proxy=$https_proxy --driver-opt '"env.no_proxy='$no_proxy'"'

to set proxy, I get
invalid driver option env.http_proxy for docker-container driver

[sirlatrom]

@sirlatrom when I use
docker buildx create --driver-opt env.http_proxy=$http_proxy --driver-opt env.https_proxy=$https_proxy --driver-opt '"env.no_proxy='$no_proxy'"'

to set proxy, I get
invalid driver option env.http_proxy for docker-container driver

@spejic88: I recommend you open a new issue with a reproducible example, and make sure to specify what component versions are in use.

[tangx]

Container driver does not use the settings from docker daemon config. There isn't currently a way to set proxy settings in buildkit config but it could be added. Or we could just add it to driver-opt in the container level.

An ugly workaround currently is to create a new buildkit image based on moby/buildkit with extra proxy settings in the ENV. And then use buildx create --driver-opt image=yourimage

edit: for clarity. The config currently allows setting mirrors to registries but doesn't allow setting up global HTTP_PROXY variables. If you only need mirrors you can use --config pointing to config file like https://github.com/moby/buildkit/blob/master/docs/buildkitd.toml.md

Sir @tonistiigi ,

I try to use registry mirror in --config=config.toml .

But it seem do not work. it's slowly as usual.

configs and commands as below

config.toml

# registry configures a new Docker register used for cache import or output.
[registry.aliyun]
  mirrors = ["wlzfs4t4.mirror.aliyuncs.com"]
  # http = true
  # insecure = true
  # ca=["/etc/config/myca.pem"]
  # [[registry."docker.io".keypair]]
  #   key="/etc/config/key.pem"
  #   cert="/etc/config/cert.pem"

Makefile

build:
	docker buildx build --platform=linux/amd64 --load --tag tangx/debian:10 .

build-cache-from:
	docker buildx build --platform=linux/amd64 --tag tangx/debian:10 --cache-from=wlzfs4t4.mirror.aliyuncs.com/library/debian:10 .

qcloud-builder:
	docker buildx create --use --name qcloud-mirrors --config=config.toml
	$(MAKE) ls

ls:
	docker buildx ls

Dockerfile

FROM debian:10

command

# make qcloud-builder

docker buildx create --use --name qcloud-mirrors --config=config.toml
qcloud-mirrors
make ls
make[1]: Entering directory '/data/alpine'
docker buildx ls
NAME/NODE         DRIVER/ENDPOINT             STATUS   PLATFORMS
qcloud-mirrors *  docker-container
  qcloud-mirrors0 unix:///var/run/docker.sock inactive
default           docker
  default         default                     running  linux/amd64, linux/386
make[1]: Leaving directory '/data/alpine'


######

# make build

docker buildx build --platform=linux/amd64 --load --tag tangx/debian:10 .
[+] Building 96.4s (3/4)
 => [internal] load build definition from Dockerfile                                                                                      0.0s
 => => transferring dockerfile: 31B                                                                                                       0.0s
 => [internal] load .dockerignore                                                                                                         0.0s
 => => transferring context: 2B                                                                                                           0.0s
 => [internal] load metadata for docker.io/library/debian:10                                                                              6.6s
 => [1/1] FROM docker.io/library/debian:10@sha256:68f4e2259032a4e6f5035804e64438b52af8dd5889528b305b9059183ea4cd2a                       89.8s
 => => resolve docker.io/library/debian:10@sha256:68f4e2259032a4e6f5035804e64438b52af8dd5889528b305b9059183ea4cd2a                        0.0s
 => => sha256:68f4e2259032a4e6f5035804e64438b52af8dd5889528b305b9059183ea4cd2a 1.85kB / 1.85kB                                            0.0s
 => => sha256:aaaaf56b44807c64d294e6c8059b479f35350b454492398225034174808d1726 529B / 529B                                                0.0s
 => => sha256:31dd5ebca5efc5e96a425402fa85e492b02c8fe757dfd3edfdea2a7c67322909 2.81MB / 50.39MB                                          75.9s
 => => sha256:ae8514941ea4f23d4948150debf0f92a427c136aa4e7fb85f9c56bba09452572 1.50kB / 1.50kB                                            0.0s

docker info

docker info
Client:
 Debug Mode: false
 Plugins:
  buildx: Build with BuildKit (Docker Inc., v0.4.1)

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 6
 Server Version: 19.03.12
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 4.15.0-88-generic
 Operating System: Ubuntu 18.04.4 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 1
 Total Memory: 3.696GiB
 Name: VM-0-41-ubuntu
 ID: BKQP:QC6G:NOST:VMO6:WIO7:RNCA:GC46:VYWG:AJSP:2KWP:R4L6:REHH
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  127.0.0.0/8
 Registry Mirrors:
  https://wlzfs4t4.mirror.aliyuncs.com/
 Live Restore Enabled: true

WARNING: No swap limit support

[tonistiigi]

@tangx Not sure what registry you're trying to mirror. I see [registry.aliyun] but "aliyun" is not a valid registry host.

[tangx]

@tangx Not sure what registry you're trying to mirror. I see [registry.aliyun] but "aliyun" is not a valid registry host.

@tonistiigi , thank you.

I got it .

• target registry : docker.io,
• local proxy registry mirror : wlzfs4t4.mirror.aliyuncs.com

buildkitd.toml

# registry configures a new Docker register used for cache import or output.
[registry."docker.io"]
  mirrors = ["wlzfs4t4.mirror.aliyuncs.com"]
  # http = true
  # insecure = true
  # ca=["/etc/config/myca.pem"]
  # [[registry."docker.io".keypair]]
  #   key="/etc/config/key.pem"
  #   cert="/etc/config/cert.pem"

command result

# make
docker buildx build --platform=linux/amd64 --load --tag tangx/debian:10 .
[+] Building 22.6s (6/6) FINISHED
 => [internal] load build definition from Dockerfile                                                                                      0.0s
 => => transferring dockerfile: 51B                                                                                                       0.0s
 => [internal] load .dockerignore                                                                                                         0.0s
 => => transferring context: 2B                                                                                                           0.0s
 => [internal] load metadata for docker.io/library/debian:9                                                                              16.6s
 => [1/1] FROM docker.io/library/debian:9@sha256:e984d8384d582e61f128920ec63c18a46bb23bc67d7c29a8311a44a69077edab                         4.5s
 => => resolve docker.io/library/debian:9@sha256:e984d8384d582e61f128920ec63c18a46bb23bc67d7c29a8311a44a69077edab                         0.0s
 => => sha256:9db90d8ea0e09374bbd206cde64e3539f1e761be051cc6d1710070de95e914c3 529B / 529B                                                0.0s
 => => sha256:7e6d8ed603557d9bf077a9ace4ee506501970a4938b9a704f040ad15f22bd4e8 45.37MB / 45.37MB                                          2.3s
 => => sha256:5df937d2ac6c936d827fdb923241c10c7f7e44e89e468c2c7952fc3369fcc873 1.50kB / 1.50kB                                            0.0s
 => => sha256:e984d8384d582e61f128920ec63c18a46bb23bc67d7c29a8311a44a69077edab 1.21kB / 1.21kB                                            0.0s
 => => unpacking docker.io/library/debian:9@sha256:e984d8384d582e61f128920ec63c18a46bb23bc67d7c29a8311a44a69077edab                       1.9s
 => exporting to oci image format                                                                                                         1.5s
 => => exporting layers                                                                                                                   0.0s
 => => exporting manifest sha256:b34997be350208e597f6c6b6e8bd4d62c1d193e61016ca15fd242d31baa52992                                         0.0s
 => => exporting config sha256:4fe8baa05f597001e236089c59633a1688acbc98f6249aac499a0598107259b1                                           0.0s
 => => sending tarball                                                                                                                    1.4s
 => importing to docker                                                                                                                   0.0s

[colachg]

@tangx I have met this problem too. I have catched the network traffic but find no package to mirror.
@tonistiigi Is there any way to check the buildkitd's configuration inside the pod? (I use buildx kubernetes driver with --config to pass the configuration)

[GMartinez-Sisti]

I'm having the same issue. I'm using the buildx action with:

      - name: Set up Docker Buildx
        id: buildx
        uses: docker/setup-buildx-action@v1
        with:
          version: latest
          buildkitd-flags: --config /etc/buildkit/buildkitd.toml

$ cat /etc/buildkit/buildkitd.toml
[registry."docker.io"]
    mirrors = ["nexus.internal.domain"]

$ cat /etc/docker/daemon.json
{
    [...]
    "registry-mirrors": ["https://nexus.internal.domain"],
    [...]
}

I can see that the buildx image is pulled from my private registry, but when buildx is trying to pull images it ignores the registry. Example:

error: failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to create LLB definition: failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/library/alpine/manifests/sha256:69e70a79f2d41ab5d637de98c1e0b055206ba40a8145e7bddb55ccc04e13cf8f: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

I have no idea how to fix this without removing buildx from the build steps. Looks like buildx is ignoring the config, but if I set a wrong key it will complain about it.

#################### edit ####################

Looks like my issue is caused by buildkitd-flags: --config /etc/buildkit/buildkitd.toml. I tested it directly with docker buildx create --config /etc/buildkit/buildkitd.toml --use and it works.

[tonistiigi]

@crazy-max in last comment, does setup-buildx-action have a way to configure config. @GMartinez-Sisti If not then you might need to call buildx create inline.

closing because resolved and last comments offtopic

[tavlima]

@tonistiigi, how would we go about what you mention in this comment?

There isn't currently a way to set proxy settings in buildkit config but it could be added.

I saw @sirlatrom's #170, but that only applies to the docker-container driver, but what I need is that kind of support for the remote driver as well.

But even if similar support is added to the remote driver, I'm still concerned about duplicating the same proxy configs across all Docker daemons/CLIs that use a given buildkitd daemon instance that happens to require proxies. That doesn't look right.

Finally, I'm not sure about all possible implications of having the proxy settings in the buildkitd.toml, but I think it makes sense (assuming callers could override the values).

Thoughts?

Related:

• Documentation not clear around config.json proxy configuration and buildkit docs#15658 (I think docs could be improved)
• Client config.json proxy settings ignored #947 (this is a really useful summary, but doesn't include buildx nor I could find in the official docs)