Permission denied errors using buildkit when Dockerfile is in directory with restricted files

[bossmc]

Steps to reproduce the issue:

1. Put a Dockerfile in /tmp
2. Ensure there's some non-readable/non-accessible folder in /tmp too (e.g a systemd PrivateTmp folder)
3. From somewhere else (e.g. ~/) run DOCKER_BUILDKIT=1 docker build -f /tmp/Dockerfile .

Describe the results you received:

$ DOCKER_BUILDKIT=1 docker build -f /tmp/Dockerfile .
[+] Building 0.1s (2/2) FINISHED                                                                                                                                                                                  
 => [internal] load .dockerignore                                                                                                                                                                            0.0s
 => => transferring context: 55B                                                                                                                                                                             0.0s
 => ERROR [internal] load build definition from Dockerfile                                                                                                                                                   0.0s
 => => transferring dockerfile: 144B                                                                                                                                                                         0.0s
------
 > [internal] load build definition from Dockerfile:
------
failed to solve with frontend dockerfile.v0: failed to resolve dockerfile: failed to build LLB: error from sender: open /tmp/systemd-private-59aef507f5954ec69c62db609c0164a3-ModemManager.service-4ohfhM: permission denied

Describe the results you expected:

The image is built successfully without trying to open other files/folders in /tmp.

Additional information you deem important (e.g. issue happens only occasionally):

Only happens with DOCKER_BUIILDKIT=1, if I'm not using buildkit, everything works fine.

Output of docker version:

$ docker version
Client: Docker Engine - Community
 Version:           19.03.5
 API version:       1.40
 Go version:        go1.12.12
 Git commit:        633a0ea838
 Built:             Wed Nov 13 07:29:52 2019
 OS/Arch:           linux/amd64
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          19.03.5
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.12
  Git commit:       633a0ea838
  Built:            Wed Nov 13 07:28:22 2019
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Output of docker info:

$ docker info
Client:
 Debug Mode: false
 Plugins:
  app: Docker Application (Docker Inc., v0.8.0)
  buildx: Build with BuildKit (Docker Inc., v0.3.1-tp-docker)

Server:
 Containers: 11
  Running: 0
  Paused: 0
  Stopped: 11
 Images: 88
 Server Version: 19.03.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
 runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.4.7-050407-generic
 Operating System: Linux Mint 19.2
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 7.773GiB
 Name: antares
 ID: GEU5:P3UF:ZNZ4:JAS3:667E:A5EN:L3FO:IMQ5:7JZB:2GL6:PHXX:TB32
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 23
  Goroutines: 36
  System Time: 2020-01-10T18:02:30.961875272Z
  EventsListeners: 0
 Username: bossmc
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.):

Running on a VirtualBox VM.

[bossmc]

Further investigation reveals that the docker client is the one failing to open the folder:

$ strace docker build ...
[...]
[pid  9740] openat(AT_FDCWD, "/tmp/systemd-private-59aef507f5954ec69c62db609c0164a3-ModemManager.service-4ohfhM", O_RDONLY|O_CLOEXEC <unfinished ...>
[pid  9740] <... openat resumed> )      = -1 EACCES (Permission denied)
[...]

[bossmc]

Workaround (from @tonistiigi - thanks!) is to do:

$ DOCKER_BUILDKIT=1 docker build -f - . < /tmp/Dockerfile
[+] Building 0.1s (3/3) FINISHED                                                                                                                                                                                  
 => [internal] load build definition from Dockerfile                                                                                                                                                         0.1s
 => => transferring dockerfile: 56B                                                                                                                                                                          0.0s
 => [internal] load .dockerignore                                                                                                                                                                            0.1s
 => => transferring context: 34B                                                                                                                                                                             0.0s
 => exporting to image                                                                                                                                                                                       0.0s
 => => writing image sha256:71de1148337f4d1845be01eb4caf15d78e4eb15a1ab96030809826698a5b7e30

[tonistiigi]

@tiborvass We should make a temporary copy of Dockerfile always, (not only when set from stdin) and send the temp copy instead.

[thaJeztah]

Also related docker/cli#1938

ping @tiborvass PTAL

[cogsy23]

I'm still having this issue on Docker version 20.10.21, build baeda1f.

Adding the folder to .dockerignore doesn't get around the problem.

docker/cli#3043 describes my issue more accurately, but was closed in favour of this issue.

[thaJeztah]

@docker/build PTAL

[tuyen-vuduc]

I had the same issue. Set DOCKER_BUILDKIT=0 works for me.

[antonengelhardt]

Any updates on this? When running with sudo i run into ASL Logging error (other issue: docker/compose#9560).

How do i set DOCKER_BUILDKIT to 0? I use fish shell

[jedevc]

@thaJeztah I think we can transfer this to https://github.com/docker/buildx (buildkit would be better, but we can't transfer between orgs 😢)

I think this is due to the fact that before transferring any files to buildkit, the client walks all files in the local directory using fsutil: https://github.com/tonistiigi/fsutil/blob/9e7a6df4857652bc85225f55e2edb8aece3ecd43/send.go#L147-L168. To avoid the permissions error, we'd need to do some special error handling.

However, this isn't just as simple as just ignoring files in those directories, we'd need to correctly propagate any errors for inaccessible files in case the dockerfile tries to access them.

[thaJeztah]

@jedevc yeah, this may be a tricky one, depending on the situation. Perhaps in some cases we can ignore the failure (i.e., if /inaccessible path is excluded, or perhaps "not used" as part of a COPY / ADD ? but may be challenging to have specific exceptions for that 🤔

Let me transfer this one to the buildx repo.