-o=.: open $HOME/.local/share/docker/overlay2/066f6../work/work: permission denied

[fenollp]

Contributing guidelines

• I've read the contributing guidelines and wholeheartedly agree

I've found a bug and checked that ...

• ... the documentation does not mention anything about my problem
• ... there are no open or closed issues that are related to my problem

Description

Running plenty of jobs with build --output=$DIR .. while DOCKER_HOST=ssh://.. is set on a rootless client, executing the jobs on a (non-rootless) buildx remote, I often get export errors:

 => CACHED [product 2/3] RUN     --mount=type=cache,target=/root/.cache/yt-dlp     set -ux  && cmd="yt-dlp --cache-dir /root/.cache/yt-dlp --newline"  && case "https://www.youtube.com  0.0s
 => [product 3/3] RUN     set -ux  && if [ "${DO_NOT_REENCODE:-}" != '1' ]; then       while read -r vid; do         ffmpeg -i "$vid" -force_key_frames 'expr:gte(t,n_forced*3)' _"$  2246.4s
 => [stage-4 1/1] COPY --from=product /app/* /                                                                                                                                           8.1s 
 => ERROR exporting to client directory                                                                                                                                                 91.9s 
 => => copying files 1.44GB                                                                                                                                                             91.9s 
------                                                                                                                                                                                        
 > exporting to client directory:                                                                                                                                                             
------                                                                                                                                                                                        
ERROR: failed to solve: error from receiver: open /home/pete/.local/share/docker/overlay2/066f68c9865b9483974c0ae79d3681c48f6e90eaaf2dc1b5de124591945590e9/work/work: permission denied
Total:	0B

Expected behaviour

I expect the final export to properly download a non-zero sized file back to my buildx client.

Actual behaviour

The whole download happens (progress is shown but export fails right at the end, due to wrong permissions under $HOME/.local/share/docker/overlay2

Buildx version

github.com/docker/buildx v0.12.1 30feaa1

Docker info

Client:
 Version:    25.0.1
 Context:    rootless
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.12.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.24.3
    Path:     /home/pete/.docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 2
 Server Version: 25.0.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: false
  userxattr: true
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 71909c1814c544ac47ab91d2e8b84718e517bb99
 runc version: v1.1.11-0-g4bccb38
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  rootless
  cgroupns
 Kernel Version: 6.5.0-15-generic
 Operating System: Ubuntu 23.10
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.32GiB
 Name: solaralchemist
 ID: 168873ef-edd0-4493-94a4-b1353e896a85
 Docker Root Dir: /home/pete/.local/share/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

WARNING: No cpuset support
WARNING: No io.weight support
WARNING: No io.weight (per device) support
WARNING: No io.max (rbps) support
WARNING: No io.max (wbps) support
WARNING: No io.max (riops) support
WARNING: No io.max (wiops) support
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

Builders list

NAME/NODE  DRIVER/ENDPOINT STATUS  BUILDKIT             PLATFORMS
default                    error                        
rootless * docker                                       
  rootless rootless        running v0.12.4+3b6880d2a00f linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/amd64/v4, linux/386

Cannot load builder default: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Configuration

docker build --build-arg ARGs='https://www.youtube.com/watch?v=zg4Q0xtcFlg' --output=$HOME/ https://github.com/fenollp/dockerhost-tools--yt-dlp.git#cfd938b53f072da5fcdeaa9526435ae4577a25c0

Build logs

2024/01/29 17:45:03 http2: server: error reading preface from client dummy-1: read |0: file already closed
#0 building with "default" instance using docker driver

#1 [internal] load git source https://github.com/fenollp/dockerhost-tools--yt-dlp.git#cfd938b53f072da5fcdeaa9526435ae4577a25c0
#1 CACHED

#2 resolve image config for docker.io/docker/dockerfile:1@sha256:39b85bbfa7536a5feceb7372a0817649ecb2724562a38360f4d6a7782a409b14
#2 DONE 0.0s

#3 docker-image://docker.io/docker/dockerfile:1@sha256:39b85bbfa7536a5feceb7372a0817649ecb2724562a38360f4d6a7782a409b14
#3 CACHED

#4 [internal] load metadata for docker.io/library/alpine@sha256:82d1e9d7ed48a7523bdebc18cf6290bdb97b82302a8a9c27d4fe885949ea94d1
#4 DONE 0.0s

#5 [alpine 1/1] FROM docker.io/library/alpine@sha256:82d1e9d7ed48a7523bdebc18cf6290bdb97b82302a8a9c27d4fe885949ea94d1
#5 DONE 0.0s

#6 [tool 1/3] RUN   --mount=type=cache,target=/var/cache/apk ln -vs /var/cache/apk /etc/apk/cache &&     set -ux  && apk add             ffmpeg             gcc             git             libstdc++             mpv             musl-dev             py3-pip             python3             python3-dev             rtmpdump  && pip install --no-cache-dir                                brotli                                certifi                                phantomjs                                pycryptodomex                                websockets                                xattr  && pip install --no-cache-dir yt-dlp
#6 CACHED

#7 [atomicparsley 1/2] WORKDIR /app
#7 CACHED

#8 [atomicparsley 2/2] RUN     set -ux  && wget https://github.com/wez/atomicparsley/releases/download/20221229.172126.d813aa6/AtomicParsleyAlpine.zip  && unzip AtomicParsleyAlpine.zip
#8 CACHED

#9 [product 3/3] RUN     set -ux  && if [ "${DO_NOT_REENCODE:-}" != '1' ]; then       while read -r vid; do         ffmpeg -i "$vid" -force_key_frames 'expr:gte(t,n_forced*3)' _"$vid" && mv _"$vid" "$vid";       done < <(ls -1 *-???????????.* | grep -vE .json$);     fi
#9 CACHED

#10 [tool 3/3] RUN     set -ux  && echo --force-ipv4 >>/etc/yt-dlp.conf  && echo "--output '%(title).200s-%(id)s.%(ext)s'" >>/etc/yt-dlp.conf  && echo --audio-multistreams >>/etc/yt-dlp.conf  && echo --video-multistreams >>/etc/yt-dlp.conf  && echo --check-formats >>/etc/yt-dlp.conf  && echo --abort-on-error >>/etc/yt-dlp.conf  && echo "--sponsorblock-remove 'sponsor,interaction'" >>/etc/yt-dlp.conf  && echo --embed-chapters >>/etc/yt-dlp.conf  && echo --embed-info-json >>/etc/yt-dlp.conf  && echo --embed-metadata >>/etc/yt-dlp.conf  && echo --embed-subs >>/etc/yt-dlp.conf
#10 CACHED

#11 [product 1/3] WORKDIR /app
#11 CACHED

#12 [product 2/3] RUN     --mount=type=cache,target=/root/.cache/yt-dlp     set -ux  && cmd="yt-dlp --cache-dir /root/.cache/yt-dlp --newline"  && case "https://www.youtube.com/watch?v=zg4Q0xtcFlg" in *' -f '*) ;; *' --format '*) ;; *' --format='*) ;; *) ARGs="--format 'bv*[ext=mp4]+ba[ext=m4a]/b[ext=mp4]/bv*+ba/b' https://www.youtube.com/watch?v=zg4Q0xtcFlg" ;; esac  && cmd="$cmd '$(echo "https://www.youtube.com/watch?v=zg4Q0xtcFlg" | sed "s% %' '%g")'"  && eval $cmd
#12 CACHED

#13 [tool 2/3] COPY --from=atomicparsley /app/AtomicParsley /usr/local/bin/
#13 CACHED

#14 [stage-4 1/1] COPY --from=product /app/* /
#14 CACHED

#15 exporting to client directory
#15 copying files 3.21MB 0.1s
#15 copying files 146.66MB 5.1s
#15 copying files 307.42MB 10.2s
#15 copying files 455.66MB 15.2s
#15 copying files 608.51MB 20.3s
#15 copying files 818.83MB 25.3s
#15 copying files 999.02MB 30.5s
#15 copying files 1.20GB 35.7s
#15 copying files 1.40GB 40.8s
#15 copying files 1.44GB 55.0s done
#15 ERROR: error from receiver: open /home/pete/.local/share/docker/overlay2/066f68c9865b9483974c0ae79d3681c48f6e90eaaf2dc1b5de124591945590e9/work/work: permission denied
------
 > exporting to client directory:
------
ERROR: failed to solve: error from receiver: open /home/pete/.local/share/docker/overlay2/066f68c9865b9483974c0ae79d3681c48f6e90eaaf2dc1b5de124591945590e9/work/work: permission denied

Additional info

The last folder in the path it errors on is world-unreadable:

ls -lha /home/pete/.local/share/docker/overlay2/066f68c9865b9483974c0ae79d3681c48f6e90eaaf2dc1b5de124591945590e9/work/
total 12K
drwx------ 3 pete pete 4.0K Jan 26 18:24 .
drwx--x--- 4 pete pete 4.0K Jan 26 18:25 ..
d--------- 2 pete pete 4.0K Jan 26 18:24 work

[tonistiigi]

Looks like you are trying to write full container rootfs (or maybe even docker state directory, unclear what the working dir in your example is). This likely contains many system files and files that require specific permissions, and you don't have permissions to create such files on the client side.

There is almost never a valid use case to export full container rootfs (1.4GB) like this. If you want to export the files as one squashed bundle instead of exporting image you should use -o type=tar that exports the same files as local output would do but in a tar container so it doesn't need to replicate the individual files on client side.

[fenollp]

Ok so -o type=tar works. I no longer have the above issue.

0 0s ~ λ DOCKER_HOST=ssh://oomphy docker build --build-arg ARGs='https://www.youtube.com/watch?v=zg4Q0xtcFlg' --output=type=tar,dest=$HOME/try.tar https://github.com/fenollp/dockerhost-tools--yt-dlp.git#cfd938b53f072da5fcdeaa9526435ae4577a25c0
[+] Building 0.0s (0/0)  docker:default
[+] Building 63.9s (15/15) FINISHED                                                                                                                                            docker:default
 => CACHED [internal] load git source https://github.com/fenollp/dockerhost-tools--yt-dlp.git#cfd938b53f072da5fcdeaa9526435ae4577a25c0                                                   0.0s
 => resolve image config for docker.io/docker/dockerfile:1@sha256:39b85bbfa7536a5feceb7372a0817649ecb2724562a38360f4d6a7782a409b14                                                       0.0s
 => CACHED docker-image://docker.io/docker/dockerfile:1@sha256:39b85bbfa7536a5feceb7372a0817649ecb2724562a38360f4d6a7782a409b14                                                          0.0s
 => [internal] load metadata for docker.io/library/alpine@sha256:82d1e9d7ed48a7523bdebc18cf6290bdb97b82302a8a9c27d4fe885949ea94d1                                                        0.0s
 => [alpine 1/1] FROM docker.io/library/alpine@sha256:82d1e9d7ed48a7523bdebc18cf6290bdb97b82302a8a9c27d4fe885949ea94d1                                                                   0.0s
 => CACHED [tool 1/3] RUN   --mount=type=cache,target=/var/cache/apk ln -vs /var/cache/apk /etc/apk/cache &&     set -ux  && apk add             ffmpeg             gcc             git  0.0s
 => CACHED [atomicparsley 1/2] WORKDIR /app                                                                                                                                              0.0s
 => CACHED [atomicparsley 2/2] RUN     set -ux  && wget https://github.com/wez/atomicparsley/releases/download/20221229.172126.d813aa6/AtomicParsleyAlpine.zip  && unzip AtomicParsleyA  0.0s
 => CACHED [tool 2/3] COPY --from=atomicparsley /app/AtomicParsley /usr/local/bin/                                                                                                       0.0s
 => CACHED [tool 3/3] RUN     set -ux  && echo --force-ipv4 >>/etc/yt-dlp.conf  && echo "--output '%(title).200s-%(id)s.%(ext)s'" >>/etc/yt-dlp.conf  && echo --audio-multistreams >>/e  0.0s
 => CACHED [product 1/3] WORKDIR /app                                                                                                                                                    0.0s
 => CACHED [product 2/3] RUN     --mount=type=cache,target=/root/.cache/yt-dlp     set -ux  && cmd="yt-dlp --cache-dir /root/.cache/yt-dlp --newline"  && case "https://www.youtube.com  0.0s
 => CACHED [product 3/3] RUN     set -ux  && if [ "${DO_NOT_REENCODE:-}" != '1' ]; then       while read -r vid; do         ffmpeg -i "$vid" -force_key_frames 'expr:gte(t,n_forced*3)'  0.0s
 => CACHED [stage-4 1/1] COPY --from=product /app/* /                                                                                                                                    0.0s
 => exporting to client tarball                                                                                                                                                         63.3s
 => => sending tarball                                                                                                                                                                  63.2s
0 70s ~ λ tar tvf try.tar 
-rw-r--r-- 0/0      1443814029 2024-01-29 17:05 La Commune 1871 (1ère partie) - Peter Watkins-zg4Q0xtcFlg.mp4

However I'm not sure I understand the other elements of your response:

• that docker state dir under home ($HOME/.local/share/docker/overlay/..) isn't something I'm trying to write to (however my -o does point to $HOME). Could that be a problem?
• I'm pretty sure I'm not trying to copy out a rootfs: the stage I copy things from is FROM scratch
• I'd rather not use type=tar as it looses progress info (just says "exporting to client tarball") and I'd like final files to overwrite my local files (or to only copy non-locally available files)

Please clear up these points :)

Note: using --output=/tmp/ I get this (only after downloading all the data):

 => CACHED [stage-4 1/1] COPY --from=product /app/* /                                                                                                                                    0.0s
 => ERROR exporting to client directory                                                                                                                                                 72.8s
 => => copying files 1.44GB                                                                                                                                                             72.7s
------
 > exporting to client directory:
------
ERROR: failed to solve: error from receiver: open /tmp/snap-private-tmp: permission denied

IMO the issue I describe is a bug (it used to work!) BUT I'm nozing that my install may be the culprit.

[fenollp]

Coming back to this with a solution: I changed the output directory to a folder that only contains subpaths that my user can browse without permission issues.
Before, I used --output=type=local,dest= with $HOME but my home is cluttered with many files and some of them somehow make buildx fail. It's probably browsing it and not silently ignoring some errors...

I can probably investigate more. e.g. a tree "$HOME" >/dev/null returns a non-zero exit code.

@tonistiigi I was writing a scratch stage containing just a (large) video file back to my local $HOME. No rootfs. :)