Unprivileged podman inside docker: No such file or directory: OCI not found

[lukash]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

We are using nested unprivileged podman containers inside docker containers in GitHub Actions on the DNF project. Podman occasionally fails with the following error in that environment:

Error: executable file `behave` not found in $PATH: No such file or directory: OCI not found

Note behave is a Python test framework. I've also seen the error happen with other executables, like chown.

Here's a failed run.

Here's the action that runs podman. (the parallel ... command on the last line executes container-test, a wrapper script that executes podman run ... behave ...)

I know this is a very complex setup, I'm trying to create a local reproducer, but so far am stuck on running nested podman inside docker:

# run the top-level docker:
$ sudo docker run --rm -it -v /home/lu/dev/ci-dnf-stack:/ci: --device /dev/fuse --cap-add SYS_ADMIN --privileged my-image bash
# build the podman image:
[root@fac399fe409b /]# export STORAGE_OPTS='overlay2.mount_program=/usr/bin/fuse-overlayfs'
[root@fac399fe409b /]# podman build --force-rm --no-cache -t nst -f Dockerfile /ci
STEP 1: FROM fedora:33
Resolved "fedora" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull registry.fedoraproject.org/fedora:33...
Getting image source signatures
Copying blob 863033e5d074 done  
Copying config 5a4884db42 done  
Writing manifest to image destination
Storing signatures
STEP 2: ENV LANG C.UTF-8
--> 215cafdade0
STEP 3: ARG TYPE=nightly
--> 90a343219b0
STEP 4: RUN set -x &&     echo -e "deltarpm=0" >> /etc/dnf/dnf.conf &&     echo -e "install_weak_deps=0" >> /etc/dnf/dnf.conf
error running container: error from /usr/bin/crun creating container for [/bin/sh -c set -x &&     echo -e "deltarpm=0" >> /etc/dnf/dnf.conf &&     echo -e "install_weak_deps=0" >> /etc/dnf/dnf.conf]: writing file `/sys/fs/cgroup/buildah-buildah123702220/cgroup.procs`: Operation not supported
: exit status 1
Error: error building at STEP "RUN set -x &&     echo -e "deltarpm=0" >> /etc/dnf/dnf.conf &&     echo -e "install_weak_deps=0" >> /etc/dnf/dnf.conf": error while running runtime: exit status 1

Any advice on how to proceed with resolving this is very welcome.

Steps to reproduce the issue:

Wish I had those.

Describe the results you received:

Error: executable file `behave` not found in $PATH: No such file or directory: OCI not found

Describe the results you expected:

No error.

Additional information you deem important (e.g. issue happens only occasionally):

Happens occasionally, initially it seemed somewhat rare but the frequency seems to be increasing to the point it's very annoying to keep re-running the CI jobs.

Output of podman version:

Version:      3.2.0-rc1
API Version:  3.2.0-rc1
Go Version:   go1.16.3
Built:        Wed May  5 21:03:46 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.20.2-dev
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.27-2.fc34.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.27, commit: '
  cpus: 2
  distribution:
    distribution: fedora
    version: "34"
  eventLogger: file
  hostname: bf0738dfc076
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.4.0-1046-azure
  linkmode: dynamic
  memFree: 3322593280
  memTotal: 7292149760
  ociRuntime:
    name: crun
    package: crun-0.19.1-2.fc34.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.19.1
      commit: 1535fedf0b83fb898d449f9680000f729ba719f5
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 4294963200
  swapTotal: 4294963200
  uptime: 9m 25.36s
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay2.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.5.0-1.fc34.x86_64
      Version: |-
        fusermount3 version: 3.10.2
        fuse-overlayfs: version 1.5
        FUSE library version 3.10.2
        using FUSE kernel interface version 7.31
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: overlayfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 15
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 3.2.0-rc1
  Built: 1620248626
  BuiltTime: Wed May  5 21:03:46 2021
  GitCommit: ""
  GoVersion: go1.16.3
  OsArch: linux/amd64
  Version: 3.2.0-rc1

Package info (e.g. output of rpm -q podman or apt list podman):

podman-3.2.0-0.1.rc1.fc34.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Using latest podman in F34 and yes.

Additional environment details (AWS, VirtualBox, physical, etc.):

Fedora 34 inside docker in GitHub Actions.

[rhatdan]

This message is coming from crun. Not sure why, but we have seen problems like this in the past.
@giuseppe thoughts?

[giuseppe]

Overlay on top of overlay so we skip the mount in the inner container? Can you make sure /var/lib/containers is a bind mount and it is not on overlay?

[lukash]

@giuseppe on the host docker container, /var/lib/containers is a directory (not a bind mount AFAICS) on the / filesystem, which is an overlay:

# mount
overlay on / type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/6CEVQOPCACN33Q5KC6AGY4HJBI:/var/lib/docker/overlay2/l/ZD4VWSKIOHSDCQHWLGUZ5RTT64:/var/lib/docker/overlay2/l/Z5Q3KAHBIGPBTG5YCV635WTWPJ:/var/lib/docker/overlay2/l/XIKZUZV5LZKMB7EB6DH2VL4X37:/var/lib/docker/overlay2/l/T3HBZ2W4RQRV5NYUMBGY3ELQJE:/var/lib/docker/overlay2/l/CFJ3NUU2HGKBUSP7DGQRVYK37B:/var/lib/docker/overlay2/l/JA57YEMOQKXWUZC5EC4TWRPN57,upperdir=/var/lib/docker/overlay2/4a9913593d653eddd46ee7ca0ca196335f55aa7e5aebbddb2af300896fc1d009/diff,workdir=/var/lib/docker/overlay2/4a9913593d653eddd46ee7ca0ca196335f55aa7e5aebbddb2af300896fc1d009/work,xino=off)
...

In the guest podman container (as I'm not entirely sure which one you're asking about), /var/lib/containers doesn't exist and the root filesystem mount line is as follows:

# mount
fuse-overlayfs on / type fuse.fuse-overlayfs (rw,noatime,user_id=0,group_id=0,default_permissions,allow_other)
...

[rhatdan]

Add a bind mount from the host into the container.

mkdir /var/lib/mycontainer
docker run -v /var/lib/mycontainer:/var/lib/containers ...

Then when you run podman within the container, it will use /var/lib/containers that is not on an overlay file system.

[lukash]

@rhatdan thanks, I've added the bind mount to our workflow: https://github.com/lukash/ci-dnf-stack/blob/54c049d9936b0fcaa0307862c5fa657558cbf839/.github/workflows/ci.yml#L43

And verified that in the outer docker container the directory is not on the overlay anymore:

# mount
overlay on / type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/5YWDZK3U7MJELDTQ6GIDA35LSU:/var/lib/docker/overlay2/l/VXAJI7WE6PQY6BKAGMLN5GBJI5:/var/lib/docker/overlay2/l/YO7C5KM4LFFVMZJPFH2IXZZDCI:/var/lib/docker/overlay2/l/EZKLXDXGPVC5NZN5XRMKHU7EA3:/var/lib/docker/overlay2/l/R7G3W5NHC7SHGQLRAVPN6F7ETW:/var/lib/docker/overlay2/l/JBM6NHEMUEOIYK2I5ABQ2W4U45:/var/lib/docker/overlay2/l/ZQGV4MENUTPR5VUS3CFIN3T6AN,upperdir=/var/lib/docker/overlay2/9aca796bd9b4d886d56e4554b8bd6d0325d23cc65164ceda36db743d9a985c95/diff,workdir=/var/lib/docker/overlay2/9aca796bd9b4d886d56e4554b8bd6d0325d23cc65164ceda36db743d9a985c95/work,xino=off)
...
/dev/sdb1 on /var/lib/containers type ext4 (rw,relatime,discard)

Looked promising for a few days, but today the bug has hit again: https://github.com/rpm-software-management/ci-dnf-stack/runs/2655353414 (search for "OCI" in the upper right hand corner, this time there are three of them)

[rhatdan]

We have just published

https://www.redhat.com/sysadmin/podman-inside-container
And
https://www.redhat.com/sysadmin/podman-inside-kubernetes

Please read these and see if they help solve your problem. Reopen if you need more information.

[jayaddison]

No need to re-open this issue; however to add information about another case where this is reproducible:

A nektos/act (local GitHub Actions) build workflow for an ARM64 version of grocy-docker:

$ act -P ubuntu-latest=catthehacker/ubuntu:act-20.04 --privileged release --eventpath event.json
...
| error running container: error from crun creating container for [/bin/sh -c set -eux;        adduser -u 82 -D -S -G www-data www-data]: writing file `/sys/fs/cgroup/buildah-buildah395986857/cgroup.procs`: Operation not supported
| : exit status 1
| error building at STEP "RUN set -eux;        adduser -u 82 -D -S -G www-data www-data": error while running runtime: exit status 1
[grocy docker container/build_and_push_latest]   ❗  ::error::Error: buildah exited with code 1%0ATrying to pull docker.io/library/alpine:3.14.0...%0AGetting image source signatures%0ACopying blob sha256:58ab47519297212468320b23b8100fc1b2b96e8d342040806ae509a778a0a07a%0ACopying config sha256:b0e47758dc53e7391c7abf92fd56fd959bf37fb74574feba3d557b4182d15801%0AWriting manifest to image destination%0AStoring signatures%0Aerror running container: error from crun creating container for [/bin/sh -c set -eux;        adduser -u 82 -D -S -G www-data www-data]: writing file `/sys/fs/cgroup/buildah-buildah395986857/cgroup.procs`: Operation not supported%0A: exit status 1%0Aerror building at STEP "RUN set -eux;        adduser -u 82 -D -S -G www-data www-data": error while running runtime: exit status 1%0A

[lukash]

Thank you @rhatdan. It seems we have stopped seeing the errors in the meantime (the history is lost when we re-run a GitHub Actions job so I can't find exactly when). If it starts happening again I'll try to bring over the configuration described in the blog post to see what helps.