podman run fails inside a container with Error: crun: creating cgroup directory "/sys/fs/cgroup/devices/libpod_parent/***" #23094

tchowice · 2024-04-04T17:54:47Z

tchowice
Apr 4, 2024

Issue Description

I have a Debian 12.5 container with podman 4.3.1 installed. The host is a Redhat 8.5 running podman 4.6.1.

Within the container, when I execute podman run, I get the following error:

Error: crun: creating cgroup directory /sys/fs/cgroup/devices/libpod_parent/libpod-a0442b6cb250349aaca3435df7ef3ca0dfca4f88e885ed68575a0380554cc90d: No such file or directory: OCI runtime attempted to invoke a command that was not found

Steps to reproduce the issue

Start container on host with the following arguments: --sysctl net.ipv4.conf.default.route_localnet=1 --security-opt seccomp=unconfined --security-opt label=disable --security-opt=unmask=/proc/* --security-opt=unmask=/sys/fs/cgroup --systemd=always --cap-add SYS_ADMIN --cap-add SYS_RESOURCE --cap-add NET_ADMIN --cap-add SYS_CHROOT --device /dev/fuse --mount=type=tmpfs,destination=/var/lib/containers --user=root -v /usr/lib/systemd/system/podman.socket:/usr/lib/systemd/system/podman.socket -v /run/systemd/journal/socket:/run/systemd/journal/socket
Within the container, execute podman run --rm -d --name test_container --hostname test_container docker.io/redis:alpine

Describe the results you received

This is the output:

Trying to pull docker.io/library/redis:alpine...
Getting image source signatures
Copying blob sha256:f76326fd8e6b93c5ddd86c0795b0a04c186faf08ce032c102aa9b3671276019a
Copying blob sha256:5c3180d102093de53ebc54b965de6754cbbb344a30e2bf2f5d17cbc3ac1d50b5
Copying blob sha256:034c076ba1e793bb3f31ae4f06a1737a362a34ce9e843c6c15022cfbec955bb9
Copying blob sha256:dffcad17539bc6497d8dd4bd24f6628013eb413b988050010925cb2ce4382291
Copying blob sha256:4abcf20661432fb2d719aaf90656f55c287f8ca915dc1c92ec14ff61e67fbaf8
Copying blob sha256:5913474e0f39b23ca3d952a08c0008364c774a07984efaf8ad3a5ba8e04d31f6
Copying blob sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
Copying blob sha256:cc6fccbbefa3f40f40b56bb79e12ea0d1712b5b2421e04b1e0086c222e43da52
Copying config sha256:435993df2c8d3a1508114cea2dd12ef4d6cbab5c7238bb8e587f20b18982c834
Writing manifest to image destination
Storing signatures
failed to connect to syslog: Initialization
time="2024-04-03T21:44:59-07:00" level=warning msg="Failed to add conmon to cgroupfs sandbox cgroup: creating cgroup for pids: mkdir /sys/fs/cgroup/pids/libpod_parent: read-only file system"
Error: crun: creating cgroup directory `/sys/fs/cgroup/devices/libpod_parent/libpod-a0442b6cb250349aaca3435df7ef3ca0dfca4f88e885ed68575a0380554cc90d`: No such file or directory: OCI runtime attempted to invoke a command that was not found

Describe the results you expected

The redis container should successfully launch on the host.

podman info output

Within the container:

host:
  arch: amd64
  buildahVersion: 1.28.2
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon_2.1.6+ds1-1_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.6, commit: unknown'
  cpuUtilization:
    idlePercent: 99.64
    systemPercent: 0.1
    userPercent: 0.26
  cpus: 4
  distribution:
    codename: bookworm
    distribution: debian
    version: "12"
  eventLogger: file
  hostname: 695ee1fe277f
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 4.18.0-348.20.1.el8_5.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 1551605760
  memTotal: 6232244224
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun_1.8.1-1+deb12u1_amd64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.1
      commit: f8a096be060b22ccd3d5f3ebe44108517fbf6c30
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.2.0-1_amd64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 8355713024
  swapTotal: 8485072896
  uptime: 2537h 54m 49.00s (Approximately 105.71 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries: {}
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs_1.10-1_amd64
      Version: |-
        fusermount3 version: 3.14.0
        fuse-overlayfs: version 1.10
        FUSE library version 3.14.0
        using FUSE kernel interface version 7.31
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 3116122112
  graphRootUsed: 32768
  graphStatus:
    Backing Filesystem: tmpfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.3.1
  Built: 0
  BuiltTime: Wed Dec 31 16:00:00 1969
  GitCommit: ""
  GoVersion: go1.19.8
  Os: linux
  OsArch: linux/amd64
  Version: 4.3.1

On the host:

host:
  arch: amd64
  buildahVersion: 1.31.3
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: Unknown
    path: /usr/bin/conmon
    version: 'conmon version 2.1.8, commit: 21e9be52d14128789284e1bbf54d8c25b4820215'
  cpuUtilization:
    idlePercent: 99.64
    systemPercent: 0.1
    userPercent: 0.26
  cpus: 4
  databaseBackend: boltdb
  distribution:
    distribution: '"rhel"'
    version: "8.5"
  eventLogger: file
  freeLocks: 2046
  hostname: eq1vjenbldl3001
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1302400513
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000854465
    uidmap:
    - container_id: 0
      host_id: 1302439623
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000854465
  kernel: 4.18.0-348.20.1.el8_5.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 3952402432
  memTotal: 6232244224
  networkBackend: cni
  networkBackendInfo:
    backend: cni
    dns: {}
  ociRuntime:
    name: runc
    package: Unknown
    path: /usr/bin/runc
    version: |-
      runc version 1.1.9
      spec: 1.0.2-dev
      go: go1.20.6
      libseccomp: 2.5.1
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    path: /run/user/1302439623/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: Unknown
    version: |-
      slirp4netns version 1.2.1
      commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.1
  swapFree: 8355713024
  swapTotal: 8485072896
  uptime: 2550h 59m 57.00s (Approximately 106.25 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /home/svc_jenkins/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 2
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/svc_jenkins/.local/share/containers/storage
  graphRootAllocated: 123469824000
  graphRootUsed: 4865560576
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 11
  runRoot: /run/user/1302439623/containers
  transientStore: false
  volumePath: /home/svc_jenkins/.local/share/containers/storage/volumes
version:
  APIVersion: 4.6.1
  Built: 1696868155
  BuiltTime: Mon Oct  9 09:15:55 2023
  GitCommit: ""
  GoVersion: go1.20.6
  Os: linux
  OsArch: linux/amd64
  Version: 4.6.1
+ docker info
host:
  arch: amd64
  buildahVersion: 1.31.3
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: Unknown
    path: /usr/bin/conmon
    version: 'conmon version 2.1.8, commit: 21e9be52d14128789284e1bbf54d8c25b4820215'
  cpuUtilization:
    idlePercent: 99.64
    systemPercent: 0.1
    userPercent: 0.26
  cpus: 4
  databaseBackend: boltdb
  distribution:
    distribution: '"rhel"'
    version: "8.5"
  eventLogger: file
  freeLocks: 2046
  hostname: eq1vjenbldl3001
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1302400513
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000854465
    uidmap:
    - container_id: 0
      host_id: 1302439623
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000854465
  kernel: 4.18.0-348.20.1.el8_5.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 3952402432
  memTotal: 6232244224
  networkBackend: cni
  networkBackendInfo:
    backend: cni
    dns: {}
  ociRuntime:
    name: runc
    package: Unknown
    path: /usr/bin/runc
    version: |-
      runc version 1.1.9
      spec: 1.0.2-dev
      go: go1.20.6
      libseccomp: 2.5.1
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    path: /run/user/1302439623/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: Unknown
    version: |-
      slirp4netns version 1.2.1
      commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.1
  swapFree: 8355713024
  swapTotal: 8485072896
  uptime: 2550h 59m 57.00s (Approximately 106.25 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /home/svc_jenkins/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 2
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/svc_jenkins/.local/share/containers/storage
  graphRootAllocated: 123469824000
  graphRootUsed: 4865560576
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 11
  runRoot: /run/user/1302439623/containers
  transientStore: false
  volumePath: /home/svc_jenkins/.local/share/containers/storage/volumes
version:
  APIVersion: 4.6.1
  Built: 1696868155
  BuiltTime: Mon Oct  9 09:15:55 2023
  GitCommit: ""
  GoVersion: go1.20.6
  Os: linux
  OsArch: linux/amd64
  Version: 4.6.1

Podman in a container

Yes

Privileged Or Rootless

Rootless

Upstream Latest Release

No

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

tchowice · 2024-04-11T23:12:59Z

tchowice
Apr 11, 2024
Author

Anyone?

0 replies

2024-05-12T00:07:53Z

github-actions[bot]
bot May 12, 2024

A friendly reminder that this issue had no activity for 30 days.

0 replies

giuseppe · 2024-06-25T11:36:14Z

giuseppe
Jun 25, 2024
Maintainer

sorry for the late reply.

There is no safe delegation for cgroup v1 and you are already running with privileges, so you could just bind mount /sys/fs/cgroup from the host and achieve the same level of safety: -v /sys/fs/cgroup:/sys/fs/cgroup. Does that make any difference?

0 replies

bijianpeng · 2024-09-04T04:33:57Z

bijianpeng
Sep 4, 2024

I have also encountered the same problem, I would like to know if this issue has been resolved?

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

podman run fails inside a container with Error: crun: creating cgroup directory "/sys/fs/cgroup/devices/libpod_parent/***" #23094

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 4 comments

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

podman run fails inside a container with Error: crun: creating cgroup directory "/sys/fs/cgroup/devices/libpod_parent/***" #23094

tchowice Apr 4, 2024

Issue Description

Steps to reproduce the issue

Describe the results you received

Describe the results you expected

podman info output

Podman in a container

Privileged Or Rootless

Upstream Latest Release

Additional environment details

Additional information

Replies: 4 comments

tchowice Apr 11, 2024 Author

github-actions[bot] bot May 12, 2024

giuseppe Jun 25, 2024 Maintainer

bijianpeng Sep 4, 2024

tchowice
Apr 4, 2024

tchowice
Apr 11, 2024
Author

github-actions[bot]
bot May 12, 2024

giuseppe
Jun 25, 2024
Maintainer

bijianpeng
Sep 4, 2024