Rootless podman in podman works locally but not in k8s

[CarelGreyling]

Hi I am trying to get a rootless podman container running in another (rootless) container. The outer container runs as the jenkins user and is meant to be where I build and test our containerized services. The outer container is meant to run in a k8s cluster using cri-o as container runtime.

I am able to successfully run the outer container and run inner containers as the jenkins user when running it locally using podman. But run into permission issues when running the exact same container in the k8s cluster. What is also strange is that there are differing permissions for the overlay directory in runRoot between the 2 instances even though they run the exact same image. Changing graphDriver to vfs seem to have the same issues.

K8S outputs

Running container

jenkins@podman-rootless:~$ podman run busybox sleep 2
Trying to pull busybox:latest...
Getting image source signatures
Copying blob 3a2e9cc4b126 done
Copying config ba5dc23f65 done
Writing manifest to image destination
Storing signatures
Error: crun: open /proc/sys/net/ipv4/ping_group_range: Permission denied: OCI permission denied

runRoot and graphRoot/overlay ls -l

jenkins@podman-rootless:~$ ls -l /tmp/containers-user-435/containers
total 0
drwxr-xr-x 2 jenkins jenkins   6 Sep 26 15:40 networks
drwx------ 2 root    root      6 Sep 26 15:40 overlay
drwx--x--x 4 jenkins jenkins 150 Sep 26 15:45 overlay-containers
drwx------ 2 jenkins jenkins  54 Sep 26 15:45 overlay-layers
drwx------ 2 jenkins jenkins   6 Sep 26 15:40 overlay-locks

jenkins@podman-rootless:~$ ls -l /home/jenkins/.local/share/containers/storage/overlay
total 0
drwx------ 4 root root  55 Sep 26 15:45 1f52b8fc168f0c0928fa2179771488e290e16a9ec4aedaf26e56b1efd887ed99
drwx------ 4 root root  55 Sep 26 15:40 8b4ec4cbcd32928e99675da4d42db8fb4b18ae92efd88044477dc34df20e74cd
drwx------ 6 root root  69 Sep 26 15:40 95c4a60383f7b6eb6f7b8e153a07cd6e896de0476763bef39d0f6cf3400624bd
drwxr-xr-x 2 root root 108 Sep 26 15:45 l

podman info

jenkins@podman-rootless:~$ podman info
host:
  arch: amd64
  buildahVersion: 1.28.2
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: conmon_2.1.6+ds1-1_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.6, commit: unknown'
  cpuUtilization:
    idlePercent: 97.53
    systemPercent: 0.88
    userPercent: 1.59
  cpus: 4
  distribution:
    codename: bookworm
    distribution: debian
    version: "12"
  eventLogger: file
  hostname: podman-rootless
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 0
      size: 4294967295
    uidmap:
    - container_id: 0
      host_id: 0
      size: 4294967295
  kernel: 5.14.0-427.31.1.el9_4.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 89531879424
  memTotal: 104942047232
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun_1.8.1-1+deb12u1_amd64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.1
      commit: f8a096be060b22ccd3d5f3ebe44108517fbf6c30
      rundir: /run/user/435/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/435/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.2.0-1_amd64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 0
  swapTotal: 0
  uptime: 191h 2m 27.00s (Approximately 7.96 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries: {}
store:
  configFile: /home/jenkins/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 0
    stopped: 2
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/jenkins/.local/share/containers/storage
  graphRootAllocated: 53615788032
  graphRootUsed: 6764498944
  graphStatus:
    Backing Filesystem: overlayfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /tmp/containers-user-435/containers
  volumePath: /home/jenkins/.local/share/containers/storage/volumes
version:
  APIVersion: 4.3.1
  Built: 0
  BuiltTime: Thu Jan  1 10:00:00 1970
  GitCommit: ""
  GoVersion: go1.19.8
  Os: linux
  OsArch: linux/amd64
  Version: 4.3.1

k8s manifest

apiVersion: v1
kind: Pod
metadata:
 name: podman-rootless
spec:
  containers:
    - name: rootless
      imagePullPolicy: Always
      image: jenkins-qa-agent:jdk11-pm
      command:
        - "sh"
        - "-c"
      args:
        - "sleep 1000000"
      securityContext:
        privileged: true
        runAsUser: 435

Local outputs

Running container

jenkins@a82fabf6d6b2:~$ podman run busybox echo "hello world"
Trying to pull busybox:latest...
Getting image source signatures
Copying blob 3a2e9cc4b126 done
Copying config ba5dc23f65 done
Writing manifest to image destination
Storing signatures
hello world

runRoot and graphRoot/overlay ls -l

jenkins@a82fabf6d6b2:~$ ls -l /tmp/containers-user-435/containers
total 4
drwxr-xr-x 2 jenkins jenkins    6 Sep 26 15:46 networks
drwx------ 2 jenkins jenkins   59 Sep 26 15:46 overlay
drwx------ 7 jenkins jenkins 4096 Sep 26 15:47 overlay-containers
drwx------ 2 jenkins jenkins   54 Sep 26 15:47 overlay-layers
drwx------ 2 jenkins jenkins    6 Sep 26 15:46 overlay-locks

jenkins@d6fbaa11312d:~$ ls -l /home/jenkins/.local/share/containers/storage/overlay
total 0
drwx------ 6 jenkins jenkins 69 Sep 26 15:58 95c4a60383f7b6eb6f7b8e153a07cd6e896de0476763bef39d0f6cf3400624bd
drwx------ 4 jenkins jenkins 55 Sep 26 15:58 a803c3479840d5cdfed9b71c1fe61b735ad6ada3346c3479385026787c92a58e
drwxr-xr-x 2 jenkins jenkins 74 Sep 26 15:58 l

Podman info

jenkins@a82fabf6d6b2:~$ podman info
host:
  arch: amd64
  buildahVersion: 1.28.2
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: conmon_2.1.6+ds1-1_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.6, commit: unknown'
  cpuUtilization:
    idlePercent: 97.53
    systemPercent: 0.88
    userPercent: 1.59
  cpus: 4
  distribution:
    codename: bookworm
    distribution: debian
    version: "12"
  eventLogger: file
  hostname: a82fabf6d6b2
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 435
      size: 1
    - container_id: 1
      host_id: 10000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 435
      size: 1
    - container_id: 1
      host_id: 10000
      size: 65536
  kernel: 5.14.0-427.31.1.el9_4.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 89572589568
  memTotal: 104942047232
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun_1.8.1-1+deb12u1_amd64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.1
      commit: f8a096be060b22ccd3d5f3ebe44108517fbf6c30
      rundir: /tmp/podman-run-435/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /tmp/podman-run-435/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.2.0-1_amd64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 0
  swapTotal: 0
  uptime: 191h 4m 0.00s (Approximately 7.96 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries: {}
store:
  configFile: /home/jenkins/.config/containers/storage.conf
  containerStore:
    number: 5
    paused: 0
    running: 0
    stopped: 5
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/jenkins/.local/share/containers/storage
  graphRootAllocated: 53615788032
  graphRootUsed: 6764519424
  graphStatus:
    Backing Filesystem: overlayfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /tmp/containers-user-435/containers
  volumePath: /home/jenkins/.local/share/containers/storage/volumes
version:
  APIVersion: 4.3.1
  Built: 0
  BuiltTime: Thu Jan  1 10:00:00 1970
  GitCommit: ""
  GoVersion: go1.19.8
  Os: linux
  OsArch: linux/amd64
  Version: 4.3.1