Having trouble getting started with rootless podman running rootless podman inside a Debian 12 image

[Tronde]

Dear Community,
I would like to create a Debian 12 (Bookworm) image including a rootless Podman installation. I followed the article "How to use Podman inside of a container by Dan Walsh but got stuck. Describing my environment and my issue I hope that someone is able to help me fix it.

My environment

]$ cat /etc/fedora-release
Fedora release 40 (Forty)

]$ podman version
Client:       Podman Engine
Version:      5.2.2
API Version:  5.2.2
Go Version:   go1.22.6
Built:        Wed Aug 21 02:00:00 2024
OS/Arch:      linux/amd64

]$ podman info
host:
  arch: amd64
  buildahVersion: 1.37.2
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.10-1.fc40.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: '
  cpuUtilization:
    idlePercent: 91.98
    systemPercent: 1.74
    userPercent: 6.28
  cpus: 12
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: workstation
    version: "40"
  eventLogger: journald
  freeLocks: 2029
  hostname: jkastnin-thinkpadp1gen3.remote.csb
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 4212606
      size: 1
    - container_id: 1
      host_id: 165536
      size: 165536
    uidmap:
    - container_id: 0
      host_id: 4212606
      size: 1
    - container_id: 1
      host_id: 165536
      size: 165536
  kernel: 6.10.6-200.fc40.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 18288775168
  memTotal: 33390977024
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.12.1-1.fc40.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.12.1
    package: netavark-1.12.1-1.fc40.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.12.1
  ociRuntime:
    name: crun
    package: crun-1.15-1.fc40.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.15
      commit: e6eacaf4034e84185fd8780ac9262bbf57082278
      rundir: /run/user/4212606/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240821.g1d6142f-1.fc40.x86_64
    version: |
      pasta 0^20240821.g1d6142f-1.fc40.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /run/user/4212606/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-2.fc40.x86_64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.5
  swapFree: 8589930496
  swapTotal: 8589930496
  uptime: 2h 14m 32.00s (Approximately 0.08 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
store:
  configFile: /home/jkastnin/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/jkastnin/.local/share/containers/storage
  graphRootAllocated: 510389125120
  graphRootUsed: 141240901632
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 9
  runRoot: /run/user/4212606/containers
  transientStore: false
  volumePath: /home/jkastnin/.local/share/containers/storage/volumes
version:
  APIVersion: 5.2.2
  Built: 1724198400
  BuiltTime: Wed Aug 21 02:00:00 2024
  GitCommit: ""
  GoVersion: go1.22.6
  Os: linux
  OsArch: linux/amd64
  Version: 5.2.2

]$ rpm -q podman
podman-5.2.2-1.fc40.x86_64

The buildah commands I use to build the container image can be found in this gist.

Steps to reproduce the issue

1. Create container image 'debian_bookworm_podman' using buildah_create_debian_bookworm_with_rootless_podman.sh from this gist
2. Execute podman run --rm --security-opt label=disable --user podman --device /dev/fuse debian_bookworm_podman:latest podman system info

This leads to the following error:

time="2024-09-03T09:25:49Z" level=error msg="running `/usr/bin/newuidmap 12 0 1000 1 1 10000 5000`: newuidmap: open of uid_map failed: Permission denied\n"
Error: cannot set up namespace using "/usr/bin/newuidmap": exit status 1

Expected result

The command is successful when issued as root (without --user podman:

]$ podman run --rm debian_bookworm_podman:latest podman system info
time="2024-09-03T09:26:44Z" level=warning msg="Using rootless single mapping into the namespace. This might break some images. Check /etc/subuid and /etc/subgid for adding sub*ids if not using a network user"
host:
  arch: amd64
  buildahVersion: 1.28.2
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: conmon_2.1.6+ds1-1_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.6, commit: unknown'
  cpuUtilization:
    idlePercent: 92.43
    systemPercent: 1.66
    userPercent: 5.91
  cpus: 12
  distribution:
    codename: bookworm
    distribution: debian
    version: "12"
  eventLogger: file
  hostname: 42d15dde834e
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 0
      size: 1
    uidmap:
    - container_id: 0
      host_id: 0
      size: 1
  kernel: 6.10.6-200.fc40.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 18031177728
  memTotal: 33390977024
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun_1.8.1-1+deb12u1_amd64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.1
      commit: f8a096be060b22ccd3d5f3ebe44108517fbf6c30
      rundir: /run/user/0/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/0/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.2.0-1_amd64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 8589930496
  swapTotal: 8589930496
  uptime: 2h 27m 17.00s (Approximately 0.08 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries: {}
store:
  configFile: /root/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 510389125120
  graphRootUsed: 141397819392
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.3.1
  Built: 0
  BuiltTime: Thu Jan  1 00:00:00 1970
  GitCommit: ""
  GoVersion: go1.19.8
  Os: linux
  OsArch: linux/amd64
  Version: 4.3.1

I would expect this to work in rootless mode as well.

What am I doing wrong? Hopefully you are able to help me out here. In case important information is missing, just let me know what you need.

Thanks in advance,
Joerg

[rhatdan]

I would figure your image debian_bookworm_podman:latest is not setup correctly. Perhaps newuidmap and newgidmap are not setup with file capabilities to run with the setuid permissions.

Could you try this with quay.io/podman/stable:latest

We are doing a

rpm --setcaps shadow-utils 2>/dev/null && \

IN there to make sure the permissions are set correctly.

[rhatdan]

What error are you seeing now?

[Tronde]

podman run --rm --user podman --privileged localhost/debian_bookworm_podman:latest podman info executes without error.

podman run --rm --user podman --security-opt label=disable --device /dev/fuse localhost/debian_bookworm_podman:latest podman info still gives me:

time="2024-09-03T21:20:00Z" level=error msg="running `/usr/bin/newuidmap 12 0 1000 1 1 1 999 1000 1001 64535`: newuidmap: write to uid_map failed: Operation not permitted\n"
Error: cannot set up namespace using "/usr/bin/newuidmap": exit status 1

I found the following two articles and think about following them to figure out what capabilities I need to add:

• What capabilities do I really need in my container?
• Find what capabilities an application requires to successful run in a container

AFAIK --privileged gives the container process the same capabilities as the user that launched it. To limit them to the minimum I need to figure out what is needed by the command I like to run in the container, right?

[rhatdan]

@giuseppe ideas?

[rhatdan]

You should only need CAP_SETUID and CAP_SETGID. But I think you also need to --security-opt unmask=proc

[Tronde]

That does not seem to be enough:

]$ podman run --rm --user podman --security-opt label=disable --device /dev/fuse --cap-add=SETUID,SETGID --security-opt unmask=proc localhost/debian_bookworm_podman:latest podman info
time="2024-09-04T06:59:03Z" level=error msg="running `/usr/bin/newuidmap 13 0 1000 1 1 1 999 1000 1001 64535`: newuidmap: write to uid_map failed: Operation not permitted\n"
Error: cannot set up namespace using "/usr/bin/newuidmap": exit status 1

Update 2024-09-04T11:15+02

I did some trial and error adding one capability after another (that seem to make sense to me) and found the following command to work:

$ podman run --rm --user podman --security-opt label=disable --device /dev/fuse --cap-add=setuid,setgid,sys_admin,chown localhost/debian_bookworm_podman:latest podman info
host:
…
security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
…

I'm not sure whether this could be called "good" but anyway it seems better than using --privileged. What do you think?