Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CentOS Container with systemd = "D-Bus connection ..." #5153

Closed
dschier-wtd opened this issue Feb 10, 2020 · 10 comments
Closed

CentOS Container with systemd = "D-Bus connection ..." #5153

dschier-wtd opened this issue Feb 10, 2020 · 10 comments
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.

Comments

@dschier-wtd
Copy link

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Running CentOS Containers with enabled systemd not properly working on Fedora 31 as host.

Steps to reproduce the issue:

As rootless, using CentOS image (daemon)

  1. podman run -d --systemd=true --name c7-systemd centos:7 /sbin/init
  2. podman exec c7-systemd yum install httpd -y
  3. podman exec c7-systemd systemctl start httpd.service
Failed to get D-Bus connection: Operation not permitted
Error: non zero exit code: 1: OCI runtime error

As root, using CentOS image (daemon):

  1. podman run -d --systemd=true --name c7-systemd centos:7 /sbin/init
  2. podman exec c7-systemd yum install httpd -y
  3. podman exec c7-systemd systemctl start httpd.service
Failed to get D-Bus connection: Operation not permitted
Error: non zero exit code: 1: OCI runtime error

As rootless, using CentOS image (interactive):

  1. podman run -it --systemd=true centos:7 /sbin/init
Failed to mount cgroup at /sys/fs/cgroup/systemd: No such file or directory
[!!!!!!] Failed to mount API filesystems, freezing.

As root, using CentOS image (interactive):

  1. sudo podman run -it --systemd=true centos:7 /sbin/init
Failed to mount cgroup at /sys/fs/cgroup/systemd: No such file or directory
[!!!!!!] Failed to mount API filesystems, freezing.

Describe the results you received:

Running centos with systemd is running into oci issues, as far as i can see and seems to have improper permissions somehow.

Describe the results you expected:

Running centos with systemd should be working with the --systemd=true switch

Output of podman version:

podman version 1.8.0

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.13.6
  podman version: 1.8.0
host:
  BuildahVersion: 1.13.1
  CgroupVersion: v2
  Conmon:
    package: conmon-2.0.10-2.fc31.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.10, commit: 6b526d9888abb86b9e7de7dfdeec0da98ad32ee0'
  Distribution:
    distribution: fedora
    version: "31"
  IDMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  MemFree: 9744019456
  MemTotal: 16602099712
  OCIRuntime:
    name: crun
    package: crun-0.12.1-1.fc31.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.12.1
      commit: df5f2b2369b3d9f36d175e1183b26e5cee55dd0a
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  SwapFree: 8371826688
  SwapTotal: 8371826688
  arch: amd64
  cpus: 8
  eventlogger: journald
  hostname: nb01
  kernel: 5.4.17-200.fc31.x86_64
  os: linux
  rootless: true
  slirp4netns:
    Executable: /usr/bin/slirp4netns
    Package: slirp4netns-0.4.0-20.1.dev.gitbbd6f25.fc31.x86_64
    Version: |-
      slirp4netns version 0.4.0-beta.3+dev
      commit: bbd6f25c70d5db2a1cd3bfb0416a8db99a75ed7e
  uptime: 6h 0m 7.4s (Approximately 0.25 days)
registries:
  search:
  - docker.io
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - quay.io
store:
  ConfigFile: /var/home/dschier/.config/containers/storage.conf
  ContainerStore:
    number: 0
  GraphDriverName: overlay
  GraphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-0.7.5-2.fc31.x86_64
      Version: |-
        fusermount3 version: 3.6.2
        fuse-overlayfs: version 0.7.5
        FUSE library version 3.6.2
        using FUSE kernel interface version 7.29
  GraphRoot: /var/home/dschier/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 13
  RunRoot: /run/user/1000
  VolumePath: /var/home/dschier/.local/share/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

podman-1.8.0-2.fc31.x86_64
@openshift-ci-robot openshift-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Feb 10, 2020
@mheon
Copy link
Member

mheon commented Feb 10, 2020

Can you run sudo podman run -it --systemd=true centos:7 /sbin/init --log-level debug (this should enable debug logging in systemd itself, which should tell us what's gone wrong)

@mheon
Copy link
Member

mheon commented Feb 10, 2020

Oh, and are you running as root, or without?

@dschier-wtd
Copy link
Author

@mheon, thanks for your awesomely fast response. I have tested both.

  1. root container
sudo podman run -it --systemd=true --log-level debug centos:7 /sbin/init
DEBU[0000] Reading configuration file "/usr/share/containers/libpod.conf" 
DEBU[0000] Merged system config "/usr/share/containers/libpod.conf": &{{false false false false false true} 0 {   [] [] []}  docker://  crun map[crun:[/usr/bin/crun /usr/sbin/crun /usr/local/bin/crun /usr/local/sbin/crun /sbin/crun /bin/crun /run/current-system/sw/bin/crun] kata-fc:[/usr/bin/kata-fc] kata-qemu:[/usr/bin/kata-qemu] kata-runtime:[/usr/bin/kata-runtime] runc:[/usr/bin/runc /usr/sbin/runc /usr/local/bin/runc /usr/local/sbin/runc /sbin/runc /bin/runc /usr/lib/cri-o-runc/sbin/runc /run/current-system/sw/bin/runc]] [crun runc] [crun] [] [/usr/libexec/podman/conmon /usr/local/libexec/podman/conmon /usr/local/lib/podman/conmon /usr/bin/conmon /usr/sbin/conmon /usr/local/bin/conmon /usr/local/sbin/conmon /run/current-system/sw/bin/conmon] [PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] systemd   /var/run/libpod -1 false /etc/cni/net.d/ [/usr/libexec/cni /usr/lib/cni /usr/local/lib/cni /opt/cni/bin] podman []   k8s.gcr.io/pause:3.1 /pause false false  2048 shm    false false} 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/lib/containers/storage 
DEBU[0000] Using run root /var/run/containers/storage   
DEBU[0000] Using static dir /var/lib/containers/storage/libpod 
DEBU[0000] Using tmp dir /var/run/libpod                
DEBU[0000] Using volume path /var/lib/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] cached value indicated that overlay is supported 
DEBU[0000] cached value indicated that metacopy is being used 
DEBU[0000] cached value indicated that native-diff is not being used 
WARN[0000] Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=true 
DEBU[0000] Initializing event backend journald          
WARN[0000] Error initializing configured OCI runtime kata-runtime: no valid executable found for OCI runtime kata-runtime: invalid argument 
WARN[0000] Error initializing configured OCI runtime kata-qemu: no valid executable found for OCI runtime kata-qemu: invalid argument 
WARN[0000] Error initializing configured OCI runtime kata-fc: no valid executable found for OCI runtime kata-fc: invalid argument 
DEBU[0000] using runtime "/usr/bin/runc"                
DEBU[0000] using runtime "/usr/bin/crun"                
INFO[0000] Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman-bridge.conflist 
INFO[0000] Found CNI network cni-podman1 (type=bridge) at /etc/cni/net.d/cni-podman1.conflist 
INFO[0000] Found CNI network cni-podman2 (type=bridge) at /etc/cni/net.d/cni-podman2.conflist 
WARN[0000] Default CNI network name podman is unchangeable 
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on]docker.io/library/centos:7" 
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@5e35e350aded98340bc8fcb0ba392d809c807bc3eb5c618d4a0674d98d88bccd" 
DEBU[0000] exporting opaque data as blob "sha256:5e35e350aded98340bc8fcb0ba392d809c807bc3eb5c618d4a0674d98d88bccd" 
DEBU[0000] Using bridge netmode                         
DEBU[0000] No hostname set; container's hostname will default to runtime default 
DEBU[0000] Loading seccomp profile from "/usr/share/containers/seccomp.json" 
DEBU[0000] created OCI spec and options for new container 
DEBU[0000] Allocated lock 0 for container 2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf 
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@5e35e350aded98340bc8fcb0ba392d809c807bc3eb5c618d4a0674d98d88bccd" 
DEBU[0000] exporting opaque data as blob "sha256:5e35e350aded98340bc8fcb0ba392d809c807bc3eb5c618d4a0674d98d88bccd" 
DEBU[0000] created container "2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf" 
DEBU[0000] container "2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf" has work directory "/var/lib/containers/storage/overlay-containers/2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf/userdata" 
DEBU[0000] container "2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf" has run directory "/var/run/containers/storage/overlay-containers/2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf/userdata" 
DEBU[0000] New container created "2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf" 
DEBU[0000] container "2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf" has CgroupParent "machine.slice/libpod-2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf.scope" 
DEBU[0000] Handling terminal attach                     
DEBU[0000] Made network namespace at /var/run/netns/cni-00944fa5-e76f-d964-4ff3-d8b3a0ad57c2 for container 2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf 
INFO[0000] About to add CNI network lo (type=loopback)  
DEBU[0000] overlay: mount_data=nodev,metacopy=on,lowerdir=/var/lib/containers/storage/overlay/l/4BXP7WXWETD2SSPFLIW6JRFPWE,upperdir=/var/lib/containers/storage/overlay/98a54959b5b8ed38f4c2818c6a000473e8ad1e88f837578b5916568bf61aa56a/diff,workdir=/var/lib/containers/storage/overlay/98a54959b5b8ed38f4c2818c6a000473e8ad1e88f837578b5916568bf61aa56a/work,context="system_u:object_r:container_file_t:s0:c223,c315" 
DEBU[0000] mounted container "2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf" at "/var/lib/containers/storage/overlay/98a54959b5b8ed38f4c2818c6a000473e8ad1e88f837578b5916568bf61aa56a/merged" 
DEBU[0000] Created root filesystem for container 2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf at /var/lib/containers/storage/overlay/98a54959b5b8ed38f4c2818c6a000473e8ad1e88f837578b5916568bf61aa56a/merged 
INFO[0000] Got pod network &{Name:sad_chandrasekhar Namespace:sad_chandrasekhar ID:2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf NetNS:/var/run/netns/cni-00944fa5-e76f-d964-4ff3-d8b3a0ad57c2 Networks:[] RuntimeConfig:map[podman:{IP: MAC: PortMappings:[] Bandwidth:<nil> IpRanges:[]}]} 
INFO[0000] About to add CNI network podman (type=bridge) 
DEBU[0000] [0] CNI result: Interfaces:[{Name:cni-podman0 Mac:f6:58:ed:cc:0d:f0 Sandbox:} {Name:veth978775bd Mac:82:ef:62:75:63:8c Sandbox:} {Name:eth0 Mac:be:37:a7:0d:7a:cb Sandbox:/var/run/netns/cni-00944fa5-e76f-d964-4ff3-d8b3a0ad57c2}], IP:[{Version:4 Interface:0xc0005d2718 Address:{IP:10.88.0.19 Mask:ffff0000} Gateway:10.88.0.1}], Routes:[{Dst:{IP:0.0.0.0 Mask:00000000} GW:<nil>}], DNS:{Nameservers:[] Domain: Search:[] Options:[]} 
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode secret 
DEBU[0000] Setting CGroups for container 2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf to machine.slice:libpod:2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf 
DEBU[0000] reading hooks from /usr/share/containers/oci/hooks.d 
DEBU[0000] reading hooks from /etc/containers/oci/hooks.d 
DEBU[0000] Created OCI spec for container 2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf at /var/lib/containers/storage/overlay-containers/2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf/userdata/config.json 
DEBU[0000] /usr/bin/conmon messages will be logged to syslog 
DEBU[0000] running conmon: /usr/bin/conmon               args="[--api-version 1 -s -c 2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf -u 2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf -r /usr/bin/crun -b /var/lib/containers/storage/overlay-containers/2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf/userdata -p /var/run/containers/storage/overlay-containers/2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf/userdata/pidfile -l k8s-file:/var/lib/containers/storage/overlay-containers/2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf/userdata/ctr.log --exit-dir /var/run/libpod/exits --socket-dir-path /var/run/libpod/socket --log-level debug --syslog -t --conmon-pidfile /var/run/containers/storage/overlay-containers/2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /var/run/containers/storage --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /var/run/libpod --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mountopt=nodev,metacopy=on --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf]"
INFO[0000] Running conmon under slice machine.slice and unitName libpod-conmon-2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf.scope 
DEBU[0000] Received: 19469                              
INFO[0000] Got Conmon PID as 19465                      
DEBU[0000] Created container 2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf in OCI runtime 
DEBU[0000] Attaching to container 2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf 
DEBU[0000] connecting to socket /var/run/libpod/socket/2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf/attach 
DEBU[0000] Received a resize event: {Width:106 Height:54} 
DEBU[0000] Starting container 2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf with command [/sbin/init] 
DEBU[0000] Started container 2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf 
DEBU[0000] Enabling signal proxying                     
Failed to mount cgroup at /sys/fs/cgroup/systemd: No such file or directory
[!!!!!!] Failed to mount API filesystems, freezing.
  1. rootless container
podman run -it --systemd=true --log-level debug centos:7 /sbin/init
DEBU[0000] Reading configuration file "/var/home/dschier/.config/containers/libpod.conf" 
DEBU[0000] Merged system config "/var/home/dschier/.config/containers/libpod.conf": &{{false false false true true true} 0 {   [] [] []} /var/home/dschier/.local/share/containers/storage/volumes docker://  crun map[crun:[/usr/bin/crun /usr/local/bin/crun] runc:[/usr/bin/runc /usr/sbin/runc /usr/local/bin/runc /usr/local/sbin/runc /sbin/runc /bin/runc /usr/lib/cri-o-runc/sbin/runc /run/current-system/sw/bin/runc]] [crun runc] [crun] [] [/usr/libexec/podman/conmon /usr/local/libexec/podman/conmon /usr/local/lib/podman/conmon /usr/bin/conmon /usr/sbin/conmon /usr/local/bin/conmon /usr/local/sbin/conmon /run/current-system/sw/bin/conmon] [PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] systemd  /var/home/dschier/.local/share/containers/storage/libpod /run/user/1000/libpod/tmp -1 false /etc/cni/net.d/ [/usr/libexec/cni /usr/lib/cni /usr/local/lib/cni /opt/cni/bin]  []   k8s.gcr.io/pause:3.1 /pause true true  2048 shm journald  ctrl-p,ctrl-q false false} 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /var/home/dschier/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/home/dschier/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000                
DEBU[0000] Using static dir /var/home/dschier/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /var/home/dschier/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] No store required. Not opening container store. 
DEBU[0000] Initializing event backend journald          
DEBU[0000] using runtime "/usr/bin/runc"                
DEBU[0000] using runtime "/usr/bin/crun"                
INFO[0000] running as rootless                          
DEBU[0000] Reading configuration file "/var/home/dschier/.config/containers/libpod.conf" 
DEBU[0000] Merged system config "/var/home/dschier/.config/containers/libpod.conf": &{{false false false true true true} 0 {   [] [] []} /var/home/dschier/.local/share/containers/storage/volumes docker://  crun map[crun:[/usr/bin/crun /usr/local/bin/crun] runc:[/usr/bin/runc /usr/sbin/runc /usr/local/bin/runc /usr/local/sbin/runc /sbin/runc /bin/runc /usr/lib/cri-o-runc/sbin/runc /run/current-system/sw/bin/runc]] [crun runc] [crun] [] [/usr/libexec/podman/conmon /usr/local/libexec/podman/conmon /usr/local/lib/podman/conmon /usr/bin/conmon /usr/sbin/conmon /usr/local/bin/conmon /usr/local/sbin/conmon /run/current-system/sw/bin/conmon] [PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] systemd  /var/home/dschier/.local/share/containers/storage/libpod /run/user/1000/libpod/tmp -1 false /etc/cni/net.d/ [/usr/libexec/cni /usr/lib/cni /usr/local/lib/cni /opt/cni/bin]  []   k8s.gcr.io/pause:3.1 /pause true true  2048 shm journald  ctrl-p,ctrl-q false false} 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /var/home/dschier/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/home/dschier/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000                
DEBU[0000] Using static dir /var/home/dschier/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /var/home/dschier/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] using runtime "/usr/bin/crun"                
DEBU[0000] using runtime "/usr/bin/runc"                
DEBU[0000] parsed reference into "[overlay@/var/home/dschier/.local/share/containers/storage+/run/user/1000:overlay.mount_program=/usr/bin/fuse-overlayfs]docker.io/library/centos:7" 
DEBU[0000] parsed reference into "[overlay@/var/home/dschier/.local/share/containers/storage+/run/user/1000:overlay.mount_program=/usr/bin/fuse-overlayfs]@5e35e350aded98340bc8fcb0ba392d809c807bc3eb5c618d4a0674d98d88bccd" 
DEBU[0000] exporting opaque data as blob "sha256:5e35e350aded98340bc8fcb0ba392d809c807bc3eb5c618d4a0674d98d88bccd" 
DEBU[0000] Using slirp4netns netmode                    
DEBU[0000] No hostname set; container's hostname will default to runtime default 
DEBU[0000] Loading seccomp profile from "/usr/share/containers/seccomp.json" 
DEBU[0000] created OCI spec and options for new container 
DEBU[0000] Allocated lock 0 for container b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14 
DEBU[0000] parsed reference into "[overlay@/var/home/dschier/.local/share/containers/storage+/run/user/1000:overlay.mount_program=/usr/bin/fuse-overlayfs]@5e35e350aded98340bc8fcb0ba392d809c807bc3eb5c618d4a0674d98d88bccd" 
DEBU[0000] exporting opaque data as blob "sha256:5e35e350aded98340bc8fcb0ba392d809c807bc3eb5c618d4a0674d98d88bccd" 
DEBU[0000] created container "b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14" 
DEBU[0000] container "b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14" has work directory "/var/home/dschier/.local/share/containers/storage/overlay-containers/b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14/userdata" 
DEBU[0000] container "b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14" has run directory "/run/user/1000/overlay-containers/b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14/userdata" 
DEBU[0000] New container created "b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14" 
DEBU[0000] container "b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14" has CgroupParent "user.slice/user-1000.slice/user@1000.service/user.slice/libpod-b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14.scope" 
DEBU[0000] Handling terminal attach                     
DEBU[0000] Made network namespace at /run/user/1000/netns/cni-82407548-13f1-996e-df2a-7b778125c0d6 for container b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14 
DEBU[0000] overlay: mount_data=lowerdir=/var/home/dschier/.local/share/containers/storage/overlay/l/ARMEVO4WOHWQ5SOH5P5M46ZUPG,upperdir=/var/home/dschier/.local/share/containers/storage/overlay/ece07aa152dcd4949b45c6efe7bfab46cd5c1044147a8e2bea9864208bc339cb/diff,workdir=/var/home/dschier/.local/share/containers/storage/overlay/ece07aa152dcd4949b45c6efe7bfab46cd5c1044147a8e2bea9864208bc339cb/work,context="system_u:object_r:container_file_t:s0:c641,c875" 
DEBU[0000] mounted container "b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14" at "/var/home/dschier/.local/share/containers/storage/overlay/ece07aa152dcd4949b45c6efe7bfab46cd5c1044147a8e2bea9864208bc339cb/merged" 
DEBU[0000] slirp4netns command: /usr/bin/slirp4netns --disable-host-loopback --mtu 65520 -c -e 3 -r 4 --netns-type=path /run/user/1000/netns/cni-82407548-13f1-996e-df2a-7b778125c0d6 tap0 
DEBU[0000] Created root filesystem for container b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14 at /var/home/dschier/.local/share/containers/storage/overlay/ece07aa152dcd4949b45c6efe7bfab46cd5c1044147a8e2bea9864208bc339cb/merged 
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode secret 
DEBU[0000] Setting CGroups for container b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14 to user.slice:libpod:b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14 
DEBU[0000] Created OCI spec for container b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14 at /var/home/dschier/.local/share/containers/storage/overlay-containers/b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14/userdata/config.json 
DEBU[0000] /usr/bin/conmon messages will be logged to syslog 
DEBU[0000] running conmon: /usr/bin/conmon               args="[--api-version 1 -s -c b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14 -u b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14 -r /usr/bin/crun -b /var/home/dschier/.local/share/containers/storage/overlay-containers/b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14/userdata -p /run/user/1000/overlay-containers/b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14/userdata/pidfile -l k8s-file:/var/home/dschier/.local/share/containers/storage/overlay-containers/b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14/userdata/ctr.log --exit-dir /run/user/1000/libpod/tmp/exits --socket-dir-path /run/user/1000/libpod/tmp/socket --log-level debug --syslog -t --conmon-pidfile /run/user/1000/overlay-containers/b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/home/dschier/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1000 --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/1000/libpod/tmp --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mount_program=/usr/bin/fuse-overlayfs --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14]"
DEBU[0000] Received: 20183                              
INFO[0000] Got Conmon PID as 20179                      
DEBU[0000] Created container b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14 in OCI runtime 
DEBU[0000] Attaching to container b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14 
DEBU[0000] connecting to socket /run/user/1000/libpod/tmp/socket/b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14/attach 
DEBU[0000] Received a resize event: {Width:106 Height:54} 
DEBU[0000] Starting container b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14 with command [/sbin/init] 
DEBU[0000] Started container b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14 
DEBU[0000] Enabling signal proxying                     
Failed to mount cgroup at /sys/fs/cgroup/systemd: No such file or directory
[!!!!!!] Failed to mount API filesystems, freezing.

@mheon
Copy link
Member

mheon commented Feb 10, 2020

Ah, sorry, I think you misunderstood - systemd also uses --log-level=debug to print debug logs, so we want it at the end of the command, so it's an argument to systemd, not Podman

@mheon
Copy link
Member

mheon commented Feb 10, 2020

(From looking at the logs, I don't see anything confirming we're in systemd mode, but checking the code, I don't see any indication that we actually log when we are - an oversight on our part)

@dschier-wtd
Copy link
Author

Ahhh, understood. Unfortunately setting "--log-level=debug" for systemd in centos does not give any more details.

Maybeeee, this is os-related. I have tested Fedora and Debian in the meanwhile and both are "somewhat" working.

@mheon
Copy link
Member

mheon commented Feb 10, 2020

Ahh - are you on Fedora 31?

I believe there's a known issue with CentOS 7 (where systemd is too old to support cgroups v2) on top of Fedora 31 (defaults to cgroups v2)

@dschier-wtd
Copy link
Author

that may be a thing, will test it on the cgroup-v1 machine.

@dschier-wtd
Copy link
Author

ok, running

podman run -it --systemd=true --privileged centos:7 /sbin/init

and 

sudo podman run -it --systemd=true --privileged centos:7 /sbin/init

are working with cgroups1. Furthermore, fedora and debian images are working almost perfect, too ;)

That was very helpful, thank.

If I can add something here or support you otherwise, please feel free to give me a ping.

@rhatdan
Copy link
Member

rhatdan commented Feb 11, 2020

Yes, sadly we can not fix this issue. RHEL7/Centos7 systemd containers can not run on a cgroup V2 machine, The systemd is too old.

@rhatdan rhatdan closed this as completed Feb 11, 2020
@markwort markwort mentioned this issue Aug 6, 2021
Open
@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 23, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 23, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@rhatdan @dschier-wtd @mheon @openshift-ci-robot