Error: timed out waiting for file ... internal libpod error with 'podman exec'

[gvalkov]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Running podman exec in a detached container sometimes results in an error, such as this one:

Error: timed out waiting for file /.../containers/storage/overlay-containers/b348cea208bc5e64a6d6a4c9e48ff9677c3bc306489d6faa7964947c624d8303/userdata/22dad53c71f2b4b3ba2827b764df786fa40896c0d1a93254dd1298085cf70e3a/exit/b348cea208bc5e64a6d6a4c9e48ff9677c3bc306489d6faa7964947c624d8303: internal libpod error

It seems to be related to the amount of generated output. We see it pop up in our custom gitlab executor. When we moved one build node to CentOS 8 Stream, CI jobs that generate lots of output started triggering this error.

Steps to reproduce the issue:

1. podman run --detach --tty --name ct-test centos:8 sleep 999999999

2. Execute a long running command that generates lots of output inside the running container. For example:
   podman exec ct-test yes or
   podman --log-level debug exec ct-test /bin/bash -c 'for i in {1..10000000}; do echo $i; done'

Describe the results you received:

2811
2812
DEBU[0009] Container 93dc1fab384b45645cb2a25a47a1252db38b278ca548c7d4a683fca37fac001a exec session 872468edcfb7d0052dd09747d330fb111ceea59f7937f77838b0e584c494bb36 completed with exit code -1 
Error: timed out waiting for file /builds/containers/storage/overlay-containers/93dc1fab384b45645cb2a25a47a1252db38b278ca548c7d4a683fca37fac001a/userdata/872468edcfb7d0052dd09747d330fb111ceea59f7937f77838b0e584c494bb36/exit/93dc1fab384b45645cb2a25a47a1252db38b278ca548c7d4a683fca37fac001a: internal libpod error

Describe the results you expected:

The command should complete and all output should be returned.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:      3.0.0-dev
API Version:  3.0.0
Go Version:   go1.15.7
Built:        Tue Feb  2 23:06:33 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

podman info --debug                                                                                                                                                                                                        [16/03/21 12:06PM 125 ]
host:
  arch: amd64
  buildahVersion: 1.19.2
  cgroupManager: systemd
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.25-1.module_el8.4.0+673+eabfc99d.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.25, commit: 897f4ebd69b9e9c725621fabf1d7c918ef635a68'
  cpus: 32
  distribution:
    distribution: '"centos"'
    version: "8"
  eventLogger: file
  hostname: ci-b.ein.i.gml.ai
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 4.18.0-277.el8.x86_64
  linkmode: dynamic
  memFree: 132216782848
  memTotal: 135035236352
  ociRuntime:
    name: crun
    package: crun-0.17-1.module_el8.4.0+673+eabfc99d.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.17
      commit: 0e9229ae34caaebcb86f1fde18de3acaf18c6d9a
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    selinuxEnabled: true
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 4294963200
  swapTotal: 4294963200
  uptime: 54m 55.26s
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 1
    stopped: 1
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /builds/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageStore:
    number: 3
  runRoot: /run/containers/storage
  volumePath: /builds/containers/storage/volumes
version:
  APIVersion: 3.0.0
  Built: 1612303593
  BuiltTime: Tue Feb  2 23:06:33 2021
  GitCommit: ""
  GoVersion: go1.15.7
  OsArch: linux/amd64
  Version: 3.0.0-dev

Package info (e.g. output of rpm -q podman or apt list podman):

podman-3.0.0-0.33rc2.module_el8.4.0+673+eabfc99d.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Partially. I tried with the latest version of conman (after reading some of the comments in Issue #5339).

Additional environment details (AWS, VirtualBox, physical, etc.):

Physical host. CentOS Stream release 8.

[gvalkov]

Using conmon 2.0.20 (from CentOS 8) on this system seems to resolve the issue.

[rhatdan]

In that case this is a bug in a distro not upstream.

[asdil12]

I have the same issue on a debian 11 (bullseye) system with conmon 2.0.25.

[arodland]

I also had the same issue on debian 11 / conmon 2.0.25. Upgrading to conmon 2.1.0 from sid fixed it. I think that the issue is containers/conmon#236 which was fixed in 2.0.26 by containers/conmon@2b87314 . Distros shipping 2.0.25 in stable may want to apply this fix.

[dabrain34]

Facing the same issue with podman 3.4.4 on ubuntu 22.04. Any recommandation about the version to use of podman or conmon ?