podman 1.9: regression for rootless containers started via systemd

[dustymabe]

There was a regression introduced in podman 1.9 that causes an issue for a subset of use cases where podman is called in a rootless context before some user specific directories exist. This includes starting podman as a non-root user during systemd units like I do for my IRC client that runs on an FCOS node. A very simple reproducer for this problem is:

variant: fcos
version: 1.0.0
systemd:
  units:
    - name: serial-getty@ttyS0.service
      dropins:
      - name: autologin-core.conf
        contents: |
          [Service]
          # Override Execstart in main unit
          ExecStart=
          # Add new Execstart with `-` prefix to ignore failure
          ExecStart=-/usr/sbin/agetty --autologin core --noclear %I $TERM
          TTYVTDisallocate=no
    - name: bug.service
      enabled: true
      contents: |
        [Unit]
        Description=Test Volumes
        After=network-online.target
        Wants=network-online.target
        [Service]
        User=core
        ExecStart=podman info
        RemainAfterExit=true
        [Install]
        WantedBy=multi-user.target

After logging in the user will then see errors for any podman operation:

[core@localhost ~]$ podman ps -a
Error: could not get runtime: database libpod temporary files directory (tmpdir) "/tmp/run-1000/libpod/tmp" does not match our libpod temporary files directory (tmpdir) "/run/user/1000/libpod/tmp": database configuration mismatch

The problem is documented upstream in containers/podman#5950 and fixed in containers/podman#6125. The fix will be in the podman 1.9.2 release that is coming out today.

The problem exists in all of our current streams of FCOS. As of this writing that is:

• 32.20200505.1.0 (next)
• 31.20200505.2.0 (testing)
• 31.20200420.3.0 (stable)

[dustymabe]

To be clear, the problem is a race condition and running rootless containers via systemd is one way to trigger it. systemd is not part of the bug.

[dustymabe]

In the FCOS meeting today we decided that we'd like to do a new spin of testing to get some soak time on the fix and have it ship to our users in the next stable release (scheduled for next week).

13:29:44        dustymabe | #action dustymabe to get new podman release into testing release so we can fix stable in next weeks releases

[dustymabe]

PR for testing release at coreos/fedora-coreos-config#397

[dustymabe]

The fix for this went into testing stream release 31.20200505.2.1. Please try out the new release and report issues.

[dustymabe]

The fix for this went into stable stream release 31.20200505.3.0.