database libpod temporary files directory (tmpdir) "/tmp/run-1000/libpod/tmp" does not match our libpod temporary files directory (tmpdir) "/run/user/1000/libpod/tmp": database configuration mismatch

[adelton]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

I've first run rootless podman via su - user session from root. It was giving me warning messages

WARN[0000] The cgroupv2 manager is set to systemd but there is no systemd user session available 
WARN[0000] For using systemd, you may need to login using an user session 
WARN[0000] Alternatively, you can enable lingering with: `loginctl enable-linger 1000` (possibly as root) 
WARN[0000] Falling back to --cgroup-manager=cgroupfs    
WARN[0000] The cgroupv2 manager is set to systemd but there is no systemd user session available 
WARN[0000] For using systemd, you may need to login using an user session 
WARN[0000] Alternatively, you can enable lingering with: `loginctl enable-linger 1000` (possibly as root) 
WARN[0000] Falling back to --cgroup-manager=cgroupfs    

but it worked.

I then thought, OK, I will log in to the machine properly. I've logged in via ssh. But then podman fails with

Error: could not get runtime: database libpod temporary files directory (tmpdir) "/tmp/run-1000/libpod/tmp" does not match our libpod temporary files directory (tmpdir) "/run/user/1000/libpod/tmp": database configuration mismatch

Steps to reproduce the issue:

1. Create fresh user via useradd user
2. su - user to the user and run podman images
3. ssh user@machine and run podman images

Describe the results you received:

2. Gives me

WARN[0000] The cgroupv2 manager is set to systemd but there is no systemd user session available 
WARN[0000] For using systemd, you may need to login using an user session 
WARN[0000] Alternatively, you can enable lingering with: `loginctl enable-linger 1000` (possibly as root) 
WARN[0000] Falling back to --cgroup-manager=cgroupfs    
WARN[0000] The cgroupv2 manager is set to systemd but there is no systemd user session available 
WARN[0000] For using systemd, you may need to login using an user session 
WARN[0000] Alternatively, you can enable lingering with: `loginctl enable-linger 1000` (possibly as root) 
WARN[0000] Falling back to --cgroup-manager=cgroupfs    
REPOSITORY   TAG   IMAGE ID   CREATED   SIZE

3. gives me

Error: could not get runtime: database libpod temporary files directory (tmpdir) "/tmp/run-1000/libpod/tmp" does not match our libpod temporary files directory (tmpdir) "/run/user/1000/libpod/tmp": database configuration mismatch

Describe the results you expected:

No error, I shouldn't be bothered by podman/libpod picking different location in each case. Also, similar to #5932, the error message gives no indication about how to fix this, where that database libpod temporary files directory is configured.

Additional information you deem important (e.g. issue happens only occasionally):

Deterministic. Can be re-reproduced by removing ~/.local and repeating steps 2 and 3.

Output of podman version:

Error: could not get runtime: database libpod temporary files directory (tmpdir) "/tmp/run-1000/libpod/tmp" does not match our libpod temporary files directory (tmpdir) "/run/user/1000/libpod/tmp": database configuration mismatch

Output of podman info --debug:

Error: could not get runtime: database libpod temporary files directory (tmpdir) "/tmp/run-1000/libpod/tmp" does not match our libpod temporary files directory (tmpdir) "/run/user/1000/libpod/tmp": database configuration mismatch

Package info (e.g. output of rpm -q podman or apt list podman):

podman-1.9.0-1.fc31.aarch64

Additional environment details (AWS, VirtualBox, physical, etc.):

Physical.

[rhatdan]

@mheon was just talking about this exact situation inside of a bugzilla. Where the location of the temporary files is being stored in the database. And we don't have a good way of solving this situation.

[adelton]

@rhatdan Do you have the bugzilla link?

[rhatdan]

Not right now, but when @mheon joins, he will point to it.

[mheon]

I actually don't think this is that BZ. As I look at this more, I start to think that this is a containers.conf breakage.

@adelton Do you have a ~/.config/containers/libpod.conf and can you provide the contents if you do?

[mheon]

Thought process here: this looks like a Podman config originally created with a really old version, using a non-standard tmpdir path. I think that path is somehow being overwritten now.

[adelton]

I don't have any ~/.config/containers/libpod.conf.

The podman / libpod was freshly installed on this Fedora.

[mheon]

Hm.

Is this the first time you've run Podman as this user? It could actually be the BZ after all.

[rhatdan]

@mheon he is saying he logged in first from root and did a su - user, This succeeded in creating a container. Then he logged in directly using gdm or sshd, and it blows up. I believe this is because the su - user records it temporary storage differently then the true login users.

[mheon]

Yeah, this is definitely the same case as the BZ, then - BZ number looks to be 1824063

[marusak]

We hit this also in cockpit tests:
We take clean and fresh fedora VM, login as root and do:

sudo -i -u admin bash << EOF                                                    
podman pull docker.io/busybox                                                   
podman pull docker.io/alpine                                                    
podman pull docker.io/registry:2                                                
EOF

then we shut the VM down, boot it back up, login as admin user and when we try to do any non-root action it fails the same way:

$ podman images
Error: could not get runtime: database libpod temporary files directory (tmpdir) "/tmp/run-1000/libpod/tmp" does not match our libpod temporary files directory (tmpdir) "/run/user/1000/libpod/tmp": database configuration mismatch

It does not need to be even shut down and log in as admin, it is enough to relogin as admin and run any command.
Easily reproducible with podman-1.9.0-1.fc31.x86_64, worked with previous version.

[martinpitt]

IMHO the cleanest solution is to actually do what the warning says, and run

loginctl enable-linger 1000

or

loginctl enable-linger $(id -u targetusername)

In our previous version of the test I just tried to set $XDG_RUNTIME_DIR appropriately and mkdir it, but that's not sufficient any more. Starting a proper logind session with su - avoids all of that hackery.

[marusak]

Two possible workarounds:

1. Executing loginctl enable-linger $(id -u admin) before doing sudo -i -u admin ...
2. Or once you broke it (not using 1.) and log in as admin and actually see this issue, doing rm ~/.local/share/containers/storage/libpod/* solves the problem.

[jlebon]

Just to make sure we understand the impact, this only affects rootless containers, right?

[mheon]

This can affect root containers as well, but we're using tmpfs directories not managed by systemd there, so it's a lot harder to hit - you could deliberately produce a configuration that would hit it, but the default config never should.

[hynd]

I've been running into the same error (tho possibly via a different cause - starting rootless containers from the "system" systemd with User=xxx)...

It feels it could be some kind of race between a rootless podman initialising before systemd-logind has the opportunity to create/mount /run/user/$UID (via the user-runtime-dir service). If /run/user/$UID isn't mounted, i believe podman falls back to using /tmp/run-$UID (https://github.com/containers/common/blob/master/pkg/config/util_supported.go#L40) and stores that in boltdb, which then causes the mismatch error next time podman's invoked by the user, when /run/user/$UID is indeed mounted?

[mheon]

I think that's probably accurate. I know that we've definitely had issues with running with the User directive without having lingering enabled for the user in question. It's unclear to me whether it's actually a race, or systemd is simply not creating the directories at all (not interpreting this as a "real" login); regardless, the end result is the same (we don't have /run/user/$UID and try to fall back, but can't)

[dustymabe]

I am reproducing this on the Fedora CoreOS next stream. I am using a systemd unit to start a rootless container by using the following in my systemd unit:

[Service]             
# Run as non-root user
User=core

Version info:

$ rpm -q podman systemd
podman-1.9.1-1.fc32.x86_64
systemd-245.4-1.fc32.x86_64

[hynd]

I am reproducing this on the Fedora CoreOS next stream. I am using a systemd unit to start a rootless container by using the following in my systemd unit:

[Service]             
# Run as non-root user
User=core

Yep, that's pretty much what i'm doing (with linger enabled on the User already too). Not sure if it's the same kinda steps that @adelton is triggering it with tho?

I see /run/user/$UID created on boot as the unit fires, but it's hard to tell the exact order of things stopping/starting from journald alone. Adding an After=systemd-logind.service to the systemd unit seems to pretty reliably avoid the problem, but i'm not sure if it's actually waiting for /run/user/$UID to mount, or just shrinking the window of opportunity for any race...

[mheon]

To update with status from a discussion in IRC, we think we've identified the cause of the race, and @rhatdan is looking into a fix

[dustymabe]

The proposed change in #6125 seems to fix the issue for me.

[hynd]

Neat. I can see how it fixes the issue, just wondering if having podman's tmpdir set differently depending on the state of the filesystem the first time it was invoked could be a bit nondeterministic/confusing? (or should the tmpdir not be something we have to care much about?)

[mheon]

The DB should actually override in that case - we'll be forced to use the original temporary directory. If it actually doesn't exist, we'll still blow up, but since this seems to be a race, we should be fine.

[rhatdan]

Since #6125 is merged we can close this. This will be in podman-1.9.2 which should be done early next week.

[dustymabe]

should we close this (fixed by #6125) and maybe link to the conmon PR that fixed it originally?

[dustymabe]

should we close this (fixed by #6125) and maybe link to the conmon PR that fixed it originally?

darn.. commented on a stale page.