[podman] overlay is not supported over overlayfs

[FrankNatoli]

Have x64 Windows VirtualBox running CentOS8.
CentOS8 has no problem using docker or podman.
Containers created for CentOS8 until a few weeks ago had no problem using docker or podman.
Had GitLab crash, regenerate container, and now container image reports podman not available.
Revise Dockerfile to yum install podman in container.
Container image now can run "podman --help" but "podman ps" and "podman info" both report:
"overlay is not supported over overlayfs, a mount program is required; backing file system is unsupported for this graph driver"
This link has apparently relevant suggestions:
https://docs.docker.com/storage/storagedriver/overlayfs-driver/
However, "podman info" on CentOS8 shows no "Storage Driver:" though it does show a "graphStatus" with an xfs Backing Filesystem.
I repeat, podman runs just fine on CentOS8, despite having no "Storage Driver:" but will not run "podman ps" or "podman info" insider container.
I believe something has changed but not sure what.
Help appreciated.

[rhatdan]

The issue is /var/lib/containers within the container is not on an overlay and no longer using fuse-overlayfs because of native overlay support. You could volume mount into the container /var/lib/containers which would fix the problem, or you could force fuse-overlayfs into the container.

Did this happen when you updated to RHEL8.5?

[FrankNatoli]

I am not aware of an intentional update to RHEL8.5.
We had a GitLab crash, and when I regenerated the containers, this problem appeared out of nowhere.
Have just checked.
Inside the container, /var/lib/containers has sub-directories sigstore and storage.
Outside the container, /var/lib/containers has sub-directories sigstore, storage and cache.
"podman info" inside the container reports the overlay complaint.
"podman info" outside the container displays all the info.
Are you suggesting a "-v /var/lib/containers:/var/lib/containers" for the container "podman run" command?
Thanks for your time.

[rhatdan]

I am suggesting -v ANYCONTAINERDIRNOTONOVERLAY:/var/lib/containers:Z
You can do -v /var/lib/containers:/var/lib/containers but will need to disable SELinux if you are using it.

[FrankNatoli]

@rhatdan: sorry for delayed response, but your "-v /var/lib/containers:/var/lib/containers" resolved my problem, many thanks.

[oappicgi]

ide the container, /var/lib/containers has sub-directories sigstore and storage.
Outside the container, /var/lib/containers has sub-directories sigstore, storage and cache.
"podman info" inside the container reports the overlay complaint.

Thanks, "podman -v /var/lib/containers:/var/lib/containers" indeed removed the issue for me in linux mint 20.3

[w7089]

had the same issue in alpine container where I installed podman. What resolved this is adding below 2 lines to Dockerfile:

RUN apk add --no-cache --update  fuse-overlayfs
RUN sed -i 's/#mount_program/mount_program/' /etc/containers/storage.conf

[rhatdan]

You could also just add a volume /var/lib/containers to your Containerfile