buildah rm and rmi giving Error: removing container

[SaurabhAhuja1983]

buildah images
REPOSITORY TAG IMAGE ID CREATED SIZE
c39d3cce8877 2 hours ago 94.3 MB
$ buildah ps
CONTAINER ID BUILDER IMAGE ID IMAGE NAME CONTAINER NAME
39168f968281 * c39d3cce8877 c39d3cce8877-working-container

i am running latest version of buildah and getting following error

buildah rm --all
Error: removing container "c39d3cce8877-working-container": deleting build container "39168f968281331f06c7eff3bf3b05dfee2e033e9e0661674add0859786a3e21": unmounting "/.local/share/containers/storage/overlay/ecda84ed4e6a33f323b88b6f8a4e74d4a2264255eaf5412ff8e042d0e1efb595/merged": invalid argument

buildah rmi --all --force
Error: 1 error occurred:
* unmounting "/.local/share/containers/storage/overlay/ecda84ed4e6a33f323b88b6f8a4e74d4a2264255eaf5412ff8e042d0e1efb595/merged": invalid argument

not sure what changed in 1.31.0 and started causing this error.
Will be happy to provide more information.
I am not able to isolate the problem but it's happening on most of our builds.

[SaurabhAhuja1983]

@Luap99 i was verifying #4869 by updating buildah v 1.31.0 and it worked fine but now looks like some regression happened and we are seeing this error. It was working fine until v.130.0

[Luap99]

Can you provide a full reproducer with the exact buildah commands to reproduce? How was the buildah container created?

[rhatdan]

buildah info would be helpful as well.

[SaurabhAhuja1983]

buildah info
{
    "host": {
        "CgroupVersion": "v2",
        "Distribution": {
            "distribution": "ubuntu",
            "version": "22.04"
        },
        "MemFree": 25517293568,
        "MemTotal": 65891975168,
        "OCIRuntime": "runc",
        "SwapFree": 0,
        "SwapTotal": 0,
        "arch": "amd64",
        "cpus": 16,
        "hostname": "agent-748f579658-znk7q",
        "kernel": "5.15.106-flatcar",
        "os": "linux",
        "rootless": true,
        "uptime": "36h 54m 49s (Approximately 1.50 days)",
        "variant": ""
    },
    "store": {
        "ContainerStore": {
            "number": 0
        },
        "GraphDriverName": "overlay",
        "GraphOptions": [
            "overlay.ignore_chown_errors=true"
        ],
        "GraphRoot": "/home/zr/.local/share/containers/storage",
        "GraphStatus": {
            "Backing Filesystem": "extfs",
            "Native Overlay Diff": "true",
            "Supports d_type": "true",
            "Using metacopy": "false"
        },
        "ImageStore": {
            "number": 0
        },
        "RunRoot": "/tmp/containers-user-1001/containers"
    }
}

[SaurabhAhuja1983]

buildah version
Version:         1.31.0
Go Version:      go1.20.6
Image Spec:      1.1.0-rc.3
Runtime Spec:    1.1.0-rc.3
CNI Spec:        1.0.0
libcni Version:  v1.1.2
image Version:   5.26.1
Git Commit:      d0de60b
Built:           Wed Jul 12 15:51:17 2023
OS/Arch:         linux/amd64
BuildPlatform:   linux/amd64

[SaurabhAhuja1983]

We are using buildah to build container. It's a big app. I will try to isolate the problem today and will try to come up with a small Dockerfile and see if i can reproduce the problem.

But most of our apps are giving this error.

[edsantiago]

Looks like containers/podman#18831 , one of our most popular flakes.

[edsantiago]

I used to see this at least once a day. Have not seen it since containers/storage#1724 two weeks ago. I'm tentatively calling it fixed. Thank you @giuseppe!