Error setting extended attributes on "/catatonit" #18543
Comments
|
@AmedeeBulle This is properly better handled by going through the Red Hat channels (i.e. bugzilla) |
Fixes: containers/podman#18543 Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
|
A friendly reminder that this issue had no activity for 30 days. |
|
I think the fix here is to ensure that (Side note: I found it non-obvious that because Anyways in the end, I think containers/storage#657 has introduced a new problem in that |
|
A friendly reminder that this issue had no activity for 30 days. |
|
Is there a workaround for this issue? |
If you don't need IMA, remove the Alternatively, build the pause container as root, save it and load it as the target user. |
|
I think another work around would be to use the k8s pause image instead, i.e. set this in containers.conf |
|
I was hit with the same error on fedora-iot, where it's not convenient to remove and re-install packages. The above k8s pause made me get running. |
|
And now seeing this on rawhide rootless |
|
As of 2024-06-07, I could file a bz against ima-evm-utils, but that seems whiny: it's their package, they can require whatever they want, I think it's our problem if we can't deal with the presence of an installed RPM. @containers/podman-maintainers PTAL, this is likely to blow up in rawhide. |
Do you see the attribute with Of course I agree that podman shouldn't fail and ignore IMA for this case. And from a quick search I couldn't find any fedora change proposals that mention IMA |
Maybe order matters? Need to install podman+catatonit after rpm-plugin-ima? Reproducer:
[edit: update: |
|
I'd argue for a revert of containers/storage#657 though it may be painful at this point...do we know if there are consumers of that? I feel like the use case of injecting IMA into a container build is a "lower level API" that should be opt-in. This could be something like
Yes, almost certainly. |
|
Well the whole thing was added because a user wanted IMA to be copied (containers/buildah#2127) so there are almost certainly users that depend on it. So reverting is a breaking change and we cannot justify that in a minor release IMO. Also I am not sure how IMA works but couldn't a policy just deny all files without the stored hash or is this not a thing? In this case dropping the signatures would be bad too. Also overall it seems to work fine as root so chaining existing behaviour is undesirable. Of course it doesn't work rootless so maybe the easiest non breaking change is to ignore the attribute when running rootless but I am not a storage maintainer so it is more important what they think. |
|
couldn't we just ignore the error when setting the IMA xattr fails with |
|
TL:DR: could we special-case the Or would that be insufficient because ordinary users tend to do If the latter, I think So the situation is that default rawhide systems might be storing IMA signatures (based on data distributed inside RPMs), but they are not enforcing signatures to be present when files are used? In the specific case of the pause image … I’m tempted to say that locally building an image at runtime, when that could happen at distribution build time is clearly suboptimal (or we might not need to create a container image just to be able to create a container, when If IMA were enforcing: First, AFAIK IMA does not authenticate directories, and there are known exploits of that. I don’t know what enabling it buys the user. I’d strongly recommend a more comprehensive authentication mechanism, like dm-verity. My understanding (per containers/storage#1608 (comment) ) is that in general, for rootless pulls / builds we are in a mostly impossible situation with IMA (given current kernel feature set). Also, none of the popular images contain embedded IMA signatures. So we would end up with a container host only being able to run specially-built images. On net, given the weak security value of IMA, and the extra steps necessary to make it enforcing, I think I’m fine with also asking users to set |
|
Friendly ping that this still needs someone to fix it. We added a work around in our CI images but it is really not sustainable that many podman users have to apply a workaround in order to use pods. |
|
We have this on the agenda for the cabal tomorrow BTW |
Most important change is removing the RPM IMA workaround (containers#18543). Let's see if podman still works on rawhide. Also: pasta bumped to 08-21 (from 08-14) Source: containers/automation_images#384 Signed-off-by: Ed Santiago <santiago@redhat.com>
|
Ping, rawhide is close to releasing, and this is still not fixed. Podman users on f41 may be in for a very unpleasant surprise. |
|
We decided to work on removing the pause image entirely, but that sounds like it will take time we may not have. Is there a temporary workaround we can use while that work gets scheduled? |
This issue is affecting me right now on Fedora IoT 40 (x86_64), but @Luap99's suggestion fixed it for me:
So one option for a temporary workaround would be to add a pause image to [engine]
infra_image="registry.fedoraproject.org/pause:41"This does mean that you would need an internet connection the first time that you launch a container, so another option would be to somehow include the image directly in the |
|
You could also pre-pull the pause image into containers-storage, for at least the rootful case. |
|
If you are using a bootc or image mode deployment. |
|
Removing |
|
I think we want to make it selective (root still gets IMA, rootless does not) but I think that's our way forward. |
|
My proposal would be to first make this a storage.conf option like Then the internal pause image build flips on that storage option. |
|
The storage library isn't the only place where we read and write those (as @edsantiago noticed, we have logic in buildah that also handles these there), and I'm not generally fond of the idea of having a configuration setting that needs to be set one way for root and one way for everyone else. |
Hmm but once we had the option, buildah could read it from c/storage right?
Note I'm not currently suggesting that the default value for this would change depending on root vs not; it's more targeting the pause image build (but allowing general configuration). To say it a different way, we would also not inject |
This is specifically for the IMA xattrs, which cannot be set except as root; as rootless, they fail with EPERM. We never noticed this before (likely because IMA xattrs seem uncommon in the wild) but if there is a file in an image with an IMA xattr rootless Podman becomes completely unable to use the image. This is particularly relevant because the catatonit binary Podman uses for building its pause image has started to include an IMA xattr on Fedora Rawhide, which is breaking rootless Podman there rather badly. Since this cannot work as rootless, it seems simplest to try to set the xattr, but tolerate failure iff the error is EPERM and we are not run as root. Fixes: containers/podman#18543 Signed-off-by: Matt Heon <mheon@redhat.com>
Rootless users cannot set the `security.ima` xattr on files (presumably for security reasons, they get an EPERM on trying to do so). We will normally try and preserve that xattr, so when trying to add a file with an IMA xattr to a build on a Buildah without this patch, you get an error. With this patch, the error is downgraded to a warning, as it's better to successfully build with a missing xattr than blocking all builds which want to include the offending file. The urgency on this has become somewhat higher as it seems like F41/Rawhide are installing rpm-plugin-ima by default, which is setting IMA xattrs on some files that Podman relies on - for example, the catatonit binary we use for pid pause images. Without this patch, building the pause image as rootless will always fail on a system with rpm-plugin-ima installed. Fixes: containers/podman#18543 Signed-off-by: Matt Heon <mheon@redhat.com>
Rootless users cannot set the `security.ima` xattr on files (presumably for security reasons, they get an EPERM on trying to do so). We will normally try and preserve that xattr, so when trying to add a file with an IMA xattr to a build on a Buildah without this patch, you get an error. With this patch, the error is downgraded to a warning, as it's better to successfully build with a missing xattr than blocking all builds which want to include the offending file. The urgency on this has become somewhat higher as it seems like F41/Rawhide are installing rpm-plugin-ima by default, which is setting IMA xattrs on some files that Podman relies on - for example, the catatonit binary we use for pid pause images. Without this patch, building the pause image as rootless will always fail on a system with rpm-plugin-ima installed. Fixes: containers/podman#18543 Signed-off-by: Matt Heon <mheon@redhat.com>
Rootless users cannot set the `security.ima` xattr on files (presumably for security reasons, they get an EPERM on trying to do so). We will normally try and preserve that xattr, so when trying to add a file with an IMA xattr to a build on a Buildah without this patch, you get an error. With this patch, the error is downgraded to a warning, as it's better to successfully build with a missing xattr than blocking all builds which want to include the offending file. The urgency on this has become somewhat higher as it seems like F41/Rawhide are installing rpm-plugin-ima by default, which is setting IMA xattrs on some files that Podman relies on - for example, the catatonit binary we use for pid pause images. Without this patch, building the pause image as rootless will always fail on a system with rpm-plugin-ima installed. Fixes: containers/podman#18543 Signed-off-by: Matt Heon <mheon@redhat.com>
Rootless users cannot set the `security.ima` xattr on files (presumably for security reasons, they get an EPERM on trying to do so). We will normally try and preserve that xattr, so when trying to add a file with an IMA xattr to a build on a Buildah without this patch, you get an error. With this patch, the error is downgraded to a warning, as it's better to successfully build with a missing xattr than blocking all builds which want to include the offending file. The urgency on this has become somewhat higher as it seems like F41/Rawhide are installing rpm-plugin-ima by default, which is setting IMA xattrs on some files that Podman relies on - for example, the catatonit binary we use for pid pause images. Without this patch, building the pause image as rootless will always fail on a system with rpm-plugin-ima installed. Fixes: containers/podman#18543 Signed-off-by: Matt Heon <mheon@redhat.com>
Rootless users cannot set the `security.ima` xattr on files (presumably for security reasons, they get an EPERM on trying to do so). We will normally try and preserve that xattr, so when trying to add a file with an IMA xattr to a build on a Buildah without this patch, you get an error. With this patch, the error is downgraded to a warning, as it's better to successfully build with a missing xattr than blocking all builds which want to include the offending file. The urgency on this has become somewhat higher as it seems like F41/Rawhide are installing rpm-plugin-ima by default, which is setting IMA xattrs on some files that Podman relies on - for example, the catatonit binary we use for pid pause images. Without this patch, building the pause image as rootless will always fail on a system with rpm-plugin-ima installed. Fixes: containers/podman#18543 <MH: Cherry picked back to release-1.37, conflicts fixed> Signed-off-by: Matt Heon <mheon@redhat.com>
Bump debian tar timebomb, remove manual crun install as the package is stable now and most importantly remove IMA workaround as the issue[1], we will see if that is true. [1] containers/podman#18543 Signed-off-by: Paul Holzinger <pholzing@redhat.com>
Bump debian tar timebomb, remove manual crun install as the package is stable now and most importantly remove IMA workaround as the issue[1], we will see if that is true. [1] containers/podman#18543 Signed-off-by: Paul Holzinger <pholzing@redhat.com>
Issue Description
This is the same issue as described in #18064, however I discovered that it is not specific to Oracle Linux, It can easily be reproduced in RHEL 9.1 and 9.2.
When IMA signature is present on the catatonit executable (
/usr/libexec/catatonit/catatonitfor RHEL 9.1/usr/libexec/podman/catatonitfor RHEL 9.2), one cannot create a rootless pod (see error below).IMA signatures are set
rpm-plugin-imapackage is already installedIt seems that
rpm-plugin-imaisn't installed by default on RHEL systems, which makes that you don't always run into the issue.As far as I can see, the copy of IMA attributes has been introduced by containers/storage#657.
Since the IMA attributes can only be copied as root, shouldn't we skip the copy (or ignore
EPERMerror) in rootless mode?Steps to reproduce the issue
Describe the results you received
See above
Describe the results you expected
A pod created
podman info output
Podman in a container
No
Privileged Or Rootless
Rootless
Upstream Latest Release
No
Additional environment details
No response
Additional information
No response
The text was updated successfully, but these errors were encountered: