-
Notifications
You must be signed in to change notification settings - Fork 59
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
RFE: safe SELinux policy updates #701
Comments
If you need to install a custom SELinux policy, currently the safest way to do this is to package it in an RPM with a scriptlet to |
This comment has been minimized.
This comment has been minimized.
What's the workaround for this?
As in, how to do this non-persistently so I still get selinux policy updates? |
Hi guys, I'm a member of the Red Hat SELinux team and have been assigned to try to find a solution for this issue. I'm able to reproduce it on a FCOS VM, for example by installing a dummy SELinux module and doing an Basically, it should be enough to run I found some code in rpm-ostree that appears to do something like this, but it doesn't seem to help in this case. I would welcome any pointers/explanations/documentation links that could help me understand the problem better. Also if you already have any ideas or outlines for a potential solution, please share it here and hopefully we can brainstorm something together. It is our priority to do as much as we can to help solve this, so I'm offering myself as a resource :) |
That code only runs on the main "base image build" side, not the client side currently. And the reason for that is what we really want here is conditional logic:
(When generating a base image/ostree-commit, we know we need to compile the policy so it's unconditional) To say this another way, we don't want everyone without local policy customizations to pay the cost of recompiling policy on update - a core principle here we're aiming for with rpm-ostree is "image based updates" by default where client systems are just replicating an image and not running lots of arbitrary code as root on each machine.
And yes, this is the other problem. Right now we handle
One approach may be to speculatively recompile the policy at Or alternatively, detect this during the next boot and reconcile at that time. |
Ok, thanks for the explanations, things are starting to click into place.
Oh, so I missed the fact that the policy customizations might be added/changed between staging the upgrade and rebooting... That makes things a bit more tricky...
This looks like the most appealing option to me. We can't fully avoid delaying the reboot either way and I think delaying boot is a bit better than delaying shutdown. But here the question is when to rebuild the policy. Systemd loads the policy when switching root, so ideally we would rebuild policy before that (e.g. by chrooting into /sysroot and relabeling /etc/selinux), which is tricky as we'd need to add some script/unit to initrd (OTOH, ostree already seems to have some hook(s) there, so perhaps this is not a big deal). If we rebuild policy after switching root, then there will be a window of time when the system would run with the uncustomized policy and I'm not sure if we can easily prevent systemd from starting other services before the new policy is built and loaded... I took a shot at the rebuild-before-switch-root approach using this very rough PoC: It is of course nowhere near production ready and doesn't yet have any logic to detect when to rebuild, but it proves that it is feasible to do the rebuild before switching root. One catch though is that rpm-ostree doesn't update the initrd automatically unless the user runs Assuming this is an acceptable approach, there is then the question of where and how to best implement this. Maybe it could be just added to |
I think the most robust approach is to make the RPM case work well and easy to do as explained in comment #701 (comment). This will make this change explicit in rpm-ostree (overlayed package) and keep the policy rebuild happening at an opportune time: as part of the local rpm-ostree compose during updates. For that to work well, we need SELinux userspace tools to be able to easily produce those RPMs instead of directly doing policy changes, especially in a binary form as it is done right now by semanage. If we can have a declarative, plain text way of configuring booleans this also becomes simpler. Removing all dependencies to Python for those tools would also be a big plus for us too. Sorry if this is starting to look like a wish-list. |
Also related: https://github.com/overdrop/overdrop-sebool |
I had thought of that option (provide tools that generate RPMs for customizations and install/remove/update those), but it seems somewhat overcomplicated to generate a spec file, run Ensuring that whatever is in /etc on boot is reflected in the policy that gets loaded on boot feels to me like the most direct way to address the problem. With classic services, you expect that when they are started, they load the present configuration, refreshing any cached state if needed. This seems to be one of the principles that CoreOS/ostree relies on, so IMHO it makes sense to make SELinux policy management (which you can think of as a special kind of "service" started on boot) more in line with this principle.
Yes, as a follow up it would be great to refactor the SELinux tooling to allow having the policy and various bits configuration split between /usr and /etc, but that on its own doesn't solve the issue with SELinux policy package updates and thus I would consider it a separate RFE.
Let's focus on one thing at a time, please :) But feel free to reach out internally to our team's PO (Lukas Vrabec, @wrabcak) to ensure that these tasks are properly tracked on our side. (FWIW, we are already tracking a requirement to remove libsemanage -> policycoreutils executable calls, though it's currently parked.) |
Also not a fan of generating RPMs for this. Feels like if we have room for experimentation, we should aim for something much nicer. Though I also agree that ideally we'd do this some time before rebooting, and even more ideally at My strawman would be something like:
So from the SELinux side, the only main work needed would be the no-op detection bit. WDYT? |
That said, overall I'm not opposed to the approach you've taken in your POC to regenerate on boot. As you've said, the conditional rebuild bit would also be needed there, so it sounds like it could be a stepping stone to something like #701 (comment) ? |
@jlebon Sounds good to me. I personally dislike the idea of preparing for the next boot during shutdown (on a workstation an impatient user might force a power-off when the shutdown takes longer than a few seconds, not to mention losing changes in case of a power outage), but since it's already the status quo, I'm not going to argue against it :) As you point out, the common first step is to detect and avoid the redundant policy rebuilding. I was hoping I could take a shortcut in the rebuild-needed detection by doing something like I had an idea to implement the rebuild-on-boot functionality behind a new libsemanage function (the rebuild being opt-in via |
To be clear, most of the time the work will happen at In general though, the OSTree model is all about preparing the next deployment in the background and atomically switching to it on reboot. So it's very resilient to power outages. :) If e.g. power goes out while the policy is being built for a new deployment, we'll just reboot into the same deployment we were in, without any loss (other than the CPU time we wasted in preparing the lost deployment).
Cool, SGTM! And of course, this would also be useful for non-OSTree based systems too.
It's an interesting avenue, though my concern there before opting in would still be boot delays and potential failure. Also, AFAICT the store root on non-OSTree systems is still |
Tangentially related, with https://github.com/coreos/enhancements/blob/main/os/coreos-layering.md we will also want to run the policy build as a finalization step. Also relates to ostreedev/ostree-rs-ext#159 |
Update: I have submitted patches implementing the policy rebuild optimization upstream and I think they are nearing their final form w.r.t. API/CLI: Basically, passing I have built libsemanage and policycoreutils with the proposed patches in COPR in case someone would like to try them out: I also briefly looked into what it would take to add the policy rebuild step into
What do you think would be better? (Or can you think of a better option?) Also, would you like me to draft some patches to implement this, or would one of you rather like to do it themselves? (If it should be me, then please give me some hints on how it should be implemented :) |
Awesome! Thanks a lot for working on this.
The first option SGTM. I don't see why libostree couldn't learn to use bubblewrap to run things against the pending deployment. We could even then upstream the (As you see there, we have some Rust code in rpm-ostree that wraps |
I agree the right long term direction is to drive more of what's in rpm-ostree into ostree, including the containerization bits. However...at a deeply practical level, the simplest way to go about this right now is probably |
(Tangentially related, I've been thinking we should move towards depending on |
Chatted with @cgwalters OOB about this. We think that just @WOnder93 If you're interested in working on this, that'd be great. Feel free to reach out on IRC or directly if you need any help! |
We had a chat about this, I want to note this whole issue heavily intersects with https://github.com/coreos/enhancements/blob/main/os/coreos-layering.md in that we will definitely want to support people changing SELinux booleans inside their container builds. It's an entirely different fix for this problem (and honestly, I think one that most non-single-system deployments will strongly prefer). |
@jlebon Ack, I've already started working on it. Hoping to have a PR in a couple of days... |
Whenever the user has SELinux enabled and has any local modules/modifications installed, it is necessary to rebuild the policy in the final deployment, otherwise ostree will leave the binary policy files uchanged from last deployment as it detects difference against the base content (in rpm-ostree case this is the RPM content). To avoid the situation where the policy binaries go stale once any local customization of the policy is made, try to rebuild the policy as part of sysroot_finalize_deployment(). Use the special --rebuild-if-modules-changed switch, which detects if the input module files have changed relative to last time the policy was built and skips the most time-consuming part of the rebuild process if modules are unchanged (thus making this a relatively cheap operation if the user hasn't made any modifications to the shipped policy). Partially addresses: coreos/fedora-coreos-tracker#701 Signed-off-by: Ondrej Mosnacek <omosnace@redhat.com>
Whenever the user has SELinux enabled and has any local modules/modifications installed, it is necessary to rebuild the policy in the final deployment, otherwise ostree will leave the binary policy files uchanged from last deployment as it detects difference against the base content (in rpm-ostree case this is the RPM content). To avoid the situation where the policy binaries go stale once any local customization of the policy is made, try to rebuild the policy as part of sysroot_finalize_deployment(). Use the special --rebuild-if-modules-changed switch, which detects if the input module files have changed relative to last time the policy was built and skips the most time-consuming part of the rebuild process if modules are unchanged (thus making this a relatively cheap operation if the user hasn't made any modifications to the shipped policy). Partially addresses: coreos/fedora-coreos-tracker#701 Signed-off-by: Ondrej Mosnacek <omosnace@redhat.com>
Downstream testing result refer to https://bugzilla.redhat.com/show_bug.cgi?id=2057497#c33 Is there any plan to do the fix? If can download old image like 36.20220430.1.0 from https://getfedora.org/en/coreos?stream=next, then I shall do testing to check if can reproduce on fcos |
If we wanted to simplify things we don't have to fully test it on every upgrade test. I think it would suffice to:
Asking more questions: It used to be pretty simple to tell if I (the user) had made custom policy modifications with How can I tell if I've made a local policy modification and it's been rebuilt with the latest policy delivered by the OSTree? |
Thanks @WOnder93 for the help to create https://bugzilla.redhat.com/show_bug.cgi?id=2089802 to track the problem |
As identified by the team (see BZ in previous comment) right now there is a gap where sebooleans won't get detected as changed and carried forward. We'll hold marking this as fully fixed until BZ#2089802 gets fixed. |
Hi, this issue basically blew up our OKD4 cluster upgrade because we have to apply 2 custom .cil selinux modules to our nodes because of some weird 3rd party tools we need to use in our CI containers. This is what it looks like after a manual policy build and after restoriung the binary policy file. Can you please advise how we can apply our 2 custom modules without touching the central policy file permanently? Can modules be applied non-permanently after boot e.g. through a systemd one-shot file? I know this is possible for booleans when omitting the -P flag. We urgently need those custom modules, but automatic OKD4 upgrades also must work. Any help is appreciated. Edit: Is it e.g. possible to compile our rules into a seperate policy binary which does not touch the delivered ostree one so we don't interfere with it? |
Once OKD rebases to an FCOS that includes the SELinux patches, you should be good to go. Until then, the most supported path would be to package the SELinux modules into an RPM and |
Thanks to @WOnder93's work, the SELinux policy is now recompiled when necessary. There's more work we could do here, like some of the ideas in #701 (comment), but the main issue is fixed so I'm going to close this. We can track follow-ups in separate tickets. Feel free to reopen if you disagree! |
So applying custom SELinux policies will no longer prevent further upstream policy updates from applying on Silverblue? Starting from which versions of the involved packages? Sorry, it's not quite clear to me from the discussion... |
Yeah to be sure if this can be closed I need to simple statements:
If those 2 are true, it's fine. If not, we need to reopen. |
Correct.
The ostree side is in v2022.3, which entered f36 6 months ago. The policycoreutils side was fixed in this update, which shipped 9 months ago.
Correct to both. Both actions will trigger policy rebuilds. From that point on, on every update the policy will be recompiled as needed to ensure base policy updates are baked in. |
Wait, I just realized we never added tests for this (conversation about that starts at #701 (comment)). Let's definitely make sure we do that before marking this as done. |
Thanks for the feedback, fantastic, taht is what was needed. But I agree, tests should be added to ensure this does not break, as disruptions to this usually cause very weird and hard to find dailures on production systems. |
Is there any doc or howto for custom selinux policies? |
you just do it based on the official docs of your distro or selinux themselves it is no different than e.g. doing it on RHEL 8 |
Has fixes for selinux UX. See coreos/fedora-coreos-tracker#701
Has fixes for selinux UX. See coreos/fedora-coreos-tracker#701
Right now SELinux stores its policy as binary files in
/etc/selinux/
and that's how it's delivered in the images we build/ship. If a user updates the SELinux policy with a local persistent modification (such as runningsetsebool -P
) then the binary files in/etc/selinux/
get overwritten and now they are considered to be "changed" byostree
and are no longer updated when a system upgrades because files in/etc/
are "configuration".This means if people make a permanent local policy modification they cease to get SELinux policy updates, which is bad.
To workaround this for now people can apply their local policy modifications non persistently on every boot. i.e., something like:
However, we need to solve this problem more generically. Here is a smathering of issues where this issue is touched on:
container_manage_cgroup
SELinux boolean by default #397A while back we had a conversation with the SELinux team about how to solve this problem in the future. I'm planning to take those notes and summarize the conversation here to try to push the conversation forward.
For now if you are trying to figure out if your system will no longer receive policy updates you can run
sudo ostree admin config-diff | grep selinux/targeted/policy
to see if there are diffs between the local persistent policy and what's delivered by the OSTree. For example:In order to recover your system you can switch to dynamically modifying the policy (see the FCCT with the systemd unit above) and also run
sudo rsync -rclv /usr/etc/selinux/ /etc/selinux/
. You may want to run with--dry-run
first.After a reboot you should be back to normal. If you see any extra denials in logs you may want to run
restorecon -vr /etc/ /var/
to fix the contexts on any files.The text was updated successfully, but these errors were encountered: