OKD-223: Load custom SELinux rules in SCOS and workaround afterburn failures #1556

aleskandro · 2024-07-18T21:40:53Z

Recent changes in the SELinux policy to make more service confined has
broken a lot of our code. The SELinux team is working through relaxing
the policy, but in the meantime, let's revert back the affected types to
permissive mode:

afterburn fail when trying to write to /run, /run/metadata and
/home/$user/.ssh.
See: https://issues.redhat.com/browse/RHEL-49735
coreos-installer installation fails due to various denials.
See: https://issues.redhat.com/browse/RHEL-38614
network functionality that rely on systemd-network-generator is
broken due to the latter being unable to create temporary files.
See: https://issues.redhat.com/browse/RHEL-47033

aleskandro · 2024-07-18T21:43:43Z

/cc @Prashanth684 @travier

Prashanth684 · 2024-07-18T21:47:31Z

overlay.d/50scos/usr/lib/okd/selinux/afterburn.cil

@@ -0,0 +1,7 @@
+(typeattributeset cil_gen_require var_run_t)


will these rules also persist after the bootimage pivots to the scos image?

I'd also say that we could delete this file once a proper fix lands in CS9? I didn't reproduce in a CS9 fresh env with no other modifications as the ones here, but it seems a bug to file there for afterburn or the selinux-policy package you mentioned in the issue.

once we have scos publicly available, yes we would need this. But by that time I hope that we can get the actual issue with selinux-policy sorted out.

https://github.com/fedora-selinux/selinux-policy/pull/1362/files might be the fix to downstream?

And https://github.com/fedora-selinux/selinux-policy/pull/2000/files

@travier see the two PRs above for #1556 (comment)

openshift-ci-robot · 2024-07-18T21:48:18Z

@aleskandro: This pull request references OKD-223 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the bug to target the "4.17.0" version, but no target version was set.

In response to this:

The afterburn systemd units fail because the afterburn binary's SELinux domain is restricted from changing the content of files in /run, /run/metadata, and /home/$user/.ssh. See #1555.

This commit implements a systemd unit to apply custom SELinux modules in SCOS shipped as CILs in /usr/lib/okd/selinux and adds an afterburn-custom.cil SELinux module to allow the afterburn services to succeed and the nodes to properly join a cluster.

Refers #1555

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

travier · 2024-07-19T14:20:14Z

Hum, why is this not failing in FCOS?

aleskandro · 2024-07-19T15:13:51Z

Hum, why is this not failing in FCOS?

I think it is because of previous patches to SELinux-policy that are already available in FCOS.

aleskandro · 2024-07-19T15:20:45Z

Oh here it can be:

zpytela/selinux-policy@18e59aa removed the permissive domain, but non-rawhide fedora has it yet: https://github.com/zpytela/selinux-policy/blob/f40/policy/modules/contrib/afterburn.te#L15

travier · 2024-07-19T15:39:05Z

How will all of this work in the initrd? We really need to fix this in Rawhide first.

I'm not convinced we should carry SELinux policy modules here. Those domains were in permissive mode before, let's make them permissive again instead.

aleskandro · 2024-07-19T17:08:26Z

How will all of this work in the initrd? We really need to fix this in Rawhide first.

I'm not convinced we should carry SELinux policy modules here. Those domains were in permissive mode before, let's make them permissive again instead.

I've changed the module to set the afterburn_t domain in permissive mode. I could do the same for the coreos installer one so that we have a workaround for the other issue too, until they are are solved upstream and we can revert the two policies' commits. I would still continue to propose having that unit that loops through a folder in /usr for applying custom SELinux rules that can be needed from time to time.

We have something similar in okd-machine-os for the FCOS variant.

Prashanth684 · 2024-07-22T15:42:45Z

/lgtm

Prashanth684 · 2024-07-22T15:43:23Z

How will all of this work in the initrd? We really need to fix this in Rawhide first.
I'm not convinced we should carry SELinux policy modules here. Those domains were in permissive mode before, let's make them permissive again instead.

I've changed the module to set the afterburn_t domain in permissive mode. I could do the same for the coreos installer one so that we have a workaround for the other issue too, until they are are solved upstream and we can revert the two policies' commits. I would still continue to propose having that unit that loops through a folder in /usr for applying custom SELinux rules that can be needed from time to time.

We have something similar in okd-machine-os for the FCOS variant.

@travier could we get an approval for this?

jlebon · 2024-07-22T17:11:25Z

Note that in FCOS rawhide, we're currently pinning selinux-policy due to other issues: https://github.com/coreos/fedora-coreos-config/blob/441ccdd5cc3f2e3ac0cb5e8e0bcf71b301a51c7e/manifest-lock.overrides.yaml#L17-L26

But even then, the policy between c9s and rawhide are different and not mean to be kept in sync AIUI. See the conversation in https://github.com/openshift/os/issues/1514.\

I sympathize with this. I'm not opposed in theory, but it's also a much more invasive workaround that might have subtle repercussions. Note also that not many people on the team are familiar with maintaining SELinux policies.

What we really need of course is better CI integration with c9s. This should be a big part of the bootable containers effort.

Short-term, I guess we can just host the last working version RPM somewhere else and revert #1552 ? (Or, if you have the right channels, try to get a new tag created in the CentOS Stream koji instance with tag2distrepo set up so we can just tag older packages we need in there.)

Prashanth684 · 2024-07-22T19:32:16Z

I sympathize with this. I'm not opposed in theory, but it's also a much more invasive workaround that might have subtle repercussions. Note also that not many people on the team are familiar with maintaining SELinux policies.

What repercussions are we worried about?

Short-term, I guess we can just host the last working version RPM somewhere else and revert #1552 ? (Or, if you have the right channels, try to get a new tag created in the CentOS Stream koji instance with tag2distrepo set up so we can just tag older packages we need in there.)

Isn't this equivalent to having a workaround here?

travier · 2024-07-23T13:30:26Z

As I asked before:

How will all of this work in the initrd?

I don't think this will work as this unit runs after the initrd, only in the real root.

travier · 2024-07-23T13:30:51Z

/hold
As I don't think this is the right fix.

aleskandro · 2024-07-23T16:08:55Z

As I asked before:

How will all of this work in the initrd?

I don't think this will work as this unit runs after the initrd, only in the real root.

@travier this branch is already being used in the releases that turned green from Sunday thanks to this fix (for AWS).

Do we really need this after rebasing from the FCOS boot image (and initrd) to SCOS?
Which cases in particular?
How can installing the previous RPM in the base OS tree resolve this problem for initrd too?
Is there a temporary solution that we can achieve to ensure the releases get green again before the actual and permanent fix downstreams to CS9?

Note that in FCOS rawhide, we're currently pinning selinux-policy due to other issues: https://github.com/coreos/fedora-coreos-config/blob/441ccdd5cc3f2e3ac0cb5e8e0bcf71b301a51c7e/manifest-lock.overrides.yaml#L17-L26

But even then, the policy between c9s and rawhide are different and not mean to be kept in sync AIUI. See the conversation in #1514

I sympathize with this. I'm not opposed in theory, but it's also a much more invasive workaround that might have subtle repercussions. Note also that not many people on the team are familiar with maintaining SELinux policies.

What we really need of course is better CI integration with c9s. This should be a big part of the bootable containers effort.

Short-term, I guess we can just host the last working version RPM somewhere else and revert #1552 ? (Or, if you have the right channels, try to get a new tag created in the CentOS Stream koji instance with tag2distrepo set up so we can just tag older packages we need in there.)

@jlebon the current version of this PR 'reverts' the SELinux policies to the previous ones for the afterburn_t domain (permissive). If it's invasive because we allow a lot more than what's needed, I can go back to the fine-grained policies we proposed earlier.

I also understand that maintaining SELinux policies is not the most exciting thing here. Still, it allows the SCOS release to unblock and OKD/SCOS to have dedicated SELinux policies when needed, especially since, based on what you said, the SELinux policies in Fedora C9S might not always be in sync (and well-tested as they are for RHCOS).

To have working and quickly released OKD/SCOS, we will occasionally need more workarounds dedicated to C9S than RHCOS.

This PR adds that systemd unit only to C9S builds, with the hope that whoever maintains and reviews a change in that folder will always bind a Jira or GH issue to revert the changes eventually when they are fixed and downstreamed to be consumed as a permanent solution.

We had similar cases for OKD/FCOS: https://github.com/openshift/okd-machine-os/blob/master/overlay.d/99okd/usr/lib/okd/selinux-fixes.cil.

I would avoid further delays for the release of OKD/SCOS, 4.16 and beyond when similar cases arise.

Prashanth684 · 2024-07-25T18:53:46Z

@travier @jlebon any suggestions on how this can be unblocked in the short term? I think @aleskandro has given sufficient reasoning above for the approach.

ausil · 2024-07-26T04:33:33Z

From my point of view this approach is better than setting up a repo and locking to an old version, we will be missing all of the bug fixes in the newer selinux-policy builds. This is not an issue in the FCOS portion because the bug is fixed upstream. We are missing all the other fixes in selinux-policy if we lock to an old version. The selinux team needs to prioritise backporting the upstream fix. As I understand the issue, we only hit it on reboot when moving to CentOS Stream, which is where the fix is needed.

ausil · 2024-07-31T20:54:30Z

@dustymabe @cverna This is blocking OKD releases from going stable cc @preethit

jlebon · 2024-08-06T21:14:53Z

How about having just the overlay.d/50scos/usr/lib/okd/selinux/afterburn-permissive.cil file, and a postprocess step to semodule -i it? I think it'd be a lot cleaner to have this be part of the baked policy instead of forcing a local policy rebuild.

aleskandro · 2024-08-06T21:44:33Z

How about having just the overlay.d/50scos/usr/lib/okd/selinux/afterburn-permissive.cil file, and a postprocess step to semodule -i it? I think it'd be a lot cleaner to have this be part of the baked policy instead of forcing a local policy rebuild.

Hi @jlebon, that is better. Done

Thanks for following up.

aleskandro · 2024-08-07T02:44:34Z

/retest-required

jlebon · 2024-08-07T03:06:56Z

Thanks! I've reworked this as follow:

There's actually no need to ship this in /usr. Doing semodule -i will automatically compile and copy the results into /etc/selinux/targeted/active/ along with the other packages, so it'd just be duplicating it. Instead, we now just create a temporary file to be able to install it, and then delete the file.
I've added additional workarounds for other outstanding SELinux issues.
I've moved the postprocess script to manifest-c9s.yaml since this affects both the c9s and okd-c9s variants.

Recent changes in the SELinux policy to make more service confined has broken a lot of our code. The SELinux team is working through relaxing the policy, but in the meantime, let's revert back the affected types to permissive mode: 1. afterburn fail when trying to write to `/run`, `/run/metadata` and `/home/$user/.ssh`. See: https://issues.redhat.com/browse/RHEL-49735 2. coreos-installer installation fails due to various denials. See: https://issues.redhat.com/browse/RHEL-38614 3. network functionality that rely on systemd-network-generator is broken due to the latter being unable to create temporary files. See: https://issues.redhat.com/browse/RHEL-47033 Co-authored-by: Jonathan Lebon <jonathan@jlebon.com>

openshift-ci-robot · 2024-08-07T03:09:35Z

@aleskandro: This pull request references OKD-223 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the bug to target the "4.17.0" version, but no target version was set.

In response to this:

Recent changes in the SELinux policy to make more service confined has
broken a lot of our code. The SELinux team is working through relaxing
the policy, but in the meantime, let's revert back the affected types to
permissive mode:

afterburn fail when trying to write to /run, /run/metadata and
/home/$user/.ssh.
See: https://issues.redhat.com/browse/RHEL-49735

coreos-installer installation fails due to various denials.
See: https://issues.redhat.com/browse/RHEL-38614

network functionality that rely on systemd-network-generator is
broken due to the latter being unable to create temporary files.
See: https://issues.redhat.com/browse/RHEL-47033

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

jlebon · 2024-08-07T03:10:00Z

/hold cancel

We're currently hitting OpenSSL issues there: openshift#1540

jlebon · 2024-08-07T03:13:03Z

I've also inlined the first commit of #1544 here. This PR should now be all we need to unblock building c9s-based bootimages in the CoreOS pipeline.

jlebon · 2024-08-07T12:15:08Z

CI failing on #1551. I've verified this PR locally.

/override ci/prow/rhcos-9-build-test-metal
/override ci/prow/rhcos-9-build-test-qemu
/override ci/prow/scos-9-build-test-metal
/override ci/prow/scos-9-build-test-qemu

/approve
/lgtm

openshift-ci · 2024-08-07T12:15:36Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aleskandro, jlebon, Prashanth684

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [jlebon]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci · 2024-08-07T12:15:40Z

@jlebon: Overrode contexts on behalf of jlebon: ci/prow/rhcos-9-build-test-metal, ci/prow/rhcos-9-build-test-qemu, ci/prow/scos-9-build-test-metal, ci/prow/scos-9-build-test-qemu

In response to this:

CI failing on #1551. I've verified this PR locally.

/override ci/prow/rhcos-9-build-test-metal
/override ci/prow/rhcos-9-build-test-qemu
/override ci/prow/scos-9-build-test-metal
/override ci/prow/scos-9-build-test-qemu

/approve
/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-ci · 2024-08-07T12:15:44Z

@aleskandro: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

aleskandro · 2024-08-07T12:18:58Z

Looks good to me!

Thanks @jlebon

openshift-ci bot requested review from marmijo and prestist July 18, 2024 21:41

aleskandro force-pushed the afterburn-fix-scos branch 2 times, most recently from d18f4f0 to 11f7e8c Compare July 18, 2024 21:43

openshift-ci bot requested review from Prashanth684 and travier July 18, 2024 21:43

Prashanth684 reviewed Jul 18, 2024

View reviewed changes

aleskandro changed the title ~~Load custom SELinux rules in SCOS and workaround afterburn failures~~ OKD-223: Load custom SELinux rules in SCOS and workaround afterburn failures Jul 18, 2024

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jul 18, 2024

aleskandro force-pushed the afterburn-fix-scos branch from 11f7e8c to d82d6ec Compare July 18, 2024 22:14

aleskandro force-pushed the afterburn-fix-scos branch from d82d6ec to 7d26dfa Compare July 19, 2024 14:39

aleskandro force-pushed the afterburn-fix-scos branch from 7d26dfa to 173a514 Compare July 19, 2024 15:26

aleskandro force-pushed the afterburn-fix-scos branch from 173a514 to d3f25ca Compare July 19, 2024 15:50

aleskandro force-pushed the afterburn-fix-scos branch 2 times, most recently from fd52060 to eb2bd38 Compare July 21, 2024 00:34

openshift-ci bot assigned Prashanth684 Jul 22, 2024

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jul 22, 2024

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 23, 2024

aleskandro force-pushed the afterburn-fix-scos branch from eb2bd38 to b19d7f6 Compare August 6, 2024 21:42

openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label Aug 6, 2024

jlebon force-pushed the afterburn-fix-scos branch from b19d7f6 to 733c199 Compare August 7, 2024 03:06

jlebon force-pushed the afterburn-fix-scos branch from 733c199 to 9ba8b71 Compare August 7, 2024 03:09

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 7, 2024

kola-denylist: add fips.* on c9s

2f5ea5f

We're currently hitting OpenSSL issues there: openshift#1540

jlebon mentioned this pull request Aug 7, 2024

NO-JIRA: kola-denylist: add fips.* and ext.config.shared.networking.* on c9s #1544

Closed

openshift-ci bot assigned jlebon Aug 7, 2024

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Aug 7, 2024

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 7, 2024

openshift-merge-bot bot merged commit 1eaeec6 into openshift:master Aug 7, 2024
7 checks passed

travier mentioned this pull request Aug 22, 2024

NO-JIRA: Revert "c9s: work around various SELinux issues" #1568

Open

OKD-223: Load custom SELinux rules in SCOS and workaround afterburn failures #1556

OKD-223: Load custom SELinux rules in SCOS and workaround afterburn failures #1556

Conversation

aleskandro commented Jul 18, 2024 • edited by jlebon Loading

aleskandro commented Jul 18, 2024

Prashanth684 Jul 18, 2024

Choose a reason for hiding this comment

aleskandro Jul 18, 2024

Choose a reason for hiding this comment

aleskandro Jul 18, 2024

Choose a reason for hiding this comment

Prashanth684 Jul 18, 2024

Choose a reason for hiding this comment

aleskandro Jul 18, 2024

Choose a reason for hiding this comment

aleskandro Jul 18, 2024

Choose a reason for hiding this comment

aleskandro Jul 19, 2024

Choose a reason for hiding this comment

openshift-ci-robot commented Jul 18, 2024 • edited by openshift-ci bot Loading

travier commented Jul 19, 2024

aleskandro commented Jul 19, 2024

aleskandro commented Jul 19, 2024 • edited Loading

travier commented Jul 19, 2024 • edited Loading

aleskandro commented Jul 19, 2024 • edited Loading

Prashanth684 commented Jul 22, 2024

Prashanth684 commented Jul 22, 2024

jlebon commented Jul 22, 2024

Prashanth684 commented Jul 22, 2024

travier commented Jul 23, 2024

travier commented Jul 23, 2024

aleskandro commented Jul 23, 2024

Prashanth684 commented Jul 25, 2024

ausil commented Jul 26, 2024

ausil commented Jul 31, 2024

jlebon commented Aug 6, 2024

aleskandro commented Aug 6, 2024

aleskandro commented Aug 7, 2024

jlebon commented Aug 7, 2024

openshift-ci-robot commented Aug 7, 2024 • edited by openshift-ci bot Loading

jlebon commented Aug 7, 2024

jlebon commented Aug 7, 2024

jlebon commented Aug 7, 2024

openshift-ci bot commented Aug 7, 2024

openshift-ci bot commented Aug 7, 2024

openshift-ci bot commented Aug 7, 2024

aleskandro commented Aug 7, 2024

aleskandro commented Jul 18, 2024 •

edited by jlebon

Loading

openshift-ci-robot commented Jul 18, 2024 •

edited by openshift-ci bot

Loading

aleskandro commented Jul 19, 2024 •

edited

Loading

travier commented Jul 19, 2024 •

edited

Loading

aleskandro commented Jul 19, 2024 •

edited

Loading

openshift-ci-robot commented Aug 7, 2024 •

edited by openshift-ci bot

Loading