core: wrap and intercept `groupadd` calls in scriptlets #3778

lucab · 2022-06-20T07:51:39Z

This re-routes groupadd calls in scriptlets, in order to inspect
group creation by RPMs and automatically generate corresponding
sysusers.d fragments.
It uses a new hidden scriptlet-intercept subcommand, which can
be re-used in future to intercept additional commands (e.g.
useradd).

lucab · 2022-06-20T15:25:45Z

Result from current F36 FCOS built with this PR:

$ grep -ni '' /usr/lib/sysusers.d/30-pkg-group-*.conf

/usr/lib/sysusers.d/30-pkg-group-clevis.conf:1:# Generated by rpm-ostree
/usr/lib/sysusers.d/30-pkg-group-clevis.conf:2:g clevis -

/usr/lib/sysusers.d/30-pkg-group-dbus.conf:1:# Generated by rpm-ostree
/usr/lib/sysusers.d/30-pkg-group-dbus.conf:2:g dbus 81

/usr/lib/sysusers.d/30-pkg-group-dnsmasq.conf:1:# Generated by rpm-ostree
/usr/lib/sysusers.d/30-pkg-group-dnsmasq.conf:2:g dnsmasq -

/usr/lib/sysusers.d/30-pkg-group-docker.conf:1:# Generated by rpm-ostree
/usr/lib/sysusers.d/30-pkg-group-docker.conf:2:g docker -

/usr/lib/sysusers.d/30-pkg-group-printadmin.conf:1:# Generated by rpm-ostree
/usr/lib/sysusers.d/30-pkg-group-printadmin.conf:2:g printadmin -

/usr/lib/sysusers.d/30-pkg-group-rpc.conf:1:# Generated by rpm-ostree
/usr/lib/sysusers.d/30-pkg-group-rpc.conf:2:g rpc 32

/usr/lib/sysusers.d/30-pkg-group-sshd.conf:1:# Generated by rpm-ostree
/usr/lib/sysusers.d/30-pkg-group-sshd.conf:2:g sshd 74

/usr/lib/sysusers.d/30-pkg-group-tss.conf:1:# Generated by rpm-ostree
/usr/lib/sysusers.d/30-pkg-group-tss.conf:2:g tss 59

/usr/lib/sysusers.d/30-pkg-group-utempter.conf:1:# Generated by rpm-ostree
/usr/lib/sysusers.d/30-pkg-group-utempter.conf:2:g utempter 35

/usr/lib/sysusers.d/30-pkg-group-utmp.conf:1:# Generated by rpm-ostree
/usr/lib/sysusers.d/30-pkg-group-utmp.conf:2:g utmp 22

lucab · 2022-07-15T15:02:07Z

This should be ready to go, the CI is currently failing on coreos/fedora-coreos-tracker#1258.

cgwalters

I think we need to handle the case where a package already contains a sysusers fragment and is running groupadd? Does that case exist? Seems like e.g. we should avoid writing duplicate sysusers fragments.

cgwalters · 2022-07-19T18:04:23Z

rust/src/builtins/scriptlet_intercept/groupadd.rs

+                .long("password")
+                .takes_value(true),
+        )
+        .arg(Arg::new("system").short('r').long("system"))


I think we should either:

Error if this flag isn't passed

No-op if this flag isn't passed

WDYT?

Good idea, I think longer term we'd want to error out.
I'm not sure if we currently have scriptlets not using -r, so in the short term I'd just log a warning and proceed with the logic, so that we can adjust affected packages (if any).

rust/src/builtins/scriptlet_intercept/groupadd.rs

cgwalters · 2022-07-19T18:09:10Z

rust/src/builtins/scriptlet_intercept/groupadd.rs

+    static SYSUSERS_DIR: &str = "usr/lib/sysusers.d";
+
+    // The filename of the configuration fragment is in fact a public
+    // API, because users may have masked it in /etc. Do not change this.


Hmm. Why would someone do that?

This could be one approach for handling users/groups mismatches like coreos/fedora-coreos-tracker#1201 (either as a transitory measure or a permanent override).

rust/src/builtins/scriptlet_intercept/groupadd.rs

cgwalters · 2022-07-19T18:11:11Z

rust/src/builtins/scriptlet_intercept/groupadd.rs

+
+    // The filename of the configuration fragment is in fact a public
+    // API, because users may have masked it in /etc. Do not change this.
+    let filename = format!("30-pkg-group-{groupname}.conf");


For tmpfiles snippets we derive the filename from the package name - I think we could do that here by passing it through the environment.

I don't have a strong opinion on that though.

I considered doing that, I explored the pkgname-through-env approach and it's indeed feasible.

I'm still oscillating on the file-naming decision though.
My current stance is that having {groupname} in the file helps with idempotence, as a single package may be doing multiple groupadd. Current naming convention avoid interferences and allows for atomic-replaces without parsing/merging.
One third option I considered is the {pkgname}-{groupname} format, but I didn't see many additional benefits in doing that.

I'm open to more brainstorming on this topic. /cc @jlebon.

My current stance is that having {groupname} in the file helps with idempotence, as a single package may be doing multiple groupadd.

Makes sense to me!

Hmm, looking at /usr/lib/sysusers.d in my local FSB and FCOS, I don't see any config using priority prefixes in their names (except Zincati). So even with 30-, we'd be at the top of the list.

One third option I considered is the {pkgname}-{groupname} format, but I didn't see many additional benefits in doing that.

That'd be my vote. I think the additional benefit of being able to tell which package we derived from is nice (given that we don't currently include the package name in the generated fragment either). It would also better conform to the guideline established in sysusers.d(5):

Each configuration file shall be named in the style of package.conf or package-part.conf. The second variant should be used when it is desirable to make it easy to override just this part of configuration.

(Though it's a bit fuzzy here because we're doing it for them, hehe, but it's in the same spirit I think).

Ok, it looks like there is consensus about adding the package name in the filename. Sorry I didn't see the last comment here before rebasing, I'll wire the pkgname in a followup.

Regarding the numerical prefix and suggested naming pattern, there has been some friction reported upstream already: systemd/systemd#21155

In our case, I already know we'd need to properly priority-sort several sources with (possibly conflicting) definitions:

packaged fragment (e.g. https://pagure.io/setup/pull-request/39)

file overlays (e.g. overlay/sysusers: add fragments for all FCOS system groups fedora-coreos-config#1836)

groupadd translation (this PR)

While we aren't strictly following the suggested pattern from the docs, the numerical prefix at least allows predictable sorting/overriding which is helpful in the above case of multiple/conflicting sources of truth.

lucab · 2022-07-26T09:20:40Z

I think we need to handle the case where a package already contains a sysusers fragment and is running groupadd? Does that case exist? Seems like e.g. we should avoid writing duplicate sysusers fragments.

That case exists. But in general it's ok to have multiple group re-definitions as long as they are consistent (e.g. even setup and systemd do define overlapping groups).

We can add some heuristic for this, but I'm unsure how much deep you want to go. The general case would involve parsing all sysusers.d entries in /usr and /etc, then doing equivalence check for u and g entries, then eventually skipping, on each groupadd call.

Also keep in mind that we found several cases where the groupadd/useradd wouldn't match the sysusers.d content, and we are still actively working around those in FCOS until they are fixed in Fedora.

My hope is that eventually scriptlets would stop directly calling groupadd/useradd, and we'd just use the packaged sysusers.d fragment.

cgwalters

One final thought; should this be configurable? I hesitate to add a lot of options, but OTOH, in theory this could break someone, and it'd be handy to have an option to turn it off without downgrading.

Alternatively, it could be opt-in.

lucab · 2022-07-26T15:02:08Z

At some point we will definitely have a configuration knob to switch from nss-altfiles to plain systemd-sysusers.

The logic in this PR though is not interesting to configure in itself. It shouldn't cause regressions (as the fragment are not really used right now). But I do understand your concerns.
I also know this just a small step in the larger sysusers.d rework. At the same time I'd like to land this and keep iterating.
Thus for the moment my suggestion would be to put this logic (and the next steps in this journey) behind an environment feature flag, until I've reached a point where all pieces are in place and known working. At that point we can a top-level knob and perform the actual migration.

cgwalters · 2022-07-26T15:54:43Z

Thus for the moment my suggestion would be to put this logic (and the next steps in this journey) behind an environment feature flag,

I'm fine with that. Could also be a hidden option to compose tree. We also already have experimental, not explicitly stabilized treefile options.

jlebon

Cool!

Some comments. I see this was approved already, so I'm fine deferring them to a followup.

jlebon · 2022-07-19T16:52:13Z

src/app/libmain.cxx

@@ -84,6 +84,8 @@ static RpmOstreeCommand commands[] = {
  /* Rust-implemented commands; they're here so that they show up in `rpm-ostree
   * --help` alongside the other commands, but the command itself is fully
   *  handled Rust side. */
+  { "scriptlet-intercept", static_cast<RpmOstreeBuiltinFlags> (RPM_OSTREE_BUILTIN_FLAG_HIDDEN),


Note if the command is hidden, there's no reason to have it here since it won't show up anyway in the --help output.

jlebon · 2022-07-26T21:37:47Z

rust/src/builtins/scriptlet_intercept/groupadd.rs

+
+    // The filename of the configuration fragment is in fact a public
+    // API, because users may have masked it in /etc. Do not change this.
+    let filename = format!("30-pkg-group-{groupname}.conf");


Hmm, looking at /usr/lib/sysusers.d in my local FSB and FCOS, I don't see any config using priority prefixes in their names (except Zincati). So even with 30-, we'd be at the top of the list.

One third option I considered is the {pkgname}-{groupname} format, but I didn't see many additional benefits in doing that.

That'd be my vote. I think the additional benefit of being able to tell which package we derived from is nice (given that we don't currently include the package name in the generated fragment either). It would also better conform to the guideline established in sysusers.d(5):

Each configuration file shall be named in the style of package.conf or package-part.conf. The second variant should be used when it is desirable to make it easy to override just this part of configuration.

(Though it's a bit fuzzy here because we're doing it for them, hehe, but it's in the same spirit I think).

jlebon · 2022-07-26T21:39:23Z

rust/src/builtins/scriptlet_intercept/groupadd.rs

+}
+
+/// CLI parser, matches <https://linux.die.net/man/8/groupadd>.
+fn cli_cmd() -> Command<'static> {


Hmm, shouldn't we error out if any of the switches we don't actually support are used? And I guess one way to do that is to just drop those switches from the clap definition.

Likely yes. I'll do that in a followup PR together with enforcing the --system flag, as soon as I've verified that all packages in the FCOS base image are fine.

This re-routes `groupadd` calls in scriptlets, in order to inspect group creation by RPMs and automatically generate corresponding sysusers.d fragments. It uses a new hidden `scriptlet-intercept` subcommand, which can be re-used in the future to intercept additional commands (e.g. `useradd`). For the moment, this logic is experimental and gated behind an environment feature flag.

openshift-ci bot added the do-not-merge/work-in-progress label Jun 20, 2022

lucab force-pushed the ups/intercept-groupadd branch 2 times, most recently from 53600a3 to 0e7b906 Compare June 22, 2022 08:51

lucab mentioned this pull request Jul 12, 2022

overlay/sysusers: add fragments for all FCOS system groups coreos/fedora-coreos-config#1836

Merged

lucab force-pushed the ups/intercept-groupadd branch 2 times, most recently from 5af74ef to fa2cdc6 Compare July 15, 2022 09:51

lucab changed the title ~~[WIP] core: wrap and intercept groupadd calls in scriptlets~~ core: wrap and intercept groupadd calls in scriptlets Jul 15, 2022

openshift-ci bot removed the do-not-merge/work-in-progress label Jul 15, 2022

cgwalters reviewed Jul 19, 2022

View reviewed changes

cgwalters previously approved these changes Jul 26, 2022

View reviewed changes

jlebon reviewed Jul 26, 2022

View reviewed changes

lucab dismissed cgwalters’s stale review via 770b7e4 July 27, 2022 08:52

lucab force-pushed the ups/intercept-groupadd branch from fa2cdc6 to 770b7e4 Compare July 27, 2022 08:52

lucab enabled auto-merge July 27, 2022 11:02

cgwalters approved these changes Jul 27, 2022

View reviewed changes

lucab merged commit 7a29de1 into coreos:main Jul 27, 2022

lucab deleted the ups/intercept-groupadd branch July 27, 2022 15:21

This was referenced Jul 29, 2022

libpriv/scripts: inject pkgname in scriptlets environment #3891

Merged

core: intercept groupadd/useradd/usermod calls in scriptlets #3762

Open

lucab mentioned this pull request Aug 11, 2022

tests/basic: useradd: lock /etc/group.lock already used by PID 4 coreos/fedora-coreos-tracker#1250

Open

lucab mentioned this pull request Oct 26, 2022

Rework useradd(etc) sysusers integration to work in container flow #4092

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

core: wrap and intercept `groupadd` calls in scriptlets #3778

core: wrap and intercept `groupadd` calls in scriptlets #3778

lucab commented Jun 20, 2022 •

edited

Loading

lucab commented Jun 20, 2022 •

edited

Loading

lucab commented Jul 15, 2022

cgwalters left a comment

cgwalters Jul 19, 2022

lucab Jul 26, 2022

cgwalters Jul 19, 2022

lucab Jul 26, 2022

cgwalters Jul 19, 2022

lucab Jul 26, 2022

cgwalters Jul 26, 2022

jlebon Jul 26, 2022

lucab Jul 27, 2022

lucab commented Jul 26, 2022 •

edited

Loading

cgwalters left a comment

lucab commented Jul 26, 2022

cgwalters commented Jul 26, 2022

jlebon left a comment

jlebon Jul 19, 2022

jlebon Jul 26, 2022

jlebon Jul 26, 2022

lucab Jul 27, 2022

core: wrap and intercept groupadd calls in scriptlets #3778

core: wrap and intercept groupadd calls in scriptlets #3778

Conversation

lucab commented Jun 20, 2022 • edited Loading

lucab commented Jun 20, 2022 • edited Loading

lucab commented Jul 15, 2022

cgwalters left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lucab commented Jul 26, 2022 • edited Loading

cgwalters left a comment

Choose a reason for hiding this comment

lucab commented Jul 26, 2022

cgwalters commented Jul 26, 2022

jlebon left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

core: wrap and intercept `groupadd` calls in scriptlets #3778

core: wrap and intercept `groupadd` calls in scriptlets #3778

lucab commented Jun 20, 2022 •

edited

Loading

lucab commented Jun 20, 2022 •

edited

Loading

lucab commented Jul 26, 2022 •

edited

Loading