Have Systemd Fail when Enabling a Non-existent Service via Presets

[Bubblemelon]

Feature Request

We would like a mode where it's possible via presets to require a unit to exist (site-specific config as opposed to generic OS vendor config)

Problem Overview:

Ignition is a provisioning tool in Container Linux - it runs in the initramfs, and supports generating systemd units, as well as enabling already extant ones.

Ignition only writes out to preset file, and does not directly enable extant units.

Ignition does not fail when a non-existent service is enabled via a ignition config, for the following reason:

Ignition is within the initramfs and has no way of knowing what will be generated by Systemd generators in the real root.

If an instance is not provisioned exactly as the ignition config specifies, then this should fail the provisioning.

Hence, in order to to fail the provisioning this needs to happen within the real root during the stage where systemd presets runs.

This is the issue is related to:
Ignition Bug #2243

Suggested Solution:

We suggest adding/implementing a new variant of the enable keyword, e.g. enable-required that has the same semantic as enabled, except it explicitly fails if the unit does not exist. We don't yet know what the presets stage failing should look like.

[poettering]

Hmpf, that would would be quite a change in semantics. Currently the idea is that admins can drop in their preset files long before installing any package, and then as packages are installed they are turned on or left off depending on what the preset files says. In fact if you look into the preset files various distros ship you will notice that the preset files in the majority list services most folks will probably not installed...

Adding a scheme to make "systemctl preset" fail if something listed doesn't exist would be a very different concept, because everything would be turned from its feet onto its head: now the preset files could be dropped in only after all packages are installed, because otherwise they might cause failure early on.

So, we can hardly ensure preset files are installed both only before and only after all packages are installed, because that's contradictory. Which makes it kinda clear to me that this is a very different concept from preset files...

Also, what's supposed to happen if these units don't exist? should the system refuse to boot? should it just be logged and the system should proceed?

It appears to me that what you are asking for is actually orthogonal to the preset logic and there's no reason to combine them in one. i.e. if all you want to check is whether a specific list of units exist at all, then this could be its own service indepdendent of presets...

Anyway, not grokking the usecase fully, and not what "failing" is supposed to entail?

[cgwalters]

This is for https://coreos.com/ignition/docs/latest/

Note the suggestion here is a new keyword enable-required - this isn't proposing any change to how "traditional" preset files work.

Also, what's supposed to happen if these units don't exist? should the system refuse to boot?

That's what Ignition does (intentionally) yes.

It appears to me that what you are asking for is actually orthogonal to the preset logic and there's no reason to combine them in one. i.e. if all you want to check is whether a specific list of units exist at all, then this could be its own service indepdendent of presets...

Probably yes, but in the end systemd's ConditionFirstBoot logic is going to be walking over all the units anyways, and Ignition is already writing out a preset file today.

[bgilbert]

So, we can hardly ensure preset files are installed both only before and only after all packages are installed, because that's contradictory.

Ignition is designed for use with a monolithic OS, where every package that will ever be installed is shipped in the OS image.

[Bubblemelon]

Sorry for the confusion, I've clarified the request.

In brief, we are suggesting to add a new mode such as enable-required which extends from the current existing mode enable but does not change anything to enable itself.

[Bubblemelon]

@ajeddeloh suggested adding a service that starts really early (the stage after the generators are done in the real root), where iterates through the preset file checking to see if the services on the presets file matches with any units. If it doesn't, then point to a .target and fail the boot.

What are the implications of this ? AFAIK, the units are iterated through and checked to see if it needs enabling or not based on the preset file. The service that Andrew is suggesting does the checking the other way round.

Also please fill in if I've missed any details. 😄

cc: @peterbaouoft

[ajeddeloh]

To elaborate on my idea:
Have a unit with ConditionFirstBoot=true and OnFailure=emergency.target. Order it very early in the bootup process, perhaps Before=sysinit.target and RequiredBy=sysinit.target. This unit fails if there are any units that don't match what they should be as defined by the presets file.

It's really not ideal since other units could mess with the boot process, try to disable it, etc. Ideally it'd be something that runs before systemd begins starting units.

[keszybz]

I agree with @poettering here: this is fairly specific functionality that can be achieved using existing mechanisms. We should not add more logic to systemd unless it something generally useful.

What @ajeddeloh suggests is better. It's more versatile too, because the logic can check more than just if the units exist, but for example if they are not modified from their pristine vendor state, or anything else.

I'll suggest a slightly different implementation though: not with a unit, but with a generator.
My assumption is that there's a preset file written by ignition that lists all the units.

# /usr/lib/systemd/system-generators/check-if-everything-is-enabled
#!/usr/bin/python3
import subprocess
import sys
import os

preset = "/etc/systemd/system-preset/00-ignition.preset"
failure_target='emergency.target'

if len(sys.argv) > 2:
   # use "early-dir" which has higher priority
   outdir = sys.argv[2]
else:
   outdir = sys.argv[1]

for line in open(preset):
     if line.startswith('enable '):
          unit = line.split()[1]
          c = subprocess.run(['systemctl', 'is-enabled', '--root=/', unit], stdout=subprocess.DEVNULL)
          if c.returncode != 0:              
                # Houston, we have a problem
                print(f'{unit} is not enabled, redirecting boot to {failure_target}')
                os.symlink(f'/usr/lib/systemd/system/{failure_target}', f'{outdir}/default.target')
                break

[cgwalters]

I'll suggest a slightly different implementation though: not with a unit, but with a generator.
My assumption is that there's a preset file written by ignition that lists all the units.

Hmm. With this model we can't check units that might come from another generator (not sure if that's a problem), and it feels...weird to be basically just talking back to pid1 over DBus. But yeah, doing it via a generator or unit we know will work in general.

[keszybz]

it feels...weird to be basically just talking back to pid1 over DBus

That's forbidden (from a generator). This does not communicate with pid1, it only queries the filesystem (--root=/).

With this model we can't check units that might come from another generator

It is certainly possible to check this, e.g. even by looking for the enablement symlink manually, and checking if it in /run/systemd/generator.*. But this seems such a niche case that I wouldn't bother.

Note: my code above contains an error, or at least a shortcoming. Generators need to print to /dev/kmsg to be properly logged.