Provide a standard way to change kernel command line options via Ignition

[martinpitt]

EDIT: Current solution: https://docs.fedoraproject.org/en-US/fedora-coreos/kernel-args/

---

On FCOS there is no official way how to change the kernel command line, e. g. the mitigations=auto,nosmt option. The only place where they occur is in /boot/loader.0/entries/ostree-{1,2}-fedora-coreos.conf, but that's not very predictable, nor is it clear whether modifying these will also apply to installing kernel updates. Right now, none of the standard mechanisms (grubby, /etc/default/grub, /boot/grub/grubenv) are available.

Background: In Cockpit we have a UI for detecting/enabling/disabling the nosmt option. Changing a kernel command line option is annoyingly hard, we have quite an elaborate script for that.

Could grubby be added to FCOS images, or if not, what's the recommended way to persist kernel options? Thanks!

[lucab]

Thanks for the report.
On FCOS, rpm-ostree owns the kernel command-line arguments. This is exposed to consumers via rpm-ostree kargs (or DBus).

That said, I agree the UX here could be enhanced by:

• supporting default dropins, see Support kargs.d directories for default kargs ostreedev/ostree#1836
• adding a page to FCOS docs describing how kargs are handled

[martinpitt]

rpm-ostree kargs sounds promising, I'll have a look at that. Thank you!

[cgwalters]

coreos/rpm-ostree#1789 also redirects grubby.

[martinpitt]

I added rpm-ostree kargs to cockpit's kernelopt.sh script, now this works great on FCOS. Thanks for your help!

[cgwalters]

Just to xref this, for RHCOS (OpenShift) we support kernelArguments in MachineConfig.

[cgwalters]

Going to reopen this since I think we do want to standardize an Ignition method for this for FCOS and ideally have that same method apply to RHCOS.

[cgwalters]

For anyone who wants to change kargs today with FCOS, you'll basically need to write a systemd unit that runs rpm-ostree kargs --append ... --reboot, and you'll also need to use e.g. ConditionFirstBoot=yes.

[kelvinfan001]

I think we do want to standardize an Ignition method for this for FCOS and ideally have that same method apply to RHCOS.

Would this "standard way" be support for kargs.d as implemented in ostreedev/ostree#2217? This way, once Ignition supports rebooting in the initramfs, we can use Ignition configs to specify our desired default kargs, write a file into kargs.d, create a new deployment, and reboot.

[jlebon]

Would this "standard way" be support for kargs.d as implemented in ostreedev/ostree#2217? This way, once Ignition supports rebooting in the initramfs, we can use Ignition configs to specify our desired default kargs, write a file into kargs.d, create a new deployment, and reboot.

I think we'd probably want to optimize it so that we don't create a new deployment at all. So e.g. the user's Ignition config drops files in kargs.d, we notice it, add the kargs to the BLS, and reboot. All before ever switching root.

[kelvinfan001]

After some discussion today, we've decided that for CoreOS purposes, the BLS will still be the source of truth. In FCOS, there will be no new API or standard way for changing/setting kargs through OSTree. Setting/editing custom kernel arguments on first boot will be handled by Ignition.

[cgwalters]

To remove the default mitigations=auto,nosmt default, currently you can write a systemd unit in ignition that does e.g.

[Unit]
ConditionKernelCommandLine=mitigations

[Service]
ExecStart=/usr/bin/rpm-ostree kargs --remove=mitigations --reboot
RemainAfterExit=yes

If you're running other units that might depend on this, you should change them to order After= this unit.

[jlebon]

The rpm-ostree kargs approach is currently documented here: https://docs.fedoraproject.org/en-US/fedora-coreos/kernel-args/.

[bgilbert]

This will be addressed via Ignition's new kernel argument support (coreos/ignition#1168), which has been implemented in an experimental config spec but not yet stabilized. Leaving this open until the functionality is stabilized and documented.

[sohankunkerkar]

This will be addressed via Ignition's new kernel argument support (coreos/ignition#1168), which has been implemented in an experimental config spec but not yet stabilized. Leaving this open until the functionality is stabilized and documented.

@bgilbert This feature has already landed in the latest stable release: https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/34.20210529.3.0/x86_64/commitmeta.json

[bgilbert]

@sohankunkerkar Yep, and we've added documentation with a suitable warning. However, we can't unconditionally recommend the functionality until Ignition spec 3.3.0 is stabilized, since Ignition configs using the experimental spec will break at that point.

[dustymabe]

v3.3.0 stabilization happened in ignition 2.11.0 which is in testing. See the proposed docs change: coreos/fedora-coreos-docs#307

[dustymabe]

The fix for this went into testing stream release 34.20210626.2.0. Please try out the new release and report issues.

[dustymabe]

The fix for this went into stable stream release 34.20210626.3.1.

[dustymabe]

Docs will get updated soon when the new butane hits Fedora repos: coreos/fedora-coreos-docs#307 (comment)