add /usr/lib/bootc/kargs.d support

[lukewarmtemp]

Fixes #255. Allows users to create files within /usr/lib/bootc/kargs.d with kernel arguments. These arguments can now be applied on a switch, upgrade, or edit.

General process:

• use ostree commit of fetched container image to return the file tree
• navigate to /usr/lib/bootc/kargs.d
• read each file within the directory
• push the contents of each file (kargs) into a vector containing all the desired kargs
• pass the kargs to the stage() function

Example Containerfile:

FROM quay.io/fedora/fedora-coreos:stable

RUN mkdir -p /usr/lib/bootc/kargs.d
RUN cat <<EOF >> /usr/lib/bootc/kargs.d/00-console.toml
kargs = ["console=ttyS0,114800n8"]
supported-arch = ["x86_64"]
EOF

RUN cat <<EOF >> /usr/lib/bootc/kargs.d/01-mitigations.toml
kargs = ["mitigations=on", "systemd.unified_cgroup_hierarchy=0"]
supported-arch = ["x86_64", "aarch64"]
EOF

Looking for some input as to how files within /usr/lib/bootc/kargs.d should be formatted (which will also affect how the contents of the file are parsed)

Signed-off-by: Luke Yang luyang@redhat.com

[cgwalters]

Thanks for looking at this!

[cgwalters]

Marking as needs-work based on https://github.com/containers/bootc/pull/401/files#r1529248230

[cgwalters]

Let's bikeshed some of the file format/semantics.

One angle: I think there are use cases for conditionally-applied kernel arguments. One that very much comes up is the console= kernel argument being different on architectures and platforms - and dealing with that is painful for people deploying across platforms. We could just say "You define your own build process which writes to /usr/lib/bootc/kargs.d" which would be fine...but, something like what systemd does with .unit files or perhaps with .link files could make sense.

IOW

[match]
architecture = x86_64

kargs = ["console=ttyS0,114800n8"]

maybe?

BTW, a whole thing running through all of this is that I think a number of bootc use cases are better off actually generating custom kernel binaries, where you don't need or want this at all; that's something to keep in mind.

[cgwalters]

related to console https://github.com/coreos/fedora-coreos-config/blob/testing-devel/platforms.yaml

[lukewarmtemp]

@cgwalters Do you know if there's there a way for Rust to detect what platform your system is running? I'm able to find and match against the arch of the system using std::env::consts::ARCH, but not sure if there's something similar for platform.

[jeckersb]

@cgwalters Do you know if there's there a way for Rust to detect what platform your system is running? I'm able to find and match against the arch of the system using std::env::consts::ARCH, but not sure if there's something similar for platform.

There's a bunch of things you can trigger conditional compilation on, such as target_os, maybe that can cover what you need?

[cgwalters]

By "platform" I meant e.g. AWS vs GCP vs OpenStack etc. We're not going to build distinct bootc binaries for each of those, and runtime detection is "interesting". What Fedora CoreOS and derivatives do is https://github.com/coreos/fedora-coreos-tracker/blob/main/internals/README-internals.md#ignitionplatformid

At a practical level though I think we will likely end up with distinct container builds per "platform" in this sense, so we probably don't need to do dynamic dispatch here.

That said, for base images that do include ignition they'll need to have ignition.platform.id anyways...so in theory what we could support is a generic matching against an existing kernel argument (which would make this whole thing slightly recursive of course).

[jeckersb]

By "platform" I meant e.g. AWS vs GCP vs OpenStack etc.

Ah gotcha. That's what I get for not reading back far enough for context. Carry on!

[cgwalters]

It's a bit ugly but we could also consider matching what Cargo does for architecture conditionals... https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html#platform-specific-dependencies

Maybe even lift the code. Dunno.

I think my instinct here is that anything we do with kargs we should do with install configs too, as that's also a made up TOML thing but we may as well try to be consistent.

Specifically it makes sense to me to support "install only" kargs having architecture conditionals too.

[cgwalters]

Any new thoughts relating to this?

I guess a downside of using the modules directory (aka /usr/lib/modules/$kver) is that anyone doing a derived build would need to be more careful to detect the container kernel version as is done for e.g. https://docs.fedoraproject.org/en-US/bootc/initramfs/#_regenerating_the_initrd and drop into that directory vs just a straight COPY 20-mykargs.toml /usr/lib/bootc/kargs.d.

So...I don't have anything better than /usr/lib/bootc/kargs.d. I feel somewhat uncomfortable in inventing a new thing though. One thing I'd like to do is try to standardize some of the bootc stuff in OCI upstream, and it'd be around exactly things like this.

[cgwalters]

One thing I'd like to do is try to standardize some of the bootc stuff in OCI upstream, and it'd be around exactly things like this.

➡️ https://groups.google.com/a/opencontainers.org/g/dev/c/OtLlhn4jxBg

[cgwalters]

Thanks, I think this is generally looking good; the big gap I see is around integration testing. I'd hoped to be able to give you better guidance and tooling here by now. We'll continue to track that in #543

Short term, I think we can probably merge, and e.g. @henrywang can look at updating the integration tests there.

We're also going to need changes to the docs.

Can you also take an action item to do a merge request to https://gitlab.com/fedora/bootc/examples say?

[cgwalters]

This looks correct, though I would say we should probably add a convenience helper to this in ostree.

[lukewarmtemp]

the big gap I see is around integration testing

Oh sorry, I was going to look at this but it slipped my mind. I can probably have a crack at it to see if I can get it working. If I can't see to get it, I'll probably reach out to @henrywang.

Can you also take an action item to do a merge request to https://gitlab.com/fedora/bootc/examples say?

Will do.

[lukewarmtemp]

MR updating bootc examples: https://gitlab.com/fedora/bootc/examples/-/merge_requests/48

[cgwalters]

I tested this out and hit:

OSTree:ERROR:src/libostree/ostree-repo-file.c:828:ostree_repo_file_tree_query_child: assertion failed: (self->tree_contents)
Bail out! OSTree:ERROR:src/libostree/ostree-repo-file.c:828:ostree_repo_file_tree_query_child: assertion failed: (self->tree_contents)

[cgwalters]

I also hit on the fact that my test machine didn't come up across an update, and I think this is the cause: this must be return Ok(existing_kargs); right?

[lukewarmtemp]

I think it might be

kargs.append(&mut existing_kargs);
return Ok(kargs);

Since the kargs vec contains the existing kargs on the running system.