Ignition blueprint config support on rhel9

[7flying]

This allows a user to configure the system via edge-simplified-installer
using an ignition configuration specified in the blueprint.

This ignition config can be embedded in the ISO as a Base64
encoded file (ignition.embedded.data) or as a file
containing the URL where the ignition config file is served
(ignition.embedded.url).
This embedded config (either data or url) ends up in the root
of the ISO.

The user can also instead specify an URL serving an ignition
config file that will passed as a karg and be fetched at first
boot (ignition.firstboot.url).

---

depends on #3130 (that PR sets the things needed to run ignition, this PR allows the user to configure the things to pass to ignition, so this PR doesn't break anything)

---

This pull request includes:

• adequate testing for the new functionality or fixed issue
• adequate documentation informing people about the change such as
• submit a PR for the guides repository if this PR changed any behavior described there: https://www.osbuild.org/guides/ --> see blueprint-reference: add Ignition config section guides#96

[7flying]

@runcom , regarding the kernel arguments, I know that in #3130 via support kargs in blueprint commit we can already do this, but from the user point of view it might be more straightforward to add it via the specific blueprint option so that you have less room to mess things up. What do you think?

Also, the PRs are going to collide in the ignition_stage.go file, that is intentional to handle all the ignition stuff in one place.

[bcl]

Pleas also remember to update the documentation in the guide - https://github.com/osbuild/guides/blob/main/osbuild-composer/src/blueprint-reference/blueprint-reference.md

[teg]

regarding the kernel arguments, I know that in #3130 via support kargs in blueprint commit we can already do this, but from the user point of view it might be more straightforward to add it via the specific blueprint option so that you have less room to mess things up. What do you think?

If we can, we should try to give users high-level options that we translate to kargs (or other things), so totally agree with this principle :)

[teg]

Could ignition_data be called just data? As it is already in the ignition section. In general, how does this compare to how CoreOS handles ignition configuration? Do you have any thoughts on this @cgwalters ?

[7flying]

Could ignition_data be called just data? As it is already in the ignition section. In general, how does this compare to how CoreOS handles ignition configuration? Do you have any thoughts on this @cgwalters ?

Sure thing, will change it 👍
We are using some of the scripts that the CoreOS team uses to handle ignition (https://github.com/fedora-iot/ignition-edge), small changes here and there but it should be very similar.

[cgwalters]

This ignition config can be embedded in the ISO as a Base64
encoded file (ignition.embedded.ignition_data)

Where is this file physically present? It's not clear to me from the PR. Is that referencing a kernel argument?

You may already be aware but I think it's still worth cross-referencing here: For CoreOS we¹ developed an IMO neat system where an Ignition config can be dynamically embedded in an ISO image: https://coreos.github.io/coreos-installer/cmd/iso/#coreos-installer-iso-ignition-embed

The way this works is by sticking the config into a "reserved" area of the ISO. Given that Edge is already using coreos-installer, this could make sense to also support (would I think require a bit of work to replicate bits of cosa buildextend-live in the ISO).

One advantage of this is that it gives a clear model for building a "generic" ISO that can have e.g. machine-specific customization injected later.

But - that's all specific to the ISO.

To support the generic functionality of embedding a custom Ignition config "statically", it can be injected into /usr/lib/dracut/modules.d/40ignition-conf today. Note on CoreOS derivatives we have:

[root@cosa-devsh ~]# cat /usr/lib/dracut/modules.d/40ignition-conf/00-core.ign 
{
  "ignition": {
    "version": "3.0.0"
  },
  "passwd": {
    "users": [
      {
        "name": "core",
        "gecos": "CoreOS Admin",
        "groups": [
          "adm",
          "sudo",
          "systemd-journal",
          "wheel"
        ]
      }
    ]
  }
}

(This config should likely not be present on Edge derivatives as the "core" user is obviously a continuation of a CoreOS default and when moving to full custom images it's likely not desired)

But, other "static" configuration can be injected in the same way.

¹ Specifically IIRC @bgilbert and @jlebon, I was only tangentially involved

[7flying]

This ignition config can be embedded in the ISO as a Base64
encoded file (ignition.embedded.ignition_data)

Where is this file physically present? It's not clear to me from the PR. Is that referencing a kernel argument?

We put the file in the root of the ISO:

[root@citest-1 si]# ll /mnt/disk/
total 802405
drwxr-xr-x. 1 root root      2048 Nov 29 06:30 EFI
-r--r--r--. 1 root root      2048 Nov 29 06:34 boot.catalog
-rw-r--r--. 1 root root 821653588 Nov 29 06:33 disk.img.xz
-rw-r--r--. 1 root root      1629 Nov 29 06:30 ignition_config
drwxr-xr-x. 1 root root      2048 Nov 29 06:33 images

It does not reference a kernel argument.

You may already be aware but I think it's still worth cross-referencing here: For CoreOS we¹ developed an IMO neat system where an Ignition config can be dynamically embedded in an ISO image: https://coreos.github.io/coreos-installer/cmd/iso/#coreos-installer-iso-ignition-embed

I wasn't aware of that at all! CC'ing @runcom ! From a quick look at it, if I understood it correctly, it would require producing the ISO first and then the installer should be ran later on to embed the config file. We do not run the installer until we are installing the image, so maybe that "later" (regarding the customisation injection that is made later on) wouldn't work for us since we just want to have a .iso with everything in there that can be installed and that's it.
But, again, CC'ing @runcom since I do have less context.

The way this works is by sticking the config into a "reserved" area of the ISO. Given that Edge is already using coreos-installer, this could make sense to also support (would I think require a bit of work to replicate bits of cosa buildextend-live in the ISO).

One advantage of this is that it gives a clear model for building a "generic" ISO that can have e.g. machine-specific customization injected later.

But - that's all specific to the ISO.

To support the generic functionality of embedding a custom Ignition config "statically", it can be injected into /usr/lib/dracut/modules.d/40ignition-conf today. Note on CoreOS derivatives we have:

[root@cosa-devsh ~]# cat /usr/lib/dracut/modules.d/40ignition-conf/00-core.ign 
{
  "ignition": {
    "version": "3.0.0"
  },
  "passwd": {
    "users": [
      {
        "name": "core",
        "gecos": "CoreOS Admin",
        "groups": [
          "adm",
          "sudo",
          "systemd-journal",
          "wheel"
        ]
      }
    ]
  }
}

(This config should likely not be present on Edge derivatives as the "core" user is obviously a continuation of a CoreOS default and when moving to full custom images it's likely not desired)

But, other "static" configuration can be injected in the same way.

¹ Specifically IIRC @bgilbert and @jlebon, I was only tangentially involved

[jlebon]

I wasn't aware of that at all! CC'ing @runcom ! From a quick look at it, if I understood it correctly, it would require producing the ISO first and then the installer should be ran later on to embed the config file. We do not run the installer until we are installing the image, so maybe that "later" (regarding the customisation injection that is made later on) wouldn't work for us since we just want to have a .iso with everything in there that can be installed and that's it. But, again, CC'ing @runcom since I do have less context.

I'm not familiar with the Edge install flow, but I can explain the CoreOS install flow this fits in and you can see from there whether it makes sense to adapt to it as well.

Users download a generic live ISO from e.g. getfedora.org (or for RHCOS, e.g. the mirror). They can then customize that ISO using coreos-installer iso customize. Users can choose how much to customize it. All this is done "offline", without having to actually boot the ISO. Crucially, the --dest-device switch will make it so that simply booting the ISO will initiate an install and reboot. All this to say that the Ignition embedding feature that Colin mentioned above provides a lot of power (that's what backs most of the iso customize features).

Though it's not clear to me from this PR whether the Ignition config we're talking about is for the ISO or for the installed system. Obviously a lot of this flexibility comes from the fact that CoreOS ISOs themselves run Ignition (and then we talk about "Ignition configs within Ignition configs" when embedding the target system's config within the ISO's config, but that's what the iso customize work is intended to simplify).

[7flying]

I wasn't aware of that at all! CC'ing @runcom ! From a quick look at it, if I understood it correctly, it would require producing the ISO first and then the installer should be ran later on to embed the config file. We do not run the installer until we are installing the image, so maybe that "later" (regarding the customisation injection that is made later on) wouldn't work for us since we just want to have a .iso with everything in there that can be installed and that's it. But, again, CC'ing @runcom since I do have less context.

I'm not familiar with the Edge install flow, but I can explain the CoreOS install flow this fits in and you can see from there whether it makes sense to adapt to it as well.

Users download a generic live ISO from e.g. getfedora.org (or for RHCOS, e.g. the mirror). They can then customize that ISO using coreos-installer iso customize. Users can choose how much to customize it. All this is done "offline", without having to actually boot the ISO. Crucially, the --dest-device switch will make it so that simply booting the ISO will initiate an install and reboot. All this to say that the Ignition embedding feature that Colin mentioned above provides a lot of power (that's what backs most of the iso customize features).

Though it's not clear to me from this PR whether the Ignition config we're talking about is for the ISO or for the installed system. Obviously a lot of this flexibility comes from the fact that CoreOS ISOs themselves run Ignition (and then we talk about "Ignition configs within Ignition configs" when embedding the target system's config within the ISO's config, but that's what the iso customize work is intended to simplify).

The ignition configuration file that we are embedding into the ISO is read by the coreos-installer and handled from there, so it is read when we are installing the system, so I guess that it is for the ISO, not for the installed system, since it is in the process of being installed.

What Colin proposes is another way of giving the users a way to customise their system, which is a separate method from our intended "installation flow". It would definitely work, but in our use cases it makes more sense to have as the last step in the process a "final ISO" that the users can grab and install.

In our build process we add specific RPMs needed for Edge via Image Builder, tweak the partition tables as we need them to be and so on, and users can start adding some customisations via blueprints there; now we are giving them the possibility of using Ignition too.

They only need to deal with Image Builder to produce images and use our r4e Butane specification to generate an Ignition file with the things that we currently support, they generate the ISO and that's it.

Otherwise at this point in time they would need to deal with IM + butane + coreos-installer iso customize.

[teg]

Thanks for our input on this @cgwalters and @jlebon!

I think you are right @7flying , that we want to make any customisations (and hence ignition insertion) at image build time, rather than after the fact, so static ISOs + dynamic customisations doesn't really make sense.

That said, it might be worth considering putting our ignition file in the same "reserved" location that the coreos tool does? I think this is sort of an internal detail, so could be done in a follow-up, but if it makes sense to be consistent we should be :)

I think a good mental model would be that the R4E image that comes out of image builder is structured the same way as a coreos installer ISO after the dynamic configuration has been applied. Unless there are technical quirks I'm not aware of that would make this not worth it, of course.

[7flying]

Rebased due to #3166

[7flying]

Thanks for our input on this @cgwalters and @jlebon!

I think you are right @7flying , that we want to make any customisations (and hence ignition insertion) at image build time, rather than after the fact, so static ISOs + dynamic customisations doesn't really make sense.

That said, it might be worth considering putting our ignition file in the same "reserved" location that the coreos tool does? I think this is sort of an internal detail, so could be done in a follow-up, but if it makes sense to be consistent we should be :)

Having an standardised way of doing things sounds always good to me :), but I have to check with others on the team.

I think a good mental model would be that the R4E image that comes out of image builder is structured the same way as a coreos installer ISO after the dynamic configuration has been applied. Unless there are technical quirks I'm not aware of that would make this not worth it, of course.

Right now I have no clue about how to do this in osbuild-composer, we might need to modify/extend one of osbuild's copy stages to handle that placement (size/alignment stuff I guess), but that would be it.

[cgwalters]

I think it's not really important to match the same mechanism right now; i.e. I would definitely not block on it. It's something that can be done later.

[teg]

I think it's not really important to match the same mechanism right now; i.e. I would definitely not block on it. It's something that can be done later.

Yup, I agree with this. Just something to keep in mind as an aspirational goal ;)

[achilleas-k]

Looks good but the test script needs to be conditioned to only run on RHEL 9 since the new features weren't added to RHEL 8.
Also, can we drop the second-to-last commit with the manifest generation? The manifests haven't changed, it's just some metadata key reordering and it doesn't affect any tests, so no need to add extra noise.

[achilleas-k]

Looks good. Thanks.

[7flying]

thanks for the review @achilleas-k !