Add support for otk-defined distributions (COMPOSER-2299)

[achilleas-k]

This PR adds the initial (but not complete) support for defining distributions using otk yaml files. I've been experimenting with multiple ideas for how to do this and I think this is the nicest to work with while keeping disruption of current internal distro definitions to a minimum.

There's still a lot more to do, but I wanted to get this in front of people to discuss design choices before moving on with the full solution. Meanwhile, I intend to start converting image types to otk files to evaluate the integration.

Notable design choices worth discussing

• Distributions are defined under an otk root directory (which will be configurable and we could support multiple paths). Distributions, architectures, and image types are discovered using the directory structure: <distro ID>/<architecture>/<image type name>.yaml.
• Each distribution must provide a properties.yaml in its root (<distro ID>/properties.yaml) which defines values required by osbuild-composer. Currently this includes name (which can differ from the distro ID), os_version, release_version, and runner.
  • Note that the current values might not all be necessary, but we will probably discover that we need some global distro properties like this to be defined somewhere. For example, module_platform_id is definitely not needed since otk is taking care of depsolving for us. But we will almost certainly need a way to define the canonical distro name in order to do host distro selection.
• Blueprint customizations are added via an entrypoint yaml file which includes the image type entrypoint (more on this below).

Keep in mind that while we transition to a fully otk-defined state, we will need to keep the instantiation and serialisation flow of the internal manifests and a lot of the interfaces associated with that. Once we finish transitioning, a lot of this should become simpler.

Blueprint customizations

When compiling an otk manifest, an otk-customizations.yaml is created in a temporary directory which includes all the Blueprint customizations and an otk.include to the target image type entrypoint. The customizations file is used as the entrypoint to the compile command.

For example, given the following Blueprint:

[customizations]
hostname = "hal-9000"

[customizations.kernel]
append = "debug"

[customizations.locale]
keyboard = "uk"

when compiling a Fedora 40 x86-64 qcow2, an otk-customizations.yaml is created with:

otk.define.customizations:
  hostname: "hal-9000"
  kernel_append: "debug"
  keyboard: "uk"

otk.include: `/otkroot/fedora-40/x86_64/qcow2.yaml`

(where /otkroot is the root where we define our otk distros)
and the manifest is compiled with:

otk compile --target osbuild /path/to/otk-customizations.yaml

The qcow2 yaml (or the files it includes) should then be properly handling these values. For example:

otk.target.osbuild:
  pipelines:
    - name: os
      stages:
        - otk.op.if-set:
          what: "${keyboard}"
          then:
            - type: org.osbuild.keymap
              options:
                keymap: "${keyboard}"

One issue with this approach however is that this puts customizations at the lowest priority since they're defined first, so any defines that follow will override the value. One way around this issue is prefixing these or namespacing them under an object that is treated with higher priority (e.g. modifications.keyboard), which is something we already discussed in the otk specifications.

[achilleas-k]

Oh nice. Tests are failing now because we depend on otk 😆

I'll probably mock the otk calls for testing the Serialize() code path.

[github-actions]

This PR is stale because it has been open 30 days with no activity. Remove "Stale" label or comment or this will be closed in 7 days.