Integration of ue-rs with update-engine post-install script

[wrl]

On updates, Flatcar should download the wanted extension images when the Nebraska response returns them as additional packages. There are two types of extension images, the OEM extensions and the optional Flatcar extensions (e.g., K8s). Only one OEM is downloaded, as specified through the ID= field from /oem/oem-release. The optional extensions are specified in /etc/flatcar/enabled-sysext.conf (contains the names per line and supports # comments).
The ticket for this is #768
As subtask, we want to use a special ue-rs binary in the post-install action to download the extension images. This means less changes to update-engine but still some are needed.

From update-engine we should pass the XML response to the special binary when invoking it from the start of the post-install action. Note that this binary is to be loaded from the new partition and its temporary mount point. When no XML response is available in the post install action because the update-engine calling it is too old, we have to fall back to a hardcoded URL scheme to download the extension update payloads. What parts are done in bash vs what parts are done in the special ue-rs binary is a tradeoff. During development it will help if the ue-rs binary can be started manually with arguments that overwrite the defaults so that it's suitable for local development.
#768 describes two paths where the extracted OEM extension images will be stored after protobuf parsing and signature verification; the OEM path is preferred, the rootfs path is a fallback when the OEM partition is too small. The post-install action will also check if both the old and new OS versions have a sysext available and then create the active-oem-OEMID flag file for the sysext to be used on next boot.

[wrl]

@pothos could you please fill in specifics?

[pothos]

It's hard to come up with the exact integration because it's a tradeoff of what goes where - some parts are easy in bash, some are better done in Rust. We have to develop both together and can't use a perfect waterfall model.
I've written the above issue description now but it's more a summary of #768 that focuses on the post-inst script and the role of the binary.
One more thing we need is signed extension images to test with.

[pothos]

Prepared in flatcar/update_engine#24 - for now done with curl in bash

Added a comment there on where the integration would happen:

sysext_download() {
    local name="$1" # Payload name
    local target="$2" # Path to write the payload to, writing does not need to be atomic because the caller later does an atomic move
    local from="${3-}" # Either path to XML dump or the constant "release-server"/"bincache-server"
    … 
    # TODO: Replace the below with invoking an ue-rs helper binary for downloading the payload "name", either from the XML data or the release server ("from"), and write unpacked, verified file to "target"

The integration could look like this (assuming the subcommands payload-from-url and payload-from-xml):

    if [ "${from}" = "release-server" ]; then
      url="https://update.release.flatcar-linux.net/${FLATCAR_BOARD}/${NEXT_VERSION}/${name}"
    elif [ "${from}" = "bincache-server" ]; then
      url="https://bincache.flatcar-linux.net/images/${FLATCAR_BOARD/-usr}/${NEXT_VERSION}/${name}"
    fi
    if [ "${url}" = "" ]; then
        ue-rs-fetch payload-from-url --payload-url "${url}" --target-file "${target}" --pub-key /usr/share/update_engine/update-payload-key.pub.pem
    else
        ue-rs-fetch payload-from-xml --xml-dump "${from}" --payload-name "${name}" --target-file "${target}" --pub-key /usr/share/update_engine/update-payload-key.pub.pem
    fi

[pothos]

I haven't checked what the ue-rs download code looks like but in the above linked postinst action we do retry for downloads (curl -fsSL --retry-delay 1 --retry 60 --retry-connrefused --retry-max-time 60 --connect-timeout 20) but not for verification. If ue-rs has no download retry we can loop around the whole thing but for that it would be nice to know based on the exit code whether an error is a download or a verification error. If you can add download retries without much effort, even better.

[pothos]

The other integration point is here in the initrd: https://github.com/flatcar/bootengine/blob/2c0968a0c26a3f39721e76fb99368ee0fabffc7c/dracut/99setup-root/initrd-setup-root-after-ignition#L24