Can not install os-extensions on first boot with rpm-ostree install --apply-live --cache-only --allow-inactive ...

[dwarf-king-hreidmar]

BLUF: I'm trying to install some local RPMs on my Raspberry Pi at fist boot and I was directed at the os-extentions / layering docs here. I'm experiencing issues running rpm-ostree install from systemd while the same command works on the cli after boot. Sorry if this is the wrong place but I figure others might try the docs and have an issue too. I also think there is a bigger pattern here around how to customize the install. Most solutions focus on connected workflows. I know there is work to make creating a custom coreos image easier so to be fair this is more of a stop gap I'm hoping. Using a remote zezere server is out for me since I can't use it for unattended installs. Thanks for your patience.

Some history:
Initial thread on fedora discussion: https://discussion.fedoraproject.org/t/fedora-coreos-and-wifi/38550
Similar request to add wifi that I commented on (was rejected and thats fine); #862
Suggested solution: #862 (comment)
Another suggested solution that isn't quite fully realized yet so you'll hit a dead end when trying to use a custom made podman image with coreos-installer: coreos/layering-examples#14 (comment)
Latest Ask Fedora Post specifically about this issue: https://ask.fedoraproject.org/t/rpm-ostree-layering-not-working-when-running-from-a-systemd-unit-file/25161

System:
Fedora CoreOS 36
RPi4
Installing the follwing packages
NetworkManager-wifi-1.36.4-1.fc36.aarch64.rpm
iw-59-4.fc36.aarch64.rpm
wireless-regdb-2021.08.28-2.fc36.noarch.rpm
wpa_supplicant-2.10-4.fc36.aarch64.rpm

What I tried:
My latest attempt is focused on using a systemd unit file to run rpm-ostree to install some local rpms similar to this: Adding OS extensions to the host system :: Fedora Docs. The only difference is for my exec I’m using rpm-ostree install --apply-live --cache-only --allow-inactive --idempotent /var/tmp/$localfile.rpm.

I’ve tried chaing exec start to hard code the timestamp name.
I’ve tried editing the rpm-ostree install command to contain a specific file in /var/tmp (which does exist).
I’ve tried with and without the “before zincatai” line
I’ve tried running by rebooting
I’ve tried running using systemctl start $service and the errors are the same when i do that

My systemd unit file:

        [Unit]
        Description=Layer wifi rpm-ostree
        # We run before `zincati.service` to avoid conflicting rpm-ostree
        # transactions.
        Wants=network-online.target
        After=network-online.target
        Before=zincati.service
        ConditionPathExists=!/var/lib/%N.stamp

        [Service]
        Type=oneshot
        RemainAfterExit=yes
        # `--allow-inactive` ensures that rpm-ostree does not return an error
        # if the package is already installed. This is useful if the package is
        # added to the root image in a future Fedora CoreOS release as it will
        # prevent the service from failing.
        ExecStart=/usr/bin/rpm-ostree install --apply-live --cache-only --allow-inactive /var/tmp/*.rpm
        ExecStart=/bin/touch /var/lib/%N.stamp

What happens:

1. When I run this I get the following error that I haven’t been able to find anywhere online:
2. rpm-ostree[2166]: error: Preparing D-Bus arguments: No such file or directory (os error 2).
3. I can run the command on the command line no problem and my wifi comes up right away.

What I want to happen:
I need this to run as a one shot on first boot though to get my wifi tools installed to enable unattended setup.

[dustymabe]

Any chance you have a copy of your Butane/Ignition config you're willing to share?

[dwarf-king-hreidmar]

Sure thing though I should note that I hopped on the server and tried a few variations on the systemd unit file in case it was starting too soon. To be clear this unit file fails when I run it manually after boot but the command runs fine on the cli. I've also tried with hard coded stamp files and without the --apply-live flag

variant: fcos
version: 1.3.0
systemd:
  units:
    # Installing vim as a layered package with rpm-ostree
    - name: rpm-ostree-install-wifi.service
      enabled: true
      contents: |
        [Unit]
        Description=Layer wifi rpm-ostree
        # We run before `zincati.service` to avoid conflicting rpm-ostree
        # transactions.
        Wants=network-online.target
        After=network-online.target
        Before=zincati.service
        After=dbus-broker.service
        ConditionPathExists=!/var/lib/%N.stamp

        [Service]
        Type=oneshot
        RemainAfterExit=yes
        # `--allow-inactive` ensures that rpm-ostree does not return an error
        # if the package is already installed. This is useful if the package is
        # added to the root image in a future Fedora CoreOS release as it will
        # prevent the service from failing.
        ExecStart=/usr/bin/rpm-ostree install --apply-live --cache-only --allow-inactive --idempotent /var/tmp/*.rpm
        ExecStart=/bin/touch /var/lib/%N.stamp

        [Install]
        WantedBy=multi-user.target```

[dustymabe]

Where do the files in /var/tmp/*.rpm come from? Are they written by Ignition (and not included in your above snippet)?

[dustymabe]

I think

ExecStart=/usr/bin/rpm-ostree install --apply-live --cache-only --allow-inactive --idempotent /var/tmp/*.rpm

Should probably be

ExecStart=/usr/bin/bash -c "rpm-ostree install --apply-live --cache-only --allow-inactive --idempotent /var/tmp/*.rpm"

IOW the *.rpm won't expand unless it's run in a shell I don't think.

[dwarf-king-hreidmar]

To get the rpms in the correct place I :

1. Download all the rpms and dependencies
2. use the coreos installer to write to my sd card
3. mount the proper partition written by the coreos installer
4. drop my rpms so that they will show up on /var/tmp when the system boots for the first time.

[dwarf-king-hreidmar]

IOW the *.rpm won't expand unless it's run in a shell I don't think.

This is an amazing catch. I could have sworn I tried this with a specific RPM in the command once and still go the same error. I must have had another typo when I tested it. Thanks so much.

Do you think its worth adding to the docs to help folks manage adding custom rpms in the same way ?

[dustymabe]

IOW the *.rpm won't expand unless it's run in a shell I don't think.

This is an amazing catch. I could have sworn I tried this with a specific RPM in the command once and still go the same error. I must have had another typo when I tested it. Thanks so much.

No problem. Glad we figured it out.

Do you think its worth adding to the docs to help folks manage adding custom rpms in the same way ?

The reason the existing docs don't use bash -c is because they don't require any shell expansion (i.e. the users just put what RPMs they want to install on the command line there).

As far as enhancing the docs for this specific use case, I'm not really sure. We do want to make the UX better for layering packages and I imagine that could also include making it easier to "pre-load" content at the coreos-installer phase and have it applied on first boot.

Not sure if we want to take this approach specifically and add it to our docs. While it does work, it's a bit cumbersome and we should probably try to serve this use case in a less painful way.

[dwarf-king-hreidmar]

I agree. This is cumbersome. I'd love for there to be a better way. I probably don't have the skill to make that better way a reality but if folks change there mind I'm happy to write the docs. Its particularly cumbersome when you need the layering to get networking working