No Updated Grub Menu Entry After 'rpm-ostree upgrade' Finishes With No Apparent Errors

[dsreyes1014]

Host system details

OS Name - Fedora Linux 38.20231012.0 (Silverblue)
OS Type - 64-bit
GNOME Version - 44.5
Windowing System - Wayland
Kernel Version - Linux 6.5.6-200.fc38.x86_64

Provide the output of rpm-ostree status.

$ rpm-ostree status
State: idle
Deployments:
● fedora:fedora/38/x86_64/silverblue
                  Version: 38.20231012.0 (2023-10-12T00:52:57Z)
               BaseCommit: 5abd4778d26b8dae81b4b08ead17e1e1e7a3d4507811a13928d9fb3b3b1b5f2e
             GPGSignature: Valid signature by 6A51BBABBA3D5467B6171221809A8D7CEB10B464
          LayeredPackages: langpacks-en
                   Pinned: yes

  fedora:fedora/38/x86_64/silverblue
                  Version: 38.20231011.0 (2023-10-11T00:51:10Z)
               BaseCommit: 63e3ef2664856a1cf18e2ba8c4332855ecea82a3c0fd958c786f3768db7cb35e
             GPGSignature: Valid signature by 6A51BBABBA3D5467B6171221809A8D7CEB10B464
          LayeredPackages: langpacks-en

Expected

Expecting an updated menu entry to boot into after an update.

Actual Behavior

Update doesn't get added to grub menu and stays stuck on 38.20231012.0

Steps to reproduce it

1. rpm-ostree upgrade
2. systemctl reboot

Provide any additional data that may help debug this - which specific version of
an RPM is in the repo, or any host system configuration.

Please let me know what else is needed to add to help debug this. I have no layered packages installed. All the apps I have installed are flatpaks.

Would you like to work on the issue?

I won't be able to fix this myself as I don't have the skills. Would need help with this bug on my particular system.

[cgwalters]

Can you check the output of journalctl -u ostree-finalize-staged?

[jmarrero]

Looks like I am hitting this too on 2023.11

jmarrero@fedora ~> journalctl -u ostree-finalize-staged
Nov 03 20:26:03 fedora systemd[1]: Finished ostree-finalize-staged.service - OSTree Finalize Staged Deployment.
Nov 03 20:26:30 fedora systemd[1]: Stopping ostree-finalize-staged.service - OSTree Finalize Staged Deployment...
Nov 03 20:26:30 fedora ostree[4704]: Finalizing staged deployment
Nov 03 20:26:30 fedora ostree[4704]: Copying /etc changes: 36 modified, 2 removed, 56 added
Nov 03 20:26:30 fedora ostree[4704]: Copying /etc changes: 36 modified, 2 removed, 56 added
Nov 03 20:26:31 fedora ostree[4704]: Finalized deployment
Nov 03 20:26:31 fedora ostree[4704]: error: Bootloader write config: grub2-mkconfig: Child process exited with code 1
Nov 03 20:26:31 fedora systemd[1]: ostree-finalize-staged.service: Control process exited, code=exited, status=1/FAILURE
Nov 03 20:26:31 fedora systemd[1]: ostree-finalize-staged.service: Failed with result 'exit-code'.
Nov 03 20:26:31 fedora systemd[1]: Stopped ostree-finalize-staged.service - OSTree Finalize Staged Deployment.
Nov 03 20:26:31 fedora systemd[1]: ostree-finalize-staged.service: Consumed 1.194s CPU time.
-- Boot baf5bb395de343ef9e9623329ce522aa --
Nov 03 20:30:40 fedora systemd[1]: Finished ostree-finalize-staged.service - OSTree Finalize Staged Deployment.
Nov 03 20:36:02 fedora systemd[1]: Stopping ostree-finalize-staged.service - OSTree Finalize Staged Deployment...
Nov 03 20:36:02 fedora ostree[8074]: Finalizing staged deployment
Nov 03 20:36:02 fedora ostree[8074]: Copying /etc changes: 35 modified, 0 removed, 55 added
Nov 03 20:36:02 fedora ostree[8074]: Copying /etc changes: 35 modified, 0 removed, 55 added
Nov 03 20:36:03 fedora ostree[8074]: Finalized deployment

jmarrero@fedora ~> sudo env OSTREE_DEBUG_GRUB2=1 ostree admin finalize-staged
[sudo] password for jmarrero: 
Copying /etc changes: 32 modified, 0 removed, 56 added
/usr/sbin/grub2-probe: error: failed to get canonical path of `overlay'.
error: Bootloader write config: grub2-mkconfig: Child process exited with code 1

Looks like this might be a bit different from #3715 as the error I get from Grub seems different. I also tried the two workaround listed in that issue, with no luck.

[jlebon]

Do you actually need dual-booting? If not, you can stop using grub2-mkconfig entirely by switching to sysroot.bootloader=none but you need to make sure you have a new enough GRUB first (see fedora-silverblue/issue-tracker#120). If you're not sure, a safer option that should work is a dropin to ostree-finalize-staged.service that sets Environment=GRUB_DISABLE_OS_PROBER=true.

[jmarrero]

@jlebon when you say dual-booting you mean two different OSes? I just use grub to select the different ostree deployments. I just installed silverblue on this new laptop a few weeks back. If you mean it thinks It needs to support multiple OSes I wonder what I did different vs my previous machine.

[cgwalters]

Copying /etc changes: 32 modified, 0 removed, 56 added
/usr/sbin/grub2-probe: error: failed to get canonical path of `overlay'.
error: Bootloader write config: grub2-mkconfig: Child process exited with code 1

This is triggered by combination of composefs (which uses overlayfs) and the legacy case of not having sysroot.bootloader=none. os-prober is getting confused by the composefs (really overlayfs).

WARNING: These instructions may cause failure
If you have a new enough GRUB (and anyone having done a fresh Fedora install since at least F37 does) then you can:

sudo ostree config set sysroot.bootloader none

Now, this issue is quite similar to coreos/coreos-installer#1203 - we may need to teach os-prober to do the same. (See also containers/composefs#144 )

[jmarrero]

This left me without being able to boot and with a grub prompt. I will try to replicate the issue in a VM once I finish getting my machine running again. Given that I had enabled composefs I wonder if this is related at all from what the OP was experiencing.

[cgwalters]

It's definitely a different triggering cause yeah. I hid the instructions above for now.

[MateusRodCosta]

I tried composefs due to https://blogs.gnome.org/alexl/2024/01/15/testing-composefs-in-silverblue/ and got similar results.

Initially I wasn't able to create new deployments and it also apparently prevented me from getting new commits at all until I rebased.

I also managed to bork GRUB and has to boot on a live session to tell it to regenerate the grub config.

[ruihe774]

I find that it is not relevant to os-prober and setting GRUB_DISABLE_OS_PROBER=true will not work. It is caused by line 142 of /usr/sbin/grub2-mkconfig:

# Device containing our userland.  Typically used for root= parameter.
GRUB_DEVICE="`${grub_probe} --target=device /`"
GRUB_DEVICE_UUID="`${grub_probe} --device ${GRUB_DEVICE} --target=fs_uuid 2> /dev/null`" || true
GRUB_DEVICE_PARTUUID="`${grub_probe} --device ${GRUB_DEVICE} --target=partuuid 2> /dev/null`" || true