Dual boot support for bootupd with static GRUB config installations

[AdamWill]

bootupd & static GRUB configs don't support dual boot yet.

---

Previous title: Install of Silverblue installer image with bootupd enabled fails due to grub2-mkconfig failure

Describe the bug
After https://pagure.io/workstation-ostree-config/c/ad61c4d56e45eef8a7ac8af47f72fb4f463892e9?branch=main landed and the openQA ostree installer test started generating ostrees with bootupd, install of the Silverblue image built around the ostree started to fail. The ostree admin instutil set-kargs command that is run to configure the kernel args on the bootupd path fails, apparently because a grub2-mkconfig command it runs fails. It does not tell me why the grub2-mkconfig run fails.

To Reproduce
Please describe the steps needed to reproduce the bug:

1. Get or build a Silverblue installer image containing an ostree with bootupd enabled, like this one
2. Try and install it. Observe the failure during post-install bootloader configuration

Expected behavior
The install should complete successfully.

Additional context
I tried reproducing the issue locally, then switching to a VT and running the failing ostree admin instutil set-kargs command locally, after chroot /mnt/sysroot. That does indeed fail, with the error error: Bootloader write config: grub2-mkconfig: Child process exited with code 1. It does not pass along anything about the grub2-mkconfig failure, and /mnt/sysroot is read-only so I can't do any hacks to try and get the output from grub2-mkconfig dumped anywhere, really. If I run the ostree command with -v it gives me more verbose output before the grub2-mkconfig command fails, but nothing more useful about the actual failure:

OT: using fuse: 0
OT: Deployment (long id) unlocked=0
OT: Using bootloader configuration: auto
OT: Using bootloader: OstreeBootloaderGrub2
OT: Using grub2-mkconfig chroot: (null)

@travier has reverted the bootupd change once more for now, but we should figure out the failure so it can be turned on. Again.

[travier]

Looks like it failed because it used a previous commit without bootupd installed thus anaconda could not detect it and use the new bootupd aware installation method.

We'll give it a try again now that the compose build failures should be fixed.

[travier]

Audit PR: https://src.fedoraproject.org/rpms/audit/pull-request/11
Bootupd PR: https://pagure.io/workstation-ostree-config/pull-request/453

[AdamWill]

No, that diagnosis definitely does not sound right.

I checked the last pass and first fail in openQA when this started happening, and bootupd being present in the ostree is exactly the difference: bootupd was not in the ostree in the last pass, and was in the ostree in the first failure.

anaconda uses exactly the presence of bootupd in the ostree as its basis for deciding whether to go down this path at all: rhinstaller/anaconda@8e690d5

if bootupd isn't there, it won't go down this path, it will use its conventional bootloader install code.

And the openQA test builds an ostree before it builds the ostree installer image. It cannot really 'use an old ostree' like the compose process can, it doesn't have any. The only ostree it can use is the one it builds. If that step fails, the test will die and never reach the ostree installer build phase.

[AdamWill]

The ISO I uploaded for testing is from this openQA test. If you check its ostree build log, you will see bootupd listed there.

[travier]

Ah thanks, I had missed that this was from an image built during the openqa test run. Then this likely means that we'll have to do another Anaconda fix. CC @jkonecny12

[jkonecny12]

I see. I wonder how to resolve this. I guess we need to add a check if it is container source?

[jkonecny12]

I wonder @travier should we use the bootupd for the ostree standard installation or only for containers?

[jkonecny12]

Also would it be possible to remove bootupd from the ostree base layer? What is the purpose.

We are currently over capacity, if possible we need to minimize our work.

[travier]

I wonder @travier should we use the bootupd for the ostree standard installation or only for containers?

We don't have that option as the same manifests are used for both.

Also would it be possible to remove bootupd from the ostree base layer? What is the purpose.

We are currently over capacity, if possible we need to minimize our work.

The goal of https://fedoraproject.org/wiki/Changes/FedoraSilverblueBootupd is to fix:

• Can't update dbx via fwupdmgr #355
• Use bootupd and remove ostree-grub2 (was: All deployments are shown twice in grub) #120

We've reverted the change so it's not in right now, but it's a core part of the Ostree Native Containers work.

[jkonecny12]

@AdamWill could you please provide us the ISO which were failing? We need something to debug the issue.

[AdamWill]

I already did, in the initial issue description, where it says "Get or build a Silverblue installer image containing an ostree with bootupd enabled, like this one".

[jkonecny12]

I see now, I guess I'm just blind :( . Thanks!

[travier]

DEBUG:anaconda.modules.payloads.payload.rpm_ostree.rpm_ostree:Internal mounts are set to: ['/mnt/sysroot/usr', '/mnt/sysroot/dev', '/mnt/sysroot/proc', '/mnt/sysroot/run', '/mnt/sysroot/sys', '/mnt/sysroot/var', '/mnt/sysroot/boot', '/mnt/sysroot/home', '/mnt/sysroot/sysroot']
INFO:anaconda.core.threads:Thread Done: AnaTaskThread-PrepareOSTreeMountTargetsTask-1 (140269177538240)                                                                                                                                                                                                                        
DEBUG:dasbus.connection:Publishing an object at /org/fedoraproject/Anaconda/Modules/Payloads/Task/10.                                                          
DEBUG:dasbus.connection:Publishing an object at /org/fedoraproject/Anaconda/Modules/Payloads/Task/11.                                                                                                                                                                                                                          
DEBUG:dasbus.connection:Publishing an object at /org/fedoraproject/Anaconda/Modules/Payloads/Task/12.                                                                                                                                                                                                                          
INFO:anaconda.core.threads:Running Thread: AnaTaskThread-CopyDriverDisksFilesTask-1 (140269177538240)                                                          
INFO:anaconda.modules.common.task.task:Copy driver disks files                                                                                                 
INFO:anaconda.core.threads:Thread Done: AnaTaskThread-CopyDriverDisksFilesTask-1 (140269177538240)                                                             
INFO:anaconda.core.threads:Running Thread: AnaTaskThread-ChangeOSTreeRemoteTask-2 (140269177538240)                                                                                                                                                                                                                            
INFO:anaconda.modules.common.task.task:Change OSTree remote                                                                                                     
INFO:anaconda.core.threads:Thread Done: AnaTaskThread-ChangeOSTreeRemoteTask-2 (140269177538240)                                                                                                                                                                                                                               
INFO:anaconda.core.threads:Running Thread: AnaTaskThread-ConfigureBootloader-1 (140269177538240)                                                                
INFO:anaconda.modules.common.task.task:Configure OSTree bootloader
INFO:program:Running in chroot '/mnt/sysroot'... bootupctl backend install --auto --write-uuid --device /dev/vda /                                                                                                                                                                                                             
INFO:program:Installed: grub.cfg                                                                                                                                
DEBUG:program:Return code: 0                                                                                                                                    
INFO:program:Running in chroot '/mnt/sysroot'... ostree admin instutil set-kargs rhgb quiet root=UUID=d2d7152e-6bdb-4f1d-affc-83f2b7114103 rootflags=subvol=root rw
INFO:program:error: Bootloader write config: grub2-mkconfig: Child process exited with code 1                                                                  
DEBUG:program:Return code: 1   
INFO:anaconda.core.threads:Thread Failed: AnaTaskThread-ConfigureBootloader-1 (140269177538240)                                                                                                                                                                                                                                
ERROR:anaconda.modules.common.task.task:Thread AnaTaskThread-ConfigureBootloader-1 has failed: Traceback (most recent call last):                              
  File "/usr/lib64/python3.12/site-packages/pyanaconda/core/threads.py", line 280, in run                                                                                                                                                                                                                                      
    threading.Thread.run(self)                                                  
  File "/usr/lib64/python3.12/threading.py", line 1010, in run                                                                                                  
    self._target(*self._args, **self._kwargs)                                                                                                                  
  File "/usr/lib64/python3.12/site-packages/pyanaconda/modules/common/task/task.py", line 94, in _thread_run_callback                                          
    self._task_run_callback()                                                   
  File "/usr/lib64/python3.12/site-packages/pyanaconda/modules/common/task/task.py", line 107, in _task_run_callback                                           
    self._set_result(self.run())                                               
                     ^^^^^^^^^^                                                
  File "/usr/lib64/python3.12/site-packages/pyanaconda/modules/payloads/payload/rpm_ostree/installation.py", line 505, in run                                                                                                                                                                                                  
    self._set_kargs()                                                          
  File "/usr/lib64/python3.12/site-packages/pyanaconda/modules/payloads/payload/rpm_ostree/installation.py", line 568, in _set_kargs                                                                                                                                                                                           
    safe_exec_with_redirect("ostree", set_kargs_args, root=self._sysroot)                                                                                      
  File "/usr/lib64/python3.12/site-packages/pyanaconda/modules/payloads/payload/rpm_ostree/installation.py", line 54, in safe_exec_with_redirect                                                                                                                                                                               
    raise PayloadInstallationError(                                            
pyanaconda.modules.common.errors.installation.PayloadInstallationError: The command 'ostree admin instutil set-kargs rhgb quiet root=UUID=d2d7152e-6bdb-4f1d-affc-83f2b7114103 rootflags=subvol=root rw' exited with the code 1.

INFO:anaconda.core.threads:Thread Done: AnaTaskThread-ConfigureBootloader-1 (140269177538240)                                                                                                                                                                                                                                  
WARNING:dasbus.server.handler:The call org.fedoraproject.Anaconda.Task.Finish has failed with an exception:                                                                                                                                                                                                                    
Traceback (most recent call last):                                             
  File "/usr/lib/python3.12/site-packages/dasbus/server/handler.py", line 455, in _method_callback                                                                                                                                                                                                                             
    result = self._handle_call(                                                
             ^^^^^^^^^^^^^^^^^^                                                
  File "/usr/lib/python3.12/site-packages/dasbus/server/handler.py", line 265, in _handle_call                                                                                                                                                                                                                                 
    return handler(*parameters, **additional_args)                                                                                                             
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                             
  File "/usr/lib64/python3.12/site-packages/pyanaconda/modules/common/task/task_interface.py", line 114, in Finish                                                                                                                                                                                                             
    self.implementation.finish()                                               
  File "/usr/lib64/python3.12/site-packages/pyanaconda/modules/common/task/task.py", line 173, in finish                                                                                                                                                                                                                       
    thread_manager.raise_if_error(self._thread_name)                                                                                                           
  File "/usr/lib64/python3.12/site-packages/pyanaconda/core/threads.py", line 171, in raise_if_error                                                                                                                                                                                                                           
    raise exc_info[1]                                                          
  File "/usr/lib64/python3.12/site-packages/pyanaconda/core/threads.py", line 280, in run                                                                                                                                                                                                                                      
    threading.Thread.run(self)                                                 
  File "/usr/lib64/python3.12/threading.py", line 1010, in run                                                                                                 
    self._target(*self._args, **self._kwargs)                                                                                                                  
  File "/usr/lib64/python3.12/site-packages/pyanaconda/modules/common/task/task.py", line 94, in _thread_run_callback                                                                                                                                                                                                          
    self._task_run_callback()                                                  
  File "/usr/lib64/python3.12/site-packages/pyanaconda/modules/common/task/task.py", line 107, in _task_run_callback                                                                                                                                                                                                           
    self._set_result(self.run())                                               
                     ^^^^^^^^^^                                                
  File "/usr/lib64/python3.12/site-packages/pyanaconda/modules/payloads/payload/rpm_ostree/installation.py", line 505, in run                                                                                                                                                                                                  
    self._set_kargs()                                                          
  File "/usr/lib64/python3.12/site-packages/pyanaconda/modules/payloads/payload/rpm_ostree/installation.py", line 568, in _set_kargs                                                                                                                                                                                           
    safe_exec_with_redirect("ostree", set_kargs_args, root=self._sysroot)                                                                                      
  File "/usr/lib64/python3.12/site-packages/pyanaconda/modules/payloads/payload/rpm_ostree/installation.py", line 54, in safe_exec_with_redirect                                                                                                                                                                               
    raise PayloadInstallationError(                                            
pyanaconda.modules.common.errors.installation.PayloadInstallationError: The command 'ostree admin instutil set-kargs rhgb quiet root=UUID=d2d7152e-6bdb-4f1d-affc-83f2b7114103 rootflags=subvol=root rw' exited with the code 1.

[travier]

$ grub2-mkconfig
### BEGIN /etc/grub.d/15_ostree ###
/etc/grub.d/15_ostree: line 31: /etc/default/grub: No such file or directory

😕

[travier]

[anaconda root@fedora /]# rpm -ql grub2-tools | grep default
/etc/default/grub
[anaconda root@fedora /]# rpm -qf /etc/default/grub
grub2-tools-2.06-110.fc40.x86_64

🤔

[travier]

$ ostree ls --repo=repo 50ddc44f72905d9b463ffdf27119e670d7911629ae143644ab3809307a8746ff /usr/etc/default
d00755 0 0      0 /usr/etc/default
-00644 0 0    123 /usr/etc/default/useradd

🤔 🤔 🤔

[travier]

OK, this is gone in latest Rawhide commits as well so this is not a bootupd issue.

[travier]

Somehow, this file is not installed anymore: https://src.fedoraproject.org/rpms/grub2/blob/rawhide/f/grub2.spec#_436

[travier]

Hum, it's not there on my F39 system either.

[travier]

OK, it's anaconda writing this file in the GRUB2 installation path and likely not in the bootupd one: https://github.com/rhinstaller/anaconda/blob/c0d803edeeb8d7797492250353e52180f5ad6a33/pyanaconda/modules/storage/bootloader/grub2.py#L253

[travier]

CC @jkonecny12

[travier]

I suspect rhinstaller/anaconda#5350

[travier]

OK, so we need to setup this config but only if the legacy ostree GRUB hooks/config are there (the ones that we want to remove after this change lands and people have time to update their bootloaders).

[travier]

Or we can update this condition / check here: https://github.com/ostreedev/ostree/blob/main/src/boot/grub2/grub2-15_ostree#L31

[travier]

Not 100% sure yet if my understanding is correct but made: ostreedev/ostree#3150

[AdamWill]

you're correct that anaconda writes it, here. GRUB2.write_defaults() is called by GRUB2.write_config(), which is called by GRUB2.write(), which is called by InstallBootloaderTask.run(), using self._storage.bootloader as the bootloader class (so only when that is a GRUB2 instance). That's called by BootloaderModule.install_bootloader_with_tasks(), which has claimed to be "just a temporary method" since March 2019. That's called by BootloaderInterface.InstallBootloaderWithTasks(), which is called by RunInstallationTask._prepare_installation() (using an internal method). See? Simple...:P

The thing that is InstallBootloaderTask._storage is probably an instance of InstallerStorage, whose bootloader property is the result of BootLoaderFactory.create_boot_loader(), which via a slightly funky bit of logic usually winds up autodetecting the bootloader class via get_class_by_platform(): BootloaderModule winds up trying to do BootLoaderFactory.set_default_class("DEFAULT") via BootloaderType.DEFAULT, which the factory's get_class_by_name doesn't have a mapping for, so it returns None, which means BootLoaderFactory.get_class() falls through to get_class_by_platform()...all unless the user specifies extlinux or sdboot via configuration.

AAANYHOOO, what's actually happening here is that all of this is getting skipped because we hit a check in InstallBootloaderTask.run() (remember, that's the thing that calls write(), that calls write_config(), that writes the file). There's a check if self._payload_type == PAYLOAD_TYPE_RPM_OSTREE and have_bootupd(self._sysroot): which bails out of run() before it calls self._bootloader.write().

[travier]

Thanks for digging 🕳️ into this!

[travier]

I've verified that ostreedev/ostree#3150 fixes the installation

[travier]

And I've tested basic bootupd function as well! Will try an update now.

[cgwalters]

Adding bootupd to a payload currently also defaults to --with-static-configs which I think is really the correct way to do things by default for dedicated Linux boxes. However...there may be people relying on more esoteric features of dynamic grub configs. For example, grub does this thing where it scans all your block devices and is like "Oh I see a Haiku install here, I'll make an entry for that" etc. And similarly for Windows I think.

But IMO, the right way to handle Windows at least is via the EFI bootloader, not grub.

Having new desktop (silverblue, etc.) installs use bootupd in this way will be awesome because it will align everything and work the same way as FCOS and how bootc does it.

But...transitioning old installs seems tricky and potentially risky.

[travier]

Considering that we "already" have dual booting issues on Silverblue (#284), I'm tempted to say that we'll not fix this for Fedora 40. We should be able to document adding a Windows boot entry to the static GRUB config for new installations.

As we're not force upgrading users (yet), systems that were installed before will keep the dynamic GRUB configs for now.

[travier]

I've tested installing Fedora Silverblue 37, updating all the way to 39 then rebasing to Rawhide and adopting using bootupd adopt-and-update.

[jkonecny12]

Great work @travier! Sorry that I wasn't able to help you more :(.

I wonder what is the correct behavior here... Should Anaconda create the /etc/default/grub or not? For OSTree based installations I would guess that this file shouldn't be created or it should be handled by bootupd?

[travier]

For bootupd installations, we don't need anaconda to create this file right now as we will use a static GRUB config and BLS to get the ostree boot entries.

This will not generate boot entries for Windows, etc. for dual boot but we already don't have that working well so it's not a great loss.

I'll try to document how to manually add boot entries for Windows & Co.

[travier]

Related to ostreedev/ostree#3198 for GRUB static config + dual boot with composefs enabled.