Fixes #36833 - Add SecureBoot support for arbitrary operating systems to "Grub2 UEFI" PXE loaders #877

goarsna · 2023-10-16T07:21:49Z

This feature consists of two patches, one for foreman and one for smart-proxy.

This patch introduces a new loader of kind :PXEGrub2TargetOS which allows to provide host-specific Network Bootstrap Programs (NPB) in order to enable network based installations for SecureBoot-enabled hosts.

SecureBoot expects to follow a chain of trust from the start of the host to the loading of Linux kernel modules. The very first shim that is loaded basically determines which distribution is allowed to be booted or kexec'ed until next reboot.

The existing "Grub2 UEFI SecureBoot" is not sufficiant as it limits the possible installations to the vendor of the Foreman (Smart Proxy) host system.

Providing shim and GRUB2 by the vendor of the to-be-installed operating system allows Foreman to install any operating system on SecureBoot-enabled hosts over network.

To achieve this, the host's DHCP filename option is set to a shim path in a directory that is host-specific (contains MAC address). Corresponding shim and GRUB2 binaries are copied into that directory along with the generated GRUB2 configuration files as we know from "Grub2 UEFI".

The required binaries must be provided once in the so called "bootloader universe". This directory can be configured via the settings file /etc/foreman-proxy/settings.d/tftp.yml and defaults to /usr/local/share/bootloader-universe/<os>/. These binaries can be manually retrieved from the installation media and is not part of this patchset.

Full example:

[root@vm ~]# hammer host info --id 241 | grep -E "(MAC address|Operating System)"
    MAC address:  00:50:56:b4:75:5e
    Operating System:       Ubuntu 22.04 LTS

[root@vm ~]# tree /usr/local/share/bootloader-universe/ /usr/local/share/bootloader-universe/
|-- centos
|   |-- grubx64.efi
|   `-- shimx64.efi
`-- ubuntu
    |-- grubx64.efi
    `-- shimx64.efi

[root@vm ~]# hammer host update --id 241 --build true

[root@vm ~]# tree /var/lib/tftpboot/grub2/00-50-56-b4-75-5e/ /var/lib/tftpboot/grub2/00-50-56-b4-75-5e/
|-- grub.cfg
|-- grub.cfg-00:50:56:b4:75:5e
|-- grub.cfg-01-00-50-56-b4-75-5e
|-- grubx64.efi
|-- shimx64.efi
`-- targetos

[root@vm ~]# grep -B2 00-50-56-b4-75-5e /var/lib/dhcpd/dhcpd.leases
  hardware ethernet 00:50:56:b4:75:5e;
  fixed-address 192.168.145.84;
        supersede server.filename = "grub2/00-50-56-b4-75-5e/shimx64.efi";

[root@vm ~]# pesign -S -i /var/lib/tftpboot/grub2/00-50-56-b4-75-5e/grubx64.efi | grep Canonical The signer's common name is Canonical Ltd. Secure Boot Signing (2021 v1)

theforeman-bot · 2023-10-16T07:21:53Z

Can one of the admins verify this patch?

goarsna · 2023-10-17T12:53:30Z

I have opened a PR to document this feature in foreman-documentation:
theforeman/foreman-documentation#2145

goarsna · 2023-10-20T08:41:44Z

This PR also requires the changes from Foreman PR #9864

ekohl

I'd advertise this as an optional capability. In https://theforeman.org/2019/04/smart-proxy-capabilities-explained.html I've written about this. The benefit is two fold. On the one hand, it can disabled if bootloader_universe is not set. Foreman can detect this. It can also help with n-1 support where Foreman x.y runs with a Foreman Proxy x.(y-1).

One example where you can see it in Foreman is https://github.com/theforeman/foreman/blob/66f0c87b26c9b3a125bd216eb26c9a28b06effb3/app/models/concerns/orchestration/dhcp.rb#L84-L86 but Katello also uses it for the various content types.

modules/tftp/server.rb

modules/tftp/tftp_api.rb

goarsna · 2024-01-25T14:43:40Z

I encountered an error on a standalone Smart Proxy with the check if the bootloader directory is configured and fixed it.

goarsna · 2024-02-15T08:45:30Z

Hi there,
I have rethought the whole concept of how we wanted to add SecureBoot support for arbitrary operating systems due to the discussions in Foreman Community Forum and came up with an idea how to completely integrate what we want into the existing PXEGrub2 PXE loaders.

What's left to do for now:

I will have to take a look at the tests after my changes.
As this approach generates MAC specific directories directly inside /var/lib/tftpboot/ the ownership of this directory has to be changed to user foreman-proxy.
One open request by @lzap was to provide differentiated bootloader files in the bootloader_universe not only by OS vendor but also by OS version.

@sbernhard @ekohl What do you think? Please share your thoughts. I'm looking forward to input and discussion.

goarsna · 2024-02-28T20:34:19Z

Implemented changes as discussed in the course of the Foreman Community Forum thread.

Please let me know, if anything is missing or further changes are required.

goarsna · 2024-02-29T12:52:32Z

Added deletion of host specific bootloader files in case host leaves build mode.

sbernhard · 2024-03-01T15:41:20Z

[test smart-proxy]

ekohl · 2024-03-06T14:43:14Z

ok to test

… to "Grub2 UEFI" PXE loaders In the course of theforeman/foreman#9864 and theforeman/smart-proxy#877, SecureBoot support for arbitrary operating systems has been added to the "Grub2 UEFI" PXE loaders. This patch adds a new parameter 'bootloader_universe' to the TFTP configuration and a directory 'host_config' inside the TFTP root directory, that are both required by the aforementioned PRs.

goarsna · 2024-03-10T20:23:01Z

Squashed and changed commit message to reflect current implementation. => Any objections against merging this? What do you think, @nofaralfasi and @stejskalleos?

stejskalleos

I'm testing the PR (together with Foreman), and I'm getting this error on Smart Proxy:

2024-06-27T11:05:48 98344422 [D] TFTP: Host is not in build mode.
2024-06-27T11:05:48 98344422 [D] TFTP:   => Removing host specific bootloader files from "/var/lib/tftpboot//host_config/00-aa-bb-6b-06-0f/grub2"
2024-06-27T11:05:48 98344422 [E] TFTP: Failed to create pxe config file: Permission denied @ dir_s_mkdir - /var/lib/tftpboot//host_config
2024-06-27T11:05:48 98344422 [W] Error details for TFTP: Failed to create pxe config file: Permission denied @ dir_s_mkdir - /var/lib/tftpboot//host_config: <Errno::EACCES>: Permission denied @ dir_s_mkdir - /var/lib/tftpboot//host_config

# config/settings.d/tftp.yml

:enabled: true
:tftproot: /var/lib/tftpboot/
:tftp_servername: 192.168.190.1
:bootloader_universe: /var/lib/tftpboot/ducky_mcface/

Some notes to this:

Host is not in build mode. That's not true; I literally created the host
It looks like it's ignoring the value of bootloader_universe, I don't see any ducky_mcface in the logs
I checked permissions & ownership of /var/lib/tftpboot, and it's fine, see:

drwxr-xr-x. 1 root     root        196 Jun 27 10:55 .
drwxr-xr-x. 1 root     root       1064 May  2 09:28 ..
drwxr-xr-x. 1 lstejska lstejska    142 Jun 24 10:08 boot
-rwxr-xr-x. 1 lstejska lstejska  25108 May 14 14:46 chain.c32
drwxr-xr-x. 1 lstejska lstejska      0 Jun 27 10:55 ducky_mcface
drwxr-xr-x. 1 lstejska lstejska    122 Jun 18 13:53 grub
drwxr-xr-x. 1 lstejska lstejska   1058 Jun 24 10:08 grub2
-rwxr-xr-x. 1 lstejska lstejska 115452 May 14 14:46 ldlinux.c32
-rwxr-xr-x. 1 lstejska lstejska 178492 May 14 14:46 libcom32.c32
-rwxr-xr-x. 1 lstejska lstejska  23632 May 14 14:46 libutil.c32
-rwxr-xr-x. 1 lstejska lstejska  26240 May 14 14:46 menu.c32
-rw-r--r--. 1 lstejska lstejska  42600 May 14 14:46 pxelinux.0
drwxr-xr-x. 1 lstejska lstejska    424 Jun 27 11:05 pxelinux.cfg

@nofaralfasi does it work for you or it's something funky on my setup?

stejskalleos · 2024-06-27T09:24:08Z

Followup to my previous comment, if I comment the bootloader_universe setting, I still get the error:

2024-06-27T11:20:02 a6b61ad4 [W] TFTP: Failed to create pxe config file: Permission denied @ dir_s_mkdir - /var/lib/tftpboot//host_config: <Errno::EACCES>: Permission denied @ dir_s_mkdir - /var/lib/tftpboot//host_config

If I create the host_config directory under /var/lib/tftpboot, it will work.

If the setting is not set/enabled, provisioning should work as usual, keeping the backward compatibility.

stejskalleos · 2024-06-27T11:28:19Z

Ha, Checking the puppet-foreman_proxy installer will handle the creation of the host_config directory.

Please ignore my previous comment; I'll continue with the testing.

stejskalleos · 2024-06-27T12:50:12Z

modules/tftp/server.rb

+
+        File.write(File.join(pxeconfig_dir_mac, 'os_info'), "#{os} #{version} #{arch}")
+      else
+        logger.debug "TFTP: Host is not in build mode."


Everytime I create new host, I'll see this message in the log.

2024-06-27T14:47:21 d924fd47 [D] TFTP: Host is not in build mode. 2024-06-27T14:47:21 d924fd47 [D] TFTP: => Removing host specific bootloader files from "/var/lib/tftpboot//host_config/00-aa-bb-6b-06-dd/grub2" 2024-06-27T14:47:21 d924fd47 [D] TFTP: /var/lib/tftpboot//host_config/00-aa-bb-6b-06-dd/grub2/grub.cfg created successfully 2024-06-27T14:47:21 d924fd47 [D] TFTP: /var/lib/tftpboot//host_config/00-aa-bb-6b-06-dd/grub2/grub.cfg-01-00-aa-bb-6b-06-dd created successfully 2024-06-27T14:47:21 d924fd47 [D] TFTP: /var/lib/tftpboot//host_config/00-aa-bb-6b-06-dd/grub2/grub.cfg-00:aa:bb:6b:06:dd created successfully 2024-06-27T14:47:21 d924fd47 [D] TFTP: /var/lib/tftpboot//grub2/grub.cfg-01-00-aa-bb-6b-06-dd created successfully 2024-06-27T14:47:21 d924fd47 [D] TFTP: /var/lib/tftpboot//grub2/grub.cfg-00:aa:bb:6b:06:dd created successfully 2024-06-27T14:47:21 d924fd47 [I] Finished POST /tftp/PXEGrub2/00:aa:bb:6b:06:dd with 200 (1.59 ms)

Why is it needed?

I don't see it. When I create a new host, I see the Host is in build mode. message in the log. However, I would expect to see this message (Host is not in build mode.) when the host is not in build mode (if I cancel it manually, or after the provisioning ends for this host). Instead, I get the Host is in build mode. message regardless. This wasn't the case in the past; I think the recent changes are causing this issue.

Hey, I tried to verify it and I can confirm, this worked differently before (at least to the best of my knowledge and my notes 😅). 🤔

I think the recent changes are causing this issue.

I have to further investigate that and have to take a look at the history of the PR.
The initial idea of the "delete the host specific bootloader files in case build is false" was to remove the existing bootloader files in case a host is rebuilt. But my current guess is that there is in general no need to rely on the build flag for that. It would be a way more atomic (and simpler) approach to simply delete the existing host specific bootloader files before deploying new bootloader files in the host specific directory. But I have to verify all of this.

Unfortunately today is my day off, therefor I will have to take a closer look on Monday. 🙈

Ok, now I am wiser 😅

Indeed it was intentional to delete the bootloader files when the host leaves Build Mode (i.e. when provisioning is finished) but with the knowledge I gained in the course of these PRs this doesn't make sense at all as also the provisioned hosts require the files in the host_config/<MAC>/grub2 directory to boot.
So it seems that the deletion of the bootloader files has never worked as it was intended to as then no host that uses any Grub2 UEFI* PXE loader could have been rebooted after being deployed.

I propose to remove the deletion depending on the build flag, but instead to delete the files every time before they get deployed (i.e. every time the TFTP configuration is deployed). By this we guarantee a consistent state of the files in the Bootloader Universe and in the host_config/<MAC>/grub2 directories.

...this doesn't make sense at all as also the provisioned hosts require the files in the host_config/<MAC>/grub2 directory to boot. So it seems that the deletion of the bootloader files has never worked as it was intended to as then no host that uses any Grub2 UEFI* PXE loader could have been rebooted after being deployed.

Why do the provisioned hosts require the files in the host_config/<MAC>/grub2 directory to boot?
The provisioned hosts should boot from the local hard drive and not go through the entire provisioning process each time they are booted.

Note: During the boot process, by default the machine first tries to boot from the network (i.e., it tries to download the boot*.efi files). However, even if these files are missing, the boot process will still be successful because it will fall back to booting from the local hard drive.

The provisioned hosts should boot from the local hard drive and not go through the entire provisioning process each time they are booted.

This is basically correct.

We all agree on that we need a way to control a host's boot sequence on the server side and to make a host boot into the locally installed system or, by purpose, to boot into an installation.

So your idea of refusing to deliver any NBP when build mode is disabled is not bad. But ...

Note: During the boot process, by default the machine first tries to boot from the network (i.e., it tries to download the boot*.efi files). However, even if these files are missing, the boot process will still be successful because it will fall back to booting from the local hard drive.

This may depend on the platform (different hypervisors incl. UEFI implementations, different bare-metal vendors, etc.). I assume you are right in most of the cases, but I wouldn't put my hand in the fire for that. Also, similar to PR 10126, one would need to make sure that the boot order entries are correct: first network boot, then local disk boot.

There exists this efibootmgr_netboot snippet to configure the boot order, but I never tried this in production.

As we noticed that PR 10126 behaves differently for different distribution vendors - which would not occur with your proposal at all - we were thinking of simply loading the local GRUB2 configuration file (PR 10207). Since this in turn is described via network loaded GRUB2 configuration file, we are more flexible in making the host finding the desired boot sequence.

And allows us to mitigate the issue of an unclear fallback situation.

I'll try to summarize what we discussed here:
Since provisioned hosts also need shim/grub to boot, it's necessary to retain these binaries after provisioning. However, this raises concerns about storage usage and manual updates if binaries are updated.

After our DM discussion, we opted for a solution: implementing a dedicated bootloader_universe directory within the TFTP root (e.g., /var/lib/tftpboot/bootloader_universe). This approach involves using symlinks for shim/grub for each host consistently, as detailed here.

Reasons for this decision include:

This maintains the default behavior in Foreman (prior to this PR). Previously, we never deleted the shim/grub.

Even with the correct boot order, when a machine boots, it first tries to download the shim/grub. Therefore, we want to keep these files even when the host is not in build mode.

Using symlinks instead of copies saves storage space and simplifies maintenance if updates are required.

In the event of a certificate revocation, there's a workaround to deploy the correct shim/grub elsewhere and manually relink all affected hosts to ensure functionality.

modules/tftp/server.rb

ekohl

I didn't highlight every case of it, but some repeating patterns:

Please use File.join consistently
Please keep your logging statements concise and ideally map them to exactly one action
Consider using single quotes (') instead of double quotes (") in your logging statements

modules/tftp/server.rb

ekohl · 2024-07-08T15:20:57Z

modules/tftp/server.rb

+        logger.debug "TFTP:   - \"#{bootloader_path}/*\" => \"#{pxeconfig_dir_mac}/\""
+        FileUtils.cp_r("#{bootloader_path}/.", "#{pxeconfig_dir_mac}/", remove_destination: true)
+      else
+        pxeconfig_dir = pxeconfig_dir()


I think this is a bit confusing and would simplify it to just

Suggested change

pxeconfig_dir = pxeconfig_dir()

dir = pxeconfig_dir

I was thinking about it and I would like to keep it as is. Looking at the code line where we create the symlinks from the pxeconfig_dir_mac to the pxeconfig_dir it's IMO more obvious where we link to when we talk about the pxeconfig_dir.

modules/tftp/server.rb

Affects "Grub2 UEFI" PXE loaders * PR in foreman: theforeman/foreman#9864 * PR in smart-proxy: theforeman/smart-proxy#877 * RFC: https://community.theforeman.org/t/add-secureboot-support-for-arbitrary-distributions/32601/1

… to "Grub2 UEFI" PXE loaders In the course of theforeman/foreman#9864 and theforeman/smart-proxy#877, SecureBoot support for arbitrary operating systems has been added to the "Grub2 UEFI" PXE loaders. This patch adds a new parameter 'bootloader_universe' to the TFTP configuration and a directory 'host_config' inside the TFTP root directory, that are both required by the aforementioned PRs.

… to "Grub2 UEFI" PXE loaders In the course of theforeman/foreman#9864 and theforeman/smart-proxy#877, SecureBoot support for arbitrary operating systems has been added to the "Grub2 UEFI" PXE loaders. This patch adds the 'bootloader-universe' and 'host-config' directories inside the TFTP root, that are both required by the aforementioned PRs.

… to "Grub2 UEFI" PXE loaders This feature consists of four patches, one each for foreman, smart-proxy, foreman-installer, and puppet-foreman_proxy. This patch adds support for individual Network Bootstrap Programs (NBP) in order to enable network based installations of SecureBoot enabled hosts for arbitrary operating systems. SecureBoot expects to follow a chain of trust from the initial boot of the host to the loading of Linux kernel modules. The very first shim that is loaded determines which distribution is allowed to be booted or kexec'ed until next reboot. Currently the "Grub2 UEFI SecureBoot" PXE loaders use NBPs provided by the vendor of the Foreman/Smart Proxy host system. All hosts receive and execute the same binary. On SecureBoot enabled hosts, this limits installations to operating systems by the vendor of the Foreman/ Smart Proxy host system. Providing shim and GRUB2 by the vendor of the operating system to be installed allows Foreman to install any operating system on SecureBoot enabled hosts over network. To achieve this, the host's DHCP filename option is set to a shim/GRUB2 binary in a host specific directory based on their MAC address. Corresponding shim and GRUB2 binaries are copied into that directory along with the generated GRUB2 configuration files. When provisioning a host, the Smart Proxy checks in a dedicated directory inside the TFTP root - the so called "bootloader universe" - if NBPs are present matching the operating system, operating system version, and architecture of the host to be installed. If this is the case, these NBPs are copied from the bootloader universe directory to the host specific directory. If not, as a fallback the default NBPs provided by the vendor of the Foreman/Smart Proxy host system are copied from the `:tftproot:/grub2` directory to the host specific directory. Up to now, shim and GRUB2 binaries have to be retrieved and set up in the bootloader universe directory manually according to the documentation. An automatic way to provide OS dependent NBPs will be added in future. In case there are no NBPs present in the bootloader universe matching the operating system, operating system version, and architecture of the host to be installed, the behaviour of the "Grub2 UEFI" PXE loaders does not change to the behavior prior to this feature. Implementation notes: --------------------- * To be future proof (e.g. to be able to provide NBPs in the bootloader universe for other PXE loaders without running into any filename conflicts) and for better structure, the PXE kind is prepended as a first directory level inside the bootloader universe. * The operating system version inside the bootloader universe consists of the major and minor version (if applicable) of the operating system separated by a dot (`.`). If no NBPs are configured for a specific operating system version the fallback directory `default` is used. * To simplify things on Foreman side in future, symlinks are used for the shim (boot-sb.efi) and GRUB2 (boot.efi) binaries. * Inside the TFTP root directory a new directory `host-config` is created for storing all the host specific directories. * Inside the TFTP root directory a new directory `bootloader-universe` is created for storing all the OS specific boot files. * For storage efficiency the shim and GRUB2 binaries from the bootloader universe or the `:tftproot:/grub2` directory are symlinked to the host specific directory. Full example: ------------- [root@vm ~]# hammer host info --id 241 | grep -E "(MAC address|Operating System)" MAC address: 00:50:56:b4:75:5e Operating System: AlmaLinux 8.9 [root@vm ~]# tree /var/lib/tftpboot/bootloader-universe/ /var/lib/tftpboot/bootloader-universe/ └── pxegrub2 └── almalinux ├── 8.9 │ └── x86_64 │ ├── boot.efi -> grubx64.efi │ ├── boot-sb.efi -> shimx64.efi │ ├── grubx64.efi │ └── shimx64.efi └── default └── x86_64 ├── boot.efi -> grubx64.efi ├── boot-sb.efi -> shimx64.efi ├── grubx64.efi └── shimx64.efi [root@vm ~]# hammer host update --id 241 --build true [root@vm ~]# tree /var/lib/tftpboot/host-config /var/lib/tftpboot/host-config └── 00-50-56-a3-41-a8 └── grub2 ├── boot.efi -> ../../../bootloader-universe/grubx64.efi ├── boot-sb.efi -> ../../../bootloader-universe/shimx64.efi ├── grub.cfg ├── grub.cfg-00:50:56:a3:41:a8 ├── grub.cfg-01-00-50-56-a3-41-a8 ├── grubx64.efi -> ../../../bootloader-universe/grubx64.efi ├── os_info └── shimx64.efi -> ../../../bootloader-universe/shimx64.efi [root@vm ~]# grep -B2 00-50-56-b4-75-5e /var/lib/dhcpd/dhcpd.leases hardware ethernet 00:50:56:b4:75:5e; fixed-address 192.168.145.84; supersede server.filename = "host-config/00-50-56-b4-75-5e/grub2/boot-sb.efi"; [root@vm ~]# pesign -S -i /var/lib/tftpboot/host-config/00-50-56-b4-75-5e/grub2/boot-sb.efi | grep "Microsoft Windows UEFI Driver Publisher" The signer's common name is Microsoft Windows UEFI Driver Publisher

goarsna · 2024-08-08T13:24:09Z

Thanks @ekohl for your review! I hope I could clarify all points I resolved.

I also updated the PR accordingly to reflect the outcome of the above mentioned discussion.

This includes:

The bootloader-universe is a dedicated (unconfigurable) directory inside the TFTP root.
The host-config directory is now written with a dash.
We use symlinks in the host-config directory to enhance storage efficiency.

nofaralfasi · 2024-08-12T15:27:51Z

modules/tftp/server.rb

+
+    def pxeconfig_dir(mac = nil)
+      if mac
+        File.join(path, 'host_config', dashed_mac(mac).downcase, 'grub2')


Did you mean host-config?

Suggested change

File.join(path, 'host_config', dashed_mac(mac).downcase, 'grub2')

File.join(path, 'host-config', dashed_mac(mac).downcase, 'grub2')

nofaralfasi · 2024-08-13T14:29:28Z

modules/tftp/server.rb

+          symlink_path = File.join(pxeconfig_dir_mac, file_name)
+
+          logger.debug "TFTP: Creating relative symlink: '#{symlink_path}' -> '#{source_file}'"
+          FileUtils.ln_sr(source_file, symlink_path, force: true)


I'm using Ruby version 3.0.4, and I don't have access to FileUtils.ln_sr. This method has been available since FileUtils v1.7.0: FileUtils v1.7.0 release, but I currently have v1.6.0

We should support FileUtils version 1.4.1 and above, as discussed here.

nofaralfasi · 2024-08-13T15:25:09Z

When trying to create a new host, I'm getting the following error:

[W] TFTP: Failed to setup host specific bootloader directory: wrong number of arguments (given 1, expected 7): <ArgumentError>: wrong number of arguments (given 1, expected 7)
/smart-proxy/modules/tftp/server.rb:73:in `setup_bootloader'
/smart-proxy/modules/tftp/tftp_api.rb:23:in `block in create'
/smart-proxy/lib/proxy/helpers.rb:19:in `log_halt'
/smart-proxy/modules/tftp/tftp_api.rb:23:in `create'
/smart-proxy/modules/tftp/tftp_api.rb:52:in `block in <class:Api>'

pr-processor bot added the Not yet reviewed label Oct 16, 2023

goarsna mentioned this pull request Oct 20, 2023

Fixes #36834 - Add SecureBoot support for arbitrary operating systems to "Grub2 UEFI" PXE loaders theforeman/foreman#9864

Open

ekohl requested changes Oct 20, 2023

View reviewed changes

modules/tftp/server.rb Outdated Show resolved Hide resolved

modules/tftp/server.rb Outdated Show resolved Hide resolved

modules/tftp/server.rb Outdated Show resolved Hide resolved

pr-processor bot added Waiting on contributor and removed Not yet reviewed labels Oct 20, 2023

pr-processor bot added Needs re-review and removed Waiting on contributor labels Nov 2, 2023