Future between bootc and OSTree

[antheas]

So I spent the last 1-2 weeks looking into the rpm-ostree, built my own Fedora based ostree image using tree files, and rebased between it and a variety of ublue images around 30 times. This is as part of testing in optimizing Bazzite's update process (with some relevant findings that are not part of this issue) but I also caught a bit of the atomic bug.

In any case, the topic of this issue is getting some understanding between the future relationship of OSTree and bootc.

The documentation mentions that OSTree is an implementation detail of bootc. And perhaps for end users it is.

However, the backend plays a large role for the base image builder/OS maintainers, as it affects the following:

• bootloader (both stanzas and options)
• kernel configuration
  • Multiple kernels anyone?
• the performance characteristics of the final image
  • e.g., overlayfs vs OSTree have inverse performance characteristics as layer number increases
• Live ISO configuration (which is something I want to look into)
  • An OSTree Live ISO can deploy its OSTree to the target system (theoretically)
  • A container based ISO that boots a container can deploy that container
  • An OSTree based bootc ISO cannot deploy the OSTree and does not have space to deploy a container
• On a certain level, building the image itself (although this should change soon ?)

Therefore, it would be nice to know what the future of OSTree itself as a backend is, before investing time and resources into it.

Is there a plan for transitioning into an overlayfs podman based backend?

If yes, are there performance benefits to it? What is the timeline? 1 year, 2 years, 3 years?

Are there downsides? Nested whiteouts, kernel instability, runtime performance degradation?

If not, that is good for OSTree, and in that case, how is the podman integration looking? In order for the "build your OCI image" to become a thing, especially for layering packages, running RUN yay -S <mypackage> should be faster or just as fast as deploying an OSTree repo, running yay -S <mypackage> under a chroot with a local cache, and deploying the result. While not duplicating storage (#512). Otherwise, the latter will be preferred.

[cgwalters]

(converted to a discussion)

Is there a plan for transitioning into an overlayfs podman based backend?

At a technical level right now, the way we handle selinux at least is going to make it hard to get away from "flattening" the rootfs by default.

But there's really a spectrum here - the c/storage backend could in theory also do some "eager flattening" too.

Anyways, see #215 for an important preparatory step for this.

[antheas]

Hi Colin, thanks for replying

SELinux is a bit of a tangent but that is your main blocker, you're indeed planning to transition away from OSTree as a backend

As for the pull request, from what I understand is that you replace ostree-rs-ext's role with a fork to podman and committing the uncompressed layers.

Still, the replacement is an interface between Podman and OSTree. An interface that is fuzzier, since you fork instead of embedding.

So let's circle back to the main point of the discussion.

What is the future role of OSTree in this project and the future of OSTree in general? As you're one of the main maintainers in both

[cgwalters]

What is the future role of OSTree in this project and the future of OSTree in general? As you're one of the main maintainers in both

We shipped ostree in at least RHEL9 and need to support it for quite a long time. But definitely the goal of this project is to merge and unify with the container stacks. There's quite a lot of threads in this.

One big sub-thread is that it's really that composefs becomes a key technical heart of both. For that, see e.g. containers/composefs#125 and related threads.

[antheas]

Indeed I did review the composefs, ostree, and podman threads. It is good that at least over say the next 10 years ostree will not be abandoned. However, I will agree with you that it is a bit murky right now.

What I am looking for is:

• a multi-deployment backend (A, B, C, D, E... not A, B)
• deployments should share storage at least partially
  • both a container layer cache (e.g. /var/lib/containers) and a shared content store (e.g., OSTree) are fine
  • I am happy to sacrifice storage space for update speed
• supports streaming updates (ostree-rs-ext is fine)
• supports at least partially diffed updates
  • both on wire and on disk (e.g, applying images should be diffed too, not just downloading them)*
  • chunked layers like rpm-ostree are fine, zstd:chunked is a nice to have but I do not feel it contributes that much to a properly designed image if the user has a fast connection and the image is chunked correctly*.
• verity is great to have provided the following:
  • it is precomputed on the server (e.g., composefs image is part of the container image)
  • it is lazy implemented
• LiveISO would be great both as a rescue media and for trying out installs (and perhaps here overlayfs will be problematic).

*: my custom Bazzite build with tree-files updates in 30s-1m, whereas the original Bazzite image updates in 6m on a fast ssd/1gb connection. Around 2 minutes of those are due to the download, the rest are for committing the new layers, which i do not expect to be fixed with diffed layers

And indeed a combination of OSTree, composefs, and podman fits all these requirements. So it would be nice to know what the plans for this project are. As for bootc itself, it does not fit the above requirements just due to not having a fleshed out API, but I'm happy to prototype around it until and if it does (e.g., bootc rollback just got added).

Multi-deployment tangent

As for the reason for A, B, C, D: AB is fine for stable incremental updates, but for desktop use where the image is extendable and or the update path is bleeding edge (e.g., with a Containerfile) its very easy for update drift to make the image unbootable. Also, for AB there are other solutions such as Valve's and Collabora's one. Which do indeed fit all but this requirement.

Example: right now there is an intermittent black screen issue for certain devices on Bazzite:

• This black screen issue freezes both TTY access and access to the desktop
• User is on version 1 as A, which does not have the issue
• User updates to version 2 as B, which does have the issue but the system boots fine
• User updates to version 3 which overrides A, and after 3 boots his system gets a black screen
• It is not possible to rollback interactively anymore
  • If they reboot multiple times, it does boot and they can fix it

Some Aspects of the above are problematic:

1. GRUB does not support touchscreens (no keyboards on handheld) and on half of the boots even external keyboards do not work (what?).
   • this is the stock Silverblue GRUB shim in case you want to look into it
2. Even if GRUB works, the stable deployment or a rescue deployment that should be there are missing
3. Things get murkier as the users start to layer stuff, and perhaps we need to save them from themselves by always making sure the original deployment is bootable

Bonus points: some users with nvidia eGPUs would like to have a separate Nvidia image to boot when they are docked, but the rpm-ostree "User Interface" (GRUB Stanzas + rpm-ostree pin <idx>) does not make this easy