ostree on Beaglebone Black

[jallwine]

I'm looking into what it would take to use ostree to manage full system atomic updates on Beaglebone boards. The official Beaglebone images for their boards are generated using scripts from this repository: https://github.com/beagleboard/image-builder
They create a root filesystem using debootstrap and apt-get and create a /boot directory where the kernel, initrd, device trees and device tree overlays are placed. A /boot/uEnv.txt file is created with a number of environment variables that specify which device tree to load and which overlays to apply. I'm trying to figure out how to adapt those scripts to make the rootfs and boot process work with ostree. Does anyone have any insights into what needs to happen to make that work?

[wmanley]

Step one would be to commit your rootfs into an ostree. You will have to adapt it so ostree will deploy it correctly

• This covers much of it: https://ostree.readthedocs.io/en/stable/manual/adapting-existing/
• There's a bit here about where you put your kernel and initrd: https://ostree.readthedocs.io/en/stable/manual/deployment/#the-system-boot
• And a bit here about where to put device-trees: deploy: support devicetree directory #2001

[jallwine]

Thanks @wmanley! When it says:

This configuration file will include a special ostree= kernel argument that allows the initramfs to find (and chroot() into) the specified deployment.

Do I need to ensure my initrd does that?

[wmanley]

I don't know about initrd as I don't use it. See #1401.

[mike-sul]

This meta layer https://github.com/advancedtelematic/meta-updater is a good example of OSTree application for full system atomic updates, and this is an example of its utilization in the scope of an end-to-end product https://github.com/foundriesio/lmp-manifest.

[dbnicholson]

@jallwine for the uEnv.txt, I was sure we had managed the environment variables at Endless, but when I looked closer there wasn't any of that. Apparently we've always just embedded the correct details into u-boot or something like that since we've controlled the hardware.

However, since we initially started using ostree, what you can do is populate /usr/lib/ostree-boot/uEnv.txt in the rootfs and ostree will merge this together with the arguments it needs to specify the kernel_image, ramdisk_image and bootargs. See https://github.com/ostreedev/ostree/blob/master/tests/test-admin-deploy-uboot.sh#L42 for an example in the test suite.

[jallwine]

I've taken my first concrete steps towards trying to get an OSTree managed Beaglebone image going. I've posted on the Beaglebone forums about it here: https://forum.beagleboard.org/t/ostree-on-beaglebone-boards/30458

If anyone has any information that could help, I'd appreciate it!

[dbnicholson]

When you create the initramfs, you need to put the stuff from ostree in it. What the ostree initramfs integration does is run ostree-prepare-root just before the pivot_root. This is what resolves the ostree deployment and sets up the mounts correctly. From your script it looks like you're just using the stock initramfs, but you need to generate an initramfs with the appropriate contents.

[jallwine]

That makes sense! Do you know of documentation or an example of this somewhere? I've never had to mess with initramfs.

[dbnicholson]

It's pretty system dependent, but ostree's initramfs integration is provided by dracut. The way you'd probably do this is chroot into the unpacked image and then run dracut --add ostree /usr/lib/modules/$kernel_version/initramfs.img $kernel_version. But if beaglebone doesn't use dracut or doesn't provide the ostree dracut integration then you'll need to do more surgery.

[jallwine]

It sounds like surgery may be necessary. The ostree docs mention switchroot. So I could expand the initramfs and copy that over to it, but how do I ensure it runs?

Beaglebone does use systemd, is there some magic that could be done with systemd without dracut?

[jallwine]

Looks like sticking it in /scripts/init-bottom allows it to run. Now I need to dial in what it's supposed to do.

[jallwine]

It booted! I had to insert the switchroot.sh script in between the overlayroot and udev scripts in the ORDER file in /scripts/init-bottom. Making progress...

[jallwine]

Now I need to figure out how to properly mount all the necessary directories. /var seems particularly important to get right as it's one of the few writable locations (and seems to contain other important directories such as /home), so let's start there.

The switchroot.ch script I'm using (the original is here) performs a bind mount for /var. I switched out the $sysroot variable for $rootmnt, as that's what I seemed to have access to in the initramfs. Unfortunately, $rootmnt seems to be read only, so the mkdir -p roothome mnt opt home fails. Does this have to be done here? If so, it seems like I either need to use a different variable or change whatever mounts $rootmnt to mount it as writable. Also, it's unclear to me how /var should be treated in the build stage. Do I check in /var into ostree, delete it or copy its contents somewhere else? When I create the client image, do I do anything with /var? Any information anyone has is appreciated!

[jallwine]

Adding a mount -o remount,rw $rootmnt $rootmnt to switchroot.sh seemed to help.

[jallwine]

I have a symbolic link for /lib directly into the deployment as U-boot is looking for overlay files in /lib/firmware prior to the kernel loading or initramfs. I'd like to link to a location that won't change (or rather will automatically change) when a new deployment is made active. I'm looking for something similar to how the /boot/loader folder is automatically switched to point to /boot/loader.1 or /boot/loader.0. Is there something comparable for the root directory of the active deployment?

[jallwine]

When I boot into the Beaglebone with my makeshift ostree setup, I'm unable to run ostree admin config-diff. It errors saying error: Not currently booted into an OSTree system and no --os= argument given. Any suggestions?

[AdrianVovk]

Is there something comparable for the root directory of the active deployment?

I don't think there is

error: Not currently booted into an OSTree system and no --os= argument given

Sounds like your makeshift setup isn't creating /run/ostree-booted. Why don't you use ostree-prepare-root and ostree-system-generator? https://github.com/ostreedev/ostree/tree/main/src/switchroot

[AdrianVovk]

When I create the client image, do I do anything with /var?

I think the best practice is that, in the OS image, /var should be an empty mount point. You should have something to populate /var on first boot. For example, systemd-tmpfiles or an init script that copies a default "/var" layout from /usr

[AdrianVovk]

Beaglebone does use systemd, is there some magic that could be done with systemd without dracut?

What is your initramfs like? Does it have systemd embedded in it? Something else? What does the initramfs do for you outside of an OSTree system? Why not use dracut?

[jallwine]

Sounds like your makeshift setup isn't creating /run/ostree-booted. Why don't you use ostree-prepare-root and ostree-system-generator? https://github.com/ostreedev/ostree/tree/main/src/switchroot

I'd be happy to with some instruction. I spliced in the switchroot.sh script into my initramfs, which runs and sets up my root directory such that my system starts, but I don't know how to use the other parts.

What is your initramfs like? Does it have systemd embedded in it? Something else? What does the initramfs do for you outside of an OSTree system? Why not use dracut?

I'm not sure how to answer those. I'm not familiar with initramfs, so I'm largely wandering in the dark. I'm doing what I can to adapt existing Beaglebone images to work with an OSTree managed system. Scripts in this repository generate the Beaglebone images. I'm trying to take an existing Beaglebone image and convert it to an image suitable to be managed with OSTree. It doesn't seem like those scripts use dracut. This is the original initramfs and this one is largely the same, but with the switchroot.sh script spliced in (this one is a tar.gz file rather than a cpio archive if that's easier).

[AdrianVovk]

@jallwine If you use dracut, ostree support should come basically for free. Your initramfs looks super custom, and I have no idea how it works. However, no dracut and no systemd. However, you can probably add "touch /run/ostree-booted" somewhere during your boot process (maybe even in switchroot.sh) to fix the "not booted with ostree" error.

Or just port to dracut. It seems like you're trying to do just that based on your GitHub activity

[jallwine]

Thanks Adrian! Yeah, I’m giving dracut a shot. My initial test for a regular running system seemed to work fine when I installed dracut and generated the initramfs with it. The version of OSTree on my system, though, wasn’t built with dracut support so I’m going through the process of building it from source and then seeing how it works with the ostree-managed system. I’d like to use well supported tools where possible, so this seems like the better route if I get it working.

[jallwine]

That seems to work flawlessly! I have my system booting up like a regular image. All services seem to start up properly and I'm able to connect like normal.

I believe I'm to the point that I need to implement a custom bootloader configuration into ostree. @wmanley, in this thread, you offered to help guide the implementation of a Raspberry Pi bootloader. I believe I need to do the same thing for the Beaglebone. I have a running ostree system with a loader directory that I manually set up (see here). I think those steps need to be performed by ostree admin deploy in order for a client Beaglebone to be able to update. I haven't looked too hard at the bootloader implementations, yet. Do you have any thoughts before I go any deeper? I imagine it will be fairly straight forward, but likely a bit more verbose than those few lines of bash. Any help you can provide to get me started is appreciated!

[wmanley]

I could be wrong, but seeing as you have a uEnv.txt I'd guess you should be able to use ostree's existing u-boot support. It should even be automatically detected:

ostree/src/libostree/ostree-bootloader-uboot.c

Lines 56 to 57 in 87db562

	if (!glnx_fstatat_allow_noent (self->sysroot->sysroot_fd, uboot_config_path, &stbuf, AT_SYMLINK_NOFOLLOW, error))
	return FALSE;

[jallwine]

The official BeagleBone images use a custom version of U-Boot that look for specific variables defined in the uEnv.txt file and expect certain files to be in /boot so I need to make a few extra symlinks to other files and directories.

[jallwine]

It would be nice to be able to run a bash script to help prepare the loader directory rather than implementing everything in C. This would make it a lot more flexible as any language could be used and it would be a lot easier to implement simple things like generating symlinks or parsing a file, etc.

[AdrianVovk]

@jallwine Take a look at the zipl bootloader backend, then. zipl is just like normal BLS, but then you need to run a command to make the bootloader actually update some cache and accept the config. Sounds a lot like your situation

[jallwine]

Thanks @AdrianVovk, that sounds promising, I’ll take a look.

[jallwine]

It looks like I may be able to simply use that bootloader and provide an executable called zipl that is the bash script I'm talking about. Are there environment variables available that would know the deployment and /boot/loader.# paths or would I need to pass those in some how?

[dbnicholson]

I think you could probably use switchroot.sh like you were trying, but I think it's been unmaintained for a while. So it's likely missing some more recent changes like /run/ostree-booted. If you want to go that route, I'd browse the git history in src/switchroot and see if there are other changes particularly to ostree-prepare-root.c and ostree-remount.c that would need to be adapted. That said, if you can get going with dracut you'll be in a better place since that's how most people handle this.

I also agree with using the u-boot loader provided with ostree. You can optionally provide /usr/lib/ostree-boot/uEnv.txt and it will be appended to the ostree managed one in /boot/loader/uEnv.txt. See https://github.com/ostreedev/ostree/blob/main/src/libostree/ostree-bootloader-uboot.c for details. In our Endless image builder we just make a symlink from /boot/uEnv.txt to /boot/loader/uEnv.txt. The ostree test suite appears to do the same - https://github.com/ostreedev/ostree/blob/main/tests/libtest.sh#L362. The whole bootloader area is sadly underdocumented.

[jallwine]

Thanks @dbnicholson! I had success using dracut, so I think I’m going to stick with that.

As for the boot loader, I’ll give /usr/lib/ostree-boot/uEnv.txt a shot, but I also need to create a number of symlinks in order for the BeagleBone U-Boot to boot the way a normal image does. I could manually create them after an ostree admin deploy, but ideally, the creation of these links would happen prior to the /boot/loader symlink changing to be as atomic as possible. It sounds like I’ll need a new boot loader configuration that is a combination of the current U-Boot and zipl ones.

[jallwine]

Alternatively, if the loader directory automatically symlinked to the deployment directory (i.e. /boot/loader/deploy symlinked to /ostree/boot.#/STATE_ROOT/CHECKSUM/0) then I could create permanent symlinks in the real /boot and I wouldn’t need a special boot loader configuration.

[jallwine]

I modified the OSTree U-Boot bootloader configuration to check for the existence of a /usr/lib/ostree-boot/setup.sh script and run it if it exists, after writing the uEnv.txt file to the loader directory. This allows the setup script to completely replace the uEnv.txt file if it requires (like in my case). The script is passed the sysroot path, the deployment path and the /boot/loader.# path so any symlinks to various files/directories can be created in the correct locations.

https://github.com/PocketNC/ostree/blob/main/src/libostree/ostree-bootloader-uboot.c