Hook up blueprint filesystem customizations #124

bcl · 2024-01-16T01:54:06Z

Really simple change :) The testing feels a bit klunky, suggestions welcome. I thought about checking the manifest, but it boots the image anyway so checking the output of df seemed to be the most straight forward way.

test/test_smoke.py

achilleas-k · 2024-01-16T13:53:11Z

One tricky part here is that we have rules in osbuild/images about what partitions a user can (not) add to avoid making unbootable images. Unfortunately those aren't exposed (yet) so we can't apply them here.

Maybe we can only allow cutomizing the size of / for now to unblock issues with images failing to build when the base container is too big and then work on bringing in the rules from images.

bcl · 2024-01-16T16:47:22Z

One tricky part here is that we have rules in osbuild/images about what partitions a user can (not) add to avoid making unbootable images. Unfortunately those aren't exposed (yet) so we can't apply them here.

It feels weird to essentially be partially re-implenting things here (and will be totally confusing to users) so I think exposing more of osbuild/images is the right way to go.

bcl · 2024-01-23T21:58:02Z

PR to move the policies into the public pkg - osbuild/images#395

bcl · 2024-01-24T00:47:25Z

NOTE: CI tests will fail until images is updated

github-actions · 2024-02-23T04:07:03Z

This PR is stale because it has been open 30 days with no activity. Remove "Stale" label or comment or this will be closed in 7 days.

bcl · 2024-03-21T00:13:26Z

I've lost track of the status of this, so if you want it don't wait on me :)

cgwalters

This looks pretty small and sane to me, I think we also just need to update the docs, right? (Can be done post-merge)

test/test_manifest.py

cgwalters · 2024-03-25T17:50:06Z

Hm, testing farm was failing with

OSError: [Errno 30] Read-only file system: 'boot.1'
But I'm not totally sure that was the root cause?

cgwalters · 2024-03-25T18:34:06Z

I've rebased this on git main, just fixing up the (textual, but not logical) conflict with the new "double rootfs size default") code. Not tested locally beyond building, but let's see what CI says!

mvo5 · 2024-03-26T11:14:45Z

I've rebased this on git main, just fixing up the (textual, but not logical) conflict with the new "double rootfs size default") code. Not tested locally beyond building, but let's see what CI says!

Once we move to the bootc install to-filesystem flow (which is hopefully soon) we will no longer write a custom /etc/fstab which is crucial for this feature to work in the general case. I think we have some options:

add policy so that we only allow to customize the size of "/", "/boot" and disallow adding extra partitions (probably a good starting point)
support writing a custom /etc/fstab from images, I managed to find a bit of time today and looked over the ostree docs and played with guestfish with a fresh image that never booted. The only way to customize /etc/fstab I found was to write into the deployment root /ostree/deploy/default/deploy/d3d806...0/etc/fstab of the unbooted disk. Any other path ("/", "/sysroot", "/ostree/deploy/default/etc/fstab") I experiented with got "shadowed" or was wrong on the first boot. We do have support for this in osbuild, so adding this way of customizing is straightforward to add but I feel uneasy about it because it seems to me that bootc should be in control of it's content and just "side-loading" like this does break any integrity checking.
with (2) solved, it seems we also need some code to deal with custom mountpoints(?). I.e. it seems to support e.g. /data we also need to create the mountpoint in the deploy root?

Please let me know if I'm missing something here or thinking too complicated.

cgwalters · 2024-03-26T12:06:37Z

but I feel uneasy about it because it seems to me that bootc should be in control of it's content and just "side-loading" like this does break any integrity checking.

I feel like we keep re-treading this, not sure how to solve the logical disconnect. I'll try to write some more about this in the bootc docs. Now for sure I would like to support better things that just "unmanaged regular files" in /etc (configmaps perhaps or other things).

But for broad/generic use cases, we simply have to support machine-local state - and for maximum compatibility we can't just require everything to learn to do configmaps or sysexts or something that doesn't boil down to supporting vi /etc/somefile.conf.

For example Anaconda today is writing an /etc/fstab always...which tangentially I needed to do an elaborate workaround for a corner case and hopefully as we align things here when Anaconda learns to use bootc install to-filesystem too we can centralize this logic more.

The only way to customize /etc/fstab I found was to write into the deployment root /ostree/deploy/default/deploy/d3d806...0/etc/fstab of the unbooted disk.

The configuration definitely needs to be in the deployment root, yes.

Also BTW let's not get too hung up on /etc/fstab specifically as e.g. .mount units also make sense - but all of this is really just a special case of "config files in /etc/ and kernel arguments" which both can form machine-local state necessary for storage, networking etc.

mvo5 · 2024-03-26T13:59:25Z

[..]

The configuration definitely needs to be in the deployment root, yes.

Also BTW let's not get too hung up on /etc/fstab specifically as e.g. .mount units also make sense - but all of this is really just a special case of "config files in /etc/ and kernel arguments" which both can form machine-local state necessary for storage, networking etc.

Thank you! Sorry that we keep coming back to this, but hopefully this was the disconnect I had. In my (wrong) mental model I assumed that there is

/ostree/deploy/default/deploy/ which shouldn't be touched
extra directory with overrides to the above dir to add local state like /etc/fstab (this is why I messed around in /etc on the not-yet-booted-disk)

It seems that ostree is smart enough to know if files in (1) get changed and tracks them as local state. I hope I got this right now and sorry that it took me a while.

cgwalters · 2024-03-26T14:05:36Z

It seems that ostree is smart enough to know if files in (1) get changed and tracks them as local state.

Yes, https://containers.github.io/bootc/filesystem.html#etc ➡️ https://ostreedev.github.io/ostree/atomic-upgrades/#assembling-a-new-deployment-directory (I'll expand this more over there)

ostree's semantics here are more arguably more "package system" like (vs "simple image system" as is emphasized by e.g. systemd's push for empty /etc) - and this definitely has tradeoffs, but again in a nutshell we support dropping arbitrary files there and they persist.

cgwalters · 2024-04-12T21:22:38Z

I rebased this again...I think it still makes sense. (Sorry about being initially semi-against...I had just really hoped we also started with the container-based configuration but that is just messier than I thought)

mvo5 · 2024-04-15T16:00:40Z

I rebased this again...I think it still makes sense. (Sorry about being initially semi-against...I had just really hoped we also started with the container-based configuration but that is just messier than I thought)

Once we move to bootc install to-filesystem (hopefully really soon) this will need something like osbuild/images#597 first.

github-actions · 2024-05-16T04:07:24Z

This PR is stale because it has been open 30 days with no activity. Remove "Stale" label or comment or this will be closed in 7 days.

This also allows the root filesystem size to be increased if the default of 10GiB isn't enough.

Make sure / is larger than the default of 10GiB and make sure there is a new /var/log mountpoint.

This requires a new release of images that exposes the policies package.

Users cannot create a mountpoint on /ostree, make sure that an error is returned when this happens.

mvo5 · 2024-05-28T07:32:25Z

I rebased this again (and slightly tweaked tiny bits). Unfortunately it needs more work because:

...
Installing image: docker://quay.io/centos-bootc/centos-bootc:stream9
ERROR Installing to filesystem: Verifying empty rootfs: Non-empty root filesystem; found "var"
Traceback (most recent call last):
  File "/run/osbuild/bin/org.osbuild.bootc.install-to-filesystem", line 53, in <module>
    r = main(args["options"], args["inputs"], args["paths"])
  File "/run/osbuild/bin/org.osbuild.bootc.install-to-filesystem", line 48, in main
    subprocess.run(pargs, env=env, check=True)
  File "/usr/lib64/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['bootc', 'install', 'to-filesystem', '--source-imgref', 'containers-storage:[overlay@/run/osbuild/containers/storage+/run/containers/storage]3b612dd1fae2437c00ae3187d0e63daa7a94711560fb1712389edd4121668c96', '--skip-fetch-check', '--generic-image', '--karg', 'rw', '--karg', 'console=tty0', '--karg', 'console=ttyS0', '--karg', 'systemd.journald.forward_to_console=1', '--target-imgref', 'quay.io/centos-bootc/centos-bootc:stream9', '/run/osbuild/mounts']' returned non-zero exit status 1.

I attach the generated osbuild manifest with the exiting code:
fs-customizations.json

The issue is that we run "org.osbuild.bootc.install-to-filesystem" with all the mounts, i.e. "/", "/boot" but also "/var/log" now.
So AFAICT "images" need sto be changed so that:

the "install-to-filesystem" stage only takes the "essential" mounts (/, /boot/, /boot/efi)
after "install-to-filesystem" ran we need a "org.osbuild.mkdir" stage for the extra mount points that also only mounts the "essential" mounts

That should be enough (but is a bit of extra work).

This commit tweaks the test setup slightly to use pytest.raises for exception checking and also run test_mount_ostree_error() only for the centos image (as the error checking/policies are exactly the same for both images).

The "images" library does not support custom mount points for bootc based images just yet. The reason is that images will generate an osbuild manifest that contains all the "mounts" for the generated disk. This means that with an extra partition like `/var/log` this is visible for the "bootc install-to-filesystem" stage. And that will trip up bootc because it validates the content of the target directory. Example error with `/var/log` as a custom mount point: ``` ... Installing image: docker://quay.io/centos-bootc/centos-bootc:stream9 ERROR Installing to filesystem: Verifying empty rootfs: Non-empty root filesystem; found "var" Traceback (most recent call last): File "/run/osbuild/bin/org.osbuild.bootc.install-to-filesystem", line 53, in <module> r = main(args["options"], args["inputs"], args["paths"]) File "/run/osbuild/bin/org.osbuild.bootc.install-to-filesystem", line 48, in main subprocess.run(pargs, env=env, check=True) File "/usr/lib64/python3.9/subprocess.py", line 528, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['bootc', 'install', 'to-filesystem', '--source-imgref', 'containers-storage:[overlay@/run/osbuild/containers/storage+/run/containers/storage]3b612dd1fae2437c00ae3187d0e63daa7a94711560fb1712389edd4121668c96', '--skip-fetch-check', '--generic-image', '--karg', 'rw', '--karg', 'console=tty0', '--karg', 'console=ttyS0', '--karg', 'systemd.journald.forward_to_console=1', '--target-imgref', 'quay.io/centos-bootc/centos-bootc:stream9', '/run/osbuild/mounts']' returned non-zero exit status 1. ``` So AFAICT "images" need sto be changed so that: 1. The "install-to-filesystem" stage only takes the "essential" mounts (/, /boot/, /boot/efi) 2. After "install-to-filesystem" ran we need a "org.osbuild.mkdir" stage for the extra mount points that also only mounts the "essential" mounts As a first step on the journy this commit limits customizations to "/" and "/boot" which is already very useful as many people have asked for precisely those.

mvo5 · 2024-05-28T10:41:23Z

As suggested by @ochosi I tweaked the PR now so that we initially only allow to customize "/" and "/boot" which will already cover many use-cases. Then we can continue on this once we have updated images to support what bib needs.

mvo5

Fwiw, this looks good to me now but please count me as only a a +0.5 because I tweaked/rebased this code a bit.

achilleas-k

I agree, let's enable / and /boot and enable more later.

bcl requested review from achilleas-k and mvo5 January 16, 2024 01:54

achilleas-k reviewed Jan 16, 2024

View reviewed changes

test/test_smoke.py Outdated Show resolved Hide resolved

bcl force-pushed the main-customfs branch from 4222c65 to bf16d2d Compare January 24, 2024 00:46

bcl requested a review from achilleas-k January 24, 2024 00:46

github-actions bot added the Stale Issue or PR with no activity for extended period of time label Feb 23, 2024

bcl force-pushed the main-customfs branch from bf16d2d to 740ce0e Compare February 27, 2024 15:44

bcl removed the Stale Issue or PR with no activity for extended period of time label Feb 27, 2024

bcl force-pushed the main-customfs branch from 740ce0e to 4b0adc8 Compare February 27, 2024 15:51

bcl marked this pull request as ready for review February 27, 2024 15:52

bcl force-pushed the main-customfs branch 2 times, most recently from b8bd52f to 2e5dc6e Compare February 28, 2024 16:50

achilleas-k force-pushed the main-customfs branch from 2e5dc6e to 06214f5 Compare March 19, 2024 16:57

achilleas-k previously approved these changes Mar 19, 2024

View reviewed changes

achilleas-k enabled auto-merge March 19, 2024 16:57

ondrejbudai disabled auto-merge March 19, 2024 16:59

ondrejbudai mentioned this pull request Mar 19, 2024

bib: add --filesystems commandline option for basic customizations #283

Closed

bcl force-pushed the main-customfs branch from 06214f5 to 2d65062 Compare March 21, 2024 00:12

cgwalters mentioned this pull request Mar 25, 2024

Dynamic root partition size #52

Open

cgwalters previously approved these changes Mar 25, 2024

View reviewed changes

test/test_manifest.py Show resolved Hide resolved

cgwalters dismissed stale reviews from achilleas-k and themself via 82547de March 25, 2024 17:54

cgwalters force-pushed the main-customfs branch from 2d65062 to 82547de Compare March 25, 2024 17:54

cgwalters force-pushed the main-customfs branch from 82547de to 07d3375 Compare March 25, 2024 18:11

cgwalters force-pushed the main-customfs branch from 07d3375 to e26533f Compare April 12, 2024 21:20

mvo5 mentioned this pull request Apr 29, 2024

Provide better error for containers without a rootfs default #400

Merged

github-actions bot added the Stale Issue or PR with no activity for extended period of time label May 16, 2024

achilleas-k removed the Stale Issue or PR with no activity for extended period of time label May 16, 2024

bcl added 4 commits May 28, 2024 08:40

Use blueprint filesystem customizations to create partitions

b5bbe60

This also allows the root filesystem size to be increased if the default of 10GiB isn't enough.

test: Check the qcow2 image filesystem changes

452ac25

Make sure / is larger than the default of 10GiB and make sure there is a new /var/log mountpoint.

Check blueprint mountpoints against OSTree mountpoint policies

419111c

This requires a new release of images that exposes the policies package.

test: Add a test for mountpoint policy violations

95e8e12

Users cannot create a mountpoint on /ostree, make sure that an error is returned when this happens.

mvo5 force-pushed the main-customfs branch from e26533f to 5fc5e6b Compare May 28, 2024 07:18

mvo5 added 3 commits May 28, 2024 12:31

bib: make image_test.go follow the $pkg_test go pattern

017c424

test: small pytest related tweaks

191a221

This commit tweaks the test setup slightly to use pytest.raises for exception checking and also run test_mount_ostree_error() only for the centos image (as the error checking/policies are exactly the same for both images).

mvo5 force-pushed the main-customfs branch from 5fc5e6b to 75fb7cd Compare May 28, 2024 10:39

mvo5 approved these changes May 28, 2024

View reviewed changes

mvo5 requested a review from achilleas-k May 28, 2024 10:45

achilleas-k approved these changes May 30, 2024

View reviewed changes

mvo5 added this pull request to the merge queue Jun 3, 2024

Merged via the queue into osbuild:main with commit 06e1b2a Jun 3, 2024
6 of 7 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Hook up blueprint filesystem customizations #124

Hook up blueprint filesystem customizations #124

bcl commented Jan 16, 2024

achilleas-k commented Jan 16, 2024

bcl commented Jan 16, 2024

bcl commented Jan 23, 2024

bcl commented Jan 24, 2024

github-actions bot commented Feb 23, 2024

bcl commented Mar 21, 2024

cgwalters left a comment

cgwalters commented Mar 25, 2024

cgwalters commented Mar 25, 2024

mvo5 commented Mar 26, 2024

cgwalters commented Mar 26, 2024

mvo5 commented Mar 26, 2024

cgwalters commented Mar 26, 2024

cgwalters commented Apr 12, 2024

mvo5 commented Apr 15, 2024

github-actions bot commented May 16, 2024

mvo5 commented May 28, 2024

mvo5 commented May 28, 2024

mvo5 left a comment

achilleas-k left a comment

Hook up blueprint filesystem customizations #124

Hook up blueprint filesystem customizations #124

Conversation

bcl commented Jan 16, 2024

achilleas-k commented Jan 16, 2024

bcl commented Jan 16, 2024

bcl commented Jan 23, 2024

bcl commented Jan 24, 2024

github-actions bot commented Feb 23, 2024

bcl commented Mar 21, 2024

cgwalters left a comment

Choose a reason for hiding this comment

cgwalters commented Mar 25, 2024

cgwalters commented Mar 25, 2024

mvo5 commented Mar 26, 2024

cgwalters commented Mar 26, 2024

mvo5 commented Mar 26, 2024

cgwalters commented Mar 26, 2024

cgwalters commented Apr 12, 2024

mvo5 commented Apr 15, 2024

github-actions bot commented May 16, 2024

mvo5 commented May 28, 2024

mvo5 commented May 28, 2024

mvo5 left a comment

Choose a reason for hiding this comment

achilleas-k left a comment

Choose a reason for hiding this comment