-
Notifications
You must be signed in to change notification settings - Fork 48
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
edge-raw-image with custom partition mount points does not boot after upgrade #352
Comments
@mcattamoredhat what upgrade was done on the new deployment(ostree: 0)? |
seems the custom lvm :
|
The workaround mentioned in 337 works ,
It needs to be embedded inside osbuild-composer to ensure lvs are mounted correctly |
I wonder if we should add it transparently ourselves when filesystem customizations are needed so that the user doesn't need to remember that the service is needed. |
I bet what's going on here is the osbuild pipeline is only making these directories on top of the deployed disk image (i.e. equivalent of anaconda The osbuild pipelines need to change to create this directory as part of the ostree commit instead. |
okay, We have enabled filesystem customization for deployments(raw,image, iso) only. Image builder actually consumes the base-commit(which does not have any data about lvm ) to build the deployments with the lvm data. |
If I understand this problem correctly this is about creating the mount points in the filesystem for custom mounts (eg /data)? |
@nullr0ute it fails on upgrade, as dirs gets deleted, needs to be created explicitly to mount the volume, else system does not boot and plunges into emergency shell. |
So based on the 👍 whether it's a LVM partition (or NFS or some other storage) is irrelevant here. The user has to know where they want to mount things and the mount point should be created as part of the ostree stage as not as part of raw/iso etc. otherwise as @cgwalters says it's going to be a problem. How we fix that I don't know, maybe we need to interpret the blueprint for both stages, maybe we need a "custom mount points" blueprint to ensure the directories are created (including the correct permissions). Any other solution, such as custom systemd services are just working around the problem and will no doubt cause other issues. |
So there are two things that's biting me:
I guess it can be tested out but if @achilleas-k or @cgwalters you are already aware of the process, please share your thoughts. |
I wouldn't think so in this case.
That is a good question
Either way, we need to test this to decide how we are going to tackle the rest. |
So I tested it by bypassing the |
Even we apply them to a commit pipelines, I am bit unsure if it would be of much use and user may want different mount-points from same base commit.
Its wont be possible to sync both the stages which will render system unbootable, Possible Solution 🤔 |
Overriding the customization checker to pass these through doesn't help because the customizations have no effect on ostree commits. IB doesn't do anything with that customization when it's building a commit. Which brings me to
What would that do? Is it just a matter of creating the directories? The user would have to know that they will have to add the partitions/mountpoints to the image blueprint as well, so we'd have to document this at least. Also the same fs customizations would have to be added to any upgrade blueprints for the same reason. Not an issue, just making sure I understand all the implications. |
Does running |
I haven't dug into this deeply but AIUI (from previous conversations) osbuild has an architecture where it wants to create a full final filesystem tree, then copy that to disk, and then it's at that point we'd run post-copy. This means that we'd still have the problem that we allow uncontrolled mutation of the toplevel filesystem root for disk images. The most robust way to fix that would be to try to better honor ostree's rules in pipeline builds. Specifically when generating disk images, in the default configuration we want to ensure that only |
Do we want to disallow this? Adding partitions and mountpoints to the root of ostree-based disk images was enabled somewhat recently as a feature requirement for edge. If this is in conflict with ostree's rules, or if it's "more correct" to move that configuration to the commit/container and block filesystem customizations on disk builds, then we should do that. My main question is: What does a filesystem customization look like when applied to a commit? Is it just about creating directories? We've talked about some form of metadata describing a partition table for containers, but afaik there's nothing like that for the base ostree case that we're using now for R4E and Fedora IoT. |
The rule basically is "the commit should be source of truth", with local state in Now except, we just added a giant ability to relax this in ostreedev/ostree#3114 ...but that still has the semantic that content placed there is dropped on upgrades.
I wouldn't say block all filesystem customizations on disk builds. Today for example, we (should) support configuring e.g. subdirectories of A simple way to look at this is for cases like Fedora CoreOS where the commit/image is shipped from upstream, we still allow choose their backing filesystem type and create sub-mounts of The problem case comes more about toplevel mounts. I should also clarify that we enabled Since this is about toplevel mounts, I'd today disallow them in disk image builds, unless |
No, |
So even we add a directory in commit it will be lost in the upgrade if the upgrade blueprint doesn't have the same fs/directory customization. So unless we have a fix for 337 I think we can: |
I think that we have already internally discussed with the team that greenboot has a defined use case and that it mustn't deviate from that role. |
agreed, greenboot's scope doesn't include things like this and shouldn't be leveraged |
Thoughts regarding option 1? File and Service customizations are not allowed in deployment stages, so the effective solution will be to embed a generic service(template unit file to create directories of the filesystem mountpoints) in commit. |
I don't consider adding a service file fragile at all, so I don't mind taking that path, but I would add it in the the deployment. Also, as I said above, we wouldn't be adding it with the blueprint (the user shouldn't be required to know that the service file is needed), but rather detect that the user is requesting filesystem customizations in the deployment blueprint and add the required service file for those to work internally in a transparent way. |
Yes. Except...there's also the approach of moving the partition creation to firstboot, and injecting logic to do that into the commit. This is the approach taken by Ignition and systemd-repart. In general I prefer that approach because it keeps the disk image simpler - or stated another way, it more strongly enforces the decoupling of the commit (container image) and disk image. If there's a way to enforce that the corresponding toplevel directories are exist in the commit when specifying disk customizations, that seems like a decent fix. In a container flow that's basically just |
The transparent way - I imagine is to magically make it appear in the blueprint
That's the idea, to be implement by a unit file that creates the mountpoint. |
No, just adding it internally programatically, if we show a blueprint that is different from what the user has requested that is going to cause problems too. When we see in the blueprint that file customizations are added, we also include the service file internally. |
@achilleas-k @ondrejbudai just FYI if you have any inputs/recommendations given the way we build commit/disk in osbuild and given the above suggestions? |
So the approach we decided is, When user adds fs-customization in deployment blueprint, osbuild will add and enable unit file , something like this in background
Question is how we enable it,
|
I would say doing it "properly" (with an install section) is preferable. I agree, it means it can be disabled and break the system, but it also means it can be disabled the right way if necessary (like if the user decides they don't want/need the partition anymore). We would have to document this of course but I'd rather it behaved like any other unit. On the topic of creating the unit itself: I would be in favour of making an osbuild stage for this. A very specific stage that creates this service file for a set of mountpoints. But we can do it with file customizations in this repository first, for development, testing, and to get it out quickly, and move it down to osbuild later. |
I think we should do this. On the composer level it would mean enabling filesystem customizations for ostree commit types. But we would have to consider what to do with the same customizations for deployment types (disks). Do we keep them? Do we handle conflicts? Or should we deprecate them and inform users that they should be putting their partitioning info in the commit? |
this, to me, is really counter intuitive :/ I understand it could be the way to go, but I kind of expect partitioning decisions to be made in deployment right? |
Don't get me wrong, I'm more on this side too. It makes more sense to me as well that given a base ostree commit or container, you can deploy it in any number of scenarios with different partitioning layouts. I'm trying to reconcile the user experience with the technology decisions we're making here (so I too would love to hear what Ben has to say). I admit I'm not completely aware of what expectations we've created for users and how we can change that (or even if we should). But I do think we should be thinking ahead about what it means when we make these changes. So, more concretely: If the decision here is "partitioning decisions should be made at ostree base image creation time", what does that mean for the existing image building process of:
Questions:
I could probably come up with more qs but that's a good starter for now I think 😆 |
The thing is: We can't change user partitions across upgrades by default no matter what. So remember, even if partitioning is specified in a disk image today, in-place upgrades won't get any changes made there. This architecture is pretty clear with e.g. Anaconda and kickstart - Anaconda is only used once, and not thereafter. Ignition is also designed to run just once partly for this reason, but Ignition also does support "initialize idempotently" - i.e. if the partition already exists in the expected format, it is reused (and hence data is preserved). The I don't have a strong opinion about this; if we were to take a dependency on repart it'd need some analysis. It could also be done by injecting custom systemd units to create the partitions, and mount units using
It will probably have to continue to be if it was before, right?
No, all that state would best be computed on firstboot (as is done with ignition and repart).
In the end, moving the logic to firstboot necessarily implies some difficulty in trying to make that a "build" time check. Honestly I would focus more on making it convenient for people to have an edit-compile-debug cycle (where "debug" means booting for real).
You're asking what happens if a user specifies a mountpoint a blueprint for creating a commit at one point, and then removes it later? I'd expect the mountpoint to drop out, yes. |
Having this the filesystem customization in commit comes with difficulties:
wont be poosible
I would prefer repart or x-systemd.makefs over custom systemd unit, probably they are more tried and tested , repart's implementation seems to be straightforward but analysis will be to see if the other mountpoints remains intact, modifying |
#399 should fix the issue. |
Filesystem customization mountpoints does not persists any upgrade due to the property of ostree which only persists data on /var and /etc. This feature will ingest the filesystem customization and create a unit file which can create the required mountpoints if it doesnot exist. fixes: osbuild#352 Signed-off-by: Sayan Paul <paul.sayan@gmail.com>
Filesystem customization mountpoints does not persists any upgrade due to the property of ostree which only persists data on /var and /etc. This feature will ingest the filesystem customization and create a unit file which can create the required mountpoints if it doesnot exist. fixes: osbuild#352 Signed-off-by: Sayan Paul <paul.sayan@gmail.com>
Filesystem customization mountpoints does not persists any upgrade due to the property of ostree which only persists data on /var and /etc. This feature will ingest the filesystem customization and create a unit file which can create the required mountpoints if it doesnot exist. fixes: osbuild#352 Signed-off-by: Sayan Paul <paul.sayan@gmail.com>
Filesystem customization mountpoints does not persists any upgrade due to the property of ostree which only persists data on /var and /etc. This feature will ingest the filesystem customization and create a unit file which can create the required mountpoints if it doesnot exist. fixes: osbuild#352 Signed-off-by: Sayan Paul <paul.sayan@gmail.com>
Filesystem customization mountpoints does not persists any upgrade due to the property of ostree which only persists data on /var and /etc. This feature will ingest the filesystem customization and create a unit file which can create the required mountpoints if it doesnot exist. fixes: osbuild#352 Signed-off-by: Sayan Paul <paul.sayan@gmail.com>
Filesystem customization mountpoints does not persists any upgrade due to the property of ostree which only persists data on /var and /etc. This feature will ingest the filesystem customization and create a unit file which can create the required mountpoints if it doesnot exist. fixes: osbuild#352 Signed-off-by: Sayan Paul <paul.sayan@gmail.com>
Filesystem customization mountpoints does not persists any upgrade due to the property of ostree which only persists data on /var and /etc. This feature will ingest the filesystem customization and create a unit file which is stored in /etc/systemd/system and can create the required mountpoints if it does not exist. fixes: osbuild#352 Signed-off-by: Sayan Paul <paul.sayan@gmail.com>
We have been testing locally RHEL 9: Filesystem customizations for edge-raw-image #255 RHEL-9.4
osbuild-104-1.20240108gitc62e555.el9.noarch
osbuild-composer-98-1.20240108git0169b7b.el9.x86_64
weldr-client-35.9-1.el9.x86_64
With blueprint:
After booting the image both in BIOS and UEFI mode, we see the corresponding logical volumes created.
Nevertheless, the system is not able to boot the deployment ostree:0 after upgrade.
On the other hand, system deployment ostree:1 is able to boot with no problem, as a matter of fact is possible to see the custom lvs.
Due to virsh console not showing any useful output, it has been difficult to track the root cause of the issue.
May you please help to check this issue when you have time?
The text was updated successfully, but these errors were encountered: