MCO-1026: Enhancement for bootc integration into MCO #1631
Conversation
|
@inesqyx: This pull request references MCO-1026 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.17.0" version, but no target version was set. In response to this: Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
openshift-ci-robot
added
the
jira/valid-reference
openshift-ci
bot
added
the
do-not-merge/work-in-progress
|
Skipping CI for Draft Pull Request. |
inesqyx
force-pushed
the
MCO-1026-bootc-enhancement
branch
from
6260522
to
ad46715
Compare
|
Tagging manually for review, apology for the spam: @cheesesashimi @yuqi-zhang @sinnykumari @mrunalp @jlebon |
|
Update on Kernel Argument: |
There was a problem hiding this comment.
Some suggestions inline, thanks for starting this enhancement!
My main question after reading is I'm not really sure if we're trying to describe the tech preview (and just enabling bootc in the MCO) or the overall plan for bootc via this enhancement. I think it would help me understand better if we directly state that, and have clearly defined goals around it
|
|
||
| ### Topology Considerations | ||
|
|
||
| #### Hypershift / Hosted Control Planes |
There was a problem hiding this comment.
I think we will have to align with Hypershift in the future. As it stands Hypershift wouldn't utilize this codepath
There was a problem hiding this comment.
Hey, Jerry! Do you by any chance know if anyone from the Hypershift team would be interested in this as well? We could include him/her/them into this conversation and review process to get better alignment in early stages.
Agreed, yeah. I think for this enhancement, we should take a broader approach and talk more about the overall story and phases than focus on the bootc part. E.g. we could retitle it "Image-mode integration into MCO" and have phases like (this is a rough list, we'd need to agree on importance and ordering):
Each one of those phases has a lot of details to work through, and ironically the last phase to move from calling |
inesqyx
force-pushed
the
MCO-1026-bootc-enhancement
branch
from
ad46715
to
57dc5f6
Compare
|
@inesqyx: This pull request references MCO-1026 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.17.0" version, but no target version was set. In response to this: Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
inesqyx
force-pushed
the
MCO-1026-bootc-enhancement
branch
from
78cc643
to
80442b2
Compare
enhancements/machine-config/alignment-with-image-mode-rhel-for-the-mco.md
Show resolved
Hide resolved
enhancements/machine-config/alignment-with-image-mode-rhel-for-the-mco.md
Outdated
Show resolved
Hide resolved
enhancements/machine-config/alignment-with-image-mode-rhel-for-the-mco.md
Outdated
Show resolved
Hide resolved
enhancements/machine-config/alignment-with-image-mode-rhel-for-the-mco.md
Outdated
Show resolved
Hide resolved
enhancements/machine-config/alignment-with-image-mode-rhel-for-the-mco.md
Outdated
Show resolved
Hide resolved
enhancements/machine-config/alignment-with-image-mode-rhel-for-the-mco.md
Outdated
Show resolved
Hide resolved
enhancements/machine-config/alignment-with-image-mode-rhel-for-the-mco.md
Outdated
Show resolved
Hide resolved
enhancements/machine-config/alignment-with-image-mode-rhel-for-the-mco.md
Outdated
Show resolved
Hide resolved
enhancements/machine-config/alignment-with-image-mode-rhel-for-the-mco.md
Outdated
Show resolved
Hide resolved
enhancements/machine-config/alignment-with-image-mode-rhel-for-the-mco.md
Outdated
Show resolved
Hide resolved
enhancements/machine-config/alignment-with-image-mode-rhel-for-the-mco.md
Outdated
Show resolved
Hide resolved
enhancements/machine-config/alignment-with-image-mode-rhel-for-the-mco.md
Outdated
Show resolved
Hide resolved
enhancements/machine-config/alignment-with-image-mode-rhel-for-the-mco.md
Outdated
Show resolved
Hide resolved
enhancements/machine-config/alignment-with-image-mode-rhel-for-the-mco.md
Outdated
Show resolved
Hide resolved
enhancements/machine-config/alignment-with-image-mode-rhel-for-the-mco.md
Outdated
Show resolved
Hide resolved
enhancements/machine-config/alignment-with-image-mode-rhel-for-the-mco.md
Outdated
Show resolved
Hide resolved
enhancements/machine-config/alignment-with-image-mode-rhel-for-the-mco.md
Outdated
Show resolved
Hide resolved
enhancements/machine-config/alignment-with-image-mode-rhel-for-the-mco.md
Show resolved
Hide resolved
enhancements/machine-config/alignment-with-image-mode-rhel-for-the-mco.md
Show resolved
Hide resolved
inesqyx
force-pushed
the
MCO-1026-bootc-enhancement
branch
from
80442b2
to
4f18add
Compare
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
|
||
| ### User Stories | ||
|
|
||
| * As an OpenShift cluster administrator I would like MCO to leverage RHEL disk images to boot/update my nodes |
There was a problem hiding this comment.
| * As an OpenShift cluster administrator I would like MCO to leverage RHEL disk images to boot/update my nodes | |
| * As an OpenShift cluster administrator I would like MCO to leverage image-mode RHEL to boot/update my nodes |
Disk images are only involved in the first boot of the node. After that it's container images.
|
|
||
| After On Clustering GA, we will gradually deprecate the current configuration method of direct file writing to disk and move to layered update path (Phase 1) by default. At the end of this phase, OCP nodes will be configured in a full image-based way and will have a RHEL-bootc layer, a CoreOS layer, an OCP node layer, and day-2 configuration layers added on top built and rolled out by On Cluster Layering discussed in Phase 1. Please note here that the proposal and implementation details of the first three layers (the RHEL-bootc layer, the CoreOS layer and the OCP node layer) are explained in the Splitting the RHCOS Image into layers enhancement, thus a non-goal for this enhancement. Some of the gaps we observed here are: | ||
| * "Default" MCO managed configuration moved to /usr and shipped as a container image layer: Includes all MCO specific config rendered to /usr + MachineConfigs which are rendered to /etc -> /usr/etc (to be confirmed) | ||
| * Configs that involve file writing to immutable local file system directories need to be handled: The immutable CoreOS model guards the immutability of certain directories. With that being said, if we write a file (e.g. SSH key) to an immutable directory during an OS image build and use rpm-ostree to rebase onto the newly built OS image, that file will not actually be there when the node reboots. |
There was a problem hiding this comment.
I think I understand what you're getting at here, but let me clarify a few things.
In a container build, there are no immutable directories. That's why you can e.g. install packages or change stuff in /usr.
/var is special. It's not immutable either but in this context, files written to it will simply be ignored when deploying to the node because nodes have their own /var that wins.
SSH keys are a big one, but it's basically any path in the MachineConfig under /var will have this issue. I think in the short/mid-term, we'll have to keep having the MCO manually write those to the nodes (though that could still be done transactionally). But ideally longer term, we can work towards moving all big use cases for this to other directories under /usr or /etc or to systemd-tmpfiles dropins instead.
There was a problem hiding this comment.
|
|
||
| ### Phase 3: Deprecate rpm-ostree and Move Layered Image Update Path to Bootc | ||
|
|
||
| Upon completion of the above, the last step is to update the image rollout mechanism. In the foreseeable future, we will be expecting a dual-stream RHCOS where rpm-ostree will be gradually deprecated and transition its responsibility to bootc. Following this trend, MCO will also enable image deployment by both “rpm-ostree rebase” and “bootc switch” and, in the end, settle down on “bootc switch” for image rollout. |
There was a problem hiding this comment.
What is the "dual-stream RHCOS" referring to here? I.e. that we'll have e.g. 9.x and 10.x versions? Why wouldn't they be handled the same way?
| ### Workflow Description | ||
|
|
||
| 1. The cluster administrator wants to add new configuration to the OS on each of their cluster nodes. | ||
| 2. The cluster administrator specifies base OS image pull spec & secret, base OS extension pull spec, rendered image pull spec & secret, rendered image push spec & secret and custom content through a MachineOSConfig object. |
There was a problem hiding this comment.
Hopefully we don't have a need for the extensions container in the future.
Also ideally there's an optimized path for the common case where the rendered image pull and push secrets are the same?
enhancements/machine-config/alignment-with-image-mode-rhel-for-the-mco.md
Show resolved
Hide resolved
enhancements/machine-config/alignment-with-image-mode-rhel-for-the-mco.md
Show resolved
Hide resolved
| After On Clustering GA, we will gradually deprecate the current configuration method of direct file writing to disk and move to layered update path (Phase 1) by default. At the end of this phase, OCP nodes will be configured in a full image-based way and will have a RHEL-bootc layer, a CoreOS layer, an OCP node layer, and day-2 configuration layers added on top built and rolled out by On Cluster Layering discussed in Phase 1. Please note here that the proposal and implementation details of the first three layers (the RHEL-bootc layer, the CoreOS layer and the OCP node layer) are explained in the Splitting the RHCOS Image into layers enhancement, thus a non-goal for this enhancement. Some of the gaps we observed here are: | ||
| * "Default" MCO managed configuration moved to /usr and shipped as a container image layer: Includes all MCO specific config rendered to /usr + MachineConfigs which are rendered to /etc -> /usr/etc (to be confirmed) | ||
| * Configs that involve file writing to immutable local file system directories need to be handled: The immutable CoreOS model guards the immutability of certain directories. With that being said, if we write a file (e.g. SSH key) to an immutable directory during an OS image build and use rpm-ostree to rebase onto the newly built OS image, that file will not actually be there when the node reboots. | ||
| * Address functionality gap between bootc and rpm-ostree: figure out a story for kernel argument changes as a container image layer/label |
There was a problem hiding this comment.
There's some support for this now: https://containers.github.io/bootc/building/kernel-arguments.html
| - A: Go bindings for bootc is not a must and, as a result, will be outside of the scope of this enhancement and will not be implemented. The main purpose of creating go bindings for rpm-ostree/bootc is to have an easier way to parse the created json and to read the status of rpm-ostree/bootc. This can be done via creating a simple wrapper function inside of the MCO. It will make sense to separate it out to a standalone helper/library in the future when the demand raises, but is a non-goal for now. | ||
|
|
||
| * Discussion question for gaps highlighted in phase 2 | ||
| - Q: A problem with the current model is that configs that involve file writing to local file system directories are not respected by “rpm-ostree rebase” / “bootc switch”. For example, if we write an SSH key to “/home/core/.ssh/authorized_keys'' directory during an OS image build and use rpm-ostree to rebase onto the newly built OS image, that file will not actually be there when the node reboots. Will this be handled/considered in the future planning of RHCOS and bootc? |
There was a problem hiding this comment.
This is related to my earlier comment about /var.
| - Everything rebootless should not be in the image, unless we have live image changes. It is better for those changes to be in writable directories on the disk. Need to keep in mind the priority order of /usr vs /etc. | ||
| - MCO would have to apply its own internal configs first then the MachineConfigs. | ||
| - [Actionable item] Get a list of files managed by the MCO to further refine this. |
There was a problem hiding this comment.
Avoiding reboots when possible isn't unique to OCP of course. I think ideally here we try to lower this functionality into bootc directly. See e.g. containers/bootc#76 related to this.
enhancements/machine-config/alignment-with-image-mode-rhel-for-the-mco.md
Show resolved
Hide resolved
No description provided.