-
Notifications
You must be signed in to change notification settings - Fork 120
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add a doc for container provisioning and updates #540
Open
cgwalters
wants to merge
1
commit into
coreos:main
Choose a base branch
from
cgwalters:doc-firstboot-rebase-container
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+128
−1
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,127 @@ | ||
= Deriving from Fedora CoreOS as a bootable container image | ||
|
||
NOTICE: At the current time, the suggested model here does not integrate with the current "update barriers" model used for default Fedora CoreOS updates. While relatively rare so far, this means that some container-based updates may see issues that were otherwise gated. This issue is being worked on. More information in https://github.com/coreos/fedora-coreos-tracker/issues/1263[this issue]. | ||
|
||
== Bootable containers overview | ||
|
||
The operating system images used for Fedora CoreOS are available as a standard container image; there is one per stream. See xref:update-streams.adoc[Update Streams]. | ||
|
||
- quay.io/fedora/fedora-coreos:stable | ||
- quay.io/fedora/fedora-coreos:next | ||
cgwalters marked this conversation as resolved.
Show resolved
Hide resolved
|
||
- quay.io/fedora/fedora-coreos:testing | ||
|
||
== Creating custom builds | ||
|
||
See the https://github.com/coreos/layering-examples[layering examples] git repository. While the examples there tend to use Containerfile/Dockerfile style builds, you can use any tool which can build containers and push the result to a registry. | ||
|
||
The default suggested approach is to have container tags be the source of truth for initating upgrades, and to have the client systems poll. Example systemd units for this are in [https://github.com/coreos/layering-examples/tree/main/autoupdate-host]; they boil down to a systemd `.service` which just runs `rpm-ostree upgrade --reboot` and a corresponding `.timer` unit to run it once a day. | ||
|
||
Of course, client side logic could be much more complex, including a privileged container image that runs completely arbitrary logic. | ||
|
||
== Using Ignition to switch to a bootable container image | ||
|
||
The https://coreos.github.io/rpm-ostree/container/[rpm-ostree container page] describes the commands for interacting with bootable ostree container images. | ||
|
||
This systemd unit will run only on the first boot, and switch a "stock" Fedora CoreOS node to your custom image. | ||
|
||
[source,yaml] | ||
---- | ||
variant: fcos | ||
version: 1.4.0 | ||
systemd: | ||
units: | ||
# Our custom unit for rebasing | ||
- name: rebase-custom.service | ||
enabled: true | ||
contents: | | ||
[Unit] | ||
Description=Fetch and deploy target image | ||
# Only run on the firstboot | ||
ConditionFirstBoot=true | ||
After=network-online.target | ||
[Service] | ||
# This ordering is important | ||
After=ignition-firstboot-complete.service | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think this is now There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. |
||
Type=oneshot | ||
RemainAfterExit=yes | ||
ExecStart=rpm-ostree rebase --reboot ostree-unverified-registry:quay.io/example/example-derived:latest | ||
[Install] | ||
WantedBy=multi-user.target | ||
---- | ||
|
||
== Adding pull secrets for private container images | ||
|
||
If your registry requires authentication, then you can add a pull secret. See https://coreos.github.io/rpm-ostree/container/#registry-authentication[container pull secrets]. | ||
|
||
[source,yaml] | ||
---- | ||
variant: fcos | ||
version: 1.4.0 | ||
storage: | ||
files: | ||
- path: /etc/ostree/auth.json | ||
mode: 0600 | ||
contents: | ||
inline: > | ||
{ | ||
"auths": { | ||
"quay.io": { | ||
"auth": "..." | ||
} | ||
} | ||
} | ||
---- | ||
|
||
== Maintaining custom builds | ||
|
||
As the upstream container image (e.g. quay.io/fedora/fedora-coreos:stable) changes for security and functionality improvements, you are responsible for re-building your derived image. Implementing this depends on your chosen container build system. | ||
|
||
However, key to this model is the assumption that if you're deploying Fedora CoreOS, you have already invested in container infrastructure and can manage the operating system updates in the same way as application containers. | ||
|
||
== Understanding Ignition versus container content | ||
|
||
=== Storage | ||
|
||
First, if you wish to configure storage (see xref:storage.adoc[Configuring Storage] ) that must still be done via the initial Ignition configuration. | ||
If you wish to change a system's storage configuration, the current recommendation is still that that be done via full system reprovisioning. | ||
|
||
=== Filesystem | ||
|
||
Ignition's philosophy is that it runs exactly once, and any subsequent changes should generally be done by node reprovisioning. This can | ||
be very expensive in some environments. Storing configuration directly | ||
with the OS content allows strongly coupling them and ensuring that | ||
the state of your machines is exactly what you build and replicate | ||
as a container image. | ||
|
||
Note however that content injected via Ignition will still persist as | ||
machine-local state into subsequent OS updates (including by default | ||
our unit to perform the switch, which will remain inactive; though your | ||
container could run code to delete or mask the unit). | ||
|
||
=== Machine local state | ||
|
||
A powerful ability of using OCI containers as a mechanism for host | ||
systems management is that one can create chains of derived images; | ||
for example, you might have a "base" image derived from Fedora CoreOS | ||
that include e.g. basic agents or logging configuration, but then further builds | ||
that customize that base for particular sites or cloud environments. | ||
|
||
It's even possible to create builds which are targeted solely for | ||
a specific machine. | ||
|
||
However, this may not always be ergonomic to do. Ignition here | ||
can serve as a mechanism to apply machine-specific state. Commonly, | ||
this machine specific state does not need to change often. A classic | ||
example here is configuring `/etc/hostname`. | ||
|
||
=== Bootstrap configuration to fetch container images | ||
|
||
The pull secret described above is also a form of configuration on | ||
the system that must be present (in some environments) to pull the container | ||
image, and hence must be configured via Ignition today. | ||
|
||
==== Machine local bootstrap configuration | ||
|
||
There is a class of configuration which combines these two previous types, | ||
such as static IP addressing. Here, Ignition again makes sense | ||
to perform this configuration. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shouldn't this be linked from the nav tree? Not sure where it'd belong... Going along with the experimental labeling comment, maybe easiest is to have a new "Experimental features" parent with this as the first child?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What would you consider experimental about this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Or, to flip it around - in your view, what would be the criteria to "graduate"?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It just doesn't seem to me like we've worked through all the details. Container layering impacts multiple parts of FCOS we're deeply opinionated about. We started on this in e.g. coreos/fedora-coreos-tracker#1219 and coreos/fedora-coreos-tracker#1263 but then... fires happened.
At a minimum, I think we should resolve coreos/fedora-coreos-tracker#1263. But also, I think we would need a larger rework of what tests we run and how the docs are structured so it's properly integrated in our provisioning and configuration story. (But ideally, we also introduce a better UX for this stuff.)
How about something like this near the top:
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah but, speaking bluntly I feel like that's not going to happen unless I keep pushing it, hence this PR.
OK, I have my opinion written there.
That's true, filed coreos/fedora-coreos-tracker#1484
That's partly in coreos/butane#428
Mmmm. I guess that's a pivotal decision here. I am not forseeing any changes which would require reprovisioning. Are you?
This part is covered below there too, but sure we can emphasize it. I tossed up this page in https://hackmd.io/QM1V-FujTmalikgi5JQHPw to avoid round trips.
(yeah it's not actually markdown sadly, but I'm just using it as a realtime editor)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You're welcome to help work through the details, too. 🙂
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK, I added a note to the top here which covers what is IMO the top last issue we considered "blocking" around the barriers. It's just up front about that outstanding bug, but I'm hopeful we'll have more infrastructure there by the time we see the need for another barrier.