Stream introspection and forced upgrades (stream-switching)

[lucab]

This is a spin-off of one of the topics from #83 (comment). See bottom of https://meetbot.fedoraproject.org/teams/fedora_coreos_meeting/fedora_coreos_meeting.2019-03-13-16.32.log.html for context.

We should decide, implement and document the following things:

• how to inject stream label into OS content via coreos-assembler
• how to retrieve the above information (read-only) on a running node
• how users would perform a manual operation to switch stream

This is because the update story for FCOS is slightly different than it used to be on CL. In particular, for a node to update from current OS version to a new release, there must be an edge in the cincinnati graph going current->new. However, cross-stream edges are problematic as they would bridge different upgrades path, and nodes could end up erratically jumping across streams.

As such, we will likely start with nodes introspecting their own stream and sticking to that for upgrades. For a node to switch stream, a manual rpm-ostree upgrade plus reboot would be required.

[jlebon]

Related discussions in coreos/coreos-assembler#159.

[jlebon]

For a node to switch stream, a manual rpm-ostree upgrade plus reboot would be required.

Did you mean rpm-ostree rebase here?

[jlebon]

So the "obvious answers" (or at least something to bootstrap this discussion) here are:

how to inject stream label into OS content via coreos-assembler
how to retrieve the above information (read-only) on a running node

This could be part of the commit metadata, or keyed off of the branch name itself. The former is clearly nicer if we want to keep the ref as an implementation detail (theoretically, we could have Zincati rebase across refs on update if necessary; though changing refs would break manual rpm-ostree upgrade invocations). Keeping stream information in the commit metadata I think is valuable on its own though, just from a "build tracking" point of view.

how users would perform a manual operation to switch stream

Offhand, I don't think we need to have a super polished story around this given that it's likely not a common operation, right? Seems like documenting rpm-ostree rebase as the official method, along with the list of stream <--> ref we support would suffice?

[lucab]

Those seem reasonable to me. Are you thinking of storing the label in the treefile itself and then plumbing it through the layer till reaching the commit metadata? Or is there some more clever approach, maybe with support for templating-and-substitution via coreos-assembler?

[jlebon]

So one straightforward approach would be to do this as part of the pipeline, since it's already parameterized on the stream: e.g. expose rpm-ostree's --add-metadata-from-json (which we already use) at the CLI level of cosa build.

Or maybe a cleaner approach is indeed to expand the treefile spec so we can inject arbitrary metadata, just like --add-metadata-from-json. This has the key advantage of keeping it all as part of fedora-coreos-config, so manual cosa runs will still embed it. We already have stream-specific manifests where those could go (see this related commit message).

[lucab]

@jlebon from your comment above, I seem to understand that the plan is more or less fixed on letting rpm-ostree compose pick up the value and embed it in the metadata, so that a running node can introspect its own stream by looking for a well-known key via rpm-ostree status (or similar).

I don't have a good opinion on whether the stream label should be part of versioned content, or just injected on the fly (@ajeddeloh @bgilbert @arithx maybe have some feedback from CL process?).
I guess that choice is tied to how we perform promotions across channels, and in which channel we label locally-built images (the latter are not meant to update as they aren't in the update-graph, so it would be good for zincati to be able to detect them).

[jlebon]

I guess that choice is tied to how we perform promotions across channels

Right yeah, it works because we don't promote content/artifacts directly, but rather the source config itself (which includes the lockfile), and then trigger a new build. So any particular build is tied to a specific stream.

and in which channel we label locally-built images (the latter are not meant to update as they aren't in the update-graph, so it would be good for zincati to be able to detect them).

Yeah, good point. I agree with you that we don't want locally-built images to magically auto-update. Hmm, though wouldn't the version not being part of the update graph exactly the right way to detect this?

[lucab]

Hmm, though wouldn't the version not being part of the update graph exactly the right way to detect this?

It would then become trickier:

• if we don't mangle neither stream label nor version value, a local build may or may not upgrade depending on its ostree commit checksum (depending on bit-by-bit reproducibility )
• if we don't mangle stream but we mangle version value, it would get the full update-graph from cincinnati but won't find itself in that

In both cases, this requires server-side workload for an result which is not meant to be used. Additionally, the approach we pick here can be re-used for metrics-client to not send us data for ephemeral dev VMs.

[jlebon]

Had a chat with @bgilbert about this today. Here's what we came up with:

• Zincati will pull the stream label from the commit metadata and pass it to Cincinnati without extra processing. Zincati itself doesn't need to have stream-specific knowledge.
• Pinger and Zincati will take a simple enable/disable flag in their config file, defaulted in /usr and overridable in /etc.
• The fedora-coreos-config manifest will conditionally configure the default enable/disable flag for Pinger and Zincati on a per-stream basis using a drop-in in /usr. So for example, the testing manifest will enable both of them, while the testing-devel one will not. So by default, a cosa build of testing-devel will have them off. But developers can still turn them on if testing Pinger/Zincati themselves. Conversely, a cosa build of testing will have them on. We'll work on a separate proposal for detecting developer builds and disabling them by default as well.