streams: switch between streams using Zincati configurations

[zonggen]

Feature Request

As a user of Zincati and FCOS, having an alternative option to automatically switch streams and keep the system updated with the Cincinnati backend might be a good feature to support.

Example Usage

Provide a /etc/zincati/config.d/10-stream-testing.toml with following content:

[stream]
target = fedora/x86_64/coreos/testing

This will tell Zincati to switch to testing stream and automatically update and reboot the system as it would with a sudo rpm-ostree rebase "fedora/x86_64/coreos/testing".

[zonggen]

This conversation came up from the 1:1 with @jtligon so I summarized the idea and see if it could be something I can be working on.

cc @lucab

[lucab]

The overall feature request is an intentional NACK, by design.
However I agree this should be written down in some design doc, so thanks for opening the ticket. I'll iteratively braindump in here, so let's keep this open for a bit and later transform to a markdown doc.

1. Zincati consumes target revisions/commits from Cincinnati. The agent doesn't know about repo layout/refs.
2. Cincinnati provides us a graph of update edges. A graph is constrained to a single stream, to keep RelEng complexity (rollouts and barriers) at hand.
   a. There is no information on which update edges are safe to follow across streams. Actually identifying those safe edges would require a large CI matrix at each FCOS release event.
   b. Just pointing Zincati to a different Cincinnati update stream won't work (missing current OS release as a source node)
   c. This feature would require a forest of DAGs and knowledge about which edges are safe, from any node in any stream to any node in any stream. This has exponential complexity.
3. Cross-stream rebases may result in downgrades (e.g next -> stable). Automatic downgrades are a huge footgun that we are trying to avoid as hard as we can.
4. Zincati is configured on first-boot via Ignition. It makes little sense to start with a fresh node on a stream and configure it to auto-update to something else as soon as possible.
5. Cross-stream rebases should be an infrequent day2 operation, which requires anyway some active intervention (setting the target stream somehow). As such it is not a good fit for file configuration.
6. Container Linux used to have something similar. However, we traded off that part when we went for an explicit update graph. It was a conscious decision, where we gained the ability to have barriers and multiple parallel ongoing rollouts.

[lucab]

@dustymabe at some point in the past came up with two feature requests similar to this one (which I haven't fully slept on enough nor opened tickets for) which are bit more feasible:

• CLI to query Zincati to know what is the current commit on a stream (to use as a target for rpm-ostree rebase)
• CLI to tell Zincati to remember current OS version/stream (useful for switching back to the original stream, after a manual rebase)

Both of them require introducing a DBus server/client architecture (we currently lack a good async Rust library for this). The latter also requires a good amount of state-machine groundwork to allow safe concurrent mutations based on external events from DBus.

[lucab]

Self note: we should also add a devdoc about what Zincati defines as downgrades (graph-based) and what rpm-ostree defines as downgrades (timestamp-based).
This is relevant here because when crossing an update stream we cannot identify downgrades based on the graph (which we don't have, see above), and timestamp-based logic can have both false-positives and false-negatives (see next->stable example above, where comparing release timestamps isn't directly meaningful).

[jtligon]

Thanks for the comments @lucab. this makes things much clearer.

[freedge]

Zincati is configured on first-boot via Ignition. It makes little sense to start with a fresh node on a stream and configure it to auto-update to something else as soon as possible.

I have a use case where I provision a coreos vm, however there is no connectivity to updates.coreos.fedoraproject.org or the various mirrors it returns, instead the vm has connectivity to quay.io (ideally through a mirror).
In this case, once provisioned using the testing image, the update needs to be done using
rpm-ostree rebase ostree-unverified-registry:quay.io/coreos/fedora-coreos:testing.

For this usecase, it looks like the feature as described in the "example usage" would be helpful.

[cgwalters]

Closing per above discussion.

For the container tooling, we won't be integrating zincati with rebasing containers. Instead see coreos/fedora-coreos-docs#540