Upgrade Testing to a future release

[ajeddeloh]

The first FCOS preview release is not able to upgrade automatically due to a bug. We should add testing to ensure that FCOS can update out of the the version under test.

Couple of things to consider:

• Do we test zincati as well as the rpmostree process?
• What do we test upgrading to? A couple proposed options are:
• The latest nightly testing-devel
• A special static ostree ref that just reports "the upgrade works"

[lucab]

In an out-of-band chat, @jlebon and @arithx came up with an interesting suggestion: we can mimic an update out of the the version under test by using the previous release in the channel as a target.
This is equivalent to a downgrade and not supported in general (due to software in general not being able to handle downgrades of state from newer versions), but for the sake of testing the update process it should work without major issues.

[jlebon]

Right, yeah. I also really want us to exercise upgrade testing on testing-devel. To do this we can:

1. store testing-devel commits in a separate OSTree repo (which is actually called for in the stream tooling doc, though I think that decision predated Move OSTree commit into build directory coreos-assembler#515)
2. have a dumber Cincinnati (or the real Cincinnati) that just always returns an edge from the test node's version to the latest version.

[lucab]

I set up item 2 above in Fedora Communishift. It can be used with:

# cat /etc/zincati/config.d/99-updates-latest.toml
updates.enabled = true
cincinnati.base_url= "https://fcos-updates-latest.apps.os.fedorainfracloud.org"

As the OSTree repo is not yet up, Zincati currently fails with No such branch 'fedora/x86_64/coreos/testing-devel' in repository summary.

[jlebon]

I opened an issue for tracking the OSTree repo annex work: #266

[jlebon]

At least for release builds, we'll probably want to make sure we test updates from older releases too. Not sure how comprehensive we want to be, and it will take longer and longer as we add more update barriers (though those shouldn't be too frequent). But at least testing from the initial release would be nice. Each release carries state forward that might be incompatible with latest releases. See #322.

[jlebon]

Related:
coreos/fedora-coreos-cincinnati#8
coreos/fedora-coreos-cincinnati#9

[jlebon]

coreos/mantle#1168

[dustymabe]

@jlebon I've seen upgrade testing in the pipeline. Is this done already?

[jlebon]

I think we can mark this as closed by coreos/mantle#1168 and have a separate tickets for upgrade tests that will leverage the compose repo.