Create confidence/testing tiers for platforms supported by Fedora CoreOS

[travier]

Describe the enhancement

We now support a good number of platforms but only fully test some of them (AWS, GCP, OpenStack and QEMU/KVM). The release pipeline also takes more time to complete for each new platform that we add support for.

One option to limit complexity, release overhead and convey more clearly to users the level of testing for each platform would be to introduce Rust-like tiered platform support:

• Tier 1:
  • Tests are run on the platform in CI for each release.
  • Confidence in the platform support is very high.
  • Tests failures are blocking release (exception of known issues)
• Tier 2:
  • Artifacts are produced from tested bits but not directly tested on the platform.
  • Confidence in the platform support is high.
  • Although they are not blocking releases due to lack of CI integration, issues on those platforms have a high-priority.
• Tier 3:
  • Artifacts are produced from tested bits but not directly tested on the platform.
  • Confidence in the platform support is not well known.
  • Issues on those platforms have a lower priority and help is welcomed.

The current supported platforms would fall into the following tiers:

• Tier 1: AWS, GCP, QEMU/KVM
• Tier 2: OpenStack, VMware, BareMetal, Alibaba Cloud, Azure, Exoscale, IBM Cloud, Vultr, DigitalOcean,
• Tier 3: ?

We could then decide that:

• Tier 3 platform artifacts are only built after Tier 1 & 2 ones are done for a release and failures are non blocking nor critical which would reduce release overhead.
• Tier 2 & 3 platform artifacts are not built by CI for testing PRs or for testing / mechanical streams which would reduce CI overhead.

Moving platforms from one tier to another is only based on testing support and any help is welcomed.

[jlebon]

Tier 2: VMware, BareMetal, Alibaba Cloud, Azure, Exoscale, IBM Cloud, Vultr, DigitalOcean,

Hmm, I think bare metal should go in Tier 1. (Though it's true that our CI doesn't actually test on a bare metal machine directly, but the intent is definitely for bare metal to be fully supported. And maybe we do add CI that runs on bare metal machines in e.g. AWS or Packet the future!)

[travier]

See also #719

[travier]

From this week community meeting:

<dustymabe> it seems like this issue has a lot of components
<dustymabe> 1) is it a good idea to break things into tiers?
<dustymabe> 2) how do we present that information to the community
<dustymabe> 3) how do we define the tiers and more strategically test/use resources

We discussed point 1 and agreed that we should move forward with this.

We also agreed that we should probably change the naming from support to confidence or testing tiers.

I've updated the first entry in this issue with the output from the discussion.

[bgilbert]

Discussed at today's community meeting. Notes:

• If we want to allow omitting broken tier 3 artifacts from a release, we should teach the stream metadata generator to continue pointing to the previous release for those artifacts. The generator currently doesn't have any notion of history.
  • Artifact build failures don't seem likely in practice, so it'd be easier to continue treating them as blocking for now.
• The platform tier assignments and definitions could be presented in a docs table, a coreos@ mailing list post, on the download page, and in the getting-started docs for each platform.