FCOS will have multiple streams:
| Type | Name (SCM branch and ostree ref) | ostree repo |
|---|---|---|
| Production | next | prod |
| Production | testing | prod |
| Production | stable | prod |
| Development | testing-devel | annex |
| Development | next-devel | annex |
| Mechanical | rawhide | annex |
| Mechanical | branched | annex |
Development and mechanical streams are subject to change.
We need a way to both (1) fix the content set for a particular stream release, and (2) integrate new content into development streams.
- git
- rpm-ostree treefiles: manifest fed to rpm-ostree that contains the list of packages to use during a compose. Example.
- rpm-ostree treefile locks: pending rpm-ostree patch adding "lockfile" functionality similar to Cargo.lock/Gopkg.lock. This essentially means that the rpm-ostree compose is guaranteed to use specific package versions (or fail) as described in the lockfile. (To be clear, all of the below could probably be done without a lock file, since the treefile supports fully specifying the NEVRA, but having a separate lockfile allows for more sophisticated tooling and a cleaner treefile.)
- Koji tags: a way to track packages built in Koji. Koji is capable of creating yum repos from such tags. RPM builds may be "tagged" in so that the next repo regeneration includes it.
- dist-git: git where RPM spec files are kept and Koji builds source from.
Mechanical streams are not curated; they're automated nightly snapshots of the underlying repos. They source their RPMs from the regular Fedora repos (using 30 here to mean $currentrelease):
- rawhide <- f32
- branched <- f31 when a branch exists, otherwise tracks rawhide
Production streams are intended for production use. They source their RPMs from a single Koji tag, coreos-pool, from which we create a yum repo:
- next <- coreos-pool
- testing <- coreos-pool
- stable <- coreos-pool
Development streams are nightly snapshots of content headed for the production streams. There's one development stream at the base of each promotion path; thus, stable doesn't have one because it promotes from testing instead. next-devel will only be maintained in the periods where next is independent of testing. Development streams source their RPMs from a single Koji tag, coreos-pool, from which we create a yum repo:
- next-devel <- coreos-pool
- testing-devel <- coreos-pool
The Koji tag ensures that (1) packages are not automatically garbage collected, (2) stream builds are reproducible (up to the GC retention policy we agree upon), and (3) packages are added to the pool (and thus into the production streams) in a controlled manner.
There is also a second Koji tag, coreos-release, for packages which have been included in a production build. Packages in coreos-release have a longer TTL than coreos-pool, increasing the time that official builds can be reproduced by others.
We maintain a git repository containing the rpm-ostree treefile and lockfiles. This could be fedora-coreos-config. We have one branch for each stream, and no main branch.
For the mechanical streams, a nightly job will run the compose from the corresponding yum repos and SCM refs. This job will output a lockfile for each CPU architecture. Those lockfiles will be committed to Git to preserve a record of the build's contents, and the builds will be pushed to the corresponding ostree refs. The branched lockfile will also be PR'd to the {testing-devel, next-devel} branch, the latter only during the part of the cycle where next-devel is maintained. We want to keep the development branches ready to release, so those PRs are not merged unless green.
The lockfiles produced from the automatic snapshot will never be hand-modified, and in the next/testing/stable branches will never be modified at all except during promotions. Instead, pins (to older NEVRAs) and updates (to newer ones) will be hand-maintained in the Git branches in a separate lockfile that overrides the autogenerated ones. These overrides will be the major distinction between the mechanical refs and the "curated" (development/production) refs. Each curated branch will have one override file, which can carry both CPU-architecture-independent and architecture-specific overrides.
When it's time to cut e.g. a promoted testing release, we push a merge commit to the testing branch which takes the testing-devel branch's tree in entirety. (That is, we use the Git "theirs" merge strategy, which, helpfully, doesn't exist.) We then tag the Git commit on the testing branch and do an official build and CI run. In the absence of flakes (ha!) this will pass because the commit is known to be green. If the build is bad, we abandon the release and iterate. Otherwise, we proceed with releasing artifacts.
If we need to update an existing testing release, there's no need for another merge commit; we just commit changes to the testing branch and tag the release from there.
When a lockfile update is merged, this triggers a process which watches for pushes to the curated branches, and adds all the builds from the updated lockfile to coreos-pool.
During production builds, the pipeline tags packages from the lockfile into coreos-release.
Update the development treefile as usual. On the next bot push, the lockfile will be updated to include that package entry.
To focus development effort, there will be one base treefile shared across all branches, whose canonical copy will live in the testing-devel branch. Changes will automatically be mirrored to next-devel and to the mechanical branches. To address divergence across Fedora releases, each branch will also have an overlay treefile (possibly empty):
- testing-devel
- next-devel -> automatically mirrored to branched
- rawhide
There are two cases:
- Backporting an already built Fedora package into a prod release:
- We bump the override file entry for that specific package
- The package automatically gets koji-tagged on push
- Backporting a patch which isn't in a Fedora package (or the package was already updated too far):
- We build the RPM from a dist-git branch
- We add the RPM to the koji tag (though the build target destination tag could already be set to remove this manual tagging step)
- We bump the override file entry for that specific package