-
Notifications
You must be signed in to change notification settings - Fork 59
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Stream and release metadata #98
Comments
If we want to ship ostree static deltas, release metadata should also record the expected predecessor(s) of a release. This will usually be the previous release on the stream, not the previous release on the snapshot branch. (For example, the predecessor of 30.5.2 might be 30.4.3.) Note that "release" really means release; abandoned unreleased versions (which can occur in CL; not sure if they will with FCOS) shouldn't count as predecessors. In some cases there will be multiple predecessors, notably when rollout of the previous release was halted due to a regression. cc @sinnykumari |
coreos-assembler's builds.json + meta.json is one model. It's OK. The git model could also be nice although it requires more work for endpoints (it's not just a REST API to consume), though obviously one could write one on top. Modeling the releases in ostree itself has some nice properties although it's also not REST. I think coreos-assembler should probably define a schema for "streams" that point to builds.json; obviously one can model it as just separate dirs but having it be discoverable (like ostree summary files list multiple refs) would be nice. I guess all of this gets to Dusty's push to model as much data as possible in ostree. That's certainly possible - but it would mean e.g. ostree commit metadata would have e.g. AMI ids. Which...is probably fine, in the end the data isn't going to be more than 10k I'd guess which is way way smaller than the statically linked binaries in the OS. |
Let me just add two concrete use cases we had with CoreOS/Container Linux (CL) which may be useful as a litmus test: With CL it was challenging to discover all of the related files to an OEM to know which files one would need to sync. When reflecting on the distribution repos there have been times when file name changes were necessitated or when new platforms had files added. Using OpenStack and PXE platforms as an example, in the case of OpenStack all that is needed is to pull down the virtual machine image while with PXE one has to pull down the Kernel and the Initramfs. While one may pull down a specific version of an OpenStack image it will quickly get out of date meaning that immediately after install a host has to go down for an update needlessly wasting time during provisioning. Our solution was to present two scripts. A generic one which could check the ETag on In the case of PXE we then had two distinct paths of execution: Either using a PXE load to then bootstrap the host or using it as a live running system. In the latter case this allowed organizations to use Container Linux as a bootstrapping platform for their own clouds. Multiple public cloud providers used Container Linux in combination with a containerized agent to perform dynamic bootstrapping of hosts (at least one of them used this with OpenStack Ironic). This presented an interesting problem though, if a user was not persisting data to disk then there is no mechanism for Omaha (or Cincinnati) based updates. This meant that the synchronization of OS components MUST occur at the infrastructure level. For this users would then be required to synchronize down the components of an OEM. As (throughout the life of Container Linux) metadata was not generally presented on a per-OEM basis, this meant users had to report to string matching of content to identify the content they needed to mirror. One example is this script -
|
Here are some sample metadata blobs for discussion. I've written them in YAML for clarity, but the published documents would be JSON. Stream metadata# Include stream name so the document is self-contained
stream: stable
metadata:
last-modified: "2019-06-04T16:18:34Z"
architectures:
x86_64:
artifacts:
# Some of these will be useful for many users, such as qemu or
# openstack. Some will likely only be useful for cloud operators,
# such as digitalocean or packet. Some, such as aws, are useful
# for users in special situations.
aws:
release: 30.1.2.3
formats:
# Generally one format per platform, but allow for future expansion
# without obscuring the platform ID (as on Container Linux)
"vmdk.xz":
# Generally only one artifact, but not always
disk:
location: https://artifacts.example.com/dsB2fnzP7KhqzQ5a.vmdk.xz
signature: https://artifacts.example.com/dsB2fnzP7KhqzQ5a.vmdk.xz.sig
sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
azure:
release: 30.1.2.3
formats:
"vdi.xz":
disk:
location: https://artifacts.example.com/aeng0xah6vaaVosh.vdi.xz
signature: https://artifacts.example.com/aeng0xah6vaaVosh.vdi.xz.sig
sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
digitalocean:
release: 30.1.2.3
formats:
"raw.xz":
disk:
location: https://artifacts.example.com/ichaloomuHax9ahR.raw.xz
signature: https://artifacts.example.com/ichaloomuHax9ahR.raw.xz.sig
sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
gcp:
release: 30.1.2.3
formats:
"tar.gz":
disk:
location: https://artifacts.example.com/ais7tah1aa7Ahvei.tar.gz
signature: https://artifacts.example.com/ais7tah1aa7Ahvei.tar.gz.sig
sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
metal:
release: 30.1.2.3
formats:
"raw.xz":
disk:
location: https://artifacts.example.com/xTqYJZKCPNvoNs6B.raw.xz
signature: https://artifacts.example.com/xTqYJZKCPNvoNs6B.raw.xz.sig
sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
iso:
disk:
location: https://artifacts.example.com/ADE5GO3bjAXeDcLO.iso
signature: https://artifacts.example.com/ADE5GO3bjAXeDcLO.iso.sig
sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
pxe:
kernel:
location: https://artifacts.example.com/hkIj8FkCydT3lV9h
signature: https://artifacts.example.com/hkIj8FkCydT3lV9h.sig
sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
initramfs:
location: https://artifacts.example.com/a9ytS8yB4cGZpca1.cpio.gz
signature: https://artifacts.example.com/a9ytS8yB4cGZpca1.cpio.gz.sig
sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
"installer.iso":
disk:
location: https://artifacts.example.com/KwKye6YW4SIIPrhY.iso
signature: https://artifacts.example.com/KwKye6YW4SIIPrhY.iso.sig
sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
installer-pxe:
kernel:
location: https://artifacts.example.com/EtqI0KsLIwZOHlCx
signature: https://artifacts.example.com/EtqI0KsLIwZOHlCx.sig
sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
initramfs:
location: https://artifacts.example.com/EhoS1x66RVA2k8y6.cpio.gz
signature: https://artifacts.example.com/EhoS1x66RVA2k8y6.cpio.gz.sig
sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
openstack:
release: 30.1.2.3
formats:
"qcow.xz":
disk:
location: https://artifacts.example.com/oKooheogobofai8l.qcow.xz
signature: https://artifacts.example.com/oKooheogobofai8l.qcow.xz.sig
sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
packet:
release: 30.1.2.3
formats:
"raw.xz":
disk:
location: https://artifacts.example.com/Oofohng0xo2phai5.raw.xz
signature: https://artifacts.example.com/Oofohng0xo2phai5.raw.xz.sig
sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
qemu:
release: 30.1.2.3
formats:
"qcow.xz":
disk:
location: https://artifacts.example.com/Siejeeb6ohpu8Eel.qcow.xz
signature: https://artifacts.example.com/Siejeeb6ohpu8Eel.qcow.xz.sig
sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
virtualbox:
release: 30.1.2.3
formats:
ova:
disk:
location: https://artifacts.example.com/yohsh2haiquaeYah.ova
signature: https://artifacts.example.com/yohsh2haiquaeYah.ova.sig
sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
vmware:
release: 30.1.2.3
formats:
ova:
disk:
location: https://artifacts.example.com/quohgh8ei0uzaD5a.ova
signature: https://artifacts.example.com/quohgh8ei0uzaD5a.ova.sig
sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
images:
# Cloud images to be launched directly by users. These are in a
# separate section because they might not always in sync with the
# release artifacts above.
aws:
regions:
us-east-1:
# We know the release because we uploaded it, so might as well
# list it.
release: 30.1.2.3
image: ami-0123456789abcdef
us-east-2:
release: 30.1.2.3
image: ami-0123456789abcdef
azure:
# We could give a specific image URN here, but we probably want
# users to always use a Marketplace URN. So this is a static
# string, and represents advice rather than a value we might
# change.
image: Fedora:CoreOS:stable:latest
gcp:
# We could give a specific image name here, but we probably want
# users to always use an image family. So this is a static string,
# and represents advice rather than a value we might change.
image: projects/fedora-cloud/global/images/family/fedora-coreos-stable
digitalocean:
# We don't control platform ingest, so an image slug is probably
# the best we can do.
image: fedora-coreos-stable
packet:
# Images don't have addressable versions, so an operating system
# slug is the best we can do.
image: fedora_coreos_stable
updates:
# Primarily meant as input to Cincinnati
release: 30.1.2.3 We could also include artifact Override fileThe simplest possible implementation of the override file would just be partial stream metadata, overriding corresponding parts of the automatically generated one. A fancier one would allow references to older releases, but that functionality could be added later if needed. Release metadataThis seems largely redundant with release: 30.1.2.3
stream: stable
metadata:
last-modified: "2019-06-04T16:18:34Z"
architectures:
x86_64:
commit: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
media:
aws:
artifacts:
"vmdk.xz":
disk:
location: https://artifacts.example.com/dsB2fnzP7KhqzQ5a.vmdk.xz
signature: https://artifacts.example.com/dsB2fnzP7KhqzQ5a.vmdk.xz.sig
sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
images:
us-east-1:
image: ami-0123456789abcdef
us-east-2:
image: ami-0123456789abcdef
azure:
artifacts:
"vdi.xz":
disk:
location: https://artifacts.example.com/aeng0xah6vaaVosh.vdi.xz
signature: https://artifacts.example.com/aeng0xah6vaaVosh.vdi.xz.sig
sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
images:
global:
image: Fedora:CoreOS:Stable:30.1.2.3
digitalocean:
artifacts:
"raw.xz":
disk:
location: https://artifacts.example.com/ichaloomuHax9ahR.raw.xz
signature: https://artifacts.example.com/ichaloomuHax9ahR.raw.xz.sig
sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
gcp:
artifacts:
"tar.gz":
disk:
location: https://artifacts.example.com/ais7tah1aa7Ahvei.tar.gz
signature: https://artifacts.example.com/ais7tah1aa7Ahvei.tar.gz.sig
sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
image: projects/fedora-cloud/global/images/fedora-coreos-stable-30-1-2-3
metal:
artifacts:
"raw.xz":
disk:
location: https://artifacts.example.com/xTqYJZKCPNvoNs6B.raw.xz
signature: https://artifacts.example.com/xTqYJZKCPNvoNs6B.raw.xz.sig
sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
iso:
disk:
location: https://artifacts.example.com/ADE5GO3bjAXeDcLO.iso
signature: https://artifacts.example.com/ADE5GO3bjAXeDcLO.iso.sig
sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
pxe:
kernel:
location: https://artifacts.example.com/hkIj8FkCydT3lV9h
signature: https://artifacts.example.com/hkIj8FkCydT3lV9h.sig
sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
initramfs:
location: https://artifacts.example.com/a9ytS8yB4cGZpca1.cpio.gz
signature: https://artifacts.example.com/a9ytS8yB4cGZpca1.cpio.gz.sig
sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
"installer.iso":
disk:
location: https://artifacts.example.com/KwKye6YW4SIIPrhY.iso
signature: https://artifacts.example.com/KwKye6YW4SIIPrhY.iso.sig
sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
installer-pxe:
kernel:
location: https://artifacts.example.com/EtqI0KsLIwZOHlCx
signature: https://artifacts.example.com/EtqI0KsLIwZOHlCx.sig
sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
initramfs:
location: https://artifacts.example.com/EhoS1x66RVA2k8y6.cpio.gz
signature: https://artifacts.example.com/EhoS1x66RVA2k8y6.cpio.gz.sig
sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
openstack:
artifacts:
"qcow.xz":
disk:
location: https://artifacts.example.com/oKooheogobofai8l.qcow.xz
signature: https://artifacts.example.com/oKooheogobofai8l.qcow.xz.sig
sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
packet:
artifacts:
"raw.xz":
disk:
location: https://artifacts.example.com/Oofohng0xo2phai5.raw.xz
signature: https://artifacts.example.com/Oofohng0xo2phai5.raw.xz.sig
sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
qemu:
artifacts:
"qcow.xz":
disk:
location: https://artifacts.example.com/Siejeeb6ohpu8Eel.qcow.xz
signature: https://artifacts.example.com/Siejeeb6ohpu8Eel.qcow.xz.sig
sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
virtualbox:
artifacts:
ova:
disk:
location: https://artifacts.example.com/yohsh2haiquaeYah.ova
signature: https://artifacts.example.com/yohsh2haiquaeYah.ova.sig
sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
vmware:
artifacts:
ova:
disk:
location: https://artifacts.example.com/quohgh8ei0uzaD5a.ova
signature: https://artifacts.example.com/quohgh8ei0uzaD5a.ova.sig
sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 |
Some discussion in #187. |
In an OOB discussion we were discussing having an index for the release metadata that would be consumed by tooling to determine the release metadata endpoint for specific OSTree commits / versions. This index will be generated by
|
@arithx do we plan to use the "index by hash" capability of this for something? If not, an alternate approach is to have (hash,version,metadata) as the same level in a single object, and keep them in an array with bottom-append (this is what cincinnati does). |
@lucab my intention was the allow the tooling (like cincinnati), that primarily operate on the hash, to do a direct lookup rather than having to search through the entire array for one containing the right hash. |
Both the stream and release metadata should have some indication of freshness. A last-modified timestamp is straightforward and seems like a good idea. I like the idea of also adding a serial number for easy reference, though that would make updates more complicated (since generating an updated artifact would require fetching the current one). Thoughts? |
Could we start with the just the timestamp and go from there? |
Updated the release metadata index structure again in the comment. |
@jlebon Yes, let's do that. I've added timestamps to both documents, under |
@bgilbert: for the release index I'd be okay with using a similar structure but I'd change the name of the key from |
I was trying to avoid the dashes/underscores/camel-case debate. 😁 But I agree that e.g. |
Updated both documents to use |
Per discussions in the [fedora-coreos-tracker](coreos/fedora-coreos-tracker#98 (comment)), modify the build metadata structure to better support multi-arch.
Per discussions in the [fedora-coreos-tracker](coreos/fedora-coreos-tracker#98 (comment)), modify the build metadata structure to better support multi-arch.
Per discussions in the [fedora-coreos-tracker](coreos/fedora-coreos-tracker#98 (comment)), modify the build metadata structure to better support multi-arch.
Followup from a private discussion: all the manifests above are meant for automatic/machine consumption, so there is the additional topic of signing them (and related key management). For the moment, we are ensuring via TLS that those cannot be tampered on the wire. /cc @dustymabe |
Created #213 for #98 (comment). |
I only partially followed this discussion originally. Is it plume that is creating this metadata from the cosa builds? |
Plume generates release metadata and fedora-coreos-stream-generator generates stream metadata. stream metadata also gets stored into fedora-coreos-streams repo. Some discussion related to stream metadata is in issue #193 |
OK, a lot of useful links there, thanks! I think my high level concern here is simple: What's shared between the FCOS and RHCOS teams today is mostly https://github.com/coreos/coreos-assembler - and of that only a subset (i.e. not plume). Or to rephrase: There's a lot of stuff that got invented in this issue that hence isn't shared. As was noted in this comment
Hmm...a lot of this seems to be trying to support having different platforms at different build IDs, but that seems like a very unusual case? |
And to fully xref the reason I came here is thinking about coreos/coreos-assembler#719 |
It's an unusual case but a critical one. For example, we can't release Container Linux if any AWS region is down. |
I've thought about that problem a lot too for RHCOS. But the thing is we have a clear distinction between "bootimages" and updates. And there's no reason for FCOS not to have that as well, right? |
FCOS releases install and upgrade images together, as CL does. This is partly to avoid confusion, and partly because there are workflows where users deploy new OS versions solely via image launches and not via upgrades. |
So I guess the question is to what degree do we view "strong bootimage and update binding" as a property of "FCOS tooling" (down to the cosa level e.g.) or is it something that's more of a policy defined in the FCOS pipeline? Actually this is a very interesting discussion because today for RHCOS we do usually want "strong bootimage binding"...I think. We haven't really discussed it but my instinct is that if we allowed the bootimage versions on different platforms to float and the update stream to float...we'd be confused fast. On the other hand, personally I'm not opposed to the idea of potentially having "per platform bootimage versions". But it's also because today we don't need to release bootimages often that we don't need to worry about a particular platform's transient availability. |
Ah, yeah. It's completely a policy question. We do want the ability to release install images and updates separately, it'll just be an uncommon case. |
Clearly at this point the ship has sailed for FCOS and the user-visible output is release streams. OK. Today for RHCOS the bootimages are hidden in the installer and also published in an ad-hoc fashion to https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/ I'm looking at fixing this as part of openshift/enhancements#201 and I'm debating whether and how we introduce release streams for RHCOS. A proposal:
|
SGTM! |
I'm looking at this again and just some notes to myself; there's really 3 levels of thing (for each stream) type:
I'm struggling to try to articulate the rationale in having both cosa |
A major factor is that not all builds are released. It's not uncommon that we do e.g. two stable builds before a release. The release index (
Stream metadata could in theory be directly updated from cosa's |
|
For reference, https://github.com/coreos/fedora-coreos-tracker/tree/master/metadata is the canonical place where we describe FCOS metadata. |
Thanks! I created openshift/os#477 based on this so far. |
There are two types of metadata relevant to FCOS releases. We should publish each type in a unified machine-readable format.
Proposal
Release metadata
Release metadata is associated with a particular OS version. It should include:
Release metadata is populated incrementally, but each field should be immutable once added. For example, we might need to publish an urgent security release while an AWS region is down. In this case, we'd provide the available AMI IDs, and backfill the metadata once we got an AMI published into the lagging region. Platforms with slow publishing processes (such as AWS Marketplace) might always need their image IDs backfilled after the release.
There's been some discussion of storing release metadata as ostree detached metadata (coreos/coreos-assembler#189 (comment)) or in a Git repo (coreos/coreos-assembler#203). Perhaps release metadata should be signed.
Stream metadata
Stream metadata is associated with a stream. It should give the currently-recommended OS version, image ID, and artifact URL for the stream, separately for each platform (and region, where appropriate). Most of the time, stream metadata will just be a refactored version of the release metadata for the current release on the stream. However, operational issues may require divergence from the current release, such as:
Stream metadata might also include the target OS version for updates, to be consumed by the Cincinnati graph builder (#83). Operational issues may sometimes require this to be different from the target version for new installs.
Tooling should perform on-demand sync of stream metadata from current release metadata, subject to overrides specified via explicit configuration. Those overrides should allow replacing certain stream metadata elements with corresponding elements from a specified older release, or from manually specified metadata.
Stream metadata for each production stream should be available at a well-known HTTP endpoint, perhaps in the form of a static JSON document generated by a site builder. Perhaps it should be signed. Metadata history should be recorded, perhaps in a Git repo.
cc @cgwalters
The text was updated successfully, but these errors were encountered: