Nicer support for pinning install to a specific version

[ashak]

Feature Request

Have a nicer way to specify which version of FCOS to install so that pinning nodes to a specific version is easier.

Desired Feature

Coming from Container Linux I expected to be able to tell coreos-installer that I want to install a specific version from the stable stream.

Ideally I would simply say --stream stable --version 31.20200310.3.0 and there would be an additional karg, something like coreos.inst.version that I could use during PXE boot to get this passed through to coreos-installer.

Other Information

Whilst this seems to be possible via --image-url (or by using the coreos.inst.image_url karg), I had to ask on IRC how to do it and what I should be providing for the image-url as it was not 100% obvious what this should be. It's also prone to error (don't ask how I know that).

Rebuilding or installing a new coreos node is a reasonably common thing for me. Whilst in an ideal world I would simply rebuild is using the stable stream, it's certainly more preferable to me to be able to choose the newest version at a time that's more suitable to me rather than have it forced on me whilst i'm likely in the middle of some other task.

From IRC (so that the info isn't lost):

< kaeso[m]> ashak: I'm a bit torn on having install --version as we generally want users to stay on top of the stream
< kaeso[m]> ashak: however I think we could at least improve tooling to make it easier to know the URL for image pinning. From the CLI, similar to list-stream

[lucab]

The one quoted above was my instinctive reaction on IRC. My main concern is not making it too tempting to just install any arbitrary old OS image.
A flow like "tell me the current URL so that I can pin+download+install" sounds slightly better to me.

Leaving some room for @bgilbert and @jlebon reactions though.

[bgilbert]

The Fedora CoreOS stream metadata, which coreos-installer uses to find the image to install, is designed to provide the current recommended image on each stream and does not record historical information. There is no supported mechanism for obtaining artifact URLs for an old release.

The reasoning is that old OS releases are not maintained, may have security issues, and should not be used. That was also true on Container Linux, but users tended to pin old releases anyway. So we've tried to emphasize a flow where users always pull the latest recommended image and don't have to think about OS versions at all.

As an alternative to manually caching the URL, you could coreos-installer download the artifact and then host it on your own web server. I'm not completely opposed to a subcommand that dumps out an artifact URL from stream metadata, but that seems a bit low-level for coreos-installer.

[ashak]

I think anyone wanting to do this (and I can't believe i'm the only one) understands that the older releases aren't maintained and may have security issues but are making a decision to prioritise the stability and reproducibility of their systems over these things.

I also understand the idea of wanting users to simply track the latest release, for people who can handle that it must be great. But I don't want to do that for the reason above. There's no guarantee that a new release, even in the stable branch, isn't going to break for some reason on my systems. For me that's a much greater risk than running a few week old software that may have some security issues that aren't of a concern in my environment.

I certainly wouldn't want to be in a position where i'm in the middle of some task that requires me to rebuild some of my nodes only to then get distracted by having to work out why my nodes are coming back broken and then have to spend time fixing that instead of whatever it is i'm supposed to be doing. I think from what you're saying that I also wouldn't have some sort of roll back option in that case because the stream metadata would now only link to the newest version which is now broken for me (unless of course i'm already caching a copy of the URL of the older version). Which seems like a disaster waiting to happen.

I also understand keeping on top of security issues. For me that's handled by looking at what's in any new release and then prioritising upgrade to the newer version should there be any reason that we believe that affects our system (such as security issues) that's more important than other things.

[jlebon]

I think this would be fixed by #187 and crucially #197. IOW, once those are in, you can use the same boot media over and over to install the exact same FCOS version matching the boot media itself.

[lucab]

@ashak couple of followup questions:

• how are you running coreos-installer? Via FCOS live-ISO or live-PXE, via a container somewhere, or directly the binary?
• how do you plan to consumer the pinned image (plus signature)? Do you intend to download it all the time from our buckets, or were you planning to mirror it locally?

[ashak]

@lucab

• Live PXE with kargs to auto install to disk
• Right now my systems just grab it off the internet everytime just like the did with Container Linux, i've not thought about how that will work with pinned FCOS versions. I don't know if the old builds stay around for me to continue that or whether i'd need to grab them and keep them locally (which is something we'd planned to do with CL at some point and presumably we'd also do with FCOS too once we get around to it, unless of course we need to do it sooner due to old builds being cleaned up).

[ashak]

@jlebon That certainly looks like it could work for us since we already have pxe kernel/initramfs locally for the version that we're running. I hadn't noticed the issues you mention, so was unaware of this.

[lucab]

@jlebon is your plan to have osmet covering the live-PXE case too?

@bgilbert I had in mind such a verb for URL dumping, parametrized on stream+arch+platform+format. The alternative is telling people to jq on the stream metadata with a path filter, I guess.

[jlebon]

@jlebon is your plan to have osmet covering the live-PXE case too?

Yup, the osmet file will be in the same CPIO as the root squashfs (see coreos/coreos-assembler#1244 and coreos/fedora-coreos-config#322).

[dustymabe]

#187 and #197 are merged now and will be in the next release of coreos-installer. Then FCOS testing and FCOS stable.

[lucab]

I think I've found another example of this at coreos/fedora-coreos-docs#168.

So far the scope of this verb would be to help pinning references to:

• PXE artifacts (URL)
• DigitalOcean images (URL)
• AWS images (AMI-ID / ARN)

[bgilbert]

@ashak In current releases of FCOS, you can grab the URLs for the PXE artifacts (note that there are now three artifacts: kernel, initramfs, and rootfs) and either 1) cache the artifacts locally, or 2) download from those URLs every time. (Old releases are not currently garbage-collected; see coreos/fedora-coreos-tracker#99.) If you install from the PXE image and do not specify a stream/image-url/image-file, coreos-installer will automatically install the same release as the booted image.

[ashak]

I think not specifying a stream / image-url / image-file and getting the same version installed as you have booted from is entirely fine and would work for my scenario. I don't think this was the case when I opened this issue? In which version did this become the case?

Since I already cache the PXE artifacts locally it seems likely that i'm now already in a state where rebuilding my systems and booting them from the same PXE artifacts is achieving what I needed

[bgilbert]

@ashak That functionality reached the stable stream in 31.20200505.3.0, after you filed your issue.

[jlebon]

Cool, let's close this then?

So far the scope of this verb would be to help pinning references to:

With the bare metal workflow resolved as discussed above, is there a need for an explicit verb still? ISTM like it's understood users can pin to specific AMIs/URLs for some time (at least until they're GC'ed; we should firm up our policy there and make that public).

[ashak]

@bgilbert nice one thanks.
@jlebon Yes, closing, this solved my problem