rhcos: implement image discovery for new pipeline #554

crawford · 2018-10-26T23:41:22Z

RHCOS has moved over to the 2.0 pipeline which has a nicer discovery
mechanism for images. This uses that new pipeline to fetch the latest
AMIs and QCOW images for the latest build. Unfortunately, there is no
"latest" alias anymore, so the installer has to fetch resources from the
Internet before it can prompt the user. This will go away once the
installer starts pinning to a specific RHCOS release.

crawford · 2018-10-26T23:48:49Z

2018/10/26 23:43:40 error: unable to signal to artifacts container to terminate in pod release-latest, triggering deletion: could not run remote command: unable to upgrade connection: container not found ("artifacts")
2018/10/26 23:43:40 error: unable to retrieve artifacts from pod release-latest: could not read gzipped artifacts: unable to upgrade connection: container not found ("artifacts")
2018/10/26 23:43:46 Ran for 1m48s
error: could not run steps: failed to wait for release pod to complete: could not wait for pod completion: pod release-latest was already deleted

I'm not sure what happened there...

/retest

crawford · 2018-10-27T01:11:00Z

/cc @cgwalters

wking · 2018-10-27T05:35:33Z

pkg/asset/installconfig/platform.go

@@ -288,13 +289,24 @@ func (a *platform) libvirtPlatform() (*types.LibvirtPlatform, error) {
 		return nil, err
 	}

+	// TODO: Ideally, this would live inside of a closure which is passed to
+	//       asset.GenerateUserProvidedAsset and only called if the environment
+	//       variable isn't present. As this exists, it ruins the abstraction.


This sort of problem goes away with #556, where all dependencies and defaults are only rendered as needed 🎉

wking · 2018-10-27T06:07:19Z

pkg/rhcos/ami.go

 // AMI calculates a Red Hat CoreOS AMI.
+// TODO: Use the new RHCOS build pipeline to discover AMIs. The pipeline isn't
+//       currently exposed publicly and CI isn't on the VPN, so this cannot be
+//       updated without breaking the tests.


This is only workable if we have the same internal and external release cadence. See discussion in #281 and openshift/os#353.

Right. I will look at making more of the pipeline public. There are some prerequisites for this.

How hard would it be to put some of this data in the pull secret though or so?

How hard would it be to put some of this data in the pull secret though or so?

What data? A URI with an initial entry-point for this discovery?

What data? A URI with an initial entry-point for this discovery?

I'd say all the release metadata - including the meta.json fetched by this PR, which includes AMI ids, along with link to qcow2 etc.

Hmm. Or maybe the pull secret grants access to a server running on api.ci or wherever that hands out the metadata.

Well, one thing this circles back to is...we could stick all of the relevant metadata in a container, which the pull secret already grants access to of course.

Would having this be in a container be ergonomic for the installer? It'd depend on oc adm release extract?

This all sounds far too complex for what we are after. A few notes on the discussion:

We shouldn't depend on the URL remaining unknown in order to protect the RHCOS images. Hiding this in a container behind a pull-secret doesn't buy much.

The RHCOS image reference should come from the CVO manifest payload (just like all of the other resources). We need to wait until MAO can create masters before we can make this jump since I don't want the installer parsing through the CVO manifest payload.

A lot of this is temporary scaffolding, so I'd rather we focus the discussion on the longer-term vision rather than this code which we know we are going to throw away in a few weeks.

The RHCOS image reference should come from the CVO manifest payload (just like all of the other resources). We need to wait until MAO can create masters before we can make this jump since I don't want the installer parsing through the CVO manifest payload.

What approach would you use for the always-pre-MAO bootstrap (which needs an AMI too ;)?

We shouldn't depend on the URL remaining unknown in order to protect the RHCOS images. Hiding this in a container behind a pull-secret doesn't buy much.

True long term.

The RHCOS image reference should come from the CVO manifest payload

Agree. What's fuzzy for me is the dependencies required for that. It also forces some deep issues like...when we branch OKD, are we going to try to maintain a separate OS stream for that? Or do we just keep the same OS and ship updated kubelets too? Or a mix where we ship the same "base OS", and just pull different kubelets? We've talked about this in the past, and what hurts my head is that the complexity jumps up when one notes we have two major version streams for RHCOS today.

Could you guys confirm, CVO manifest payload = origin-v4.0:release image?

wking · 2018-10-27T06:15:48Z

I think the new pipeline also lets us replace this hack with Content-Encoding (discussed in 7abe94d, #280).

pkg/rhcos/qemu.go

pkg/rhcos/builds.go

ashcrow · 2018-10-30T16:04:53Z

Needs a rebase

RHCOS has moved over to the 2.0 pipeline which has a nicer discovery mechanism for images. This uses that new pipeline to fetch the latest QCOW images for the latest build. Unfortunately, there is no "latest" alias anymore, so the installer has to fetch resources from the Internet before it can prompt the user. This will go away once the installer starts pinning to a specific RHCOS release.

wking · 2018-10-31T18:11:05Z

pkg/rhcos/ami.go

-		if nextImage.ImageId == nil || nextImage.CreationDate == nil {
-			continue
+	for _, ami := range meta.AMIs {
+		if ami.Name == region {


Do we want to land this before the AMIs are being pushed to other regions?

$ BUILD=$(curl -sL https://releases-rhcos.svc.ci.openshift.org/storage/releases/maipo/builds.json | jq -r '.builds[0]') $ echo "${BUILD}" 47.38 $ curl -sL "https://releases-rhcos.svc.ci.openshift.org/storage/releases/maipo/${BUILD}/meta.json" | jq .amis [ { "name": "us-east-1", "hvm": "ami-03afbcc8b72560ac8" } ]

I'd rather stick to scraping AWS until we get that, because scraping AWS has a chance at working in other regions if folks can copy the AMI over to the target region. That would allow us to use the other regions we're trying to use for openshift-dev.

We can add an environment variable for the AMI. We already have the override for the QEMU image.

We can add an environment variable for the AMI. We already have the override for the QEMU image.

I'm ok with that, although see this comment about not having AMI/image knobs and assuming that there's a default AMI/image that is (1) available and (2) healthy enough to pivot.

We're close to landing the multi-region again although this is going to slow our pipeline notably. There are two things we need to do; first on the RHCOS side, implement tags, and only upload to all the regions for tags (e.g. once a day at most?).

Second though we need to do openshift/os#289 or possibly teach the installer to take a set of override RPMs and apply them via Ignition? Then I think we can slow the cadence of "promoted" builds, and anyone who wants to test bleeding edge OS work can use libvirt.

abhinavdahiya · 2018-10-31T21:01:34Z

/lgtm

openshift-ci-robot · 2018-10-31T21:01:43Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: abhinavdahiya, crawford

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [abhinavdahiya,crawford]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

wking · 2018-10-31T22:09:03Z

Do we want to land an environment variable override? Or are we ok re-breaking @ironcladlou and others who work in other regions until we get the multi-region pipeline going?

cgwalters · 2018-11-01T12:52:41Z

An env override makes sense to me, but it'd be even more powerful to have an override for the URL; then one could use it to override every format. The only slight downside to this would be that one would have to hand-craft a meta.json but that's not hard.

Keep the environment variable (with a warning about using it), but drop the interactive prompt. The default is solid, and users manipulating it are more likely to break something (e.g. by continuing to use the old v1 pipeline), while the installer can update it's default to track the RHCOS folks (e.g. like 1117821, rhcos: implement image discovery for new pipeline, 2018-10-26, openshift#554).

Keep the environment variable (with a warning about using it), but drop the interactive prompt. The default is solid, and users manipulating it are more likely to break something (e.g. by continuing to use the old v1 pipeline), while the installer can update its default to track the RHCOS folks (e.g. like 1117821, rhcos: implement image discovery for new pipeline, 2018-10-26, openshift#554).

openshift-ci-robot requested review from aaronlevy and staebler October 26, 2018 23:41

openshift-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Oct 26, 2018

openshift-ci-robot requested a review from cgwalters October 27, 2018 01:11

wking reviewed Oct 27, 2018

View reviewed changes

wking mentioned this pull request Oct 27, 2018

Libvirt release channels? openshift/os#354

Closed

wking reviewed Oct 27, 2018

View reviewed changes

openshift-ci-robot requested a review from smarterclayton October 29, 2018 18:33

wking reviewed Oct 30, 2018

View reviewed changes

pkg/rhcos/qemu.go Outdated Show resolved Hide resolved

wking reviewed Oct 30, 2018

View reviewed changes

pkg/rhcos/builds.go Outdated Show resolved Hide resolved

abhinavdahiya mentioned this pull request Oct 30, 2018

deploy etcd as static pod openshift/machine-config-operator#148

Merged

cgwalters mentioned this pull request Oct 30, 2018

pkg/rhcos/ami: Update name filter to "redhat-coreos-*" #572

Merged

wking reviewed Oct 31, 2018

View reviewed changes

openshift-ci-robot assigned abhinavdahiya Oct 31, 2018

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Oct 31, 2018

openshift-merge-robot merged commit b4fe4d2 into openshift:master Oct 31, 2018

crawford deleted the os branch October 31, 2018 21:49

wking mentioned this pull request Nov 7, 2018

pkg/asset/installconfig/platform: Remove libvirt image prompt #637

Merged

wking mentioned this pull request Nov 16, 2018

libvirt: only cache images in libvirt storage pool #687

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rhcos: implement image discovery for new pipeline #554

rhcos: implement image discovery for new pipeline #554

crawford commented Oct 26, 2018

crawford commented Oct 26, 2018

crawford commented Oct 27, 2018

wking Oct 27, 2018

wking Oct 27, 2018 •

edited

Loading

cgwalters Oct 29, 2018

wking Oct 29, 2018

cgwalters Oct 29, 2018

cgwalters Oct 29, 2018

crawford Oct 30, 2018

wking Oct 30, 2018

cgwalters Oct 30, 2018

dgoodwin Nov 2, 2018

cgwalters Nov 2, 2018

wking commented Oct 27, 2018

ashcrow commented Oct 30, 2018

wking Oct 31, 2018 •

edited

Loading

crawford Oct 31, 2018

wking Oct 31, 2018

cgwalters Oct 31, 2018

abhinavdahiya commented Oct 31, 2018

openshift-ci-robot commented Oct 31, 2018

wking commented Oct 31, 2018

cgwalters commented Nov 1, 2018

rhcos: implement image discovery for new pipeline #554

rhcos: implement image discovery for new pipeline #554

Conversation

crawford commented Oct 26, 2018

crawford commented Oct 26, 2018

crawford commented Oct 27, 2018

Choose a reason for hiding this comment

wking Oct 27, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wking commented Oct 27, 2018

ashcrow commented Oct 30, 2018

wking Oct 31, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abhinavdahiya commented Oct 31, 2018

openshift-ci-robot commented Oct 31, 2018

wking commented Oct 31, 2018

cgwalters commented Nov 1, 2018

wking Oct 27, 2018 •

edited

Loading

wking Oct 31, 2018 •

edited

Loading