Add "Build OKD-on-Fedora-CoreOS in Prow" proposal

[LorbusChris]

OKD is the OpenShift community distribution of Kubernetes.

As part of the version 4 effort, the OKD working group has decided to target Fedora CoreOS (FCOS) as the primary base operating system for OKD control plane and worker nodes.

This document contains a proposal to add missing FCOS support to key components (machine-config-operator and installer),
create OKD artifacts in OpenShift's Prow instance from the canonical OpenShift master and release branches alongside OCP CI artifacts,
and regularily create OKD release payloads from there.

Rendered

[LorbusChris]

I've updated the proposal to reflect the current status and the work left to get us to OKD 4 GA.

/cc @smarterclayton @miabbott @ashcrow @runcom @darkmuggle @vrutkovs @sdodson @crawford @abhinavdahiya @cgwalters

[vrutkovs]

I'd prefer the build-time flag, similar to libvirt approach:

• Rework existing installer to have all ignition operators in a single file (Abstract Ignition operations in a single file installer#3248)
• Create Ignition 3 implementation of the same functions, hidden under okd build flag
  WIP at https://github.com/vrutkovs/installer/commits/fcos-4.5
• Build installer:okd image with this flag enabled
• Add isOKD internal setting and conditionally enable okd-specific features:
  • use fcos.json instead of rhcos.json
  • allow empty pull secret
  • restrict to community operators
• Enable libvirt provider in OKD installer

[LorbusChris]

+1 from me. @abhinavdahiya, WDYT?

[smarterclayton]

Why not rename rhcos.json to machine-os.json and make that the build time replacement instead?

Branches on “is okd” raise red flags for me - the imply OKD isn’t the same source.

The restriction to community operators can be handled dynamically in many cases (does the payload include the operator).

The pull secret is going to be relevant for some deployments no matter what - perhaps instead of making the check conditional we instead check whether the pull of the release image requires auth as a gate during pre validation (which has the side benefit of verifying the credentials)

[cgwalters]

Bikeshed: bootimage.json

[vrutkovs]

Why not rename rhcos.json to machine-os.json and make that the build time replacement instead?

Sounds good to me

Branches on “is okd” raise red flags for me - the imply OKD isn’t the same source.

We still need to branch on few options here - i.e. default SDN, Ignition v3 etc. IsOKD is not exposed externally, cannot be changed via config and being set on build time

The restriction to community operators can be handled dynamically in many cases (does the payload include the operator).

So instead of installer injecting operators we'd have to have a setting other operators would read?

we instead check whether the pull of the release image requires auth as a gate during pre validation

This check would happen in the installer, right?

[LorbusChris]

Are there any blockers to this?

[vrutkovs]

Looks good, found few typos

[LorbusChris]

@staebler I've added you to the reviewers from the Installer side.

Tl;dr the largest remaining piece of this is openshift/installer#4453.

[mjturek]

I'm interested in building for other architectures. I'm tasked with investigating OKD support for ppc64le.

After a conversation @vrutkovs and @LorbusChris, it seems the goal would be to do nightly builds of OKD on ppc64le in cadence with x86_64. This means we would need infrastructure where we could build these containers, and a release controller.

Can others weigh in on where we may be able to do this? Any input is welcome and appreciated!

[cgwalters]

I think we'd want to ask either Fedora infra or CentOS infra about capacity for this - that's where the community ppc64le/s390x hardware is as far as I know.

[derekwaynecarr]

the enhancement should be a statement of intent or direction, and so holding this out while the installer change is reviewed seems unnecessary. are there any further updates required, or we good to merge and iterate?

[LorbusChris]

@derekwaynecarr I think this is good to merge.

Setting up OKD builds on alt arches is not part of this proposal, and can be done in a separate one if needed.

[russellb]

Merging based on the last comments from @derekwaynecarr and @LorbusChris

/approve
/lgtm

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: russellb, vrutkovs

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [russellb]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[vrutkovs]

/refresh
/retest

[vrutkovs]

/hold

Needs a few trailing spaces removed

[openshift-bot]

/retest

Please review the full test history for this PR and help us cut down flakes.

[LorbusChris]

I've fixed the linter issues.

/hold cancel

[russellb]

/lgtm

[darkmuggle]

This enhancement was not approved and it's not implementable. Stakeholders (Installer, MCO, CoreOS, among others) have not agreed, nor have their reviews been obtained.

[russellb]

This enhancement was not approved and it's not implementable. Stakeholders (Installer, MCO, CoreOS, among others) have not agreed, nor have their reviews been obtained.

This was merged primarily as a starting point to document the current reality (OKD does already exist ...) and make it easier to iterate on the doc. You should not interpret this as any big new decision. I've spoken with @vrutkovs about some updates needed here to better capture status and open questions and have asked him to tag you on the update.