Registries should get their list of images to use in plugins/devfiles/sidecars from the Che operator's CSV's RELATED_IMAGE_* env vars

[nickboldt]

Is your enhancement related to a problem? Please describe.

Today, we have to generate digests in Che and CRW registries, which means for every sidecar, machine exec, or theia respin, we have to rebuild the registries that refer to digests in order to stay current. THEN we have to rebuild the operator-metadata too, and there's a chance that things can get misaligned, such that the digest in the registry != the digest in the metadata container's CSV. (This happens pretty often in CRW when we're having outages and the build flow breaks down.)

Describe the solution you'd like

Rather than hardcoding image references in the registries, which need to be overwritten by sed scripts at build time, and which need to be kept current every time a referenced image changes, could we:

• use RELATED_IMAGE_image_name_here variables in the registries, then
• at runtime, resolve those env var variables with their actual values from the operator deployment pod

This would also centralize the information into a single place, so there's no risk of image refs in plugin reg or devfile reg getting out of sync.

And with the upcoming devfile 2.0 convention, where there's a devfile.yaml in the che-theia plugin's latest/ folder, we would also avoid having THAT file be out of date relative other images.

Describe alternatives you've considered

I can't think of a down side to this plan, other than that running a devfile or plugin registry locally will be slightly more challenging as you'd have to also have an operator + metadata deployment in your cluster. Gone would be the days of being able to run a registry container by itself, outside a cluster. (Does that break any existing tests?)

Also impacted would be the script that generates a list of external image references, which I believe we started doing not for any API or data flow reasons, but so that it was easier to audit if the image references were the same in all three containers where we need them (devfile reg, plugin reg, and operator-metadata). So we would either...

• update this script to resolve RELATED_IMAGE_* env vars to their actual URLs,
• maybe generate a json file, mapping the env var to its tag/digest real values, like this, or
• remove them entirely from the registries.

Additional context

See also:

• Kubernetes Image Puller (KIP) should get its list of images to pull from the Che operator's CSV's RELATED_IMAGE_* Automatically configure Kubernetes Image Puller with the list of images used by Che #19657
• Expose DWO and DWCO images in CSV as RELATED_IMAGE_* env vars (like all the other images deployed by the Che Operator) Expose DWO and DWCO images in CSV as RELATED_IMAGE_* env vars (like all the other images deployed by the Che Operator) #19667
• Verify that registries are able to use RELATED_IMAGE_* values in operator-metatada image CRW-1157

For downstream, this would also allow Freshmaker to be able to rebuild any (or all) CRW image for CVE fixes, then simply respin a new operator-metadata container to provide updated digests.

[nickboldt]

While this is tagged as kind/enhancement it will also allow us to benefit downstream by having the ability to get automated respins for CVE fixes.

Once https://projects.engineering.redhat.com/browse/CLOUDWF-3111 is live and we re-enable it, we can get respins of any/all containers in CRW, with an updated operator metadata + CSV + IIB, but the registries will still refer to old digests, which would result in a partial respin scenario... new plugin-java11 sidecar in the CSV, but old one ref'd by registries, for example.

In that case, airgap might be broken if image puller is pulling the newer images based on the CSV list of related images, but the old ones are referenced in the registries.

So... now that the CLOUDWF fix is live, if we want to enable this, fixing this issue currently blocks using Freshmaker for CRW.

[nickboldt]

Due to upcoming infra changes reported in https://projects.engineering.redhat.com/browse/CLOUDBLD-6237 we need to escalate this to highest priority for the next CRW release. I've set P1 because there's no P0, but if you want to move it to blocker, I'm +1 on that.

[ericwill]

Given that the downstream registries are a fork or upstream, do we need this logic in upstream?

[nickboldt]

I believe this is done as part of CRW-1157.

[svor]

In downstream it was done as part of https://issues.redhat.com/browse/CRW-1157 and as I know for upstream it was done by Deploy team, @tolusha correct me if I'm wrong.

I think we can clothe this one