Chectl function to fetch the list of needed images and download them (performance tuning, OCP 3.11 airgap)

[MichalMaler]

Is your enhancement related to a problem? Please describe.

Right now we have a table that should list the recommend list of images with digest for image puller to pre-pull:
https://www.eclipse.org/che/docs/che-7/administration-guide/caching-images-for-faster-workspace-start/#deploying-image-puller-using-openshift-templates_caching-images-for-faster-workspace-start

This table is almost identical to the table of images needed for an installation in restricted environments.
So, the problem is to, from the doc perspective, maintain two almost identical tables in two files that are also used in downstream, but with different images.

1. These tables are frequently updated and their keeping up is problematic.

2. The whole section is complex and instead of having multiple procedures about how to get those images pulled (where some are pulled by imagepuller, some have to be downloaded by skopeo), it would be better to have this "in code".
   https://www.eclipse.org/che/docs/che-7/administration-guide/caching-images-for-faster-workspace-start/

Describe the solution you'd like

I would like to request a script that would fetch the list of needed images, download them, provide the list of remaining images from which a user can pick and download, based on a user's use-case.

This script would be a new feature in chectl. It would be great if it could have two options:
a) download images for faster caching
b) download images for use in restricted environments

Describe alternatives you've considered

Doc alternative:

1. Reduce Table 3. List of recommended images to pre-pull to list ONLY the 2 brokers and Che-Theia.

• add attributes for versions to keep this up to date. Latest brokers can be found from either https://github.com/eclipse/che-plugin-broker/tags

• or by checking tags on https://quay.io/repository/eclipse/che-plugin-metadata-broker?tab=tags

• tags for theia can be found here: https://github.com/eclipse/che-theia/tags or https://quay.io/repository/eclipse/che-theia?tab=tags

2. add a section about curling the whole list of images (with a note that some might not be pullable this way, like machine-exec)

3. Add a note that if the image puller can't pull images and keep them alive (no sleep command, etc.), then the backup process is to simply use skopeo copy --all to mirror images into your cluster's image registry; this is ALSO how to do airgap setup for OCP 3.11, so might be able to reuse content/link to other section

Additional context

https://issues.redhat.com/browse/RHDEVDOCS-2289

[MichalMaler]

@themr0c @nickboldt @rhopp

[nickboldt]

Not sure how these are different:

a) download images for faster caching
b) download images for use in restricted environments

Is the intent that the new chectl server:images command (I'm guessing that this new command would go under server since it's related to deployments and server administration):

• would read the CSV data embedded in chectl (or pull from an external URL provided via --starting-csv=starting-csv, the same flag used with server:start), then
• skopeo copy --all those images into the connected OCP cluster's image registry, using the SAME digest names (and maybe optionally could use flags instead?)

If so, then the cached images would be effective for both online/offline & airgapped/non-airgapped environments. So (a) and (b) are the same solution for different needs.

Note too that for OCP 4.3+, this functionality is already provided by OCP itself via OLM, so this is really only a requirement for OCP 3.11 and maybe for k8s deployments (?).

Or if there's a way to make triggering the OLM mirroring easier via chectl, then perhaps we would want two paths inside chectl server:images - one that calls the OLM API and one that does skopeo copy --all for OCP 3.11.

WDYT @tolusha ?

[MichalMaler]

Yeah, that's right, Nick. I am not a PRO, so that's why I tagged you :) Fabrice is the author of this idea, btw, so all the glory and fame -> @themr0c :)

[themr0c]

@nickboldt I believe that would be really a good point to have one single command able to fetch images for both user stories.

[themr0c]

• New features in the Operator will solve the problem for image puller.

[ibuziuk]

in terms of the image-puller the following user-story #18133 should tackle the case when the image puller is installed as an optional dependecy of the che-operator and automatically detect the default list of image for pre-pulling

[nickboldt]

After discussion in Che community call, better clarity between these two use cases:

a) download images for faster caching
b) download images for use in restricted environments

(a) in this scenario, the point is NOT ONLY to pull images into the cluster's registry, but to also spread those to all nodes in the cluster so that startup of workspaces is faster; eg., if your users are all python and java devs, you'd tell image puller to pull those two sidecar images

(b) in this scenario, the point is to pull images into the cluster's registry, since you don't have access to the outside world. If you THEN want to use image puller to spread some of them around the cluster nodes, you can then do so.

So, the need to have docs explaining the two different scenarios persists, where (b) is a superset of images used in (a).

Having a tool / script to fetch the complete list of images is still useful for both scenarios, but since image puller is likely to be used for only a subset of sidecar images, perhaps we want:

• section of doc explains how to get images for airgap installation on OCP 3.11
• section of doc explains how to use image puller for sidecar images AND LINKS TO the above section (so the image list / script is not in two places and is more easily maintained)

[themr0c]

Airgap: on OCP 4 also, when using the CLI.

[che-bot]

Issues go stale after 180 days of inactivity. lifecycle/stale issues rot after an additional 7 days of inactivity and eventually close.

Mark the issue as fresh with /remove-lifecycle stale in a new comment.

If this issue is safe to close now please do so.

Moderators: Add lifecycle/frozen label to avoid stale mode.