Document image build process

[ryanj]

Document the image build process (#88)

CRC tarball -> crc binary -> crc.qcow2 -> disk.raw -> crc.tar.gz -> GCP Image (build) -> GCP Image (GUID) -> GCP instance (launched via instruqt) -> GCP instance (READY)

Solution Outline:

GCP setup

1. Log in to GCP and create a new project for this work
2. Install the gcloud command line tools: https://cloud.google.com/sdk/docs/install
3. Log in to GCP via the command line: gcloud auth login
4. Set your default project scope: gcloud config set project rhd-devx-instruqt

CRC VM image extraction

Image Extraction workflow (tested on Fedora35):

1. Install RH VPN services: https://redhat.service-now.com/help?id=kb_article_view&sys_kb_id=04168d681b830990839e32a3cc4bcbb9
2. Download the latest CRC binary from the cdk build server (VPN required): http://cdk-builds.usersys.redhat.com/builds/crc/releases/
3. Extract the CRC installation tarball tar xvf ~/Downloads/crc-linux-amd64.tar.xz
4. Establish a $PATH-accessible folder for your local binaries: mkdir -p ~/.local/bin
5. Export CRC_RELEASE export CRC_RELEASE=2.10.1 and OS_RELEASE export OPENSHIFT_RELEASE=4.11.7
6. Move the CRC executable to a PATH accessible location: mv ~/Downloads/crc-linux-${CRC_RELEASE}-amd64/crc ~/.local/bin
7. Run crc setup to download the latest CRC qcow2 VM image to your local machine
8. The extracted VM disk image should be visible here: cd ~/.crc/cache/crc_libvirt_${OPENSHIFT_RELEASE}_amd64/ && ls -alh
9. Convert the image to RAW format: qemu-img convert -O raw ~/.crc/cache/crc_libvirt_${OPENSHIFT_RELEASE}_amd64/crc.qcow2 ~/.crc/cache/crc_libvirt_${OPENSHIFT_RELEASE}_amd64/disk.raw
10. Tar the raw disk image tar cvf crc.tar disk.raw
11. Compress the tarball: gzip crc.tar
12. Check to make sure the tarball includes a single file named disk.raw: tar tfz crc.tar.gz

Import the raw CRC disk image to GCP

1. Try listing images from the command line. This step should give you an opportunity to enable the Compute API for your project: gcloud compute images list --no-standard-images | grep openshift.

2. export CRC_DASH_RELEASE=2-10-1

3. export OPENSHIFT_DASH_RELEASE=4-11-7

4. Use Google Cloud Storage to create a new storage bucket for the raw image: gsutil mb -l europe-west1 gs://openshift-${OPENSHIFT_DASH_RELEASE}

5. Upload the raw image to the new cloud storage bucket gsutil cp ~/.crc/cache/crc_libvirt_${OPENSHIFT_RELEASE}_amd64/crc.tar.gz gs://openshift-${OPENSHIFT_DASH_RELEASE}

6. Review what you've uploaded to the bucket: gcloud alpha storage ls gs://openshift-${OPENSHIFT_DASH_RELEASE}

7. After the upload has completed, create a new Image in region europe-west1 by selecting it from the bucket gs://openshift-${OPENSHIFT_DASH_RELEASE}: gcloud compute images create --storage-location europe-west1 openshift-${OPENSHIFT_DASH_RELEASE}-disk --source-uri gs://openshift-${OPENSHIFT_DASH_RELEASE}/crc.tar.gz

8. Confirm that the new image is available in GCP: gcloud compute images list --no-standard-images | grep openshift

GCP Image Build

Establish a build script and automation for converting the image ending with "-disk" into a pre-booted image with a name that ends with "-GUID" (use GCP cloud build, packer, or ssh?). The "-GUID" version will be booted by instruqt

1. Launch a GCE instace using the new image: gcloud compute instances create openshift-${OPENSHIFT_DASH_RELEASE}-build --image openshift-${OPENSHIFT_DASH_RELEASE}-disk --zone=europe-west1-c --machine-type n1-standard-8
2. ssh -i ~/.crc/cache/crc_libvirt_${OPENSHIFT_RELEASE}_amd64/id_ecdsa_crc core@EXTERNAL_IP
3. sudo su - then run the following script: https://github.com/openshift-labs/learn-instruqt/blob/main/scripts/build/setup.sh
4. systemctl reset-failed
5. exit
6. Add pull secret to openshift-config namespace:
   https://github.com/openshift-labs/learn-instruqt/blob/main/scripts/pullsecret.json
7. sudo chown core.core /var/lib/kubelet/config.json
8. https://docs.google.com/document/d/1Iz2nAfhz4_G0W1L50CelDckdXjx6WByD4D4lRE4eEko/edit
9. export KUBECONFIG=/opt/kubeconfig
10. oc delete secret pull-secret -n openshift-config && oc create secret generic pull-secret --from-file=.dockerconfigjson=/var/lib/kubelet/config.json --type=kubernetes.io/dockerconfigjson -n openshift-config
11. Add admin, developer, and user1 accounts: https://github.com/openshift-instruqt/instruqt/pull/117/files#diff-e438bfd9dad2f72ff1626b329d6d466d2afc779b8209f3866498f64fd1742dc0R42-R68
12. history -c
13. sudo shutdown -h now
14. Wait for the VM to shut down, then generate a new image from the updated VM disk gcloud compute images create openshift-${OPENSHIFT_DASH_RELEASE}-GUID --project=rhd-devx-instruqt --source-disk=openshift-${OPENSHIFT_DASH_RELEASE}-build --source-disk-zone=europe-west1-c --storage-location=europe-west1

Optimization goals

• Update SSL Need to enable LE certificate for OpenShift Web Console #42
  • install zeroSSL
  • https://cert-manager.io/docs/configuration/acme/http01/#podtemplate
• Automate the build process

Client Image Build

• include developer deps: git, oc, kubectl, npm, etc...
  • see https://github.com/openshift-instruqt/instruqt/tree/master/scripts

Image startup optimizations

• update machine size
• Include client image
• Establish a method for testing the GCP images using Instruqt (see https://play.instruqt.com/openshift/tracks/getting-started-4-10)

[ryanj]

^ these notes should be moved to a markdown doc somewhere in the repo

[blues-man]

@ryanj I was able to keep the same hostname while recreating the new image like this:

hostnamectl set-hostname --static crc-gh9wd-master-0
hostnamectl set-hostname --transient crc-gh9wd-master-0

@ryanj actually the hostname then mismatches with kubelet:

[root@crc-gh9wd-master-0 ~]# oc get nodes
NAME                 STATUS     ROLES           AGE   VERSION
crc-5nvrm-master-0   NotReady   master,worker   11d   v1.23.5+3afdacb

[praveenkumar]

About CRC VM image extraction step I think you don't need the vpn to download the bundle because after crc release it is publicly available and if you want some different bundle then only use vpn and get the linux bundle from internal server http://cdk-builds.usersys.redhat.com/builds/crc/bundles/openshift/ location.

Downloading bundle for a latest crc release should be simply.

$ openshiftVersion=$(curl -sL https://mirror.openshift.com/pub/openshift-v4/clients/crc/latest/release-info.json | jq -r .version.openshiftVersion)
$ curl -L -O https://mirror.openshift.com/pub/openshift-v4/clients/crc/bundles/openshift/"${openshiftVersion}"/crc_libvirt_"${openshiftVersion}"_amd64.crcbundle