Chectl fails for "Eclipse Che pod bootstrap: Downloading Images -> "helm:latest"

[frankae]

Describe the bug

ISSUE in all of the below scenarios is the same. The "helm:latest" can't be pulled. Before Friday, 12th Feb 2021 that had worked. That was reported in different flavors at different times. Here something changed

While doing some tests to explore the opportunity to use che in our company I had made some installations on different systems with individuell setups for

• docker-desktop with

  • Hyper-V, and with
  • WSL2 (Ubuntu 20.04 LTS)

• Ubuntu 20.04 LTS on

  • bare Metall, and with
  • snap juju, and
  • snap microk8s,

both with --classic approach.
After that the procedure was the same for all tests.

1. Download chectl as described in the installation sections.

2. Update chectl to the latest version.

3. Deploy che-server via chectl server:deploy in the following variants
   ai. docker-desktop: chectl server:deploy -p docker-desktop -i helm
   aii. docker-desktop: chectl server:deploy -p docker-desktop -i helm -m
   bi. UBUNTU 20.04.1 LTS (snap microk8s): chectl server:deploy -p microk8s -i helm
   bii. UBUNTU 20.04.1 LTS (snap microk8s): chectl server:deploy -p microk8s -i helm -m
   ci. UBUNTU 20.04.1 LTS (snap juju, lxd-based charmed kubernetes):
   chectl server:deploy -p k8s -i helm
   cii. UBUNTU 20.04.1 LTS (snap juju, lxd-based charmed kubernetes):
   chectl server:deploy -p k8s -i helm -m

4. Output from chectl server:deploy (microk8s, which is the identical for all tests:
   chectl server:deploy -p microk8s -i helm
   › Current Kubernetes context: 'loaded-context'
   › Installer type is set to: 'operator'
   Enable CLI usage data to be sent to Red Hat online services. [y/n]: y
   ✔ Verify Kubernetes API...OK
   ✔ 👀 Looking for an already existing Eclipse Che instance
   ✔ Verify if Eclipse Che is deployed into namespace "eclipse-che"...it is not
   ✔ ✈️ MicroK8s preflight checklist
   ✔ Verify if kubectl is installed
   ✔ Verify if microk8s is installed
   ✔ Verify if microk8s is running
   ↓ Start microk8s [skipped]
   → MicroK8s is already running.
   ✔ Check Kubernetes version: Found v1.20.1-34+e7db93d188d0d1.
   ✔ Verify if microk8s ingress and storage addons is enabled
   ✔ Enable microk8s ingress addon
   ↓ Enable microk8s storage addon [skipped]
   → Storage addon is already enabled.
   ✔ Retrieving microk8s IP and domain for ingress URLs...172.31.3.62.nip.io.
   ↓ Check if cluster accessible [skipped]
   ✔ Following Eclipse Che logs
   ✔ Start following Operator logs...done
   ✔ Start following Eclipse Che server logs...done
   ✔ Start following Postgres logs...done
   ✔ Start following Keycloak logs...done
   ✔ Start following Plugin registry logs...done
   ✔ Start following Devfile registry logs...done
   ✔ Start following namespace events...done
   › Warning: Consider using the more reliable 'OLM' installer when deploying a stable
   › release of Eclipse Che (--installer=olm).
   ✔ 🏃 Running the Eclipse Che operator
   ✔ Create Namespace (eclipse-che)...Done.
   ✔ Create ServiceAccount che-operator in namespace eclipse-che...done.
   ✔ Read Roles and Bindings...done.
   ✔ Creating Roles and Bindings...done.
   ✔ Create CRD checlusters.org.eclipse.che...done.
   ✔ Waiting 5 seconds for the new Kubernetes resources to get flushed...done.
   ✔ Create deployment che-operator in namespace eclipse-che...done.
   ✔ Operator pod bootstrap
   ✔ Scheduling...done
   ✔ Downloading images...done
   ✔ Starting...done
   ✔ Prepare Eclipse Che cluster CR...Done.
   ✔ Create the Custom Resource of type checlusters.org.eclipse.che in the namespace ec
   lipse-che...done.
   ❯ ✅ Post installation checklist
   ✔ PostgreSQL pod bootstrap
   ✔ Scheduling...done
   ✔ Downloading images...done
   ✔ Starting...done
   ✔ Keycloak pod bootstrap
   ✔ Scheduling...done
   ✔ Downloading images...done
   ✔ Starting...done
   ✔ Devfile registry pod bootstrap
   ✔ Scheduling...done
   ✔ Downloading images...done
   ✔ Starting...done
   ✔ Plugin registry pod bootstrap
   ✔ Scheduling...done
   ✔ Downloading images...done
   ✔ Starting...done
   ❯ Eclipse Che pod bootstrap
   ✔ Scheduling...done
   ✖ Downloading images...failed
   → Failed to download image, reason: ImagePullBackOff, message: Back-off pulling
   …
   Starting
   Eclipse Che status check
   Retrieving Keycloak admin credentials
   Retrieving Che self-signed CA certificate
   Prepare post installation output
   › Error: Error: Failed to download image, reason: ImagePullBackOff, message:
   › Back-off pulling image "helm:latest".
   › Command server:deploy failed. Error log: /home/devfrank/.cache/chectl/error.log

5. Output from evfrank@ubu2004-che-mk:~$ kubectl describe -n eclipse-che pod che-8678f8f794-rs9v9 > /tmp/runbooks_describe_pod.txt

evfrank@ubu2004-che-mk:~$ more /tmp/runbooks_describe_pod.txt**
Name: che-8678f8f794-rs9v9
Namespace: eclipse-che
Priority: 0
Node: ubu2004-che-mk/172.31.3.62
Start Time: Mon, 15 Feb 2021 13:53:48 +0000
Labels: app=che
app.kubernetes.io/component=che
app.kubernetes.io/instance=che
app.kubernetes.io/managed-by=che-operator
app.kubernetes.io/name=che
component=che
pod-template-hash=8678f8f794
Annotations: cni.projectcalico.org/podIP: 10.1.25.87/32
cni.projectcalico.org/podIPs: 10.1.25.87/32
Status: Pending
IP: 10.1.25.87
IPs:
IP: 10.1.25.87
Controlled By: ReplicaSet/che-8678f8f794
Containers:
che:
Container ID:
Image: helm:latest
Image ID:
Ports: 8080/TCP, 8000/TCP, 8888/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP
State: Waiting
Reason: ImagePullBackOff
Ready: False
Restart Count: 0
Limits:
cpu: 1
memory: 1Gi
Requests:
cpu: 100m
memory: 512Mi
Liveness: http-get http://:8080/api/system/state delay=400s timeout=3s period=10s #success=1 #failure=3
Readiness: http-get http://:8080/api/system/state delay=25s timeout=5s period=10s #success=1 #failure=18
Environment Variables from:
che ConfigMap Optional: false
Environment:
CHE_SELF__SIGNED__CERT: <set to the key 'ca.crt' in secret 'self-signed-certificate'> Optional: true
CHE_GIT_SELF__SIGNED__CERT:
CHE_GIT_SELF__SIGNED__CERT__HOST:
CHE_KEYCLOAK_ADMIN__PASSWORD: <set to the key 'password' in secret 'che-identity-secret'> Optional: false
CHE_KEYCLOAK_ADMIN__USERNAME: <set to the key 'user' in secret 'che-identity-secret'> Optional: false
CM_REVISION: 8570,7998
KUBERNETES_NAMESPACE: eclipse-che (v1:metadata.namespace)
CHE_JDBC_USERNAME: <set to the key 'user' in secret 'che-postgres-secret'> Optional: false
CHE_JDBC_PASSWORD: <set to the key 'password' in secret 'che-postgres-secret'> Optional: false
Mounts:
/public-certs from che-public-certs (rw)
/var/run/secrets/kubernetes.io/serviceaccount from che-token-xzc8m (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
che-public-certs:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: ca-certs-merged
Optional: false
che-token-xzc8m:
Type: Secret (a volume populated by a Secret)
SecretName: che-token-xzc8m
Optional: false
QoS Class: Burstable
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message

---

Normal Scheduled 60m default-scheduler Successfully assigned eclipse-che/che-8678f8f794-rs9v9 to ubu2004-c
he-mk
Normal Pulling 59m (x4 over 60m) kubelet Pulling image "helm:latest"
Warning Failed 59m (x4 over 60m) kubelet Error: ErrImagePull
Warning Failed 59m (x4 over 60m) kubelet Failed to pull image "helm:latest": rpc error: code = Unknown desc
= failed to pull and unpack image "docker.io/library/helm:latest": failed to resolve reference "docker.io/library/helm:latest": pul
l access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Warning Failed 25m (x151 over 60m) kubelet Error: ImagePullBackOff
Normal BackOff 4m38s (x215 over 60m) kubelet Back-off pulling image "helm:latest"

ISSUE in all scenarios is the same. The "helm:latest" can't be pulled. Before Friday, 12th Feb 2021 that had worked.

Che version

• [ x] latest
• nightly
• other: please specify

Steps to reproduce

Expected behavior

Runtime

• [x ] kubernetes (include output of kubectl version)
• Openshift (include output of oc version)
• minikube (include output of minikube version and kubectl version)
• minishift (include output of minishift version and oc version)
• [x ] docker-desktop + K8S (include output of docker version and kubectl version)
• [ x] other: (please specify) microk8s

Screenshots

Installation method

• [x ] chectl, see above
  • provide a full command that was used to deploy Eclipse Che (including the output)
  • provide an output of chectl version command
• OperatorHub
• I don't know

Environment

• [x ] my computer
  • [ x] Windows
  • [ x] Linux
  • macOS
• Cloud
  • Amazon
  • Azure
  • GCE
  • other (please specify)
• other: please specify

Eclipse Che Logs

Additional context

[sparkoo]

thank you for reporting. Can you please try it with stable version? Just update chectl to latest stable version with chectl update stable should be enough.

[frankae]

Hi,
had done a quick test on the Windows10 docker-desktop:

1. chectl server:delete , and yes
2. chectl update stable
3. chectl server:deploy -p docker-desktop -i helm -m

Result: as before.
In "Eclipse Che pod bootstrap", "Downloading images ...failed"

[sparkoo]

@frankae oh I see it now. You're using -i helm, but -i is parameter for che-server image. You should use -a helm or --installer=helm. See chectl --help for further details.