On minikube, not able to start a workspace with che installed with chectl next

[dkwon17]

Describe the bug

On a minikube cluster, I'm not able to start a workspace with Che installed with chectl next due to the workspace pod terminating:

I'm not able to reproduce this issue on an OpenShift environment.

Che version

next (development version)

Steps to reproduce

1. Start a minikube cluster:

minikube start --addons=ingress --vm=true --cpus=4 --memory=16384

2. Deploy che with chectl next channel:

chectl server:deploy -p minikube

3. After installation is complete, open the Che dashboard and start any sample project.

4. The workspace does not start. The workspace pod terminates due to a failing init container: remote-runtime-injector:

$ kubectl get event --namespace admin-che --field-selector involvedObject.name=workspace63360a48062b4ea8-b548f9cd8-p8nqp
LAST SEEN   TYPE      REASON          OBJECT                                          MESSAGE
5s          Normal    Scheduled       pod/workspace63360a48062b4ea8-b548f9cd8-p8nqp   Successfully assigned admin-che/workspace63360a48062b4ea8-b548f9cd8-p8nqp to minikube
4s          Normal    Pulling         pod/workspace63360a48062b4ea8-b548f9cd8-p8nqp   Pulling image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next"
4s          Normal    Pulled          pod/workspace63360a48062b4ea8-b548f9cd8-p8nqp   Successfully pulled image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next" in 428.49452ms
3s          Normal    Created         pod/workspace63360a48062b4ea8-b548f9cd8-p8nqp   Created container remote-runtime-injector
3s          Normal    Started         pod/workspace63360a48062b4ea8-b548f9cd8-p8nqp   Started container remote-runtime-injector
3s          Normal    Killing         pod/workspace63360a48062b4ea8-b548f9cd8-p8nqp   Stopping container remote-runtime-injector
1s          Normal    Killing         pod/workspace63360a48062b4ea8-b548f9cd8-p8nqp   Stopping container remote-runtime-injector

Expected behavior

The workspace pod should not terminate. The workspace should start and the editor should open.

Runtime

minikube

Screenshots

No response

Installation method

chectl/next

Environment

macOS

Eclipse Che Logs

No response

Additional context

No response

[l0rd]

@dkwon17 do you have the remote-runtime-injector container logs?

[dkwon17]

@l0rd these were the logs I was able to get before the the pod was gone (I've set controller.devfile.io/debug-start: "true" for the devworkspace. related doc):

remote-runtime-injector /plugins/bash-debug-0.3.9.vsix
remote-runtime-injector /plugins/mads-hartmann.bash-ide-vscode-1.10.2.vsix
remote-runtime-injector VSIX files are already in folder /plugins, skipping the download
Stream closed EOF for admin-che/workspace1b1581fdaa91472b-99c686948-s5crz (remote-runtime-injector)

[dkwon17]

I have been able to reproduce this issue on chectl stable version (7.50) on a minikube cluster, but with one more step:

Before starting a workspace, create this config map on the user namespace:

cat << EOF | kubectl apply -n admin-che -f -
kind: ConfigMap
apiVersion: v1
metadata:
  name: my-config
  labels:
    controller.devfile.io/mount-to-devworkspace: 'true'
    controller.devfile.io/watch-configmap: 'true'
  annotations:
    controller.devfile.io/mount-as: env
data:
  TESTENV: test
EOF

(the above assumes that admin-che is the user namespace)

This configmap has the correct labels and annotations for the Devworkspace operator to set the TESTENV environment variable to the workspace containers.

Multiple workpsace pods will start up and terminate one by one. But, after deleting the configmap above the workspace pod will start up successfully.

[dkwon17]

It looks like the cause of this issue in Che installed via chectl next, is because this merged PR for che-operator: eclipse-che/che-operator#1443 creates a similar config map in the user namespace: che-idle-settings:

kind: ConfigMap
apiVersion: v1
metadata:
  name: che-idle-settings
  namespace: {USERNAMESPACE}
  labels:
    app.kubernetes.io/component: user-settings
    app.kubernetes.io/name: eclipse-che
    app.kubernetes.io/part-of: che.eclipse.org
    controller.devfile.io/mount-to-devworkspace: 'true'
    controller.devfile.io/watch-configmap: 'true'
  annotations:
    controller.devfile.io/mount-as: env
data:
  SECONDS_OF_DW_INACTIVITY_BEFORE_IDLING: '1800'
  SECONDS_OF_DW_RUN_BEFORE_IDLING: '-1'

[dkwon17]

I noticed that this issue also happens when using the che-incubator/che-code/insiders editor

Events:

│   Type    Reason     Age   From               Message                                                                                    │
│   ----    ------     ----  ----               -------                                                                                    │
│   Normal  Scheduled  5s    default-scheduler  Successfully assigned admin-che/workspace38941c7afd73486d-864b9f6ccb-f4zgw to minikube     │
│   Normal  Pulling    2s    kubelet            Pulling image "quay.io/che-incubator/che-code:insiders"                                    │
│   Normal  Pulled     1s    kubelet            Successfully pulled image "quay.io/che-incubator/che-code:insiders" in 608.828519ms        │
│   Normal  Created    1s    kubelet            Created container che-code-injector                                                        │
│   Normal  Started    1s    kubelet            Started container che-code-injector                                                        │
│   Normal  Killing    1s    kubelet            Stopping container che-code-injector

Logs for che-code-injector:

│ che-code-injector listing all files copied                                                                                               │
│ che-code-injector total 28                                                                                                               │
│ che-code-injector drwxrwxrwx 6 root root 4096 Jul 19 16:26 .                                                                             │
│ che-code-injector drwxr-xr-x 1 root root 4096 Jul 19 16:29 ..                                                                            │
│ che-code-injector drwxr-xr-x 2 1234 root 4096 Jul 19 16:26 bin                                                                           │
│ che-code-injector drwxr-xr-x 7 1234 root 4096 Jul 19 16:26 checode-linux-libc                                                            │
│ che-code-injector drwxr-xr-x 7 1234 root 4096 Jul 19 16:26 checode-linux-musl                                                            │
│ che-code-injector -rwxr-xr-x 1 1234 root 1355 Jul 19 16:29 entrypoint-volume.sh                                                          │
│ che-code-injector drwx------ 4 1234 root 4096 Jul 19 16:26 remote                                                                        │
│ Stream closed EOF for admin-che/workspace38941c7afd73486d-7d75784d7d-lz5x4 (che-code-injector)

[dkwon17]

I cannot reproduce this issue on chectl next anymore after the DWO update on che-operator, but I experience this issue on chectl stable (Che 7.51) on minikube.

A temporary workaround that I've used to get workspaces starting on minikube with Che 7.51 was to update DWO by running:

$ DWO_VERSION=v0.15.2
$ kubectl patch deployment devworkspace-controller-manager -n devworkspace-controller --patch '{"spec": {"template": {"spec": {"containers": [{"name": "devworkspace-controller","image": "quay.io/devfile/devworkspace-controller:'${DWO_VERSION}'", "env": [{"name": "RELATED_IMAGE_devworkspace_webhook_server","value": "quay.io/devfile/devworkspace-controller:'${DWO_VERSION}'"}, {"name": "RELATED_IMAGE_project_clone", "value": "quay.io/devfile/project-clone:'${DWO_VERSION}'"}]}]}}}}';

[tolusha]

@dkwon17
Can we close this issue with fix version 7.52.0 ?

[dkwon17]

Hi @tolusha thank you for the reminder, we can close this issue