Che-server fails to bootstrap on minikube v1.9.2 #16936

Divya063 · 2020-05-14T16:27:10Z

Describe the bug

Deploying Che with self signed TLS certificates on Kubernetes fails with timeout error Timeout set to pod wait timeout 300000. podExist: false, currentPhase: undefined.

Che version

latest
nightly
other: please specify

Steps to reproduce

Deploy Che using chectl chectl server:start --platform=minikube --installer=operator --self-signed-cert

Expected behavior

Runtime

kubernetes (include output of kubectl version)
Openshift (include output of oc version)
minikube (include output of minikube version and kubectl version)
minishift (include output of minishift version and oc version)
docker-desktop + K8S (include output of docker version and kubectl version)
other: (please specify)

Screenshots

Installation method

chectl
- provide a full command that was used to deploy Eclipse Che (including the output)
- provide an output of chectl version command
OperatorHub
I don't know

Environment

chectl logs

Set current context to 'minikube'
  ✔ Verify Kubernetes API...OK
  ✔ 👀  Looking for an already existing Eclipse Che instance
    ✔ Verify if Eclipse Che is deployed into namespace "che"...it is not
  ✔ ✈️  Minikube preflight checklist
    ✔ Verify if kubectl is installed
    ✔ Verify if minikube is installed
    ✔ Verify if minikube is running
    ↓ Start minikube [skipped]
      → Minikube is already running.
    ✔ Check Kubernetes version: Found v1.18.0.
    ✔ Verify if minikube ingress addon is enabled
    ✔ Enable minikube ingress addon
    ✔ Retrieving minikube IP and domain for ingress URLs...192.168.64.2.nip.io.
    ↓ Check if cluster accessible [skipped]
Eclipse Che logs will be available in '/var/folders/ck/7h90dz716lx9d9lcdqz_zhd40000gp/T/chectl-logs/1589471418284'
  ✔ Start following logs
    ✔ Start following Operator logs...done
    ✔ Start following Eclipse Che logs...done
    ✔ Start following Postgres logs...done
    ✔ Start following Keycloak logs...done
    ✔ Start following Plugin registry logs...done
    ✔ Start following Devfile registry logs...done
  ✔ Start following events
    ✔ Start following namespace events...done
  ✔ 🏃‍  Running the Eclipse Che operator
    ✔ Copying operator resources...done.
    ✔ Create Namespace (che)...It already exists.
    ✔ Checking for pre-created TLS secret... "che-tls" secret found
    ↓ Checking certificate [skipped]
    ✔ Create ServiceAccount che-operator in namespace che...It already exists.
    ✔ Create Role che-operator in namespace che...It already exists.
    ✔ Create ClusterRole che-operator...It already exists.
    ✔ Create RoleBinding che-operator in namespace che...It already exists.
    ✔ Create ClusterRoleBinding che-operator...It already exists.
    ✔ Create CRD checlusters.org.eclipse.che...It already exists.
    ✔ Waiting 5 seconds for the new Kubernetes resources to get flushed...done.
    ✔ Create deployment che-operator in namespace che...It already exists.
    ✔ Create Eclipse Che cluster eclipse-che in namespace che...It already exists.
  ❯ ✅  Post installation checklist
    ❯ Eclipse Che pod bootstrap
      ✖ scheduling
        → ERR_TIMEOUT: Timeout set to pod wait timeout 300000. podExist: false, currentPhase: undefined
        downloading images
        starting
      Retrieving Eclipse Che server URL
      Eclipse Che status check
 ›   Error: Error: ERR_TIMEOUT: Timeout set to pod wait timeout 300000. podExist: false, currentPhase: undefined
 ›   Installation failed, check logs in '/var/folders/ck/7h90dz716lx9d9lcdqz_zhd40000gp/T/chectl-logs/1589471418284'

minikube version

minikube version: v1.9.2

chectl version

chectl/7.12.2 darwin-x64 node-v10.13.0

The text was updated successfully, but these errors were encountered:

tolusha · 2020-05-15T05:08:11Z

@Divya063
Thank you for reporting.
Currently it is not possible to deploy Eclipse Che on minikube 1.9.2 due to kubernetes/minikube#7218
There is workaround kubernetes/minikube#7218 (comment)
Another option: try minikube 1.8.2

Divya063 · 2020-05-15T10:46:03Z

@tolusha Thank you for your response! I'm now using minikube 1.8.2 and still facing the same error.

chectl server:start --platform=minikube --installer=operator --self-signed-cert                                 ─╯
Set current context to 'minikube'
  ✔ Verify Kubernetes API...OK
  ✔ 👀  Looking for an already existing Eclipse Che instance
    ✔ Verify if Eclipse Che is deployed into namespace "che"...it is not
  ✔ ✈️  Minikube preflight checklist
    ✔ Verify if kubectl is installed
    ✔ Verify if minikube is installed
    ✔ Verify if minikube is running
    ↓ Start minikube [skipped]
      → Minikube is already running.
    ✔ Check Kubernetes version: Found v1.18.0.
    ✔ Verify if minikube ingress addon is enabled
    ✔ Enable minikube ingress addon
    ✔ Retrieving minikube IP and domain for ingress URLs...192.168.64.2.nip.io.
    ↓ Check if cluster accessible [skipped]
Eclipse Che logs will be available in '/var/folders/ck/7h90dz716lx9d9lcdqz_zhd40000gp/T/chectl-logs/1589539010389'
  ✔ Start following logs
    ✔ Start following Operator logs...done
    ✔ Start following Eclipse Che logs...done
    ✔ Start following Postgres logs...done
    ✔ Start following Keycloak logs...done
    ✔ Start following Plugin registry logs...done
    ✔ Start following Devfile registry logs...done
  ✔ Start following events
    ✔ Start following namespace events...done
  ✔ 🏃‍  Running the Eclipse Che operator
    ✔ Copying operator resources...done.
    ✔ Create Namespace (che)...It already exists.
    ✔ Checking for pre-created TLS secret... "che-tls" secret found
    ↓ Checking certificate [skipped]
    ✔ Create ServiceAccount che-operator in namespace che...It already exists.
    ✔ Create Role che-operator in namespace che...It already exists.
    ✔ Create ClusterRole che-operator...It already exists.
    ✔ Create RoleBinding che-operator in namespace che...It already exists.
    ✔ Create ClusterRoleBinding che-operator...It already exists.
    ✔ Create CRD checlusters.org.eclipse.che...It already exists.
    ✔ Waiting 5 seconds for the new Kubernetes resources to get flushed...done.
    ✔ Create deployment che-operator in namespace che...It already exists.
    ✔ Create Eclipse Che cluster eclipse-che in namespace che...It already exists.
  ❯ ✅  Post installation checklist
    ❯ Eclipse Che pod bootstrap
      ✖ scheduling
        → ERR_TIMEOUT: Timeout set to pod wait timeout 300000. podExist: false, currentPhase: undefined
        downloading images
        starting
      Retrieving Eclipse Che server URL
      Eclipse Che status check
 ›   Error: Error: ERR_TIMEOUT: Timeout set to pod wait timeout 300000. podExist: false, currentPhase: undefined
 ›   Installation failed, check logs in '/var/folders/ck/7h90dz716lx9d9lcdqz_zhd40000gp/T/chectl-logs/1589539010389'

tolusha · 2020-05-15T11:58:30Z

@Divya063
I don't see that keycloak, postgres and other pods starting up.
Is that the very first install logs ?

Divya063 · 2020-05-15T12:44:26Z

@tolusha After switching to minikube 1.8.2 this is the very first install logs.

tolusha · 2020-05-15T15:44:01Z

@Divya063
And that is very strange, because normally we have logs like that:

 ✅  Post installation checklist
    ✔ PostgreSQL pod bootstrap
      ✔ scheduling...done.
      ✔ downloading images...done.
      ✔ starting...done.
    ✔ Keycloak pod bootstrap
      ✔ scheduling...done.
      ✔ downloading images...done.
      ✔ starting...done.
    ✔ Devfile registry pod bootstrap
      ✔ scheduling...done.
      ✔ downloading images...done.
      ✔ starting...done.
    ✔ Plugin registry pod bootstrap
      ✔ scheduling...done.
      ✔ downloading images...done.
      ✔ starting...done.
    ❯ Eclipse Che pod bootstrap
      ✔ scheduling...done.
      ✔ downloading images...done.
      ⠋ starting
      Retrieving Eclipse Che ser

Any manual work before deploying?

Divya063 · 2020-05-15T17:36:59Z

@tolusha When I first ran chectl server:start --platform=minikube --installer=operator --self-signed-cert using minikube 1.9.2 it was stuck at the PostgreSQL pod:

✅  Post installation checklist
    ✔ PostgreSQL pod bootstrap
      ✔ scheduling...

Don't know if its relevant, After that I started following this guide - https://che.eclipse.org/running-eclipse-che-on-kubernetes-using-docker-desktop-for-mac-5d972ed511e1 but some commands were not working. One of the command was this helm upgrade — install che — namespace che — set global.cheWorkspacesNamespace=”che” — set global.ingressDomain=${CHE_DOMAIN}.nip.io ./ and the error I got was

Release “che” does not exist. Installing it now.
Error: found in requirements.yaml, but missing in charts/ directory: che-devfile-registry, che-plugin-registry, che-postgres, che-keycloak, che-jaeger, prometheus, grafana`

To fix that I used helm dependency update

amisevsk · 2020-05-15T20:59:20Z

@Divya063 Just to check, did you do a minikube delete before starting the 1.8 cluster? I don't believe the k8s version can be changed without recreating the whole VM.

The logs

    ✔ Copying operator resources...done.
    ✔ Create Namespace (che)...It already exists.
    ✔ Checking for pre-created TLS secret... "che-tls" secret found
    ↓ Checking certificate [skipped]
    ✔ Create ServiceAccount che-operator in namespace che...It already exists.
    ✔ Create Role che-operator in namespace che...It already exists.
    ✔ Create ClusterRole che-operator...It already exists.
    ✔ Create RoleBinding che-operator in namespace che...It already exists.
    ✔ Create ClusterRoleBinding che-operator...It already exists.
    ✔ Create CRD checlusters.org.eclipse.che...It already exists.
    ✔ Waiting 5 seconds for the new Kubernetes resources to get flushed...done.
    ✔ Create deployment che-operator in namespace che...It already exists.
    ✔ Create Eclipse Che cluster eclipse-che in namespace che...It already exists.

make it look like there's already some Che deployment elements present in the cluster

Divya063 · 2020-05-16T07:59:05Z

@amisevsk I didn't do minikube delete. Well, I started again by doing minikube delete:
At first it got stuck at Keycloak pod :

Logs

chectl server:start --platform=minikube --installer=operator --self-signed-cert
Set current context to 'minikube'
  ✔ Verify Kubernetes API...OK
  ✔ 👀  Looking for an already existing Eclipse Che instance
    ✔ Verify if Eclipse Che is deployed into namespace "che"...it is not
  ✔ ✈️  Minikube preflight checklist
    ✔ Verify if kubectl is installed
    ✔ Verify if minikube is installed
    ✔ Verify if minikube is running
    ↓ Start minikube [skipped]
      → Minikube is already running.
    ✔ Check Kubernetes version: Found v1.17.3.
    ✔ Verify if minikube ingress addon is enabled
    ✔ Enable minikube ingress addon
    ✔ Retrieving minikube IP and domain for ingress URLs...192.168.64.3.nip.io
.
    ↓ Check if cluster accessible [skipped]
Eclipse Che logs will be available in '/var/folders/ck/7h90dz716lx9d9lcdqz_zhd40000gp/T/chectl-logs/1589612585730'
  ✔ Start following logs
    ✔ Start following Operator logs...done
    ✔ Start following Eclipse Che logs...done
    ✔ Start following Postgres logs...done
    ✔ Start following Keycloak logs...done
    ✔ Start following Plugin registry logs...done
    ✔ Start following Devfile registry logs...done
  ✔ Start following events
    ✔ Start following namespace events...done
  ✔ 🏃‍  Running the Eclipse Che operator
    ✔ Copying operator resources...done.
    ✔ Create Namespace (che)...It already exists.
    ✔ Checking for pre-created TLS secret... "che-tls" secret found
    ↓ Checking certificate [skipped]
    ✔ Create ServiceAccount che-operator in namespace che...done.
    ✔ Create Role che-operator in namespace che...done.
    ✔ Create ClusterRole che-operator...done.
    ✔ Create RoleBinding che-operator in namespace che...done.
    ✔ Create ClusterRoleBinding che-operator...done.
    ✔ Create CRD checlusters.org.eclipse.che...done.
    ✔ Waiting 5 seconds for the new Kubernetes resources to get flushed...done
.
    ✔ Create deployment che-operator in namespace che...done.
    ✔ Create Eclipse Che cluster eclipse-che in namespace che...done.
  ❯ ✅  Post installation checklist
    ✔ PostgreSQL pod bootstrap
      ✔ scheduling...done.
      ✔ downloading images...done.
      ✔ starting...done.
    ❯ Keycloak pod bootstrap
      ✔ scheduling...done.
      ✖ downloading images
        → ERR_TIMEOUT: Timeout set to pod wait timeout 300000
        starting
      Devfile registry pod bootstrap
      Plugin registry pod bootstrap
      Eclipse Che pod bootstrap
      Retrieving Eclipse Che server URL
      Eclipse Che status check
 ›   Error: Error: ERR_TIMEOUT: Timeout set to pod wait timeout 300000
 ›   Installation failed, check logs in
 ›   '/var/folders/ck/7h90dz716lx9d9lcdqz_zhd40000gp/T/chectl-logs/1589612585
 ›   730'

I started again, it jumped to Eclipse Che pod, but failed to start:

Logs

chectl server:start --platform=minikube --installer=operator --self-signed-cert
Set current context to 'minikube'
  ✔ Verify Kubernetes API...OK
  ✔ 👀  Looking for an already existing Eclipse Che instance
    ✔ Verify if Eclipse Che is deployed into namespace "che"...it is not
  ✔ ✈️  Minikube preflight checklist
    ✔ Verify if kubectl is installed
    ✔ Verify if minikube is installed
    ✔ Verify if minikube is running
    ↓ Start minikube [skipped]
      → Minikube is already running.
    ✔ Check Kubernetes version: Found v1.17.3.
    ✔ Verify if minikube ingress addon is enabled
    ✔ Enable minikube ingress addon
    ✔ Retrieving minikube IP and domain for ingress URLs...192.168.64.3.nip.io
.
    ↓ Check if cluster accessible [skipped]
Eclipse Che logs will be available in '/var/folders/ck/7h90dz716lx9d9lcdqz_zhd40000gp/T/chectl-logs/1589613321876'
  ✔ Start following logs
    ✔ Start following Operator logs...done
    ✔ Start following Eclipse Che logs...done
    ✔ Start following Postgres logs...done
    ✔ Start following Keycloak logs...done
    ✔ Start following Plugin registry logs...done
    ✔ Start following Devfile registry logs...done
  ✔ Start following events
    ✔ Start following namespace events...done
  ✔ 🏃‍  Running the Eclipse Che operator
    ✔ Copying operator resources...done.
    ✔ Create Namespace (che)...It already exists.
    ✔ Checking for pre-created TLS secret... "che-tls" secret found
    ↓ Checking certificate [skipped]
    ✔ Create ServiceAccount che-operator in namespace che...It already exists.
    ✔ Create Role che-operator in namespace che...It already exists.
    ✔ Create ClusterRole che-operator...It already exists.
    ✔ Create RoleBinding che-operator in namespace che...It already exists.
    ✔ Create ClusterRoleBinding che-operator...It already exists.
    ✔ Create CRD checlusters.org.eclipse.che...It already exists.
    ✔ Waiting 5 seconds for the new Kubernetes resources to get flushed...done
.
    ✔ Create deployment che-operator in namespace che...It already exists.
    ✔ Create Eclipse Che cluster eclipse-che in namespace che...It already exi
sts.
  ❯ ✅  Post installation checklist
    ❯ Eclipse Che pod bootstrap
      ✔ scheduling...done.
      ✔ downloading images...done.
      ✖ starting
        → ERR_TIMEOUT: Timeout set to pod ready timeout 130000
      Retrieving Eclipse Che server URL
      Eclipse Che status check
 ›   Error: Error: ERR_TIMEOUT: Timeout set to pod ready timeout 130000
 ›   Installation failed, check logs in
 ›   '/var/folders/ck/7h90dz716lx9d9lcdqz_zhd40000gp/T/chectl-logs/1589613321
 ›   876'

Next time I ran the command, the logs depicts that Eclipse Che is already deployed.

chectl server:start --platform=minikube --installer=operator --self-signed-cert
Set current context to 'minikube'
  ✔ Verify Kubernetes API...OK
  ✔ 👀  Looking for an already existing Eclipse Che instance
    ✔ Verify if Eclipse Che is deployed into namespace "che"
      ✔ Found running Eclipse Che deployment
      ✔ Found running postgres deployment
      ✔ Found running keycloak deployment
      ✔ Found running plugin registry deployment
      ✔ Found running devfile registry deployment
 ›   Warning: Deployed Eclipse Che is found and the specified installation
 ›   parameters will be ignored
  ✔ 👀  Starting already deployed Eclipse Che
    ✔ Scaling up Eclipse Che Deployments...done.
  ❯ ✅  Post installation checklist
    ❯ Eclipse Che pod bootstrap
      ✔ scheduling...done.
      ✔ downloading images...done.
      ✖ starting
        → ERR_TIMEOUT: Timeout set to pod ready timeout 130000
      Retrieving Eclipse Che server URL
      Eclipse Che status check
 ›   Error: Error: ERR_TIMEOUT: Timeout set to pod ready timeout 130000
 ›   Installation failed, check logs in
 ›   '/var/folders/ck/7h90dz716lx9d9lcdqz_zhd40000gp/T/chectl-logs/1589614448
 ›   795'

tolusha · 2020-05-18T05:38:28Z

pls do.
kubectl delete namespace che
chectl server:start --platform=minikube --installer=operator --k8spodwaittimeout 600000

skabashnyuk · 2020-05-18T08:30:13Z

--k8spodwaittimeout

that didn't help me.

gattytto · 2020-05-23T03:57:55Z

to fix deploy using custom storage resources (storage class and persistent volume for workspaces and che Postgres), and add storage class and workspacestorageclass to chectl, if using che-operator then edit the cr.yaml accordingly

gattytto · 2020-05-23T04:01:35Z

che-incubator/chectl#461

gattytto · 2020-05-23T04:06:04Z

@skabashnyuk #16545 (comment) here's details on how to. I think there's docs pr on this too but I can't find the link.

tolusha · 2020-05-25T06:16:59Z

https://www.eclipse.org/che/docs/che-7/installing-che-using-storage-classes/

See eclipse-che/che#16936 (comment) for details

tolusha · 2020-07-08T14:09:11Z

Fixed by che-incubator/chectl#752

Divya063 added the kind/bug Outline of a bug - must adhere to the bug report template. label May 14, 2020

che-bot added the status/need-triage An issue that needs to be prioritized by the curator responsible for the triage. See https://github. label May 14, 2020

tolusha mentioned this issue May 18, 2020

postgres-storage storage class missing when starting Che on Minikube #16960

Closed

skabashnyuk added the severity/P1 Has a major impact to usage or development of the system. label May 18, 2020

tolusha changed the title ~~Che-server fails to bootstrap on Kubernetes [Timeout]~~ Che-server fails to bootstrap on minikube v1.9.2 May 25, 2020

tolusha removed the status/info-needed More information is needed before the issue can move into the “analyzing” state for engineering. label May 26, 2020

tolusha mentioned this issue May 26, 2020

Che fails to deploy on minikube #17013

Closed

5 tasks

olberger added a commit to olberger/antidote-selfmedicate that referenced this issue Jun 9, 2020

Use minikube 1.8.2 since latest can't provision workspaces currently

73878a8

See eclipse-che/che#16936 (comment) for details

tolusha mentioned this issue Jun 10, 2020

Support minikube 1.9+ when using chectl #17131

Closed

tolusha closed this as completed Jul 8, 2020

tolusha added this to the 7.16 milestone Jul 8, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Che-server fails to bootstrap on minikube v1.9.2 #16936

Che-server fails to bootstrap on minikube v1.9.2 #16936

Divya063 commented May 14, 2020

tolusha commented May 15, 2020

Divya063 commented May 15, 2020 •

edited

Loading

tolusha commented May 15, 2020

Divya063 commented May 15, 2020

tolusha commented May 15, 2020

Divya063 commented May 15, 2020 •

edited

Loading

amisevsk commented May 15, 2020

Divya063 commented May 16, 2020

tolusha commented May 18, 2020

skabashnyuk commented May 18, 2020

gattytto commented May 23, 2020

gattytto commented May 23, 2020

gattytto commented May 23, 2020

tolusha commented May 25, 2020

tolusha commented Jul 8, 2020

Che-server fails to bootstrap on minikube v1.9.2 #16936

Che-server fails to bootstrap on minikube v1.9.2 #16936

Comments

Divya063 commented May 14, 2020

Describe the bug

Che version

Steps to reproduce

Expected behavior

Runtime

Screenshots

Installation method

Environment

chectl logs

minikube version

chectl version

tolusha commented May 15, 2020

Divya063 commented May 15, 2020 • edited Loading

tolusha commented May 15, 2020

Divya063 commented May 15, 2020

tolusha commented May 15, 2020

Divya063 commented May 15, 2020 • edited Loading

amisevsk commented May 15, 2020

Divya063 commented May 16, 2020

tolusha commented May 18, 2020

skabashnyuk commented May 18, 2020

gattytto commented May 23, 2020

gattytto commented May 23, 2020

gattytto commented May 23, 2020

tolusha commented May 25, 2020

tolusha commented Jul 8, 2020

Divya063 commented May 15, 2020 •

edited

Loading

Divya063 commented May 15, 2020 •

edited

Loading