SPI Kubernetes infrastructure

[l0rd]

Currently only Docker and OpenShift infrastructure have been implemented. Once OpenShift infrastructure is mature enough we should extract the Kubernetes parts and make a new infrastructure. OpenShift infrastructure should be an extension of Kubernetes.

• Support Docker images recipes
• Support k8s recipes
• Implement a deployment yml to deploy Che on minikube using kubectl
• Provide the documentation to run Che on minikube
• Che Volumes don't work because of PVC permissions issue Directories provisioned by hostPath provisioner are only writeable by root kubernetes/minikube#1990.
  As a workaround added an ability to configure context security for pods. Setting runAsUser and FsGroup to 0 fixes issue on minikube.
• Add nginx connection configuration that doesn't kill websocket connections
• Improve Kubernetes infrastructure by adding an ability to configure ingresses implementation (now nginx is hardcoded)
• Improve Kubernetes infrastructure by adding an ability to configure TLS for ingresses
• Implement a Helm Kubernetes Chart to distribute Che
• Improve dashboard to support kubernetes recipes
• Run QA on kubernetes infrastructure
• Maybe we need minikube addon for Che
• Improve Kubernetes infrastructure by adding an ability to set hostname for ingresses (hostname based single port)
• Add checking that Kubernetes installation is configured with all components for Che Kubernetes infrastructure (like ingress controllers, DNS service discovery)
• Kubernetes secret token is mounted to all workspaces pods. It gives access to Kubernetes API from Pods. Check whether it is a problem and disable mounting of the token if it is possible
• Dashboard doesn't reconnect to a WebSocket endpoint if the previous connection failed Dashboard doesn't reconnect to a WebSocket endpoint if the previous connection failed #8675
• Add an ability to use host-based routing instead of path-based for k8s infra Add an ability to use host-based routing instead of path-based for k8s infra #8694
• Add an ability to deploy Multiuser Che on Kubernetes Deploy Multi-User Che on Kubernetes #8745
• add ability to configure workspace external IP on k8s infra #9131 add ability to configure workspace external IP on k8s infra

[garagatyi]

@l0rd @gorkem
Currently, I'm thinking how to implement k8s infra and it would be very useful to understand what for it is needed.
Do you have any inputs on what are requirements for this infra? Or maybe you have user stories?

[gorkem]

I think minikube and kubeadm are the environments that we are targeting for this work. This SPI will be used by those who wants to have an independent k8s installation to run Che or as a starting point for the other k8s based solutions.

[l0rd]

Here is a list of things I think we should implement:

• Support Docker images recipes
• Support k8s recipes
• Implement a deployment yml to deploy Che on minikube using kubectl
• Implement a Helm Kubernetes Chart to distribute Che
• Provide the documentation to run Che on minikube

@garagatyi I planned to work on that too next sprint and we may work together on these tasks wdyt?

@gorkem maybe you meant kubectl instead of kubeadm?

[garagatyi]

We can share these tasks, sure. But this is probably needed to be decided on a teams level to achieve better utilization of resources of teams. @skabashnyuk WDYT?

[gorkem]

@l0rd I think kubeadm is the one used for installing clusters right?

[andrewhowdencom]

There's a certified conformance program if you're looking for a "standard Kubernetes": https://github.com/cncf/k8s-conformance

heptio/buoy checks clusters against a checklist, might help here. Could use the rule "where heptio/buoy passes, this should work"

[garagatyi]

@andrewhowdencom sorry, I didn't get what issue suggested solution is supposed to solve, can you elaborate?

[andrewhowdencom]

I think minikube and kubeadm are the environments that we are targeting for this work. This SPI will be used by those who wants to have an independent k8s installation to run Che or as a starting point for the other k8s based solutions.

It was my understanding that there was some notion of defining what a suitable test environment for the API work for che should be.

To determine in what environments Kubernetes is supposed to work, you could use the guideline "Does it adhere to the Kubernetes Conformance guidelines". In this case, it would not rather be kubeadm, gke or any other specific cluster, but rather clusters that are "known correct".

The point about heptio/buoy was to resolve any ambiguity whether it should be some handling built in to che to handle ambiguity between k8s environments (for example, rbac here and not there, or alpha features being enabled). The goal being "Che should operate in environments in which buoy indicates there are no issues":

Reference: https://github.com/heptio/sonobuoy

If I misread this, I'm sorry <3 I was just trying to pass on information that I thought may be useful. I'm excited to see this work completed!

[garagatyi]

@andrewhowdencom thank you for the explanation. Not sure we need to test environment for the compatibility with k8s TCK. Anyone can correct me if I missed something.
If you are referring discussion from the Che bi-weekly meeting about testing k8s infrastructure we were discussing TCK for Che SPI itself, not k8s.

[gorkem]

@garagatyi I think the question is what kubernetes platform/distribution should we test Che SPI against. There are over 50 of those nowadays and we can not test them all.

@andrewhowdencom Thanks for bringing up sonobuoy, it is interesting. I think we should study it.

[sleshchenko]

So, @garagatyi and I did some investigation and work on Kubernetes infrastructure.

Today we have a branch with Kubernetes infrastructure. There are few more pieces of stuff that required to be reworked a bit. PR will be created soon.
So Kubernetes infrastructure supports dockerimage and kubernetes(yaml list with kubernetes objects) recipes.
I've tested it using the following configuration:
Che is running on Docker on my localhost. Kubernetes infrastructure is configured to start workspaces on my local minikube (Che creates workspaces Kubernetes Objects on a minikube).

OpenShift infrastructure extends Kubernetes and supports dockerimage, kubernetes, openshift recipes.

The list of tasks what is left to do is in the issue description.

[sleshchenko]

So, PR with k8s infrastructure is merged.
I've updated tasks list in the issue description.
What's new there:

• Dashboard doesn't reconnect to a WebSocket endpoint if the previous connection failed Dashboard doesn't reconnect to a WebSocket endpoint if the previous connection failed #8675
  Because of some reason WebSocket connections are interrupted often, it's not an issue for IDE because it reconnects automatically. But Dashboard doesn't do it and loses events from workspace master. So Dashboard should be improved. And maybe it makes sense to investigate why WebSocket connections are interrupted.
  I suppose it happens because of nginx ingress controller. K8s infra adds new ingress for each workspace server and nginx controller restarts itself when new ingress rule is added. So, there is an ability to configure ingress controller for created workspaces ingresses. So only Dashboard should be fixed, and WebSocket connections can be made stable with another ingress controller (I suppose traefik ingress controller can be used for such purposes)

• One more issue that I've faced is PVC permissions issue on minikube. Che Volumes is mounted in containers with root permissions even when precreate subpaths feature is turned on. I guess it is related to Directories provisioned by hostPath provisioner are only writeable by root kubernetes/minikube#1990. As a workaround added an ability to configure security context for pods, and for minikube root user is configured.

• Add an ability to use host-based routing instead of path-based for k8s infra Add an ability to use host-based routing instead of path-based for k8s infra #8694
  Without this feature, workspaces start well but some of the sample applications fail to work correctly.

• @riuvshin is working on Run QA on kubernetes infrastructure set up k8s environment for testing CHE on CI #8578

• Here is an PR for issue Implement a Helm Kubernetes Chart to distribute Che. @perspectivus1 should recreate it in Eclipse Che repository

[sleshchenko]

Update:

• Moved tasks into the epic issue description.
• Added an issue for deploying Multi-User Che on a Kubernetes.

[riuvshin]

linked issue that blocks us to run QA tests on CI against k8s infra #9131, as I understand this PR: #8822 should cover it.

[carloreggiani]

Not sure this is the right thread, but I'm trying to to deploy Eclipse Che on Docker EE 2.0 Kubernetes cluster (and also Azure AKS), but docs explaing only minikube conf: any documentation expecially for networking configuration to access Eclipse Che services (some trouble for me to configure che-ingress)?

[l0rd]

@carloreggiani you can use the Helm Chart https://github.com/eclipse/che/blob/master/deploy/kubernetes/helm/che/readme.md

[carloreggiani]

It was my firts try, @l0rd, but I'm trying to understand how to configure on Docker EE kubernetes the RBCA prerequisite steps (I'm connected ad admin user to the cluster):

kubectl create clusterrolebinding add-on-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:default
error: no matches for rbac.authorization.k8s.io/, Kind=ClusterRoleBinding

[garagatyi]

@l0rd I think we can close this issue now. WDYT?

[garagatyi]

Closing the issue since it is already implemented.