envd kubernetes design discussion

[gaocegege]

Ref #303

[gaocegege]

There are two options on how to support ssh in a Kubernetes pod:

• Use libcontainerssh, it runs a backend server containerssh, as the proxy. When users run ssh user@host, the proxy creates a pod, then attaches to it.
  
• embed a sshd into the image (like what we do in envd). Then users can ssh to the pod directly.

[gaocegege]

The benefits of a proxy server:

• Easy to audit and control. The server has the full info about the connections. Thus we can implement GC (tear down when the ssh connection is idle) in this layer.
• SSO can be supported.

[gaocegege]

Or, this ssh-proxy backend can be used instead of kubernetes backend: https://containerssh.io/getting-started/sshproxy/

We still embed a sshd in the image, then use contaienrssh as the proxy server for the sshd in the pod.

[gaocegege]

[gaocegege]

Why do we need an envd API server?

• audit
• admin dashboard
• RBAC and user auth
• resource management

SSH

We embed an ssh server to the image and a pub key. The envd API server should use the private key provided by the envd user, as the auth method, to connect to the backend pod.

Ref https://containerssh.io/getting-started/sshproxy/

Need to investigate:

• How to work as the ssh proxy to forward the private key to the backend pod.

Reconnection

The backend pod will be garbage collected if it is not used again for at least 30 minutes. If the user reconnects to the pod, envd API server needs to decide which one to re-attach.

Thus the backend pod should have a unique identifier. When the user run envd attach (Or maybe envd ssh), the only information given to envd API server, is the name of the project, and the username (if user and org are supported in the API server). envd API server should get the correct pod from the info.

Data integration

runtime.docker.mount and other docker-specific funcs will be ignored in kubernetes context. runtime.kubernetes.mount() should create/use the corresponding PV/PVC in the backend pod.

runtime.kubernetes.mount("")

port forwarding (jupyter)

We may need to use ssh port forwarding (a.k.a ssh tunneling) for this. Because users may run new services in runtime. Exposing ports on pod cannot be updated dynamically.

But, we need to think: Is it reasonable to support dynamic ports if we expect infra as code?

[gaocegege]

How to integrate ssh auth with the auth in envd