kube exec in web UI fails with "disconnected" error

[webvictim]

Expected behavior

kubectl exec in the web UI should work.

Current behavior

Attempting to exec into a pod provided by a locally-running kubernetes_service fails with an error:

Web UI:

Proxy logs:

2024-06-18T18:00:17Z DEBU [WEB]       New kube exec request for namespace=example-grafana pod=example-grafana-5564998b54-frh7d container=, sid=a3e742fd-205c-41eb-96d3-204c6a93a34b, websid=bb5823383bc25b4390
21b6918341f7328a94af13de98a38eadb340af2539b1e8. web/apiserver.go:3385
2024-06-18T18:00:17Z DEBU [POD]       Creating websocket stream for a kube exec request web/kube.go:188
2024-06-18T18:00:17Z DEBU [POD]       Starting websocket ping loop with interval 5m0s. web/ws_io.go:75
2024-06-18T18:00:17Z DEBU [CLIENT]    MFA requirement from CreateAuthenticateChallenge, MFARequired=MFA_REQUIRED_NO client/cluster_client.go:587
2024-06-18T18:00:17Z DEBU [POD]       Web kube exec request URL: https://localhost:3080/api/v1/namespaces/example-grafana/pods/example-grafana-5564998b54-frh7d/exec?command=%2Fbin%2Fbash&stdin=true&stdout=true&t
ty=true web/kube.go:295
2024-06-18T18:00:17Z DEBU [PROXY:PRO] Ignoring unsupported cluster name name "kube-teleport-proxy-alpn.teleport.cluster.local". pid:7.1 authclient/tls.go:108
2024-06-18T18:00:17Z ERRO [POD]       "failed exec command streaming\n\tunable to upgrade streaming request: websocket: bad handshake" web/kube.go:165
2024-06-18T18:00:17Z WARN [POD]       Unable to send error to terminal: Expected binary message, got -1 error:[
ERROR REPORT:
Original Error: *websocket.netError set tcp 10.100.43.197:3080: use of closed network connection
Stack Trace:
    github.com/gravitational/teleport/lib/web/terminal/terminal.go:338 github.com/gravitational/teleport/lib/web/terminal.(*WSStream).Write
    strings/replace.go:319 strings.stringWriter.WriteString
    strings/replace.go:371 strings.(*genericReplacer).WriteString
    strings/replace.go:103 strings.(*Replacer).WriteString
    github.com/gravitational/teleport/lib/web/terminal/terminal.go:118 github.com/gravitational/teleport/lib/web/terminal.(*WSStream).WriteError
    github.com/gravitational/teleport/lib/web/terminal/terminal.go:165 github.com/gravitational/teleport/lib/web/terminal.(*WSStream).processMessages
    runtime/asm_amd64.s:1695 runtime.goexit
User Message: set tcp 10.100.43.197:3080: use of closed network connection] terminal/terminal.go:119
2024-06-18T18:00:17Z DEBU [POD]       Terminating websocket ping loop. web/ws_io.go:95
2024-06-18T18:00:45Z DEBU [WEB]       Could not authenticate: missing session cookie web/a

This may be the case when TLS is terminated in front of Teleport - I don't see the same issue when running Teleport on a single EC2 node with locally-terminated TLS. This test setup is using the teleport-cluster Helm chart on an EKS cluster with ingress-nginx and cert-manager running for ingress/TLS termination.

There is also another error message that appears when trying to exec into a pod on a remotely-connected Kubernetes cluster:

2024-06-18T18:06:50Z DEBU [WEB]       New kube exec request for namespace=ingress-nginx pod=ingress-nginx-controller-c8f499cfc-ftk48 container=, sid=bd8a3389-a062-44b9-bfb3-37efab25e8cf, websid=26dbc8f51c7f0d654
2fceb386fdb4a6ded5d18cbaed65271c62bcc98b08258b1. web/apiserver.go:3385
2024-06-18T18:06:50Z DEBU [POD]       Creating websocket stream for a kube exec request web/kube.go:188
2024-06-18T18:06:50Z DEBU [POD]       Starting websocket ping loop with interval 5m0s. web/ws_io.go:75
2024-06-18T18:06:50Z DEBU [CLIENT]    MFA requirement from CreateAuthenticateChallenge, MFARequired=MFA_REQUIRED_NO client/cluster_client.go:587
2024-06-18T18:06:50Z ERRO [POD]       "failed issuing user certs\n\tKubernetes cluster \"62cd53b1-f580-438c-bd97-01da032a785c\" is not registered in this Teleport cluster; you can list registered Kubernetes clus
ters using 'tsh kube ls'" web/kube.go:165
2024-06-18T18:06:50Z WARN [POD]       Unable to send error to terminal: Expected binary message, got -1 error:[
ERROR REPORT:
Original Error: *websocket.netError set tcp 10.100.21.134:3080: use of closed network connection
Stack Trace:
    github.com/gravitational/teleport/lib/web/terminal/terminal.go:338 github.com/gravitational/teleport/lib/web/terminal.(*WSStream).Write
    strings/replace.go:319 strings.stringWriter.WriteString
    strings/replace.go:371 strings.(*genericReplacer).WriteString
    strings/replace.go:103 strings.(*Replacer).WriteString
    github.com/gravitational/teleport/lib/web/terminal/terminal.go:118 github.com/gravitational/teleport/lib/web/terminal.(*WSStream).WriteError
    github.com/gravitational/teleport/lib/web/terminal/terminal.go:165 github.com/gravitational/teleport/lib/web/terminal.(*WSStream).processMessages
    runtime/asm_amd64.s:1695 runtime.goexit
User Message: set tcp 10.100.21.134:3080: use of closed network connection] terminal/terminal.go:119
2024-06-18T18:06:50Z DEBU [POD]       Terminating websocket ping loop. web/ws_io.go:95

Bug details:

• Teleport version: 16.0.1

[webvictim]

This turned out to be an issue with missing Kubernetes permissions, confirmed with tsh:

gus@apollo:~ % tsh kube login gus-teleport-dev2.example.com
Logged into Kubernetes cluster "gus-teleport-dev2.example.com".

Your Teleport cluster runs behind a layer 7 load balancer or reverse proxy.

To access the cluster, use "tsh kubectl" which is a fully featured "kubectl"
command that works when the Teleport cluster is behind layer 7 load balancer or
reverse proxy. To run the Kubernetes client, use:
  tsh kubectl version

Or, start a local proxy with "tsh proxy kube" and use the kubeconfig
provided by the local proxy with your native Kubernetes clients:
  tsh proxy kube -p 8443

Learn more at https://goteleport.com/docs/architecture/tls-routing/#working-with-layer-7-load-balancers-or-reverse-proxies-preview

gus@apollo:~ % tsh kubectl get pods
ERROR: Your user's Teleport role does not allow Kubernetes access. Please ask cluster administrator to ensure your role has appropriate kubernetes_groups and kubernetes_users set.

Adding cluster-admin to the kubernetes_groups in my access role and logging out/back in again fixed the issue. Exec works in the web UI now too.

Related issue: #42626