Python Kubernetes Client - 500 Internal Server Error when running exec command

[alex-astronomer]

Expected behavior:
I expect that with the right authentication and access that I can run exec commands from a Python Kubernetes client without error.

Current behavior:
This could be user error, but it's hard for me to tell without having the real error message appear. I have an API object in Python which we will call api from this point forward. I've tested a few scenarios.

---

Docker Desktop Cluster:

I can run exec commands, and any other command that I choose. No teleport enabled for local cluster. This is my baseline to determine that my Python syntax and commands are correct. All commands are working as expected, including the Websocket-based exec call.

Teleport Cluster:

I can run all commands except for exec. This returns the error with traceback:

Traceback (most recent call last):
  File "/Applications/PyCharm CE.app/Contents/plugins/python-ce/helpers/pydev/_pydevd_bundle/pydevd_exec2.py", line 3, in Exec
    exec(exp, global_vars, local_vars)
  File "<string>", line 1, in <module>
  File "/Users/alexkennedy/.pyenv/versions/creutilannot/lib/python3.10/site-packages/kubernetes/stream/stream.py", line 35, in _websocket_request
    return api_method(*args, **kwargs)
  File "/Users/alexkennedy/.pyenv/versions/creutilannot/lib/python3.10/site-packages/kubernetes/client/api/core_v1_api.py", line 994, in connect_get_namespaced_pod_exec
    return self.connect_get_namespaced_pod_exec_with_http_info(name, namespace, **kwargs)  # noqa: E501
  File "/Users/alexkennedy/.pyenv/versions/creutilannot/lib/python3.10/site-packages/kubernetes/client/api/core_v1_api.py", line 1101, in connect_get_namespaced_pod_exec_with_http_info
    return self.api_client.call_api(
  File "/Users/alexkennedy/.pyenv/versions/creutilannot/lib/python3.10/site-packages/kubernetes/client/api_client.py", line 348, in call_api
    return self.__call_api(resource_path, method,
  File "/Users/alexkennedy/.pyenv/versions/creutilannot/lib/python3.10/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
    response_data = self.request(
  File "/Users/alexkennedy/.pyenv/versions/creutilannot/lib/python3.10/site-packages/kubernetes/stream/ws_client.py", line 497, in websocket_call
    raise ApiException(status=0, reason=str(e))
kubernetes.client.exceptions.ApiException: (0)
Reason: Handshake status 500 Internal Server Error

just a generic handshake error with a 500 code.

kubectl:

To confirm that I actually have permissions to do this, I also ran commands from tsh on my shell. I have access to any resource that I need, and along with that, exec commands are working fine when using tsh.

I have also checked through kube apiserver logs as well as kube authenticator logs and it looks like the commands are not making it to Kubernetes which is why I suspect teleport in this case.

I believe that the variable here that isn't a control between the cases is SPDY vs Websockets for the exec call. It seemed strange to me that kubectl can exec but Python can't even with the exact same login process and kubeconfig and I believe that kubectl still uses SPDY for the exec, while Python uses Websockets. I was trying to find a way to call the exec function with SPDY from Python but the python-kubernetes doesn't support this (based on my own research, would like confirmation here). The one thing that I can see that changes between kubectl and python api call is SPDY vs Websockets. That's my best guess right now.

---

Bug details:

• Teleport version - 8.0.7
• Recreation steps
  • Base requirements:
    • Teleport cluster, pointing to a kubernetes cluster.
    • Access and permissions for exec, namespace, and pods
  • Configure Python with the correct teleport kubeconfig by instantiating this file with
    • Set kubeconfig location
    • Login to teleport through proxy tsh login --proxy=<my proxy>
    • login to cluster using tsh kube login <cluster id>
  • Create a Python kubernetes API connection and run commands
    • `api.read_namespaces("") - should work fine
    • api.read_namespaced_pod("<pod name>", "<ns>") - should work fine

    •   stream(
          api.connect_post_namespaced_pod_exec,
          name="<pod name>",
          namespace="<ns>",
          command=["echo", "Hello"],
          stdout=True,
          stderr=True,
        )
      

    • This command raises the generic 500 internal server error exception

[alex-astronomer]

Just realized there's a lot of issues that got opened today regarding the same problem. Closing.

[alex-astronomer]

#15474
#15463

[alex-astronomer]

Actually not sure that this is 100% related yet, so reopening to track.

[tigrato]

#15463 is precisely the same issue.
We currently only support SPDY and not WebSocket for exec

#15474 is a tracking issue for a future websocket protocol

[alex-astronomer]

Any recommendations then on a client that uses SPDY? Or is using a Python client just not possible right now with Teleport and Kubernetes? @tigrato

[tigrato]

Any recommendations then on a client that uses SPDY? Or is using a Python client just not possible right now with Teleport and Kubernetes? @tigrato

For now, I think that only Go's client uses SPDY