Proposal: A way to control idle connection timeouts

[danazkari]

This is an AWESOME tool to provide devs an easy and secure way of using our development environment on our k8s cluster.

The issue

Idle connection timeout is very short, and we can't change streaming-connection-idle-timeout setting in our managed kubernetes cluster.

Proposal

Some kind of flag to either control the port-forwarding tunnel idle timeout (not sure how feasible this would be tho), or one that tells the tool to reconnect the service upon disconnecting.

I'm not a fan of either of those two options but perhaps this is a good conversation starter to find a solution.

[cjimti]

kubefwd port-forwards to Pods. I have not done any type of exhaustive research on this, but in my initial work, I did not find a reliable way to know what constitutes a complete failure or temporary disruption and what to do next. Since kubefwd was designed as a development utility one of the main use cases involves the ability to observe the behavior of the pods you are connected to, something difficult to do when communicating with a server that has dozens or hundreds of pods backing it (unless you have a well oiled, distributed tracing implementation). Automatically disconnecting from on Pod and connecting to another may break that use cast, leaving the developer observing a Pod they are no longer connected to.

I like the feature and would love to have a flag that enables it. There are times that I don't care what Pod I am connected to and only concerned for the connection at the service level.

At this point, kubefwd only knows a connection is down when an attempt to connect is made; this may be some time after a Pod has crashed. Would kubefwd need to watch for changes on connected Pods.

[danazkari]

Yeah I didn't think this would be an easy task by any stretch, just thought it was necessary to voice what could be a very VERY cool feature, I can't promise I can do anything about it, but I'll surf through the code trying to get how kubefwd works and perhaps offer my brain to bounce ideas around and see if there's any feasible way of doing this.

At least for the use case I'm handling right now, my team of developers don't really care to which pod they're connecting to, as long as they get an endpoint to the deployed service they can reference in their local development (having all of our setup running locally, really upsets our laptops hehe)

What I've noticed is that the timeout is precisely 5 minutes and has to do with the --streaming-connection-timeout-idle perhaps a child process can be spawned to check every X amount of minutes to check connectivity to a service, and if it's down try to make the connection again? I might be talking complete trash tho haha, haven't looked at the code long enough yet to know if this even makes any sense at all.

[cjimti]

I'll have to explore the timeout issue you are having. I typically run kubefwd aginst custom cluster I build and at times I have had kubefwd attached for days with no problem.

[danazkari]

That's interesting, I think this "bug" is only true for managed k8s instances (GKE, AKS or EKS), where one doesn't have control of how that's setup. I haven't tried in GKE or EKS yet but I'd imagine they would behave similarly.

I'm jealous of your k8s instance running kubewd for days 😛

[photomoose]

I'm also experiencing the disconnection after 5 minutes of idle time in Azure AKS. I use kubefwd extensively, but it does become a bit annoying having to restart kubefwd and enter my password each time (so it can update the hosts file). I haven't looked in any depth whether I can configure the kubelet timeout setting yet.

[cjimti]

The 5-minute timeout looks to be a kubelet setting (feature). I use kubeadm to build custom clusters which must set this timeout to infinity.
https://stackoverflow.com/questions/47484312/kubectl-port-forwarding-timeout-issue

I plan to explore some options for re-connecting soon.

[cjimti]

Release 1.11.0 should resolve this. I have not extensive testing and will re-open the issue if needed.

[deadwards90]

I'm afraid it doesn't reconnect automatically on timeout. Using Azure AKS which has the 5 minute window. I have seen it automatically reconnect when a pod is deployed though.

[hdiass]

I can confirm that not reconnecting on the 5 minute timeout. Works only on pod refresh only.

[deadwards90]

I can actually see it reconnecting now, after updating to 1.17.0, however it does take some time.

INFO[07:38:04] Port-Forward: 127.1.27.1 pdfs:80 to pod pdfs-7d79bb499f-7vtlg:80
ERRO[07:46:29] Runtime error: lost connection to pod
WARN[07:46:29] Stopped forwarding pod pdfs-7d79bb499f-7vtlg for pdfs.namespace.cluster
INFO[07:58:04] Port-Forward: 127.1.27.1 pdfs:80 to pod pdfs-7d79bb499f-7vtlg:80

I think this is probably what #153 is trying to solve.

[tvvignesh]

@cjimti Hi. Is this fixed? I still find port forwarding timing out in my GKE cluster every few minutes and it does not reconnect with v1.17.3.

If I stop it and then start it again, it works. But the annoying thing is that I have to do this every 5 minutes or so 😢

[cjimti]

I run version 1.17.3 and forward ~30 services (pods) ~6-8 hours a day. I only drop connections If I suspend my computer or lose my internet connection completely, which is rare.

There was some work to check connections and re-connect a few versions ago that caused more problems than it solved. I'm not an expert on socat and the networking around you and GKE and why you would lose connection so often.

Let me know if you have tried version 1.17.3. It does not make attempts to check the connections and re-connect, yet in my experience has been more stable.

[tvvignesh]

@cjimti Hi. Thanks a lot for your response. I am using 1.17.3 as mentioned above already and while it works and is stable, the issue of timeout is not with Kubefwd directly but due to --streaming-connection-idle-timeout as mentioned here https://stackoverflow.com/questions/47484312/kubectl-port-forwarding-timeout-issue which I guess is around 4-5 minutes in GKE and I am guessing that is the reason due to which the connect breaks.

Actually, I had similar problems when forwarding manually like this (without Kubefwd):

#!/bin/sh
kubectl --context <mycontext> --namespace <mynamespace> port-forward $(kubectl get pods --context <mycontext> --namespace <mynamespace> --selector "statefulset.kubernetes.io/pod-name=dgraph-dgraph-alpha-0,release=dgraph" --output jsonpath="{.items[0].metadata.name}") 9999:8080

But to get over this problem and reconnect, I wrapped it within a while like this and it reconnects without issues:

#!/bin/sh
while true; do kubectl --context <mycontext> --namespace <mynamespace> port-forward $(kubectl get pods --context <mycontext> --namespace <mynamespace> --selector "statefulset.kubernetes.io/pod-name=dgraph-dgraph-alpha-0,release=dgraph" --output jsonpath="{.items[0].metadata.name}") 9999:8080; done

I was wondering how to do the same with Kubefwd since I cant wrap it in a while (it does not exit on forwarding failures) or was wondering if there was an inbuilt way to do this so that Kubefwd can reconnect periodically rather than me having to modify the flag in the kubelet to increase the timeout which is not recommended.

Really love kubefwd. Btw, while not related to this issue, is there any way to use kubefwd over remote ssh?

I have my setup like this where I use a remote VM to do all my development:
My PC->Remote SSH to VM in GCP->Kubefwd->GKE Cluster

Now, while kubefwd works well in remote VM, I was wondering if there was a way to forward the same to my local PC as well so that I need not open an RDP to access the browser remotely when needed.

Thanks.

[alexef]

@cjimti re-opening an old thread as we're also constantly affected by this. Would it be possible to have kubefwd fail (and terminate the process) on a forwarding failure?

What we see is:

ERR: Runtime: lost connection to pod
WARN: Stopped forwarding pod ...

and then kubefwd just hangs forever.