Connecting to api server fails with HTTP/2 handshake error

[mikhail-barg]

Describe the bug
I'm getting the following exception while connecting to my K8s cluster:

System.Net.Http.HttpRequestException: 'An HTTP/2 connection could not be established because the server did not complete the HTTP/2 handshake.'

The code:

KubernetesClientConfiguration config = KubernetesClientConfiguration.BuildConfigFromConfigFile(kubeConfFilePath);
Console.WriteLine($"Connecting to {config.CurrentContext}, {config.Host}, {config.Namespace}, {config.Username}");
config.SkipTlsVerify = true;	//does not change anything
using (IKubernetes client = new Kubernetes(config))
{
	V1NamespaceList namespaces = client.CoreV1.ListNamespace(); //this fails

Whole stacktrace is:

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.<WaitWithCancellationAsync>d__1.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1.ConfiguredValueTaskAwaiter.GetResult()
   at System.Net.Http.HttpConnectionPool.<GetHttp2ConnectionAsync>d__79.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1.ConfiguredValueTaskAwaiter.GetResult()
   at System.Net.Http.HttpConnectionPool.<SendWithVersionDetectionAndRetryAsync>d__83.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1.ConfiguredValueTaskAwaiter.GetResult()
   at System.Net.Http.RedirectHandler.<SendAsync>d__4.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at System.Net.Http.HttpClient.<<SendAsync>g__Core|83_0>d.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at k8s.Kubernetes.<SendRequestRaw>d__39.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at k8s.AbstractKubernetes.<k8s-ICoreV1Operations-ListNamespaceWithHttpMessagesAsync>d__21.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at k8s.CoreV1OperationsExtensions.<ListNamespaceAsync>d__15.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at k8s.CoreV1OperationsExtensions.ListNamespace(ICoreV1Operations operations, Nullable`1 allowWatchBookmarks, String continueParameter, String fieldSelector, String labelSelector, Nullable`1 limit, String resourceVersion, String resourceVersionMatch, Nullable`1 timeoutSeconds, Nullable`1 watch, Nullable`1 pretty)
   at K8sApiTest.Program.Main(String[] args) in Z:\kube-test\src\K8sApiTest\Program.cs:line 18

Kubernetes C# SDK Client Version
9.0.38

Server Kubernetes Version
1.25.4

Dotnet Runtime Version
net6

Expected behavior
I expected to be able to connect to the api server

KubeConfig
my kubeconf file looks like this:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0t...
    server: https://<some_ip>:6443
  name: kubernetes-test
contexts:
- context:
    cluster: kubernetes-test
    user: kubernetes-test-admin
  name: kubernetes-test-admin@kubernetes-test
current-context: kubernetes-test-admin@kubernetes-test
kind: Config
preferences: {}
users:
- name: kubernetes-test-admin
  user:
    client-certificate-data: LS0...
    client-key-data: LS0...

Where do you run your app with Kubernetes SDK (please complete the following information):

• OS: Windows 8.1 Prof
• Environment - just a plain app from MS VS
• Cloud - No

Additional context
I doubt that there's a problem with my kubeconf file, because both Lens and KubeNav have no problems connecting to the API server using the same file.

Here are screenshots from wireshark:

My app with the problem:

Kubenav with no problem:

(black rects mask my machine, green ones mask api server)

[tg123]

looks like server side rejected tls1.2

[tg123]

seems windows8 does not support tls1.3, is your kubenav running on same machine?

[mikhail-barg]

@tg123 yes, my kubenav is on the same machine

[mikhail-barg]

My first thought also was the TLS version problem, but as far as I can understand the dumps, SSL handshake completes sucessfully even thoughs it's tls1.2 (I think it's allowed on the api server). If that was not the case, I'd be receiving SSL errors, not HTTP/2, right?

[tg123]

6443 -> you RST, which means server closed the connection
i do not think handshake succ, you can test with browser as well

[mikhail-barg]

Not sure what to test though, but I did open the https://<apiserver-url>:6443 in Chrome, and (after skipping server certificate validation failed warning), I got a (probably expected) response:

{
    "kind": "Status",
    "apiVersion": "v1",
    "metadata": {},
    "status": "Failure",
    "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
    "reason": "Forbidden",
    "details": {},
    "code": 403
}

In the wireshark I see that the connection is done via TLS 1.3, and it seems to be keepalived. And I did run it from the same machine, yes.

Should I check something else?

[mikhail-barg]

Okay, I did the following from a c# program:

HttpClientHandler handler = new HttpClientHandler() {
	ServerCertificateCustomValidationCallback = HttpClientHandler.DangerousAcceptAnyServerCertificateValidator
};

using (HttpClient client = new HttpClient(handler))
{
	using (HttpResponseMessage result = await client.GetAsync("https://<apiserver-url>:6443"))
	{
		Console.WriteLine(result.StatusCode);
		Console.WriteLine(await result.Content.ReadAsStringAsync());
	}
}

and it gives me the same HTTP response

{
	"kind": "Status",
	"apiVersion": "v1",
	"metadata": {},
	"status": "Failure",
	"message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
	"reason": "Forbidden",
	"details": {},
	"code": 403
}

And in wireshark I see that it's done via TLS 1.2

[mikhail-barg]

Ok, I am able to reproduce the problem using the following change to the code above:

using (HttpRequestMessage req = new HttpRequestMessage(HttpMethod.Get, "https://<apiserver-url>:6443") {
	Version = new Version(2, 0)
})
{
	using (HttpResponseMessage result = await client.SendAsync(req))

So switching request to Http/2 causes the same problem as I experience with the client library.
Not sure where to go next then.

[tg123]

anything special with your api server, the sdk assumes it supports http2 and did not have n option to change it at the moment.

[mikhail-barg]

anything special with your api server

don't think so. Just a vanilla K8s, bootstrapped via kubeadm.

The only thing that comes to mind, is that we've set it up with --skip-phases=addon/kube-proxy to use Cilium for CNI, but I don't think it's a culprit.

[mikhail-barg]

I also have no luck in googling any HTTP/2 issues with kube api server, except for this, but it's from K3s, which we don't use.

[mikhail-barg]

OK, some more info. I tried running my original app from Win 11 machine, and it does connect successfully using the same kubeconf file. So it seems that it's really not a HTTP/2 problem.

[tg123]

could you please take a look at api server to see why it does not like windows 8.1

[mikhail-barg]

Yes, but I'm not sure what to look at. Maybe you could give some hints? I think, we have all TLS versions and all ciphers enabled at apiserver.

[tg123]

https://kubernetes.io/docs/tasks/debug/debug-cluster/#control-plane-nodes

LMK if it helps, so we can do something to improve sdk's compatibility

[mikhail-barg]

It seems that there's a specific requirement in rfc7540 that says that in case of HTTP/2 over TLS1.2 there's a list of additionally forbidden ciphers.

It looks that what I'm facing is that my client and server negotiate a cypher that is OK in general, but is not allowed for HTTP/2. So when client requests an HTTP/2 connection AFTER SSL handshake, api server rejects it because of inappropriate cipher used for the channel.

The part of a problem is that SslClientAuthenticationOptions.CipherSuitesPolicy is not available on Windows platform so I cannot specify exact ciphers on the client side.

So I'll be checking the --tls-cipher-suites option of api-server, but for now I have no luck finding a proper ciphers subset.

Just for a reference, here's a win 8.1 ciphers list.

[tg123]

good catch
will add an option to disable http2
the reason we force http2 is to have the connections keep alive, otherwise loadbalancers might kick the watch connections.

[tg123]

add some config to bypass force http2
btw win8.1 is eol Jan 2023

[mikhail-barg]

Thanks for considering and implementing this feature! Hope to see a package release soon!

the reason we force http2 is to have the connections keep alive, otherwise loadbalancers might kick the watch connections.

Hm, I think HTTP keepalive (not closing TCP connection after request completed) is available as an option for HTTP/1.1, see for example this. So you probably mean the option of sending PING frames, which is HTTP/2 specific?

btw win8.1 is eol Jan 2023

Thanks for highlighting this! Will see if there's an upgrade possible.

[tg123]

long discussion about keep alive in #533

the tcp keep alive is not easy to setup, thus, the sdk prefers https now

[tg123]

close as disable http2 option added

[mikhail-barg]

Just checked the new release, and it works! Finally! Thanks again!