Missing http/2 support in kube-apiserver #5148

kmadac · 2022-02-21T22:34:34Z

Environmental Info:
K3s Version:
v1.22.6+k3s1 (3228d9c)

Node(s) CPU architecture, OS, and Version:
Linux ubuntu2004.localdomain 5.4.0-48-generic #52-Ubuntu SMP Thu Sep 10 10:58:49 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:
Single node vagrant VM

Describe the bug:
We need to deploy application (https://github.com/gardener/dashboard) which tries to communicate with k8s via http/2 protocol. That apllication can't run in k3s cluster. We found out that it fails when tries to communicate with kube api via http/2 protocol and for some reason it fails. We can reporoduce the issue with simple curl command.

Steps To Reproduce:

Installed K3s: curl -sfL https://get.k3s.io | sh -
Run container with curl: kubectl run tmp-shell --rm -i --tty --image nicolaka/netshoot -- /bin/bash
Run curl request:
curl --http2-prior-knowledge -sSk "https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_PORT_443_TCP_PORT/" -vv

Expected behavior:
request will return

*   Trying 10.43.0.1:443...
* Connected to 10.43.0.1 (10.43.0.1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: O=k3s; CN=k3s
*  start date: Feb 21 22:14:50 2022 GMT
*  expire date: Feb 21 22:14:50 2023 GMT
*  issuer: CN=k3s-server-ca@1645481690
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
> GET / HTTP/1.1
> Host: 10.43.0.1
> User-Agent: curl/7.80.0
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Mark bundle as not supporting multiuse
< HTTP/1.1 401 Unauthorized
< Audit-Id: d2c98f42-2f18-41e7-8772-4538b5a26d41
< Cache-Control: no-cache, private
< Content-Type: application/json
< Date: Mon, 21 Feb 2022 22:31:44 GMT
< Content-Length: 165
< 
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {
    
  },
  "status": "Failure",
  "message": "Unauthorized",
  "reason": "Unauthorized",
  "code": 401
* Connection #0 to host 10.43.0.1 left intact
}

Actual behavior:

*   Trying 10.43.0.1:443...
* Connected to 10.43.0.1 (10.43.0.1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: O=k3s; CN=k3s
*  start date: Feb 21 22:14:50 2022 GMT
*  expire date: Feb 21 22:14:50 2023 GMT
*  issuer: CN=k3s-server-ca@1645481690
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* OpenSSL SSL_write: Broken pipe, errno 32
* Failed sending HTTP2 data
* Connection #0 to host 10.43.0.1 left intact
curl: (16) OpenSSL SSL_write: Broken pipe, errno 32

Additional context / logs:
No other logs available, but it works as expected on kubernetes deployed by kubespray.

Backporting

Needs backporting to older releases

The text was updated successfully, but these errors were encountered:

brandond · 2022-02-21T23:45:52Z

Have you opened an issue with the other project? There's nothing in the Kubernetes spec or conformance tests that requires it to be made available via http/2.

kmadac · 2022-02-22T08:06:25Z

Yes I did. No reaction so far :(.

I created issue here as well to find out more information about the subject like whether is it possible to enable http/2 on k3s side? Or is it necessary to recompile it? Is this difference from original kube-apiserver intentional in order to make binaries smaller?

fix issue k3s-io#5148 Signed-off-by: Kamil Madac <kamil.madac@gmail.com>

fix issue #5148 Signed-off-by: Kamil Madac <kamil.madac@gmail.com>

VestigeJ · 2022-03-22T20:39:17Z

Validated

Validated this against VERSION=v1.21.11-rc1+k3s1, VERSION=v1.22.8-rc1+k3s1, and VERSION=v1.23.5-rc1+k3s1

$ kgn

NAME               STATUS   ROLES                       AGE     VERSION
ip-12-3-23-15   Ready    <none>                      67s     v1.21.11-rc1+k3s1
ip-12-3-25-9    Ready    control-plane,etcd,master   9m38s   v1.21.11-rc1+k3s1
ip-12-3-27-11   Ready    control-plane,etcd,master   4m33s   v1.21.11-rc1+k3s1

$ uname -a

Linux ip-12-3-25-9 5.11.0-1020-aws #21~20.04.2-Ubuntu SMP Fri Oct 1 13:01:34 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux

$ kubectl run tmp-shell --rm -i --tty --image nicolaka/netshoot -- /bin/bash
If you don't see a command prompt, try pressing enter.
bash-5.1# curl --http2-prior-knowledge -sSk

"https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_PORT_443_TCP_PORT/" -vv
*   Trying 10.43.0.1:443...
* Connected to 10.43.0.1 (10.43.0.1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: O=k3s; CN=k3s
*  start date: Mar 22 18:37:57 2022 GMT
*  expire date: Mar 22 18:43:14 2023 GMT
*  issuer: CN=k3s-server-ca@1647974277
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0xffff8f6d1a90)
> GET / HTTP/2
> Host: 10.43.0.1
> user-agent: curl/7.80.0
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Empty reply from server
* Closing connection 0
* TLSv1.3 (OUT), TLS alert, close notify (256):
curl: (52) Empty reply from server

$ kgn

NAME               STATUS                     ROLES                       AGE    VERSION
ip-12-3-23-15   Ready                      <none>                      99m    v1.22.8-rc1+k3s1
ip-12-3-25-9    Ready,SchedulingDisabled   control-plane,etcd,master   107m   v1.22.8-rc1+k3s1
ip-12-3-27-11   Ready                      control-plane,etcd,master   102m   v1.22.8-rc1+k3s1

$ kubectl run tmp-shell --rm -i --tty --image nicolaka/netshoot -- /bin/bash
If you don't see a command prompt, try pressing enter.
bash-5.1# curl --http2-prior-knowledge -sSk

"https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_PORT_443_TCP_PORT/" -vv
*   Trying 10.43.0.1:443...
* Connected to 10.43.0.1 (10.43.0.1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: O=k3s; CN=k3s
*  start date: Mar 22 18:37:57 2022 GMT
*  expire date: Mar 22 18:43:14 2023 GMT
*  issuer: CN=k3s-server-ca@1647974277
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0xffff8aeaca90)
> GET / HTTP/2
> Host: 10.43.0.1
> user-agent: curl/7.80.0
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Empty reply from server
* Closing connection 0
* TLSv1.3 (OUT), TLS alert, close notify (256):
curl: (52) Empty reply from server

$ kgn

NAME               STATUS                     ROLES                       AGE    VERSION
ip-12-3-23-15   Ready                      <none>                      111m   v1.23.5-rc1+k3s1
ip-12-3-25-9    Ready,SchedulingDisabled   control-plane,etcd,master   119m   v1.23.5-rc1+k3s1
ip-12-3-27-11   Ready                      control-plane,etcd,master   114m   v1.23.5-rc1+k3s1

$ kubectl run tmp-shell --rm -i --tty --image nicolaka/netshoot -- /bin/bash
If you don't see a command prompt, try pressing enter.
bash-5.1# curl --http2-prior-knowledge -sSk

"https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_PORT_443_TCP_PORT/" -vv
*   Trying 10.43.0.1:443...
* Connected to 10.43.0.1 (10.43.0.1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: O=k3s; CN=k3s
*  start date: Mar 22 18:37:57 2022 GMT
*  expire date: Mar 22 18:43:14 2023 GMT
*  issuer: CN=k3s-server-ca@1647974277
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0xffff9e6cba90)
> GET / HTTP/2
> Host: 10.43.0.1
> user-agent: curl/7.80.0
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 401 
< audit-id: e8e702ca-33cb-45c4-924e-836a858ece7e
< cache-control: no-cache, private
< content-type: application/json
< content-length: 157
< date: Tue, 22 Mar 2022 20:37:30 GMT
< 
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "Unauthorized",
  "reason": "Unauthorized",
  "code": 401
* Connection #0 to host 10.43.0.1 left intact

kmadac added a commit to kmadac/k3s that referenced this issue Feb 22, 2022

Adding http/2 support to API server

120ba64

fix issue k3s-io#5148 Signed-off-by: Kamil Madac <kamil.madac@gmail.com>

kmadac added a commit to kmadac/k3s that referenced this issue Feb 22, 2022

Add http/2 support to API server

ee59c6c

fix issue k3s-io#5148 Signed-off-by: Kamil Madac <kamil.madac@gmail.com>

kmadac mentioned this issue Feb 22, 2022

Add http/2 support to API server #5149

Merged

dereknola pushed a commit that referenced this issue Mar 1, 2022

Add http/2 support to API server (#5149)

3332484

fix issue #5148 Signed-off-by: Kamil Madac <kamil.madac@gmail.com>

brandond added this to the v1.23.5+k3s1 milestone Mar 22, 2022

ShylajaDevadiga assigned VestigeJ Mar 22, 2022

VestigeJ closed this as completed Mar 22, 2022

mikhail-barg mentioned this issue Dec 5, 2022

Connecting to api server fails with HTTP/2 handshake error kubernetes-client/csharp#1112

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Missing http/2 support in kube-apiserver #5148

Missing http/2 support in kube-apiserver #5148

kmadac commented Feb 21, 2022

brandond commented Feb 21, 2022 •

edited

Loading

kmadac commented Feb 22, 2022

VestigeJ commented Mar 22, 2022

Missing http/2 support in kube-apiserver #5148

Missing http/2 support in kube-apiserver #5148

Comments

kmadac commented Feb 21, 2022

brandond commented Feb 21, 2022 • edited Loading

kmadac commented Feb 22, 2022

VestigeJ commented Mar 22, 2022

brandond commented Feb 21, 2022 •

edited

Loading