Sporadic status 400 on concurrent HTTP2 POST requests

[PSanetra]

NGINX Ingress controller version: 0.29.0

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.10", GitCommit:"575467a0eaf3ca1f20eb86215b3bde40a5ae617a", GitTreeState:"clean", BuildDate:"2019-12-16T16:29:58Z", GoVersion:"go1.12.14", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.8", GitCommit:"c52f59bbba5fbf21fbb18e9a06f96e563fe4c20a", GitTreeState:"clean", BuildDate:"2020-01-31T20:00:26Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration: Azure Kubernetes Service
• OS (e.g. from /etc/os-release): Ubuntu 16.04.6 LTS
• Kernel (e.g. uname -a): 4.15.0-1066-azure

What happened:

Concurrent HTTP2 POST requests fail sporadically with an HTTP 400 status code.
The nginx-ingress-controller logs the following messages when this error is happening (with verbose logging enabled):

10.240.0.4 - - [02/Apr/2020:15:37:46 +0000] "POST /xxx HTTP/2.0" 400 157 "-" "Go-http-client/2.0" 792 0.000 [example] [] - - - - ee85f511e37d1f7dcd07b95b144245a
[...]
2020/04/02 15:37:46 [info] 41#41: *1235 client prematurely closed stream: only 0 out of 777 bytes of request body received, client: 123.123.123.123, server: example.com, request: "POST /xxx HTTP/2.0", host: "example.com"

What you expected to happen:

Requests should not fail with an HTTP 400 status code.

How to reproduce it:

Not sure :-/

Anything else we need to know:

This issue is not happening when I disable HTTP2 via the use-http2 configuration option.

I guess that this issue is caused by golang/go#25009 and/or by golang/go#32441.

It would be nice to keep this issue open until the related Go issues are fixed and close this after the nginx-ingress controller uses a Go version where those issues are fixed.

/kind bug

[aledbf]

@PSanetra this does not seems related to ingress-nginx? Not sure what we can do about an issue in Go.

[PSanetra]

@aledbf this issue is related to ingress-nginx as it makes using HTTP 2 with ingress-nginx unreliable.

You can use the new Go version as soon as it provides a way to fix this issue. We also don't know how the fix will look like. Maybe you will need to change some code. Therefore I suggest to keep this issue open until a fix is possible.

[aledbf]

You can use the new Go version as soon as it provides a way to fix this issue.

I am sorry, I still don't understand how this is related to the Go issue. There is no Go code in the requests, only NGINX, which is a binary, not written in Go.

[PSanetra]

ok, I am not 100% sure how ingress-nginx is implemented, but isn't there some Go code involved?
If not, why does the log message of the ingress controller contain the information "Go-http-client/2.0"?

[aledbf]

ok, I am not 100% sure how ingress-nginx is implemented, but isn't there some Go code involved?

Go is used only to communicate with the k8s API server and configure nginx. When you send traffic to the service/s defined in ingress objects, go is not involved at all, only NGINX.

"Go-http-client/2.0"?

If you see that in the log of the ingress controller pod, it means the client connecting to ingress-nginx is using a Go user-agent.

[PSanetra]

ah ok, so this is an issue of the original client.
Thanks for the clarification!