x/net/http2: handle servers that delay sending END_STREAM in HEAD responses #53253
Comments
|
The h2_bundle.go file should have instructions on how it was generated in its header |
|
It does indeed! If I try It works, but I get a whole lot of diff lines like this showing that it doesn't believe it is part of the @@ -742,7 +743,7 @@ type http2ClientConnPool interface {
// call, so the caller should not omit it. If the caller needs
// to, ClientConn.RoundTrip can be called with a bogus
// new(http.Request) to release the stream reservation.
- GetClientConn(req *Request, addr string) (*http2ClientConn, error)
+ GetClientConn(req *http.Request, addr string) (*http2ClientConn, error)
MarkDead(*http2ClientConn)
}
I tried mucking around with the Next hint? |
|
Try |
cherrymui
added
the
NeedsInvestigation
|
cc @neild |
cherrymui
changed the title
|
It looks like this is the correct command line now @@ -2,7 +2,7 @@
// +build !nethttpomithttp2
// Code generated by golang.org/x/tools/cmd/bundle. DO NOT EDIT.
-// $ bundle -o=h2_bundle.go -prefix=http2 -tags=!nethttpomithttp2 golang.org/x/net/http2
+// $ bundle -dst net/http -o=h2_bundle.go -prefix=http2 -tags=!nethttpomithttp2 golang.org/x/net/http2
// Package http2 implements the HTTP/2 protocol.
//However I can't get it to check out any other version than latest which is scuppering my plans to try a bisect Any ideas? |
|
the version should be controlled by |
|
Though it may have been easier for your project to do use http2.ConfigureServer instead of changing the bundled version? |
|
For the benefit of those who come after I used t.TLSNextProto = nil
err := http2.ConfigureTransport(t)
if err != nil {
log.Fatal(err)
}And used a And that make it a whole lot easier to bisect - thanks for the hint @seankhliao The commit which causes the regression is this one by @neild Unfortunately it is a large commit... |
|
I've been trying to pin down exactly what the slowdown is. It seems that each HTTP request / response has a roughly 5 second time penalty. I've managed to make a little reproduction of this which you can run yourselves. I put the code in a gist also (along with go.mod and go.sum): https://gist.github.com/ncw/016e5ccdc69000225b03db43e00f8c3a package main
import (
"flag"
"fmt"
"io"
"log"
"net/http"
"os"
"time"
"golang.org/x/net/http2"
)
func get(url string) {
t0 := time.Now()
resp, err := http.Head(url)
dt := time.Since(t0)
if err != nil {
log.Printf("Failed to fetch %q: %v", url, err)
return
}
defer resp.Body.Close()
_, err = io.Copy(os.Stdout, resp.Body)
if err != nil && err != io.EOF {
log.Printf("Failed to read from %q: %v", url, err)
return
}
fmt.Println()
log.Printf("%q: read OK", url)
log.Printf("%q: request -> response took %v", url, dt)
}
func main() {
flag.Parse()
// Use the specific version of x/net/http2 in go.mod rather
// than the one bundled with the go compiler
t := http.DefaultTransport.(*http.Transport)
t.TLSNextProto = nil
err := http2.ConfigureTransport(t)
if err != nil {
log.Fatal(err)
}
for _, arg := range flag.Args() {
get(arg)
get(arg)
get(arg)
get(arg)
get(arg)
}
}If you run this at x/net Whereas if you run it at x/net Hopefully this is enough info for you to reproduce and investigate. |
|
Just turning on GODEBUG highlights the difference: require golang.org/x/net v0.0.0-20210929193557-e81a3d93ecf6 require golang.org/x/net v0.0.0-20220607020251-c690dde0001d |
|
It looks more like the current http2 code is correct and expects an END_STREAM on the final HEADERS frame which isn't happening. |
|
When I upgraded to go1.18.3, I also had problems with tls speed, but go1.18.2 was fine |
|
Thanks for the nice reproduction case. It looks like the problem is, as @fraenkel observes, that the client is waiting for an END_STREAM that the server isn't sending. Since this is a HEAD request, and responses to HEAD requests aren't allowed to contain a body, there's no reason for the server not to immediately send an END_STREAM. I don't know why it isn't. The Go HTTP/2 client is waiting for the stream to be cleaned up before returning from https://go.googlesource.com/net/+/refs/heads/master/http2/transport.go#1164 I don't remember exactly why this wait is here. It does ensure that in the simple case of a request and response with no body, the request stream has been cleaned up before Possible options:
I'm leaning towards the second option (reset the stream). |
|
Change https://go.dev/cl/411474 mentions this issue: |
|
I just noticed This seems like it might be a bug in curl though as the debug says it is waiting for 6 more bytes (the I'll try reporting this to Scaleway also now I have a curl repro. |
|
Interesting, --head completes in .763s and -X hEAD finishes in 5s. |
|
I think the I made a similar object on AWS s3 if you want to compare So I don't think you can reproduce the problem with |
|
@neild What is the current status here? This issue is currently in the 1.19 milestone. Should it move to 1.20? To Backlog? Thanks. |
|
This can move to 1.20; the condition has existed since 1.18 and is limited to interactions with buggy (IMO) servers. We should still put in a workaround. |
|
Change https://go.dev/cl/415454 mentions this issue: |
|
It's not obvious to me what the right fix here is, or if we should change anything at all in the Go HTTP/2 client. I tested a few major websites to see how they respond to HTTP/2 HEAD requests. www.google.com sends a combined END_HEADERS+END_STREAM frame, but www.apache.org, www.amazon.com, and www.microsoft.com all send a frame with END_HEADERS immediately followed by a frame with END_STREAM. Nobody I can find (aside from Scaleway) delays between END_HEADERS and END_STREAM, which I still think seems like a clear bug. (AWS S3 doesn't appear to directly support HTTP/2, by the way, so I can't directly compare it.) It is important that a request stream be closed by the time we return from Things we could do:
|
|
Regarding curl: I haven't looked into curl's source to see exactly what it's doing, but it looks like An HTTP/2 connection can support some number of simultaneous streams, where each stream corresponds to an HTTP request. HTTP/2 clients must take care not to exceed the maximum number of streams permitted on a connection. An HTTP/2 stream ends either when both endpoints send a frame with the Scaleway's HTTP/2 server is responding to HEAD requests by sending response headers, pausing for several seconds, and finally sending a frame with the A client could note that the server has finished responding to the HEAD request without closing the stream and send a The closest thing we could do to matching curl's behavior is to return immediately from So I don't think it's accurate to say that curl has a workaround, and it is not feasible for |
neild
changed the title
|
Hello 👋 So indeed it looks like there is an "issue" on our (Scaleway) S3 endpoint. I'll take a look and see what can be done. However, a |
|
The problem with sending a |
|
True! I might have a fix though, still need to dig a bit but it should fix this issue. Can't really say for sure when it'll go to production, but hopefully in the next 2 months! |
|
Do we think this could be the root cause for this nasty gRPC bug[1]? cc @neild |
AFAIK, the Go gRPC implementation shares no code with |
|
Hello! This is now fixed on Scaleway's S3 I think the issue can be closed! |
What version of Go are you using (
go version)?Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (
go env)?go envOutputWhat did you do?
A user reported a performance regression between rclone v1.57 and v1.58 in the forum.
I ascertained that this wasn't a regression in rclone, but turned out to be a regression in the go standard library from
go1.17togo1.18. I managed to bisect the problem to this commit:7109323 is the first bad commit from #48564 and #23559
It turns out that this http2 upgrade tanked the performance of rclone with this HTTP2 server. I haven't measured exactly the performance but it seems at least 10 times slower per file uploaded by rclone. (Edit it seems to add 5 seconds of latency to each HTTP roundtrip - see later).
I'm not really sure why though, and I need some help investigating further. I tried turning on
GODEBUG=http2debug=2and I couldn't see anything obvious (no error messages). I can post these, but I need to sanitize tokens out of them first.I could bisect the
golang.org/x/netbundle commit further, but I don't know how to gluex/netinto the go source - a hint here would be appreciated too!The text was updated successfully, but these errors were encountered: