4xx and Unexpected EOF

[mwpolcik]

Hey everyone, I started to use fasthttp in one of my projects and I am seeing quite often this error:

error when serving connection "172.17.0.2:80"<->"10.192.10.31:48432": unexpected EOF

To give you more context:

1. Application is sitting behind AWS Application load balancer with idle timeout set to 60s.
2. App is sometimes returning 4xx codes when unexpected EOF occurs.
3. Server config:

func getServer(handlerFunc func(ctx *fasthttp.RequestCtx)) *fasthttp.Server {
	return &fasthttp.Server{
		Handler:            cors(handlerFunc),
		ReadTimeout:        10 * time.Second,
		WriteTimeout:       10 * time.Second,
		IdleTimeout:        15 * time.Second,
		TCPKeepalive:       true,
		TCPKeepalivePeriod: 30 * time.Second,
		LogAllErrors:       false,
		DisableKeepalive:   false,
		Name:               "my-server",
	}
}

4. Handler logic on each endpoint is quite similar:

type processArgs struct {
	Body          []byte
	UserAgent     []byte
	XForwardedFor []byte
}

func newProcessArgs(ctx *fasthttp.RequestCtx) *processArgs {
	args := &processArgs{}
	args.Body = append(args.Body, ctx.PostBody()...)
	args.UserAgent = append(args.UserAgent, ctx.UserAgent()...)
	args.XForwardedFor = append(args.XForwardedFor, ctx.Request.Header.Peek(fasthttp.HeaderXForwardedFor)...)
	return args
}

// Handler handles session request
func (h Handler) Handler(ctx *fasthttp.RequestCtx) {
	args := newProcessArgs(ctx)
	go h.process(args)
}

I am reading what I need from request, start go routine to process data and return right away.

I was trying different configurations for server, but I am still getting unexpected EOF and I am not sure what I am doing wrong here. Any help appreciated.

[dgrr]

The handler is leaving just after you start the goroutine. I don't know what do you mean with return right away, but if your h.process call expects to send data to the user you must wait for the goroutine to finish. Anyway, that should not cause the unexpected EOF error. The error should be caused because the request is malformed or something like that.

Maybe you can try by adding a proxy in between AWS and fasthttp to log the requests.

[mwpolcik]

I don't want to send anything to the user, i just need to send back 200. I get data for processing, but client does not need to know anything about what happened later.

[dgrr]

I remember a user having troubles with AWS using keepalive messages. Try disabling the keepalive.

[mwpolcik]

I have seen this thread, but I really would like to avoid disabling keepalive as this would kill performance. Still, I will try this and see what happens. Thanks.

[erikdubbelboer]

Sounds like this is the AWS loadbalancer closing the connection while fasthttp is trying to read the next request.

Are you able to modify the fasthttp source you compile with to test something for me?

I'm thinking this line:

fasthttp/server.go

Line 1900 in be93fbd

if err != io.EOF {

should probably be:

if err != io.EOF && err != io.ErrUnexpectedEOF {

Or if that doesn't fix it I'm wondering if the AWS loadbalancer maybe sometimes opens a connection and then closes it again without sending a request.

[erikdubbelboer]

Try this branch if you can: https://github.com/valyala/fasthttp/tree/aws-unexpected-eof

[mwpolcik]

Thanks, I will try to release this as fast as I can and get back to you. As I can't replicate this behavior locally, I will need to verify this on live system.

[mwpolcik]

Hey @erikdubbelboer , I have release version based on your branch and I am still getting this:

2019/10/01 06:48:10 error when serving connection "172.17.0.3:80"<->"10.192.10.31:3960" after 509.322864ms: unexpected EOF
2019/10/01 06:48:33 error when serving connection "172.17.0.3:80"<->"10.192.11.142:36916" after 6.651099834s: unexpected EOF
2019/10/01 06:48:44 error when serving connection "172.17.0.3:80"<->"10.192.11.142:37214" after 4.581654206s: unexpected EOF
2019/10/01 06:48:47 error when serving connection "172.17.0.3:80"<->"10.192.10.31:4202" after 15.721600213s: unexpected EOF

In case there is no easy fix for that is there a way to suppress this errors ?

[erikdubbelboer]

Sorry for the slow reply.

Interesting to see that the connection gets closed with EOF so soon already. I'm thinking the loadbalancer closes connections that haven't seen any requests yet. We ignore EOF errors if it's after the first request as that will just be a keep-alive connection closing. But we print this error if the connection is closed before any request arrives at all.

I have updated https://github.com/valyala/fasthttp/tree/aws-unexpected-eof once more. Would you mind testing it another time?

[kevburnsjr]

Judging by the frequency of this error in @mwpolcik's logs (4+/min), I'm going to wager that this is a high traffic public facing API (>100krpm).

Theory 1

• Client begins a request to ALB from cellphone while driving into a tunnel.
• ALB streams headers and partial body to backend but can't finish sending the body.
• Backend waits for rest of body but it never arrives.
• ALB closes the connection.

Theory 2

• Some client is sending invalid content-length headers to ALB.
• ALB doesn't sanitize content-length.

---

See PR #682

[mwpolcik]

Hey @erikdubbelboer and @kevburnsjr . Sorry for not updating this issue earlier, got caught up. I think in most cases Theory 1 is correct. After some more investigations we were able to find that most of the requests that are getting 4xx are in most cases malformed in some way. For now we've tweaked a little bit our general configuration:

1. idle timeout on server to be higher than idle timeout on ALB. In my case server: 120s and 60s on ALB. Which frankly is just common sense, but we've missed this.
2. What made also huge drop for Unexpected EOF is not passing unwanted requests to server - we are now using ALB also as reverse-proxy and only passing what we need.

@kevburnsjr you are right about nature of the application. We are getting 6000 QPS right now and with the above configuration we are expecting to double this number without any problems as logs are quite clean right now.

In short - we can close this issue and anyone who will have similar issues can learn from our mistakes.

[erikdubbelboer]

@mwpolcik just wondering, are you still using the aws-unexpected-eof branch or did you switch to master?