encoding/json: Encoder.Encode newlines can cause issues with Go HTTP clients

[johnmaguire]

What version of Go are you using (go version)?

$ go version
go version go1.19 darwin/arm64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="arm64"
GOBIN=""
GOCACHE="/Users/jmaguire/Library/Caches/go-build"
GOENV="/Users/jmaguire/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/jmaguire/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/jmaguire/go"
GOPRIVATE="github.com/DefinedNet"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/opt/homebrew/Cellar/go/1.19/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/opt/homebrew/Cellar/go/1.19/libexec/pkg/tool/darwin_arm64"
GOVCS=""
GOVERSION="go1.19"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/2t/rbxbv8612sq9_rdrxhtw8rzc0000gn/T/go-build1461363701=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Filing this task primarily as documentation in case someone else runs into this.

encoding/json.Encoder.Encode writes a JSON message plus a trailing newline, as described in the docs.

encoding/json.Decoder.Decode reads a JSON message from a buffer, as described.

net/http.Client.Do states that the Body must both be read to EOF and closed in order for a connection to be reused. If this does not occur the next request to Do for that server will fail with an error.

If the Body is not both read to EOF and closed, the Client's underlying RoundTripper (typically Transport) may not be able to re-use a persistent TCP connection to the server for a subsequent "keep-alive" request.

Looking at the implementation details of Decode, it will read 512 bytes at a time. This means that if the buffer contains a single JSON message which has a byte length that is a multiple of 512, Decoder will read the JSON message and leave the trailing newline: https://go.dev/play/p/ZHrpoVGlizA

If an HTTP client uses Decode to parse a JSON message returned by a server it will occasionally fail due to the body not being read to EOF.

What did you expect to see?

No errors returned from subsequent HTTP client requests.

What did you see instead?

Intermittent EOF errors.

[mvdan]

For the newlines, see #37083.

Sounds like you should make sure you decode the entire body rather than just one JSON value. For that, see #36225.

[johnmaguire]

Sounds like you should make sure you decode the entire body rather than just one JSON value. For that, see #36225.

Thanks - this is the approach we've taken to solve the problem. I appreciate the linked issue, it speaks to exactly this issue. I filed this task in hopes of helping the next person who Googles for intermittent EOF errors during HTTP requests as it's the intersection of these 3 common APIs which created the problem.

[mvdan]

Thanks for confirming. I'll close this as a duplicate of #36225, then.