TCPWriter: Close connections on network timeout errors #267
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Generate changelog in
|
tabboud
commented
Mar 16, 2021
| @@ -106,7 +106,7 @@ func (d *TCPWriter) Write(p []byte) (n int, err error) { | |||
| n, err := conn.Write(envelope[total:]) | |||
| total += n | |||
| if err != nil { | |||
| if nerr, ok := err.(net.Error); !(ok && (nerr.Temporary() || nerr.Timeout())) { | |||
| if nerr, ok := err.(net.Error); !ok || !nerr.Temporary() { | |||
There was a problem hiding this comment.
I am slightly nervous that a timeout error could also be considered temporary, but looking at the tls.Conn code it doesn't seem like that is the case. Mind double checking this logic?
There was a problem hiding this comment.
I just went ahead and added the explicit check for a timeout as well to be safe.
bmoylan
approved these changes
Mar 17, 2021
There was a problem hiding this comment.
Reviewed 2 of 3 files at r2, 1 of 1 files at r3.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @tabboud)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Before this PR
Connections could get stuck in a "connection timed out" state and the underlying net.Conn would never be reset causing logs to be dropped.
According to the docs for
tls.Conn.SetDeadline()(https://pkg.go.dev/crypto/tls#Conn.SetDeadline), after a write has timed out once, the TLS state is corrupt and thus the same error is returned. My hypothesis is that once we hit the timeout, we got stuck in this state since we never closed the connection.After this PR
==COMMIT_MSG==
Close TCP connections if there are network timeouts
==COMMIT_MSG==
Notes
I wasn't able to repro the exact issues we were seeing below, since the
connection timed outerror comes from the ETIMEDOUT syscall on Linux.However when digging into
syscall.ETIMEDOUT, I realized it happens to implement thenet.Errorinterface and returns true forTimeout()and ALSO returns true forTemporary()if there was a timeout!Relevant issue: golang/go#31449
Possible downsides?
This change is