Skip to content

Commit

Permalink
consul/connect: avoid warn messages on connect proxy errors
Browse files Browse the repository at this point in the history
When creating a TCP proxy bridge for Connect tasks, we are at the
mercy of either end for managing the connection state. For long
lived gRPC connections the proxy could reasonably expect to stay
open until the context was cancelled. For the HTTP connections used
by connect native tasks, we experience connection disconnects.
The proxy gets recreated as needed on follow up requests, however
we also emit a WARN log when the connection is broken. This PR
lowers the WARN to a TRACE, because these disconnects are to be
expected.

Ideally we would be able to proxy at the HTTP layer, however Consul
or the connect native task could be configured to expect mTLS, preventing
Nomad from MiTM the requests.

We also can't mange the proxy lifecycle more intelligently, because
we have no control over the HTTP client or server and how they wish
to manage connection state.

What we have now works, it's just noisy.

Fixes #10933
  • Loading branch information
jrasell committed Aug 5, 2021
1 parent 62057de commit a946419
Show file tree
Hide file tree
Showing 2 changed files with 10 additions and 8 deletions.
3 changes: 3 additions & 0 deletions .changelog/10951.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
```release-note:improvement
consul/connect: Reduced the noise of log messages emitted for connect native tasks
```
15 changes: 7 additions & 8 deletions client/allocrunner/consul_grpc_sock_hook.go
Original file line number Diff line number Diff line change
Expand Up @@ -313,10 +313,10 @@ func proxyConn(ctx context.Context, logger hclog.Logger, destAddr string, conn n
defer cancel()
n, err := io.Copy(dest, conn)
if ctx.Err() == nil && err != nil {
logger.Warn("error proxying to Consul", "error", err, "dest", destAddr,
"src_local", conn.LocalAddr(), "src_remote", conn.RemoteAddr(),
"bytes", n,
)
// expect disconnects when proxying http
logger.Trace("error message received proxying to Consul",
"msg", err, "dest", destAddr, "src_local", conn.LocalAddr(),
"src_remote", conn.RemoteAddr(), "bytes", n)
return
}
logger.Trace("proxy to Consul complete",
Expand All @@ -332,10 +332,9 @@ func proxyConn(ctx context.Context, logger hclog.Logger, destAddr string, conn n
defer cancel()
n, err := io.Copy(conn, dest)
if ctx.Err() == nil && err != nil {
logger.Warn("error proxying from Consul", "error", err, "dest", destAddr,
"src_local", conn.LocalAddr(), "src_remote", conn.RemoteAddr(),
"bytes", n,
)
logger.Trace("error message received proxying from Consul",
"msg", err, "dest", destAddr, "src_local", conn.LocalAddr(),
"src_remote", conn.RemoteAddr(), "bytes", n)
return
}
logger.Trace("proxy from Consul complete",
Expand Down

0 comments on commit a946419

Please sign in to comment.