Enable TCP keepalive on listener sockets. #2638

erwbgy · 2020-06-29T07:18:48Z

Fixes #2633
Signed-off-by: Keith Burdis keith.burdis@gs.com

codecov · 2020-06-29T07:22:50Z

Codecov Report

Merging #2638 into master will increase coverage by 0.11%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #2638      +/-   ##
==========================================
+ Coverage   76.77%   76.88%   +0.11%     
==========================================
  Files          71       72       +1     
  Lines        5674     5702      +28     
==========================================
+ Hits         4356     4384      +28     
  Misses       1227     1227              
  Partials       91       91

Impacted Files	Coverage Δ
internal/envoy/listener.go	`97.73% <100.00%> (+<0.01%)`	⬆️
internal/envoy/stats.go	`100.00% <100.00%> (ø)`
internal/envoy/tcp_keepalive.go	`100.00% <100.00%> (ø)`

jpeach

This looks very reasonable to me (barring a couple of style nits). I'll leave it open for other maintainers to review, however.

I'm not too concerned about compatibility, since all the socket options are documented as being from Linux 2.2 or 2.4 days. Hopefully no-one is running kernels that old. The values seem sensible to me.

I would like to see this documented, though I'll cop to there not currently being an obvious place to put this documentation. Let's at least file an issue to remember that we should add docs.

internal/envoy/tcp_keepalive.go

erwbgy · 2020-07-04T14:09:55Z

@jpeach @youngnick I saw some discussion on Slack about putting this behind a feature flag because these socket options don't work on Windows. Another way would be to check runtime.GOOS and only set the socket options on Linux, which should be a simple update to this pull request. Thoughts on that?

erwbgy · 2020-07-04T14:13:13Z

@pims Did you get a chance to test if this works on MacOS?

pims · 2020-07-04T14:18:08Z

@erwbgy I should be able to test this weekend. I’ll update the PR.

I think build flags might also be a good way to gate this without a dedicated contour flag.

pims · 2020-07-04T15:02:33Z

Hmm, so this doesn't build on MacOS:

internal/envoy/tcp_keepalive.go:40:17: undefined: syscall.TCP_KEEPIDLE
internal/envoy/tcp_keepalive.go:48:17: undefined: syscall.TCP_KEEPINTVL
internal/envoy/tcp_keepalive.go:58:17: undefined: syscall.TCP_KEEPCNT

if we rename tcp_keepalive.go to tcp_keepalive_linux.go and add the //+build linux directive
as well as create a tcp_keepalive.go with the following:

package envoy

import (
	envoy_api_v2_core "github.com/envoyproxy/go-control-plane/envoy/api/v2/core"
)

func TCPKeepaliveSocketOptions() []*envoy_api_v2_core.SocketOption {
    // should we return nil instead?
    return []*envoy_api_v2_core.SocketOption{}
}

it builds correctly. I still need to test it though.

erwbgy · 2020-07-04T18:46:46Z

Thanks @pims, learnt something new today as I hadn't had the need for build constraints before.

jpeach · 2020-07-05T01:52:41Z

Hmm, so this doesn't build on MacOS:
internal/envoy/tcp_keepalive.go:40:17: undefined: syscall.TCP_KEEPIDLE
internal/envoy/tcp_keepalive.go:48:17: undefined: syscall.TCP_KEEPINTVL
internal/envoy/tcp_keepalive.go:58:17: undefined: syscall.TCP_KEEPCNT
if we rename tcp_keepalive.go to tcp_keepalive_linux.go and add the //+build linux directive
as well as create a tcp_keepalive.go with the following:
package envoy

import (
	envoy_api_v2_core "github.com/envoyproxy/go-control-plane/envoy/api/v2/core"
)

func TCPKeepaliveSocketOptions() []*envoy_api_v2_core.SocketOption {
    // should we return nil instead?
    return []*envoy_api_v2_core.SocketOption{}
}
it builds correctly. I still need to test it though.

Sorry, I should have picked up this issue during review :(

What we should do is carry our own constants for the socket options. If we conditionally compile the whole of the keep alive options file, then Envoy would get programmed differently depending on what platform Contour is running on, which is not the result that we want.

We could use +build to define the socket option number, but since we will need to hard-code them for non-linux platforms, I'm not sure there is much benefit in that. Basically what we need is a set of constants per target Envoy platform.

jpeach

As discussed in main thread, Contour should program Envoy the same way, irrespective of the platform Contour was built on.

pims · 2020-07-05T05:01:40Z

🤷 syscall.TCP_KEEPINTVL is available on // +build aix freebsd linux netbsd which I'd guess would cover 99.9% of use cases of Contour/k8s + Envoy.
My main concern was the ability to build/run contour on MacOS for development purposes.

pims · 2020-07-05T05:25:02Z

#define	TCP_KEEPINTVL		0x101	/* interval between keepalives */
#define	TCP_KEEPCNT		0x102	/* number of keepalives before close */

from https://opensource.apple.com/source/xnu/xnu-2422.115.4/bsd/netinet/tcp.h

Couldn't find references to TCP_KEEPIDLE on MacOS but I could have missed it.

Edit:

	TCP_KEEPALIVE                     = 0x10
	TCP_KEEPCNT                       = 0x102
	TCP_KEEPINTVL                     = 0x101

from https://github.com/golang/go/blob/50bd1c4d4eb4fac8ddeb5f063c099daccfb71b26/src/syscall/zerrors_darwin_arm64.go

erwbgy · 2020-07-05T08:51:59Z

Looks like we can get these values from https://github.com/felixge/tcpkeepalive/.

erwbgy · 2020-07-05T10:01:52Z

Added constants for the TCP keepalive options, defaulting to the syscall values for Linux and BSD with MacOS (Darwin) being different. Apparently MacOS calls the TCP_KEEPIDLE socket option TCP_KEEPALIVE.

The GOOS values not covered are aix, android, illumos, js, plan9, solaris, windows. Do we need to be concerned about those?

jpeach · 2020-07-05T23:17:22Z

Added constants for the TCP keepalive options, defaulting to the syscall values for Linux and BSD with MacOS (Darwin) being different. Apparently MacOS calls the TCP_KEEPIDLE socket option TCP_KEEPALIVE.

Remember that the socket options are being used to program Envoy, so we need to send the right options for the platform that Envoy is running on, not the ones for the platform that Contour is running on. There's no in-band way to detect the Envoy platform AFAIK, so the only option is to assume that it is running on Linux (which would be true in the majority of cases).

erwbgy · 2020-07-06T09:11:22Z

Is it not valid to assume that Contour and Envoy would run on the same platform?

Do we need to support any platforms other than Linux? For development or runtime or both?

jpeach · 2020-07-07T01:27:48Z

Is it not valid to assume that Contour and Envoy would run on the same platform?

Not always. The main case is developer usage, where you may run a working copy of Contour on macOS and Envoy in a kind cluster.

Do we need to support any platforms other than Linux? For development or runtime or both?

For Contour, we should support macOS as a developer convenience. I don't know of any use case for supporting Envoy on non-Linux.

erwbgy · 2020-07-08T07:31:40Z

Thanks @jpeach I think I've got it now. We only support Envoy on Linux so always configure Linux TCP keep-alive socket options regardless of the platform that Contour is running on.

stevesloka

This lgtm with just one thought on docs & configuring.

internal/envoy/tcp_keepalive.go

jpeach

Please hold until I get a chance to do a final review pass. Thanks.

jpeach

This looks good; we just need to rethink the docs.

Please add the keepalive options to the Listener returned by envoy.StatsListener().

Can you please rebase and squash the commits. Add a commit message that discusses the important aspects of the change for the benefit of future maintainers.

site/_resources/envoy.md

erwbgy · 2020-07-15T16:12:14Z

Rebased and squashed the commits with a single commit message.

…stener sockets. With these changes Contour configures TCP keepalives for Envoy listener sockets so that half-open connections with clients (downstream in Envoy speak) are flushed which helps Envoy work better with networks that drop long running idle TCP connections. Once a TCP connection has been idle for 45 seconds a keepalive packet is sent every 5 seconds and if 9 packets receive no response then the connection will be terminated by Envoy. Envoy is only supported on Linux so Contour always configures Linux TCP keep-alive socket options regardless of the platform that Contour is running on. Fixes #2633 Signed-off-by: Keith Burdis <keith.burdis@gs.com>

jpeach reviewed Jun 30, 2020

View reviewed changes

internal/envoy/tcp_keepalive.go Outdated Show resolved Hide resolved

internal/envoy/tcp_keepalive.go Outdated Show resolved Hide resolved

jpeach mentioned this pull request Jul 5, 2020

Add macOS build checks to CI #2651

Closed

jpeach suggested changes Jul 5, 2020

View reviewed changes

jpeach mentioned this pull request Jul 5, 2020

Add configuration support to tune or disable listener Keepalives. #2652

Open

stevesloka reviewed Jul 8, 2020

View reviewed changes

internal/envoy/tcp_keepalive.go Show resolved Hide resolved

jpeach suggested changes Jul 8, 2020

View reviewed changes

jpeach suggested changes Jul 13, 2020

View reviewed changes

site/_resources/envoy.md Outdated Show resolved Hide resolved

erwbgy changed the base branch from master to gh-pages July 15, 2020 15:54

erwbgy changed the base branch from gh-pages to master July 15, 2020 15:55

erwbgy added 2 commits July 16, 2020 17:24

Merge branch 'master' into master

a36fd97

jpeach added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Jul 17, 2020

jpeach approved these changes Jul 17, 2020

View reviewed changes

jpeach merged commit e7f3456 into projectcontour:master Jul 17, 2020

stevesloka mentioned this pull request Jul 24, 2020

TCP Socket Keepalives running on osx not working #2729

Closed

youngnick mentioned this pull request Aug 12, 2020

Ability to set keepalive timings on downstream connection #2788

Closed

egerkke mentioned this pull request Jul 1, 2022

Make configurable Envoy listener socketOptions to allow DSCP marking for QoS purposes #4605

Closed

kfaseela mentioned this pull request Aug 11, 2022

Support DSCP Marking on external interfaces for QoS purposes istio/istio#40397

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable TCP keepalive on listener sockets. #2638

Enable TCP keepalive on listener sockets. #2638

erwbgy commented Jun 29, 2020

codecov bot commented Jun 29, 2020 •

edited

Loading

jpeach left a comment

erwbgy commented Jul 4, 2020

erwbgy commented Jul 4, 2020

pims commented Jul 4, 2020

pims commented Jul 4, 2020

erwbgy commented Jul 4, 2020

jpeach commented Jul 5, 2020

jpeach left a comment

pims commented Jul 5, 2020

pims commented Jul 5, 2020 •

edited

Loading

erwbgy commented Jul 5, 2020

erwbgy commented Jul 5, 2020

jpeach commented Jul 5, 2020

erwbgy commented Jul 6, 2020

jpeach commented Jul 7, 2020

erwbgy commented Jul 8, 2020

stevesloka left a comment

jpeach left a comment

jpeach left a comment

erwbgy commented Jul 15, 2020

Enable TCP keepalive on listener sockets. #2638

Enable TCP keepalive on listener sockets. #2638

Conversation

erwbgy commented Jun 29, 2020

codecov bot commented Jun 29, 2020 • edited Loading

Codecov Report

jpeach left a comment

Choose a reason for hiding this comment

erwbgy commented Jul 4, 2020

erwbgy commented Jul 4, 2020

pims commented Jul 4, 2020

pims commented Jul 4, 2020

erwbgy commented Jul 4, 2020

jpeach commented Jul 5, 2020

jpeach left a comment

Choose a reason for hiding this comment

pims commented Jul 5, 2020

pims commented Jul 5, 2020 • edited Loading

erwbgy commented Jul 5, 2020

erwbgy commented Jul 5, 2020

jpeach commented Jul 5, 2020

erwbgy commented Jul 6, 2020

jpeach commented Jul 7, 2020

erwbgy commented Jul 8, 2020

stevesloka left a comment

Choose a reason for hiding this comment

jpeach left a comment

Choose a reason for hiding this comment

jpeach left a comment

Choose a reason for hiding this comment

erwbgy commented Jul 15, 2020

codecov bot commented Jun 29, 2020 •

edited

Loading

pims commented Jul 5, 2020 •

edited

Loading