net: add multi listener impl for net.Listener #311

aroradaman · 2024-05-25T09:43:17Z

This adds an implementation of net.Listener which uses synchronous multiplexing to listen on and accept connections from multiple addrs.

What type of PR is this?
/kind feature

What this PR does / why we need it:
This PR adds an implementation of net.Listener which uses synchronous multiplexing to listen to and accept connections from multiple addresses.
Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Release note:

NONE

aroradaman · 2024-05-25T09:43:51Z

/cc @thockin @danwinship @aojea

aroradaman · 2024-05-25T09:46:06Z

For the testing part should I directly add an integration test that listens on net.InterfaceAddrs on multiple ports and then try to connect to server for validation?

aroradaman · 2024-05-25T10:07:06Z

/cc @uablrek

aroradaman · 2024-05-25T10:16:03Z

For windows, we might need a goroutine-based async multiplexer as select is not supported.

net/multi_listen_darwin.go

aojea · 2024-05-26T23:00:08Z

do we really need this?

aroradaman · 2024-05-27T04:36:43Z

do we really need this?

Currently, for v1alpha2 I spawn multiple helathz and metrics servers in go-routines, this could really help there.

Also, Tim suggested this here

uablrek · 2024-05-27T05:22:10Z

do we really need this?

My vote is on a library function because of testability. I can imagine some "what if"s that should be tested in (good) unit-tests at one place.

I also want to emphasise my "context" comment:

Also, I think your NewMultiListener should take a context as parameter. net.Listen uses ListenConfig which feels like a work-around for keeping backward compatibility.

danwinship · 2024-06-04T22:50:19Z

do we really need this?

Currently, for v1alpha2 I spawn multiple helathz and metrics servers in go-routines, this could really help there.

Also, Tim suggested this here

KEP-2438 Dual-Stack API Server needs it too

danwinship · 2024-06-04T22:52:25Z

But you really shouldn't be implementing this in terms of syscall. Use goroutines. Trust the runtime.

aojea · 2024-06-11T15:29:17Z

net/multi_listen.go

+		stopCh:       make(chan struct{}),
+	}
+
+	for _, address := range addresses {


we may need to filter duplicate addresses? or do we want to fail (I think it will fail to create a listener in the same address twice)?

I think the caller be responsible for that (not passing the same address twice).
I can't imagine a case where there is a need to listen on the same address twice, if it is passed it might be because of some bug and we should error out instead of silently ignoring the dup address, right?

In the same area: return an error if the address slice is empty. I think it works like this now: (ml, nil) ia returned, and a subsequent Accept() hangs forever. While this is logical (sort of), it's probably not what the caller intended.

net/multi_listen.go

uablrek · 2024-06-12T06:23:09Z

You must have support for context. Please see #311 (comment). I will make a better review later.

uablrek · 2024-06-12T10:03:18Z

I think the multi listener must be backward compatible. That means that a call like:

l,e := MultiListen(ctx, []string{":8080"})

must work, and listen on both families, and:

l,e := MultiListen(ctx, []string{"0.0.0.0:8080"})
// or
l,e := MultiListen(ctx, []string{"[::]:8080"})

must also listen on both families.

net/multi_listen.go

thockin · 2024-06-18T19:14:24Z

net/multi_listen.go

+			_ = ml.Close()
+			return nil, err
+		}
+		switch IPFamilyOf(ParseIPSloppy(host)) {


I think I'd rather see this as an argument, rather than try to figure it out. It seems like an opportunity to get it wrong, when we don't need to.

net/multi_listen.go

thockin · 2024-06-18T19:24:19Z

net/multi_listen.go

+	return ml, nil
+}
+
+// Accept is part of net.Listener interface.


I would explain here the specific semantics - this will wait for ANY of the sub-listeners to get a connection.

thockin · 2024-06-18T19:25:59Z

net/multi_listen.go

+var _ net.Listener = &multiListener{}
+
+// MultiListen returns net.Listener which can listen for and accept
+// TCP connections on multiple addresses.


I think this comment could be more verbose to explain the semantics

thockin · 2024-06-18T19:46:04Z

net/multi_listen.go

+
+}
+
+// Close is part of net.Listener interface.


Comment, this will close ALL of the sub-listeners

uablrek · 2024-06-19T11:35:27Z

The multi-listener will have different behavior compared to the standard Listener, as mentioned in #311 (comment). I can't see a way to make them exactly the same, but the differences must be investigated and documented. Otherwise users will assume that they are totally compatible. I can make some tests, but not this week.

Related is how TCP backlog works in Linux. I thought I knew (I didn't), and have been hit by half-open connection problems at least once (in ~20y, so it is very rare).

net/multi_listen.go

danwinship · 2024-06-22T14:53:53Z

net/multi_listen_test.go

+
+var _ net.Listener = &fakeListener{}
+
+func TestMultiListen(t *testing.T) {


I think we need some tests of accept-while-close, close-with-pending-connections, etc

and some tests of HTTP serving on a multi-listener

and some tests of HTTP serving on a multi-listener

do you mean an integration test that uses real HTTP servers?

I think it can still be a unit test (since this repo doesn't really have integration tests). Just use :0 for a random port. Create 2 or 3 such ports, pass the multi-listener to http.Serve and then connect to each of the ports that were allocated, ensuring they get the correct HTTP response.

aroradaman · 2024-06-23T07:18:03Z

Actually, I don't think you have to store it. Just create the Listerners with something like:
l,e := net.ListenConfig{}.Listen(ctx, network, address)
Then I suppose all Accept() will exit with an error if the ctx is canceled

@uablrek golang/go#50861

uablrek · 2024-06-23T15:33:51Z

Actually, I don't think you have to store it. Just create the Listerners with something like:
l,e := net.ListenConfig{}.Listen(ctx, network, address)
Then I suppose all Accept() will exit with an error if the ctx is canceled
@uablrek golang/go#50861

Is there any way to break a blocking Accept() (except close the Listener)??

danwinship · 2024-06-24T14:58:03Z

OK, so...

We could do:

MultiListen(ctx context.Context, network string, addresses []string) (net.Listener, error)

where the semantics are basically:

for i := range addresses {
        lc.Listen(ctx, network, addresses[i])
}

with no added intelligence.

If we want per-address network fields, then I think a more logical API would be drop MultiListen and instead have the caller do

ml := netutils.NewMultiListener()
err := ml.Listen(ctx, "tcp4", "10.0.0.0:80")
if err != nil ...
err = ml.Listen(ctx, "tcp6", "[::]:80")
if err != nil ...
...

I think both of these are equally unambiguous, though the latter is more flexible.

I can imagine that for some callers, being able to separately listen on "tcp4", "0.0.0.0:80" and "tcp6", "[::]:80" might make more sense given their internal organization?

Regardless of which of the above approaches we go to, the other question is: what should happen if you listen on both "tcp", "0.0.0.0:80" and "tcp", "[::]:80" (which is one of the bugs that got us here, kubernetes/kubernetes#114702)? Should MultiListener DTRT here, or should it just do literally what you asked, and end up returning an error (meaning that it's up to callers to know better than to do that).

thockin · 2024-06-24T17:27:51Z

Taking (or building) a list of individual Listener is about as flexible as gets. It's not what I meant, but I can see the appeal. Given that the primary consumer of this API will be kubernetes, I suspect that's more flexible that we need, but it seems low-cost to do it that way. I see now the argument that dual-stack is a primary motivator, so a single network argument for a list of IPs may be pointless.

WRT zero addresses, if we make the caller call Listen(), then it should be on them to not collide - we should not be too smart. Effectively this becomes an accept-mux - we need to nail those semantics.

net/multi_listen.go

This adds an implementation of net.Listener which listens on and accepts connections from multiple addresses. Signed-off-by: Daman Arora <aroradaman@gmail.com> Co-authored-by: Tim Hockin <thockin@google.com>

aroradaman · 2024-07-10T18:41:28Z

net/multi_listen.go

+				// This assumes that ANY error from Accept() will terminate the
+				// sub-listener. We could maybe be more precise, but it
+				// doesn't seem necessary.
+				terminate := err != nil


@thockin should I just return the error to the consumer instead of closing the listener here?
The behaviour of multiListener will be similar to net.LIstener if we return the error.

no, I think we need to return it up thru the channel so the main Accpet can return it

thockin

Thanks!

/lgtm
/approve

k8s-ci-robot · 2024-07-10T23:47:36Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aroradaman, thockin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [thockin]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added kind/feature Categorizes issue or PR as related to a new feature. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels May 25, 2024

k8s-ci-robot requested review from apelisse and dims May 25, 2024 09:43

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label May 25, 2024

k8s-ci-robot requested review from aojea, danwinship and thockin May 25, 2024 09:43

k8s-ci-robot requested a review from uablrek May 25, 2024 10:07

aroradaman force-pushed the net-multi-listen branch from 86f290e to 66a431e Compare May 25, 2024 10:14

aroradaman force-pushed the net-multi-listen branch 2 times, most recently from 3327c9c to 6bd4edf Compare May 25, 2024 10:55

uablrek reviewed May 25, 2024

View reviewed changes

net/multi_listen_darwin.go Outdated Show resolved Hide resolved

aroradaman force-pushed the net-multi-listen branch 2 times, most recently from 0d98132 to 19a9056 Compare June 11, 2024 14:35

aojea reviewed Jun 11, 2024

View reviewed changes

net/multi_listen.go Outdated Show resolved Hide resolved

aroradaman force-pushed the net-multi-listen branch from 19a9056 to ed136b1 Compare June 11, 2024 17:38

aojea reviewed Jun 11, 2024

View reviewed changes

net/multi_listen.go Outdated Show resolved Hide resolved

aroradaman force-pushed the net-multi-listen branch from 4108967 to 8cd5597 Compare June 17, 2024 07:14

aroradaman requested a review from uablrek June 17, 2024 07:15

thockin reviewed Jun 18, 2024

View reviewed changes

danwinship reviewed Jun 22, 2024

View reviewed changes

aroradaman force-pushed the net-multi-listen branch from 8cd5597 to 062710e Compare June 24, 2024 20:09

k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jun 24, 2024

aroradaman force-pushed the net-multi-listen branch from 062710e to 7c2ee8d Compare June 24, 2024 20:12

aroradaman requested review from danwinship and thockin June 24, 2024 20:15

thockin reviewed Jun 24, 2024

View reviewed changes

aroradaman force-pushed the net-multi-listen branch 3 times, most recently from d6eb366 to 32b2fda Compare June 29, 2024 19:27

net: add multi listener impl for net.Listener

27da63f

This adds an implementation of net.Listener which listens on and accepts connections from multiple addresses. Signed-off-by: Daman Arora <aroradaman@gmail.com> Co-authored-by: Tim Hockin <thockin@google.com>

aroradaman force-pushed the net-multi-listen branch from 32b2fda to 27da63f Compare June 29, 2024 19:31

aroradaman requested a review from thockin June 29, 2024 19:34

aroradaman commented Jul 10, 2024

View reviewed changes

thockin reviewed Jul 10, 2024

View reviewed changes

k8s-ci-robot assigned thockin Jul 10, 2024

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 10, 2024

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 10, 2024

k8s-ci-robot merged commit d4aae2b into kubernetes:master Jul 10, 2024
14 checks passed

aroradaman mentioned this pull request Jul 11, 2024

bump k8s.io/utils kubernetes/kubernetes#126018

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

net: add multi listener impl for net.Listener #311

net: add multi listener impl for net.Listener #311

aroradaman commented May 25, 2024

aroradaman commented May 25, 2024

aroradaman commented May 25, 2024

aroradaman commented May 25, 2024

aroradaman commented May 25, 2024

aojea commented May 26, 2024

aroradaman commented May 27, 2024

uablrek commented May 27, 2024

danwinship commented Jun 4, 2024

danwinship commented Jun 4, 2024

aojea Jun 11, 2024

aroradaman Jun 11, 2024

aojea Jun 12, 2024

uablrek Jun 15, 2024

uablrek commented Jun 12, 2024

uablrek commented Jun 12, 2024

thockin Jun 18, 2024

thockin Jun 18, 2024

thockin Jun 18, 2024

thockin Jun 18, 2024

uablrek commented Jun 19, 2024

danwinship Jun 22, 2024

thockin Jun 22, 2024

aroradaman Jun 24, 2024

thockin Jun 24, 2024

aroradaman commented Jun 23, 2024

uablrek commented Jun 23, 2024 •

edited

Loading

danwinship commented Jun 24, 2024

thockin commented Jun 24, 2024

aroradaman Jul 10, 2024

thockin Jul 10, 2024

thockin left a comment

k8s-ci-robot commented Jul 10, 2024


		var _ net.Listener = &fakeListener{}

		func TestMultiListen(t *testing.T) {

net: add multi listener impl for net.Listener #311

net: add multi listener impl for net.Listener #311

Conversation

aroradaman commented May 25, 2024

aroradaman commented May 25, 2024

aroradaman commented May 25, 2024

aroradaman commented May 25, 2024

aroradaman commented May 25, 2024

aojea commented May 26, 2024

aroradaman commented May 27, 2024

uablrek commented May 27, 2024

danwinship commented Jun 4, 2024

danwinship commented Jun 4, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

uablrek commented Jun 12, 2024

uablrek commented Jun 12, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

uablrek commented Jun 19, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aroradaman commented Jun 23, 2024

uablrek commented Jun 23, 2024 • edited Loading

danwinship commented Jun 24, 2024

thockin commented Jun 24, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thockin left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Jul 10, 2024

uablrek commented Jun 23, 2024 •

edited

Loading