Uneven incoming connection distribution to worker processes with SO_REUSEPORT

[ghost]

The usage for option SO_REUSEPORT was recently updated in
9eb1bc9

However, there is an additional issue with using the specific option in the current-code flow which affect high volume traffic sites being serviced by Hitch.

The initial patch in Linux kernel 3.9 for the said option SO_REUSEPORT does explain the intended use to be a case where we can have multiple (unrelated) processes bind to the same IP:PORT pair if and only if the first process chooses to create the first binding with this specific option enabled and the connection would be distributed among the interest processes based on a hash of source/destination ip:port pairs.

The behavior observed in Debian (Jessie/Whezzy) is that it leads to uneven distribution (pretty skewed) of incoming connection to different worker processes, with the initial/first process get 'most' of the connection (till the RX queue associated with the first core gets full)

[ghost]

Submitted pull request #144 to move front-end listener socket initialisation to individual worker processes. At a beta deployment in production this change seems to have resolved the issue related to uneven distribution of incoming connection across Hitch worker processes.

Took the hint for this change from https://lwn.net/Articles/542629/

"Multiple servers (processes or threads) can bind to the same port if they each set the option as follows:
int sfd = socket(domain, socktype, 0);
int optval = 1;
setsockopt(sfd, SOL_SOCKET, SO_REUSEPORT, &optval, sizeof(optval));
bind(sfd, (struct sockaddr *) &addr, addrlen);"

In order to appropriately use SO_REUSEPORT each worker/thread should have a different socket-handle, this does move away from the current Hitch implementation where we had a socket initialised and bonded only once in master process.

[ghost]

Any chance of this particular change also being pushed upstream.

[p-id]

Thanks for taking this up.

I can confirm that this does indeed work and help a bit in current stable Debian derivative production environment since past year or so...