use SO_RCVBUFFORCE to increase receive buffer size

[marten-seemann]

SO_RCVBUFFORCE (since Linux 2.6.14): Using this socket option, a privileged (CAP_NET_ADMIN) process can perform the same task as SO_RCVBUF, but the rmem_max limit can be overridden.

[MarcoPolo]

I'm interested in picking this up. 🧑‍🏭

[otbutz]

@marten-seemann could you add a Linux section to https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes which explains the CAP_NET_ADMIN behaviour?

[marten-seemann]

@otbutz Can you suggest some text?

[otbutz]

As of quic-go v0.19.x, you might see warnings about the receive and send buffer sizes.

Experiments have shown that QUIC transfers on high-bandwidth connections can be limited by the size of the UDP receive and send buffer. The receive buffer holds packets that have been received by the kernel, but not yet read by the application (quic-go in this case). The send buffer holds packets that have been sent by quic-go, but not sent out by the kernel. In both cases, once these buffers fill up, the kernel will drop any new incoming packet.

Therefore, quic-go tries to increase the buffer size. The way to do this is an OS-specific, and we currently have an implementation for linux, windows and darwin. However, an application is only allowed to do increase the buffer size up to a maximum value set in the kernel. Unfortunately, on Linux this value is rather small, too small for high-bandwidth QUIC transfers.

Linux

If the process has the CAP_NET_ADMIN privilege, quic-go is able to override the maximum buffer size enforced by the kernel on its own. If this is not possible it is recommended to increase the maximum buffer size by running:

sysctl -w net.core.rmem_max=2621440
sysctl -w net.core.wmem_max=2621440

These commands would increase the maximum receive and the send buffer size to 2.5 MiB.

To make these changes persistent across reboots, you should create a sysctl config like /etc/sysctl.d/60-quic.conf with the following content:

net.core.rmem_max=2621440
net.core.wmem_max=2621440

BSD

Taken from: https://medium.com/@CameronSparr/increase-os-udp-buffers-to-improve-performance-51d167bb1360

On BSD/Darwin systems you need to add about a 15% padding to the kernel limit socket buffer. Meaning if you want a 25MB buffer (8388608 bytes) you need to set the kernel limit to 26214400*1.15 = 30146560. This is not documented anywhere but happens in the kernel here.

To update the value immediately to 2.5M, type the following commands as root:

sysctl -w kern.ipc.maxsockbuf=3014656

Add the following lines to the /etc/sysctl.conf file to keep this setting across reboots:

kern.ipc.maxsockbuf=3014656

[otbutz]

Do we need a section for Windows?

e.g https://lcm-proj.github.io/lcm/content/multicast-setup.html#kernel-udp-receive-buffer-sizing