Communication problems (GSO/IO) between lite-rpc and quic-forward-proxy on IPv6 private network

[riordanp]

When deploying lite-rpc and quic-forward-proxy to Fly on separate VMs, we noticed submitted TXs timing out.
After some debugging, it seems this call to write_all on the SendStream can produce this error:

sendmsg error: Os { code: 22, kind: InvalidInput, message: "Invalid argument" }, Transmit: { destination: [fdaa:0:a53d:a7b:17b:3cd9:fe0f:2]:11111, src_ip: None, enc: Some(Ect0), len: 1291, segment_size: Some(1200) }

After which the future never returns, blocking any further messages from the broadcast_receiver being processed. After restarting lite-rpc maybe one or two batches are successfully forwarded before this error occurs and no more batches are forwarded.

Adding a timeout to this future allows sending to continue, but produces the following error logs in quic-forward-proxy:

ERROR handle_client_connection: solana_lite_rpc_quic_forward_proxy::inbound::proxy_listener: failed to accept stream: timed out
thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: Read(ConnectionLost(TimedOut))', quic-forward-proxy/src/inbound/proxy_listener.rs:106:85
ERROR solana_lite_rpc_quic_forward_proxy::inbound::proxy_listener: failed handling connection from client: error accepting stream - skip

Tests on IPv4 and local IPv6 outside of the Fly network have been successful, pointing to an incompatibility with the Fly IPv6 private network, or kernel network parameters on the VMs.

[grooviegermanikus]

sendmsg is implemented in quinn-udp to allow for large package sizes

we suspect that the issue is related to Generic Segmentation Offload (GSO) available in recent OS's

[grooviegermanikus]

Quinn added opt-out to to 0.9 quinn-rs/quinn#1671

will be part of quinn-proto 0.9.6:
cargo update -p quinn-proto@0.9.5
or
cargo update -p quinn-proto

[grooviegermanikus]

opened PR to allow to completely disable GSO on Quinn-Level