artiq-zynq performance regression with 0.7.0

[sbourdeauducq]

See https://git.m-labs.hk/M-Labs/artiq-zynq/issues/127

[Dirbaio]

Interesting.
Is "H" a Linux machine and "D" a smoltcp device? Assuming so:

• H2D could be due to Delayed ACK if the rx buffer in the device is small. Can you try disabling it?
• D2H could be due to tcp: don't send data outside the remote window #387 (but unlikely because the rx buffer in the Linux side is probably huge), or maybe tcp: Add RTT estimation. #406 if the link is lossy (the RTT estimation will probably detect a higher RTO, the previous hardcoded RTO of 100ms was very agressive?).

Could you provide packet captures of before and after? It would help significantly in figuring it out

[astro]

Regression occurs since the merge of #404.

[dnadlinger]

@Dirbaio: Just to confirm your interpretation, H would indeed have been a Linux box ("host") running the kernel TCP stack, and D the embedded ARM device running smoltcp.

[Dirbaio]

Regression occurs since the merge of #404.

Okay, makes sense.

• How is the workload like? Is it one big transfer in one direction, or many small transfers in request/response fashion? What's the TCP buffer sizes, what's the size of the writes?
• Can you provide packet capture?
• Have you tried disabling delayed ACK? socket.set_ack_delay(None)

[dnadlinger]

(@astro/… can hopefully provide details, as I'm only peripherally involved with this.)

[astro]

The workload sizes are 1MB, and 1KB, echoed in a request/response fashion. TCP buffer sizes are 64KB.

socket.set_ack_delay(None) mitigated the regression for us. A performance regression in the default settings is still worth investigating, so don't close this issue yet.

[astro]

I've done some loopback tests with delay and FaultInjector in an std environment but was not yet able to reproduce throughput differences between running with and without delayed acks.

[Dirbaio]

It's expected ACK delay causes issues depending on the workload (usually chatty request/response workloads). There's no default settings that will suit everyone, so it's expected people will have to tune them. Also, Linux defaults to ack_delay on FWIW.

If you believe your workload shouldn't be impacted by ack_delay but still get a performance regression, please open a new issue with logs and packet capture.