Encrypted overlay performance regression affecting 1.2.0 and 1.3.0

[awh]

If the router is started with a password, performance is reminiscent of that observed when ethtool is unavailable:

vagrant@host1:~$ weave launch --password foo
vagrant@host1:~$ docker $(weave config) run -ti --name alpha --rm dwragg/iperf3 -c omega.weave.local
Connecting to host omega.weave.local, port 5201
[  4] local 10.40.0.0 port 52733 connected to 10.32.0.1 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   176 KBytes  1.44 Mbits/sec   38   2.65 KBytes       
[  4]   1.00-2.00   sec  90.2 KBytes   739 Kbits/sec   16   2.65 KBytes       
[  4]   2.00-3.00   sec  87.5 KBytes   717 Kbits/sec   19   3.98 KBytes       
[  4]   3.00-4.00   sec   131 KBytes  1.08 Mbits/sec   16   6.63 KBytes

The problem goes away if fast datapath is completely disabled:

vagrant@host1:~$ WEAVE_NO_FASTDP=1 weave launch --password foo
vagrant@host1:~$ docker $(weave config) run -ti --name alpha --rm dwragg/iperf3 -c omega.weave.local
Connecting to host omega.weave.local, port 5201
[  4] local 10.32.0.1 port 52102 connected to 10.40.0.0 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  15.0 MBytes   126 Mbits/sec   43   98.4 KBytes       
[  4]   1.00-2.00   sec  12.8 MBytes   107 Mbits/sec   13    117 KBytes       
[  4]   2.00-3.00   sec  14.0 MBytes   118 Mbits/sec    4    105 KBytes

Modifying the weave script and making ethtool_tx_off_fastdp equal to ethtool_tx_off_bridge fixes the issue; simply executing ethtool -K $ROUTER_HOSTNETNS_IFNAME tx off in the script does not.

[awh]

Gah not sure why GH didn't close this, it's in the fixes list for #1758...

[dpw]

Could you elaborate on how #1758 addresses this issue?

[awh]

Could you elaborate on how #1758 addresses this issue?

pcap is no longer used in conjunction with the OVS datapath - see this commit comment e12dbb4

[dpw]

And you tested that #1758 doesn't suffer from this performance problem, for traffic over encrypted connections?

[awh]

And you tested that #1758 doesn't suffer from this performance problem, for traffic over encrypted connections?

Yes - in fact it was whilst performance testing #1758 that I discovered the issue. I posted some mesh mode plugin + optional encryption numbers to Slack yesterday:

Mesh mode plugin, untrusted link (bear in mind that compared to the WEAVE_NO_FASTDP=1 numbers above, these figures are impacted by the intermediary bridge netdev in addition to the switch from pcap):

vagrant@host1:~$ docker run --net=weave -h alpha.weave.local $(weave dns-args) -ti dwragg/iperf3 -c omega
Connecting to host omega, port 5201
[  4] local 10.40.0.0 port 52884 connected to 10.32.0.1 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  10.7 MBytes  89.8 Mbits/sec   13    114 KBytes       
[  4]   1.00-2.00   sec  10.9 MBytes  91.3 Mbits/sec   38   95.5 KBytes       
[  4]   2.00-3.00   sec  11.2 MBytes  93.9 Mbits/sec   16   94.2 KBytes       

Mesh mode plugin, trusted link:

vagrant@host1:~$ docker run --net=weave -h alpha.weave.local $(weave dns-args) -ti dwragg/iperf3 -c omega
Connecting to host omega, port 5201
[  4] local 10.40.0.0 port 52890 connected to 10.32.0.1 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   157 MBytes  1.32 Gbits/sec   26    797 KBytes       
[  4]   1.00-2.00   sec   163 MBytes  1.37 Gbits/sec    1    736 KBytes       
[  4]   2.00-3.00   sec   163 MBytes  1.37 Gbits/sec    0    838 KBytes       
[  4]   3.00-4.00   sec   161 MBytes  1.35 Gbits/sec   66    660 KBytes