Very slow over SSH port forwarding #53
Comments
|
You might want to try using WireGuard instead, if your setup allows it. It will drop packets that are not authenticated for their address on the VPN. This also works better for more permanent setups, and has higher tolerance for unstable networks. |
|
FWIW, I am getting very slow speeds when caching is enabled, too. And without it, throughput isn’t great either: Roughly 50-60 MiB/s instead of the usual 100, lower with smaller msize. I assume the protocol is very latency-sensitive then? |
|
The network is not unstable or has a high latency (our two data centers are only 150 meters apart). WireGuard or any other VPN is too complex and permanent for my setup. I choose 9P because it is small, doesn't require dependencies like rpcbind and only requires one port.
|
|
On Sunday, March 31, 2019 5:04:38 PM CEST TobiasKarnat wrote:
The network is not unstable or has a high latency (our two data centers are
only 150 meters apart). It's interesting what causes the slowness. CPU is
not maxed out, so something else is the cause.
Given the nature of the protocol, even Ethernet latencies are enough to make
it slow. It’s almost as bad as SFTP.
|
|
Just curious - did the slow test perform OK when running without the loopback tunnel? If you're sharing root file systems, my guess is 9P is a bad fit. And the observation that only with really low latency can you get away with it is probably spot on! Not sure if it helps in your particular case, but on our big linux clusters, we shared root file systems with NFS for many years, briefly experimented with 9P (far too slow!), and more recently settled on sharing one read-only ext4 image to many clients with iSCSI, then layering on a RAM-based block device (like zram) on the client at the block layer, to make it appear writeable. Nothing beats networking at the block layer for this type of workload. |
|
Out of curiosity, @garlick have you got a write up about your experiments and stuff? |
|
Ah right, I never restored the google code wiki stuff here. Following artifacts in #20 - this may be of interest |
|
Without the tunnel it's faster by a factor of 10. There are some Linux patches (not stable yet) for async rpc: Would these help and would changes to diod be necessary?
|
|
Finally put up those wiki pages here |
I don't know offhand if those patches would be likely to help. I would need to refresh my memory of the kernel code before I could grok what is going on there. The kernel certainly didn't block before waiting for RPC responses, so it must be about not giving up the CPU when there is more work to do? Since 9P semantics already support overlapped RPCs I don't see how this could affect the diod code. How are you measuring performance? E.g. what workload are you putting on the file system? |
|
Found the solution: -o cache=loose caused the slowdown (not the ssh session). For benchmark I just mounted the remote share and did dd: I loose the filesystem cache by omitting the option. |
Throughput is very slow ~ 9 MiB/s with SSH port forwarding. CPU utilization diod+ssh only ~ 35%-
Can I do anything to optimize the speed? With scp I can get ~ 120 MiB/s (AES-NI on both sides).
My idea was that it could be caused by the large default MTU size (65536) of the loopback interface on the server side and I used iptables to rewrite the MSS-value of the initializing SYN-paket:
iptables -A OUTPUT -o lo -p tcp --dport 564 --syn -j TCPMSS --set-mss 1412
It does rewrite the initial MSS, but doesn't help to improve performance.
Also keeping the Nagle algorithm doesn't help, but decreases performance to ~ 2 MiB/s
https://unix.stackexchange.com/questions/434825/ssh-speed-greatly-improved-via-proxycommand-but-why
Ubuntu 18.04.2 with HWE-Kernel 4.18
Client Configuration:
ssh -L 1564:localhost:564 -NT -o StrictHostKeyChecking=accept-new servername &
mount -t 9p -n -o aname=/,access=client,cache=loose,msize=65536,port=1564 127.0.0.1 /mnt
Server Configuration:
/etc/diod.conf
listen = { "127.0.0.1:564" }
auth_required = 0
exports = { "ctl", "/", "/var" }
The text was updated successfully, but these errors were encountered: