Devices with flaky connections appear "Online" but are unreachable

[xginn8]

As mentioned in https://www.flowdock.com/app/rulemotion/resin-tech/threads/wdSU6cV3l2S2RgB9VL1ukrAb-V-,

i've noticed there are a number of devices that appear online, but upon trying to connect i get the following error:

❯ balena ssh {{UUID}}
error: host error:
Connection to ssh.balena-devices.com closed.

According to https://github.com/balena-io/open-balena-vpn/blob/master/config/confd_env_backend/templates/server.conf.tmpl#L14, the openvpn keepalive behavior should be configured to 10/60 by default, so presumably these devices are pinging back just fine. Occasionally there will be out-of-order messages given the "Online" state:

21 Jan 2020 15:03:59 vpn[95]: debug: [vpn-124120] successfully updated state for device: uuid={{uuid}} worker_id=2 connected=true virtual_address={{vip}}
21 Jan 2020 15:43:00 vpn[98]: debug: [vpn-124118] successfully updated state for device: uuid={{uuid}} worker_id=3 connected=true virtual_address={{vip}}
21 Jan 2020 15:43:02 vpn[95]: debug: [vpn-124120] successfully updated state for device: uuid={{uuid}} worker_id=2 connected=false

Even pinging from the VPN instance directly shows slow networking:

--- {{uuid}}.vpn ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 64ms
rtt min/avg/max/mdev = 2365.627/2991.429/3526.436/482.054 ms, pipe 4

The devices affected are running balenaOS 2.2.0, fwiw.