ffmuc-mesh-vpn-wireguard: enable loadbalancing

[awlx]

This enables loadbalancing to actually use the supernodes to the best extend.

[blocktrron]

Do you have empiric data on the effectiveness of communicating load-information to clients?

I did think a lot about whether this is necessary on tunnel establishment but it all came down to concluding statistics to do the work for me should suffice, especially since selection logic is uniform (well, mostly uniform given the connectees client version).

This also has the advantage of having the chance by statistic to establish a peer-connection to a different peer in case we can not establish a connection to the single host your broker now communicates to the client. This could be fixed by a ordered list of candidates with a decrementing selection-weight. However it is arguably, if this is something to consider.

Coming from distributing clients on a wireless network, the major benefit I'd extrapolate from such a solution is if it would allow to continuously evaluate the need of reselection on the connection-terminating end, especially since judging effectiveness of target selection primarily depends on bandwidth usage, which is not known on connection establishment neither static. But this is a different topic. Nevertheless, did you put this into consideration and if so, did you draw a conclusion on the topic?

[awlx]

This PR is especially meant to avoid situations like the one above when we do Supernode maintenances and all nodes pile up on one GW after the gateways rebooted.

We didn't plan yet to have a gateway switch after the initial connect.

[T0biii]

Another feature is that we can add gateways without new firmware

[T0biii]

I think we should still change this for v1 because I recently found out that this random is "random" based on the system time, maybe there is another option to determine the random without depending on the system time. Which is why so many clients choose the same gateway from the list but this shouldn't be a blocker for a merge as the v2 offers significantly better advantages for load balancing

[grische]

It seems that our awk implementation for srand only uses second-precision, that is indeed not very helpful:

# i=0; while [[ $i -lt 15 ]]; do date -Ins | tr -d '\n' && echo -n " rand()=" && awk 'BEGIN{srand(); print rand()}'; i=$((i+1)); done
2024-03-26T13:32:18,000000000+01:00 rand()=0.764051
2024-03-26T13:32:18,000000000+01:00 rand()=0.764051
2024-03-26T13:32:18,000000000+01:00 rand()=0.764051
2024-03-26T13:32:18,000000000+01:00 rand()=0.764051
2024-03-26T13:32:18,000000000+01:00 rand()=0.764051
2024-03-26T13:32:18,000000000+01:00 rand()=0.764051
2024-03-26T13:32:18,000000000+01:00 rand()=0.764051
2024-03-26T13:32:18,000000000+01:00 rand()=0.764051
2024-03-26T13:32:18,000000000+01:00 rand()=0.764051
2024-03-26T13:32:18,000000000+01:00 rand()=0.764051
2024-03-26T13:32:18,000000000+01:00 rand()=0.764051
2024-03-26T13:32:18,000000000+01:00 rand()=0.764051
2024-03-26T13:32:18,000000000+01:00 rand()=0.764051
2024-03-26T13:32:18,000000000+01:00 rand()=0.764051
2024-03-26T13:32:18,000000000+01:00 rand()=0.764051

I can confirm this is the same with Ubuntu 22.04 on a desktop system (but it does show nanoseconds on date --iso=ns):

$ i=0; while [[ $i -lt 15 ]]; do date -Ins | tr -d '\n' && echo -n " rand()=" && awk 'BEGIN{srand(); print rand()}'; i=$((i+1)); done
2024-03-26T13:53:25,716105751+01:00 rand()=0.49871
2024-03-26T13:53:25,717883612+01:00 rand()=0.49871
2024-03-26T13:53:25,719541005+01:00 rand()=0.49871
2024-03-26T13:53:25,721102173+01:00 rand()=0.49871
2024-03-26T13:53:25,722682426+01:00 rand()=0.49871
...

---

As $RANDOM is not available in busybox sh, an alternative would be to use busybox' hexdump + /dev/urandom:

# i=0; while [[ $i -lt 15 ]]; do i=$((i+1)); date -Ins | tr -d '\n' && echo -n " rand()=" && hexdump -n 4 -e '"%u"' </dev/urandom && echo; done
2024-03-26T13:36:00,000000000+01:00 rand()=2986735940
2024-03-26T13:36:00,000000000+01:00 rand()=234726918
2024-03-26T13:36:00,000000000+01:00 rand()=4249724515
2024-03-26T13:36:00,000000000+01:00 rand()=165910888
2024-03-26T13:36:00,000000000+01:00 rand()=2380433412
2024-03-26T13:36:00,000000000+01:00 rand()=3772558562
2024-03-26T13:36:00,000000000+01:00 rand()=277626919
2024-03-26T13:36:00,000000000+01:00 rand()=2142351523
2024-03-26T13:36:00,000000000+01:00 rand()=1096050454
2024-03-26T13:36:00,000000000+01:00 rand()=1464321489
2024-03-26T13:36:00,000000000+01:00 rand()=2153772455
2024-03-26T13:36:00,000000000+01:00 rand()=2246216180
2024-03-26T13:36:00,000000000+01:00 rand()=3724409659
2024-03-26T13:36:00,000000000+01:00 rand()=2829435099
2024-03-26T13:36:00,000000000+01:00 rand()=4106529008

[grische]

I opened a separate PR for this: #101

[blocktrron]

@awlx Does the loadbalancing solve the issue in case the gateways are offline at moment of re-evaluation anyhow? Or do you advertise Gateways which will re-appear soon even in case they are currently offline?

Nothing blocking this PR (It's your package anyhow), but i have a hard time understanding how this fixes the problem you describe. The piling up happens because the gateway-selection is limited in time of re-evaluation. But the piling up would happen with loadbalancing also, as Gateways which are offline would not be considered as candidates?

[awlx]

If the gateways are offline for longer than a minute yes. But we time the reboots so always 3 gateways are online :).

[blocktrron]

Understood. But how does this differ from a random reselection of the Gateway list on the device then? This would also give a reasonable chance of each node selecting a different gateway, doesn't it?

[maurerle]

If every node selects a random gateway from a list, and all nodes are rebooted - you get a random distribution.

If you have maintenance on one of n nodes, you would still have all nodes establish a connection to the available n-1 nodes. Once they established the connection the are not reconnecting, therefore the last maintained gateway stays with less clients.
This can be fixed using the loadbalancing here.

[maurerle]

I would eventually handle the case of "Parsing wgkex broker data failed" and fall back to v1 instead of not connecting at all?

[awlx]

Sounds like a good idea :).

[awlx]

@grische @DasSkelett some thing to consider?

[T0biii]

i tried to move the call for v1 and v2 in own functions.
So v2 can call v1 when it failes:

https://github.com/DasSkelett/community-packages/pull/1/files
Here is a logread from the checkuplink:
https://gist.github.com/T0biii/1c14b56fc8488cae95181cfa359f53e0
Firmware to test:
stabel:
https://github.com/T0biii/site-ffm/releases/tag/v2024.2.0
next:
https://github.com/T0biii/site-ffm/releases/tag/v2024.2.0-next

[grische]

@maurerle thanks a lot, that's a very good idea. I added the patches + some suggestions of mine on top of the current PR.

[blocktrron]

@maurerle But the loadbalancing does not engage when a connection to one of the remaining gateways has been established:

community-packages/ffmuc-mesh-vpn-wireguard-vxlan/files/lib/gluon/gluon-mesh-wireguard-vxlan/checkuplink

Line 162 in 160bb63

exit 0

I feel like I don't see the point everyone else is seeing, but for your scenario to be viable, a node would require to re-evaluate itÄs existing choice, doesn't it?

[awlx]

The random distribution is also a problem, when we do maintenance in one PoP the nodes would try to reach the "deactivated" gateways and take longer to reconnect. With the new approach they don't even see the gateways in maintenance, it's also easier to add or remove gateways.

This is the first iteration of loadbalancing, in the end we also want the nodes to re-evaluate every X minutes if they are still on the best GW, but this requires quite some tuning otherwise we have a flappy node.

[blocktrron]

Thanks for the explanation, now it is more clear to me what you try to achieve.