dqlite: Use go-dqlite LTS and build libdqlite from LTS branch

[roosterfish]

No description provided.

[masnax]

As mentioned here we should test go-dqlite behaviour for any changes to leader detection, and update our default settings appropriately.

We should also ensure that clusters that are formed with different versions of go-dqlite but no associated API extensions or schema updates can form clusters properly, and are able to properly detect the leader, given that they may have different implementations.

[masnax]

Granted I'm not sure if the leader detection changes even made it into the LTS, they might just be in v3.

[roosterfish]

As mentioned here we should test go-dqlite behaviour for any changes to leader detection, and update our default settings appropriately.

Looks v2 was cut right from the main branch (which now is v3), the delta is quite small, mostly docs changes https://github.com/canonical/go-dqlite/tree/v2.

We should also ensure that clusters that are formed with different versions of go-dqlite

I have tested this using a cluster of three on v3 and switching one of the members to v2. It properly joins without errors.

[masnax]

I have tested this using a cluster of three on v3 and switching one of the members to v2. It properly joins without errors.

This is going to be racy. We'd need to trigger a join in the middle of a leader election to be sure.
I'm thinking back to this issue: canonical/lxd#12571

[masnax]

Looks v2 was cut right from the main branch (which now is v3), the delta is quite small, mostly docs changes https://github.com/canonical/go-dqlite/tree/v2.

https://github.com/canonical/go-dqlite/releases/tag/v2.0.0 indicates the leader detection changes were indeed included.

canonical/go-dqlite#320

[roosterfish]

This is going to be racy

I guess it's safe to say that right now both v2 and v3 contain technically the same bits (besides the doc and pipeline changes I mentioned).

Where do you think are we missing other bits that could have been changed which are now untested (when it comes to election etc.)?

[masnax]

This is going to be racy

I guess it's safe to say that right now both v2 and v3 contain technically the same bits (besides the doc and pipeline changes I mentioned).

Where do you think are we missing other bits that could have been changed which are now untested (when it comes to election etc.)?

I'm talking about v1 clashing with v2/v3. A fresh cluster being formed likely won't be an issue since v1 won't be in any stable track, but during upgrades this might become problematic.

For example, imagine a 3 node microcluster with go-dqlite v1, and you want to move it to the LTS. As the nodes are upgraded, will there be quorum loss or incorrect leader detection as the old and new nodes have different mechanisms for determining the dqlite leader.

[roosterfish]

I see. But this is more like a general problem if you happen to have a fairly old microcluster running?
No matter if you update to the latest v1 (which now is v3), or v2 it can cause the issues you are describing.

[masnax]

I see. But this is more like a general problem if you happen to have a fairly old microcluster running? No matter if you update to the latest v1 (which now is v3), or v2 it can cause the issues you are describing.

This would be a problem for any existing MicroCloud user as they would be on a pre-v2 go-dqlite version.