Speed up leader search

[cole-miller]

This PR contains two changes that are intended to address #300 by cutting the number of dqlite connections that need to be opened in typical cases.

The first change is to remember the address of the cluster leader whenever we connect to it successfully, and to try this remembered address first when opening a new leader connection. When the cluster is stable, this means that we open only one connection in Connector.Connect; previously we would open on the order of N (= cluster size) connections. The tradeoff is that Connector.Connect is now slower and less efficient when the cluster is not stable. This optimization applies to both client.FindLeader and the driver implementation.

The second change is to enable client.FindLeader to reuse an existing connection to the leader instead of opening a new one. This is valid because the operations that can be performed on the connection using the returned Client do not depend on the logical state of the connection (open database, prepared statements). When the leader is stable, this saves one new connection per client.FindLeader call after the first change has been implemented. The long-lived connection is checked for validity and leadership before returning it to be reused.

Both of these changes rely on storing some new state in the NodeStore using some fun embedding tricks. I did it this way because the NodeStore is the only object that is passed around to all the right places and lives long enough to persist state between connection attempts.

On my machine, using @masnax's repro script, these two changes combined cut the spikes in CPU usage from 30% to 8-10%, with the first change being responsible for most of that improvement. The remaining spike is due to opening N connections (in parallel) within makeRolesChanges, and could perhaps be dealt with by augmenting the NodeStore further with a pool of connections to all nodes instead of just the last known leader, but I've left that for a possible follow-up.

Signed-off-by: Cole Miller cole.miller@canonical.com

[just-now]

+1

[cole-miller]

Pushed new commits to update the interface following discussion with @marco6. Instead of attaching leader tracking to the NodeStore, now we attach it to protocol.Connector. This is a more natural separation of concerns and cleans things up nicely within the protocol package. It requires some changes to client and driver to take advantage of leader tracking, since previously these packages would just create one-off protocol.Connectors instead of reusing them.

For the driver package, the speedup from leader tracking remains transparent---we store a *protocol.Connector in the driver.Connector, and the database/sql connection pool will internally reuse a driver.Connector many times.

For the client package, there's nowhere in the signature of FindLeader that we could smuggle in a persistent *protocol.Connector, so I've just introduced a new LeaderConnector type that takes care of this. App has been updated to keep a LeaderConnector around.

We save slightly less work with the new design because client and driver can't share a LeaderTracker (although in the future I'd love to turn LeaderConnector into a place for sharing common functionality between client and driver, instead of duplicating options).