Use connectivity store to manage connections to endpoints

[clangenb]

We have already several issues regarding fail over connections to endpoints, or tracking block production of our chain, I would like to aggregate those issues here, and formulate an idea for tackling these. I found the currently unused connectivity_store implemented by @azackmatoff. While it doesn't fulfil the all the specs, I believe that this approach is what we want.

Connectivity Store Specs:

• Needs to know all RPC nodes of a network
• Needs to track block production of the nodes and return a bestEndpoint which is the one with the highest best block number. This endpoint is used for all networking.
• In case the bestEndpoint changes, it handles resubscription of all websocket subscriptions.
• Needs to identify block production issues, aka if all nodes stop producing blocks
• Needs to know when the app/phone itself is offline

Maybe specs:

• Handle the extrinsic queue

[clangenb]

@brenzi other specs?

[brenzi]

I think the extrinsic queue is a separate concern, shouldn't mix.

Do we need to test rpc endpoints to see if indexing is enabled? Or do we assume hardcoding good endpoints only?

Is the set dynamic? Can the user change the endpoint set manually?

[brenzi]

I might suggest randomizing the updated endpoints instead of finding the BEST one and direct all traffic there by this choice

[clangenb]

I think the extrinsic queue is a separate concern, shouldn't mix.

I think so too.

Do we need to test rpc endpoints to see if indexing is enabled? Or do we assume hardcoding good endpoints only?

I think we should hardcode in the beginning, as it is simpler. Yes, I think we should only use endpoints that have offchain indexing enabled. Good point.

Is the set dynamic? Can the user change the endpoint set manually?

I think we should not be dynamic in the beginning. User can choose IMO, but this is also an extension.

I might suggest randomizing the updated endpoints instead of finding the BEST one and direct all traffic there by this choice

This is a good idea, although I would rather do: randomize "good" endpoints, good implying that they are producing blocks and are within close range of the bestHeader number known by the store.