Location of participants in Lotus stack, multiple identities in one instance

[Kubuxu]

For now, I've assumed that the f3 active participant would live in Lotus.
This might not have been as good of an assumption.

An active participant is tied to SP code and identity; that flow lives in lotus-miner/lotus-provider.
At the same time, the lotus miner is not connected with the global pubsub (AFAIK).

A single Lotus node can also host multiple providers, which, if f3 continues to live there, would necessitate either multiple concurrent instances in one Louts node or f3 being able to handle multiple identities.

As far as I know, the protocol flow is independent of our own identity, which should make running multiple identities at the same much easier. We could abstract out signing and VRF generation as part of the broadcast operation. Essentially, the instance itself stops caring about our own identity.

[Kubuxu]

The design I settled on:

• gpbft.Participant is unaware of the ID it is running as the protocol is independent of our own decisions
  • this requires a slight refactor and cleanup in gpbft to remove ParticipantID
• gpbft requests a given message to be broadcasted, that message is universal (not connected with any ID), this is passed to the Host which knows as which IDs it wants to broadcast, and uses powertable from gpbft to resolve IDs to public keys
• The host then builds a payload to be signed with that key. Signing can happen over RPC boundary (necessary for offloading keys to lotus-miner). This results in serialized payloads that are to be signed with the given key and returned back to the host for broadcasting
• When these payloads are returned to be broadcasted, they are also immediately processed as incoming messages

See #188 for PR on it. The suggestion there was to separate the PR into:

1. async delivery of local messages
2. removal of ID from gpbft.Participant
3. addition of the message builder pattern

[anorth]

This has proven slightly trickier than we thought. The protocol code has assumptions that it can update state as a result of its own decisions more or less synchronously. This used to be achieved by internally receiving messages sent by self synchronously. This changed to async as part of #259 but means there's now a race between receiving those messages sent to self and receiving alarms #316.

I explored an internal send-to-self of unvalidated messages, but immediately ran into not knowing the node's own power, so the state updates are impossible.

Some potential paths forward that I see:

• Attempt to remove any algorithmic assumption of being able to update state as a result of the participant's own decisions. I don't know how easy this will be. We haven't tried. Possibly related to Running gpbft without active participation (passive observation) #319.
• When requesting broadcast of a message, synchronously receive back from the host the information needed to update state. This is a list of participant IDs and power amounts that the host will subsequently sign and broadcast on behalf of. This would still leave unknown: tickets for CONVERGE message/s.

[masih]

@anorth @Kubuxu In light of #334 Is there still remaining work here? If not can we close this issue?