race condition in cluster manger

[woodsaj]

if a node receives a LocalState update and a NotifyLeave message at the same time, a race condition exists resulting in the node that has left not being removed members map. This stale member information is then propagated through the cluster.

eg, assume a 3 node cluster. nodeA, nodeB, nodeC

• nodeA leaves the cluster (normal shutdown)
• nodeB initiates a push/pull sync with nodeC
• nodeC responds with the LocalState payload before it processes the notifyLeave message from nodeA
• nodeB processes the notifyLeave message, removing nodeA from the members table
• nodeB processes the LocalState payload from nodeC, sees a new node called nodeA and adds it to its members table.
• nodeC initiates a push/pull sync with nodeB
• nodeB responds with the localState payload that includes nodeA as a member of the cluster
• nodeC processes the localState payload and adds nodeA to its members table.