[TierTwo] Fix and improve several synchronization problems

[furszy]

A never ending story, more improvements and corrections for the tier two initial synchronization.

This time, the following points are tackled:

1. Fix a bad workflow that causes the budget data initial synchronization to be skipped.
   if we received for example a proposal +50 seconds ago, while we are synchronizing another asset, then once the budget sync starts, it will look like the sync is finished when it's not. Not waiting to receive any budget data, declaring the sync completed.

2. Decrease mnw sync threshold to 4 peers to "minimize" the insane amount of network flood and initial tier two sync delay.
   MNW initial sync is mostly for historical reasons, getting +20k items per peer (+120k items total for the previous threshold of 6) is a high amount of duplicated data relayed over the wire. 80k is still very high but.. it's something that we have to live with until v6.0 (which removes entirely the need of mnw messages..).

3. Fix extra sync attempt for the mnlist asset badly incremented in the sporks sync flow.

4. Connect the not-connected ProcessSyncStatusMsg (ex masternodesync.ProcessMessage()). Thus why the tier two sync stats counters are always in 0, this function was being skipped.

5. The node, after triggering the internal budget data relay process (relaying budget data to every known peer), is invalidly blocking every remote peer, for an hour, for follow-up budget sync requests.

6. The orphans votes maps are linking a single prop/bud parent hash to a single vote, discarding the previous stored item every time that a new orphan arrives. Blocking any follow-up reception of the orphan votes that were discarded, prohibiting its connection.
 (edge case scenario: splitting risk)

7. Added test coverage for:

   1. Single proposal sync based on an orphan proposal vote reception.
   2. Single budget finalization sync based on an orphan finalization vote reception.
   3. Orphan proposal votes and orphan budget fin votes connection when parent is received.


8. Removed unneeded extra items network sync roundtrip: No need to trigger an inv-based sync when the peer sent us a single proposal sync request. And stop walking through the budgets and proposals maps, locking their mutexes, on every single peer unique proposal/bud_fin item request.


9. Generalized the budget full sync items relay process.

Let's aim to include this work on the coming v5.4.0.

[furszy]

Added another commit, co-authored with zebra.
The node, after triggering the internal budget data relay process (relaying budget data to every known peer), is invalidly blocking every remote peer, for an hour, for follow-up budget sync requests.

-- Added issue to PR description as well --

[furszy]

Final update: fixed an scenario which was causing the node to reject and not connect budget system votes, removed extra tier two sync msgs roundtrips (unneeded network flood), expanded test coverage and more:

1. The orphans votes maps are linking a single prop/bud parent hash to a single vote, discarding the previous stored item every time that a new orphan arrives. Blocking any follow-up reception of the orphan votes that were discarded, prohibiting its connection.
 (edge case scenario: splitting risk)

2. Added test coverage for:

   1. Single proposal sync based on an orphan proposal vote reception.
   2. Single budget finalization sync based on an orphan finalization vote reception.
   3. Orphan proposal votes and orphan budget fin votes connection when parent is received.


3. Removed unneeded extra items network sync roundtrip: No need to trigger an inv-based sync when the peer sent us a single proposal sync request. And stop walking through the budgets and proposals maps, locking their mutexes, on every single peer unique proposal/bud_fin item request.


4. Generalized the budget full sync items relay process.

[random-zebra]

Great set of changes 👍 Code ACK with a couple of suggestions.
Will run some test.

[furszy]

After further convos with zebra about the orphan maps cleanup functionality, added the following changes:

1. Limit orphan maps to 10k entries max.
2. Cleanup orphan maps if the parent item wasn't received after an hour from the last received vote time.

Ready to go now.

[furszy]

updated per feedback, seen maps cleaned during the orphan votes cleanup process as well.

[furszy]

Last update:

• Fixed a serious bug that essentially blocks the node from relaying budget data to other peers during the incremental sync process.
• Added test coverage for the budget incremental synchronization process (which is the last budget sync flow that was not covered by tests).
• And cherry-picked zebra's 94e5228 which prevents the removal of valid votes from the seen map. Which fixes another problematic budget incremental sync issue, as the node isn't able to answer to known valid item requests.

Let's give it a good testnet testing round now :).

[random-zebra]

ACK 6f79df3

[Fuzzbawls]

Having spent the past few days stress testing this on mainnet (which has a much more realistic amount of layer 2 overall data than testnet), there were indeed some inconsistent results prior to the most recent push, some of which have stabilized when doing repeated testing.

Results so far (each test was done 50 times on linux and 50 times on macOS):

• Incremental sync for a node already running after first syncing layer 2 data: GOOD
• Initial sync after immediately restarting a node that had a synced layer 2 data previously: GOOD
• Initial sync with a node that has no on-disk layer 2 cache data: INTERMITTENT ^

^ Some test runs without any on-disk layer 2 cache data resulted in a partial budget vote count, others did not. the ones that "failed" had a noticeably extended period of time spent on the MNW portion of the sync.

This test does not yet have a large number of test runs yet due to the extended amount of intentional downtime required

• Initial sync after starting a node with 130+ minutes of downtime and previous on-disk layer 2 cache data: INTERMITTEENT

Mostly on macOS, encountered numerous times where the initial layer 2 sync ended with partial vote counts. An incremental sync afterwards would recoup the missing votes.

I also have observed a peer-banning issue right after the node catches up with the chain tip and starts the layer 2 sync that looks to be related to mnping messages, but unrelated to the changes here.

[furszy]

Are you testing this PR between updated nodes or only against the current network?

Some important points that are all correlated if you are connecting your node to the network:

• The full incremental sync flow as is now deployed in the network does not work (only the partial sync works under certain circumstances). The clear of the votes seen maps right before the data relay impedes non-updated peers in the network from syncing anyone surrounding them. This is solved by 34e9c23, thus why the connection between updated nodes is needed in order to fulfill the entire process successfully.

• For the very same reason, going into the intermittentencies topic: as there are peers that have the seen maps empty, and as this maps are used to relay the get data requests, there are a good number of peers in the network that can't respond to budget vote data requests (until someone else relay the items to them again..). Replying to each request with "not found" messages (thus why detected [Net_processing] Do not log "notfound" msg as unknown. #2676). You can check it enabling the 'net' logging level and looking for the "not found" response when you see the initial sync intermittentencies, your node will receive one "not found" per vote item request.

• Another point over this topic is that non-update peers in the network are blocking budget items requests for an hour if the node asks for single items as well. So, when the mnw process takes a while, and the node receives single proposal item (before the budget sync step): he will ask for the proposal/budget, the remote peer will answer it back and block any follow-up budget sync request, making the next tier two full budget sync request useless as the remote peer will not answer it. (This is solved by 5f596b8 and further cleaned by 31e40a4).

[Fuzzbawls]

Threw more nodes running this into the connection pool, all good now.

ACK 6f79df3