Profile, benchmark, and add more load tests for portions of the p2p stack

[evan-forbes]

The MConnection is used as a final queue for all incoming and outgoing packets. For this reason, it seems at least within the realm of possibility that it plays a role of what I refer to as "The Syndrome", which is when we see validators unable to reach consensus despite bandwidth or compute limitations not being hit.

To better understand if the MConnection is playing a role in events that trigger "The Syndrome", such as the inscription events a few weeks ago, we should create simple load unit tests/benchmarks, while also profiling these benchmarks to see if there are any unexpected effective mutexes. The same goes for other low level transport logic, such as the MultiplexTransport.

[staheri14]

I have been scrutinizing the code to pinpoint potential bottlenecks (outlined below) that could impede message transmission (on the sender side), thereby hindering network efficiency and increasing block time. I will then conduct various test scenarios (in the upcoming PRs) based on these findings aimed at stretching these bottlenecks to their limits. By measuring the time taken for each, I will determine which elements are most responsible for slowing down message transmission. Subsequently, I will conduct a similar analysis on the receiving flow.

• In MConnection, messages are transmitted using channel IDs. Messages for each channel ID are placed in a queue, with every channel ID having its own associated queue. If the method Send is called, there is a defaultSendTimeout. If the messages are not successfully queued within this timeout period, the queuing will fail. These queues have the potential to become bottlenecks in a highly loaded network.
• Messages are prioritized for sending based on their channel priority. If this prioritization is not properly enforced, important messages may not be sent in a timely manner, potentially hindering the progress of consensus as intended.
• All messages, irrespective of their channel, are sent through the MConnection's bufConnWriter. They are buffered prior to being sent to the remote party. If the buffer isn't full yet and contains an important message, like a validator vote, it may not be sent until the buffer is full. It's important to note that a ping-pong protocol operates between the two nodes, which ensures that the connection’s buffer is flushed at least after each ping interval. This interval is set to the defaultPingInterval, which is 60 second. Additionally, there is a flushTimer logic that clears data in the buffer every FlushThrottleTimeout. Therefore, data is flushed at the earliest occurrence of either the defaultPingInterval or the FlushThrottleTimeout.
• The MConnection's send and receive rate also play an important role and may become a bottleneck when the network load is high.