Use EMA to compute QUIC streamer load for staked connections #34586
Conversation
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## master #34586 +/- ##
=========================================
- Coverage 81.8% 81.8% -0.1%
=========================================
Files 823 823
Lines 222725 222910 +185
=========================================
+ Hits 182332 182447 +115
- Misses 40393 40463 +70 |
| let mut last_throttling_instant = tokio::time::Instant::now(); | ||
| let mut streams_in_current_interval = 0; | ||
| let staked_stream = matches!(peer_type, ConnectionPeerType::Staked) && params.stake > 0; |
There was a problem hiding this comment.
why do we need to check both of these things this deep in the call stack? seems like it should've been resolved much higher
There was a problem hiding this comment.
When we initially did the QUIC implementation, I remember that peers with stake = 0 were still marked as ConnectionPeerType::Staked.
The inverse is not true though. I can change it to this. Would that be prefered?
let staked_stream = params.stake > 0;
There was a problem hiding this comment.
let's leave it as is for now and fix ConnectionPeerType::Staked where params.stake == 0 in a followup. either the enum is misnamed or we're patching up a bug all over the place
|
@lijunwangs, @t-nelson, do you have any other feedback? |
|
can we switch all of the |
Done. |
|
lgtm! how are the canary boxes looking? |
The nodes are up, but I am not seeing much streams getting opened. Likely because the clients are not sending transactions directly to these machine. On tpu-forwards port I do see some streams, but very sporadic. So the EMA tends to be closer to 0 all the time. I'll try to run a private cluster. |
The testing on private cluster showed that the number of streams in an EMA interval was always tending to be 1 or 0. This is because the streams are short lived, as each transaction opens a stream, sends the transaction and closes the stream. It makes more sense to accumulate number of streams that were opened in the recent EMA interval, and 0 it out once the EMA has been updated. |
…labs#34586) * Use EMA to compute QUIC streamer load for staked connections * change min load to 25% of max load * reduce max PPS from 500K to 250K * update ema_function to account for missing intervals * replace f64 math with u128 * track streams across all connections for a peer * u128 -> u64 * replace ' as ' type conversion to from and try_from * add counter for u64 overflow * reset recent stream load on ema interval * do not use same counter for unstaked connections from a peer IP
Problem
The limit for staked connection streams doesn't use current load. If the load is less, more streams can be allowed to the connections.
Summary of Changes
Fixes #