Too many forwards

[aeyakovenko]

Problem

Under load banking state is generating 100k+ forwards per second

Proposed Solution

Limit the forwarders rate to something reasonable like 10k per second

Discussion: https://discord.com/channels/428295358100013066/689412830075551748/1018215363068055562

[aeyakovenko]

Tag @taozhu-chicago @sakridge

[tao-stones]

Maybe it is better to measure "number of CUs forwarded" instead of "number of Transactions forwarded", especially forwarding is by account CU limits.

Current static setting that impact forward rate are:

1. forwarder buffer size: it is set to forward up to 50 blocks, equivalent up to 12K transactions all with default 200_000 CUs (mainly to accommodate txs without ComputeBudget::Requested_CU ix).
2. max rate limit: 12 MB outbound limit per second const MAX_BYTES_PER_SECOND: usize = 12_000_000;, equivalent to ~10K TXs per sec.

Can reduce forward queue (#1) to 10 blocks, or even lower to just 4 blocks. Will look into rate limit.