[Tracking issue] 2024 H2 single shard performance

[akhi3030]

This is the issue for tracking the work on increasing the single shard performance in the second half of 2024.

The high level guidance from Bowen is that we would like to see performance reaching:

• 100K native token transfers per second
• 5000 FT transfers per second
• 1000 swaps per second

We will need to build appropriate benchmarks for each of the targets as well. Note that although the work will be done primarily by the contract runtime team, it is absolutely possible that various improvements will be needed in code / components that are not "owned" by the CRT. E.g. it is expected that we might hit state witness size limits when trying to do around 5000 FT transfers per second.

Note one of the performance optimisations can include executing contracts in parallel on the shard.

We are also open to changing the hardware requirements as long as we have clearly demonstrated that we are hitting the limits of the existing hardware requirements and there isn't a good software optimisation that can help us address the limit.

Issues tracking features:

• Idea: Introduce additional parallelism within application of a single shard chunk #11319

CC: @near/contract-runtime

[akhi3030]

Notes from meeting on 2024-07-23:

• FT transfers. We are able to do 5K transfers on local machines but in order to go above 500 transfers on GCP machines, we get throttled by GCP. The workaround so far has been to use a ssh tunnel however. This might not be a good solution and we may need to setup a VPN or come up with some other solution in order to not get throttled by GCP.
• We should validate that we are able to generate 5K transfers of traffic. We can use the get_nonce method in order to do this.
• We do not have much in terms of benchmarking support for native transfers or for swaps yet.
• For FT transfers state, we have a state with 50M users over 9 smart contracts for 5 GiB of state. This is probably "too much state" i.e. we have seen that the "inflection point" or where the performance drops is at fewer users.
• In order to get tracing and visualisation to work, we are currently being rate limited by grafana. Akhi will chat with Andrei M. to see if we can get the limits increased.