Implement test for local congestion control

[aborg-dev]

We should have a test that generates an interesting workload with non-trivial congestion metrics. This test will quickly evaluate the various designs that attempt to solve the congestion problem.

The concrete test for the local congestion control milestone would evaluate liveness when the system receives 1000 transactions per second (eventually every transaction should be processed after the load is stopped). This simulates a short bursty sustained load (e.g. for a minute) at 10x of the network capacity.

It is intended as a functional correctness test, not a performance test.

As a part of the test, we should track the following:

• Delayed receipts queue size
• Transaction pool size
• Distribution of transaction latency

We should aim to reuse a framework from #8847

Some related work issues:

• Create dashboards tracking key congestion metrics #8880
• Come up with workloads/contracts that trigger various corner cases in the transaction handling pipeline #8705

[jakmeier]

I plan to start working on this on 1 May.

[jakmeier]

I have started thinking about this. Reusing #8847 sounds good and all. But I am contemplating whether it's a bit of an overkill for a congestion test that specifically doesn't care about performance. Maybe a pytest on a single machine would be simpler and give us the same results?

For now, I will try to come up with a sketch for what I think is a good test workload for local congestion. Then I will see how easy it is to integrate it in #8847.

[aborg-dev]

I have started thinking about this. Reusing #8847 sounds good and all. But I am contemplating whether it's a bit of an overkill for a congestion test that specifically doesn't care about performance. Maybe a pytest on a single machine would be simpler and give us the same results?

There is a way to launch that test on a single machine without any additional setup steps using setup-cluster flag:

nearcore/pytest/tests/loadtest/loadtest2.py

Lines 391 to 402 in a5ed70b

	if args.setup_cluster:
	config = cluster.load_config()
	nodes = cluster.start_cluster(
	2, 0, args.shards, config, [["epoch_length", 100]], {
	shard: {
	"tracked_shards": list(range(args.shards))
	} for shard in range(args.shards + 1)
	})
	if args.contract_key is None:
	signer_key = nodes[0].signer_key
	else:
	signer_key = key.Key.from_json_file(args.contract_key)

One of the reasons why we also extended it to work with a cluster started in advance is to simplify export of Grafana metrics which is tricky to do in pytest/local run environment. For this test, Grafana will at least be useful to track the size of transaction pools, though arguably we can try to query that metric directly from the test runner.

[aborg-dev]

Some thoughts on testing the transaction pool size limit (#8878):

During the test, we should vary the pool size and observe the behavior of the system. If everything works correctly, there will be a threshold such that setting the limit higher than the threshold should not influence the throughput of the network.

The threshold will be based on the chunk gas limit and will need to be high enough to saturate the chunk capacity.
My ballpark estimate, based on 300 TGas -> 4 MiB limit and assuming that 10 blocks of buffer would be enough is:
10 PGas / 300 TGas * 4 MiB = 100/3 * 4 MiB = 33 * 4 MiB = 132 MiB.

Rounding up, 150 MiB should be enough. In the test, we can try values [50MiB, 100MiB, 150MiB, 200 MiB, 500 MiB, Unlimited] and see how they affect the throughput.

[jakmeier]

With the recent progress in #9118 I will soon shift my focus over here.

The locust setup should be good enough now to simulate congestion. Next week I'll try to produce a mocknet-like setup with Prometheus / Grafana integration to look at different queue lengths. Then I will report my findings around how our system generally behaves under congestion.

From there it should be easy to come up with useful test scenarios and define them using locust. Increased receipt sizes combined with lowered TX pool size limits will be one of the first things I want to play around with.

[aborg-dev]

I've ran a quick and dirty test with the following setup:

• 1 locust master node, 16 locust worker
• 100 parallel users
• Each user issues a request "sha256("a" * 100000)" (around 100KB of payload) and waits till result

The test ran for 5 minutes, with latency creeping towards 10s timeout and the transaction pool growing to 30 MB. Then all the nodes crashed for an unknown reason. Will be looking into logs to understand why this happened.

[aborg-dev]

I did a further investigation - with 20 users, the network works without changes to latency, but with 40 users, the chunk production stalls and we get forks in the chain:

This could be potentially explained by the fact that both the network and the loadtest run on the same machine and there is resource contention between then, though I've checked that nodes are doing fine and the CPU is only 70% utilized.
I'll try to separate the load generator and see if this helps.

UPD: The node0 has became unreachable and its logs contain the following lines:

2023-06-01T15:49:54.569334Z  INFO stats: #    2426 Downloading blocks 0.00% (2 left; at 2426) 3 peers ⬇ 236 B/s ⬆ 193 B/s 0.00 bps 0 gas/s CPU: 1%, Mem: 4.87 GB

[aborg-dev]

With 1 node and 1 shard, I was reliably able to get to a regime that represents a congested network.
The exact workload was:

• 1 locust node
• 150 concurrent users
• around 12 requests per second
• Fibonacci number computation with n = 33

This load generated around 12 seconds of congestion in the delayed receipts queue. This in turn resulted in all locust requests timing out, which is expected.

The chain is utilized most of the time:

The detailed Grafana graphs for the relevant period can be found here: https://nearinc.grafana.net/goto/PRNdCRlVR?orgId=1

Next up, I want to introduce a backpressure in the delayed receipts queue and confirm that this workload generates an increasing transaction pool queue size.

I also will look into adding a metric about the delayed receipts queue size mentioned in #8880

[aborg-dev]

We now have the workload that can reproduce congestion and the bursty workload described in #8920 (comment).

I'm resolving this, as the functional part is there and it's now just a matter of analyzing how the system performs under this load.