[forge][chaos] Expand chaos simulation to six regions

[ibalajiarun]

Description

After the recent forge region simulation fixes in #13476, we can now support up to six regions from four previously. Hence, this PR. By dividing the nodes into six regions, we can now expand support to more nodes. I have tested up to 155 nodes (150 vals + 5 vfns).

For up to 100 validators, 4-regions will be the default, and for anything more 6-regions will be default.

Support for expanding over 6 regions is WIP.

[trunk-io]

⏱️ 3h 58m total CI duration on this PR

Slowest 15 Jobs	Cumulative Duration	Recent Runs
test-target-determinator	55m	⬜ 🟩 🟩 🟩 🟩 (+10 more)
forge-e2e-test / forge	33m	🟥 🟥
rust-cargo-deny	24m	⬜ 🟩 🟩 🟩 🟩 (+10 more)
check-dynamic-deps	24m	🟩 🟩 🟩 🟩 🟩 (+11 more)
rust-images / rust-all	15m	🟩
rust-images / rust-all	13m	🟩
forge-e2e-test / forge	11m	🟥
semgrep/ci	7m	🟩 🟩 🟩 🟩 🟩 (+11 more)
general-lints	6m	⬜ 🟩 🟩 🟩 🟩 (+10 more)
dispatch_event	6m	🟥
dispatch_event	5m	🟥
dispatch_event	5m	🟥
file_change_determinator	3m	🟩 🟩 🟩 🟩 🟩 (+11 more)
file_change_determinator	3m	🟩 🟩 🟩 🟩 🟩 (+10 more)
rust-move-tests	2m	🟩

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

[github-actions]

✅ Forge suite compat success on b29f09f57e898d8d211c8bc3e303f6e50bba2266 ==> cdbebc064eda0720f161e0f4d927ed00a09deb5a

Compatibility test results for b29f09f57e898d8d211c8bc3e303f6e50bba2266 ==> cdbebc064eda0720f161e0f4d927ed00a09deb5a (PR)
1. Check liveness of validators at old version: b29f09f57e898d8d211c8bc3e303f6e50bba2266
compatibility::simple-validator-upgrade::liveness-check : committed: 12769.64 txn/s, latency: 2526.79 ms, (p50: 1900 ms, p70: 2100, p90: 2600 ms, p99: 18800 ms), latency samples: 495240
2. Upgrading first Validator to new version: cdbebc064eda0720f161e0f4d927ed00a09deb5a
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 7661.42 txn/s, latency: 3585.45 ms, (p50: 3700 ms, p70: 4400, p90: 4900 ms, p99: 5000 ms), latency samples: 141360
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 7103.13 txn/s, latency: 4469.40 ms, (p50: 4600 ms, p70: 4800, p90: 6700 ms, p99: 6900 ms), latency samples: 241200
3. Upgrading rest of first batch to new version: cdbebc064eda0720f161e0f4d927ed00a09deb5a
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 6740.14 txn/s, latency: 4178.18 ms, (p50: 4700 ms, p70: 5000, p90: 5100 ms, p99: 5200 ms), latency samples: 124960
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 6544.39 txn/s, latency: 4949.27 ms, (p50: 5400 ms, p70: 5600, p90: 6400 ms, p99: 6800 ms), latency samples: 226500
4. upgrading second batch to new version: cdbebc064eda0720f161e0f4d927ed00a09deb5a
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 12197.82 txn/s, latency: 2224.30 ms, (p50: 2400 ms, p70: 2500, p90: 2600 ms, p99: 2700 ms), latency samples: 213420
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 11047.59 txn/s, latency: 2775.57 ms, (p50: 2500 ms, p70: 2700, p90: 4800 ms, p99: 6400 ms), latency samples: 360100
5. check swarm health
Compatibility test for b29f09f57e898d8d211c8bc3e303f6e50bba2266 ==> cdbebc064eda0720f161e0f4d927ed00a09deb5a passed
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions]

✅ Forge suite realistic_env_max_load success on cdbebc064eda0720f161e0f4d927ed00a09deb5a

two traffics test: inner traffic : committed: 12569.03 txn/s, submitted: 12569.40 txn/s, expired: 0.37 txn/s, latency: 3160.29 ms, (p50: 2900 ms, p70: 3000, p90: 3900 ms, p99: 9900 ms), latency samples: 4779200
two traffics test : committed: 99.97 txn/s, latency: 1852.39 ms, (p50: 1500 ms, p70: 1700, p90: 3000 ms, p99: 4400 ms), latency samples: 1720
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.255, avg: 0.227", "QsPosToProposal: max: 1.016, avg: 0.957", "ConsensusProposalToOrdered: max: 0.340, avg: 0.325", "ConsensusOrderedToCommit: max: 0.519, avg: 0.486", "ConsensusProposalToCommit: max: 0.846, avg: 0.811"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 1.07s no progress at version 1780135 (avg 0.23s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 7.78s no progress at version 1780133 (avg 7.00s) [limit 15].
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions]

✅ Forge suite framework_upgrade success on b29f09f57e898d8d211c8bc3e303f6e50bba2266 ==> cdbebc064eda0720f161e0f4d927ed00a09deb5a

Compatibility test results for b29f09f57e898d8d211c8bc3e303f6e50bba2266 ==> cdbebc064eda0720f161e0f4d927ed00a09deb5a (PR)
Upgrade the nodes to version: cdbebc064eda0720f161e0f4d927ed00a09deb5a
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1131.77 txn/s, submitted: 1133.58 txn/s, failed submission: 1.80 txn/s, expired: 1.80 txn/s, latency: 2657.48 ms, (p50: 2400 ms, p70: 3000, p90: 3800 ms, p99: 5400 ms), latency samples: 100480
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1117.57 txn/s, submitted: 1121.75 txn/s, failed submission: 4.17 txn/s, expired: 4.17 txn/s, latency: 2628.54 ms, (p50: 2600 ms, p70: 3000, p90: 4000 ms, p99: 5500 ms), latency samples: 101720
5. check swarm health
Compatibility test for b29f09f57e898d8d211c8bc3e303f6e50bba2266 ==> cdbebc064eda0720f161e0f4d927ed00a09deb5a passed
Upgrade the remaining nodes to version: cdbebc064eda0720f161e0f4d927ed00a09deb5a
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1141.49 txn/s, submitted: 1144.53 txn/s, failed submission: 3.04 txn/s, expired: 3.04 txn/s, latency: 2795.28 ms, (p50: 2400 ms, p70: 3000, p90: 4700 ms, p99: 6900 ms), latency samples: 97740
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking