admin service /threadz dumps waiting and sleeping threads on verbose=…

[msmouse]

…true

Description

the waiting threads are the key to debug a dead lock

How Has This Been Tested?

tried on forge runs

Key Areas to Review

Type of Change

• New feature

Which Components or Systems Does This Change Impact?

• Full Node (API, Indexer, etc.)

[trunk-io]

⏱️ 1h 1m total CI duration on this PR

Slowest 15 Jobs	Cumulative Duration	Recent Runs
test-target-determinator	23m	🟩 🟩 🟩 🟩 🟩
rust-cargo-deny	6m	🟩 🟩 🟩
check-dynamic-deps	6m	🟩 🟩 🟩 🟩 🟩
rust-doc-tests	5m	🟩
execution-performance / test-target-determinator	4m	🟩
check	4m	🟩
semgrep/ci	2m	🟩 🟩 🟩 🟩 🟩
rust-move-tests	2m	🟩
rust-move-tests	2m	🟩
rust-move-tests	2m	🟩
fetch-last-released-docker-image-tag	2m	🟩
general-lints	1m	🟩 🟩 🟩
file_change_determinator	1m	🟩 🟩 🟩 🟩 🟩
file_change_determinator	47s	⬜ 🟩 ⬜ 🟩 🟩
permission-check	20s	🟩 🟩 🟩 🟩 🟩 (+2 more)

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

[github-actions]

✅ Forge suite realistic_env_max_load success on 4d2e420366928e0b369e0ac37136da25157c208b

two traffics test: inner traffic : committed: 14129.62 txn/s, latency: 2809.92 ms, (p50: 2700 ms, p70: 2700, p90: 3000 ms, p99: 3300 ms), latency samples: 5372580
two traffics test : committed: 99.98 txn/s, latency: 1491.24 ms, (p50: 1500 ms, p70: 1500, p90: 1600 ms, p99: 2200 ms), latency samples: 1760
Latency breakdown for phase 0: ["MempoolToBlockCreation: max: 2.053, avg: 1.571", "ConsensusProposalToOrdered: max: 0.316, avg: 0.298", "ConsensusOrderedToCommit: max: 0.384, avg: 0.367", "ConsensusProposalToCommit: max: 0.682, avg: 0.665"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 1.20s no progress at version 2480956 (avg 0.21s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 8.47s no progress at version 2480954 (avg 8.47s) [limit 15].
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions]

✅ Forge suite framework_upgrade success on b29f09f57e898d8d211c8bc3e303f6e50bba2266 ==> 4d2e420366928e0b369e0ac37136da25157c208b

Compatibility test results for b29f09f57e898d8d211c8bc3e303f6e50bba2266 ==> 4d2e420366928e0b369e0ac37136da25157c208b (PR)
Upgrade the nodes to version: 4d2e420366928e0b369e0ac37136da25157c208b
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1170.05 txn/s, submitted: 1172.52 txn/s, failed submission: 2.47 txn/s, expired: 2.47 txn/s, latency: 2774.59 ms, (p50: 2400 ms, p70: 3000, p90: 4900 ms, p99: 6900 ms), latency samples: 104160
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1196.66 txn/s, submitted: 1198.45 txn/s, failed submission: 1.79 txn/s, expired: 1.79 txn/s, latency: 2525.83 ms, (p50: 2100 ms, p70: 2700, p90: 4200 ms, p99: 6300 ms), latency samples: 107100
5. check swarm health
Compatibility test for b29f09f57e898d8d211c8bc3e303f6e50bba2266 ==> 4d2e420366928e0b369e0ac37136da25157c208b passed
Upgrade the remaining nodes to version: 4d2e420366928e0b369e0ac37136da25157c208b
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1153.81 txn/s, submitted: 1155.58 txn/s, failed submission: 1.77 txn/s, expired: 1.77 txn/s, latency: 2655.69 ms, (p50: 2400 ms, p70: 2900, p90: 4200 ms, p99: 6000 ms), latency samples: 104420
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions]

✅ Forge suite compat success on b29f09f57e898d8d211c8bc3e303f6e50bba2266 ==> 4d2e420366928e0b369e0ac37136da25157c208b

Compatibility test results for b29f09f57e898d8d211c8bc3e303f6e50bba2266 ==> 4d2e420366928e0b369e0ac37136da25157c208b (PR)
1. Check liveness of validators at old version: b29f09f57e898d8d211c8bc3e303f6e50bba2266
compatibility::simple-validator-upgrade::liveness-check : committed: 17230.69 txn/s, latency: 1957.77 ms, (p50: 2000 ms, p70: 2100, p90: 2200 ms, p99: 2800 ms), latency samples: 557720
2. Upgrading first Validator to new version: 4d2e420366928e0b369e0ac37136da25157c208b
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 5667.78 txn/s, latency: 5058.39 ms, (p50: 5700 ms, p70: 6000, p90: 6100 ms, p99: 6200 ms), latency samples: 112580
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 5705.49 txn/s, latency: 5634.42 ms, (p50: 6100 ms, p70: 6300, p90: 6800 ms, p99: 7600 ms), latency samples: 188960
3. Upgrading rest of first batch to new version: 4d2e420366928e0b369e0ac37136da25157c208b
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 6084.50 txn/s, latency: 4617.60 ms, (p50: 5200 ms, p70: 5500, p90: 5700 ms, p99: 5900 ms), latency samples: 112160
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 6186.85 txn/s, latency: 5176.46 ms, (p50: 5700 ms, p70: 5800, p90: 6500 ms, p99: 6800 ms), latency samples: 209200
4. upgrading second batch to new version: 4d2e420366928e0b369e0ac37136da25157c208b
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 2314.02 txn/s, latency: 12832.87 ms, (p50: 13600 ms, p70: 17600, p90: 18600 ms, p99: 18900 ms), latency samples: 56640
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 8258.95 txn/s, latency: 3872.20 ms, (p50: 3200 ms, p70: 4700, p90: 5800 ms, p99: 7100 ms), latency samples: 267860
5. check swarm health
Compatibility test for b29f09f57e898d8d211c8bc3e303f6e50bba2266 ==> 4d2e420366928e0b369e0ac37136da25157c208b passed
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking