roachtest: gossip/chaos/nodes=9 failed

[cockroach-teamcity]

roachtest.gossip/chaos/nodes=9 failed with artifacts on master @ 9e1cd533828b7887a48db8635f705669287cf2d6:

(gossip.go:81).2: gossip did not stabilize (dead node 2) in 56.4s
test artifacts and logs in: /artifacts/gossip/chaos/nodes=9/cpu_arch=arm64/run_1

Parameters:

• ROACHTEST_arch=arm64
• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=false
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-39970

[arulajmani]

We hypothesized that the test was sleeping longer than it's intended 1s duration, and added more logging in #126087 to prove this. It seems like we are indeed sleeping for longer than 1s, but it still doesn't account for the 56s wait to print the "checking if gossip excludes dead node" log line.

I'm not sure what to make of this, but I'll close this out as a duplicate of #124828.

2024/07/02 09:43:18 gossip.go:92: 1: gossip not ok (dead node 2 present) (3s)
2024/07/02 09:43:18 gossip.go:132: sleeping for 1s (3s)
2024/07/02 09:44:11 gossip.go:126: checking if gossip excludes dead node 2 (56s)