roachtest: tpcc/w=100/nodes=3/chaos=true failed

[cockroach-teamcity]

SHA: https://github.com/cockroachdb/cockroach/commits/9f084ad576e85756c5c5a7e41335d9aa2d3eee30

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpcc/w=100/nodes=3/chaos=true PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1112101&tab=buildLog

The test failed on master:
	test.go:743,cluster.go:1195,tpcc.go:130,tpcc.go:182: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1112101-tpcc-w-100-nodes-3-chaos-true:4 -- ./workload check tpcc --warehouses=100 {pgurl:1} returned:
		stderr:
		
		stdout:
		I190128 15:31:29.230660 1 workload/tpcc/tpcc.go:288  check 3.3.2.1 took 29.119545ms
		I190128 15:31:32.440868 1 workload/tpcc/tpcc.go:288  check 3.3.2.2 took 3.21016191s
		Error: check failed: 3.3.2.2: inequality at idx 435: order: 3010.000000, newOrder: 3010.000000, district-1: 3009.000000
		Error:  exit status 1
		: exit status 1

[nvanbenschoten]

Duplicate of #34025. Keeping open for now since it has a different symptom.

[cockroach-teamcity]

SHA: https://github.com/cockroachdb/cockroach/commits/395d842feb97c5bd8cad2b32b71a5156c03061eb

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpcc/w=100/nodes=3/chaos=true PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1115923&tab=buildLog

The test failed on master:
	test.go:743,cluster.go:1226,tpcc.go:118,cluster.go:1564,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1115923-tpcc-w-100-nodes-3-chaos-true:4 -- ./workload run tpcc --warehouses=100 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=30m0s {pgurl:1-3} returned:
		stderr:
		
		stdout:
		    0.0      0.0 orderStatus
		   4m36s  4638542            0.0           41.6      0.0      0.0      0.0      0.0 payment
		   4m36s  4638542            0.0            4.6      0.0      0.0      0.0      0.0 stockLevel
		E190130 15:28:41.709374 1 workload/cli/run.go:402  error in newOrder: dial tcp 10.128.0.38:26257: connect: connection refused
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		   4m37s  4656783            0.0            4.5      0.0      0.0      0.0      0.0 delivery
		   4m37s  4656783            0.0           43.6      0.0      0.0      0.0      0.0 newOrder
		   4m37s  4656783            0.0            4.6      0.0      0.0      0.0      0.0 orderStatus
		   4m37s  4656783            0.0           41.5      0.0      0.0      0.0      0.0 payment
		   4m37s  4656783            0.0            4.5      0.0      0.0      0.0      0.0 stockLevel
		E190130 15:28:42.709447 1 workload/cli/run.go:402  error in orderStatus: dial tcp 10.128.0.38:26257: connect: connection refused
		: signal: killed
	test.go:743,cluster.go:1585,tpcc.go:128,tpcc.go:182: unexpected node event: 1: dead

[nvanbenschoten]

Previous failure was

panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x50 pc=0x190242d]

goroutine 719576 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Recover(0xc0001c5830, 0x3898dc0, 0xc00f811050)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:183 +0x11f
panic(0x2d24dc0, 0x546ff60)
	/usr/local/go/src/runtime/panic.go:513 +0x1b9
github.com/cockroachdb/cockroach/pkg/storage.(*truncateDecision).raftSnapshotsForIndex(0xc0060625f0, 0x0, 0xc0054edb00)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_log_queue.go:226 +0x3d

Fixed by #34399.