roachtest: tpccbench/nodes=3/cpu=4 failed

[cockroach-teamcity]

SHA: https://github.com/cockroachdb/cockroach/commits/395d842feb97c5bd8cad2b32b71a5156c03061eb

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=3/cpu=4 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1115923&tab=buildLog

The test failed on master:
	test.go:743,cluster.go:1585,tpcc.go:705,tpcc.go:391: unexpected node event: 3: dead

[nvanbenschoten]

panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x50 pc=0x190242d]

goroutine 422647 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Recover(0xc0001c5cb0, 0x3898dc0, 0xc016d831a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:183 +0x11f
panic(0x2d24dc0, 0x546ff60)
	/usr/local/go/src/runtime/panic.go:513 +0x1b9
github.com/cockroachdb/cockroach/pkg/storage.(*truncateDecision).raftSnapshotsForIndex(0xc0178e5360, 0x0, 0xc00eed3140)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_log_queue.go:226 +0x3d
github.com/cockroachdb/cockroach/pkg/storage.(*truncateDecision).NumNewRaftSnapshots(0xc0178e5360, 0xc01020de20)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_log_queue.go:254 +0x34
github.com/cockroachdb/cockroach/pkg/storage.(*truncateDecision).String(0xc0178e5360, 0xc013873400, 0xc005efc000)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_log_queue.go:273 +0x24a
github.com/cockroachdb/cockroach/pkg/storage.(*raftLogQueue).process(0xc0000940c0, 0x3898d80, 0xc013873440, 0xc005efc000, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_log_queue.go:449 +0x3d0
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processReplica(0xc00074c500, 0x3898d80, 0xc013873440, 0xc005efc000, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:751 +0x345
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1.2(0x3898dc0, 0xc016d831a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:645 +0xb8
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0001c5cb0, 0x3898dc0, 0xc016d831a0, 0xc013c9c810, 0x2b, 0x0, 0x0, 0xc01020de00)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:323 +0xe6
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:318 +0x134

Duplicate of #34398, fixed by #34399.