roachtest: tpccbench/nodes=3/cpu=16 failed [GCE disk space issue]

[cockroach-teamcity]

roachtest.tpccbench/nodes=3/cpu=16 failed with artifacts on master @ 40f11fead0a0453969634f8ddb0502c1f78b2806:

		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:905
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:779
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2054
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.loadTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:969
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1067
		  | main.(*monitorImpl).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:106
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (6) output in run_222026.000266563_n1_cockroach_workload_fixtures_import_tpcc
		Wraps: (7) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-3804493-1637959164-28-n4cpu16:1 -- ././cockroach workload fixtures import tpcc --warehouses=3000 --checks=false returned
		  | stderr:
		  | I211126 22:20:27.112826 1 ccl/workloadccl/fixture.go:345  [-] 1  starting import of 9 tables
		  | I211126 22:20:27.473164 25 ccl/workloadccl/fixture.go:502  [-] 2  imported 160 KiB in warehouse table (3000 rows, 0 index entries, took 218.486404ms, 0.71 MiB/s)
		  | I211126 22:20:28.337918 26 ccl/workloadccl/fixture.go:502  [-] 3  imported 3.0 MiB in district table (30000 rows, 0 index entries, took 1.08328872s, 2.73 MiB/s)
		  | I211126 22:20:37.937705 31 ccl/workloadccl/fixture.go:502  [-] 4  imported 7.8 MiB in item table (100000 rows, 0 index entries, took 10.682880376s, 0.73 MiB/s)
		  | I211126 22:21:16.510970 30 ccl/workloadccl/fixture.go:502  [-] 5  imported 383 MiB in new_order table (27000000 rows, 0 index entries, took 49.256201837s, 7.78 MiB/s)
		  | I211126 22:29:06.737742 29 ccl/workloadccl/fixture.go:502  [-] 6  imported 4.7 GiB in order table (90000000 rows, 90000000 index entries, took 8m39.4829546s, 9.29 MiB/s)
		  | I211126 22:30:14.617607 28 ccl/workloadccl/fixture.go:502  [-] 7  imported 6.5 GiB in history table (90000000 rows, 0 index entries, took 9m47.362916915s, 11.25 MiB/s)
		  | W211126 22:42:06.464705 1 ccl/workloadccl/fixture.go:553  [-] 8  error enabling automatic stats: dial tcp 127.0.0.1:26257: connect: connection refused
		  | Error: importing fixture: importing table customer: dial tcp 127.0.0.1:26257: connect: connection refused
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ``````
		  |   | ././cockroach workload fixtures import tpcc --warehouses=3000 --checks=false
		  |   | ``````
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (8) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) *cluster.WithCommandDetails (8) *exec.ExitError

	cluster.go:1343,context.go:91,cluster.go:1333,test_runner.go:867: dead node detection: 3: dead (exit status 1)

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

• roachtest: tpccbench/nodes=3/cpu=16/admission failed #72860 roachtest: tpccbench/nodes=3/cpu=16/admission failed [C-test-failure O-roachtest O-robot branch-release-21.2]
• roachtest: tpccbench/nodes=3/cpu=16 failed #72859 roachtest: tpccbench/nodes=3/cpu=16 failed [C-test-failure O-roachtest O-robot branch-release-21.2]

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

[AlexTalks]

This test, similar to other tpccbench test failures in #73205 and #73222, seems to be failing due to lack on disk space, specifically on GCE runs (as AWS runs are passing cleanly). Hopefully these should be fixed by #73248 - currently investigating.