roachtest: tpccbench/nodes=3/cpu=16/no-admission failed [GCE disk space issue] #73205

cockroach-teamcity · 2021-11-26T22:45:22Z

roachtest.tpccbench/nodes=3/cpu=16/no-admission failed with artifacts on master @ 40f11fead0a0453969634f8ddb0502c1f78b2806:

		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:905
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:779
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2054
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.loadTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:969
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1067
		  | main.(*monitorImpl).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:106
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (6) output in run_222326.961210894_n1_cockroach_workload_fixtures_import_tpcc
		Wraps: (7) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-3804493-1637959164-29-n4cpu16:1 -- ././cockroach workload fixtures import tpcc --warehouses=3000 --checks=false returned
		  | stderr:
		  | I211126 22:23:28.076992 1 ccl/workloadccl/fixture.go:345  [-] 1  starting import of 9 tables
		  | I211126 22:23:30.582952 86 ccl/workloadccl/fixture.go:502  [-] 2  imported 160 KiB in warehouse table (3000 rows, 0 index entries, took 2.347626758s, 0.07 MiB/s)
		  | I211126 22:23:33.282231 87 ccl/workloadccl/fixture.go:502  [-] 3  imported 3.0 MiB in district table (30000 rows, 0 index entries, took 5.047027428s, 0.59 MiB/s)
		  | I211126 22:23:33.282895 92 ccl/workloadccl/fixture.go:502  [-] 4  imported 7.8 MiB in item table (100000 rows, 0 index entries, took 5.047544657s, 1.54 MiB/s)
		  | I211126 22:24:06.629364 91 ccl/workloadccl/fixture.go:502  [-] 5  imported 383 MiB in new_order table (27000000 rows, 0 index entries, took 38.394015956s, 9.98 MiB/s)
		  | I211126 22:31:31.894945 90 ccl/workloadccl/fixture.go:502  [-] 6  imported 4.7 GiB in order table (90000000 rows, 90000000 index entries, took 8m3.65964246s, 9.98 MiB/s)
		  | I211126 22:32:52.132528 89 ccl/workloadccl/fixture.go:502  [-] 7  imported 6.5 GiB in history table (90000000 rows, 0 index entries, took 9m23.897243751s, 11.72 MiB/s)
		  | W211126 22:45:05.443347 1 ccl/workloadccl/fixture.go:553  [-] 8  error enabling automatic stats: dial tcp 127.0.0.1:26257: connect: connection refused
		  | Error: importing fixture: importing table stock: dial tcp 127.0.0.1:26257: connect: connection refused
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ``````
		  |   | ././cockroach workload fixtures import tpcc --warehouses=3000 --checks=false
		  |   | ``````
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (8) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) *cluster.WithCommandDetails (8) *exec.ExitError

	cluster.go:1343,context.go:91,cluster.go:1333,test_runner.go:867: dead node detection: 1: dead (exit status 1)

Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/kv-triage _{This test on roachdash | Improve this report!}

The text was updated successfully, but these errors were encountered:

cockroach-teamcity · 2021-11-27T10:38:10Z

roachtest.tpccbench/nodes=3/cpu=16 failed with artifacts on master @ b450fea83a7db1e06403b2563c13f38c9284b932:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpccbench/nodes=3/cpu=16/run_1
	monitor.go:128,tpcc.go:1069,tpcc.go:905,test_runner.go:779: monitor failure: unexpected node event: 2: dead (exit status 1)
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
		  | main.(*monitorImpl).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1069
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:905
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:779
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (2) monitor failure
		Wraps: (3) unexpected node event: 2: dead (exit status 1)
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

	cluster.go:1343,context.go:91,cluster.go:1333,test_runner.go:867: dead node detection: 2: dead (exit status 1)

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: tpccbench/nodes=3/cpu=16/admission failed #72860 roachtest: tpccbench/nodes=3/cpu=16/admission failed [C-test-failure O-roachtest O-robot branch-release-21.2]
roachtest: tpccbench/nodes=3/cpu=16 failed #72859 roachtest: tpccbench/nodes=3/cpu=16 failed [C-test-failure O-roachtest O-robot branch-release-21.2]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2021-11-27T12:21:07Z

roachtest.tpccbench/nodes=3/cpu=16/no-admission failed with artifacts on master @ b450fea83a7db1e06403b2563c13f38c9284b932:

		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:905
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:779
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2054
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.loadTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:969
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1067
		  | main.(*monitorImpl).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:106
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (6) output in run_115912.561281890_n1_cockroach_workload_fixtures_import_tpcc
		Wraps: (7) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-3807010-1638000226-19-n4cpu16:1 -- ././cockroach workload fixtures import tpcc --warehouses=3000 --checks=false returned
		  | stderr:
		  | I211127 11:59:13.730141 1 ccl/workloadccl/fixture.go:345  [-] 1  starting import of 9 tables
		  | I211127 11:59:14.140456 52 ccl/workloadccl/fixture.go:502  [-] 2  imported 160 KiB in warehouse table (3000 rows, 0 index entries, took 236.942441ms, 0.66 MiB/s)
		  | I211127 11:59:14.184640 53 ccl/workloadccl/fixture.go:502  [-] 3  imported 3.0 MiB in district table (30000 rows, 0 index entries, took 281.074584ms, 10.51 MiB/s)
		  | I211127 11:59:24.004905 58 ccl/workloadccl/fixture.go:502  [-] 4  imported 7.8 MiB in item table (100000 rows, 0 index entries, took 10.101191816s, 0.77 MiB/s)
		  | I211127 11:59:57.222179 57 ccl/workloadccl/fixture.go:502  [-] 5  imported 383 MiB in new_order table (27000000 rows, 0 index entries, took 43.31843719s, 8.84 MiB/s)
		  | I211127 12:07:34.061088 56 ccl/workloadccl/fixture.go:502  [-] 6  imported 4.7 GiB in order table (90000000 rows, 90000000 index entries, took 8m20.157369166s, 9.65 MiB/s)
		  | I211127 12:09:02.716656 55 ccl/workloadccl/fixture.go:502  [-] 7  imported 6.5 GiB in history table (90000000 rows, 0 index entries, took 9m48.813043262s, 11.23 MiB/s)
		  | W211127 12:20:50.618125 1 ccl/workloadccl/fixture.go:553  [-] 8  error enabling automatic stats: dial tcp 127.0.0.1:26257: connect: connection refused
		  | Error: importing fixture: importing table customer: dial tcp 127.0.0.1:26257: connect: connection refused
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ``````
		  |   | ././cockroach workload fixtures import tpcc --warehouses=3000 --checks=false
		  |   | ``````
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (8) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) *cluster.WithCommandDetails (8) *exec.ExitError

	cluster.go:1343,context.go:91,cluster.go:1333,test_runner.go:867: dead node detection: 3: dead (exit status 1)

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2021-11-28T09:21:59Z

roachtest.tpccbench/nodes=3/cpu=16 failed with artifacts on master @ 3b30a0e12f9a14b08ee8ad55b50299aca50c67a2:

		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:905
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:779
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2054
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.loadTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:969
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1067
		  | main.(*monitorImpl).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:106
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (6) output in run_090004.507267095_n1_cockroach_workload_fixtures_import_tpcc
		Wraps: (7) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-3807326-1638083317-29-n4cpu16:1 -- ././cockroach workload fixtures import tpcc --warehouses=3000 --checks=false returned
		  | stderr:
		  | I211128 09:00:05.620717 1 ccl/workloadccl/fixture.go:345  [-] 1  starting import of 9 tables
		  | I211128 09:00:06.946028 114 ccl/workloadccl/fixture.go:502  [-] 2  imported 160 KiB in warehouse table (3000 rows, 0 index entries, took 1.14377969s, 0.14 MiB/s)
		  | I211128 09:00:07.502346 115 ccl/workloadccl/fixture.go:502  [-] 3  imported 3.0 MiB in district table (30000 rows, 0 index entries, took 1.700024281s, 1.74 MiB/s)
		  | I211128 09:00:12.589552 120 ccl/workloadccl/fixture.go:502  [-] 4  imported 7.8 MiB in item table (100000 rows, 0 index entries, took 6.786945527s, 1.15 MiB/s)
		  | I211128 09:00:46.591006 119 ccl/workloadccl/fixture.go:502  [-] 5  imported 383 MiB in new_order table (27000000 rows, 0 index entries, took 40.78852105s, 9.39 MiB/s)
		  | I211128 09:07:29.670350 118 ccl/workloadccl/fixture.go:502  [-] 6  imported 4.7 GiB in order table (90000000 rows, 90000000 index entries, took 7m23.867917411s, 10.88 MiB/s)
		  | I211128 09:09:44.424800 117 ccl/workloadccl/fixture.go:502  [-] 7  imported 6.5 GiB in history table (90000000 rows, 0 index entries, took 9m38.622427824s, 11.42 MiB/s)
		  | W211128 09:21:42.874240 1 ccl/workloadccl/fixture.go:553  [-] 8  error enabling automatic stats: dial tcp 127.0.0.1:26257: connect: connection refused
		  | Error: importing fixture: importing table stock: dial tcp 127.0.0.1:26257: connect: connection refused
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ``````
		  |   | ././cockroach workload fixtures import tpcc --warehouses=3000 --checks=false
		  |   | ``````
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (8) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) *cluster.WithCommandDetails (8) *exec.ExitError

	cluster.go:1343,context.go:91,cluster.go:1333,test_runner.go:867: dead node detection: 2: dead (exit status 1)

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: tpccbench/nodes=3/cpu=16/admission failed #72860 roachtest: tpccbench/nodes=3/cpu=16/admission failed [C-test-failure O-roachtest O-robot branch-release-21.2]
roachtest: tpccbench/nodes=3/cpu=16 failed #72859 roachtest: tpccbench/nodes=3/cpu=16 failed [C-test-failure O-roachtest O-robot branch-release-21.2]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2021-11-28T09:25:07Z

roachtest.tpccbench/nodes=3/cpu=16/no-admission failed with artifacts on master @ 3b30a0e12f9a14b08ee8ad55b50299aca50c67a2:

		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:905
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:779
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2054
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.loadTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:969
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1067
		  | main.(*monitorImpl).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:106
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (6) output in run_090308.207134803_n1_cockroach_workload_fixtures_import_tpcc
		Wraps: (7) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-3807326-1638083317-26-n4cpu16:1 -- ././cockroach workload fixtures import tpcc --warehouses=3000 --checks=false returned
		  | stderr:
		  | I211128 09:03:09.279062 1 ccl/workloadccl/fixture.go:345  [-] 1  starting import of 9 tables
		  | I211128 09:03:12.856135 19 ccl/workloadccl/fixture.go:502  [-] 2  imported 160 KiB in warehouse table (3000 rows, 0 index entries, took 3.426397855s, 0.05 MiB/s)
		  | I211128 09:03:14.162297 20 ccl/workloadccl/fixture.go:502  [-] 3  imported 3.0 MiB in district table (30000 rows, 0 index entries, took 4.732484838s, 0.62 MiB/s)
		  | I211128 09:03:14.439490 25 ccl/workloadccl/fixture.go:502  [-] 4  imported 7.8 MiB in item table (100000 rows, 0 index entries, took 5.009555795s, 1.55 MiB/s)
		  | I211128 09:03:48.912961 24 ccl/workloadccl/fixture.go:502  [-] 5  imported 383 MiB in new_order table (27000000 rows, 0 index entries, took 39.483083503s, 9.70 MiB/s)
		  | I211128 09:11:50.264375 23 ccl/workloadccl/fixture.go:502  [-] 6  imported 4.7 GiB in order table (90000000 rows, 90000000 index entries, took 8m40.834497014s, 9.27 MiB/s)
		  | I211128 09:12:54.147717 22 ccl/workloadccl/fixture.go:502  [-] 7  imported 6.5 GiB in history table (90000000 rows, 0 index entries, took 9m44.71791097s, 11.31 MiB/s)
		  | W211128 09:24:46.454740 1 ccl/workloadccl/fixture.go:553  [-] 8  error enabling automatic stats: dial tcp 127.0.0.1:26257: connect: connection refused
		  | Error: importing fixture: importing table customer: dial tcp 127.0.0.1:26257: connect: connection refused
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ``````
		  |   | ././cockroach workload fixtures import tpcc --warehouses=3000 --checks=false
		  |   | ``````
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (8) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) *cluster.WithCommandDetails (8) *exec.ExitError

	cluster.go:1343,context.go:91,cluster.go:1333,test_runner.go:867: dead node detection: 3: dead (exit status 1)

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2021-11-29T09:16:50Z

roachtest.tpccbench/nodes=3/cpu=16 failed with artifacts on master @ 2c014c47c1a242f504f6d595bfd79c0edc20b90a:

		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:905
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:779
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2054
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.loadTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:969
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1067
		  | main.(*monitorImpl).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:106
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (6) output in run_085455.272529126_n1_cockroach_workload_fixtures_import_tpcc
		Wraps: (7) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-3810278-1638169717-28-n4cpu16:1 -- ././cockroach workload fixtures import tpcc --warehouses=3000 --checks=false returned
		  | stderr:
		  | I211129 08:54:56.385121 1 ccl/workloadccl/fixture.go:345  [-] 1  starting import of 9 tables
		  | I211129 08:55:01.065323 114 ccl/workloadccl/fixture.go:502  [-] 2  imported 160 KiB in warehouse table (3000 rows, 0 index entries, took 4.513915447s, 0.03 MiB/s)
		  | I211129 08:55:02.259365 115 ccl/workloadccl/fixture.go:502  [-] 3  imported 3.0 MiB in district table (30000 rows, 0 index entries, took 5.7078314s, 0.52 MiB/s)
		  | I211129 08:55:02.472413 120 ccl/workloadccl/fixture.go:502  [-] 4  imported 7.8 MiB in item table (100000 rows, 0 index entries, took 5.920700271s, 1.31 MiB/s)
		  | I211129 08:55:32.074493 119 ccl/workloadccl/fixture.go:502  [-] 5  imported 383 MiB in new_order table (27000000 rows, 0 index entries, took 35.522841068s, 10.78 MiB/s)
		  | I211129 09:03:13.474699 118 ccl/workloadccl/fixture.go:502  [-] 6  imported 4.7 GiB in order table (90000000 rows, 90000000 index entries, took 8m16.92307703s, 9.71 MiB/s)
		  | I211129 09:04:20.453888 117 ccl/workloadccl/fixture.go:502  [-] 7  imported 6.5 GiB in history table (90000000 rows, 0 index entries, took 9m23.902341627s, 11.72 MiB/s)
		  | W211129 09:16:34.799766 1 ccl/workloadccl/fixture.go:553  [-] 8  error enabling automatic stats: dial tcp 127.0.0.1:26257: connect: connection refused
		  | Error: importing fixture: importing table stock: dial tcp 127.0.0.1:26257: connect: connection refused
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ``````
		  |   | ././cockroach workload fixtures import tpcc --warehouses=3000 --checks=false
		  |   | ``````
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (8) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) *cluster.WithCommandDetails (8) *exec.ExitError

	cluster.go:1343,context.go:91,cluster.go:1333,test_runner.go:867: dead node detection: 3: dead (exit status 1)

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: tpccbench/nodes=3/cpu=16/admission failed #72860 roachtest: tpccbench/nodes=3/cpu=16/admission failed [C-test-failure O-roachtest O-robot branch-release-21.2]
roachtest: tpccbench/nodes=3/cpu=16 failed #72859 roachtest: tpccbench/nodes=3/cpu=16 failed [C-test-failure O-roachtest O-robot branch-release-21.2]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2021-11-29T09:22:30Z

roachtest.tpccbench/nodes=3/cpu=16/no-admission failed with artifacts on master @ 2c014c47c1a242f504f6d595bfd79c0edc20b90a:

		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:905
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:779
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2054
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.loadTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:969
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1067
		  | main.(*monitorImpl).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:106
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (6) output in run_090030.319658445_n1_cockroach_workload_fixtures_import_tpcc
		Wraps: (7) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-3810278-1638169717-19-n4cpu16:1 -- ././cockroach workload fixtures import tpcc --warehouses=3000 --checks=false returned
		  | stderr:
		  | I211129 09:00:31.645612 1 ccl/workloadccl/fixture.go:345  [-] 1  starting import of 9 tables
		  | I211129 09:00:35.049361 69 ccl/workloadccl/fixture.go:502  [-] 2  imported 160 KiB in warehouse table (3000 rows, 0 index entries, took 3.247567069s, 0.05 MiB/s)
		  | I211129 09:00:35.763765 70 ccl/workloadccl/fixture.go:502  [-] 3  imported 3.0 MiB in district table (30000 rows, 0 index entries, took 3.961972283s, 0.75 MiB/s)
		  | I211129 09:00:35.981221 75 ccl/workloadccl/fixture.go:502  [-] 4  imported 7.8 MiB in item table (100000 rows, 0 index entries, took 4.179231946s, 1.86 MiB/s)
		  | I211129 09:01:16.906373 74 ccl/workloadccl/fixture.go:502  [-] 5  imported 383 MiB in new_order table (27000000 rows, 0 index entries, took 45.104462336s, 8.49 MiB/s)
		  | I211129 09:09:34.798065 73 ccl/workloadccl/fixture.go:502  [-] 6  imported 4.7 GiB in order table (90000000 rows, 90000000 index entries, took 9m2.996156272s, 8.89 MiB/s)
		  | I211129 09:10:34.038482 72 ccl/workloadccl/fixture.go:502  [-] 7  imported 6.5 GiB in history table (90000000 rows, 0 index entries, took 10m2.236621854s, 10.98 MiB/s)
		  | W211129 09:22:12.662461 1 ccl/workloadccl/fixture.go:553  [-] 8  error enabling automatic stats: dial tcp 127.0.0.1:26257: connect: connection refused
		  | Error: importing fixture: importing table order_line: dial tcp 127.0.0.1:26257: connect: connection refused
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ``````
		  |   | ././cockroach workload fixtures import tpcc --warehouses=3000 --checks=false
		  |   | ``````
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (8) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) *cluster.WithCommandDetails (8) *exec.ExitError

	cluster.go:1343,context.go:91,cluster.go:1333,test_runner.go:867: dead node detection: 3: dead (exit status 1)

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

AlexTalks mentioned this issue Nov 30, 2021

roachtest: tpccbench/nodes=3/cpu=16 failed [GCE disk space issue] #73204

Closed

AlexTalks changed the title ~~roachtest: tpccbench/nodes=3/cpu=16/no-admission failed~~ roachtest: tpccbench/nodes=3/cpu=16/no-admission failed [GCE disk space issue] Nov 30, 2021

AlexTalks removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Nov 30, 2021

This was referenced Nov 30, 2021

roachtest: oom due to WorkloadKVConverter #68965

Closed

roachtest: tpccbench/nodes=9/cpu=4/multi-region failed [self-delegated snaps] #72083

Closed

healthy-pod linked a pull request Nov 30, 2021 that will close this issue

roachtest: ensure ssd count is not zero when using local ssd #73248

Merged

healthy-pod closed this as completed Nov 30, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

roachtest: tpccbench/nodes=3/cpu=16/no-admission failed [GCE disk space issue] #73205

roachtest: tpccbench/nodes=3/cpu=16/no-admission failed [GCE disk space issue] #73205

cockroach-teamcity commented Nov 26, 2021

cockroach-teamcity commented Nov 27, 2021

cockroach-teamcity commented Nov 27, 2021

cockroach-teamcity commented Nov 28, 2021

cockroach-teamcity commented Nov 28, 2021

cockroach-teamcity commented Nov 29, 2021

cockroach-teamcity commented Nov 29, 2021

roachtest: tpccbench/nodes=3/cpu=16/no-admission failed [GCE disk space issue] #73205

roachtest: tpccbench/nodes=3/cpu=16/no-admission failed [GCE disk space issue] #73205

Comments

cockroach-teamcity commented Nov 26, 2021

cockroach-teamcity commented Nov 27, 2021

cockroach-teamcity commented Nov 27, 2021

cockroach-teamcity commented Nov 28, 2021

cockroach-teamcity commented Nov 28, 2021

cockroach-teamcity commented Nov 29, 2021

cockroach-teamcity commented Nov 29, 2021