roachtest: clearrange/checks=true failed #75701

cockroach-teamcity · 2022-01-29T15:55:45Z

roachtest.clearrange/checks=true failed with artifacts on master @ bb85ddc4fc0ad8761d7aba61315da22b433022e3:

The test failed on branch=master, cloud=gce:
test timed out (see artifacts for details)

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: clearrange/zfs/checks=true failed #70306 roachtest: clearrange/zfs/checks=true failed [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

/cc @cockroachdb/storage _{This test on roachdash | Improve this report!}

The text was updated successfully, but these errors were encountered:

cockroach-teamcity · 2022-01-30T12:09:15Z

roachtest.clearrange/checks=true failed with artifacts on master @ a5158c4c93e9dca858b38ef7e94321f0a0a2b5c5:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/clearrange/checks=true/run_1
	cluster.go:1915,clearrange.go:86,clearrange.go:39,test_runner.go:780: one or more parallel execution failure
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).ParallelE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2035
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Parallel
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:1919
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Start
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/roachprod/install/cockroach.go:164
		  | github.com/cockroachdb/cockroach/pkg/roachprod.Start
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:659
		  | main.(*clusterImpl).StartE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1873
		  | main.(*clusterImpl).Start
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1914
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runClearRange
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:86
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClearRange.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:39
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:780
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (2) one or more parallel execution failure
		Error types: (1) *withstack.withStack (2) *errutil.leafError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: clearrange/zfs/checks=true failed #70306 roachtest: clearrange/zfs/checks=true failed [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-02-01T11:47:20Z

roachtest.clearrange/checks=true failed with artifacts on master @ 8548987813ff9e1b8a9878023d3abfc6911c16db:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /artifacts/clearrange/checks=true/run_1
	cluster.go:1915,clearrange.go:86,clearrange.go:39,test_runner.go:780: one or more parallel execution failure
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).ParallelE
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2035
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Parallel
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:1919
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Start
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/install/cockroach.go:164
		  | github.com/cockroachdb/cockroach/pkg/roachprod.Start
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:659
		  | main.(*clusterImpl).StartE
		  | 	main/pkg/cmd/roachtest/cluster.go:1873
		  | main.(*clusterImpl).Start
		  | 	main/pkg/cmd/roachtest/cluster.go:1914
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runClearRange
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:86
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClearRange.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:39
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:780
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) one or more parallel execution failure
		Error types: (1) *withstack.withStack (2) *errutil.leafError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: clearrange/zfs/checks=true failed #70306 roachtest: clearrange/zfs/checks=true failed [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-02-01T12:36:20Z

roachtest.clearrange/checks=true failed with artifacts on master @ 95c82a7dbc6d6fecb63edbd666c9986d1ad3885b:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/clearrange/checks=true/run_1
	cluster.go:1915,clearrange.go:86,clearrange.go:39,test_runner.go:780: one or more parallel execution failure
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).ParallelE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2035
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Parallel
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:1919
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Start
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/roachprod/install/cockroach.go:164
		  | github.com/cockroachdb/cockroach/pkg/roachprod.Start
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:659
		  | main.(*clusterImpl).StartE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1873
		  | main.(*clusterImpl).Start
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1914
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runClearRange
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:86
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClearRange.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:39
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:780
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (2) one or more parallel execution failure
		Error types: (1) *withstack.withStack (2) *errutil.leafError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: clearrange/zfs/checks=true failed #70306 roachtest: clearrange/zfs/checks=true failed [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-02-02T10:44:00Z

roachtest.clearrange/checks=true failed with artifacts on master @ 8548987813ff9e1b8a9878023d3abfc6911c16db:

		  | I220202 10:43:34.205696 88 workload/pgx_helpers.go:79  [-] 29  pgx logger [error]: Exec logParams=map[args:[-9105963748652747532 e3] err:unexpected EOF sql:kv-2]
		  | I220202 10:43:34.205711 346 workload/pgx_helpers.go:79  [-] 30  pgx logger [error]: Exec logParams=map[args:[-8814401828926534952 d4] err:unexpected EOF sql:kv-2]
		  | I220202 10:43:34.205727 87 workload/pgx_helpers.go:79  [-] 31  pgx logger [error]: Exec logParams=map[args:[-1639961889401908716 91] err:unexpected EOF sql:kv-2]
		  | I220202 10:43:34.205741 344 workload/pgx_helpers.go:79  [-] 32  pgx logger [error]: Exec logParams=map[args:[-3057243357848094483 35] err:unexpected EOF sql:kv-2]
		  | I220202 10:43:34.205756 340 workload/pgx_helpers.go:79  [-] 33  pgx logger [error]: Exec logParams=map[args:[-3001014459915439206 be] err:unexpected EOF sql:kv-2]
		  | I220202 10:43:34.205770 339 workload/pgx_helpers.go:79  [-] 34  pgx logger [error]: Exec logParams=map[args:[-8786960124782161664 19] err:unexpected EOF sql:kv-2]
		  | Error: unexpected EOF
		  | COMMAND_PROBLEM: exit status 1
		  |    7: 
		  | UNCLASSIFIED_PROBLEM: context canceled
		  |    8: 
		  | UNCLASSIFIED_PROBLEM: context canceled
		  |    9: 
		  | UNCLASSIFIED_PROBLEM: context canceled
		  |   10: 
		  | UNCLASSIFIED_PROBLEM: context canceled
		Wraps: (4) secondary error attachment
		  | COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 6. Command with error:
		  |   | ``````
		  |   | ./cockroach workload run kv --concurrency=32 --duration=1h
		  |   | ``````
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString

	monitor.go:127,clearrange.go:207,clearrange.go:39,test_runner.go:780: monitor failure: monitor command failure: unexpected node event: 6: dead (exit status 10)
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runClearRange
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:207
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClearRange.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:39
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func3
		  | 	main/pkg/cmd/roachtest/monitor.go:202
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) monitor command failure
		Wraps: (5) unexpected node event: 6: dead (exit status 10)
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *errors.errorString

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: clearrange/zfs/checks=true failed #70306 roachtest: clearrange/zfs/checks=true failed [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-02-02T12:13:53Z

roachtest.clearrange/checks=true failed with artifacts on master @ 16859b99637d1adedb915a67998749dbfa4c0663:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/clearrange/checks=true/run_1
	cluster.go:1915,clearrange.go:86,clearrange.go:39,test_runner.go:780: one or more parallel execution failure
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).ParallelE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2035
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Parallel
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:1919
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Start
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/roachprod/install/cockroach.go:164
		  | github.com/cockroachdb/cockroach/pkg/roachprod.Start
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:659
		  | main.(*clusterImpl).StartE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1873
		  | main.(*clusterImpl).Start
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1914
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runClearRange
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:86
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClearRange.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:39
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:780
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (2) one or more parallel execution failure
		Error types: (1) *withstack.withStack (2) *errutil.leafError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: clearrange/zfs/checks=true failed #70306 roachtest: clearrange/zfs/checks=true failed [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-02-04T14:48:10Z

roachtest.clearrange/checks=true failed with artifacts on master @ 8548987813ff9e1b8a9878023d3abfc6911c16db:

The test failed on branch=master, cloud=gce:
test timed out (see artifacts for details)

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: clearrange/checks=true failed #76045 roachtest: clearrange/checks=true failed [C-test-failure O-roachtest O-robot T-storage branch-release-21.1 release-blocker]
roachtest: clearrange/zfs/checks=true failed #70306 roachtest: clearrange/zfs/checks=true failed [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-02-05T14:59:27Z

roachtest.clearrange/checks=true failed with artifacts on master @ 8548987813ff9e1b8a9878023d3abfc6911c16db:

		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | .go:386  [-] 2 +	github.com/klauspost/compress/zstd/external/com_github_klauspost_compress/zstd/blockdec.go:117 +0x187
		  | I220205 14:53:43.209340 21 util/log/clog.go:386  [-] 2 +
		  | I220205 14:53:43.209340 21 util/log/clog.go:386  [-] 2 +goroutine 88 [chan receive, 389 minutes]:
		  | I220205 14:53:43.209340 21 util/log/clog.go:386  [-] 2 +github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000b40cc0)
		  | I220205 14:53:43.209340 21 util/log/clog.go:386  [-] 2 +	github.com/klauspost/compress/zstd/external/com_github_klauspost_compress/zstd/blockdec.go:211 +0x94
		  | I220205 14:53:43.209340 21 util/log/clog.go:386  [-] 2 +created by github.com/klauspost/compress/zstd.newBlockDec
		  | I220205 14:53:43.209340 21 util/log/clog.go:386  [-] 2 +	github.com/klauspost/compress/zstd/external/com_github_klauspost_compress/zstd/blockdec.go:117 +0x187
		  | I220205 14:53:43.209340 21 util/log/clog.go:386  [-] 2 +
		  | I220205 14:53:43.209340 21 util/log/clog.go:386  [-] 2 +goroutine 89 [chan receive, 389 minutes]:
		  | I220205 14:53:43.209340 21 util/log/clog.go:386  [-] 2 +github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000b40d80)
		  | I220205 14:53:43.209340 21 util/log/clog.go:386  [-] 2 +	github.com/klauspost/compress/zstd/external/com_github_klauspost_compress/zstd/blockdec.go:211 +0x94
		  | I220205 14:53:43.209340 21 util/log/clog.go:386  [-] 2 +created by github.com/klauspost/compress/zstd.newBlockDec
		  | I220205 14:53:43.209340 21 util/log/clog.go:386  [-] 2 +	github.com/klauspost/compress/zstd/external/com_github_klauspost_compress/zstd/blockdec.go:117 +0x187
		  | I220205 14:53:43.209340 21 util/log/clog.go:386  [-] 2 +
		  | I220205 14:53:43.209340 21 util/log/clog.go:386  [-] 2 +goroutine 90 [chan receive, 389 minutes]:
		  | I220205 14:53:43.209340 21 util/log/clog.go:386  [-] 2 +github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000b40e40)
		  | I220205 14:53:43.209340 21 util/log/clog.go:386  [-] 2 +	github.com/klauspost/compress/zstd/external/com_github_klauspost_compress/zstd/blockdec.go:211 +0x94
		  | I220205 14:53:43.209340 21 util/log/clog.go:386  [-] 2 +created by github.com/klauspost/compress/zstd.newBlockDec
		  | I220205 14:53:43.209340 21 util/log/clog.go:386  [-] 2 +	github.com/klauspost/compress/zstd/external/com_github_klauspost_compress/zstd/blockdec.go:117 +0x187
		  | I220205 14:53:43.209340 21 util/log/clog.go:386  [-] 2 +
		  | I220205 14:53:43.209340 21 util/log/clog.go:386  [-] 2 +goroutine 91 [select]:
		  | I220205 14:53:43.209340 21 util/log/clog.go:386  [-] 2 +go.opencensus.io/stats/view.(*worker).start(0xc000b72d00)
		  | I220205 14:53:43.209340 21 util/log/clog.go:386  [-] 2 +	go.opencensus.io/stats/view/external/io_opencensus_go/stats/view/worker.go:276 +0xb9
		  | I220205 14:53:43.209340 21 util/log/clog.go:386  [-] 2 +created by go.opencensus.io/stats/view.init.0
		  | I220205 14:53:43.209340 21 util/log/clog.go:386  [-] 2 +	go.opencensus.io/stats/view/external/io_opencensus_go/stats/view/worker.go:34 +0x92
		  | I220205 14:53:43.209340 21 util/log/clog.go:386  [-] 2 +
		  | I220205 14:53:43.209340 21 util/log/clog.go:386  [-] 2 +goroutine 22 [select, 389 minutes]:
		  | I220205 14:53:43.209340 21 util/log/clog.go:386  [-] 2 +github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup.func2()
		  | I220205 14:53:43.209340 21 util/log/clog.go:386  [-] 2 +	github.com/cockroachdb/cockroach/pkg/cli/cli.go:349 +0xab
		  | I220205 14:53:43.209340 21 util/log/clog.go:386  [-] 2 +created by github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup
		  | I220205 14:53:43.209340 21 util/log/clog.go:386  [-] 2 +	github.com/cockroachdb/cockroach/pkg/cli/cli.go:347 +0x1f6
		  | I220205 14:53:43.209340 21 util/log/clog.go:386  [-] 2 +
		  | I220205 14:53:43.209340 21 util/log/clog.go:386  [-] 2 +goroutine 23 [select, 389 minutes]:
		  | I220205 14:53:43.209340 21 util/log/clog.go:386  [-] 2 +database/sql.(*DB).connectionOpener(0xc000041a00, {0x6058730, 0xc000bcd140})
		  | I220205 14:53:43.209340 21 util/log/clog.go:386  [-] 2 +	GOROOT/src/database/sql/sql.go:1196 +0x93
		  | I220205 14:53:43.209340 21 util/log/clog.go:386  [-] 2 +created by database/sql.OpenDB
		  | I220205 14:53:43.209340 21 util/log/clog.go:386  [-] 2 +	GOROOT/src/database/sql/sql.go:794 +0x188
		  |
		  | stdout:
		Wraps: (4) SSH_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./cockroach workload fixtures import bank --payload-bytes=10240 --ranges=10 --rows=65104166 --seed=4 --db=bigbank
		  | ``````
		Wraps: (6) exit status 255
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError

	test_runner.go:909,test_runner.go:808: test timed out (6h30m0s)

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: clearrange/checks=true failed #76045 roachtest: clearrange/checks=true failed [C-test-failure O-roachtest O-robot T-storage branch-release-21.1 release-blocker]
roachtest: clearrange/zfs/checks=true failed #70306 roachtest: clearrange/zfs/checks=true failed [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-02-06T09:33:54Z

roachtest.clearrange/checks=true failed with artifacts on master @ 8548987813ff9e1b8a9878023d3abfc6911c16db:

		  | I220206 09:33:21.055983 240 workload/pgx_helpers.go:79  [-] 6  pgx logger [error]: Exec logParams=map[args:[2712643461550494102 a7] err:unexpected EOF sql:kv-2]
		  | I220206 09:33:21.055946 328 workload/pgx_helpers.go:79  [-] 7  pgx logger [error]: Exec logParams=map[args:[-6723764846806033098 91] err:unexpected EOF sql:kv-2]
		  | Error: unexpected EOF
		  | COMMAND_PROBLEM: exit status 1
		  |    5: 
		  | UNCLASSIFIED_PROBLEM: context canceled
		  |    6: 
		  | UNCLASSIFIED_PROBLEM: context canceled
		  |    7: 
		  | UNCLASSIFIED_PROBLEM: context canceled
		  |    8: 
		  | UNCLASSIFIED_PROBLEM: context canceled
		  |    9: 
		  | UNCLASSIFIED_PROBLEM: context canceled
		  |   10: 
		  | UNCLASSIFIED_PROBLEM: context canceled
		Wraps: (4) secondary error attachment
		  | COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 4. Command with error:
		  |   | ``````
		  |   | ./cockroach workload run kv --concurrency=32 --duration=1h
		  |   | ``````
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString

	monitor.go:127,clearrange.go:207,clearrange.go:39,test_runner.go:779: monitor failure: monitor command failure: unexpected node event: 4: dead (exit status 10)
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runClearRange
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:207
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClearRange.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:39
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func3
		  | 	main/pkg/cmd/roachtest/monitor.go:202
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) monitor command failure
		Wraps: (5) unexpected node event: 4: dead (exit status 10)
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *errors.errorString

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: clearrange/checks=true failed #76045 roachtest: clearrange/checks=true failed [C-test-failure O-roachtest O-robot T-storage branch-release-21.1 release-blocker]
roachtest: clearrange/zfs/checks=true failed #70306 roachtest: clearrange/zfs/checks=true failed [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-02-08T09:44:06Z

roachtest.clearrange/checks=true failed with artifacts on master @ a7b1c17c240c935056f87f29dfc1ee75611c371e:

		  | I220208 09:43:27.075445 345 workload/pgx_helpers.go:79  [-] 31  pgx logger [error]: Exec logParams=map[args:[234563670358188951 5f] err:unexpected EOF sql:kv-2]
		  | I220208 09:43:27.075461 359 workload/pgx_helpers.go:79  [-] 32  pgx logger [error]: Exec logParams=map[args:[6094485276672755140 13] err:unexpected EOF sql:kv-2]
		  | I220208 09:43:27.075478 344 workload/pgx_helpers.go:79  [-] 33  pgx logger [error]: Exec logParams=map[args:[-9018874366736943282 0e] err:unexpected EOF sql:kv-2]
		  | I220208 09:43:27.075494 341 workload/pgx_helpers.go:79  [-] 34  pgx logger [error]: Exec logParams=map[args:[1138836797285360995 cf] err:unexpected EOF sql:kv-2]
		  | Error: unexpected EOF
		  | COMMAND_PROBLEM: exit status 1
		  |    6: 
		  | UNCLASSIFIED_PROBLEM: context canceled
		  |    7: 
		  | UNCLASSIFIED_PROBLEM: context canceled
		  |    8: 
		  | UNCLASSIFIED_PROBLEM: context canceled
		  |    9: 
		  | UNCLASSIFIED_PROBLEM: context canceled
		  |   10: 
		  | UNCLASSIFIED_PROBLEM: context canceled
		Wraps: (4) secondary error attachment
		  | COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 5. Command with error:
		  |   | ``````
		  |   | ./cockroach workload run kv --concurrency=32 --duration=1h
		  |   | ``````
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString

	monitor.go:127,clearrange.go:207,clearrange.go:39,test_runner.go:779: monitor failure: monitor command failure: unexpected node event: 5: dead (exit status 10)
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runClearRange
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:207
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClearRange.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:39
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func3
		  | 	main/pkg/cmd/roachtest/monitor.go:202
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) monitor command failure
		Wraps: (5) unexpected node event: 5: dead (exit status 10)
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *errors.errorString

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: clearrange/checks=true failed #76045 roachtest: clearrange/checks=true failed [C-test-failure O-roachtest O-robot T-storage branch-release-21.1 release-blocker]
roachtest: clearrange/zfs/checks=true failed #70306 roachtest: clearrange/zfs/checks=true failed [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

_{This test on roachdash | Improve this report!}

nicktrav · 2022-02-11T21:24:17Z

Looks like we're down to just this clearrange roachtest failing, and it seems a lot more intermittent than it was previously.

Currently running a few stress jobs to see what the replica imbalance looks like (see #75664, which came out of #68303).

nicktrav · 2022-02-12T00:22:33Z

I did three more clearrange/checks=true again on current master and they all passed and the replica distributions looked fine for each run.

Going to close this one out and let it re-open if / when it fires again.

blathers-crl bot added the T-storage Storage Team label Jan 29, 2022

cockroach-teamcity mentioned this issue Feb 4, 2022

roachtest: clearrange/checks=true failed #76045

Closed

nicktrav closed this as completed Feb 12, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

roachtest: clearrange/checks=true failed #75701

roachtest: clearrange/checks=true failed #75701

cockroach-teamcity commented Jan 29, 2022

cockroach-teamcity commented Jan 30, 2022

cockroach-teamcity commented Feb 1, 2022

cockroach-teamcity commented Feb 1, 2022

cockroach-teamcity commented Feb 2, 2022

cockroach-teamcity commented Feb 2, 2022

cockroach-teamcity commented Feb 4, 2022

cockroach-teamcity commented Feb 5, 2022

cockroach-teamcity commented Feb 6, 2022

cockroach-teamcity commented Feb 8, 2022

nicktrav commented Feb 11, 2022

nicktrav commented Feb 12, 2022

roachtest: clearrange/checks=true failed #75701

roachtest: clearrange/checks=true failed #75701

Comments

cockroach-teamcity commented Jan 29, 2022

cockroach-teamcity commented Jan 30, 2022

cockroach-teamcity commented Feb 1, 2022

cockroach-teamcity commented Feb 1, 2022

cockroach-teamcity commented Feb 2, 2022

cockroach-teamcity commented Feb 2, 2022

cockroach-teamcity commented Feb 4, 2022

cockroach-teamcity commented Feb 5, 2022

cockroach-teamcity commented Feb 6, 2022

cockroach-teamcity commented Feb 8, 2022

nicktrav commented Feb 11, 2022

nicktrav commented Feb 12, 2022