Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: clearrange/zfs/checks=false failed #68420

Closed
cockroach-teamcity opened this issue Aug 4, 2021 · 47 comments
Closed

roachtest: clearrange/zfs/checks=false failed #68420

cockroach-teamcity opened this issue Aug 4, 2021 · 47 comments
Assignees
@bananabrick
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. T-storage Storage Team

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Aug 4, 2021 •
edited by cockroach-jira-scripts
Loading

roachtest.clearrange/zfs/checks=false failed with artifacts on master @ eef03a46f2e43ff70485dadf7d9ad445db05cab4:

The test failed on branch=master, cloud=gce:
test timed out (see artifacts for details)
Reproduce

See: roachtest README

See: CI job to stress roachtests

For the CI stress job, click the ellipsis (...) next to the Run button and fill in: * Changes / Build branch: master * Parameters / `env.TESTS`: `^clearrange/zfs/checks=false$` * Parameters / `env.COUNT`: <number of runs>

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

Jira issue: CRDB-9033
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Aug 4, 2021
@blathers-crl blathers-crl bot added the T-storage Storage Team label Aug 4, 2021
@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/zfs/checks=false failed with artifacts on master @ 50ef2fc205baa65c5a740c2d614fe1de279367e9:

The test failed on branch=master, cloud=gce:
test timed out (see artifacts for details)
Reproduce

See: roachtest README

See: CI job to stress roachtests

For the CI stress job, click the ellipsis (...) next to the Run button and fill in: * Changes / Build branch: master * Parameters / `env.TESTS`: `^clearrange/zfs/checks=false$` * Parameters / `env.COUNT`: <number of runs>

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/zfs/checks=false failed with artifacts on master @ 62ec88c61edcaa023a579199cc5b43d3ee951cef:

The test failed on branch=master, cloud=gce:
test timed out (see artifacts for details)
Reproduce

See: roachtest README

See: CI job to stress roachtests

For the CI stress job, click the ellipsis (...) next to the Run button and fill in: * Changes / Build branch: master * Parameters / `env.TESTS`: `^clearrange/zfs/checks=false$` * Parameters / `env.COUNT`: <number of runs>

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/zfs/checks=false failed with artifacts on master @ 847514dab6354d4cc4ccf7b2857487b32119fb37:

		3: 14399
		8: 13983
		4: 13911
		10: 14242
		5: 14673
		2: 14514
		Error: UNCLASSIFIED_PROBLEM: 9: dead (exit status 137)
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) secondary error attachment
		  | 1: dead (exit status 137)
		  | (1) attached stack trace
		  |   -- stack trace:
		  |   | main.glob..func14
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1171
		  |   | main.wrap.func1
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:279
		  |   | github.com/spf13/cobra.(*Command).execute
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856
		  |   | github.com/spf13/cobra.(*Command).ExecuteC
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960
		  |   | github.com/spf13/cobra.(*Command).Execute
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
		  |   | main.main
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:2094
		  |   | runtime.main
		  |   | 	/usr/local/go/src/runtime/proc.go:225
		  |   | runtime.goexit
		  |   | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		  | Wraps: (2) 1: dead (exit status 137)
		  | Error types: (1) *withstack.withStack (2) *errutil.leafError
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1171
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:279
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:2094
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:225
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (4) 9: dead (exit status 137)
		Error types: (1) errors.Unclassified (2) *secondary.withSecondaryError (3) *withstack.withStack (4) *errutil.leafError
Reproduce

See: roachtest README

See: CI job to stress roachtests

For the CI stress job, click the ellipsis (...) next to the Run button and fill in: * Changes / Build branch: master * Parameters / `env.TESTS`: `^clearrange/zfs/checks=false$` * Parameters / `env.COUNT`: <number of runs>

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

@bananabrick bananabrick self-assigned this Aug 10, 2021
@bananabrick bananabrick mentioned this issue Aug 11, 2021
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/zfs/checks=false failed with artifacts on master @ ee3efd6b1e24a3e1676778f5028fa0a35266f683:

The test failed on branch=master, cloud=gce:
test timed out (see artifacts for details)
Reproduce

See: roachtest README

See: CI job to stress roachtests

For the CI stress job, click the ellipsis (...) next to the Run button and fill in: * Changes / Build branch: master * Parameters / `env.TESTS`: `^clearrange/zfs/checks=false$` * Parameters / `env.COUNT`: <number of runs>

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/zfs/checks=false failed with artifacts on master @ 04a41e7915f4a89dcc1d0dbd92466c6adf79ec9f:

The test failed on branch=master, cloud=gce:
test timed out (see artifacts for details)
Reproduce

See: roachtest README

See: CI job to stress roachtests

For the CI stress job, click the ellipsis (...) next to the Run button and fill in: * Changes / Build branch: master * Parameters / `env.TESTS`: `^clearrange/zfs/checks=false$` * Parameters / `env.COUNT`: <number of runs>

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/zfs/checks=false failed with artifacts on master @ 7897f24246bef3cb94f9f4bfaed474ecaa9fdee6:

		  | (1) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*crdbInstallHelper).startNode
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:412
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.Cockroach.Start.func1
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:166
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*SyncedCluster).ParallelE.func1.1
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cluster_synced.go:1709
		  |   | runtime.goexit
		  |   | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		  | Wraps: (2) ~ ./cockroach.sh
		  |   | �[0;1;31mJob for cockroach.service failed because the control process exited with error code.�[0m
		  |   | �[0;1;31mSee "systemctl status cockroach.service" and "journalctl -xe" for details.�[0m
		  | Wraps: (3) exit status 1
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *exec.ExitError: 
		  | 8: ~ ./cockroach.sh: exit status 1
		  | (1) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*crdbInstallHelper).startNode
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:412
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.Cockroach.Start.func1
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:166
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*SyncedCluster).ParallelE.func1.1
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cluster_synced.go:1709
		  |   | runtime.goexit
		  |   | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		  | Wraps: (2) ~ ./cockroach.sh
		  |   | �[0;1;31mJob for cockroach.service failed because the control process exited with error code.�[0m
		  |   | �[0;1;31mSee "systemctl status cockroach.service" and "journalctl -xe" for details.�[0m
		  | Wraps: (3) exit status 1
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *exec.ExitError: 
		  | 9: ~ ./cockroach.sh: exit status 1
		  | (1) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*crdbInstallHelper).startNode
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:412
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.Cockroach.Start.func1
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:166
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*SyncedCluster).ParallelE.func1.1
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cluster_synced.go:1709
		  |   | runtime.goexit
		  |   | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		  | Wraps: (2) ~ ./cockroach.sh
		  |   | �[0;1;31mJob for cockroach.service failed because the control process exited with error code.�[0m
		  |   | �[0;1;31mSee "systemctl status cockroach.service" and "journalctl -xe" for details.�[0m
		  | Wraps: (3) exit status 1
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *exec.ExitError: 
		  | I210820 08:18:38.240165 1 (gostd) cluster_synced.go:1677  [-] 1  command failed
		Wraps: (2) exit status 1
		Error types: (1) *cluster.WithCommandDetails (2) *exec.ExitError
Reproduce

See: roachtest README

See: CI job to stress roachtests

For the CI stress job, click the ellipsis (...) next to the Run button and fill in: * Changes / Build branch: master * Parameters / `env.TESTS`: `^clearrange/zfs/checks=false$` * Parameters / `env.COUNT`: <number of runs>

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/zfs/checks=false failed with artifacts on master @ 11e0a4da82124e70e772a009011ca7a4007bff85:

		  | (1) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*crdbInstallHelper).startNode
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:412
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.Cockroach.Start.func1
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:166
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*SyncedCluster).ParallelE.func1.1
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cluster_synced.go:1709
		  |   | runtime.goexit
		  |   | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		  | Wraps: (2) ~ ./cockroach.sh
		  |   | �[0;1;31mJob for cockroach.service failed because the control process exited with error code.�[0m
		  |   | �[0;1;31mSee "systemctl status cockroach.service" and "journalctl -xe" for details.�[0m
		  | Wraps: (3) exit status 1
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *exec.ExitError: 
		  | 8: ~ ./cockroach.sh: exit status 1
		  | (1) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*crdbInstallHelper).startNode
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:412
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.Cockroach.Start.func1
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:166
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*SyncedCluster).ParallelE.func1.1
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cluster_synced.go:1709
		  |   | runtime.goexit
		  |   | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		  | Wraps: (2) ~ ./cockroach.sh
		  |   | �[0;1;31mJob for cockroach.service failed because the control process exited with error code.�[0m
		  |   | �[0;1;31mSee "systemctl status cockroach.service" and "journalctl -xe" for details.�[0m
		  | Wraps: (3) exit status 1
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *exec.ExitError: 
		  | 9: ~ ./cockroach.sh: exit status 1
		  | (1) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*crdbInstallHelper).startNode
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:412
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.Cockroach.Start.func1
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:166
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*SyncedCluster).ParallelE.func1.1
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cluster_synced.go:1709
		  |   | runtime.goexit
		  |   | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		  | Wraps: (2) ~ ./cockroach.sh
		  |   | �[0;1;31mJob for cockroach.service failed because the control process exited with error code.�[0m
		  |   | �[0;1;31mSee "systemctl status cockroach.service" and "journalctl -xe" for details.�[0m
		  | Wraps: (3) exit status 1
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *exec.ExitError: 
		  | I210821 08:02:42.148843 1 (gostd) cluster_synced.go:1677  [-] 1  command failed
		Wraps: (2) exit status 1
		Error types: (1) *cluster.WithCommandDetails (2) *exec.ExitError
Reproduce

See: roachtest README

See: CI job to stress roachtests

For the CI stress job, click the ellipsis (...) next to the Run button and fill in: * Changes / Build branch: master * Parameters / `env.TESTS`: `^clearrange/zfs/checks=false$` * Parameters / `env.COUNT`: <number of runs>

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/zfs/checks=false failed with artifacts on master @ d18da6c092bf1522e7a6478fe3973817e318c247:

		  | (1) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*crdbInstallHelper).startNode
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:412
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.Cockroach.Start.func1
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:166
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*SyncedCluster).ParallelE.func1.1
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cluster_synced.go:1709
		  |   | runtime.goexit
		  |   | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		  | Wraps: (2) ~ ./cockroach.sh
		  |   | �[0;1;31mJob for cockroach.service failed because the control process exited with error code.�[0m
		  |   | �[0;1;31mSee "systemctl status cockroach.service" and "journalctl -xe" for details.�[0m
		  | Wraps: (3) exit status 1
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *exec.ExitError: 
		  | 8: ~ ./cockroach.sh: exit status 1
		  | (1) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*crdbInstallHelper).startNode
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:412
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.Cockroach.Start.func1
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:166
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*SyncedCluster).ParallelE.func1.1
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cluster_synced.go:1709
		  |   | runtime.goexit
		  |   | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		  | Wraps: (2) ~ ./cockroach.sh
		  |   | �[0;1;31mJob for cockroach.service failed because the control process exited with error code.�[0m
		  |   | �[0;1;31mSee "systemctl status cockroach.service" and "journalctl -xe" for details.�[0m
		  | Wraps: (3) exit status 1
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *exec.ExitError: 
		  | 9: ~ ./cockroach.sh: exit status 1
		  | (1) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*crdbInstallHelper).startNode
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:412
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.Cockroach.Start.func1
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:166
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*SyncedCluster).ParallelE.func1.1
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cluster_synced.go:1709
		  |   | runtime.goexit
		  |   | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		  | Wraps: (2) ~ ./cockroach.sh
		  |   | �[0;1;31mJob for cockroach.service failed because the control process exited with error code.�[0m
		  |   | �[0;1;31mSee "systemctl status cockroach.service" and "journalctl -xe" for details.�[0m
		  | Wraps: (3) exit status 1
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *exec.ExitError: 
		  | I210822 08:32:05.171158 1 (gostd) cluster_synced.go:1677  [-] 1  command failed
		Wraps: (2) exit status 1
		Error types: (1) *cluster.WithCommandDetails (2) *exec.ExitError
Reproduce

See: roachtest README

See: CI job to stress roachtests

For the CI stress job, click the ellipsis (...) next to the Run button and fill in: * Changes / Build branch: master * Parameters / `env.TESTS`: `^clearrange/zfs/checks=false$` * Parameters / `env.COUNT`: <number of runs>

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/zfs/checks=false failed with artifacts on master @ 61bd543ba7288c8f0eed6cddded7b219c9d1fcd4:

		  | (1) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*crdbInstallHelper).startNode
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:412
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.Cockroach.Start.func1
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:166
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*SyncedCluster).ParallelE.func1.1
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cluster_synced.go:1709
		  |   | runtime.goexit
		  |   | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		  | Wraps: (2) ~ ./cockroach.sh
		  |   | �[0;1;31mJob for cockroach.service failed because the control process exited with error code.�[0m
		  |   | �[0;1;31mSee "systemctl status cockroach.service" and "journalctl -xe" for details.�[0m
		  | Wraps: (3) exit status 1
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *exec.ExitError: 
		  | 8: ~ ./cockroach.sh: exit status 1
		  | (1) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*crdbInstallHelper).startNode
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:412
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.Cockroach.Start.func1
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:166
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*SyncedCluster).ParallelE.func1.1
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cluster_synced.go:1709
		  |   | runtime.goexit
		  |   | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		  | Wraps: (2) ~ ./cockroach.sh
		  |   | �[0;1;31mJob for cockroach.service failed because the control process exited with error code.�[0m
		  |   | �[0;1;31mSee "systemctl status cockroach.service" and "journalctl -xe" for details.�[0m
		  | Wraps: (3) exit status 1
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *exec.ExitError: 
		  | 9: ~ ./cockroach.sh: exit status 1
		  | (1) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*crdbInstallHelper).startNode
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:412
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.Cockroach.Start.func1
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:166
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*SyncedCluster).ParallelE.func1.1
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cluster_synced.go:1709
		  |   | runtime.goexit
		  |   | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		  | Wraps: (2) ~ ./cockroach.sh
		  |   | �[0;1;31mJob for cockroach.service failed because the control process exited with error code.�[0m
		  |   | �[0;1;31mSee "systemctl status cockroach.service" and "journalctl -xe" for details.�[0m
		  | Wraps: (3) exit status 1
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *exec.ExitError: 
		  | I210823 08:13:52.974657 1 (gostd) cluster_synced.go:1677  [-] 1  command failed
		Wraps: (2) exit status 1
		Error types: (1) *cluster.WithCommandDetails (2) *exec.ExitError
Reproduce

See: roachtest README

See: CI job to stress roachtests

For the CI stress job, click the ellipsis (...) next to the Run button and fill in: * Changes / Build branch: master * Parameters / `env.TESTS`: `^clearrange/zfs/checks=false$` * Parameters / `env.COUNT`: <number of runs>

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/zfs/checks=false failed with artifacts on master @ 42e5f9492d0d8d93638241303bca984fe78baae3:

		10: 13523
		7: 13544
		6: 13446
		5: 14035
		3: 13822
		2: 13444
		Error: UNCLASSIFIED_PROBLEM: 4: dead (exit status 137)
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) secondary error attachment
		  | 1: dead (exit status 137)
		  | (1) attached stack trace
		  |   -- stack trace:
		  |   | main.glob..func14
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1173
		  |   | main.wrap.func1
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:281
		  |   | github.com/spf13/cobra.(*Command).execute
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856
		  |   | github.com/spf13/cobra.(*Command).ExecuteC
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960
		  |   | github.com/spf13/cobra.(*Command).Execute
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
		  |   | main.main
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:2107
		  |   | runtime.main
		  |   | 	/usr/local/go/src/runtime/proc.go:225
		  |   | runtime.goexit
		  |   | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		  | Wraps: (2) 1: dead (exit status 137)
		  | Error types: (1) *withstack.withStack (2) *errutil.leafError
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1173
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:281
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:2107
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:225
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (4) 4: dead (exit status 137)
		Error types: (1) errors.Unclassified (2) *secondary.withSecondaryError (3) *withstack.withStack (4) *errutil.leafError
Reproduce

See: roachtest README

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

@jbowens jbowens removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Sep 7, 2021
@cockroach-teamcity cockroach-teamcity mentioned this issue Sep 11, 2021
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/zfs/checks=false failed with artifacts on master @ 6eb9df1f3d460f904b5254c3d32c745eecbedd3f:

		Wraps: (2) output in run_081106.676010339_n1_cockroach_workload_fixtures_import_bank
		Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-3429354-1631428311-31-n10cpu16:1 -- ./cockroach workload fixtures import bank --payload-bytes=10240 --ranges=10 --rows=65104166 --seed=4 --db=bigbank returned
		  | stderr:
		  | I210912 08:11:07.773821 1 ccl/workloadccl/fixture.go:345  [-] 1  starting import of 1 tables
		  | Error: importing fixture: importing table bank: dial tcp 127.0.0.1:26257: connect: connection refused
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ``````
		  |   | ./cockroach workload fixtures import bank --payload-bytes=10240 --ranges=10 --rows=65104166 --seed=4 --db=bigbank
		  |   | ``````
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError

	cluster.go:1249,context.go:89,cluster.go:1237,test_runner.go:866: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-3429354-1631428311-31-n10cpu16 --oneshot --ignore-empty-nodes: exit status 1 1: dead (exit status 137)
		6: 13621
		5: 13904
		10: 13532
		3: 13222
		2: 13318
		7: 13360
		4: 13555
		8: 13602
		9: 13220
		Error: UNCLASSIFIED_PROBLEM: 1: dead (exit status 137)
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1173
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:281
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:2107
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:225
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (3) 1: dead (exit status 137)
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/zfs/checks=false failed with artifacts on master @ d49fadb6f1c67d99ce91b719bac44b5640fa8e01:

The test failed on branch=master, cloud=gce:
test timed out (see artifacts for details)
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/zfs/checks=false failed with artifacts on master @ 676dc076dc6a137d88eac9217764bbb63beae86f:

The test failed on branch=master, cloud=gce:
test timed out (see artifacts for details)
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/zfs/checks=false failed with artifacts on master @ a465b0d5d51a6534d39e0683fd037d132bcbd576:

	cluster.go:1944,clearrange.go:139,monitor.go:106,errgroup.go:57: output in run_092348.093109847_n1-10_cockroach_workload_run_kv: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-3485082-1632378740-33-n10cpu16:1-10 -- ./cockroach workload run kv --concurrency=32 --duration=1h returned: context canceled
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2034
		  | main.(*clusterImpl).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1942
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runClearRange.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:139
		  | main.(*monitorImpl).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:106
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (2) output in run_092348.093109847_n1-10_cockroach_workload_run_kv
		Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-3485082-1632378740-33-n10cpu16:1-10 -- ./cockroach workload run kv --concurrency=32 --duration=1h returned
		  | stderr:
		  |
		  | stdout:
		  | teamcity-3485082-1632378740-33-n10cpu16: ./cockroach workload run kv.................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
		Wraps: (4) secondary error attachment
		  | signal: killed
		  | (1) signal: killed
		  | Error types: (1) *exec.ExitError
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString

	monitor.go:128,clearrange.go:207,clearrange.go:53,test_runner.go:777: monitor failure: monitor task failed: pq: query execution canceled due to statement timeout
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
		  | main.(*monitorImpl).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runClearRange
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:207
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClearRange.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:53
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (4) monitor task failed
		Wraps: (5) pq: query execution canceled due to statement timeout
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/zfs/checks=false failed with artifacts on master @ ecaa4920192549af1418923648769ede3de1c9c8:

The test failed on branch=master, cloud=gce:
test timed out (see artifacts for details)
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/zfs/checks=false failed with artifacts on master @ 80844e44f4f9d0e565c6e810bd4c782be61899bf:

The test failed on branch=master, cloud=gce:
test timed out (see artifacts for details)
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/checks=false failed with artifacts on master @ aef9aa08e12d50ca6401440cac5b57d5a5d6b720:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/clearrange/checks=false/run_1
	cluster.go:1944,clearrange.go:69,clearrange.go:38,test_runner.go:777: output in run_074524.238654916_n1_cockroach_workload_fixtures_import_bank: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-3544338-1633501097-32-n10cpu16:1 -- ./cockroach workload fixtures import bank --payload-bytes=10240 --ranges=10 --rows=65104166 --seed=4 --db=bigbank returned: exit status 20
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2034
		  | main.(*clusterImpl).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1942
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runClearRange
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:69
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClearRange.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:38
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (2) output in run_074524.238654916_n1_cockroach_workload_fixtures_import_bank
		Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-3544338-1633501097-32-n10cpu16:1 -- ./cockroach workload fixtures import bank --payload-bytes=10240 --ranges=10 --rows=65104166 --seed=4 --db=bigbank returned
		  | stderr:
		  | I211006 07:45:25.289224 1 ccl/workloadccl/fixture.go:345  [-] 1  starting import of 1 tables
		  | I211006 08:28:43.582435 56 ccl/workloadccl/fixture.go:502  [-] 2  imported 622 GiB in bank table (65104166 rows, 0 index entries, took 43m18.280073349s, 245.12 MiB/s)
		  | I211006 08:28:43.588206 1 ccl/workloadccl/fixture.go:354  [-] 3  imported 622 GiB bytes in 1 tables (took 43m18.297535913s, 245.12 MiB/s)
		  | I211006 08:28:43.617040 1 workload/workloadsql/workloadsql.go:120  [-] 4  starting 9 splits
		  | W211006 08:30:27.322258 16 workload/workloadsql/workloadsql.go:157  [-] 5  ALTER TABLE bank SCATTER FROM (13020832) TO (13020832): read tcp 127.0.0.1:47612 -> 127.0.0.1:26257: read: connection reset by peer
		  | W211006 08:30:27.322273 118 workload/workloadsql/workloadsql.go:157  [-] 6  ALTER TABLE bank SCATTER FROM (26041664) TO (26041664): read tcp 127.0.0.1:47610 -> 127.0.0.1:26257: read: connection reset by peer
		  | Error: executing ALTER TABLE bank SPLIT AT VALUES (6510416): dial tcp 127.0.0.1:26257: connect: connection refused
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ``````
		  |   | ./cockroach workload fixtures import bank --payload-bytes=10240 --ranges=10 --rows=65104166 --seed=4 --db=bigbank
		  |   | ``````
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/zfs/checks=false failed with artifacts on master @ cc6296c24ddb048215dabe5cc41339f306db4f41:

The test failed on branch=master, cloud=gce:
test timed out (see artifacts for details)
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/zfs/checks=false failed with artifacts on master @ 7e4ba61845bb47cc2d7146d1bbd70fd53eff5457:

The test failed on branch=master, cloud=gce:
test timed out (see artifacts for details)
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/zfs/checks=false failed with artifacts on master @ 965b99b1a05eb893ee908d852680824a0bd4ffe3:

The test failed on branch=master, cloud=gce:
test timed out (see artifacts for details)
Help

See: roachtest README

|

See: How To Investigate (internal)

Same failure on other branches

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/checks=false failed with artifacts on master @ 40f11fead0a0453969634f8ddb0502c1f78b2806:

The test failed on branch=master, cloud=gce:
test timed out (see artifacts for details)
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/checks=false failed with artifacts on master @ b450fea83a7db1e06403b2563c13f38c9284b932:

The test failed on branch=master, cloud=gce:
test timed out (see artifacts for details)
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/checks=false failed with artifacts on master @ 3b30a0e12f9a14b08ee8ad55b50299aca50c67a2:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/clearrange/checks=false/run_1
	cluster.go:1966,clearrange.go:70,clearrange.go:38,test_runner.go:779: output in run_083818.550177647_n1_cockroach_workload_fixtures_import_bank: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-3807326-1638083317-31-n10cpu16:1 -- ./cockroach workload fixtures import bank --payload-bytes=10240 --ranges=10 --rows=65104166 --seed=4 --db=bigbank returned: exit status 20
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2054
		  | main.(*clusterImpl).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1964
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runClearRange
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:70
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClearRange.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:38
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:779
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_083818.550177647_n1_cockroach_workload_fixtures_import_bank
		Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-3807326-1638083317-31-n10cpu16:1 -- ./cockroach workload fixtures import bank --payload-bytes=10240 --ranges=10 --rows=65104166 --seed=4 --db=bigbank returned
		  | stderr:
		  | I211128 08:38:19.809469 1 ccl/workloadccl/fixture.go:345  [-] 1  starting import of 1 tables
		  | Error: importing fixture: importing table bank: dial tcp 127.0.0.1:26257: connect: connection refused
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ``````
		  |   | ./cockroach workload fixtures import bank --payload-bytes=10240 --ranges=10 --rows=65104166 --seed=4 --db=bigbank
		  |   | ``````
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError

	cluster.go:1343,context.go:91,cluster.go:1333,test_runner.go:867: dead node detection: 1: dead (exit status 1)
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/checks=false failed with artifacts on master @ 2c014c47c1a242f504f6d595bfd79c0edc20b90a:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/clearrange/checks=false/run_1
	cluster.go:1966,clearrange.go:70,clearrange.go:38,test_runner.go:779: output in run_083827.744997725_n1_cockroach_workload_fixtures_import_bank: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-3810278-1638169717-31-n10cpu16:1 -- ./cockroach workload fixtures import bank --payload-bytes=10240 --ranges=10 --rows=65104166 --seed=4 --db=bigbank returned: exit status 20
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2054
		  | main.(*clusterImpl).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1964
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runClearRange
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:70
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClearRange.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:38
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:779
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_083827.744997725_n1_cockroach_workload_fixtures_import_bank
		Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-3810278-1638169717-31-n10cpu16:1 -- ./cockroach workload fixtures import bank --payload-bytes=10240 --ranges=10 --rows=65104166 --seed=4 --db=bigbank returned
		  | stderr:
		  | I211129 08:38:28.906970 1 ccl/workloadccl/fixture.go:345  [-] 1  starting import of 1 tables
		  | Error: importing fixture: importing table bank: dial tcp 127.0.0.1:26257: connect: connection refused
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ``````
		  |   | ./cockroach workload fixtures import bank --payload-bytes=10240 --ranges=10 --rows=65104166 --seed=4 --db=bigbank
		  |   | ``````
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError

	cluster.go:1343,context.go:91,cluster.go:1333,test_runner.go:867: dead node detection: 1: dead (exit status 1)
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/checks=false failed with artifacts on master @ dadff0d6e20211f260facf0755518e85f50b124d:

	cluster.go:1962,clearrange.go:140,monitor.go:106,errgroup.go:57: output in run_095643.909874438_n1-10_cockroach_workload_run_kv: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-3867360-1638947909-30-n10cpu16:1-10 -- ./cockroach workload run kv --concurrency=32 --duration=1h returned: context canceled
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2050
		  | main.(*clusterImpl).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1960
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runClearRange.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:140
		  | main.(*monitorImpl).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:106
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_095643.909874438_n1-10_cockroach_workload_run_kv
		Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-3867360-1638947909-30-n10cpu16:1-10 -- ./cockroach workload run kv --concurrency=32 --duration=1h returned
		  | stderr:
		  |
		  | stdout:
		  | teamcity-3867360-1638947909-30-n10cpu16: ./cockroach workload run kv......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
		Wraps: (4) secondary error attachment
		  | signal: killed
		  | (1) signal: killed
		  | Error types: (1) *exec.ExitError
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString

	monitor.go:128,clearrange.go:208,clearrange.go:38,test_runner.go:779: monitor failure: monitor task failed: pq: query execution canceled due to statement timeout
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
		  | main.(*monitorImpl).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runClearRange
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:208
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClearRange.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:38
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (4) monitor task failed
		Wraps: (5) pq: query execution canceled due to statement timeout
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/checks=false failed with artifacts on master @ 78419450178335b31f542bd1b14fefdf4ecee0e8:

		  | I220115 10:44:04.866089 319 workload/pgx_helpers.go:72  [-] 12  pgx logger [error]: Exec logParams=map[args:[5638997613164904430 9f] err:unexpected EOF sql:kv-2]
		  | I220115 10:44:04.866204 97 workload/pgx_helpers.go:72  [-] 13  pgx logger [error]: Exec logParams=map[args:[-3503708016888177558 23] err:unexpected EOF sql:kv-2]
		  | I220115 10:44:04.866166 333 workload/pgx_helpers.go:72  [-] 14  pgx logger [error]: Exec logParams=map[args:[-3265339289772281963 cc] err:unexpected EOF sql:kv-2]
		  | Error: unexpected EOF
		  | I220115 10:44:04.866082 323 workload/pgx_helpers.go:72  [-] 15  pgx logger [error]: Exec logParams=map[args:[-2972596694405450509 86] err:unexpected EOF sql:kv-2]
		  | COMMAND_PROBLEM: exit status 1
		  |    6: 
		  | UNCLASSIFIED_PROBLEM: context canceled
		  |    7: 
		  | UNCLASSIFIED_PROBLEM: context canceled
		  |    8: 
		  | UNCLASSIFIED_PROBLEM: context canceled
		  |    9: 
		  | UNCLASSIFIED_PROBLEM: context canceled
		  |   10: 
		  | UNCLASSIFIED_PROBLEM: context canceled
		Wraps: (4) secondary error attachment
		  | COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 5. Command with error:
		  |   | ``````
		  |   | ./cockroach workload run kv --concurrency=32 --duration=1h
		  |   | ``````
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString

	monitor.go:127,clearrange.go:207,clearrange.go:39,test_runner.go:780: monitor failure: monitor command failure: unexpected node event: 5: dead (exit status 10)
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runClearRange
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:207
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClearRange.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:39
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func3
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:202
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (4) monitor command failure
		Wraps: (5) unexpected node event: 5: dead (exit status 10)
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *errors.errorString
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/checks=false failed with artifacts on master @ 5ad21e3896ee809e9c3ebc28bb22166f1275acca:

		  | I220116 10:35:53.958833 247 workload/pgx_helpers.go:72  [-] 27  pgx logger [error]: Exec logParams=map[args:[-6265789633912520877 a3] err:unexpected EOF sql:kv-2]
		  | I220116 10:35:53.958853 333 workload/pgx_helpers.go:72  [-] 28  pgx logger [error]: Exec logParams=map[args:[-7876477857276334369 79] err:unexpected EOF sql:kv-2]
		  | I220116 10:35:53.958884 336 workload/pgx_helpers.go:72  [-] 29  pgx logger [error]: Exec logParams=map[args:[3949141417247107132 d6] err:unexpected EOF sql:kv-2]
		  | I220116 10:35:53.958905 323 workload/pgx_helpers.go:72  [-] 30  pgx logger [error]: Exec logParams=map[args:[-4069442148262535618 13] err:unexpected EOF sql:kv-2]
		  | I220116 10:35:53.958925 322 workload/pgx_helpers.go:72  [-] 31  pgx logger [error]: Exec logParams=map[args:[415744693094110907 1b] err:unexpected EOF sql:kv-2]
		  | I220116 10:35:53.958944 325 workload/pgx_helpers.go:72  [-] 32  pgx logger [error]: Exec logParams=map[args:[8427313232314314118 0d] err:unexpected EOF sql:kv-2]
		  | I220116 10:35:53.958972 335 workload/pgx_helpers.go:72  [-] 33  pgx logger [error]: Exec logParams=map[args:[-4464207775372267026 8c] err:unexpected EOF sql:kv-2]
		  | I220116 10:35:53.958991 250 workload/pgx_helpers.go:72  [-] 34  pgx logger [error]: Exec logParams=map[args:[-4043224230521627396 a2] err:unexpected EOF sql:kv-2]
		  | Error: unexpected EOF
		  | COMMAND_PROBLEM: exit status 1
		  |    8: 
		  | UNCLASSIFIED_PROBLEM: context canceled
		  |    9: 
		  | UNCLASSIFIED_PROBLEM: context canceled
		  |   10: 
		  | UNCLASSIFIED_PROBLEM: context canceled
		Wraps: (4) secondary error attachment
		  | COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 7. Command with error:
		  |   | ``````
		  |   | ./cockroach workload run kv --concurrency=32 --duration=1h
		  |   | ``````
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString

	monitor.go:127,clearrange.go:207,clearrange.go:39,test_runner.go:780: monitor failure: monitor command failure: unexpected node event: 7: dead (exit status 10)
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runClearRange
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:207
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClearRange.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:39
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func3
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:202
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (4) monitor command failure
		Wraps: (5) unexpected node event: 7: dead (exit status 10)
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *errors.errorString
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/checks=false failed with artifacts on master @ 4b41789120e019ab015e6dbb924df763897ebadb:

		  |  2043.0s        0            0.0         1060.1      0.0      0.0      0.0      0.0 write
		  |  2044.0s        0            0.0         1059.6      0.0      0.0      0.0      0.0 write
		  |  2045.0s        0            0.0         1059.1      0.0      0.0      0.0      0.0 write
		  | I220117 09:52:47.238784 333 workload/pgx_helpers.go:72  [-] 3  pgx logger [error]: Exec logParams=map[args:[2964951993166812657 69] err:ERROR: result is ambiguous (error=rpc error: code = Unavailable desc = error reading from server: EOF [exhausted]) (SQLSTATE 40003) sql:kv-2]
		  | I220117 09:52:47.238786 335 workload/pgx_helpers.go:72  [-] 6  pgx logger [error]: Exec logParams=map[args:[3008164110191694223 5c] err:ERROR: result is ambiguous (error=rpc error: code = Unavailable desc = error reading from server: EOF [exhausted]) (SQLSTATE 40003) sql:kv-2]
		  | I220117 09:52:47.238783 81 workload/pgx_helpers.go:72  [-] 8  pgx logger [error]: Exec logParams=map[args:[3201363600701051851 16] err:ERROR: result is ambiguous (error=rpc error: code = Unavailable desc = error reading from server: EOF [exhausted]) (SQLSTATE 40003) sql:kv-2]
		  | I220117 09:52:47.238785 322 workload/pgx_helpers.go:72  [-] 7  pgx logger [error]: Exec logParams=map[args:[3444184268271550182 c3] err:ERROR: result is ambiguous (error=rpc error: code = Unavailable desc = error reading from server: EOF [exhausted]) (SQLSTATE 40003) sql:kv-2]
		  | I220117 09:52:47.238783 76 workload/pgx_helpers.go:72  [-] 4  pgx logger [error]: Exec logParams=map[args:[4164019100877029105 f1] err:ERROR: result is ambiguous (error=rpc error: code = Unavailable desc = error reading from server: EOF [exhausted]) (SQLSTATE 40003) sql:kv-2]
		  | I220117 09:52:47.238783 341 workload/pgx_helpers.go:72  [-] 5  pgx logger [error]: Exec logParams=map[args:[3402140211336010515 7e] err:ERROR: result is ambiguous (error=rpc error: code = Unavailable desc = error reading from server: EOF [exhausted]) (SQLSTATE 40003) sql:kv-2]
		  | Error: ERROR: result is ambiguous (error=rpc error: code = Unavailable desc = error reading from server: EOF [exhausted]) (SQLSTATE 40003)
		  | COMMAND_PROBLEM: exit status 1
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./cockroach workload run kv --concurrency=32 --duration=1h
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

	monitor.go:127,clearrange.go:207,clearrange.go:39,test_runner.go:780: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runClearRange
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:207
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClearRange.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:39
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:780
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:171
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:80
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:6498
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:238
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/checks=false failed with artifacts on master @ 365b4da8bd02c06ee59d2130a56dec74ffc9ce21:

		  | I220118 10:16:20.261715 49 workload/pgx_helpers.go:72  [-] 17  pgx logger [error]: Exec logParams=map[args:[-531632558795619032 ac] err:unexpected EOF sql:kv-2]
		  | I220118 10:16:20.257872 45 workload/pgx_helpers.go:72  [-] 18  pgx logger [error]: Exec logParams=map[args:[4639513351833619430 81] err:unexpected EOF sql:kv-2]
		  | I220118 10:16:20.257828 359 workload/pgx_helpers.go:72  [-] 19  pgx logger [error]: Exec logParams=map[args:[-6647534625571820937 a0] err:unexpected EOF sql:kv-2]
		  | I220118 10:16:20.261741 352 workload/pgx_helpers.go:72  [-] 20  pgx logger [error]: Exec logParams=map[args:[-7831809228977036274 2c] err:unexpected EOF sql:kv-2]
		  | I220118 10:16:20.261766 355 workload/pgx_helpers.go:72  [-] 21  pgx logger [error]: Exec logParams=map[args:[2859166911093030182 c2] err:unexpected EOF sql:kv-2]
		  | I220118 10:16:20.261783 361 workload/pgx_helpers.go:72  [-] 22  pgx logger [error]: Exec logParams=map[args:[-6029439150509853361 8f] err:unexpected EOF sql:kv-2]
		  | I220118 10:16:20.261800 358 workload/pgx_helpers.go:72  [-] 23  pgx logger [error]: Exec logParams=map[args:[-296956208564693601 62] err:unexpected EOF sql:kv-2]
		  | I220118 10:16:20.261818 343 workload/pgx_helpers.go:72  [-] 24  pgx logger [error]: Exec logParams=map[args:[529200908579190623 f6] err:unexpected EOF sql:kv-2]
		  | I220118 10:16:20.261837 345 workload/pgx_helpers.go:72  [-] 25  pgx logger [error]: Exec logParams=map[args:[-226136554201026465 42] err:unexpected EOF sql:kv-2]
		  | I220118 10:16:20.261854 356 workload/pgx_helpers.go:72  [-] 26  pgx logger [error]: Exec logParams=map[args:[-5470521325140306528 ea] err:unexpected EOF sql:kv-2]
		  | I220118 10:16:20.261872 43 workload/pgx_helpers.go:72  [-] 27  pgx logger [error]: Exec logParams=map[args:[-6532190395182479007 e4] err:unexpected EOF sql:kv-2]
		  | I220118 10:16:20.261890 344 workload/pgx_helpers.go:72  [-] 28  pgx logger [error]: Exec logParams=map[args:[-6803246642505559059 e8] err:unexpected EOF sql:kv-2]
		  | I220118 10:16:20.261909 354 workload/pgx_helpers.go:72  [-] 29  pgx logger [error]: Exec logParams=map[args:[4352811922722313548 36] err:unexpected EOF sql:kv-2]
		  | Error: unexpected EOF
		  | I220118 10:16:20.261925 342 workload/pgx_helpers.go:72  [-] 30  pgx logger [error]: Exec logParams=map[args:[216439865641540565 f1] err:unexpected EOF sql:kv-2]
		  | COMMAND_PROBLEM: exit status 1
		Wraps: (4) secondary error attachment
		  | COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 10. Command with error:
		  |   | ``````
		  |   | ./cockroach workload run kv --concurrency=32 --duration=1h
		  |   | ``````
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString

	monitor.go:127,clearrange.go:207,clearrange.go:39,test_runner.go:780: monitor failure: monitor command failure: unexpected node event: 10: dead (exit status 10)
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runClearRange
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:207
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClearRange.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:39
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func3
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:202
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (4) monitor command failure
		Wraps: (5) unexpected node event: 10: dead (exit status 10)
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *errors.errorString
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Jan 19, 2022
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/checks=false failed with artifacts on master @ da01e4c0545f191a0573e1d097ff0366769e0d6b:

		  | Error: unexpected EOF
		  | COMMAND_PROBLEM: exit status 1
		  |    4: 
		  | UNCLASSIFIED_PROBLEM: context canceled
		  |    5: 
		  | UNCLASSIFIED_PROBLEM: context canceled
		  |    6: 
		  | UNCLASSIFIED_PROBLEM: context canceled
		  |    7: 
		  | UNCLASSIFIED_PROBLEM: context canceled
		  |    8: 
		  | UNCLASSIFIED_PROBLEM: context canceled
		  |    9: 
		  | UNCLASSIFIED_PROBLEM: context canceled
		  |   10: 
		  | UNCLASSIFIED_PROBLEM: context canceled
		Wraps: (4) secondary error attachment
		  | COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 3. Command with error:
		  |   | ``````
		  |   | ./cockroach workload run kv --concurrency=32 --duration=1h
		  |   | ``````
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString

	monitor.go:127,clearrange.go:207,clearrange.go:39,test_runner.go:780: monitor failure: monitor command failure: unexpected node event: 3: dead (exit status 10)
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runClearRange
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:207
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClearRange.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:39
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func3
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:202
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (4) monitor command failure
		Wraps: (5) unexpected node event: 3: dead (exit status 10)
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *errors.errorString
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/checks=false failed with artifacts on master @ 58ceac139a7e83052171121b28026a7366f16f7e:

		  | I220121 10:34:15.191434 218 workload/pgx_helpers.go:72  [-] 21  pgx logger [error]: Exec logParams=map[args:[-6508079655690867455 46] err:unexpected EOF sql:kv-2]
		  | I220121 10:34:15.191455 363 workload/pgx_helpers.go:72  [-] 22  pgx logger [error]: Exec logParams=map[args:[1581112652989666144 b4] err:unexpected EOF sql:kv-2]
		  | I220121 10:34:15.191474 220 workload/pgx_helpers.go:72  [-] 23  pgx logger [error]: Exec logParams=map[args:[-948097495605556355 83] err:unexpected EOF sql:kv-2]
		  | I220121 10:34:15.191494 375 workload/pgx_helpers.go:72  [-] 24  pgx logger [error]: Exec logParams=map[args:[1697394334348767092 ae] err:unexpected EOF sql:kv-2]
		  | I220121 10:34:15.191524 367 workload/pgx_helpers.go:72  [-] 25  pgx logger [error]: Exec logParams=map[args:[1828462869120443484 6a] err:unexpected EOF sql:kv-2]
		  | I220121 10:34:15.191544 370 workload/pgx_helpers.go:72  [-] 26  pgx logger [error]: Exec logParams=map[args:[-6449735435425593323 85] err:unexpected EOF sql:kv-2]
		  | I220121 10:34:15.191565 362 workload/pgx_helpers.go:72  [-] 27  pgx logger [error]: Exec logParams=map[args:[8562420703358228377 5d] err:unexpected EOF sql:kv-2]
		  | I220121 10:34:15.191583 369 workload/pgx_helpers.go:72  [-] 28  pgx logger [error]: Exec logParams=map[args:[-6348158259531913573 3e] err:unexpected EOF sql:kv-2]
		  | I220121 10:34:15.191612 361 workload/pgx_helpers.go:72  [-] 29  pgx logger [error]: Exec logParams=map[args:[-4648337083416143401 93] err:unexpected EOF sql:kv-2]
		  | I220121 10:34:15.191630 358 workload/pgx_helpers.go:72  [-] 30  pgx logger [error]: Exec logParams=map[args:[1712553258208951372 3f] err:unexpected EOF sql:kv-2]
		  | I220121 10:34:15.191650 222 workload/pgx_helpers.go:72  [-] 31  pgx logger [error]: Exec logParams=map[args:[-1607890077470497407 e5] err:unexpected EOF sql:kv-2]
		  | I220121 10:34:15.191668 219 workload/pgx_helpers.go:72  [-] 32  pgx logger [error]: Exec logParams=map[args:[-6368305136099591442 d0] err:unexpected EOF sql:kv-2]
		  | I220121 10:34:15.191697 357 workload/pgx_helpers.go:72  [-] 33  pgx logger [error]: Exec logParams=map[args:[1857822206165222767 49] err:unexpected EOF sql:kv-2]
		  | I220121 10:34:15.191714 354 workload/pgx_helpers.go:72  [-] 34  pgx logger [error]: Exec logParams=map[args:[1730943711847276015 46] err:unexpected EOF sql:kv-2]
		  | Error: unexpected EOF
		  | COMMAND_PROBLEM: exit status 1
		Wraps: (4) secondary error attachment
		  | COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 10. Command with error:
		  |   | ``````
		  |   | ./cockroach workload run kv --concurrency=32 --duration=1h
		  |   | ``````
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString

	monitor.go:127,clearrange.go:207,clearrange.go:39,test_runner.go:780: monitor failure: monitor command failure: unexpected node event: 10: dead (exit status 10)
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runClearRange
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:207
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClearRange.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:39
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func3
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:202
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (4) monitor command failure
		Wraps: (5) unexpected node event: 10: dead (exit status 10)
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *errors.errorString
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/checks=false failed with artifacts on master @ dc07599dc9db1acd5afa3a6537297815f25c1fca:

		  | I220122 10:44:59.066440 342 workload/pgx_helpers.go:79  [-] 23  pgx logger [error]: Exec logParams=map[args:[-7784643919100477327 2c] err:unexpected EOF sql:kv-2]
		  | I220122 10:44:59.071261 357 workload/pgx_helpers.go:79  [-] 24  pgx logger [error]: Exec logParams=map[args:[7526004908046120689 b9] err:unexpected EOF sql:kv-2]
		  | I220122 10:44:59.071289 344 workload/pgx_helpers.go:79  [-] 25  pgx logger [error]: Exec logParams=map[args:[-7753587312593170114 e9] err:unexpected EOF sql:kv-2]
		  | I220122 10:44:59.071308 94 workload/pgx_helpers.go:79  [-] 26  pgx logger [error]: Exec logParams=map[args:[-6903628605487882055 23] err:unexpected EOF sql:kv-2]
		  | I220122 10:44:59.071329 356 workload/pgx_helpers.go:79  [-] 27  pgx logger [error]: Exec logParams=map[args:[508541492097773595 dc] err:unexpected EOF sql:kv-2]
		  | I220122 10:44:59.071344 355 workload/pgx_helpers.go:79  [-] 28  pgx logger [error]: Exec logParams=map[args:[-6851016543514504991 ce] err:unexpected EOF sql:kv-2]
		  | I220122 10:44:59.071370 350 workload/pgx_helpers.go:79  [-] 29  pgx logger [error]: Exec logParams=map[args:[-7976976711999609109 6f] err:unexpected EOF sql:kv-2]
		  | I220122 10:44:59.071388 359 workload/pgx_helpers.go:79  [-] 30  pgx logger [error]: Exec logParams=map[args:[-8103740834491251413 a6] err:unexpected EOF sql:kv-2]
		  | I220122 10:44:59.071407 339 workload/pgx_helpers.go:79  [-] 31  pgx logger [error]: Exec logParams=map[args:[9051044643520169072 93] err:unexpected EOF sql:kv-2]
		  | I220122 10:44:59.071425 96 workload/pgx_helpers.go:79  [-] 32  pgx logger [error]: Exec logParams=map[args:[-6982137170969833586 83] err:unexpected EOF sql:kv-2]
		  | I220122 10:44:59.071455 92 workload/pgx_helpers.go:79  [-] 33  pgx logger [error]: Exec logParams=map[args:[929580363877672574 9f] err:unexpected EOF sql:kv-2]
		  | I220122 10:44:59.071490 90 workload/pgx_helpers.go:79  [-] 34  pgx logger [error]: Exec logParams=map[args:[361332709876869602 06] err:unexpected EOF sql:kv-2]
		  | Error: unexpected EOF
		  | COMMAND_PROBLEM: exit status 1
		  |   10: 
		  | UNCLASSIFIED_PROBLEM: context canceled
		Wraps: (4) secondary error attachment
		  | COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 9. Command with error:
		  |   | ``````
		  |   | ./cockroach workload run kv --concurrency=32 --duration=1h
		  |   | ``````
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString

	monitor.go:127,clearrange.go:207,clearrange.go:39,test_runner.go:780: monitor failure: monitor command failure: unexpected node event: 9: dead (exit status 10)
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runClearRange
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:207
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClearRange.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:39
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func3
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:202
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (4) monitor command failure
		Wraps: (5) unexpected node event: 9: dead (exit status 10)
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *errors.errorString
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/checks=false failed with artifacts on master @ e1068d77afbd39b162978281c9da7cbea49c1c3a:

		  |    3: 
		  | UNCLASSIFIED_PROBLEM: context canceled
		  |    4: 
		  | UNCLASSIFIED_PROBLEM: context canceled
		  |    5: 
		  | UNCLASSIFIED_PROBLEM: context canceled
		  |    6: 
		  | UNCLASSIFIED_PROBLEM: context canceled
		  |    7: 
		  | UNCLASSIFIED_PROBLEM: context canceled
		  |    8: 
		  | UNCLASSIFIED_PROBLEM: context canceled
		  |    9: 
		  | UNCLASSIFIED_PROBLEM: context canceled
		  |   10: 
		  | UNCLASSIFIED_PROBLEM: context canceled
		Wraps: (4) secondary error attachment
		  | COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ``````
		  |   | ./cockroach workload run kv --concurrency=32 --duration=1h
		  |   | ``````
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString

	monitor.go:127,clearrange.go:207,clearrange.go:39,test_runner.go:780: monitor failure: monitor command failure: unexpected node event: 1: dead (exit status 10)
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runClearRange
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:207
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClearRange.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:39
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func3
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:202
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (4) monitor command failure
		Wraps: (5) unexpected node event: 1: dead (exit status 10)
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *errors.errorString
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/checks=false failed with artifacts on master @ 8cd28089c6c7333615ba3201e841839001d2f0e1:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/clearrange/checks=false/run_1
	cluster.go:1997,clearrange.go:71,clearrange.go:39,test_runner.go:780: output in run_085044.051831842_n1_cockroach_workload_fixtures_import_bank: ./cockroach workload fixtures import bank --payload-bytes=10240 --ranges=10 --rows=65104166 --seed=4 --db=bigbank returned: COMMAND_PROBLEM: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2031
		  | main.(*clusterImpl).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1995
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runClearRange
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:71
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClearRange.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:39
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:780
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_085044.051831842_n1_cockroach_workload_fixtures_import_bank
		Wraps: (3) ./cockroach workload fixtures import bank --payload-bytes=10240 --ranges=10 --rows=65104166 --seed=4 --db=bigbank returned
		  | stderr:
		  | I220124 08:50:44.791999 1 ccl/workloadccl/fixture.go:318  [-] 1  starting import of 1 tables
		  | Error: importing fixture: importing table bank: dial tcp 127.0.0.1:26257: connect: connection refused
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./cockroach workload fixtures import bank --payload-bytes=10240 --ranges=10 --rows=65104166 --seed=4 --db=bigbank
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/checks=false failed with artifacts on master @ c4c5ca2fdd5a641433a85a28d4dfd3bd4443015d:

The test failed on branch=master, cloud=gce:
test timed out (see artifacts for details)
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/checks=false failed with artifacts on master @ 83e2df701688745fe7e7d8a0d0e5e7a4ba8633c8:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/clearrange/checks=false/run_1
	cluster.go:1915,clearrange.go:86,clearrange.go:39,test_runner.go:780: one or more parallel execution failure
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).ParallelE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2035
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Parallel
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:1919
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Start
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/roachprod/install/cockroach.go:164
		  | github.com/cockroachdb/cockroach/pkg/roachprod.Start
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:659
		  | main.(*clusterImpl).StartE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1873
		  | main.(*clusterImpl).Start
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1914
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runClearRange
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:86
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClearRange.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:39
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:780
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (2) one or more parallel execution failure
		Error types: (1) *withstack.withStack (2) *errutil.leafError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/checks=false failed with artifacts on master @ 71becf337d9d2731298dc092f3ce9cf0f0eedb2c:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/clearrange/checks=false/run_1
	cluster.go:1915,clearrange.go:86,clearrange.go:39,test_runner.go:780: one or more parallel execution failure
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).ParallelE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2035
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Parallel
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:1919
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Start
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/roachprod/install/cockroach.go:164
		  | github.com/cockroachdb/cockroach/pkg/roachprod.Start
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:659
		  | main.(*clusterImpl).StartE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1873
		  | main.(*clusterImpl).Start
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1914
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runClearRange
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:86
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClearRange.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:39
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:780
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (2) one or more parallel execution failure
		Error types: (1) *withstack.withStack (2) *errutil.leafError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/checks=false failed with artifacts on master @ bb85ddc4fc0ad8761d7aba61315da22b433022e3:

The test failed on branch=master, cloud=gce:
test timed out (see artifacts for details)
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/checks=false failed with artifacts on master @ a5158c4c93e9dca858b38ef7e94321f0a0a2b5c5:

The test failed on branch=master, cloud=gce:
test timed out (see artifacts for details)
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/checks=false failed with artifacts on master @ 8548987813ff9e1b8a9878023d3abfc6911c16db:

The test failed on branch=master, cloud=gce:
test timed out (see artifacts for details)
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/checks=false failed with artifacts on master @ 95c82a7dbc6d6fecb63edbd666c9986d1ad3885b:

The test failed on branch=master, cloud=gce:
test timed out (see artifacts for details)
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/checks=false failed with artifacts on master @ 8548987813ff9e1b8a9878023d3abfc6911c16db:

The test failed on branch=master, cloud=gce:
test timed out (see artifacts for details)
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/checks=false failed with artifacts on master @ 8548987813ff9e1b8a9878023d3abfc6911c16db:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /artifacts/clearrange/checks=false/run_1
	cluster.go:1997,clearrange.go:71,clearrange.go:39,test_runner.go:780: output in run_081629.045531912_n1_cockroach_workload_fixtures_import_bank: ./cockroach workload fixtures import bank --payload-bytes=10240 --ranges=10 --rows=65104166 --seed=4 --db=bigbank returned: COMMAND_PROBLEM: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:2031
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1995
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runClearRange
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:71
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClearRange.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:39
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:780
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_081629.045531912_n1_cockroach_workload_fixtures_import_bank
		Wraps: (3) ./cockroach workload fixtures import bank --payload-bytes=10240 --ranges=10 --rows=65104166 --seed=4 --db=bigbank returned
		  | stderr:
		  | I220204 08:16:29.881182 1 ccl/workloadccl/fixture.go:318  [-] 1  starting import of 1 tables
		  | Error: importing fixture: importing table bank: dial tcp 127.0.0.1:26257: connect: connection refused
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./cockroach workload fixtures import bank --payload-bytes=10240 --ranges=10 --rows=65104166 --seed=4 --db=bigbank
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/checks=false failed with artifacts on master @ 8548987813ff9e1b8a9878023d3abfc6911c16db:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /artifacts/clearrange/checks=false/run_1
	cluster.go:1981,clearrange.go:71,clearrange.go:39,test_runner.go:779: output in run_080326.037858248_n1_cockroach_workload_fixtures_import_bank: ./cockroach workload fixtures import bank --payload-bytes=10240 --ranges=10 --rows=65104166 --seed=4 --db=bigbank returned: COMMAND_PROBLEM: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:2015
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1979
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runClearRange
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:71
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClearRange.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:39
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:779
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_080326.037858248_n1_cockroach_workload_fixtures_import_bank
		Wraps: (3) ./cockroach workload fixtures import bank --payload-bytes=10240 --ranges=10 --rows=65104166 --seed=4 --db=bigbank returned
		  | stderr:
		  | I220205 08:03:26.804869 1 ccl/workloadccl/fixture.go:318  [-] 1  starting import of 1 tables
		  | Error: importing fixture: importing table bank: dial tcp 127.0.0.1:26257: connect: connection refused
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./cockroach workload fixtures import bank --payload-bytes=10240 --ranges=10 --rows=65104166 --seed=4 --db=bigbank
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/checks=false failed with artifacts on master @ e151a24dc1d495bf694e1e1c3ee8acb518c1424e:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /artifacts/clearrange/checks=false/run_1
	monitor.go:127,clearrange.go:207,clearrange.go:39,test_runner.go:779: monitor failure: monitor command failure: unexpected node event: 6: dead (exit status 7)
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runClearRange
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:207
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClearRange.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/clearrange.go:39
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func3
		  | 	main/pkg/cmd/roachtest/monitor.go:202
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) monitor command failure
		Wraps: (5) unexpected node event: 6: dead (exit status 7)
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *errors.errorString
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@nicktrav
Copy link
Collaborator

nicktrav commented Mar 2, 2022

I downloaded the artifacts and it looks like node 6 crashed with the panic we've been seeing elsewhere (see #77030):

F220302 12:42:59.763219 314 kv/kvserver/pkg/kv/kvserver/replica_raft.go:826 ⋮ [n6,s6,r10130/6:{-},raft] 22147  setting non-zero HardState.Commit on uninitialized replica [n6,s6,r10130/6:{-}]. HS={Term:7 Vote:0 Commit:10876}
F220302 12:42:59.763219 314 kv/kvserver/pkg/kv/kvserver/replica_raft.go:826 ⋮ [n6,s6,r10130/6:{-},raft] 22147 !goroutine 314 [running]:
F220302 12:42:59.763219 314 kv/kvserver/pkg/kv/kvserver/replica_raft.go:826 ⋮ [n6,s6,r10130/6:{-},raft] 22147 !github.com/cockroachdb/cockroach/pkg/util/log.getStacks(0x1)
F220302 12:42:59.763219 314 kv/kvserver/pkg/kv/kvserver/replica_raft.go:826 ⋮ [n6,s6,r10130/6:{-},raft] 22147 ! github.com/cockroachdb/cockroach/pkg/util/log/get_stacks.go:25 +0x8a
F220302 12:42:59.763219 314 kv/kvserver/pkg/kv/kvserver/replica_raft.go:826 ⋮ [n6,s6,r10130/6:{-},raft] 22147 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(0xc0006e6be0, {{{0xc00383b0e0, 0x24}, {0x4db18c5, 0x1}, {0x0, 0x0}, {0x0, 0x0}}, 0x16d890e5a3294be8, ...})
F220302 12:42:59.763219 314 kv/kvserver/pkg/kv/kvserver/replica_raft.go:826 ⋮ [n6,s6,r10130/6:{-},raft] 22147 ! github.com/cockroachdb/cockroach/pkg/util/log/clog.go:239 +0x97
F220302 12:42:59.763219 314 kv/kvserver/pkg/kv/kvserver/replica_raft.go:826 ⋮ [n6,s6,r10130/6:{-},raft] 22147 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepth({0x6160c78, 0xc014ea9650}, 0x1, 0x4, 0x0, {0x4e84aea, 0x45}, {0xc006a6b848, 0x2, 0x2})
F220302 12:42:59.763219 314 kv/kvserver/pkg/kv/kvserver/replica_raft.go:826 ⋮ [n6,s6,r10130/6:{-},raft] 22147 ! github.com/cockroachdb/cockroach/pkg/util/log/channels.go:60 +0x385
F220302 12:42:59.763219 314 kv/kvserver/pkg/kv/kvserver/replica_raft.go:826 ⋮ [n6,s6,r10130/6:{-},raft] 22147 !github.com/cockroachdb/cockroach/pkg/util/log.Fatalf(...)
F220302 12:42:59.763219 314 kv/kvserver/pkg/kv/kvserver/replica_raft.go:826 ⋮ [n6,s6,r10130/6:{-},raft] 22147 ! github.com/cockroachdb/cockroach/bazel-out/k8-opt/bin/pkg/util/log/log_channels_generated.go:834
F220302 12:42:59.763219 314 kv/kvserver/pkg/kv/kvserver/replica_raft.go:826 ⋮ [n6,s6,r10130/6:{-},raft] 22147 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked(0xc002876000, {0x6160c78, 0xc014ea9650}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
F220302 12:42:59.763219 314 kv/kvserver/pkg/kv/kvserver/replica_raft.go:826 ⋮ [n6,s6,r10130/6:{-},raft] 22147 ! github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:826 +0x123b
F220302 12:42:59.763219 314 kv/kvserver/pkg/kv/kvserver/replica_raft.go:826 ⋮ [n6,s6,r10130/6:{-},raft] 22147 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady(0xc006a6be70, {0x6160c78, 0xc014ea9650}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
F220302 12:42:59.763219 314 kv/kvserver/pkg/kv/kvserver/replica_raft.go:826 ⋮ [n6,s6,r10130/6:{-},raft] 22147 ! github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:556 +0x15b
F220302 12:42:59.763219 314 kv/kvserver/pkg/kv/kvserver/replica_raft.go:826 ⋮ [n6,s6,r10130/6:{-},raft] 22147 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady(0xc0001a0d00, 0x6160c78)
F220302 12:42:59.763219 314 kv/kvserver/pkg/kv/kvserver/replica_raft.go:826 ⋮ [n6,s6,r10130/6:{-},raft] 22147 ! github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:512 +0xd7
F220302 12:42:59.763219 314 kv/kvserver/pkg/kv/kvserver/replica_raft.go:826 ⋮ [n6,s6,r10130/6:{-},raft] 22147 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc00a140820, {0x6160c78, 0xc0045e7920})
F220302 12:42:59.763219 314 kv/kvserver/pkg/kv/kvserver/replica_raft.go:826 ⋮ [n6,s6,r10130/6:{-},raft] 22147 ! github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:305 +0x25d
F220302 12:42:59.763219 314 kv/kvserver/pkg/kv/kvserver/replica_raft.go:826 ⋮ [n6,s6,r10130/6:{-},raft] 22147 !github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
F220302 12:42:59.763219 314 kv/kvserver/pkg/kv/kvserver/replica_raft.go:826 ⋮ [n6,s6,r10130/6:{-},raft] 22147 ! github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:494 +0x16f
F220302 12:42:59.763219 314 kv/kvserver/pkg/kv/kvserver/replica_raft.go:826 ⋮ [n6,s6,r10130/6:{-},raft] 22147 !created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
F220302 12:42:59.763219 314 kv/kvserver/pkg/kv/kvserver/replica_raft.go:826 ⋮ [n6,s6,r10130/6:{-},raft] 22147 ! github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:483 +0x445

I also looked at the metrics, and despite the slow import (also being tracked elsewhere, see #77157), this test run looked like it was on track to complete, if not for the panic. Will close out.

@nicktrav nicktrav closed this as completed Mar 2, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bananabrick bananabrick

Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. T-storage Storage Team
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@nicktrav @jbowens @bananabrick @cockroach-teamcity