Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

roachtest: weekly/tpcc/headroom failed [infra problems] #47652

Closed
cockroach-teamcity opened this issue Apr 18, 2020 · 55 comments
Closed

roachtest: weekly/tpcc/headroom failed [infra problems] #47652

cockroach-teamcity opened this issue Apr 18, 2020 · 55 comments
Assignees
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot.
Milestone

Comments

@cockroach-teamcity
Copy link
Member

(roachtest).weekly/tpcc/headroom failed on master@a63a670b1a591be2ed38af7f98c1f85e32546596:

		  | 78224.0s        0           15.0           21.0     11.0     15.7     21.0     21.0 orderStatus
		  | 78224.0s        0          224.6          209.6     22.0     35.7     46.1     58.7 payment
		  | 78224.0s        0           21.0           21.0     88.1    234.9    260.0    260.0 stockLevel
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 78225.0s        0           10.0           21.0    104.9    209.7    209.7    209.7 delivery
		  | 78225.0s        0          196.4          209.6     46.1     96.5    113.2    121.6 newOrder
		  | 78225.0s        0           17.0           21.0     12.6     31.5     48.2     48.2 orderStatus
		  | 78225.0s        0          182.3          209.6     23.1     56.6     71.3     75.5 payment
		  | 78225.0s        0           24.0           21.0    104.9    192.9    201.3    201.3 stockLevel
		  | 78226.0s        0           22.0           21.0    113.2    159.4    167.8    167.8 delivery
		  | 78226.0s        0          191.9          209.6     48.2     88.1    100.7    142.6 newOrder
		  | 78226.0s        0           22.0           21.0     12.1     21.0     26.2     26.2 orderStatus
		  | 78226.0s        0          230.9          209.6     22.0     39.8     56.6     65.0 payment
		  | 78226.0s        0           14.0           21.0     83.9    201.3    234.9    234.9 stockLevel
		  | 78227.0s        0           22.0           21.0    117.4    192.9    218.1    218.1 delivery
		  | 78227.0s        0          189.7          209.6     56.6    125.8    151.0    167.8 newOrder
		  | 78227.0s        0           30.9           21.0     13.6     26.2     29.4     29.4 orderStatus
		  | 78227.0s        0          200.6          209.6     26.2     65.0     75.5     88.1 payment
		  | 78227.0s        0           24.0           21.0     96.5    285.2    302.0    302.0 stockLevel
		Wraps: (5) exit status 10
		Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errutil.withMessage (4) *main.withCommandDetails (5) *exec.ExitError

	cluster.go:2418,tpcc.go:168,tpcc.go:298,test_runner.go:753: monitor failure: monitor task failed: Goexit() was called
		(1) attached stack trace
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2406
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2414
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:168
		  | main.registerTPCC.func4
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:298
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:753
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2462
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2359
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:5222
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:190
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (6) Goexit() was called
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *withstack.withStack (4) *errutil.withMessage (5) *withstack.withStack (6) *errors.errorString

More

Artifacts: /weekly/tpcc/headroom

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Apr 18, 2020
@cockroach-teamcity cockroach-teamcity added this to the 20.1 milestone Apr 18, 2020
@cockroach-teamcity
Copy link
Member Author

(roachtest).weekly/tpcc/headroom failed on master@0e16cc15f139b816b8e46fe6571691a8ec0e6937:

		  | 205737.0s        0           38.9          209.4  64424.5  94489.3 103079.2 103079.2 newOrder
		  | 205737.0s        0            3.0           20.9  53687.1  60129.5  60129.5  60129.5 orderStatus
		  | 205737.0s        0           32.0          209.4  51539.6  66572.0  77309.4  77309.4 payment
		  | 205737.0s        0            5.0           20.9  55834.6  64424.5  64424.5  64424.5 stockLevel
		  | 205738.0s        0            5.0           20.9  60129.5  68719.5  68719.5  68719.5 delivery
		  | 205738.0s        0           68.1          209.4  62277.0  90194.3  90194.3  90194.3 newOrder
		  | 205738.0s        0            5.0           20.9  53687.1  68719.5  68719.5  68719.5 orderStatus
		  | 205738.0s        0           50.1          209.4  51539.6  73014.4  94489.3  94489.3 payment
		  | 205738.0s        0            8.0           20.9  49392.1  55834.6  55834.6  55834.6 stockLevel
		  | 205739.0s        0           10.0           20.9  55834.6  90194.3  90194.3  90194.3 delivery
		  | 205739.0s        0           84.0          209.4  60129.5  94489.3  98784.2 103079.2 newOrder
		  | 205739.0s        0            6.0           20.9  49392.1  60129.5  60129.5  60129.5 orderStatus
		  | 205739.0s        0           80.0          209.4  49392.1  77309.4  81604.4 103079.2 payment
		  | 205739.0s        0           13.0           20.9  45097.2  57982.1  66572.0  66572.0 stockLevel
		  | 205740.0s        0            6.0           20.9  57982.1  94489.3  94489.3  94489.3 delivery
		  | 205740.0s        0           66.9          209.4  60129.5  90194.3  98784.2 103079.2 newOrder
		  | 205740.0s        0            6.0           20.9  53687.1  60129.5  60129.5  60129.5 orderStatus
		  | 205740.0s        0           63.9          209.4  47244.6  68719.5  85899.3  94489.3 payment
		  | 205740.0s        0            5.0           20.9  49392.1  57982.1  57982.1  57982.1 stockLevel
		Wraps: (5) exit status 30
		Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errutil.withMessage (4) *main.withCommandDetails (5) *exec.ExitError

	cluster.go:2444,tpcc.go:168,tpcc.go:298,test_runner.go:753: monitor failure: monitor task failed: Goexit() was called
		(1) attached stack trace
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2432
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2440
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:168
		  | main.registerTPCC.func4
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:298
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:753
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2488
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2385
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:5222
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:190
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (6) Goexit() was called
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *withstack.withStack (4) *errutil.withMessage (5) *withstack.withStack (6) *errors.errorString

More

Artifacts: /weekly/tpcc/headroom

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).weekly/tpcc/headroom failed on master@61f18db7dd9a054d9a4648f67546202f760b5000:

		  | 146039.0s        0           11.0           21.0   4831.8   5100.3   5905.6   5905.6 orderStatus
		  | 146039.0s        0            0.0          209.6      0.0      0.0      0.0      0.0 payment
		  | 146039.0s        0           14.0           21.0   5100.3   6710.9   6979.3   6979.3 stockLevel
		  | 146040.0s        0            6.0           21.0   5905.6   7784.6   7784.6   7784.6 delivery
		  | 146040.0s        0           17.0          209.5   6174.0   9126.8  11274.3  11274.3 newOrder
		  | 146040.0s        0            1.0           21.0   4295.0   4295.0   4295.0   4295.0 orderStatus
		  | 146040.0s        0            0.0          209.6      0.0      0.0      0.0      0.0 payment
		  | 146040.0s        0           11.0           21.0   5905.6   6979.3   7516.2   7516.2 stockLevel
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 146041.0s        0            0.0           21.0      0.0      0.0      0.0      0.0 delivery
		  | 146041.0s        0            0.0          209.5      0.0      0.0      0.0      0.0 newOrder
		  | 146041.0s        0            0.0           21.0      0.0      0.0      0.0      0.0 orderStatus
		  | 146041.0s        0            0.0          209.6      0.0      0.0      0.0      0.0 payment
		  | 146041.0s        0            0.0           21.0      0.0      0.0      0.0      0.0 stockLevel
		  | 146042.0s        0           22.0           21.0   9663.7  15032.4  15569.3  15569.3 delivery
		  | 146042.0s        0          126.8          209.5  10200.5  15032.4  15569.3  15569.3 newOrder
		  | 146042.0s        0            4.0           21.0   6979.3   8589.9   8589.9   8589.9 orderStatus
		  | 146042.0s        0           85.9          209.6  10737.4  15032.4  15569.3  15569.3 payment
		  | 146042.0s        0            3.0           21.0   8053.1   9126.8   9126.8   9126.8 stockLevel
		Wraps: (5) exit status 30
		Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errutil.withMessage (4) *main.withCommandDetails (5) *exec.ExitError

	cluster.go:2456,tpcc.go:168,tpcc.go:298,test_runner.go:753: monitor failure: monitor task failed: Goexit() was called
		(1) attached stack trace
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2444
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2452
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:168
		  | main.registerTPCC.func4
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:298
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:753
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2500
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2397
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:5222
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:190
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (6) Goexit() was called
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *withstack.withStack (4) *errutil.withMessage (5) *withstack.withStack (6) *errors.errorString

More

Artifacts: /weekly/tpcc/headroom

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).weekly/tpcc/headroom failed on master@f6ce99f3e8c70059afdcb7683432cde5007c5102:

		  | 70539.0s        0           16.0           21.0     10.0     19.9     23.1     23.1 orderStatus
		  | 70539.0s        0          208.3          209.6     23.1     52.4     58.7    100.7 payment
		  | 70539.0s        0           13.0           21.0    117.4    192.9    234.9    234.9 stockLevel
		  | 70540.0s        0           26.0           21.0    109.1    201.3    209.7    209.7 delivery
		  | 70540.0s        0          207.0          209.6     56.6    130.0    142.6    151.0 newOrder
		  | 70540.0s        0           21.0           21.0     15.2     26.2     30.4     30.4 orderStatus
		  | 70540.0s        0          212.0          209.6     29.4     75.5     88.1    100.7 payment
		  | 70540.0s        0           22.0           21.0    125.8    243.3    318.8    318.8 stockLevel
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 70541.0s        0           27.0           21.0     83.9    117.4    130.0    130.0 delivery
		  | 70541.0s        0          186.0          209.5     41.9     56.6     60.8     71.3 newOrder
		  | 70541.0s        0           23.0           21.0     10.0     15.2     17.8     17.8 orderStatus
		  | 70541.0s        0          194.0          209.6     19.9     35.7     44.0     50.3 payment
		  | 70541.0s        0           29.0           21.0    100.7    142.6    159.4    159.4 stockLevel
		  | 70542.0s        0           31.0           21.0    104.9    218.1    243.3    243.3 delivery
		  | 70542.0s        0          186.7          209.5     52.4    104.9    117.4    121.6 newOrder
		  | 70542.0s        0           22.0           21.0     10.0     25.2     33.6     33.6 orderStatus
		  | 70542.0s        0          210.7          209.6     23.1     52.4     71.3     92.3 payment
		  | 70542.0s        0           16.0           21.0    117.4    159.4    226.5    226.5 stockLevel
		Wraps: (5) exit status 30
		Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errutil.withMessage (4) *main.withCommandDetails (5) *exec.ExitError

	cluster.go:2487,tpcc.go:168,tpcc.go:298,test_runner.go:753: monitor failure: monitor task failed: Goexit() was called
		(1) attached stack trace
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2475
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2483
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:168
		  | main.registerTPCC.func4
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:298
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:753
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2531
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2428
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:5222
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:190
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (6) Goexit() was called
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *withstack.withStack (4) *errutil.withMessage (5) *withstack.withStack (6) *errors.errorString

More

Artifacts: /weekly/tpcc/headroom

See this test on roachdash
powered by pkg/cmd/internal/issues

@knz
Copy link
Contributor

knz commented Jun 4, 2020

error in payment: ERROR: remote wall time is too far ahead (995.198703ms) to be trustworthy (SQLSTATE XXUUU)

This looks like clock skew. Is there anything actionable?

@cockroach-teamcity
Copy link
Member Author

(roachtest).weekly/tpcc/headroom failed on master@2dd15cd26f95801c678b1034e98a15560dc10baa:

		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *withstack.withStack (4) *errutil.withMessage (5) *withstack.withStack (6) *errors.errorString

	cluster.go:1512,context.go:135,cluster.go:1501,test_runner.go:825: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2023507-1592543750-03-n4cpu16 --oneshot --ignore-empty-nodes: exit status 1 4: skipped
		1: dead
		3: 4525
		2: dead
		Error: UNCLASSIFIED_PROBLEM: 1: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) secondary error attachment
		  | 2: dead
		  | (1) attached stack trace
		  |   | main.glob..func13
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1115
		  |   | main.wrap.func1
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:266
		  |   | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).execute
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:766
		  |   | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).ExecuteC
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:852
		  |   | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).Execute
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:800
		  |   | main.main
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1789
		  |   | runtime.main
		  |   | 	/usr/local/go/src/runtime/proc.go:203
		  |   | runtime.goexit
		  |   | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		  | Wraps: (2) 3 safe details enclosed
		  | Wraps: (3) 2: dead
		  | Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errors.errorString
		Wraps: (3) attached stack trace
		  | main.glob..func13
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1115
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:266
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:766
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:852
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:800
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1789
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:203
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (4) 3 safe details enclosed
		Wraps: (5) 1: dead
		Error types: (1) errors.Unclassified (2) *secondary.withSecondaryError (3) *withstack.withStack (4) *safedetails.withSafeDetails (5) *errors.errorString

More

Artifacts: /weekly/tpcc/headroom

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).weekly/tpcc/headroom failed on master@d3791a81c0716478de08d44459d3fcf5b4f3ea1e:

		  | 50321.0s        0            0.0          209.5      0.0      0.0      0.0      0.0 newOrder
		  | 50321.0s        0            0.0           21.0      0.0      0.0      0.0      0.0 orderStatus
		  | 50321.0s        0            0.0          209.6      0.0      0.0      0.0      0.0 payment
		  | 50321.0s        0            0.0           21.0      0.0      0.0      0.0      0.0 stockLevel
		  | 50322.0s        0            0.0           21.0      0.0      0.0      0.0      0.0 delivery
		  | 50322.0s        0            0.0          209.5      0.0      0.0      0.0      0.0 newOrder
		  | 50322.0s        0            0.0           21.0      0.0      0.0      0.0      0.0 orderStatus
		  | 50322.0s        0            0.0          209.6      0.0      0.0      0.0      0.0 payment
		  | 50322.0s        0            0.0           21.0      0.0      0.0      0.0      0.0 stockLevel
		  | 50323.0s        0            0.0           21.0      0.0      0.0      0.0      0.0 delivery
		  | 50323.0s        0            0.0          209.5      0.0      0.0      0.0      0.0 newOrder
		  | 50323.0s        0            0.0           21.0      0.0      0.0      0.0      0.0 orderStatus
		  | 50323.0s        0            0.0          209.6      0.0      0.0      0.0      0.0 payment
		  | 50323.0s        0            0.0           21.0      0.0      0.0      0.0      0.0 stockLevel
		  | 50324.0s        0           56.9           21.0  10200.5  11811.2  12348.0  12348.0 delivery
		  | 50324.0s        0          312.6          209.5  10200.5  11811.2  11811.2  11811.2 newOrder
		  | 50324.0s        0           22.0           21.0   6979.3   7784.6   7784.6   7784.6 orderStatus
		  | 50324.0s        0          220.7          209.6   9126.8  11274.3  11811.2  11811.2 payment
		  | 50324.0s        0           10.0           21.0   7516.2  10737.4  10737.4  10737.4 stockLevel
		Wraps: (5) exit status 30
		Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errutil.withMessage (4) *main.withCommandDetails (5) *exec.ExitError

	cluster.go:2484,tpcc.go:168,tpcc.go:298,test_runner.go:753: monitor failure: monitor task failed: Goexit() was called
		(1) attached stack trace
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2472
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2480
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:168
		  | main.registerTPCC.func4
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:298
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:753
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2528
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2425
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:5222
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:190
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (6) Goexit() was called
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *withstack.withStack (4) *errutil.withMessage (5) *withstack.withStack (6) *errors.errorString

More

Artifacts: /weekly/tpcc/headroom

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).weekly/tpcc/headroom failed on master@112ff533bf079fb057aadf5a998d9e1f09e6aa1f:

		  | 77925.0s        0          211.3          209.6     37.7     62.9     88.1    104.9 newOrder
		  | 77925.0s        0           16.0           21.0      8.9     10.5     26.2     26.2 orderStatus
		  | 77925.0s        0          216.3          209.7     17.8     28.3     54.5     58.7 payment
		  | 77925.0s        0           22.0           21.0     92.3    151.0    201.3    201.3 stockLevel
		  | 77926.0s        0           17.0           21.0     79.7    117.4    125.8    125.8 delivery
		  | 77926.0s        0          194.0          209.6     41.9     65.0     75.5    100.7 newOrder
		  | 77926.0s        0           22.0           21.0     10.0     13.6     14.7     14.7 orderStatus
		  | 77926.0s        0          206.0          209.7     21.0     35.7     52.4     56.6 payment
		  | 77926.0s        0           27.0           21.0     96.5    151.0    167.8    167.8 stockLevel
		  | 77927.0s        0           22.0           21.0     88.1    109.1    142.6    142.6 delivery
		  | 77927.0s        0          220.9          209.6     37.7     71.3     96.5     96.5 newOrder
		  | 77927.0s        0           20.0           21.0      8.9     16.8     21.0     21.0 orderStatus
		  | 77927.0s        0          213.9          209.7     17.8     35.7     62.9     75.5 payment
		  | 77927.0s        0           21.0           21.0    104.9    151.0    176.2    176.2 stockLevel
		  | 77928.0s        0           19.0           21.0    100.7    121.6    167.8    167.8 delivery
		  | 77928.0s        0          184.1          209.6     46.1     92.3    117.4    142.6 newOrder
		  | 77928.0s        0           25.0           21.0     13.6     25.2     31.5     31.5 orderStatus
		  | 77928.0s        0          178.1          209.7     22.0     46.1     62.9     79.7 payment
		  | 77928.0s        0           23.0           21.0    121.6    302.0    302.0    302.0 stockLevel
		Wraps: (5) exit status 20
		Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errutil.withMessage (4) *main.withCommandDetails (5) *exec.ExitError

	cluster.go:2467,tpcc.go:168,tpcc.go:298,test_runner.go:757: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2455
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2463
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:168
		  | main.registerTPCC.func4
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:298
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:757
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2511
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2425
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:5420
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:190
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1373
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *withstack.withStack (4) *errutil.withMessage (5) *withstack.withStack (6) *errors.errorString

More

Artifacts: /weekly/tpcc/headroom

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).weekly/tpcc/headroom failed on master@542c83e7db1694208e09c7fea7d350db4993b897:

		  | 	/usr/local/go/src/runtime/asm_amd64.s:1373
		Wraps: (2) 2 safe details enclosed
		Wraps: (3) output in run_063420.142_n4_workload_fixtures_load_tpcc
		Wraps: (4) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2079183-1594357816-03-n4cpu16:4 -- ./workload fixtures load tpcc --warehouses=1000  {pgurl:1} returned
		  | stderr:
		  | 22.218079 1 ccl/workloadccl/cliccl/fixtures.go:279  starting restore of 9 tables
		  | I200710 06:34:22.693936 117 ccl/workloadccl/fixture.go:586  loaded 53 KiB table warehouse in 465.548ms (1000 rows, 0 index entries, 114 KiB)
		  | I200710 06:34:43.380666 122 ccl/workloadccl/fixture.go:586  loaded 126 MiB table new_order in 21.152408627s (9000000 rows, 0 index entries, 5.9 MiB)
		  | I200710 06:36:12.697497 118 ccl/workloadccl/fixture.go:586  loaded 1006 KiB table district in 1m50.469186768s (10000 rows, 0 index entries, 9.1 KiB)
		  | W200710 06:39:39.743814 1 ccl/workloadccl/fixture.go:497  error enabling automatic stats: dial tcp 10.128.0.97:26257: connect: connection refused
		  | Error: restoring fixture: dial tcp 10.128.0.97:26257: connect: connection refused
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 4. Command with error:
		  |   | ```
		  |   | ./workload fixtures load tpcc --warehouses=1000  {pgurl:1}
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (5) exit status 20
		Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errutil.withMessage (4) *main.withCommandDetails (5) *exec.ExitError

	cluster.go:1516,context.go:135,cluster.go:1505,test_runner.go:826: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2079183-1594357816-03-n4cpu16 --oneshot --ignore-empty-nodes: exit status 1 4: skipped
		3: 4243
		2: 4278
		1: dead
		Error: UNCLASSIFIED_PROBLEM: 1: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  | main.glob..func13
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1115
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:266
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:830
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:914
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:864
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1789
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:203
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1373
		Wraps: (3) 3 safe details enclosed
		Wraps: (4) 1: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *safedetails.withSafeDetails (4) *errors.errorString

More

Artifacts: /weekly/tpcc/headroom

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).weekly/tpcc/headroom failed on master@af0031e3004327b8d09e23e99eb9659abf7d82de:

		  | 27033.0s        0            0.0          209.5      0.0      0.0      0.0      0.0 newOrder
		  | 27033.0s        0            0.0           21.0      0.0      0.0      0.0      0.0 orderStatus
		  | 27033.0s        0            0.0          209.6      0.0      0.0      0.0      0.0 payment
		  | 27033.0s        0            0.0           21.0      0.0      0.0      0.0      0.0 stockLevel
		  | 27034.0s        0            0.0           21.0      0.0      0.0      0.0      0.0 delivery
		  | 27034.0s        0            0.0          209.5      0.0      0.0      0.0      0.0 newOrder
		  | 27034.0s        0            0.0           21.0      0.0      0.0      0.0      0.0 orderStatus
		  | 27034.0s        0            0.0          209.6      0.0      0.0      0.0      0.0 payment
		  | 27034.0s        0            0.0           21.0      0.0      0.0      0.0      0.0 stockLevel
		  | 27035.0s        0            0.0           21.0      0.0      0.0      0.0      0.0 delivery
		  | 27035.0s        0            0.0          209.5      0.0      0.0      0.0      0.0 newOrder
		  | 27035.0s        0            0.0           21.0      0.0      0.0      0.0      0.0 orderStatus
		  | 27035.0s        0            0.0          209.5      0.0      0.0      0.0      0.0 payment
		  | 27035.0s        0            0.0           21.0      0.0      0.0      0.0      0.0 stockLevel
		  | 27036.0s        0           45.8           21.0  16106.1  19327.4  19327.4  19327.4 delivery
		  | 27036.0s        0          205.1          209.5  17179.9  19327.4  19327.4  19327.4 newOrder
		  | 27036.0s        0           11.9           21.0  12884.9  13958.6  13958.6  13958.6 orderStatus
		  | 27036.0s        0          276.8          209.5  16643.0  19327.4  19327.4  19327.4 payment
		  | 27036.0s        0            0.0           21.0      0.0      0.0      0.0      0.0 stockLevel
		Wraps: (5) exit status 20
		Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errutil.withMessage (4) *main.withCommandDetails (5) *exec.ExitError

	cluster.go:2494,tpcc.go:187,tpcc.go:320,test_runner.go:757: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2482
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2490
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:187
		  | main.registerTPCC.func4
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:320
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:757
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2538
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2452
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:5420
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:190
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1373
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *withstack.withStack (4) *errutil.withMessage (5) *withstack.withStack (6) *errors.errorString

More

Artifacts: /weekly/tpcc/headroom

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).weekly/tpcc/headroom failed on master@bfa6307c292ef4dfed4a53cb99f506e6dab26533:

		  | 81802.0s        0          200.0          209.6     35.7     46.1     58.7     62.9 newOrder
		  | 81802.0s        0           21.0           21.0      7.9     13.6     14.2     14.2 orderStatus
		  | 81802.0s        0          207.0          209.7     16.3     22.0     29.4     32.5 payment
		  | 81802.0s        0           26.0           21.0     88.1    117.4    130.0    130.0 stockLevel
		  | 81803.0s        0           33.0           21.0     92.3    142.6    167.8    167.8 delivery
		  | 81803.0s        0          200.9          209.6     44.0     96.5    125.8    130.0 newOrder
		  | 81803.0s        0           27.0           21.0      9.4     22.0     25.2     25.2 orderStatus
		  | 81803.0s        0          215.9          209.7     22.0     60.8     88.1    125.8 payment
		  | 81803.0s        0           21.0           21.0    113.2    234.9    285.2    285.2 stockLevel
		  | 81804.0s        0           26.1           21.0     83.9    113.2    117.4    117.4 delivery
		  | 81804.0s        0          197.4          209.6     37.7     54.5     67.1     71.3 newOrder
		  | 81804.0s        0           20.0           21.0      9.4     12.6     12.6     12.6 orderStatus
		  | 81804.0s        0          218.5          209.7     16.8     30.4     44.0     52.4 payment
		  | 81804.0s        0           27.1           21.0     88.1    130.0    130.0    130.0 stockLevel
		  | 81805.0s        0           23.0           21.0    104.9    251.7    268.4    268.4 delivery
		  | 81805.0s        0          185.0          209.6     54.5    167.8    209.7    209.7 newOrder
		  | 81805.0s        0           20.0           21.0     13.1     35.7     44.0     44.0 orderStatus
		  | 81805.0s        0          219.0          209.7     27.3     75.5    113.2    151.0 payment
		  | 81805.0s        0           23.0           21.0    130.0    335.5    402.7    402.7 stockLevel
		Wraps: (5) exit status 20
		Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errutil.withMessage (4) *main.withCommandDetails (5) *exec.ExitError

	cluster.go:2553,tpcc.go:187,tpcc.go:320,test_runner.go:757: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2541
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2549
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:187
		  | main.registerTPCC.func4
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:320
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:757
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2597
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2511
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:5420
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:190
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1373
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *withstack.withStack (4) *errutil.withMessage (5) *withstack.withStack (6) *errors.errorString

More

Artifacts: /weekly/tpcc/headroom

See this test on roachdash
powered by pkg/cmd/internal/issues

@knz
Copy link
Contributor

knz commented Jul 25, 2020

This latest error is different from the earlier failure.

It indicates a t.Fatal call in an unexpected place. @andreimatei did your PR about t.Fatal in goroutine not already address this?

@andreimatei
Copy link
Contributor

The last failure seems to be Error: error in newOrder: ERROR: remote wall time is too far ahead (996.958792ms) to be trustworthy (SQLSTATE XXUUU).
I think the monitor failure: monitor task failed: t.Fatal() was called message just indicates that we were running the workload binary in a monitor task, and that task called t.Fatal(). That's fine; the monitor knows how to deal with it.

@cockroach-teamcity
Copy link
Member Author

(roachtest).weekly/tpcc/headroom failed on master@8bdf6ba0f908f562822348300597b9f435f478d1:

		  | 80189.0s        0           17.0           21.0     13.1     22.0     25.2     25.2 orderStatus
		  | 80189.0s        0          171.0          209.7     30.4     54.5     60.8     65.0 payment
		  | 80189.0s        0           19.0           21.0    113.2    167.8    285.2    285.2 stockLevel
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 80190.0s        0           15.0           21.0    100.7    121.6    176.2    176.2 delivery
		  | 80190.0s        0          209.6          209.6     50.3     83.9     96.5    109.1 newOrder
		  | 80190.0s        0           18.0           21.0      9.4     13.1     17.8     17.8 orderStatus
		  | 80190.0s        0          223.6          209.7     24.1     52.4     62.9     96.5 payment
		  | 80190.0s        0           16.0           21.0    100.7    209.7    243.3    243.3 stockLevel
		  | 80191.0s        0           17.0           21.0    100.7    159.4    167.8    167.8 delivery
		  | 80191.0s        0          213.0          209.6     46.1     75.5     92.3    104.9 newOrder
		  | 80191.0s        0           12.0           21.0     11.5     17.8     19.9     19.9 orderStatus
		  | 80191.0s        0          209.0          209.7     23.1     41.9     54.5     79.7 payment
		  | 80191.0s        0           17.0           21.0    113.2    159.4    159.4    159.4 stockLevel
		  | 80192.0s        0           19.0           21.0    151.0    234.9    243.3    243.3 delivery
		  | 80192.0s        0          208.9          209.6     71.3    167.8    201.3    419.4 newOrder
		  | 80192.0s        0           17.0           21.0     11.5     17.8     28.3     28.3 orderStatus
		  | 80192.0s        0          230.9          209.7     33.6     83.9    113.2    130.0 payment
		  | 80192.0s        0           27.0           21.0    134.2    369.1    385.9    385.9 stockLevel
		Wraps: (5) exit status 20
		Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errutil.withMessage (4) *main.withCommandDetails (5) *exec.ExitError

	cluster.go:2553,tpcc.go:187,tpcc.go:320,test_runner.go:754: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2541
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2549
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:187
		  | main.registerTPCC.func4
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:320
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:754
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2597
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2511
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:5420
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:190
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1373
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *withstack.withStack (4) *errutil.withMessage (5) *withstack.withStack (6) *errors.errorString

More

Artifacts: /weekly/tpcc/headroom

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).weekly/tpcc/headroom failed on master@d310eb8eb6da5df4ac4407a2c90d898acdfdddf3:

		  | 26970.0s        0            0.0           20.9      0.0      0.0      0.0      0.0 orderStatus
		  | 26970.0s        0            0.0          209.5      0.0      0.0      0.0      0.0 payment
		  | 26970.0s        0            0.0           21.0      0.0      0.0      0.0      0.0 stockLevel
		  | 26971.0s        0            0.0           20.9      0.0      0.0      0.0      0.0 delivery
		  | 26971.0s        0            0.0          209.4      0.0      0.0      0.0      0.0 newOrder
		  | 26971.0s        0            0.0           20.9      0.0      0.0      0.0      0.0 orderStatus
		  | 26971.0s        0            0.0          209.4      0.0      0.0      0.0      0.0 payment
		  | 26971.0s        0            0.0           21.0      0.0      0.0      0.0      0.0 stockLevel
		  | 26972.0s        0            0.0           20.9      0.0      0.0      0.0      0.0 delivery
		  | 26972.0s        0            0.0          209.4      0.0      0.0      0.0      0.0 newOrder
		  | 26972.0s        0            0.0           20.9      0.0      0.0      0.0      0.0 orderStatus
		  | 26972.0s        0            0.0          209.4      0.0      0.0      0.0      0.0 payment
		  | 26972.0s        0            0.0           20.9      0.0      0.0      0.0      0.0 stockLevel
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 26973.0s        0            0.0           20.9      0.0      0.0      0.0      0.0 delivery
		  | 26973.0s        0            0.0          209.4      0.0      0.0      0.0      0.0 newOrder
		  | 26973.0s        0            0.0           20.9      0.0      0.0      0.0      0.0 orderStatus
		  | 26973.0s        0            0.0          209.4      0.0      0.0      0.0      0.0 payment
		  | 26973.0s        0            0.0           20.9      0.0      0.0      0.0      0.0 stockLevel
		Wraps: (5) exit status 20
		Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errutil.withMessage (4) *main.withCommandDetails (5) *exec.ExitError

	cluster.go:2556,tpcc.go:187,tpcc.go:320,test_runner.go:754: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2544
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2552
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:187
		  | main.registerTPCC.func4
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:320
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:754
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2600
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2514
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:5420
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:190
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1373
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *withstack.withStack (4) *errutil.withMessage (5) *withstack.withStack (6) *errors.errorString

More

Artifacts: /weekly/tpcc/headroom

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).weekly/tpcc/headroom failed on master@1049eaa26d983a0449049264d6c2ee2cc27454e2:

		  | 79896.0s        0           13.0           21.0      9.4     13.6     25.2     25.2 orderStatus
		  | 79896.0s        0          212.0          209.6     21.0     75.5     92.3    109.1 payment
		  | 79896.0s        0           25.0           21.0    113.2    285.2    335.5    335.5 stockLevel
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 79897.0s        0           23.0           21.0    130.0    218.1    226.5    226.5 delivery
		  | 79897.0s        0          197.8          209.6     56.6    117.4    167.8    192.9 newOrder
		  | 79897.0s        0           29.0           21.0     11.5     37.7     56.6     56.6 orderStatus
		  | 79897.0s        0          208.8          209.6     30.4     75.5    100.7    109.1 payment
		  | 79897.0s        0           11.0           21.0    104.9    125.8    125.8    125.8 stockLevel
		  | 79898.0s        0           23.0           21.0     83.9    176.2    218.1    218.1 delivery
		  | 79898.0s        0          236.3          209.6     46.1    125.8    151.0    167.8 newOrder
		  | 79898.0s        0           24.0           21.0     10.0     26.2     29.4     29.4 orderStatus
		  | 79898.0s        0          220.2          209.6     24.1     62.9    100.7    125.8 payment
		  | 79898.0s        0           25.0           21.0    109.1    285.2    302.0    302.0 stockLevel
		  | 79899.0s        0           21.0           21.0     92.3    142.6    176.2    176.2 delivery
		  | 79899.0s        0          200.7          209.6     41.9     75.5     83.9     92.3 newOrder
		  | 79899.0s        0           15.0           21.0     11.0     14.7     15.7     15.7 orderStatus
		  | 79899.0s        0          206.7          209.6     21.0     37.7     56.6     58.7 payment
		  | 79899.0s        0           14.0           21.0     92.3    201.3    226.5    226.5 stockLevel
		Wraps: (5) exit status 20
		Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errutil.withMessage (4) *main.withCommandDetails (5) *exec.ExitError

	cluster.go:2597,tpcc.go:187,tpcc.go:320,test_runner.go:754: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2585
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2593
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:187
		  | main.registerTPCC.func4
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:320
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:754
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2641
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2555
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:5228
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:190
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *withstack.withStack (4) *errutil.withMessage (5) *withstack.withStack (6) *errors.errorString

More

Artifacts: /weekly/tpcc/headroom

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).weekly/tpcc/headroom failed on master@bbbedabbf6ea0b1ff6fc799a0c04a75295a9f4c2:

		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 59177.0s        0            0.0           21.0      0.0      0.0      0.0      0.0 delivery
		  | 59177.0s        0            0.0          209.5      0.0      0.0      0.0      0.0 newOrder
		  | 59177.0s        0            0.0           21.0      0.0      0.0      0.0      0.0 orderStatus
		  | 59177.0s        0            0.0          209.5      0.0      0.0      0.0      0.0 payment
		  | 59177.0s        0            0.0           21.0      0.0      0.0      0.0      0.0 stockLevel
		  | 59178.0s        0            0.0           21.0      0.0      0.0      0.0      0.0 delivery
		  | 59178.0s        0            0.0          209.5      0.0      0.0      0.0      0.0 newOrder
		  | 59178.0s        0            0.0           21.0      0.0      0.0      0.0      0.0 orderStatus
		  | 59178.0s        0            0.0          209.5      0.0      0.0      0.0      0.0 payment
		  | 59178.0s        0            0.0           21.0      0.0      0.0      0.0      0.0 stockLevel
		  | 59179.0s        0            0.0           21.0      0.0      0.0      0.0      0.0 delivery
		  | 59179.0s        0            0.0          209.5      0.0      0.0      0.0      0.0 newOrder
		  | 59179.0s        0            0.0           21.0      0.0      0.0      0.0      0.0 orderStatus
		  | 59179.0s        0            0.0          209.5      0.0      0.0      0.0      0.0 payment
		  | 59179.0s        0            0.0           20.9      0.0      0.0      0.0      0.0 stockLevel
		Wraps: (5) exit status 20
		Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errutil.withMessage (4) *main.withCommandDetails (5) *exec.ExitError

	cluster.go:2597,tpcc.go:187,tpcc.go:320,test_runner.go:754: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2585
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2593
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:187
		  | main.registerTPCC.func4
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:320
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:754
		Wraps: (2) 1 safe detail enclosed
		Wraps: (3) monitor failure
		Wraps: (4) attached stack trace
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2641
		Wraps: (5) 1 safe detail enclosed
		Wraps: (6) monitor task failed
		Wraps: (7) attached stack trace
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2555
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:5228
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:190
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (8) 1 safe detail enclosed
		Wraps: (9) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errutil.withMessage (4) *withstack.withStack (5) *safedetails.withSafeDetails (6) *errutil.withMessage (7) *withstack.withStack (8) *safedetails.withSafeDetails (9) *errors.errorString

More

Artifacts: /weekly/tpcc/headroom

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).weekly/tpcc/headroom failed on master@123ca0782f423bc3d225a5ca798ece60c3185935:

		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 82822.0s        0           13.0           21.0     88.1    226.5    385.9    385.9 delivery
		  | 82822.0s        0          212.9          209.6     56.6    192.9    243.3    268.4 newOrder
		  | 82822.0s        0           20.0           21.0     11.0     62.9     75.5     75.5 orderStatus
		  | 82822.0s        0          205.9          209.6     26.2    117.4    142.6    159.4 payment
		  | 82822.0s        0           19.0           21.0     79.7    184.5    251.7    251.7 stockLevel
		  | 82823.0s        0           31.0           21.0    100.7    226.5    302.0    302.0 delivery
		  | 82823.0s        0          206.1          209.6     44.0    130.0    184.5    201.3 newOrder
		  | 82823.0s        0           18.0           21.0     10.5     27.3     48.2     48.2 orderStatus
		  | 82823.0s        0          211.1          209.6     19.9     58.7     79.7     92.3 payment
		  | 82823.0s        0           19.0           21.0     92.3    192.9    234.9    234.9 stockLevel
		  | 82824.0s        0           17.0           21.0    113.2    260.0    302.0    302.0 delivery
		  | 82824.0s        0          213.3          209.6     58.7    151.0    184.5    184.5 newOrder
		  | 82824.0s        0           23.0           21.0     13.6     30.4     48.2     48.2 orderStatus
		  | 82824.0s        0          207.3          209.6     28.3    100.7    130.0    159.4 payment
		  | 82824.0s        0           22.0           21.0     88.1    318.8    385.9    385.9 stockLevel
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

	cluster.go:2613,tpcc.go:187,tpcc.go:320,test_runner.go:754: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2601
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2609
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:187
		  | main.registerTPCC.func4
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:320
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:754
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2657
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2571
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:5228
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:190
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

More

Artifacts: /weekly/tpcc/headroom

See this test on roachdash
powered by pkg/cmd/internal/issues

@knz
Copy link
Contributor

knz commented Aug 31, 2020

ERROR: remote wall time is too far ahead (997.439795ms) to be trustworthy (SQLSTATE XXUUU)

again

this looks like infra flake. Do we want to mark the test as failed in that case?

@jlinder is there a way to make the test report as non-failed when this symptom is encountered?

@knz knz removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Aug 31, 2020
@tbg
Copy link
Member

tbg commented Sep 1, 2020

It seems to fail like that every week, so there's definitely something to investigate here. Can this really be true?

@cockroach-teamcity
Copy link
Member Author

(roachtest).weekly/tpcc/headroom failed on master@9fae2abb48a00c26ac0344f35cd100df7faa2aec:

		  | 79233.0s        0           16.0           21.0     96.5    226.5    302.0    302.0 stockLevel
		  | 79234.0s        0           15.0           21.0    134.2    209.7    218.1    218.1 delivery
		  | 79234.0s        0          187.4          209.5     65.0    159.4    176.2    184.5 newOrder
		  | 79234.0s        0           22.0           21.0     11.5     48.2     62.9     62.9 orderStatus
		  | 79234.0s        0          180.4          209.6     35.7     88.1    104.9    113.2 payment
		  | 79234.0s        0           18.0           21.0    100.7    201.3    209.7    209.7 stockLevel
		  | 79235.0s        0           22.0           21.0    100.7    302.0    318.8    318.8 delivery
		  | 79235.0s        0          237.9          209.5     41.9    167.8    201.3    243.3 newOrder
		  | 79235.0s        0           17.0           21.0      9.4     14.7     27.3     27.3 orderStatus
		  | 79235.0s        0          228.9          209.6     19.9     79.7    104.9    113.2 payment
		  | 79235.0s        0           22.0           21.0     83.9    159.4    201.3    201.3 stockLevel
		  | 79236.0s        0           17.0           21.0    151.0    218.1    218.1    218.1 delivery
		  | 79236.0s        0          182.8          209.5     46.1    121.6    151.0    176.2 newOrder
		  | 79236.0s        0           22.0           21.0     11.0     35.7     37.7     37.7 orderStatus
		  | 79236.0s        0          196.8          209.6     23.1     75.5     88.1     92.3 payment
		  | 79236.0s        0           22.0           21.0     79.7    226.5    260.0    260.0 stockLevel
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

	cluster.go:2648,tpcc.go:190,tpcc.go:323,test_runner.go:754: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2636
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2644
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:190
		  | main.registerTPCC.func4
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:323
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:754
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2692
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2606
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:5228
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:190
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

More

Artifacts: /weekly/tpcc/headroom

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).weekly/tpcc/headroom failed on master@339275585b7d30b9ee2d49b0c696b9ddb8d51ad4:

		  | 78207.0s        0           20.0           21.0     79.7    151.0    184.5    184.5 delivery
		  | 78207.0s        0          221.9          209.7     32.5     75.5     92.3    100.7 newOrder
		  | 78207.0s        0           19.0           21.0      6.3     11.0     11.5     11.5 orderStatus
		  | 78207.0s        0          216.9          209.8     18.9     48.2     65.0     75.5 payment
		  | 78207.0s        0           16.0           21.0     50.3     71.3     83.9     83.9 stockLevel
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 78208.0s        0           14.0           21.0     75.5    151.0    159.4    159.4 delivery
		  | 78208.0s        0          211.7          209.7     30.4     67.1     83.9    109.1 newOrder
		  | 78208.0s        0           24.0           21.0      7.3     14.2     15.2     15.2 orderStatus
		  | 78208.0s        0          202.8          209.8     17.8     39.8     54.5     75.5 payment
		  | 78208.0s        0           21.0           21.0     39.8    130.0    159.4    159.4 stockLevel
		  | 78209.0s        0           15.0           21.0     71.3    113.2    130.0    130.0 delivery
		  | 78209.0s        0          197.3          209.7     30.4     56.6     67.1    113.2 newOrder
		  | 78209.0s        0           26.0           21.0      7.1     10.5     17.8     17.8 orderStatus
		  | 78209.0s        0          205.3          209.8     16.8     41.9     48.2     50.3 payment
		  | 78209.0s        0           19.0           21.0     39.8     67.1     71.3     71.3 stockLevel
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

	cluster.go:2648,tpcc.go:164,tpcc.go:293,test_runner.go:760: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2636
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2644
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:164
		  | main.registerTPCC.func4
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:293
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:760
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2692
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2606
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:5652
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:191
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

More

Artifacts: /weekly/tpcc/headroom

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).weekly/tpcc/headroom failed on master@a786c51627fe66e47b4a4445c67b2a9077ae2a93:

		  -- stack trace:
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2305
		  | main.(*cluster).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2225
		  | main.setupTPCC.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:120
		  | main.setupTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:136
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:147
		  | main.registerTPCC.func4
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:295
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:760
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (2) output in run_083247.515_n1_cockroach_workload_fixtures_import_tpcc
		Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2586207-1610691606-03-n4cpu16:1 -- ./cockroach workload fixtures import tpcc --warehouses=1000  returned
		  | stderr:
		  | I210115 08:32:48.330527 1 ccl/workloadccl/fixture.go:342  starting import of 9 tables
		  | I210115 08:32:48.568407 104 ccl/workloadccl/fixture.go:472  imported 1006 KiB in district table (10000 rows, 0 index entries, took 229.799069ms, 4.28 MiB/s)
		  | I210115 08:32:51.282692 103 ccl/workloadccl/fixture.go:472  imported 53 KiB in warehouse table (1000 rows, 0 index entries, took 2.944094508s, 0.02 MiB/s)
		  | I210115 08:32:52.395852 109 ccl/workloadccl/fixture.go:472  imported 7.8 MiB in item table (100000 rows, 0 index entries, took 4.057172736s, 1.92 MiB/s)
		  | I210115 08:33:04.412608 108 ccl/workloadccl/fixture.go:472  imported 126 MiB in new_order table (9000000 rows, 0 index entries, took 16.073912968s, 7.81 MiB/s)
		  | I210115 08:34:05.398523 106 ccl/workloadccl/fixture.go:472  imported 2.2 GiB in history table (30000000 rows, 0 index entries, took 1m17.059703859s, 28.63 MiB/s)
		  | I210115 08:34:12.214403 107 ccl/workloadccl/fixture.go:472  imported 1.6 GiB in order table (30000000 rows, 30000000 index entries, took 1m23.87573319s, 19.02 MiB/s)
		  | I210115 08:36:49.864008 105 ccl/workloadccl/fixture.go:472  imported 17 GiB in customer table (30000000 rows, 30000000 index entries, took 4m1.52525515s, 73.10 MiB/s)
		  | I210115 08:39:24.659032 111 ccl/workloadccl/fixture.go:472  imported 17 GiB in order_line table (300011520 rows, 0 index entries, took 6m36.320359901s, 43.45 MiB/s)
		  | I210115 08:39:37.828357 110 ccl/workloadccl/fixture.go:472  imported 30 GiB in stock table (100000000 rows, 0 index entries, took 6m49.489605225s, 75.08 MiB/s)
		  | I210115 08:39:37.876986 1 ccl/workloadccl/fixture.go:351  imported 68 GiB bytes in 9 tables (took 6m49.546184068s, 169.84 MiB/s)
		  | I210115 08:39:38.832551 1 ccl/workloadccl/cliccl/fixtures.go:355  fixture is restored; now running consistency checks (ctrl-c to abort)
		  | I210115 08:39:38.877177 1 workload/tpcc/tpcc.go:389  check 3.3.2.1 took 44.57127ms
		  | I210115 08:39:55.398757 1 workload/tpcc/tpcc.go:389  check 3.3.2.2 took 16.521505268s
		  | I210115 08:39:59.305526 1 workload/tpcc/tpcc.go:389  check 3.3.2.3 took 3.906713994s
		  | I210115 08:43:35.765035 1 workload/tpcc/tpcc.go:389  check 3.3.2.4 took 3m36.459458139s
		  | I210115 08:43:54.435343 1 workload/tpcc/tpcc.go:389  check 3.3.2.5 took 18.670230824s
		  | Error: check failed: 3.3.2.5: left EXCEPT right returned nonzero results.
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ```
		  |   | ./cockroach workload fixtures import tpcc --warehouses=1000
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

More

Artifacts: /weekly/tpcc/headroom

See this test on roachdash
powered by pkg/cmd/internal/issues

@tbg
Copy link
Member

tbg commented Jan 15, 2021

	  | I210115 08:39:38.832551 1 ccl/workloadccl/cliccl/fixtures.go:355  fixture is restored; now running consistency checks (ctrl-c to abort)
		  | I210115 08:39:38.877177 1 workload/tpcc/tpcc.go:389  check 3.3.2.1 took 44.57127ms
		  | I210115 08:39:55.398757 1 workload/tpcc/tpcc.go:389  check 3.3.2.2 took 16.521505268s
		  | I210115 08:39:59.305526 1 workload/tpcc/tpcc.go:389  check 3.3.2.3 took 3.906713994s
		  | I210115 08:43:35.765035 1 workload/tpcc/tpcc.go:389  check 3.3.2.4 took 3m36.459458139s
		  | I210115 08:43:54.435343 1 workload/tpcc/tpcc.go:389  check 3.3.2.5 took 18.670230824s
		  | Error: check failed: 3.3.2.5: left EXCEPT right returned nonzero results

This looks concerning.

@yuzefovich
Copy link
Member

I believe the nonzero results failure is the same as #58935 which I have a PR open to fix. @tbg @andreimatei do you guys want this issue to be closed when that PR merges?

@tbg
Copy link
Member

tbg commented Jan 15, 2021 via email

craig bot pushed a commit that referenced this issue Jan 15, 2021
58713: ui: add eslint with shared config r=dhartunian a=dhartunian

Our UI codebase does not have standardized formatting
yet. In order to get to a point where we can enforce these
standards, we're adding the ability to run `eslint` for
UI code in `pkg/ui`.

A base set of ignored rules is added in `package.json`
in order to get a clean run. It is our intention to
eventually remove these ignored rules once we clean up
the codebase further. Currently they result in over
1500 warnings.

We are also removing the use of the `tslint` tool as
`eslint` is preferred for TypeScript linting as well.

`eslint` will now run as part of the Makefile `ui-lint`
task.

Further PRs will contain formatting fixes to get eslint
to pass.

Depends on: cockroachdb/yarn-vendored#48

Release note: none

58877: filetable: read chunks as needed r=dt a=dt

This changes filetable to read only the block needed from the table
instead of reading the whole file at once.

Release note: none.

59057: colexec: fix coalescing logic in the spilling queue r=yuzefovich a=yuzefovich

In a recently merged change to the spilling queue we introduced a bug in
the coalescing logic - we incorrectly computed the index of the "tail"
batch in case when it is the last one in the in-memory buffer. This
could lead to multiple problems (modification of the filled-up batch, of
the already dequeued batch, etc). This is now fixed.

Additionally, this commit switches to using `Copy` instead of `Append`
in that coalescing logic (since we know that there is enough capacity
for all of the tuples we're appending) and sets the item to `nil` when
it was dequeued.

Addresses: #47652.
Fixes: #58913.
Fixes: #58915.
Fixes: #58934.
Fixes: #58935.
Fixes: #58936.
Fixes: #58956.

Release note: None (no stable release with this bug)

Co-authored-by: David Hartunian <davidh@cockroachlabs.com>
Co-authored-by: David Taylor <tinystatemachine@gmail.com>
Co-authored-by: Yahor Yuzefovich <yahor@cockroachlabs.com>
@cockroach-teamcity
Copy link
Member Author

(roachtest).weekly/tpcc/headroom failed on master@baf34b241c9ee27530f0a3a988149bfd7cca3f6d:

		  | 75747.0s        0           26.9           21.0    134.2    285.2    318.8    318.8 delivery
		  | 75747.0s        0          224.5          209.7     46.1    159.4    184.5    201.3 newOrder
		  | 75747.0s        0           24.0           21.0      7.9     29.4     39.8     39.8 orderStatus
		  | 75747.0s        0          200.6          209.7     27.3    100.7    134.2    243.3 payment
		  | 75747.0s        0           14.0           21.0     50.3    167.8    184.5    184.5 stockLevel
		  | 75748.0s        0           21.0           21.0     79.7    134.2    151.0    151.0 delivery
		  | 75748.0s        0          226.9          209.7     39.8     83.9     92.3    104.9 newOrder
		  | 75748.0s        0           19.0           21.0      7.6     10.5     11.5     11.5 orderStatus
		  | 75748.0s        0          208.0          209.7     26.2     58.7     71.3     83.9 payment
		  | 75748.0s        0           24.0           21.0     54.5    167.8    184.5    184.5 stockLevel
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 75749.0s        0           20.0           21.0     83.9    134.2    184.5    184.5 delivery
		  | 75749.0s        0          196.4          209.7     35.7     58.7     67.1     67.1 newOrder
		  | 75749.0s        0           22.0           21.0      7.3     17.8     37.7     37.7 orderStatus
		  | 75749.0s        0          171.4          209.7     22.0     32.5     37.7     52.4 payment
		  | 75749.0s        0           33.1           21.0     46.1     62.9     88.1     88.1 stockLevel
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

	cluster.go:2687,tpcc.go:162,tpcc.go:327,test_runner.go:760: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2675
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2683
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:162
		  | main.registerTPCC.func4
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:327
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:760
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2731
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2645
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:5652
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:191
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

More

Artifacts: /weekly/tpcc/headroom

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).weekly/tpcc/headroom failed on master@86c619f768a4d6f4cc80efb5c9479e5fc0fd72c1:

		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 79346.0s        0           21.0           21.0     83.9    192.9    226.5    226.5 delivery
		  | 79346.0s        0          205.0          209.6     35.7    142.6    151.0    167.8 newOrder
		  | 79346.0s        0           22.0           21.0      7.9     18.9     39.8     39.8 orderStatus
		  | 79346.0s        0          184.0          209.7     21.0     62.9     88.1    121.6 payment
		  | 79346.0s        0           27.0           21.0     44.0     96.5    142.6    142.6 stockLevel
		  | 79347.0s        0           26.0           21.0    109.1    268.4    268.4    268.4 delivery
		  | 79347.0s        0          200.6          209.6     52.4    113.2    151.0    192.9 newOrder
		  | 79347.0s        0           18.0           21.0      9.4     16.8     18.9     18.9 orderStatus
		  | 79347.0s        0          241.6          209.7     31.5     75.5    113.2    125.8 payment
		  | 79347.0s        0           24.0           21.0     56.6    218.1    251.7    251.7 stockLevel
		  | 79348.0s        0           22.0           21.0     83.9    113.2    117.4    117.4 delivery
		  | 79348.0s        0          189.0          209.6     37.7     62.9     71.3     83.9 newOrder
		  | 79348.0s        0           24.0           21.0      7.1     12.1     24.1     24.1 orderStatus
		  | 79348.0s        0          227.0          209.7     21.0     41.9     54.5     60.8 payment
		  | 79348.0s        0           22.0           21.0     41.9     71.3     75.5     75.5 stockLevel
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

	cluster.go:2687,tpcc.go:162,tpcc.go:327,test_runner.go:767: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2675
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2683
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:162
		  | main.registerTPCC.func4
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:327
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:767
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2731
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2645
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:5652
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:191
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

More

Artifacts: /weekly/tpcc/headroom

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).weekly/tpcc/headroom failed on master@c584f62067a45aa540c26fc9081a83e460bfe37a:

		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 78577.0s        0           21.0           21.0    100.7    285.2    302.0    302.0 delivery
		  | 78577.0s        0          196.7          209.7     37.7    142.6    159.4    243.3 newOrder
		  | 78577.0s        0           19.0           21.0      7.3     24.1     50.3     50.3 orderStatus
		  | 78577.0s        0          215.7          209.8     22.0    109.1    130.0    184.5 payment
		  | 78577.0s        0           26.0           21.0     41.9    184.5    192.9    192.9 stockLevel
		  | 78578.0s        0           18.0           21.0     83.9    151.0    159.4    159.4 delivery
		  | 78578.0s        0          187.3          209.7     33.6     75.5     96.5    117.4 newOrder
		  | 78578.0s        0           18.0           21.0      7.6     12.6     16.8     16.8 orderStatus
		  | 78578.0s        0          231.4          209.8     19.9     46.1     52.4     58.7 payment
		  | 78578.0s        0           18.0           21.0     44.0     52.4     60.8     60.8 stockLevel
		  | 78579.0s        0           22.0           21.0     75.5     92.3    125.8    125.8 delivery
		  | 78579.0s        0          217.1          209.7     31.5     44.0     48.2     56.6 newOrder
		  | 78579.0s        0           23.0           21.0      6.6      7.9      9.4      9.4 orderStatus
		  | 78579.0s        0          187.1          209.8     18.9     27.3     39.8     48.2 payment
		  | 78579.0s        0           14.0           21.0     41.9     60.8     75.5     75.5 stockLevel
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

	cluster.go:2687,tpcc.go:162,tpcc.go:344,test_runner.go:767: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2675
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2683
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:162
		  | main.registerTPCC.func4
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:344
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:767
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2731
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2645
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:5652
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:191
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

More

Artifacts: /weekly/tpcc/headroom

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).weekly/tpcc/headroom failed on master@e9e372122a2e3db7090b5705da07128f828e2441:

		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 79349.0s        0           18.0           21.0     71.3    142.6    159.4    159.4 delivery
		  | 79349.0s        0          204.2          209.8     30.4     58.7     75.5     75.5 newOrder
		  | 79349.0s        0           11.0           21.0      7.9     10.5     14.7     14.7 orderStatus
		  | 79349.0s        0          219.3          209.8     17.8     44.0     54.5     71.3 payment
		  | 79349.0s        0           22.0           21.0     39.8     56.6     56.6     56.6 stockLevel
		  | 79350.0s        0           20.0           21.0     71.3    100.7    117.4    117.4 delivery
		  | 79350.0s        0          217.1          209.8     29.4     50.3     56.6     58.7 newOrder
		  | 79350.0s        0           20.0           21.0      7.1      9.4      9.4      9.4 orderStatus
		  | 79350.0s        0          233.1          209.8     16.8     32.5     39.8     58.7 payment
		  | 79350.0s        0           20.0           21.0     41.9     96.5    100.7    100.7 stockLevel
		  | 79351.0s        0           19.0           21.0     75.5    117.4    121.6    121.6 delivery
		  | 79351.0s        0          223.6          209.8     29.4     46.1     52.4     58.7 newOrder
		  | 79351.0s        0           23.0           21.0      6.8      9.4     11.5     11.5 orderStatus
		  | 79351.0s        0          202.6          209.8     16.8     31.5     37.7     39.8 payment
		  | 79351.0s        0           18.0           21.0     44.0     62.9     96.5     96.5 stockLevel
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

	cluster.go:2687,tpcc.go:162,tpcc.go:344,test_runner.go:767: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2675
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2683
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:162
		  | main.registerTPCC.func4
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:344
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:767
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2731
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2645
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:5652
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:191
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

More

Artifacts: /weekly/tpcc/headroom

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).weekly/tpcc/headroom failed on master@83e70ce84b740e27e721c3b73c38a4b8b515094a:

		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 77793.0s        0           25.0           21.0    142.6    285.2    285.2    285.2 delivery
		  | 77793.0s        0          202.0          209.7     39.8    151.0    176.2    184.5 newOrder
		  | 77793.0s        0           18.0           21.0      7.6     32.5     44.0     44.0 orderStatus
		  | 77793.0s        0          202.0          209.7     18.9     79.7    109.1    117.4 payment
		  | 77793.0s        0           18.0           21.0     46.1    109.1    142.6    142.6 stockLevel
		  | 77794.0s        0           23.0           21.0     88.1    134.2    192.9    192.9 delivery
		  | 77794.0s        0          211.6          209.7     33.6     79.7     96.5    100.7 newOrder
		  | 77794.0s        0           30.0           21.0      8.1     18.9     24.1     24.1 orderStatus
		  | 77794.0s        0          211.6          209.7     18.9     44.0     54.5     58.7 payment
		  | 77794.0s        0           23.0           21.0     44.0     75.5     83.9     83.9 stockLevel
		  | 77795.0s        0           25.0           21.0     79.7    176.2    192.9    192.9 delivery
		  | 77795.0s        0          229.3          209.7     35.7     75.5     88.1    104.9 newOrder
		  | 77795.0s        0           28.0           21.0      8.1     17.8     21.0     21.0 orderStatus
		  | 77795.0s        0          240.4          209.7     18.9     39.8     46.1     54.5 payment
		  | 77795.0s        0           14.0           21.0     46.1     71.3    100.7    100.7 stockLevel
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

	cluster.go:2687,tpcc.go:162,tpcc.go:344,test_runner.go:767: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2675
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2683
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:162
		  | main.registerTPCC.func4
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:344
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:767
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2731
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2645
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:5652
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:191
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

More

Artifacts: /weekly/tpcc/headroom

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).weekly/tpcc/headroom failed on master@6601d827b814d4e85a1081b03bf2562d8ac2a4ab:

		  | 84050.0s        0           22.0           21.0     52.4    109.1    109.1    109.1 stockLevel
		  | 84051.0s        0           22.0           21.0     96.5    285.2    352.3    352.3 delivery
		  | 84051.0s        0          212.9          209.7     41.9    176.2    209.7    243.3 newOrder
		  | 84051.0s        0           22.0           21.0      8.9     41.9     41.9     41.9 orderStatus
		  | 84051.0s        0          212.9          209.7     22.0     88.1    142.6    176.2 payment
		  | 84051.0s        0           19.0           21.0     50.3    151.0    167.8    167.8 stockLevel
		  | 84052.0s        0           15.0           21.0     92.3    125.8    142.6    142.6 delivery
		  | 84052.0s        0          195.1          209.7     35.7     62.9     71.3    100.7 newOrder
		  | 84052.0s        0           29.0           21.0      8.1     14.7     15.7     15.7 orderStatus
		  | 84052.0s        0          240.1          209.7     21.0     46.1     60.8     83.9 payment
		  | 84052.0s        0           18.0           21.0     50.3     75.5     75.5     75.5 stockLevel
		  | 84053.0s        0           20.0           21.0     88.1    151.0    151.0    151.0 delivery
		  | 84053.0s        0          210.0          209.7     37.7    100.7    117.4    125.8 newOrder
		  | 84053.0s        0           17.0           21.0      7.9     16.8     54.5     54.5 orderStatus
		  | 84053.0s        0          190.0          209.7     18.9     56.6     71.3     75.5 payment
		  | 84053.0s        0           17.0           21.0     54.5    142.6    176.2    176.2 stockLevel
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

	cluster.go:2688,tpcc.go:162,tpcc.go:344,test_runner.go:767: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2676
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2684
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:162
		  | main.registerTPCC.func4
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:344
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:767
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2732
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2646
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:5652
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:191
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

More

Artifacts: /weekly/tpcc/headroom

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).weekly/tpcc/headroom failed on master@b703e663da8ededaee2e28fc39a24e3880ae54cf:

		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 47725.0s        0            0.0           21.0      0.0      0.0      0.0      0.0 delivery
		  | 47725.0s        0            0.0          209.6      0.0      0.0      0.0      0.0 newOrder
		  | 47725.0s        0            0.0           21.0      0.0      0.0      0.0      0.0 orderStatus
		  | 47725.0s        0            0.0          209.6      0.0      0.0      0.0      0.0 payment
		  | 47725.0s        0            0.0           21.0      0.0      0.0      0.0      0.0 stockLevel
		  | 47726.0s        0            0.0           21.0      0.0      0.0      0.0      0.0 delivery
		  | 47726.0s        0            1.0          209.6  14495.5  14495.5  14495.5  14495.5 newOrder
		  | 47726.0s        0            0.0           21.0      0.0      0.0      0.0      0.0 orderStatus
		  | 47726.0s        0            2.0          209.6  14495.5  14495.5  14495.5  14495.5 payment
		  | 47726.0s        0            0.0           21.0      0.0      0.0      0.0      0.0 stockLevel
		  | 47727.0s        0           48.8           21.0  12348.0  14495.5  15032.4  15032.4 delivery
		  | 47727.0s        0          339.4          209.6  11811.2  14495.5  15032.4  15569.3 newOrder
		  | 47727.0s        0           10.0           21.0   9663.7  11811.2  11811.2  11811.2 orderStatus
		  | 47727.0s        0          119.4          209.6  10200.5  13421.8  15032.4  15032.4 payment
		  | 47727.0s        0           14.9           21.0  10200.5  13421.8  13421.8  13421.8 stockLevel
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

	cluster.go:2688,tpcc.go:162,tpcc.go:344,test_runner.go:767: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2676
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2684
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:162
		  | main.registerTPCC.func4
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:344
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:767
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2732
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2646
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:5652
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:191
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

More

Artifacts: /weekly/tpcc/headroom

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).weekly/tpcc/headroom failed on master@4b98115dfda02a9498f566958bd915c45ec7e449:

		  | 84756.0s        0           27.0           21.0     88.1    100.7    104.9    104.9 delivery
		  | 84756.0s        0          210.9          209.6     35.7     44.0     48.2     58.7 newOrder
		  | 84756.0s        0           23.0           21.0      7.6     10.5     12.1     12.1 orderStatus
		  | 84756.0s        0          225.9          209.7     21.0     27.3     31.5     41.9 payment
		  | 84756.0s        0           21.0           21.0     50.3     79.7     83.9     83.9 stockLevel
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 84757.0s        0           11.0           21.0    167.8    285.2    285.2    285.2 delivery
		  | 84757.0s        0          170.0          209.6     46.1    192.9    218.1    226.5 newOrder
		  | 84757.0s        0           20.0           21.0     10.0     39.8     39.8     39.8 orderStatus
		  | 84757.0s        0          199.0          209.7     27.3    113.2    159.4    218.1 payment
		  | 84757.0s        0           18.0           21.0     52.4    201.3    234.9    234.9 stockLevel
		  | 84758.0s        0           28.0           21.0    117.4    352.3    453.0    453.0 delivery
		  | 84758.0s        0          250.3          209.6     44.0    218.1    268.4    285.2 newOrder
		  | 84758.0s        0           17.0           21.0      8.4     10.5     12.1     12.1 orderStatus
		  | 84758.0s        0          229.3          209.7     23.1    142.6    176.2    201.3 payment
		  | 84758.0s        0           28.0           21.0     75.5    285.2    285.2    285.2 stockLevel
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

	cluster.go:2688,tpcc.go:162,tpcc.go:344,test_runner.go:767: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2676
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2684
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:162
		  | main.registerTPCC.func4
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:344
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:767
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2732
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2646
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:5652
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:191
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

More

Artifacts: /weekly/tpcc/headroom

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).weekly/tpcc/headroom failed on master@ee9f47b9ec9476a693464e2dcd09a01bf9d39ad2:

		  | 87935.0s        0           18.0           21.0     79.7    142.6    151.0    151.0 delivery
		  | 87935.0s        0          197.1          209.7     33.6     52.4     58.7     71.3 newOrder
		  | 87935.0s        0           21.0           21.0      7.9     10.5     11.0     11.0 orderStatus
		  | 87935.0s        0          200.1          209.7     17.8     31.5     37.7     46.1 payment
		  | 87935.0s        0           15.0           21.0     48.2     71.3     83.9     83.9 stockLevel
		  | 87936.0s        0           28.0           21.0     83.9    117.4    167.8    167.8 delivery
		  | 87936.0s        0          185.7          209.7     33.6     75.5     88.1    113.2 newOrder
		  | 87936.0s        0           30.9           21.0      7.3     10.0     16.8     16.8 orderStatus
		  | 87936.0s        0          196.7          209.7     19.9     50.3     62.9    121.6 payment
		  | 87936.0s        0           22.0           21.0     44.0     65.0     92.3     92.3 stockLevel
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 87937.0s        0           26.0           21.0    104.9    201.3    209.7    209.7 delivery
		  | 87937.0s        0          222.3          209.7     39.8     88.1    121.6    151.0 newOrder
		  | 87937.0s        0           20.0           21.0      8.9     17.8     23.1     23.1 orderStatus
		  | 87937.0s        0          214.3          209.7     23.1     48.2     54.5     62.9 payment
		  | 87937.0s        0           21.0           21.0     52.4     92.3    100.7    100.7 stockLevel
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

	cluster.go:2688,tpcc.go:162,tpcc.go:344,test_runner.go:768: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2676
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2684
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:162
		  | main.registerTPCC.func4
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:344
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:768
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2732
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2646
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:5652
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:191
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

More

Artifacts: /weekly/tpcc/headroom

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).weekly/tpcc/headroom failed on master@f84a9da9e62bad67c6a72d6c0717cd87e2372357:

		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 15441.0s        0           20.0           21.0     71.3    151.0    159.4    159.4 delivery
		  | 15441.0s        0          221.7          209.5     44.0     92.3    117.4    142.6 newOrder
		  | 15441.0s        0           26.0           21.0      6.8     17.8     19.9     19.9 orderStatus
		  | 15441.0s        0          211.7          209.7     28.3     88.1    109.1    113.2 payment
		  | 15441.0s        0           25.0           21.0     27.3     54.5     79.7     79.7 stockLevel
		  | 15442.0s        0           19.0           21.0     60.8     83.9     83.9     83.9 delivery
		  | 15442.0s        0          216.9          209.5     32.5     46.1     54.5     58.7 newOrder
		  | 15442.0s        0           26.0           21.0      6.6      8.9     14.2     14.2 orderStatus
		  | 15442.0s        0          222.9          209.7     19.9     30.4     41.9     62.9 payment
		  | 15442.0s        0           14.0           21.0     26.2     33.6     35.7     35.7 stockLevel
		  | 15443.0s        0           19.0           21.0     65.0     88.1     88.1     88.1 delivery
		  | 15443.0s        0          208.1          209.5     31.5     48.2     54.5     58.7 newOrder
		  | 15443.0s        0           16.0           21.0      6.0      8.9      8.9      8.9 orderStatus
		  | 15443.0s        0          206.1          209.7     21.0     32.5     37.7     48.2 payment
		  | 15443.0s        0           23.0           21.0     22.0     30.4     44.0     44.0 stockLevel
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

	cluster.go:2688,tpcc.go:162,tpcc.go:344,test_runner.go:768: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2676
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2684
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:162
		  | main.registerTPCC.func4
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:344
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:768
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2732
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2646
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:5652
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:191
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

More

Artifacts: /weekly/tpcc/headroom

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).weekly/tpcc/headroom failed on master@8b137b4f068a0d590a3e86ae91fd60eb84f2750a:

		  | 83374.0s        0           12.0           21.0    113.2    176.2    201.3    201.3 delivery
		  | 83374.0s        0          201.6          209.7     41.9     88.1    109.1    142.6 newOrder
		  | 83374.0s        0           25.0           21.0      9.4     25.2     27.3     27.3 orderStatus
		  | 83374.0s        0          207.6          209.7     26.2     58.7     71.3     79.7 payment
		  | 83374.0s        0           23.0           21.0     52.4     88.1    176.2    176.2 stockLevel
		  | 83375.0s        0           23.0           21.0    104.9    201.3    218.1    218.1 delivery
		  | 83375.0s        0          201.0          209.7     39.8    113.2    125.8    134.2 newOrder
		  | 83375.0s        0           16.0           21.0      7.9     13.6     24.1     24.1 orderStatus
		  | 83375.0s        0          229.0          209.7     22.0     60.8     75.5     79.7 payment
		  | 83375.0s        0           27.0           21.0     54.5    104.9    117.4    117.4 stockLevel
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 83376.0s        0           19.0           21.0     88.1    109.1    109.1    109.1 delivery
		  | 83376.0s        0          196.4          209.7     37.7     48.2     54.5     65.0 newOrder
		  | 83376.0s        0           22.0           21.0      7.6     10.5     11.0     11.0 orderStatus
		  | 83376.0s        0          210.4          209.7     21.0     29.4     58.7     60.8 payment
		  | 83376.0s        0           22.0           21.0     56.6     75.5     79.7     79.7 stockLevel
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

	cluster.go:2688,tpcc.go:162,tpcc.go:344,test_runner.go:768: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2676
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2684
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:162
		  | main.registerTPCC.func4
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:344
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:768
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2732
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2646
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:5652
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:191
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

More

Artifacts: /weekly/tpcc/headroom

See this test on roachdash
powered by pkg/cmd/internal/issues

@tbg
Copy link
Member

tbg commented Apr 28, 2021

Tracked in #62946

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot.
Projects
None yet
Development

No branches or pull requests

6 participants