roachtest: tpcc/mixed-headroom/n5cpu16 failed #43110

cockroach-teamcity · 2019-12-11T05:36:22Z

(roachtest).tpcc/mixed-headroom/n5cpu16 failed on provisional_201912102236_v20.1.0-alpha20191211@62640b24b4b12535600e670023e61f7b1c7309ce:

The test failed on branch=provisional_201912102236_v20.1.0-alpha20191211, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191211-1637473/tpcc/mixed-headroom/n5cpu16/run_1
	cluster.go:1783,tpcc.go:136,tpcc.go:137,tpcc.go:148,tpcc.go:261,test_runner.go:700: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1637473-1576023985-75-n5cpu16:5 -- ./workload fixtures load tpcc --warehouses=1000  {pgurl:1} returned:
		stderr:
		
		stdout:
		7  loaded 1006 KiB table district in 15.313403372s (10000 rows, 0 index entries, 66 KiB)
		I191211 05:36:05.694045 116 ccl/workloadccl/fixture.go:547  loaded 126 MiB table new_order in 44.524280847s (9000000 rows, 0 index entries, 2.8 MiB)
		Error: restoring fixture: pq: importing 1101 ranges: importing span /Table/59/1/3{5/59829-6/64449}: adding to batch: /Table/61/1/36/12089/0/1558478069.788080604,0 -> /TUPLE/3:3:Int/67/1:4:Bytes/S05Ueb5SQ4eZx7YRnDfml22y/1:5:Bytes/IMMuIvSzug6DnmkAvJ1rrKnb/1:6:Bytes/lzf0XDDqcglRKdMKN2Q5j2Fg/1:7:Bytes/L5t5Zty0kHBjPGEHONstSot9/1:8:Bytes/7sLX3NfvEb6hzAZ8UnysTftc/1:9:Bytes/sc4QbcDc3xfbsODqBcHwfUCq/1:10:Bytes/BT25vtCZn6e5hpX85CArH8tz/1:11:Bytes/d0KldOHZnZz9nQS12lsmAUyG/1:12:Bytes/bzWd4uHxP2fCjPmfIYk0XeMI/1:13:Bytes/FAnpxJLxivIsLPUroO2exIqE/1:14:Int/0/1:15:Int/0/1:16:Int/0/1:17:Bytes/ZTbOMhxdOVTw9ouSCbGgbwlOArCaB7YY450jEubHt: addsstable [/Table/61/1/35/59829/0,/Table/61/1/36/12088/0/NULL): closing sstable iterator: leveldb/table: invalid table (bad magic number)
		Error:  exit status 1
		: exit status 1

details

Artifacts: /tpcc/mixed-headroom/n5cpu16

make stressrace TESTS=tpcc/mixed-headroom/n5cpu16 PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS=-timeout 5m' 2>&1

_{powered by pkg/cmd/internal/issues}

The text was updated successfully, but these errors were encountered:

cockroach-teamcity · 2019-12-19T03:27:05Z

(roachtest).tpcc/mixed-headroom/n5cpu16 failed on master@2f1e342c386973c35246bef68c177fcd0b8b609b:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191218-1649932/tpcc/mixed-headroom/n5cpu16/run_1
	cluster.go:1786,tpcc.go:157,cluster.go:2145,errgroup.go:57: error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:399
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1820
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1784
		    main.runTPCC.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:157
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2145
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1337
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1649932-1576655788-79-n5cpu16:5 -- ./workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=2h0m0s {pgurl:1-4} returned:
		    stderr:
		    
		    stdout:
		    Initializing 1818 connections...
		    Error: EOF
		    Error:  exit status 1:
		  - exit status 1

Repro

Artifacts: /tpcc/mixed-headroom/n5cpu16

make stressrace TESTS=tpcc/mixed-headroom/n5cpu16 PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2019-12-20T03:15:11Z

(roachtest).tpcc/mixed-headroom/n5cpu16 failed on master@277e28b2ea78929563e85cb1c9efc573f37408a4:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191219-1653424/tpcc/mixed-headroom/n5cpu16/run_1
	cluster.go:1786,tpcc.go:157,cluster.go:2145,errgroup.go:57: error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:399
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1820
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1784
		    main.runTPCC.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:157
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2145
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1337
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1653424-1576741476-84-n5cpu16:5 -- ./workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=2h0m0s {pgurl:1-4} returned:
		    stderr:
		    
		    stdout:
		    Initializing 1818 connections...:
		  - context canceled

Repro

Artifacts: /tpcc/mixed-headroom/n5cpu16

make stressrace TESTS=tpcc/mixed-headroom/n5cpu16 PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2019-12-22T03:15:27Z

(roachtest).tpcc/mixed-headroom/n5cpu16 failed on master@dd9b1c1f40dda59ee9d446416106d311ae5ce1e6:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191221-1658785/tpcc/mixed-headroom/n5cpu16/run_1
	cluster.go:1786,tpcc.go:157,cluster.go:2145,errgroup.go:57: error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:399
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1820
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1784
		    main.runTPCC.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:157
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2145
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1658785-1576914291-84-n5cpu16:5 -- ./workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=2h0m0s {pgurl:1-4} returned:
		    stderr:
		    
		    stdout:
		    Initializing 1818 connections...
		    Error: EOF
		    Error:  exit status 1:
		  - exit status 1

Repro

Artifacts: /tpcc/mixed-headroom/n5cpu16

make stressrace TESTS=tpcc/mixed-headroom/n5cpu16 PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2019-12-23T03:14:39Z

(roachtest).tpcc/mixed-headroom/n5cpu16 failed on master@319080c701cb4e13c347e86c4afb2f1c2af78def:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191222-1660313/tpcc/mixed-headroom/n5cpu16/run_1
	cluster.go:1786,tpcc.go:157,cluster.go:2145,errgroup.go:57: error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:399
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1820
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1784
		    main.runTPCC.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:157
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2145
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1660313-1577000643-84-n5cpu16:5 -- ./workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=2h0m0s {pgurl:1-4} returned:
		    stderr:
		    
		    stdout:
		    Initializing 1818 connections...
		    Error: EOF
		    Error:  exit status 1:
		  - exit status 1

Repro

Artifacts: /tpcc/mixed-headroom/n5cpu16

make stressrace TESTS=tpcc/mixed-headroom/n5cpu16 PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2019-12-25T03:12:46Z

(roachtest).tpcc/mixed-headroom/n5cpu16 failed on master@a3f81bcbef596f8b8134641526a87f27c4ad8082:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191224-1661395/tpcc/mixed-headroom/n5cpu16/run_1
	cluster.go:1786,tpcc.go:157,cluster.go:2145,errgroup.go:57: error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:399
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1820
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1784
		    main.runTPCC.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:157
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2145
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1661395-1577173331-88-n5cpu16:5 -- ./workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=2h0m0s {pgurl:1-4} returned:
		    stderr:
		    
		    stdout:
		    Initializing 1818 connections...
		    Error: EOF
		    Error:  exit status 1:
		  - exit status 1

Repro

Artifacts: /tpcc/mixed-headroom/n5cpu16

make stressrace TESTS=tpcc/mixed-headroom/n5cpu16 PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2019-12-26T03:13:00Z

(roachtest).tpcc/mixed-headroom/n5cpu16 failed on master@c172b37ec3316ec982e1ee92d4414a76823680e6:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191225-1662271/tpcc/mixed-headroom/n5cpu16/run_1
	cluster.go:1786,tpcc.go:157,cluster.go:2145,errgroup.go:57: error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:399
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1820
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1784
		    main.runTPCC.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:157
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2145
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1662271-1577259744-84-n5cpu16:5 -- ./workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=2h0m0s {pgurl:1-4} returned:
		    stderr:
		    
		    stdout:
		    Initializing 1818 connections...
		    Error: EOF
		    Error:  exit status 1:
		  - exit status 1

Repro

Artifacts: /tpcc/mixed-headroom/n5cpu16

make stressrace TESTS=tpcc/mixed-headroom/n5cpu16 PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2019-12-27T03:06:03Z

(roachtest).tpcc/mixed-headroom/n5cpu16 failed on master@2ddb82b2a8fff0c4e42169cf4e610d33a0fbd2d3:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191226-1663134/tpcc/mixed-headroom/n5cpu16/run_1
	cluster.go:1786,tpcc.go:157,cluster.go:2145,errgroup.go:57: error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:399
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1820
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1784
		    main.runTPCC.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:157
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2145
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1663134-1577345728-85-n5cpu16:5 -- ./workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=2h0m0s {pgurl:1-4} returned:
		    stderr:
		    
		    stdout:
		    Initializing 1818 connections...
		    Error: EOF
		    Error:  exit status 1:
		  - exit status 1

Repro

Artifacts: /tpcc/mixed-headroom/n5cpu16

make stressrace TESTS=tpcc/mixed-headroom/n5cpu16 PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2019-12-28T03:14:53Z

(roachtest).tpcc/mixed-headroom/n5cpu16 failed on master@d54e001248f233734ff926daef5470487c5616b0:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191227-1664318/tpcc/mixed-headroom/n5cpu16/run_1
	cluster.go:1786,tpcc.go:157,cluster.go:2145,errgroup.go:57: error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:399
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1820
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1784
		    main.runTPCC.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:157
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2145
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1664318-1577432658-85-n5cpu16:5 -- ./workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=2h0m0s {pgurl:1-4} returned:
		    stderr:
		    
		    stdout:
		    Initializing 1818 connections...:
		  - context canceled

Repro

Artifacts: /tpcc/mixed-headroom/n5cpu16

make stressrace TESTS=tpcc/mixed-headroom/n5cpu16 PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2019-12-29T03:23:23Z

(roachtest).tpcc/mixed-headroom/n5cpu16 failed on master@8f946b8bb62629002c5e958373b81ca9e1920dd6:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191228-1665456/tpcc/mixed-headroom/n5cpu16/run_1
	cluster.go:1786,tpcc.go:157,cluster.go:2145,errgroup.go:57: error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:399
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1820
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1784
		    main.runTPCC.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:157
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2145
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1665456-1577519567-88-n5cpu16:5 -- ./workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=2h0m0s {pgurl:1-4} returned:
		    stderr:
		    
		    stdout:
		    Initializing 1818 connections...:
		  - context canceled

Repro

Artifacts: /tpcc/mixed-headroom/n5cpu16

make stressrace TESTS=tpcc/mixed-headroom/n5cpu16 PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2019-12-30T03:14:06Z

(roachtest).tpcc/mixed-headroom/n5cpu16 failed on master@edd46b1405b7928760b65ec8aad59fd22a8adb6b:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191229-1666388/tpcc/mixed-headroom/n5cpu16/run_1
	cluster.go:1786,tpcc.go:157,cluster.go:2145,errgroup.go:57: error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:399
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1820
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1784
		    main.runTPCC.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:157
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2145
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1666388-1577605410-90-n5cpu16:5 -- ./workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=2h0m0s {pgurl:1-4} returned:
		    stderr:
		    
		    stdout:
		    Initializing 1818 connections...
		    Error: EOF
		    Error:  exit status 1:
		  - exit status 1

Repro

Artifacts: /tpcc/mixed-headroom/n5cpu16

make stressrace TESTS=tpcc/mixed-headroom/n5cpu16 PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-01-01T03:03:46Z

(roachtest).tpcc/mixed-headroom/n5cpu16 failed on master@f1f15250871d124178f40bf340ef48719b50e7e5:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191231-1668189/tpcc/mixed-headroom/n5cpu16/run_1
	cluster.go:1809,tpcc.go:157,cluster.go:2168,errgroup.go:57: error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:399
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1843
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1807
		    main.runTPCC.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:157
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2168
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1668189-1577777590-88-n5cpu16:5 -- ./workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=2h0m0s {pgurl:1-4} returned:
		    stderr:
		    
		    stdout:
		    Initializing 1818 connections...:
		  - context canceled

Repro

Artifacts: /tpcc/mixed-headroom/n5cpu16

make stressrace TESTS=tpcc/mixed-headroom/n5cpu16 PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-01-02T03:14:37Z

(roachtest).tpcc/mixed-headroom/n5cpu16 failed on master@0d1ae9cbc49518180e68aa0bfbf982d73854ddae:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200101-1669512/tpcc/mixed-headroom/n5cpu16/run_1
	cluster.go:1809,tpcc.go:157,cluster.go:2168,errgroup.go:57: error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:399
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1843
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1807
		    main.runTPCC.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:157
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2168
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1669512-1577864642-85-n5cpu16:5 -- ./workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=2h0m0s {pgurl:1-4} returned:
		    stderr:
		    
		    stdout:
		    Initializing 1818 connections...
		    Error: EOF
		    Error:  exit status 1:
		  - exit status 1

Repro

Artifacts: /tpcc/mixed-headroom/n5cpu16

make stressrace TESTS=tpcc/mixed-headroom/n5cpu16 PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-01-03T03:14:43Z

(roachtest).tpcc/mixed-headroom/n5cpu16 failed on master@a99740bb34c03ab3af7558129edce9c17c88d4f9:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200102-1670273/tpcc/mixed-headroom/n5cpu16/run_1
	cluster.go:1809,tpcc.go:157,cluster.go:2168,errgroup.go:57: error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:399
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1843
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1807
		    main.runTPCC.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:157
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2168
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1670273-1577951048-85-n5cpu16:5 -- ./workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=2h0m0s {pgurl:1-4} returned:
		    stderr:
		    
		    stdout:
		    Initializing 1818 connections...:
		  - context canceled

Repro

Artifacts: /tpcc/mixed-headroom/n5cpu16

make stressrace TESTS=tpcc/mixed-headroom/n5cpu16 PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-01-04T03:25:03Z

(roachtest).tpcc/mixed-headroom/n5cpu16 failed on master@8facb469d66426a3f372a1aed4ea6f34e287b501:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200103-1671733/tpcc/mixed-headroom/n5cpu16/run_1
	cluster.go:1809,tpcc.go:157,cluster.go:2168,errgroup.go:57: error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:399
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1843
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1807
		    main.runTPCC.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:157
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2168
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1671733-1578038067-87-n5cpu16:5 -- ./workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=2h0m0s {pgurl:1-4} returned:
		    stderr:
		    
		    stdout:
		    Initializing 1818 connections...:
		  - context canceled

Repro

Artifacts: /tpcc/mixed-headroom/n5cpu16

make stressrace TESTS=tpcc/mixed-headroom/n5cpu16 PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-01-05T03:15:07Z

(roachtest).tpcc/mixed-headroom/n5cpu16 failed on master@7906cbe3d9c615430f6e892a42e6bccdad2aff00:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200104-1673622/tpcc/mixed-headroom/n5cpu16/run_1
	cluster.go:1809,tpcc.go:157,cluster.go:2168,errgroup.go:57: error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:399
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1843
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1807
		    main.runTPCC.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:157
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2168
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1673622-1578123870-87-n5cpu16:5 -- ./workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=2h0m0s {pgurl:1-4} returned:
		    stderr:
		    
		    stdout:
		    Initializing 1818 connections...
		    Error: EOF
		    Error:  exit status 1:
		  - exit status 1

Repro

Artifacts: /tpcc/mixed-headroom/n5cpu16

make stressrace TESTS=tpcc/mixed-headroom/n5cpu16 PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-01-07T02:06:43Z

(roachtest).tpcc/mixed-headroom/n5cpu16 failed on master@f180501b7c2ccb946d7f7c8944d5509b87cb0394:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200106-1674751/tpcc/mixed-headroom/n5cpu16/run_1
	cluster.go:1809,tpcc.go:157,cluster.go:2168,errgroup.go:57: error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:399
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1843
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1807
		    main.runTPCC.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:157
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2168
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1674751-1578297236-87-n5cpu16:5 -- ./workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=2h0m0s {pgurl:1-4} returned:
		    stderr:
		    
		    stdout:
		    Initializing 1818 connections...
		    Error: EOF
		    Error:  exit status 1:
		  - exit status 1

Repro

Artifacts: /tpcc/mixed-headroom/n5cpu16

make stressrace TESTS=tpcc/mixed-headroom/n5cpu16 PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-01-07T15:06:41Z

(roachtest).tpcc/mixed-headroom/n5cpu16 failed on 43008@b555d16b4455f7122f4211082512820742f8116f:

The test failed on branch=43008, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200106-1676232/tpcc/mixed-headroom/n5cpu16/run_1
	cluster.go:1809,tpcc.go:157,cluster.go:2168,errgroup.go:57: error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:399
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1843
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1807
		    main.runTPCC.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:157
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2168
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1676232-1578345768-84-n5cpu16:5 -- ./workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=2h0m0s {pgurl:1-4} returned:
		    stderr:
		    
		    stdout:
		    Initializing 1818 connections...:
		  - context canceled

Repro

Artifacts: /tpcc/mixed-headroom/n5cpu16

make stressrace TESTS=tpcc/mixed-headroom/n5cpu16 PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-01-08T02:06:47Z

(roachtest).tpcc/mixed-headroom/n5cpu16 failed on master@52261c39c65aecf960e3c8d54089a66d378af610:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200107-1676715/tpcc/mixed-headroom/n5cpu16/run_1
	cluster.go:1809,tpcc.go:157,cluster.go:2168,errgroup.go:57: error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:399
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1843
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1807
		    main.runTPCC.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:157
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2168
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1676715-1578383325-86-n5cpu16:5 -- ./workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=2h0m0s {pgurl:1-4} returned:
		    stderr:
		    
		    stdout:
		    Initializing 1818 connections...
		    Error: EOF
		    Error:  exit status 1:
		  - exit status 1

Repro

Artifacts: /tpcc/mixed-headroom/n5cpu16

make stressrace TESTS=tpcc/mixed-headroom/n5cpu16 PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-01-16T17:10:48Z

(roachtest).tpcc/mixed-headroom/n5cpu16 failed on master@e2c2be6cbef07c7f2ee6341816f81f72a206c32a:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200116-1693614/tpcc/mixed-headroom/n5cpu16/run_1
	cluster.go:1821,tpcc.go:169,tpcc.go:261,test_runner.go:716: error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:399
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1855
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1819
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:169
		    main.registerTPCC.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:261
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:716
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1693614-1579160767-82-n5cpu16:5 -- ./workload check tpcc --warehouses=909 {pgurl:1} returned:
		    stderr:
		    
		    stdout:
		    I200116 17:10:21.687209 1 workload/tpcc/tpcc.go:372  check 3.3.2.1 took 127.976672ms
		    Error: check failed: 3.3.2.1: 58 rows returned, expected zero
		    Error:  exit status 1:
		  - exit status 1

Repro

Artifacts: /tpcc/mixed-headroom/n5cpu16

make stressrace TESTS=tpcc/mixed-headroom/n5cpu16 PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-01-17T14:10:13Z

(roachtest).tpcc/mixed-headroom/n5cpu16 failed on master@96d7e50acfaa63b5b60addd7cec8e750a4a4d333:

		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1695913-1579247026-83-n5cpu16:5 -- ./workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=2h0m0s {pgurl:1-4} returned:
		    stderr:
		    
		    stdout:
		    3 stockLevel
		      232.0s        0           20.0            8.2     41.9     71.3     79.7     79.7 delivery
		      232.0s        0          141.9           72.3     30.4     41.9     52.4     67.1 newOrder
		      232.0s        0           20.0            8.1      5.8     10.5     12.1     12.1 orderStatus
		      232.0s        0          141.0           79.8     14.7     31.5     37.7     41.9 payment
		      232.0s        0           11.0            8.0     35.7     67.1     79.7     79.7 stockLevel
		    _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		      233.0s        0           13.0            8.2     46.1     62.9     79.7     79.7 delivery
		      233.0s        0          172.1           72.7     29.4     41.9     44.0     56.6 newOrder
		      233.0s        0           12.0            8.2      6.3      7.1      8.1      8.1 orderStatus
		      233.0s        0          173.1           80.2     14.7     21.0     26.2     32.5 payment
		      233.0s        0           19.0            8.1     30.4     60.8     79.7     79.7 stockLevel
		      234.0s        0           23.0            8.2     48.2     60.8     75.5     75.5 delivery
		      234.0s        0          164.0           73.1     28.3     41.9     48.2     56.6 newOrder
		      234.0s        0           17.0            8.2      6.0      9.4     16.8     16.8 orderStatus
		      234.0s        0          175.0           80.6     15.7     24.1     29.4     41.9 payment
		      234.0s        0           15.0            8.1     39.8     92.3     96.5     96.5 stockLevel
		      235.0s        0           13.0            8.3     41.9     48.2     50.3     50.3 delivery
		      235.0s        0          146.0           73.4     29.4     41.9     46.1     48.2 newOrder
		      235.0s        0           20.0            8.2      6.0     12.6     14.7     14.7 orderStatus
		      235.0s        0          154.0           80.9     14.2     23.1     32.5     37.7 payment
		      235.0s        0           15.0            8.1     39.8     65.0    104.9    104.9 stockLevel
		      236.0s        0           15.0            8.3     46.1     54.5     54.5     54.5 delivery
		      236.0s        0          157.0           73.8     28.3     46.1     52.4     67.1 newOrder
		      236.0s        0           18.0            8.3      5.8     12.1     13.6     13.6 orderStatus
		      236.0s        0          150.0           81.2     14.7     28.3     30.4     37.7 payment
		      236.0s        0           16.0            8.2     31.5     44.0     67.1     67.1 stockLevel
		    _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		      237.0s        0           17.0            8.3     48.2     52.4     54.5     54.5 delivery
		      237.0s        0          160.0           74.1     27.3     39.8     44.0     50.3 newOrder
		      237.0s        0           22.0            8.3      5.5     10.0     13.1     13.1 orderStatus
		      237.0s        0          140.0           81.5     13.6     21.0     23.1     23.1 payment
		      237.0s        0           20.0            8.2     30.4     79.7     83.9     83.9 stockLevel
		      238.0s        0           11.0            8.3     46.1     58.7     62.9     62.9 delivery
		      238.0s        0          145.0           74.4     29.4     44.0     52.4     52.4 newOrder
		      238.0s        0           12.0            8.4      6.3     14.2     16.8     16.8 orderStatus
		      238.0s        0          177.0           81.9     15.2     24.1     30.4     39.8 payment
		      238.0s        0           12.0            8.2     35.7     83.9    104.9    104.9 stockLevel
		      239.0s        0           19.0            8.4     46.1     75.5     75.5     75.5 delivery
		      239.0s        0          143.0           74.7     28.3     39.8     44.0     46.1 newOrder
		      239.0s        0           15.0            8.4      6.8     16.8     18.9     18.9 orderStatus
		      239.0s        0          165.0           82.2     13.6     19.9     22.0     26.2 payment
		      239.0s        0           15.0            8.3     48.2     88.1    104.9    104.9 stockLevel
		    Error: error in newOrder: ERROR: duplicate key value (o_w_id,o_d_id,o_id)=(700,3,3003) violates unique constraint "primary" (SQLSTATE 23505)
		    Error:  exit status 1:
		  - exit status 1

Repro

Artifacts: /tpcc/mixed-headroom/n5cpu16

make stressrace TESTS=tpcc/mixed-headroom/n5cpu16 PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-01-19T14:53:04Z

(roachtest).tpcc/mixed-headroom/n5cpu16 failed on master@3bb183403b4bc75c25fbcfc69d7d35de76d2b984:

		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1699769-1579420121-88-n5cpu16:5 -- ./workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=2h0m0s {pgurl:1-4} returned:
		    stderr:
		    
		    stdout:
		    2 stockLevel
		     1706.0s        0           17.0           19.0     52.4     75.5     79.7     79.7 delivery
		     1706.0s        0          214.0          189.1     31.5     50.3     65.0     79.7 newOrder
		     1706.0s        0           17.0           19.0      7.9     13.1     15.2     15.2 orderStatus
		     1706.0s        0          179.0          190.2     16.3     28.3     32.5     46.1 payment
		     1706.0s        0           14.0           19.0     25.2     50.3     71.3     71.3 stockLevel
		     1707.0s        0           12.0           19.0     50.3     58.7     75.5     75.5 delivery
		     1707.0s        0          207.1          189.1     31.5     52.4     60.8     83.9 newOrder
		     1707.0s        0           22.0           19.0      7.1     13.1     16.8     16.8 orderStatus
		     1707.0s        0          192.1          190.2     16.3     31.5     46.1     56.6 payment
		     1707.0s        0           25.0           19.0     27.3     60.8     79.7     79.7 stockLevel
		     1708.0s        0           20.0           19.0     52.4     67.1     92.3     92.3 delivery
		     1708.0s        0          192.7          189.1     30.4     48.2     56.6     67.1 newOrder
		     1708.0s        0           12.0           19.0      6.8     12.1     12.1     12.1 orderStatus
		     1708.0s        0          193.7          190.2     16.3     30.4     39.8     41.9 payment
		     1708.0s        0           22.0           19.0     27.3     52.4     58.7     58.7 stockLevel
		    _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		     1709.0s        0           16.0           19.0     50.3     71.3     92.3     92.3 delivery
		     1709.0s        0          202.3          189.1     32.5     60.8     75.5     83.9 newOrder
		     1709.0s        0           23.0           19.0      6.8     16.8     19.9     19.9 orderStatus
		     1709.0s        0          199.3          190.3     15.7     41.9     52.4     58.7 payment
		     1709.0s        0           13.0           19.0     37.7     50.3     71.3     71.3 stockLevel
		     1710.0s        0           20.0           19.0     48.2     65.0     75.5     75.5 delivery
		     1710.0s        0          195.7          189.1     29.4     44.0     62.9     71.3 newOrder
		     1710.0s        0           21.0           19.0      6.8     12.6     14.2     14.2 orderStatus
		     1710.0s        0          187.7          190.3     15.2     25.2     37.7     52.4 payment
		     1710.0s        0           16.0           19.0     28.3     67.1     92.3     92.3 stockLevel
		     1711.0s        0           20.0           19.0     48.2     58.7     62.9     62.9 delivery
		     1711.0s        0          181.3          189.1     30.4     46.1     56.6     75.5 newOrder
		     1711.0s        0           17.0           19.0      7.1     12.6     14.2     14.2 orderStatus
		     1711.0s        0          183.3          190.2     16.8     30.4     37.7     39.8 payment
		     1711.0s        0           10.0           18.9     25.2     54.5     54.5     54.5 stockLevel
		     1712.0s        0           24.0           19.0     50.3     79.7     88.1     88.1 delivery
		     1712.0s        0          191.7          189.1     30.4     60.8     79.7     83.9 newOrder
		     1712.0s        0           17.0           19.0      6.8     12.1     14.7     14.7 orderStatus
		     1712.0s        0          184.7          190.2     16.3     37.7     50.3     65.0 payment
		     1712.0s        0           24.0           19.0     32.5     62.9     96.5     96.5 stockLevel
		    _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		     1713.0s        0           17.0           19.0     50.3     58.7     67.1     67.1 delivery
		     1713.0s        0          168.0          189.1     28.3     41.9     50.3     56.6 newOrder
		     1713.0s        0           26.0           19.0      6.6     10.5     11.0     11.0 orderStatus
		     1713.0s        0          213.0          190.3     15.2     25.2     37.7     58.7 payment
		     1713.0s        0           18.0           19.0     27.3     46.1     60.8     60.8 stockLevel
		    Error: error in newOrder: ERROR: duplicate key value (o_w_id,o_d_id,o_id)=(882,4,3027) violates unique constraint "primary" (SQLSTATE 23505)
		    Error:  exit status 1:
		  - exit status 1

Repro

Artifacts: /tpcc/mixed-headroom/n5cpu16
roachdash

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-01-22T04:18:44Z

(roachtest).tpcc/mixed-headroom/n5cpu16 failed on master@a5b42be47cf47206d2343f40bbaf445f387f3890:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200121-1702345/tpcc/mixed-headroom/n5cpu16/run_1
	cluster.go:1821,tpcc.go:169,tpcc.go:261,test_runner.go:716: error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:399
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1855
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1819
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:169
		    main.registerTPCC.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:261
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:716
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1702345-1579636752-84-n5cpu16:5 -- ./workload check tpcc --warehouses=909 {pgurl:1} returned:
		    stderr:
		    
		    stdout:
		    I200122 04:18:19.939666 1 workload/tpcc/tpcc.go:372  check 3.3.2.1 took 152.532726ms
		    Error: check failed: 3.3.2.1: 74 rows returned, expected zero
		    Error:  exit status 1:
		  - exit status 1

Repro

Artifacts: /tpcc/mixed-headroom/n5cpu16
roachdash

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-01-22T14:57:14Z

(roachtest).tpcc/mixed-headroom/n5cpu16 failed on master@c76ad970e73e606bc55a372d93f9d2d6acb32c9c:

		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1703406-1579679704-81-n5cpu16:5 -- ./workload run tpcc --warehouses=909 --histograms=perf/stats.json  --ramp=5m0s --duration=2h0m0s {pgurl:1-4} returned:
		    stderr:
		    
		    stdout:
		     stockLevel
		     2926.0s        0           24.0           19.0     50.3     71.3     79.7     79.7 delivery
		     2926.0s        0          201.7          189.5     32.5     50.3     67.1     83.9 newOrder
		     2926.0s        0           15.0           19.1      7.1     13.6     15.2     15.2 orderStatus
		     2926.0s        0          195.7          190.3     16.3     28.3     35.7     41.9 payment
		     2926.0s        0           22.0           19.0     50.3     79.7     96.5     96.5 stockLevel
		     2927.0s        0           16.0           19.0     54.5     67.1    104.9    104.9 delivery
		     2927.0s        0          183.0          189.5     35.7     65.0     75.5     88.1 newOrder
		     2927.0s        0           18.0           19.1      7.1     13.6     14.7     14.7 orderStatus
		     2927.0s        0          202.0          190.3     16.8     35.7     50.3     71.3 payment
		     2927.0s        0           23.0           19.0     50.3     88.1     92.3     92.3 stockLevel
		     2928.0s        0           15.0           19.0     48.2     56.6     58.7     58.7 delivery
		     2928.0s        0          197.4          189.5     30.4     48.2     52.4     60.8 newOrder
		     2928.0s        0           14.0           19.1      5.8     10.0     10.5     10.5 orderStatus
		     2928.0s        0          178.3          190.3     15.7     23.1     29.4     35.7 payment
		     2928.0s        0           22.0           19.0     39.8     71.3     79.7     79.7 stockLevel
		    _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		     2929.0s        0           22.0           19.0     52.4     71.3     79.7     79.7 delivery
		     2929.0s        0          198.1          189.5     30.4     54.5     75.5     88.1 newOrder
		     2929.0s        0           17.0           19.1      6.8      8.4      9.4      9.4 orderStatus
		     2929.0s        0          189.1          190.3     16.3     33.6     56.6     60.8 payment
		     2929.0s        0           24.0           19.0     46.1     96.5    100.7    100.7 stockLevel
		     2930.0s        0           23.0           19.0     50.3     71.3     88.1     88.1 delivery
		     2930.0s        0          176.7          189.5     33.6     56.6     79.7     83.9 newOrder
		     2930.0s        0           22.0           19.1      7.1     11.0     13.1     13.1 orderStatus
		     2930.0s        0          206.7          190.3     16.8     35.7     41.9     44.0 payment
		     2930.0s        0           22.0           19.0     39.8     71.3     75.5     75.5 stockLevel
		     2931.0s        0           17.0           19.0     48.2     75.5     75.5     75.5 delivery
		     2931.0s        0          200.3          189.5     31.5     46.1     52.4     54.5 newOrder
		     2931.0s        0           11.0           19.0      7.1      8.9     11.0     11.0 orderStatus
		     2931.0s        0          182.2          190.3     15.7     25.2     28.3     30.4 payment
		     2931.0s        0           18.0           19.0     41.9     65.0     71.3     71.3 stockLevel
		     2932.0s        0           22.0           19.0     54.5     79.7    100.7    100.7 delivery
		     2932.0s        0          184.8          189.5     35.7     67.1     83.9     96.5 newOrder
		     2932.0s        0           19.0           19.0      7.9     17.8     27.3     27.3 orderStatus
		     2932.0s        0          200.7          190.3     16.8     28.3     41.9     52.4 payment
		     2932.0s        0           18.0           19.0     41.9     71.3     75.5     75.5 stockLevel
		    _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		     2933.0s        0           15.0           19.0     50.3     67.1     75.5     75.5 delivery
		     2933.0s        0          185.9          189.5     32.5     52.4     60.8     67.1 newOrder
		     2933.0s        0           27.0           19.1      7.1     15.7     16.8     16.8 orderStatus
		     2933.0s        0          182.9          190.3     16.3     26.2     30.4     35.7 payment
		     2933.0s        0           17.0           19.0     33.6     54.5     75.5     75.5 stockLevel
		    Error: error in newOrder: ERROR: duplicate key value (o_w_id,o_d_id,o_id)=(144,10,3059) violates unique constraint "primary" (SQLSTATE 23505)
		    Error:  exit status 1:
		  - exit status 1

Repro

Artifacts: /tpcc/mixed-headroom/n5cpu16
roachdash

_{powered by pkg/cmd/internal/issues}

tbg · 2020-01-23T10:37:50Z

@asubiotto there are some concerning failures here:

Error: error in newOrder: ERROR: duplicate key value (o_w_id,o_d_id,o_id)=(144,10,3059) violates unique constraint "primary" (SQLSTATE 23505)
Error: exit status 1:

but also from one higher up:

Error: check failed: 3.3.2.1: 74 rows returned, expected zero

There are some similar failures above, the first one on 1/16: #43110 (comment)

cc @nvanbenschoten

Looks like we ought to stress this baby

tbg · 2020-01-23T20:46:06Z

If I had to take a shot in the dark about the cause of this, based on these symptoms and the timeline of this failure, I'd say there's a decent chance this is related to #42969. I'd try running before and after that change and see whether we can pinpoint it. @tbg you have this running on master right now, correct?

No I'm actually had just a few failed attempts at running this today (nothing interesting.. just hard to build the right binaries and my gceworker is old enough to have mismatching kernel versions, etc etc)
I'll make sure to run around this SHA, right now I picked the first one from a week ago that had this failure mode.
I'm also heading out now so if you want to beat me to it you've got a pretty good shot!

edit: my overnight run is http://35.229.40.30:8080/

tbg · 2020-01-24T09:09:18Z

They all failed exactly with these errors above:

==> artifacts/tpcc/mixed-headroom/n5cpu16/run_1/test.log <==
                    I200123 23:32:25.302426 1 workload/tpcc/tpcc.go:372  check 3.3.2.1 took 136.317768ms
                    Error: check failed: 3.3.2.1: 84 rows returned, expected zero
                    Error:  exit status 1:
                  - exit status 1


==> artifacts/tpcc/mixed-headroom/n5cpu16/run_10/test.log <==
                    I200123 23:29:05.879142 1 workload/tpcc/tpcc.go:372  check 3.3.2.1 took 134.526421ms
                    Error: check failed: 3.3.2.1: 75 rows returned, expected zero
                    Error:  exit status 1:
                  - exit status 1


==> artifacts/tpcc/mixed-headroom/n5cpu16/run_2/test.log <==
                    I200123 23:31:33.865297 1 workload/tpcc/tpcc.go:372  check 3.3.2.1 took 127.072871ms
                    Error: check failed: 3.3.2.1: 67 rows returned, expected zero
                    Error:  exit status 1:
                  - exit status 1


==> artifacts/tpcc/mixed-headroom/n5cpu16/run_3/test.log <==
                      186.0s        0           22.0           18.3     37.7     58.7     71.3     71.3 stockLevel
                    Error: error in newOrder: ERROR: duplicate key value (o_w_id,o_d_id,o_id)=(720,2,3005) violates unique constraint "primary" (SQLSTATE 23505)
                    Error:  exit status 1:
                  - exit status 1


==> artifacts/tpcc/mixed-headroom/n5cpu16/run_4/test.log <==
                      363.0s        0           16.0           19.0     29.4     58.7     62.9     62.9 stockLevel
                    Error: error in newOrder: ERROR: duplicate key value (o_w_id,o_d_id,o_id)=(330,4,3013) violates unique constraint "primary" (SQLSTATE 23505)
                    Error:  exit status 1:
                  - exit status 1


==> artifacts/tpcc/mixed-headroom/n5cpu16/run_5/test.log <==
                       36.0s        0           20.0           16.6     44.0     71.3    100.7    100.7 stockLevel
                    Error: error in newOrder: ERROR: duplicate key value (o_w_id,o_d_id,o_id)=(36,1,3005) violates unique constraint "primary" (SQLSTATE 23505)
                    Error:  exit status 1:
                  - exit status 1


==> artifacts/tpcc/mixed-headroom/n5cpu16/run_6/test.log <==
                      772.0s        0           27.0           19.0     29.4     56.6     96.5     96.5 stockLevel
                    Error: error in newOrder: ERROR: duplicate key value (o_w_id,o_d_id,o_id)=(376,9,3023) violates unique constraint "primary" (SQLSTATE 23505)
                    Error:  exit status 1:
                  - exit status 1


==> artifacts/tpcc/mixed-headroom/n5cpu16/run_7/test.log <==
                    I200123 23:26:49.085837 1 workload/tpcc/tpcc.go:372  check 3.3.2.1 took 163.63256ms
                    Error: check failed: 3.3.2.1: 75 rows returned, expected zero
                    Error:  exit status 1:
                  - exit status 1


==> artifacts/tpcc/mixed-headroom/n5cpu16/run_8/test.log <==
                    I200123 23:33:42.935959 1 workload/tpcc/tpcc.go:372  check 3.3.2.1 took 138.020407ms
                    Error: check failed: 3.3.2.1: 70 rows returned, expected zero
                    Error:  exit status 1:
                  - exit status 1


==> artifacts/tpcc/mixed-headroom/n5cpu16/run_9/test.log <==
                      273.0s        0           19.0           18.7     30.4     58.7     71.3     71.3 stockLevel
                    Error: error in newOrder: ERROR: duplicate key value (o_w_id,o_d_id,o_id)=(82,4,3013) violates unique constraint "primary" (SQLSTATE 23505)
                    Error:  exit status 1:
                  - exit status 1

This was at e2c2be6. I will now try the parent of 379fc8f (the commit containing #42969).

tbg · 2020-01-24T12:13:27Z

One out of five runs is still outstanding, but four already passed. Pretty sure pre-Andrei it's not broken (sorry buddy)

Going to try 379fc8f now.

tbg · 2020-01-24T13:18:51Z

Yeah first one already failed:

ERROR: duplicate key value (o_w_id,o_d_id,o_id)=(128,3,3010) violates unique constraint "primary" (SQLSTATE 23505)

@andreimatei that seems to make it official. Note that there were other commits in that merge, but of all of them only #43989 remotely has the potential to have caused this bug, and I really don't think that one's it.

I'm out for the rest of the day, so I won't be looking until next week. I guess we're pretty confident that we'll find out the root cause in due course since we have a <2h repro cycle and know the exact commit that caused the problem. tobias-1579868169-05-n5cpu16 is one cluster that has this problem. Any other tobias- clusters that are left after http://35.229.40.30:8080/ goes unresponsive will have failed the run, too. Just in case either you or @nvanbenschoten want to dig later today.

andreimatei · 2020-01-24T13:39:35Z

Ugh. I'll rollback something and see what's up. Thanks for looking!

…

On Fri, Jan 24, 2020 at 8:18 AM Tobias Grieger ***@***.***> wrote: Yeah first one already failed: ERROR: duplicate key value (o_w_id,o_d_id,o_id)=(128,3,3010) violates unique constraint "primary" (SQLSTATE 23505) @andreimatei <https://github.com/andreimatei> that seems to make it official. Note that there were other commits in that merge, but of all of them only #43989 <#43989> remotely has the potential to have caused this bug, and I really don't think that one's it. I'm out for the rest of the day, so I won't be looking until next week. I guess we're pretty confident that we'll find out the root cause in due course since we have a <2h repro cycle and know the exact commit that caused the problem. tobias-1579868169-05-n5cpu16 is one cluster that has this problem. Any other tobias- clusters that are left after http://35.229.40.30:8080/ goes unresponsive will have failed the run, too. Just in case either you or @nvanbenschoten <https://github.com/nvanbenschoten> want to dig later today. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#43110?email_source=notifications&email_token=AAC4C4PW23NEEBMV5OB73M3Q7LTDZA5CNFSM4JZJOUL2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJ2YEGA#issuecomment-578126360>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAC4C4LHEYT5D3NKWCUV7D3Q7LTDZANCNFSM4JZJOULQ> .

cockroach-teamcity · 2020-01-24T17:07:24Z

(roachtest).tpcc/mixed-headroom/n5cpu16 failed on master@3c408a47707a15d4b5c92031c4eafcfa643d307c:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200124-1708438/tpcc/mixed-headroom/n5cpu16/run_1
	cluster.go:1935,tpcc.go:169,tpcc.go:261,test_runner.go:719: error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1969
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1933
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:169
		    main.registerTPCC.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:261
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:719
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1708438-1579854217-92-n5cpu16:5 -- ./workload check tpcc --warehouses=909 {pgurl:1} returned:
		    stderr:
		    
		    stdout:
		    I200124 17:06:59.469116 1 workload/tpcc/tpcc.go:372  check 3.3.2.1 took 135.653787ms
		    Error: check failed: 3.3.2.1: 64 rows returned, expected zero
		    Error:  exit status 1:
		  - exit status 1

More

Artifacts: /tpcc/mixed-headroom/n5cpu16

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

andreimatei · 2020-01-24T17:34:20Z

I'm rolling back the main commit from #42969. And I can confirm that on one of the leftover tobias clusters, TPCC check 3.3.2.1 indeed is failing...

44350: storage: revert "storage: rationalize server-side refreshes and fix bugs" r=andreimatei a=andreimatei This reverts commit 1edb0d5. Revert the main commit from #42969 (storage: rationalize server-side refreshes and fix bugs). It seems to be causing consistency problems, as seen in #43110. Co-authored-by: Andrei Matei <andrei@cockroachlabs.com>

andreimatei · 2020-01-25T00:12:24Z

#44350 rolled back the main commit from #42969. Surprisingly, that does not seem to fix it (which is good news, I guess...). #44381 reverts the rest. This does fix it.

The good news is that it repros with 4cpu clusters instead of the tests's 16cpu clusters.
The even better news is that it repros really quickly if you cut out the TPCC ramp up time and pass the no wait flag.

It's interesting that this test is a mixed-version test. I've verified that it does not seem to fail when all nodes are at HEAD.

One of the two failure modes is with duplicate key errors on newOrder. The value for the PK of the orders is computed with

UPDATE district
SET d_next_o_id = d_next_o_id + 1
WHERE d_w_id = $1 AND d_id = $2
RETURNING d_tax, d_next_o_id

This query executes as a scan followed by a put.
Superficially, one possible cause for the error would be if those of these queries return the same d_next_o_id.

That's as far as I've gotten. I suspect 0417ab0.

More later.

44381: storage: revert remaining bits of 42969 r=andreimatei a=andreimatei Revert everything from #42969 that wasn't reverted by #44350. Even after the partial revert in #44350, it appears the bug causing TPCC to fail (#43110) is still there. Release note: None Co-authored-by: Andrei Matei <andrei@cockroachlabs.com>

andreimatei · 2020-01-25T19:55:53Z

I've found it. It's about a 20.1 migrations which removes the use of the currently-named field.DeprecatedOrigTimestamp field. 19.2 nodes set that field instead of the currently-named txn.ReadTimestamp (which used to be txn.RefreshedTimestamp in 19.2). And 0417ab0 forgot about that fact, resulting in the ts cache not being properly populated for requests coming from 19.2.

cockroach-teamcity · 2020-03-08T00:48:08Z

(roachtest).tpcc/mixed-headroom/n5cpu16 failed on master@752dea867f3aeb142e98c22f8d320ce19041aa8d:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200307-1793930/tpcc/mixed-headroom/n5cpu16/run_1
	cluster.go:1935,tpcc.go:170,tpcc.go:264,test_runner.go:741: error with attached stack trace:
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2010
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1933
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:170
		    main.registerTPCC.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:264
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: output in %s
		    -- arg 1: <string>
		  - output in run_184824.903_n5_workload_check_tpcc:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2019
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2000
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1933
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:170
		    main.registerTPCC.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:264
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1793930-1583568484-84-n5cpu16:5 -- ./workload check tpcc --warehouses=909 {pgurl:1} returned:
		    stderr:
		    
		    stdout::
		  - context canceled

	cluster.go:2050,cluster.go:2069,cluster.go:2173,cluster.go:1470,context.go:135,cluster.go:1467,test_runner.go:774: context canceled

More

Artifacts: /tpcc/mixed-headroom/n5cpu16
Related:

roachtest: tpcc/mixed-headroom/n5cpu16 failed #44586 roachtest: tpcc/mixed-headroom/n5cpu16 failed C-test-failure O-roachtest O-robot branch-provisional_202001302015_v19.2.3 release-blocker
roachtest: tpcc/mixed-headroom/n5cpu16 failed #44526 roachtest: tpcc/mixed-headroom/n5cpu16 failed C-test-failure O-roachtest O-robot branch-provisional_202001281357_v19.2.3 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

tbg · 2020-03-09T10:44:42Z

No artifact because of the awkward timeout mode, but I think we can chalk this up to #45830

tbg · 2020-03-09T10:45:04Z

I've found it. It's about a 20.1 migrations which removes the use of the currently-named field.DeprecatedOrigTimestamp field. 19.2 nodes set that field instead of the currently-named txn.ReadTimestamp (which used to be txn.RefreshedTimestamp in 19.2). And 0417ab0 forgot about that fact, resulting in the ts cache not being properly populated for requests coming from 19.2.

@andreimatei did we fix this yet?

cockroach-teamcity · 2020-03-10T00:00:47Z

(roachtest).tpcc/mixed-headroom/n5cpu16 failed on master@c473f40078994551cebcbe00fdbf1fa388957658:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200309-1796240/tpcc/mixed-headroom/n5cpu16/run_1
	cluster.go:1935,tpcc.go:170,tpcc.go:264,test_runner.go:741: error with attached stack trace:
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2010
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1933
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:170
		    main.registerTPCC.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:264
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: output in %s
		    -- arg 1: <string>
		  - output in run_192126.425_n5_workload_check_tpcc:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2019
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2000
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1933
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:170
		    main.registerTPCC.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:264
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1796240-1583738442-85-n5cpu16:5 -- ./workload check tpcc --warehouses=909 {pgurl:1} returned:
		    stderr:
		    
		    stdout::
		  - context canceled

	cluster.go:2050,cluster.go:2069,cluster.go:2173,cluster.go:1470,context.go:135,cluster.go:1467,test_runner.go:774: context canceled

More

Artifacts: /tpcc/mixed-headroom/n5cpu16
Related:

roachtest: tpcc/mixed-headroom/n5cpu16 failed #44586 roachtest: tpcc/mixed-headroom/n5cpu16 failed C-test-failure O-roachtest O-robot branch-provisional_202001302015_v19.2.3 release-blocker
roachtest: tpcc/mixed-headroom/n5cpu16 failed #44526 roachtest: tpcc/mixed-headroom/n5cpu16 failed C-test-failure O-roachtest O-robot branch-provisional_202001281357_v19.2.3 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

andy-kimball · 2020-03-20T16:05:56Z

Here's another one to close once you've verified your fix.

andreimatei · 2020-03-20T16:13:36Z

I've found it. It's about a 20.1 migrations which removes the use of the currently-named field.DeprecatedOrigTimestamp field. 19.2 nodes set that field instead of the currently-named txn.ReadTimestamp (which used to be txn.RefreshedTimestamp in 19.2). And 0417ab0 forgot about that fact, resulting in the ts cache not being properly populated for requests coming from 19.2.

@andreimatei did we fix this yet?

Yeah, that was a fixed a while ago.

nvanbenschoten · 2020-03-20T20:31:07Z

Here's another one to close once you've verified your fix.

I'm kicking off a series of these tests on master to confirm.

nvanbenschoten · 2020-03-21T02:11:42Z

10/10 passing. This looks fixed.

cockroach-teamcity added C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. labels Dec 11, 2019

cockroach-teamcity added this to the 20.1 milestone Dec 11, 2019

cockroach-teamcity assigned andreimatei Dec 11, 2019

tbg added the branch-master Failures and bugs on the master branch. label Jan 22, 2020

tbg mentioned this issue Jan 24, 2020

release: v20.1.0-alpha.20200123 #43754

Closed

19 tasks

andreimatei mentioned this issue Jan 24, 2020

storage: revert "storage: rationalize server-side refreshes and fix bugs" #44350

Merged

andreimatei mentioned this issue Jan 24, 2020

storage: revert remaining bits of 42969 #44381

Merged

This was referenced Jan 30, 2020

roachtest: tpcc/mixed-headroom/n5cpu16 failed #44526

Closed

roachtest: tpcc/mixed-headroom/n5cpu16 failed #44586

Closed

nvanbenschoten unassigned andreimatei Mar 17, 2020

andy-kimball assigned nvanbenschoten Mar 20, 2020

nvanbenschoten closed this as completed Mar 21, 2020

roachtest: tpcc/mixed-headroom/n5cpu16 failed #43110

roachtest: tpcc/mixed-headroom/n5cpu16 failed #43110

Comments

cockroach-teamcity commented Dec 11, 2019

cockroach-teamcity commented Dec 19, 2019

cockroach-teamcity commented Dec 20, 2019

cockroach-teamcity commented Dec 22, 2019

cockroach-teamcity commented Dec 23, 2019

cockroach-teamcity commented Dec 25, 2019

cockroach-teamcity commented Dec 26, 2019

cockroach-teamcity commented Dec 27, 2019

cockroach-teamcity commented Dec 28, 2019

cockroach-teamcity commented Dec 29, 2019

cockroach-teamcity commented Dec 30, 2019

cockroach-teamcity commented Jan 1, 2020

cockroach-teamcity commented Jan 2, 2020

cockroach-teamcity commented Jan 3, 2020

cockroach-teamcity commented Jan 4, 2020

cockroach-teamcity commented Jan 5, 2020

cockroach-teamcity commented Jan 7, 2020

cockroach-teamcity commented Jan 7, 2020

cockroach-teamcity commented Jan 8, 2020

cockroach-teamcity commented Jan 16, 2020

cockroach-teamcity commented Jan 17, 2020

cockroach-teamcity commented Jan 19, 2020

cockroach-teamcity commented Jan 22, 2020

cockroach-teamcity commented Jan 22, 2020

tbg commented Jan 23, 2020

tbg commented Jan 23, 2020 • edited Loading

tbg commented Jan 24, 2020

tbg commented Jan 24, 2020

tbg commented Jan 24, 2020

andreimatei commented Jan 24, 2020 via email

cockroach-teamcity commented Jan 24, 2020

andreimatei commented Jan 24, 2020

andreimatei commented Jan 25, 2020 • edited Loading

andreimatei commented Jan 25, 2020

cockroach-teamcity commented Mar 8, 2020

tbg commented Mar 9, 2020

tbg commented Mar 9, 2020

cockroach-teamcity commented Mar 10, 2020

andy-kimball commented Mar 20, 2020

andreimatei commented Mar 20, 2020

nvanbenschoten commented Mar 20, 2020

nvanbenschoten commented Mar 21, 2020

tbg commented Jan 23, 2020 •

edited

Loading

andreimatei commented Jan 25, 2020 •

edited

Loading