roachtest: schemachange/mixed-versions failed #58237

cockroach-teamcity · 2020-12-23T08:10:30Z

(roachtest).schemachange/mixed-versions failed on master@cd62a02833eeb02d14944668d09ee54806927a2b:

		  -- stack trace:
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2292
		  | main.(*cluster).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2212
		  | main.runSchemaChangeWorkloadStep.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_schemachange.go:67
		  | main.(*versionUpgradeTest).run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:189
		  | main.runSchemaChangeMixedVersions
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_schemachange.go:144
		  | main.registerSchemaChangeMixedVersions.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_schemachange.go:36
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:760
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (2) output in run_071010.723_n3_workload_run_schemachange
		Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2538659-1608706971-01-n4cpu4:3 -- ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4} returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | iled to initialize the load generator: ERROR: desc 291: type ID 177 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W201223 08:09:45.161400 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: ERROR: desc 291: type ID 177 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W201223 08:09:47.114697 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: ERROR: desc 291: type ID 177 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W201223 08:09:49.365989 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: ERROR: desc 291: type ID 177 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W201223 08:09:51.304176 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: ERROR: desc 291: type ID 177 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W201223 08:09:53.377493 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: ERROR: desc 291: type ID 177 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W201223 08:09:55.339039 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: ERROR: desc 291: type ID 177 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W201223 08:09:57.176933 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: ERROR: desc 291: type ID 177 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W201223 08:09:59.451290 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: ERROR: desc 291: type ID 177 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W201223 08:10:01.628243 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: ERROR: desc 291: type ID 177 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W201223 08:10:03.869135 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: ERROR: desc 291: type ID 177 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W201223 08:10:05.956825 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: ERROR: desc 291: type ID 177 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W201223 08:10:08.175993 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: ERROR: desc 291: type ID 177 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W201223 08:10:10.238476 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: ERROR: desc 291: type ID 177 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W201223 08:10:12.009228 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: ERROR: desc 291: type ID 177 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | E201223 08:10:12.148153 1 workload/cli/run.go:382  Attempt to create load generator failed. It's been more than 1h0m0s since we started trying to create the load generator so we're giving up. Last failure: failed to initialize the load generator: ERROR: desc 291: type ID 177 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | Error: failed to initialize the load generator: ERROR: desc 291: type ID 177 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 3. Command with error:
		  |   | ```
		  |   | ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4}
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

More

Artifacts: /schemachange/mixed-versions

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

The text was updated successfully, but these errors were encountered:

cockroach-teamcity · 2020-12-29T17:10:32Z

(roachtest).schemachange/mixed-versions failed on master@365c5504b75c9a9260365a628a5110c48312178b:

		  | 35730.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35730.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35731.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35731.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35732.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35732.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35733.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35733.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35734.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35734.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35735.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35735.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35736.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35736.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35737.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35737.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35738.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35738.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35739.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35739.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35740.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35740.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35741.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35741.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35742.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35742.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35743.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35743.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35744.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35744.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35745.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35745.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35746.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35746.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35747.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35747.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35748.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35748.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35749.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35749.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35750.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35750.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		Wraps: (4) secondary error attachment
		  | signal: killed
		  | (1) signal: killed
		  | Error types: (1) *exec.ExitError
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString

More

Artifacts: /schemachange/mixed-versions

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-12-31T08:09:43Z

(roachtest).schemachange/mixed-versions failed on master@b93fd531b93cb010729cb73fe679cdff9388cf27:

		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2292
		  | main.(*cluster).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2212
		  | main.runSchemaChangeWorkloadStep.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_schemachange.go:67
		  | main.(*versionUpgradeTest).run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:189
		  | main.runSchemaChangeMixedVersions
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_schemachange.go:144
		  | main.registerSchemaChangeMixedVersions.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_schemachange.go:36
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:760
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (2) output in run_070924.359_n3_workload_run_schemachange
		Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2552340-1609398198-09-n4cpu4:3 -- ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4} returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | type ID 210 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W201231 08:08:56.642846 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: ERROR: type ID 210 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W201231 08:08:58.595937 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: ERROR: type ID 210 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W201231 08:09:00.847061 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: ERROR: type ID 210 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W201231 08:09:02.785252 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: ERROR: type ID 210 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W201231 08:09:04.856723 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: ERROR: type ID 210 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W201231 08:09:06.819472 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: ERROR: type ID 210 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W201231 08:09:08.657713 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: ERROR: type ID 210 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W201231 08:09:10.930993 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: ERROR: type ID 210 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W201231 08:09:13.105538 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: ERROR: type ID 210 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W201231 08:09:15.346753 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: ERROR: type ID 210 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W201231 08:09:17.434674 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: ERROR: type ID 210 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W201231 08:09:19.652741 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: ERROR: type ID 210 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W201231 08:09:21.713150 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: ERROR: type ID 210 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W201231 08:09:23.483842 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: ERROR: type ID 210 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W201231 08:09:25.582401 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: ERROR: type ID 210 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | E201231 08:09:25.768750 1 workload/cli/run.go:382  Attempt to create load generator failed. It's been more than 1h0m0s since we started trying to create the load generator so we're giving up. Last failure: failed to initialize the load generator: ERROR: type ID 210 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | Error: failed to initialize the load generator: ERROR: type ID 210 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 3. Command with error:
		  |   | ```
		  |   | ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4}
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

More

Artifacts: /schemachange/mixed-versions

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-01-06T17:10:16Z

(roachtest).schemachange/mixed-versions failed on master@cee475331ca3629b503cd2e7c7919b72c98a5ca5:

		  | 35845.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35845.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35845.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35846.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35846.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35846.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35847.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35847.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35847.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35848.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35848.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35848.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35849.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35849.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35849.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35850.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35850.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35850.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35851.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35851.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35851.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35852.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35852.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35852.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35853.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35853.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35853.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35854.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35854.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35854.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35855.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35855.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35855.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35856.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35856.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35856.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35857.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35857.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35857.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35858.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35858.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35858.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		Wraps: (4) secondary error attachment
		  | signal: killed
		  | (1) signal: killed
		  | Error types: (1) *exec.ExitError
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString

More

Artifacts: /schemachange/mixed-versions

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-01-07T08:07:14Z

(roachtest).schemachange/mixed-versions failed on master@15765c0fa9118885dda0bd2ad1384b8801c412c3:

		  -- stack trace:
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2292
		  | main.(*cluster).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2212
		  | main.runSchemaChangeWorkloadStep.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_schemachange.go:67
		  | main.(*versionUpgradeTest).run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:189
		  | main.runSchemaChangeMixedVersions
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_schemachange.go:144
		  | main.registerSchemaChangeMixedVersions.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_schemachange.go:36
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:760
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (2) output in run_070654.017_n1_workload_run_schemachange
		Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2564046-1610002658-02-n4cpu4:1 -- ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4} returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | iled to initialize the load generator: ERROR: desc 519: type ID 561 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W210107 08:06:27.089332 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: ERROR: desc 519: type ID 561 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W210107 08:06:29.145154 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: ERROR: desc 519: type ID 561 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W210107 08:06:31.066366 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: ERROR: desc 519: type ID 561 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W210107 08:06:33.126304 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: ERROR: desc 519: type ID 561 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W210107 08:06:35.227935 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: ERROR: desc 519: type ID 561 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W210107 08:06:37.540127 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: ERROR: desc 519: type ID 561 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W210107 08:06:39.498238 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: ERROR: desc 519: type ID 561 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W210107 08:06:41.755672 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: ERROR: desc 519: type ID 561 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W210107 08:06:43.699675 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: ERROR: desc 519: type ID 561 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W210107 08:06:45.785266 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: ERROR: desc 519: type ID 561 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W210107 08:06:47.752366 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: ERROR: desc 519: type ID 561 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W210107 08:06:49.594728 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: ERROR: desc 519: type ID 561 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W210107 08:06:51.873850 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: ERROR: desc 519: type ID 561 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W210107 08:06:54.053258 1 workload/cli/run.go:368  retrying after error while creating load: failed to initialize the load generator: ERROR: desc 519: type ID 561 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | E210107 08:06:55.468826 1 workload/cli/run.go:382  Attempt to create load generator failed. It's been more than 1h0m0s since we started trying to create the load generator so we're giving up. Last failure: failed to initialize the load generator: ERROR: desc 519: type ID 561 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | Error: failed to initialize the load generator: ERROR: desc 519: type ID 561 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ```
		  |   | ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4}
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

More

Artifacts: /schemachange/mixed-versions
Related:

roachtest: schemachange/mixed-versions failed #58530 roachtest: schemachange/mixed-versions failed C-test-failure O-roachtest O-robot branch-release-20.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-01-09T17:09:24Z

(roachtest).schemachange/mixed-versions failed on master@dbc7245c5d8c9f009072353fec261419e573032c:

		  | 35827.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35827.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35828.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35828.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35828.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35828.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35829.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35829.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35829.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35829.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35830.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35830.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35830.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35830.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35831.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35831.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35831.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35831.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35832.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35832.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35832.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35832.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35833.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35833.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35833.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35833.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35834.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35834.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35834.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35834.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35835.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35835.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35835.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35835.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35836.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35836.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35836.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35836.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35837.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35837.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35837.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35837.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		Wraps: (4) secondary error attachment
		  | signal: killed
		  | (1) signal: killed
		  | Error types: (1) *exec.ExitError
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString

More

Artifacts: /schemachange/mixed-versions

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

thoszhang · 2021-01-11T21:49:24Z

I haven't been able to figure out what's going on with the timeouts. This is different from #58344: There are no cannot be reverted, manual cleanup may be required errors in the logs, and no logs for schema changes waiting on other schema changes in the mutation queue.

One thing that's suspicious is that we get a lot of these in the logs, for the same table and version, every 5 minutes or so:

teamcity-2569739-1610175644-15-n4cpu4-0004> I210109 16:53:38.566031 8630 sql/schema_changer.go:980 ⋮ [n4,job=622938404807475204] waiting for a single version... done (at v 43)

I think these are coming from the GC job:

cockroach/pkg/sql/gcjob/gc_job.go

Lines 93 to 109 in 54dbbe5

    
           // If there are any interleaved indexes to drop as part of a table TRUNCATE 
        
           // operation, then drop the indexes before waiting on the GC timer. 
        
           if len(details.InterleavedIndexes) > 0 { 
        
           	// Before deleting any indexes, ensure that old versions of the table 
        
           	// descriptor are no longer in use. 
        
           	if err := sql.WaitToUpdateLeases(ctx, execCfg.LeaseManager, details.InterleavedTable.ID); err != nil { 
        
           		return err 
        
           	} 
        
           	if err := sql.TruncateInterleavedIndexes( 
        
           		ctx, 
        
           		execCfg, 
        
           		tabledesc.NewImmutable(*details.InterleavedTable), 
        
           		details.InterleavedIndexes, 
        
           	); err != nil { 
        
           		return err 
        
           	} 
        
           }

The issue seems to be that we're repeatedly writing the descriptor every time we check statuses for the GC job, when we should just be doing it once (I think). But I doubt this is related to the timeouts.

Edit: the weirder thing is that I grepped for INTERLEAVE in the workload log files (again, in the most recent failure) and didn't turn up anything, even though I think we should be logging all attempted DDL statements. So I don't know why we're hitting the len(details.InterleavedIndexes) > 0 case.

cockroach-teamcity · 2021-01-17T17:09:17Z

(roachtest).schemachange/mixed-versions failed on master@7b0ccdda99b81613e70f421c9374483c3feddff3:

		  | 35859.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35859.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35860.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35860.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35861.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35861.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35862.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35862.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35863.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35863.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35864.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35864.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35865.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35865.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35866.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35866.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35867.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35867.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35868.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35868.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35869.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35869.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35870.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35870.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35871.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35871.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35872.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35872.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35873.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35873.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35874.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35874.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35875.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35875.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35876.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35876.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35877.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35877.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35878.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35878.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35879.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35879.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		Wraps: (4) secondary error attachment
		  | signal: killed
		  | (1) signal: killed
		  | Error types: (1) *exec.ExitError
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString

More

Artifacts: /schemachange/mixed-versions

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

thoszhang · 2021-01-19T20:18:10Z

I'm removing the release-blocker tag. Copying what I said in #58489:

The schemachange timeouts shouldn't be a release blocker. We've been running into these timeouts in for a while, starting at around the same time that Jay added new functionality to the workload, so it's likely that the workload has been uncovering some latent problem with certain combinations of schema changes. We're going to do #59157 to mitigate the impact to the tests, and we're going to investigate further soon.

cockroach-teamcity · 2021-01-25T17:25:23Z

(roachtest).schemachange/mixed-versions failed on master@48fbbe7313fc70f889fa66c5749914f57ddd5a91:

		  | 35788.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35788.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35789.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35789.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35789.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35789.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35790.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35790.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35790.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35790.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35791.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35791.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35791.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35791.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35792.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35792.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35792.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35792.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35793.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35793.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35793.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35793.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35794.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35794.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35794.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35794.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35795.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35795.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35795.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35795.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35796.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35796.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35796.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35796.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35797.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35797.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35797.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35797.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35798.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35798.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35798.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35798.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		Wraps: (4) secondary error attachment
		  | signal: killed
		  | (1) signal: killed
		  | Error types: (1) *exec.ExitError
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString

More

Artifacts: /schemachange/mixed-versions

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-01-28T17:02:48Z

(roachtest).schemachange/mixed-versions failed on master@1a459a81dba35b6a091f0a2954aa33d50f1e5d24:

		  | 35857.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35857.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35858.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35858.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35859.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35859.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35860.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35860.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35861.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35861.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35862.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35862.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35863.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35863.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35864.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35864.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35865.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35865.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35866.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35866.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35867.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35867.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35868.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35868.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35869.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35869.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35870.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35870.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35871.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35871.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35872.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35872.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35873.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35873.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35874.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35874.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35875.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35875.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35876.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35876.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35877.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35877.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		Wraps: (4) secondary error attachment
		  | signal: killed
		  | (1) signal: killed
		  | Error types: (1) *exec.ExitError
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString

More

Artifacts: /schemachange/mixed-versions

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-02-01T17:02:38Z

(roachtest).schemachange/mixed-versions failed on master@e4ac407b14ed315662050f09670707cdf8ad9bfe:

		  | 35869.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35869.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35870.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35870.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35870.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35870.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35871.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35871.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35871.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35871.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35872.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35872.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35872.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35872.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35873.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35873.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35873.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35873.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35874.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35874.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35874.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35874.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35875.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35875.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35875.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35875.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35876.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35876.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35876.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35876.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35877.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35877.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35877.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35877.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35878.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35878.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35878.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35878.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35879.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35879.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35879.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35879.0s        7            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		Wraps: (4) secondary error attachment
		  | signal: killed
		  | (1) signal: killed
		  | Error types: (1) *exec.ExitError
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString

More

Artifacts: /schemachange/mixed-versions

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-02-06T07:11:19Z

(roachtest).schemachange/mixed-versions failed on master@81a2c26a104fa8cc7e8b530b837ffb6ff85ddc5a:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/schemachange/mixed-versions/run_1
	cluster.go:2253,mixed_version_schemachange.go:46,versionupgrade.go:192,mixed_version_schemachange.go:144,mixed_version_schemachange.go:36,test_runner.go:767: output in run_071103.153_n1-4_workload_init_schemachange: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2650259-1612594948-16-n4cpu4:1-4 -- ./workload init schemachange returned: exit status 20
		(1) attached stack trace
		  -- stack trace:
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2331
		  | main.(*cluster).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2251
		  | main.uploadAndInitSchemaChangeWorkload.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_schemachange.go:46
		  | main.(*versionUpgradeTest).run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:192
		  | main.runSchemaChangeMixedVersions
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_schemachange.go:144
		  | main.registerSchemaChangeMixedVersions.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_schemachange.go:36
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:767
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (2) output in run_071103.153_n1-4_workload_init_schemachange
		Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2650259-1612594948-16-n4cpu4:1-4 -- ./workload init schemachange returned
		  | stderr:
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ```
		  |   | ./workload init schemachange
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		  | teamcity-2650259-1612594948-16-n4cpu4: ./workload init schemachange
		  |    1: ./workload: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by ./workload)
		  | COMMAND_PROBLEM: exit status 1
		  |    2: ./workload: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by ./workload)
		  | COMMAND_PROBLEM: exit status 1
		  |    3: ./workload: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by ./workload)
		  | COMMAND_PROBLEM: exit status 1
		  |    4: ./workload: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by ./workload)
		  | COMMAND_PROBLEM: exit status 1
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

More

Artifacts: /schemachange/mixed-versions

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-02-07T07:08:27Z

(roachtest).schemachange/mixed-versions failed on master@0e98670c8fcec566937e899fdf77d2a68c702d62:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/schemachange/mixed-versions/run_1
	cluster.go:2253,mixed_version_schemachange.go:46,versionupgrade.go:192,mixed_version_schemachange.go:144,mixed_version_schemachange.go:36,test_runner.go:767: output in run_070810.071_n1-4_workload_init_schemachange: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2651620-1612681270-07-n4cpu4:1-4 -- ./workload init schemachange returned: exit status 20
		(1) attached stack trace
		  -- stack trace:
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2331
		  | main.(*cluster).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2251
		  | main.uploadAndInitSchemaChangeWorkload.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_schemachange.go:46
		  | main.(*versionUpgradeTest).run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:192
		  | main.runSchemaChangeMixedVersions
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_schemachange.go:144
		  | main.registerSchemaChangeMixedVersions.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_schemachange.go:36
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:767
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (2) output in run_070810.071_n1-4_workload_init_schemachange
		Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2651620-1612681270-07-n4cpu4:1-4 -- ./workload init schemachange returned
		  | stderr:
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ```
		  |   | ./workload init schemachange
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		  | teamcity-2651620-1612681270-07-n4cpu4: ./workload init schemachange
		  |    1: ./workload: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by ./workload)
		  | COMMAND_PROBLEM: exit status 1
		  |    2: ./workload: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by ./workload)
		  | COMMAND_PROBLEM: exit status 1
		  |    3: ./workload: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by ./workload)
		  | COMMAND_PROBLEM: exit status 1
		  |    4: ./workload: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by ./workload)
		  | COMMAND_PROBLEM: exit status 1
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

More

Artifacts: /schemachange/mixed-versions

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-02-08T06:59:38Z

(roachtest).schemachange/mixed-versions failed on master@5b33e6dfc47000de831745a851e3bf9e2cf7fd95:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/schemachange/mixed-versions/run_1
	cluster.go:2253,mixed_version_schemachange.go:46,versionupgrade.go:192,mixed_version_schemachange.go:144,mixed_version_schemachange.go:36,test_runner.go:767: output in run_065920.911_n1-4_workload_init_schemachange: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2653401-1612767119-13-n4cpu4:1-4 -- ./workload init schemachange returned: exit status 20
		(1) attached stack trace
		  -- stack trace:
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2331
		  | main.(*cluster).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2251
		  | main.uploadAndInitSchemaChangeWorkload.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_schemachange.go:46
		  | main.(*versionUpgradeTest).run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:192
		  | main.runSchemaChangeMixedVersions
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_schemachange.go:144
		  | main.registerSchemaChangeMixedVersions.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_schemachange.go:36
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:767
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (2) output in run_065920.911_n1-4_workload_init_schemachange
		Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2653401-1612767119-13-n4cpu4:1-4 -- ./workload init schemachange returned
		  | stderr:
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ```
		  |   | ./workload init schemachange
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		  | teamcity-2653401-1612767119-13-n4cpu4: ./workload init schemachange
		  |    1: ./workload: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by ./workload)
		  | COMMAND_PROBLEM: exit status 1
		  |    2: ./workload: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by ./workload)
		  | COMMAND_PROBLEM: exit status 1
		  |    3: ./workload: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by ./workload)
		  | COMMAND_PROBLEM: exit status 1
		  |    4: ./workload: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by ./workload)
		  | COMMAND_PROBLEM: exit status 1
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

More

Artifacts: /schemachange/mixed-versions

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-02-09T07:11:31Z

(roachtest).schemachange/mixed-versions failed on master@7853fd32de8b6dea869f2a2a92dcd7506f4a8998:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/schemachange/mixed-versions/run_1
	cluster.go:2253,mixed_version_schemachange.go:46,versionupgrade.go:192,mixed_version_schemachange.go:144,mixed_version_schemachange.go:36,test_runner.go:767: output in run_071107.295_n1-4_workload_init_schemachange: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2657140-1612854228-01-n4cpu4:1-4 -- ./workload init schemachange returned: exit status 20
		(1) attached stack trace
		  -- stack trace:
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2331
		  | main.(*cluster).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2251
		  | main.uploadAndInitSchemaChangeWorkload.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_schemachange.go:46
		  | main.(*versionUpgradeTest).run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:192
		  | main.runSchemaChangeMixedVersions
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_schemachange.go:144
		  | main.registerSchemaChangeMixedVersions.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_schemachange.go:36
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:767
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (2) output in run_071107.295_n1-4_workload_init_schemachange
		Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2657140-1612854228-01-n4cpu4:1-4 -- ./workload init schemachange returned
		  | stderr:
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ```
		  |   | ./workload init schemachange
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		  | teamcity-2657140-1612854228-01-n4cpu4: ./workload init schemachange
		  |    1: ./workload: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by ./workload)
		  | COMMAND_PROBLEM: exit status 1
		  |    2: ./workload: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by ./workload)
		  | COMMAND_PROBLEM: exit status 1
		  |    3: ./workload: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by ./workload)
		  | COMMAND_PROBLEM: exit status 1
		  |    4: ./workload: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by ./workload)
		  | COMMAND_PROBLEM: exit status 1
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

More

Artifacts: /schemachange/mixed-versions

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-02-14T17:00:58Z

(roachtest).schemachange/mixed-versions failed on master@9969862335cf501c1aa67a4aaf018ad5dc8e85e8:

		  | 35872.0s       13            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35872.0s       13            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35873.0s       13            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35873.0s       13            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35873.0s       13            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35873.0s       13            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35874.0s       13            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35874.0s       13            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35874.0s       13            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35874.0s       13            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35875.0s       13            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35875.0s       13            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35875.0s       13            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35875.0s       13            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35876.0s       13            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35876.0s       13            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35876.0s       13            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35876.0s       13            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35877.0s       13            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35877.0s       13            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35877.0s       13            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35877.0s       13            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35878.0s       13            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35878.0s       13            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35878.0s       13            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35878.0s       13            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35879.0s       13            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35879.0s       13            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35879.0s       13            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35879.0s       13            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35880.0s       13            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35880.0s       13            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35880.0s       13            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35880.0s       13            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35881.0s       13            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35881.0s       13            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35881.0s       13            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35881.0s       13            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35882.0s       13            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35882.0s       13            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35882.0s       13            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35882.0s       13            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		Wraps: (4) secondary error attachment
		  | signal: killed
		  | (1) signal: killed
		  | Error types: (1) *exec.ExitError
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString

More

Artifacts: /schemachange/mixed-versions
Related:

roachtest: schemachange/mixed-versions failed #60331 roachtest: schemachange/mixed-versions failed C-test-failure O-roachtest O-robot branch-release-20.2 release-blocker
roachtest: schemachange/mixed-versions failed #60322 roachtest: schemachange/mixed-versions failed C-test-failure O-roachtest O-robot branch-release-20.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-02-15T17:04:35Z

(roachtest).schemachange/mixed-versions failed on master@5971ecb9dd1a25c81cd6012d6be1ff922802eae5:

		  | 35516.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35516.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35517.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35517.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35517.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35517.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35518.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35518.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35518.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35518.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35519.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35519.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35519.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35519.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35520.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35520.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35520.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35520.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35521.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35521.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35521.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35521.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35522.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35522.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35522.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35522.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35523.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35523.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35523.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35523.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35524.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35524.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35524.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35524.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35525.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35525.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35525.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35525.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35526.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35526.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35526.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35526.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		Wraps: (4) secondary error attachment
		  | signal: killed
		  | (1) signal: killed
		  | Error types: (1) *exec.ExitError
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString

More

Artifacts: /schemachange/mixed-versions
Related:

roachtest: schemachange/mixed-versions failed #60331 roachtest: schemachange/mixed-versions failed C-test-failure O-roachtest O-robot branch-release-20.2 release-blocker
roachtest: schemachange/mixed-versions failed #60322 roachtest: schemachange/mixed-versions failed C-test-failure O-roachtest O-robot branch-release-20.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-02-21T17:10:26Z

(roachtest).schemachange/mixed-versions failed on master@64c4aef909f4382523cd9248341ca9f4448d841a:

		  | 35860.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35860.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35861.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35861.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35862.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35862.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35863.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35863.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35864.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35864.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35865.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35865.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35866.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35866.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35867.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35867.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35868.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35868.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35869.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35869.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35870.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35870.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35871.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35871.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35872.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35872.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35873.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35873.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35874.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35874.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35875.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35875.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35876.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35876.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35877.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35877.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35878.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35878.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35879.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35879.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35880.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35880.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		Wraps: (4) secondary error attachment
		  | signal: killed
		  | (1) signal: killed
		  | Error types: (1) *exec.ExitError
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString

More

Artifacts: /schemachange/mixed-versions
Related:

roachtest: schemachange/mixed-versions failed #60667 roachtest: schemachange/mixed-versions failed C-test-failure O-roachtest O-robot branch-60635 release-blocker
roachtest: schemachange/mixed-versions failed #60322 roachtest: schemachange/mixed-versions failed C-test-failure O-roachtest O-robot branch-release-20.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-02-26T17:12:54Z

(roachtest).schemachange/mixed-versions failed on master@6601d827b814d4e85a1081b03bf2562d8ac2a4ab:

		  | 35911.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35911.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35911.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35912.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35912.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35912.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35913.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35913.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35913.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35914.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35914.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35914.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35915.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35915.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35915.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35916.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35916.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35916.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35917.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35917.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35917.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35918.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35918.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35918.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35919.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35919.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35919.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35920.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35920.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35920.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35921.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35921.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35921.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35922.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35922.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35922.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35923.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35923.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35923.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35924.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35924.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35924.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		Wraps: (4) secondary error attachment
		  | signal: killed
		  | (1) signal: killed
		  | Error types: (1) *exec.ExitError
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString

More

Artifacts: /schemachange/mixed-versions

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-02-28T17:07:12Z

(roachtest).schemachange/mixed-versions failed on master@7d1324fa42732f482329a524b0166db8dd7365e6:

		  | 35950.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35950.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35951.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35951.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35952.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35952.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35953.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35953.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35954.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35954.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35955.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35955.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35956.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35956.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35957.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35957.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35958.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35958.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35959.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35959.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35960.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35960.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35961.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35961.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35962.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35962.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35963.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35963.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35964.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35964.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35965.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35965.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35966.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35966.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35967.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35967.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35968.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35968.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35969.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35969.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35970.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35970.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		Wraps: (4) secondary error attachment
		  | signal: killed
		  | (1) signal: killed
		  | Error types: (1) *exec.ExitError
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString

More

Artifacts: /schemachange/mixed-versions

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-03-02T08:05:47Z

(roachtest).schemachange/mixed-versions failed on master@9ba48738bc511ad6954682cab41e23b8492facd8:

test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/schemachange/mixed-versions/run_1
	cluster.go:2254,mixed_version_schemachange.go:67,versionupgrade.go:200,mixed_version_schemachange.go:144,mixed_version_schemachange.go:36,test_runner.go:767: output in run_070511.124_n2_workload_run_schemachange: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2733649-1614668254-05-n4cpu4:2 -- ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4} returned: exit status 20
		(1) attached stack trace
		  -- stack trace:
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2332
		  | main.(*cluster).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2252
		  | main.runSchemaChangeWorkloadStep.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_schemachange.go:67
		  | main.(*versionUpgradeTest).run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:200
		  | main.runSchemaChangeMixedVersions
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_schemachange.go:144
		  | main.registerSchemaChangeMixedVersions.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_schemachange.go:36
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:767
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (2) output in run_070511.124_n2_workload_run_schemachange
		Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2733649-1614668254-05-n4cpu4:2 -- ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4} returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | 8:04:48.020669 1 workload/cli/run.go:367  [-] 1767  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: type "enum107" (183): referencing table 287 was dropped without dependency unlinking (SQLSTATE XX000)
		  | W210302 08:04:49.897498 1 workload/cli/run.go:367  [-] 1768  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: type "enum107" (183): referencing table 287 was dropped without dependency unlinking (SQLSTATE XX000)
		  | W210302 08:04:51.829353 1 workload/cli/run.go:367  [-] 1769  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: type "enum107" (183): referencing table 287 was dropped without dependency unlinking (SQLSTATE XX000)
		  | W210302 08:04:54.159125 1 workload/cli/run.go:367  [-] 1770  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: type "enum107" (183): referencing table 287 was dropped without dependency unlinking (SQLSTATE XX000)
		  | W210302 08:04:56.030282 1 workload/cli/run.go:367  [-] 1771  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: type "enum107" (183): referencing table 287 was dropped without dependency unlinking (SQLSTATE XX000)
		  | W210302 08:04:58.218240 1 workload/cli/run.go:367  [-] 1772  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: type "enum107" (183): referencing table 287 was dropped without dependency unlinking (SQLSTATE XX000)
		  | W210302 08:05:00.334143 1 workload/cli/run.go:367  [-] 1773  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: type "enum107" (183): referencing table 287 was dropped without dependency unlinking (SQLSTATE XX000)
		  | W210302 08:05:02.278908 1 workload/cli/run.go:367  [-] 1774  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: type "enum107" (183): referencing table 287 was dropped without dependency unlinking (SQLSTATE XX000)
		  | W210302 08:05:04.597929 1 workload/cli/run.go:367  [-] 1775  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: type "enum107" (183): referencing table 287 was dropped without dependency unlinking (SQLSTATE XX000)
		  | W210302 08:05:06.737135 1 workload/cli/run.go:367  [-] 1776  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: type "enum107" (183): referencing table 287 was dropped without dependency unlinking (SQLSTATE XX000)
		  | W210302 08:05:08.866982 1 workload/cli/run.go:367  [-] 1777  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: type "enum107" (183): referencing table 287 was dropped without dependency unlinking (SQLSTATE XX000)
		  | W210302 08:05:11.192907 1 workload/cli/run.go:367  [-] 1778  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: type "enum107" (183): referencing table 287 was dropped without dependency unlinking (SQLSTATE XX000)
		  | E210302 08:05:12.550037 1 workload/cli/run.go:381  [-] 1779  Attempt to create load generator failed. It's been more than 1h0m0s since we started trying to create the load generator so we're giving up. Last failure: failed to initialize the load generator: ERROR: internal error: type "enum107" (183): referencing table 287 was dropped without dependency unlinking (SQLSTATE XX000)
		  | Error: failed to initialize the load generator: ERROR: internal error: type "enum107" (183): referencing table 287 was dropped without dependency unlinking (SQLSTATE XX000)
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 2. Command with error:
		  |   | ```
		  |   | ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4}
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

More

Artifacts: /schemachange/mixed-versions

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

thoszhang · 2021-03-02T15:25:25Z

failed to initialize the load generator: ERROR: internal error: type "enum107" (183): referencing table 287 was dropped without dependency unlinking (SQLSTATE XX000)

@ajwerner I don't think we've seen this one before.

cockroach-teamcity · 2021-03-03T08:04:18Z

(roachtest).schemachange/mixed-versions failed on master@6de4313ec216161c79fe725fcc31fc87ef1804ea:

test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/schemachange/mixed-versions/run_1
	cluster.go:2254,mixed_version_schemachange.go:67,versionupgrade.go:200,mixed_version_schemachange.go:144,mixed_version_schemachange.go:36,test_runner.go:767: output in run_070333.543_n3_workload_run_schemachange: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2738824-1614754482-06-n4cpu4:3 -- ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4} returned: exit status 20
		(1) attached stack trace
		  -- stack trace:
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2332
		  | main.(*cluster).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2252
		  | main.runSchemaChangeWorkloadStep.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_schemachange.go:67
		  | main.(*versionUpgradeTest).run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:200
		  | main.runSchemaChangeMixedVersions
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_schemachange.go:144
		  | main.registerSchemaChangeMixedVersions.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_schemachange.go:36
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:767
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (2) output in run_070333.543_n3_workload_run_schemachange
		Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2738824-1614754482-06-n4cpu4:3 -- ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4} returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | 8:03:10.631221 1 workload/cli/run.go:367  [-] 1712  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: type "enum155" (186): referencing table 475 was dropped without dependency unlinking (SQLSTATE XX000)
		  | W210303 08:03:12.602631 1 workload/cli/run.go:367  [-] 1713  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: type "enum155" (186): referencing table 475 was dropped without dependency unlinking (SQLSTATE XX000)
		  | W210303 08:03:14.809036 1 workload/cli/run.go:367  [-] 1714  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: type "enum155" (186): referencing table 475 was dropped without dependency unlinking (SQLSTATE XX000)
		  | W210303 08:03:16.861634 1 workload/cli/run.go:367  [-] 1715  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: type "enum155" (186): referencing table 475 was dropped without dependency unlinking (SQLSTATE XX000)
		  | W210303 08:03:18.689196 1 workload/cli/run.go:367  [-] 1716  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: type "enum155" (186): referencing table 475 was dropped without dependency unlinking (SQLSTATE XX000)
		  | W210303 08:03:20.683352 1 workload/cli/run.go:367  [-] 1717  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: type "enum155" (186): referencing table 475 was dropped without dependency unlinking (SQLSTATE XX000)
		  | W210303 08:03:22.966375 1 workload/cli/run.go:367  [-] 1718  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: type "enum155" (186): referencing table 475 was dropped without dependency unlinking (SQLSTATE XX000)
		  | W210303 08:03:25.260763 1 workload/cli/run.go:367  [-] 1719  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: type "enum155" (186): referencing table 475 was dropped without dependency unlinking (SQLSTATE XX000)
		  | W210303 08:03:27.480585 1 workload/cli/run.go:367  [-] 1720  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: type "enum155" (186): referencing table 475 was dropped without dependency unlinking (SQLSTATE XX000)
		  | W210303 08:03:29.326611 1 workload/cli/run.go:367  [-] 1721  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: type "enum155" (186): referencing table 475 was dropped without dependency unlinking (SQLSTATE XX000)
		  | W210303 08:03:31.630848 1 workload/cli/run.go:367  [-] 1722  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: type "enum155" (186): referencing table 475 was dropped without dependency unlinking (SQLSTATE XX000)
		  | W210303 08:03:33.641523 1 workload/cli/run.go:367  [-] 1723  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: type "enum155" (186): referencing table 475 was dropped without dependency unlinking (SQLSTATE XX000)
		  | E210303 08:03:34.951430 1 workload/cli/run.go:381  [-] 1724  Attempt to create load generator failed. It's been more than 1h0m0s since we started trying to create the load generator so we're giving up. Last failure: failed to initialize the load generator: ERROR: internal error: type "enum155" (186): referencing table 475 was dropped without dependency unlinking (SQLSTATE XX000)
		  | Error: failed to initialize the load generator: ERROR: internal error: type "enum155" (186): referencing table 475 was dropped without dependency unlinking (SQLSTATE XX000)
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 3. Command with error:
		  |   | ```
		  |   | ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4}
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

More

Artifacts: /schemachange/mixed-versions

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-03-12T18:03:35Z

(roachtest).schemachange/mixed-versions failed on master@4b98115dfda02a9498f566958bd915c45ec7e449:

test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/schemachange/mixed-versions/run_1
	cluster.go:2254,mixed_version_schemachange.go:67,versionupgrade.go:200,mixed_version_schemachange.go:144,mixed_version_schemachange.go:36,test_runner.go:767: output in run_170300.781_n2_workload_run_schemachange: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2771389-1615568102-12-n4cpu4:2 -- ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4} returned: exit status 20
		(1) attached stack trace
		  -- stack trace:
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2332
		  | main.(*cluster).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2252
		  | main.runSchemaChangeWorkloadStep.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_schemachange.go:67
		  | main.(*versionUpgradeTest).run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:200
		  | main.runSchemaChangeMixedVersions
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_schemachange.go:144
		  | main.registerSchemaChangeMixedVersions.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_schemachange.go:36
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:767
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (2) output in run_170300.781_n2_workload_run_schemachange
		Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2771389-1615568102-12-n4cpu4:2 -- ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4} returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | 8:02:37.232075 1 workload/cli/run.go:370  [-] 1767  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: type "enum132" (196): referencing table 251 was dropped without dependency unlinking (SQLSTATE XX000)
		  | W210312 18:02:39.117225 1 workload/cli/run.go:370  [-] 1768  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: type "enum132" (196): referencing table 251 was dropped without dependency unlinking (SQLSTATE XX000)
		  | W210312 18:02:41.046623 1 workload/cli/run.go:370  [-] 1769  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: type "enum132" (196): referencing table 251 was dropped without dependency unlinking (SQLSTATE XX000)
		  | W210312 18:02:43.373301 1 workload/cli/run.go:370  [-] 1770  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: type "enum132" (196): referencing table 251 was dropped without dependency unlinking (SQLSTATE XX000)
		  | W210312 18:02:45.225815 1 workload/cli/run.go:370  [-] 1771  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: type "enum132" (196): referencing table 251 was dropped without dependency unlinking (SQLSTATE XX000)
		  | W210312 18:02:47.419774 1 workload/cli/run.go:370  [-] 1772  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: type "enum132" (196): referencing table 251 was dropped without dependency unlinking (SQLSTATE XX000)
		  | W210312 18:02:49.534579 1 workload/cli/run.go:370  [-] 1773  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: type "enum132" (196): referencing table 251 was dropped without dependency unlinking (SQLSTATE XX000)
		  | W210312 18:02:51.477880 1 workload/cli/run.go:370  [-] 1774  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: type "enum132" (196): referencing table 251 was dropped without dependency unlinking (SQLSTATE XX000)
		  | W210312 18:02:53.795315 1 workload/cli/run.go:370  [-] 1775  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: type "enum132" (196): referencing table 251 was dropped without dependency unlinking (SQLSTATE XX000)
		  | W210312 18:02:55.938516 1 workload/cli/run.go:370  [-] 1776  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: type "enum132" (196): referencing table 251 was dropped without dependency unlinking (SQLSTATE XX000)
		  | W210312 18:02:58.063534 1 workload/cli/run.go:370  [-] 1777  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: type "enum132" (196): referencing table 251 was dropped without dependency unlinking (SQLSTATE XX000)
		  | W210312 18:03:00.388193 1 workload/cli/run.go:370  [-] 1778  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: type "enum132" (196): referencing table 251 was dropped without dependency unlinking (SQLSTATE XX000)
		  | E210312 18:03:02.183901 1 workload/cli/run.go:384  [-] 1779  Attempt to create load generator failed. It's been more than 1h0m0s since we started trying to create the load generator so we're giving up. Last failure: failed to initialize the load generator: ERROR: internal error: type "enum132" (196): referencing table 251 was dropped without dependency unlinking (SQLSTATE XX000)
		  | Error: failed to initialize the load generator: ERROR: internal error: type "enum132" (196): referencing table 251 was dropped without dependency unlinking (SQLSTATE XX000)
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 2. Command with error:
		  |   | ```
		  |   | ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4}
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

More

Artifacts: /schemachange/mixed-versions

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-03-13T17:07:58Z

(roachtest).schemachange/mixed-versions failed on master@4d44ddf24153d8ef8e0a996fdbe75ac5607f9574:

		  | 35891.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35891.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35891.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35892.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35892.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35892.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35893.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35893.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35893.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35894.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35894.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35894.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35895.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35895.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35895.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35896.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35896.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35896.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35897.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35897.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35897.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35898.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35898.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35898.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35899.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35899.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35899.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35900.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35900.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35900.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35901.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35901.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35901.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35902.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35902.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35902.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35903.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35903.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35903.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35904.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35904.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35904.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		Wraps: (4) secondary error attachment
		  | signal: killed
		  | (1) signal: killed
		  | Error types: (1) *exec.ExitError
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString

More

Artifacts: /schemachange/mixed-versions

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-04-25T05:14:26Z

roachtest.schemachange/mixed-versions failed with artifacts on master @ 3cd29547884feb24b150b67ca2a88073798a35fd:

		  | 35659.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35659.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35660.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35660.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35660.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35660.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35661.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35661.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35661.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35661.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35662.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35662.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35662.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35662.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35663.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35663.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35663.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35663.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35664.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35664.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35664.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35664.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35665.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35665.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35665.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35665.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35666.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35666.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35666.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35666.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35667.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35667.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35667.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35667.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35668.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35668.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35668.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35668.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35669.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35669.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35669.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35669.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		Wraps: (4) secondary error attachment
		  | signal: killed
		  | (1) signal: killed
		  | Error types: (1) *exec.ExitError
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString

Reproduce

To reproduce, try:

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh schemachange/mixed-versions

Same failure on other branches

roachtest: schemachange/mixed-versions failed #62158 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot branch-release-21.1]

/cc @cockroachdb/sql-schema _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2021-04-30T07:37:22Z

roachtest.schemachange/mixed-versions failed with artifacts on master @ d84625c42d11e38775b6a1ff3f4089e839fe3e82:

		  | W210430 07:36:26.526362 1 workload/cli/run.go:385  [-] 1731  retrying after error while creating load: failed to initialize the load generator: dial tcp 10.128.0.142:26257: connect: connection refused
		  | W210430 07:36:28.790396 1 workload/cli/run.go:385  [-] 1732  retrying after error while creating load: failed to initialize the load generator: dial tcp 10.128.0.142:26257: connect: connection refused
		  | W210430 07:36:30.893253 1 workload/cli/run.go:385  [-] 1733  retrying after error while creating load: failed to initialize the load generator: dial tcp 10.128.0.142:26257: connect: connection refused
		  | W210430 07:36:32.756626 1 workload/cli/run.go:385  [-] 1734  retrying after error while creating load: failed to initialize the load generator: dial tcp 10.128.0.142:26257: connect: connection refused
		  | W210430 07:36:35.017706 1 workload/cli/run.go:385  [-] 1735  retrying after error while creating load: failed to initialize the load generator: dial tcp 10.128.0.142:26257: connect: connection refused
		  | W210430 07:36:37.134003 1 workload/cli/run.go:385  [-] 1736  retrying after error while creating load: failed to initialize the load generator: dial tcp 10.128.0.142:26257: connect: connection refused
		  | W210430 07:36:38.980569 1 workload/cli/run.go:385  [-] 1737  retrying after error while creating load: failed to initialize the load generator: dial tcp 10.128.0.142:26257: connect: connection refused
		  | W210430 07:36:40.812191 1 workload/cli/run.go:385  [-] 1738  retrying after error while creating load: failed to initialize the load generator: dial tcp 10.128.0.142:26257: connect: connection refused
		  | W210430 07:36:42.912283 1 workload/cli/run.go:385  [-] 1739  retrying after error while creating load: failed to initialize the load generator: dial tcp 10.128.0.142:26257: connect: connection refused
		  | E210430 07:36:44.736804 1 workload/cli/run.go:399  [-] 1740  Attempt to create load generator failed. It's been more than 1h0m0s since we started trying to create the load generator so we're giving up. Last failure: failed to initialize the load generator: dial tcp 10.128.0.142:26257: connect: connection refused
		  | Error: failed to initialize the load generator: dial tcp 10.128.0.142:26257: connect: connection refused
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ``````
		  |   | ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4}
		  |   | ``````
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

	cluster.go:1716,context.go:140,cluster.go:1705,test_runner.go:858: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2940313-1619763594-01-n4cpu4 --oneshot --ignore-empty-nodes: exit status 1 1: 13787
		3: dead
		4: 12171
		2: 12953
		Error: UNCLASSIFIED_PROBLEM: 3: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1162
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:284
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:2057
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:204
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (3) 3: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError

Reproduce

To reproduce, try:

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh schemachange/mixed-versions

Same failure on other branches

roachtest: schemachange/mixed-versions failed #62158 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot branch-release-21.1]

/cc @cockroachdb/sql-schema _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2021-04-30T20:20:26Z

roachtest.schemachange/mixed-versions failed with artifacts on master @ 42e2e4265087751695afc05fc5ad8ec534fe0121:

		  | W210430 20:19:26.903104 1 workload/cli/run.go:385  [-] 1706  retrying after error while creating load: failed to initialize the load generator: dial tcp 10.128.0.42:26257: connect: connection refused
		  | W210430 20:19:29.036997 1 workload/cli/run.go:385  [-] 1707  retrying after error while creating load: failed to initialize the load generator: dial tcp 10.128.0.42:26257: connect: connection refused
		  | W210430 20:19:31.210021 1 workload/cli/run.go:385  [-] 1708  retrying after error while creating load: failed to initialize the load generator: dial tcp 10.128.0.42:26257: connect: connection refused
		  | W210430 20:19:33.086415 1 workload/cli/run.go:385  [-] 1709  retrying after error while creating load: failed to initialize the load generator: dial tcp 10.128.0.42:26257: connect: connection refused
		  | W210430 20:19:35.146424 1 workload/cli/run.go:385  [-] 1710  retrying after error while creating load: failed to initialize the load generator: dial tcp 10.128.0.42:26257: connect: connection refused
		  | W210430 20:19:37.317227 1 workload/cli/run.go:385  [-] 1711  retrying after error while creating load: failed to initialize the load generator: dial tcp 10.128.0.42:26257: connect: connection refused
		  | W210430 20:19:39.324504 1 workload/cli/run.go:385  [-] 1712  retrying after error while creating load: failed to initialize the load generator: dial tcp 10.128.0.42:26257: connect: connection refused
		  | W210430 20:19:41.307730 1 workload/cli/run.go:385  [-] 1713  retrying after error while creating load: failed to initialize the load generator: dial tcp 10.128.0.42:26257: connect: connection refused
		  | W210430 20:19:43.528084 1 workload/cli/run.go:385  [-] 1714  retrying after error while creating load: failed to initialize the load generator: dial tcp 10.128.0.42:26257: connect: connection refused
		  | E210430 20:19:43.629860 1 workload/cli/run.go:399  [-] 1715  Attempt to create load generator failed. It's been more than 1h0m0s since we started trying to create the load generator so we're giving up. Last failure: failed to initialize the load generator: dial tcp 10.128.0.42:26257: connect: connection refused
		  | Error: failed to initialize the load generator: dial tcp 10.128.0.42:26257: connect: connection refused
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 2. Command with error:
		  |   | ``````
		  |   | ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4}
		  |   | ``````
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

	cluster.go:1716,context.go:140,cluster.go:1705,test_runner.go:858: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2941631-1619809231-03-n4cpu4 --oneshot --ignore-empty-nodes: exit status 1 2: 14405
		3: dead
		1: 16321
		4: 14295
		Error: UNCLASSIFIED_PROBLEM: 3: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1162
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:284
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:2057
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:204
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (3) 3: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError

Reproduce

To reproduce, try:

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh schemachange/mixed-versions

Same failure on other branches

roachtest: schemachange/mixed-versions failed #62158 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot branch-release-21.1]

/cc @cockroachdb/sql-schema _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2021-05-01T19:22:54Z

roachtest.schemachange/mixed-versions failed with artifacts on master @ 684e753c15f3fc58df79b6ea70e7b6715eae4835:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/schemachange/mixed-versions/run_1
	mixed_version_schemachange.go:81,mixed_version_schemachange.go:36,test_runner.go:777: prev version not set for version: 21.2
		(1) attached stack trace
		  -- stack trace:
		  | main.PredecessorVersion
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:1210
		  | main.runSchemaChangeMixedVersions
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_schemachange.go:79
		  | main.registerSchemaChangeMixedVersions.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_schemachange.go:36
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (2) prev version not set for version: 21.2
		Error types: (1) *withstack.withStack (2) *errutil.leafError

Reproduce

To reproduce, try:

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh schemachange/mixed-versions

Same failure on other branches

roachtest: schemachange/mixed-versions failed #62158 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot branch-release-21.1]

/cc @cockroachdb/sql-schema _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2021-05-03T19:17:46Z

roachtest.schemachange/mixed-versions failed with artifacts on master @ dbf711a83969210bcf706d65f53f888856da6ad0:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/schemachange/mixed-versions/run_1
	mixed_version_schemachange.go:81,mixed_version_schemachange.go:36,test_runner.go:777: prev version not set for version: 21.2
		(1) attached stack trace
		  -- stack trace:
		  | main.PredecessorVersion
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:1210
		  | main.runSchemaChangeMixedVersions
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_schemachange.go:79
		  | main.registerSchemaChangeMixedVersions.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_schemachange.go:36
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (2) prev version not set for version: 21.2
		Error types: (1) *withstack.withStack (2) *errutil.leafError

Reproduce

To reproduce, try:

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh schemachange/mixed-versions

Same failure on other branches

roachtest: schemachange/mixed-versions failed #62158 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot branch-release-21.1]

/cc @cockroachdb/sql-schema _{This test on roachdash | Improve this report!}

postamar · 2021-05-03T19:46:01Z

~~It seems to me that this error will keep occurring until the 21.1.0 release, at which point we'll be able to add an entry for 21.2 in~~

cockroach/pkg/cmd/roachtest/test_runner.go

Line 1199 in 212a424

verMap := map[string]string{

~~referring to a 21.1 binary which will exist.~~

~~@jordanlewis what's the correct way to deal with this twice a year? Disable the test once the nightly gets tagged as the subsequent version until the upcoming version gets released?~~

edit: never mind! see below

irfansharif · 2021-05-03T19:49:04Z

Caused due to #64391, +cc @j-low.

cockroach-teamcity mentioned this issue Jan 7, 2021

roachtest: schemachange/mixed-versions failed #58530

Closed

cockroach-teamcity mentioned this issue Jan 12, 2021

roachtest: schemachange/mixed-versions failed #58760

Closed

thoszhang self-assigned this Jan 19, 2021

thoszhang mentioned this issue Jan 19, 2021

roachtest: set statement timeout for schemachange workload to bypass known test timeout failures #59157

Closed

thoszhang removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Jan 19, 2021

cockroach-teamcity mentioned this issue Feb 9, 2021

roachtest: schemachange/mixed-versions failed #60048

Closed

thoszhang assigned ajwerner and unassigned thoszhang Feb 9, 2021

This was referenced Feb 9, 2021

roachtest: schemachange/mixed-versions failed #60172

Closed

roachtest: schemachange/mixed-versions failed #60322

Closed

roachtest: schemachange/mixed-versions failed #60331

Closed

cockroach-teamcity mentioned this issue Feb 17, 2021

roachtest: schemachange/mixed-versions failed #60667

Closed

cockroach-teamcity mentioned this issue Mar 17, 2021

roachtest: schemachange/mixed-versions failed #62158

Closed

cockroach-teamcity mentioned this issue Apr 14, 2021

roachtest: schemachange/mixed-versions failed #63682

Closed

j-low mentioned this issue May 3, 2021

roachtest: update version map and test fixtures for 21.2 #64610

Merged

craig bot closed this as completed in e3327f0 May 4, 2021

craig bot closed this as completed in #64610 May 4, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

roachtest: schemachange/mixed-versions failed #58237

roachtest: schemachange/mixed-versions failed #58237

cockroach-teamcity commented Dec 23, 2020

cockroach-teamcity commented Dec 29, 2020

cockroach-teamcity commented Dec 31, 2020

cockroach-teamcity commented Jan 6, 2021

cockroach-teamcity commented Jan 7, 2021

cockroach-teamcity commented Jan 9, 2021

thoszhang commented Jan 11, 2021 •

edited

Loading

cockroach-teamcity commented Jan 17, 2021

thoszhang commented Jan 19, 2021

cockroach-teamcity commented Jan 25, 2021

cockroach-teamcity commented Jan 28, 2021

cockroach-teamcity commented Feb 1, 2021

cockroach-teamcity commented Feb 6, 2021

cockroach-teamcity commented Feb 7, 2021

cockroach-teamcity commented Feb 8, 2021

cockroach-teamcity commented Feb 9, 2021

cockroach-teamcity commented Feb 14, 2021

cockroach-teamcity commented Feb 15, 2021

cockroach-teamcity commented Feb 21, 2021

cockroach-teamcity commented Feb 26, 2021

cockroach-teamcity commented Feb 28, 2021

cockroach-teamcity commented Mar 2, 2021

thoszhang commented Mar 2, 2021

cockroach-teamcity commented Mar 3, 2021

cockroach-teamcity commented Mar 12, 2021

cockroach-teamcity commented Mar 13, 2021

cockroach-teamcity commented Apr 25, 2021

cockroach-teamcity commented Apr 30, 2021

cockroach-teamcity commented Apr 30, 2021

cockroach-teamcity commented May 1, 2021

cockroach-teamcity commented May 3, 2021

postamar commented May 3, 2021 •

edited

Loading

irfansharif commented May 3, 2021

roachtest: schemachange/mixed-versions failed #58237

roachtest: schemachange/mixed-versions failed #58237

Comments

cockroach-teamcity commented Dec 23, 2020

cockroach-teamcity commented Dec 29, 2020

cockroach-teamcity commented Dec 31, 2020

cockroach-teamcity commented Jan 6, 2021

cockroach-teamcity commented Jan 7, 2021

cockroach-teamcity commented Jan 9, 2021

thoszhang commented Jan 11, 2021 • edited Loading

cockroach-teamcity commented Jan 17, 2021

thoszhang commented Jan 19, 2021

cockroach-teamcity commented Jan 25, 2021

cockroach-teamcity commented Jan 28, 2021

cockroach-teamcity commented Feb 1, 2021

cockroach-teamcity commented Feb 6, 2021

cockroach-teamcity commented Feb 7, 2021

cockroach-teamcity commented Feb 8, 2021

cockroach-teamcity commented Feb 9, 2021

cockroach-teamcity commented Feb 14, 2021

cockroach-teamcity commented Feb 15, 2021

cockroach-teamcity commented Feb 21, 2021

cockroach-teamcity commented Feb 26, 2021

cockroach-teamcity commented Feb 28, 2021

cockroach-teamcity commented Mar 2, 2021

thoszhang commented Mar 2, 2021

cockroach-teamcity commented Mar 3, 2021

cockroach-teamcity commented Mar 12, 2021

cockroach-teamcity commented Mar 13, 2021

cockroach-teamcity commented Apr 25, 2021

cockroach-teamcity commented Apr 30, 2021

cockroach-teamcity commented Apr 30, 2021

cockroach-teamcity commented May 1, 2021

cockroach-teamcity commented May 3, 2021

postamar commented May 3, 2021 • edited Loading

irfansharif commented May 3, 2021

thoszhang commented Jan 11, 2021 •

edited

Loading

postamar commented May 3, 2021 •

edited

Loading