roachtest: tpcc/mixed-headroom/n5cpu16 failed

[cockroach-teamcity]

SHA: https://github.com/cockroachdb/cockroach/commits/179f29b066c266d14cfeac33ce29b2d18ba86c63

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpcc/mixed-headroom/n5cpu16 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1465495&tab=buildLog

The test failed on branch=release-2.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190904-1465495/tpcc/mixed-headroom/n5cpu16/run_1
	cluster.go:1735,tpcc.go:162,tpcc.go:164,tpcc.go:175,tpcc.go:288,test_runner.go:688: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1567577389-61-n5cpu16:5 -- ./workload fixtures load tpcc --warehouses=1000  {pgurl:1} returned:
		stderr:
		
		stdout:
		I190904 10:12:39.523069 110 ccl/workloadccl/fixture.go:544  loaded 126 MiB table new_order in 15m42.952791503s (9000000 rows, 0 index entries, 136 KiB)
		I190904 10:14:57.843431 109 ccl/workloadccl/fixture.go:544  loaded 1.3 GiB table order in 18m1.273322188s (30000000 rows, 30000000 index entries, 1.3 MiB)
		I190904 10:16:17.977356 108 ccl/workloadccl/fixture.go:544  loaded 3.8 GiB table history in 19m21.407289732s (30000000 rows, 60000000 index entries, 3.4 MiB)
		I190904 10:19:43.485745 107 ccl/workloadccl/fixture.go:544  loaded 17 GiB table customer in 22m46.915606848s (30000000 rows, 30000000 index entries, 13 MiB)
		I190904 10:28:49.677898 113 ccl/workloadccl/fixture.go:544  loaded 18 GiB table order_line in 31m53.10783794s (300011520 rows, 300011520 index entries, 9.8 MiB)
		I190904 10:30:37.703182 112 ccl/workloadccl/fixture.go:544  loaded 31 GiB table stock in 33m41.132913741s (100000000 rows, 100000000 index entries, 16 MiB)
		Error: restoring fixture: pq: syntax error at or near "inject"
		Error:  exit status 1
		: exit status 1

[cockroach-teamcity]

SHA: https://github.com/cockroachdb/cockroach/commits/179f29b066c266d14cfeac33ce29b2d18ba86c63

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpcc/mixed-headroom/n5cpu16 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1468506&tab=buildLog

The test failed on branch=provisional_201909042143_v2.1.9, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190905-1468506/tpcc/mixed-headroom/n5cpu16/run_1
	cluster.go:1735,tpcc.go:162,tpcc.go:164,tpcc.go:175,tpcc.go:288,test_runner.go:688: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1567656826-62-n5cpu16:5 -- ./workload fixtures load tpcc --warehouses=1000  {pgurl:1} returned:
		stderr:
		
		stdout:
		47 B)
		I190905 08:30:37.139456 134 ccl/workloadccl/fixture.go:544  loaded 1006 KiB table district in 6m58.588327201s (10000 rows, 0 index entries, 2.4 KiB)
		I190905 08:36:41.206250 137 ccl/workloadccl/fixture.go:544  loaded 1.3 GiB table order in 13m2.655116538s (30000000 rows, 30000000 index entries, 1.7 MiB)
		I190905 08:37:46.029537 136 ccl/workloadccl/fixture.go:544  loaded 3.6 GiB table history in 14m7.478455561s (30000000 rows, 60000000 index entries, 4.3 MiB)
		I190905 08:48:18.380438 135 ccl/workloadccl/fixture.go:544  loaded 17 GiB table customer in 24m39.82915846s (30000000 rows, 30000000 index entries, 12 MiB)
		I190905 08:51:40.182775 140 ccl/workloadccl/fixture.go:544  loaded 32 GiB table stock in 28m1.631493827s (100000000 rows, 100000000 index entries, 20 MiB)
		I190905 08:52:34.109581 141 ccl/workloadccl/fixture.go:544  loaded 22 GiB table order_line in 28m55.558563448s (300011520 rows, 300011520 index entries, 13 MiB)
		Error: restoring fixture: pq: syntax error at or near "inject"
		Error:  exit status 1
		: exit status 1

[nvanbenschoten]

It looks like we're trying to run ALTER TABLE ... INJECT STATISTICS against a cluster that doesn't support this. Both of these failures have been on release-2.1, which did support this syntax (c7482e0), but the mixed-headroom runs nodes with the predecessor version - 2.0.7. 2.0.7 did not support this syntax.

I'm just going to ignore the failure in workload.