roachtest: clearrange/checks=false failed

[cockroach-teamcity]

SHA: https://github.com/cockroachdb/cockroach/commits/98ef7abf32784b8e837d18d10173ef083010ad45

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=clearrange/checks=false PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1067238&tab=buildLog

The test failed on master:
	test.go:1101: test timed out (1h30m0s)
	test.go:703,clearrange.go:55,clearrange.go:34: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1067238-clearrange-checks-false:7 -- mkdir -p {store-dir} && gsutil -m -q cp -r gs://cockroach-fixtures/workload/bank/version=1.0.0,payload-bytes=10240,ranges=0,rows=65104166,seed=4/stores=10,bin-version=2.0-8/7/* {store-dir} returned:
		stderr:
		
		stdout:
		: signal: killed

[cockroach-teamcity]

SHA: https://github.com/cockroachdb/cockroach/commits/1da87b86059c282c2b2821d6b5b481f17c89d98e

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=clearrange/checks=false PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1069764&tab=buildLog

The test failed on master:
	test.go:1101: test timed out (1h30m0s)
	test.go:703,clearrange.go:55,clearrange.go:34: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1069764-clearrange-checks-false:5 -- mkdir -p {store-dir} && gsutil -m -q cp -r gs://cockroach-fixtures/workload/bank/version=1.0.0,payload-bytes=10240,ranges=0,rows=65104166,seed=4/stores=10,bin-version=2.0-8/5/* {store-dir} returned:
		stderr:
		
		stdout:
		: signal: killed

[tbg]

This actually did time out, and it was actually downloading the store dumps that took so long. What gives?

[petermattis]

I saw a similar problem with the backup2TB test. Perhaps Google Cloud Storage (or our bucket) was having problems. My best thought for what to do about this is #33377. Please add your thoughts there if you have any (even if they are only "+1").