roachtest: jepsen-batch3/multi-register/start-stop-2 failed [hopefully #36431] #38126

cockroach-teamcity · 2019-06-10T11:41:37Z

SHA: https://github.com/cockroachdb/cockroach/commits/3b63648f905715e6c0b055fe9acac9c5b8206196

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=jepsen-batch3/multi-register/start-stop-2 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1333520&tab=buildLog

The test failed on branch=master, cloud=gce:
	jepsen.go:253,jepsen.go:323,test.go:1248: exit status 1

The text was updated successfully, but these errors were encountered:

cockroach-teamcity · 2019-06-15T10:24:29Z

SHA: https://github.com/cockroachdb/cockroach/commits/753f245ae441a4ea5e00bdedd99145c19d5e320b

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=jepsen-batch3/multi-register/start-stop-2 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1339408&tab=buildLog

The test failed on branch=release-2.1, cloud=gce:
	jepsen.go:253,jepsen.go:323,test.go:1248: exit status 1

tbg · 2019-06-19T13:35:21Z

10:23:24 cluster.go:272: > /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1339408-jepsen-batch3:6 -- tail -n 100 /mnt/data1/jepsen/cockroachdb/invoke.log
        :value [[:write 1 8] [:write 3 5]],
        :index 49681,
        :time 176928274222},
       :pending []}
      {:model {4 6, 2 4, 0 2, 1 8, 3 5},
       :last-op
       {:process 27,
        :type :ok,
        :f :txn,
        :value [[:write 1 8] [:write 3 5]],
        :index 49681,
        :time 176928274222},
       :pending []}),
     :final-paths ()},
    :valid? true},
   49
   {:timeline {:valid? true},
    :linearizable
    {:valid? true,
     :configs
     ({:model {4 2, 0 4, 1 6, 3 0, 2 1},
       :last-op
       {:process 20,
        :type :ok,
        :f :txn,
        :value [[:write 4 2] [:write 3 0] [:write 0 4] [:write 2 1]],
        :index 3146,
        :time 13704112334},
       :pending []}),
     :final-paths ()},
    :valid? true},
   858
   {:timeline {:valid? true},
    :linearizable
    {:valid? true,
     :configs
     ({:model {2 3, 3 7, 4 8, 1 7, 0 9},
       :last-op
       {:process 6,
        :type :ok,
        :f :txn,
        :value [[:read 1 7]],
        :index 51642,
        :time 180476400380},
       :pending []}),
     :final-paths ()},
    :valid? true},
   390
   {:timeline {:valid? true},
    :linearizable
    {:valid? true,
     :configs
     ({:model {2 7, 3 2, 0 1, 1 3, 4 3},
       :last-op
       {:process 19,
        :type :ok,
        :f :txn,
        :value [[:write 4 3] [:write 0 1]],
        :index 23845,
        :time 80873793827},
       :pending []}),
     :final-paths ()},
    :valid? true},
   1259
   {:timeline {:valid? true},
    :linearizable
    {:valid? true,
     :configs
     ({:model {1 6, 2 3, 4 2, 3 1, 0 9},
       :last-op
       {:process 15,
        :type :ok,
        :f :txn,
        :value [[:write 3 1]],
        :index 75990,
        :time 277086487452},
       :pending []}),
     :final-paths ()},
    :valid? true},
   84
   {:timeline {:valid? true},
    :linearizable
    {:valid? true,
     :configs
     ({:model {1 5, 3 5, 4 6, 0 9, 2 1},
       :last-op
       {:process 24,
        :type :ok,
        :f :txn,
        :value [[:read 2 1]],
        :index 5393,
        :time 17890191036},
       :pending []}),
     :final-paths ()},
    :valid? true}},
  :failures [74 758 460 115 824 346 209]},
 :valid? false}


Analysis invalid! (ï¾‰à²¥ç›Šà²¥ï¼‰ï¾‰ â”»â”�â”»

Hopefully this is hopefully #36431

tbg · 2019-06-19T13:36:22Z

@nvanbenschoten this also failed on 2.1, but that's expected, right?

nvanbenschoten · 2019-06-19T16:28:04Z

Yes, that's expected. We didn't introduce this issue recently.

cockroach-teamcity · 2019-06-30T15:03:33Z

SHA: https://github.com/cockroachdb/cockroach/commits/86154ae6ae36e286883d8a6c9a4111966198201d

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=jepsen-batch3/multi-register/start-stop-2 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1367379&tab=buildLog

The test failed on branch=master, cloud=gce:
	jepsen.go:259,jepsen.go:321,test.go:1249: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1367379-jepsen-batch3:6 -- bash -e -c "\
		cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && \
		 ~/lein run test \
		   --tarball file://${PWD}/cockroach.tgz \
		   --username ${USER} \
		   --ssh-private-key ~/.ssh/id_rsa \
		   --os ubuntu \
		   --time-limit 300 \
		   --concurrency 30 \
		   --recovery-time 25 \
		   --test-count 1 \
		   -n 10.142.0.60 -n 10.142.0.56 -n 10.142.0.62 -n 10.142.0.58 -n 10.142.0.7 \
		   --test multi-register --nemesis start-stop-2 \
		> invoke.log 2>&1 \
		" returned:
		stderr:
		
		stdout:
		Error:  ssh verbose log retained in /root/.roachprod/debug/ssh_35.196.133.241_2019-06-30T14:46:32Z: exit status 1
		: exit status 1

cockroach-teamcity · 2019-07-04T14:03:36Z

SHA: https://github.com/cockroachdb/cockroach/commits/4878cb3e960dc26f0946e527e1836f27b3304c97

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=jepsen/multi-register/start-stop-2 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1373334&tab=buildLog

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190704-1373334/jepsen/multi-register/start-stop-2/run_1
	cluster.go:1724,jepsen.go:98,jepsen.go:138,jepsen.go:316,test_runner.go:681: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1562220659-29-n6cpu4:1-6 -- tar --transform s,^,cockroach/, -c -z -f cockroach.tgz cockroach returned:
		stderr:
		
		stdout:
		teamcity-1562220659-29-n6cpu4: tar --transform s,^,cockroa.........
		   1: 
		   2: tar: cockroach: Cannot stat: No such file or directory
		tar: Exiting with failure status due to previous errors
		ssh verbose log retained in /root/.roachprod/debug/ssh_35.227.82.203_2019-07-04T14:03:20Z: exit status 2
		   3: 
		   4: 
		   5: tar: cockroach: Cannot stat: No such file or directory
		tar: Exiting with failure status due to previous errors
		ssh verbose log retained in /root/.roachprod/debug/ssh_34.73.100.118_2019-07-04T14:03:20Z: exit status 2
		   6: tar: cockroach: Cannot stat: No such file or directory
		tar: Exiting with failure status due to previous errors
		ssh verbose log retained in /root/.roachprod/debug/ssh_35.196.208.190_2019-07-04T14:03:20Z: exit status 2
		Error:  ssh verbose log retained in /root/.roachprod/debug/ssh_35.227.82.203_2019-07-04T14:03:20Z: exit status 2
		: exit status 1

cockroach-teamcity · 2019-07-04T14:07:21Z

SHA: https://github.com/cockroachdb/cockroach/commits/4878cb3e960dc26f0946e527e1836f27b3304c97

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=jepsen/register/start-stop-2 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1373334&tab=buildLog

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190704-1373334/jepsen/register/start-stop-2/run_1
	cluster.go:1724,jepsen.go:98,jepsen.go:138,jepsen.go:316,test_runner.go:681: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1562220659-29-n6cpu4:1-6 -- tar --transform s,^,cockroach/, -c -z -f cockroach.tgz cockroach returned:
		stderr:
		
		stdout:
		teamcity-1562220659-29-n6cpu4: tar --transform s,^,cockroa.........
		   1: 
		   2: tar: cockroach: Cannot stat: No such file or directory
		tar: Exiting with failure status due to previous errors
		ssh verbose log retained in /root/.roachprod/debug/ssh_35.227.82.203_2019-07-04T14:07:06Z: exit status 2
		   3: 
		   4: 
		   5: tar: cockroach: Cannot stat: No such file or directory
		tar: Exiting with failure status due to previous errors
		ssh verbose log retained in /root/.roachprod/debug/ssh_34.73.100.118_2019-07-04T14:07:06Z: exit status 2
		   6: tar: cockroach: Cannot stat: No such file or directory
		tar: Exiting with failure status due to previous errors
		ssh verbose log retained in /root/.roachprod/debug/ssh_35.196.208.190_2019-07-04T14:07:06Z: exit status 2
		Error:  ssh verbose log retained in /root/.roachprod/debug/ssh_35.227.82.203_2019-07-04T14:07:06Z: exit status 2
		: exit status 1

cockroach-teamcity · 2019-07-06T15:20:15Z

SHA: https://github.com/cockroachdb/cockroach/commits/9322e07476de447799c5d3011eb2874930ee2993

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=jepsen/multi-register/start-stop-2 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1375546&tab=buildLog

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190706-1375546/jepsen/multi-register/start-stop-2/run_1
	jepsen.go:264,jepsen.go:316,test_runner.go:681: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1562393890-30-n6cpu4:6 -- bash -e -c "\
		cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && \
		 ~/lein run test \
		   --tarball file://${PWD}/cockroach.tgz \
		   --username ${USER} \
		   --ssh-private-key ~/.ssh/id_rsa \
		   --os ubuntu \
		   --time-limit 300 \
		   --concurrency 30 \
		   --recovery-time 25 \
		   --test-count 1 \
		   -n 10.142.0.16 -n 10.142.0.65 -n 10.142.0.69 -n 10.142.0.73 -n 10.142.0.59 \
		   --test multi-register --nemesis start-stop-2 \
		> invoke.log 2>&1 \
		" returned:
		stderr:
		
		stdout:
		Error:  ssh verbose log retained in /root/.roachprod/debug/ssh_35.196.125.127_2019-07-06T15:03:55Z: exit status 1
		: exit status 1

cockroach-teamcity · 2019-08-01T17:45:44Z

SHA: https://github.com/cockroachdb/cockroach/commits/da56c792e968574b8f1d9ef3fdb45d56a530221a

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=jepsen/register/start-stop-2 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1415578&tab=buildLog

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190801-1415578/jepsen/register/start-stop-2/run_1
	jepsen.go:264,jepsen.go:325,test_runner.go:691: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1564640260-62-n6cpu4:6 -- bash -e -c "\
		cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && \
		 ~/lein run test \
		   --tarball file://${PWD}/cockroach.tgz \
		   --username ${USER} \
		   --ssh-private-key ~/.ssh/id_rsa \
		   --os ubuntu \
		   --time-limit 300 \
		   --concurrency 30 \
		   --recovery-time 25 \
		   --test-count 1 \
		   -n 10.128.0.25 -n 10.128.0.54 -n 10.128.0.86 -n 10.128.0.88 -n 10.128.0.46 \
		   --test register --nemesis start-stop-2 \
		> invoke.log 2>&1 \
		" returned:
		stderr:
		
		stdout:
		Error:  exit status 255
		: exit status 1

nvanbenschoten · 2019-08-01T21:30:35Z

The previous failure was a duplicate of #39218, which is fixed by cockroachdb/jepsen#23.

…tervals This commit fixes the most common failure case in all of the following Jepsen failures. I'm not closing them here though, because they also should be failing due to cockroachdb#36431. Would fix cockroachdb#37394. Would fix cockroachdb#37545. Would fix cockroachdb#37930. Would fix cockroachdb#37932. Would fix cockroachdb#37956. Would fix cockroachdb#38126. Would fix cockroachdb#38763. Before this fix, we were not considering intents in a scan's uncertainty interval to be uncertain. This had the potential to cause stale reads because an unresolved intent doesn't indicate that its transaction hasn’t been committed and isn’t a causal ancestor of the scan. This was causing the `jepsen/multi-register` tests to fail, which I had previously incorrectly attributed entirely to cockroachdb#36431. This commit fixes this by returning `WriteIntentError`s for intents when they are above the read timestamp of a scan but below the max timestamp of a scan. This could have also been fixed by returning `ReadWithinUncertaintyIntervalError`s in this situation. Both would eventually have the same effect, but it seems preferable to kick off concurrency control immediately in this situation and only fall back to uncertainty handling for committed values. If the intent ends up being aborted, this could allow the read to avoid moving its timestamp. This commit will need to be backported all the way back to v2.0. Release note (bug fix): Consider intents in a read's uncertainty interval to be uncertain just as if they were committed values. This removes the potential for stale reads when a causally dependent transaction runs into the not-yet resolved intents from a causal ancestor.

40600: storage/engine: return WriteIntentError for intents in uncertainty intervals r=petermattis a=nvanbenschoten This commit fixes the most common failure case in all of the following Jepsen failures. I'm not closing them here though, because they also should be failing due to #36431. Would fix #37394. Would fix #37545. Would fix #37930. Would fix #37932. Would fix #37956. Would fix #38126. Would fix #38763. Before this fix, we were not considering intents in a scan's uncertainty interval to be uncertain. This had the potential to cause stale reads because an unresolved intent doesn't indicate that its transaction hasn’t been committed and isn’t a causal ancestor of the scan. This was causing the `jepsen/multi-register` tests to fail, which I had previously incorrectly attributed entirely to #36431. This commit fixes this by returning `WriteIntentError`s for intents when they are above the read timestamp of a scan but below the max timestamp of a scan. This could have also been fixed by returning `ReadWithinUncertaintyIntervalError`s in this situation. Both would eventually have the same effect, but it seems preferable to kick off concurrency control immediately in this situation and only fall back to uncertainty handling for committed values. If the intent ends up being aborted, this could allow the read to avoid moving its timestamp. This commit will need to be backported all the way back to v2.0. Release note (bug fix): Consider intents in a read's uncertainty interval to be uncertain just as if they were committed values. This removes the potential for stale reads when a causally dependent transaction runs into the not-yet resolved intents from a causal ancestor. 40603: make: pass TESTFLAGS to roachprod-stress, not GOFLAGS r=petermattis a=nvanbenschoten Passing the testflags through the GOFLAGS env var was causing the following error: ``` stringer -output=pkg/sql/opt/rule_name_string.go -type=RuleName pkg/sql/opt/rule_name.go pkg/sql/opt/rule_name.og.go stringer: go [list -f {{context.GOARCH}} {{context.Compiler}} -tags= -- unsafe]: exit status 1: go: parsing $GOFLAGS: non-flag "storage.test" Makefile:1496: recipe for target 'pkg/sql/opt/rule_name_string.go' failed make: *** [pkg/sql/opt/rule_name_string.go] Error 1 make: *** Waiting for unfinished jobs.... ``` Release note: None Co-authored-by: Nathan VanBenschoten <nvanbenschoten@gmail.com>

…tervals This commit fixes the most common failure case in all of the following Jepsen failures. I'm not closing them here though, because they also should be failing due to cockroachdb#36431. Would fix cockroachdb#37394. Would fix cockroachdb#37545. Would fix cockroachdb#37930. Would fix cockroachdb#37932. Would fix cockroachdb#37956. Would fix cockroachdb#38126. Would fix cockroachdb#38763. Before this fix, we were not considering intents in a scan's uncertainty interval to be uncertain. This had the potential to cause stale reads because an unresolved intent doesn't indicate that its transaction hasn’t been committed and isn’t a causal ancestor of the scan. This was causing the `jepsen/multi-register` tests to fail, which I had previously incorrectly attributed entirely to cockroachdb#36431. This commit fixes this by returning `WriteIntentError`s for intents when they are above the read timestamp of a scan but below the max timestamp of a scan. This could have also been fixed by returning `ReadWithinUncertaintyIntervalError`s in this situation. Both would eventually have the same effect, but it seems preferable to kick off concurrency control immediately in this situation and only fall back to uncertainty handling for committed values. If the intent ends up being aborted, this could allow the read to avoid moving its timestamp. This commit will need to be backported all the way back to v2.0. Release note (bug fix): Consider intents in a read's uncertainty interval to be uncertain just as if they were committed values. This removes the potential for stale reads when a causally dependent transaction runs into the not-yet resolved intents from a causal ancestor.

cockroach-teamcity added this to the 19.2 milestone Jun 10, 2019

cockroach-teamcity assigned petermattis Jun 10, 2019

cockroach-teamcity added C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. labels Jun 10, 2019

tbg changed the title ~~roachtest: jepsen-batch3/multi-register/start-stop-2 failed~~ roachtest: jepsen-batch3/multi-register/start-stop-2 failed [hopefully #36431] Jun 19, 2019

tbg assigned nvanbenschoten and unassigned petermattis Jun 19, 2019

nvanbenschoten mentioned this issue Sep 9, 2019

storage/engine: return WriteIntentError for intents in uncertainty intervals #40600

Merged

craig bot closed this as completed in d20419d Sep 9, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

roachtest: jepsen-batch3/multi-register/start-stop-2 failed [hopefully #36431] #38126

roachtest: jepsen-batch3/multi-register/start-stop-2 failed [hopefully #36431] #38126

cockroach-teamcity commented Jun 10, 2019

cockroach-teamcity commented Jun 15, 2019

tbg commented Jun 19, 2019

tbg commented Jun 19, 2019

nvanbenschoten commented Jun 19, 2019

cockroach-teamcity commented Jun 30, 2019

cockroach-teamcity commented Jul 4, 2019

cockroach-teamcity commented Jul 4, 2019

cockroach-teamcity commented Jul 6, 2019

cockroach-teamcity commented Aug 1, 2019

nvanbenschoten commented Aug 1, 2019

roachtest: jepsen-batch3/multi-register/start-stop-2 failed [hopefully #36431] #38126

roachtest: jepsen-batch3/multi-register/start-stop-2 failed [hopefully #36431] #38126

Comments

cockroach-teamcity commented Jun 10, 2019

cockroach-teamcity commented Jun 15, 2019

tbg commented Jun 19, 2019

tbg commented Jun 19, 2019

nvanbenschoten commented Jun 19, 2019

cockroach-teamcity commented Jun 30, 2019

cockroach-teamcity commented Jul 4, 2019

cockroach-teamcity commented Jul 4, 2019

cockroach-teamcity commented Jul 6, 2019

cockroach-teamcity commented Aug 1, 2019

nvanbenschoten commented Aug 1, 2019