Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

roachtest: cdc/sink-chaos/rangefeed=false failed (skipped on release-19.1) #36979

Closed
cockroach-teamcity opened this issue Apr 21, 2019 · 1 comment · Fixed by #37498
Closed
Assignees
Labels
C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot.
Milestone

Comments

@cockroach-teamcity
Copy link
Member

SHA: https://github.com/cockroachdb/cockroach/commits/df200cbf3f407dbf349aa601ff9036b4dff88e83

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=cdc/sink-chaos/rangefeed=false PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1252822&tab=buildLog

The test failed on release-2.1:
	cluster.go:1329,cdc.go:746,cdc.go:135,cluster.go:1667,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1252822-cdc-sink-chaos-rangefeed-false:4 -- ./workload run tpcc --warehouses=100 --duration=30m --tolerate-errors {pgurl:1-3}  returned:
		stderr:
		
		stdout:
		l
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		   2m21s        0            0.0            2.3      0.0      0.0      0.0      0.0 delivery
		   2m21s        0           16.0           20.9     28.3     37.7     39.8     39.8 newOrder
		   2m21s        0            3.0            2.5      6.3      6.6      6.6      6.6 orderStatus
		   2m21s        0           21.0           23.0     14.7     18.9     23.1     23.1 payment
		   2m21s        0            4.0            2.3     10.0     14.2     14.2     14.2 stockLevel
		   2m22s        0            2.0            2.3     35.7     50.3     50.3     50.3 delivery
		   2m22s        0           27.0           21.0     28.3     35.7     37.7     37.7 newOrder
		   2m22s        0            0.0            2.5      0.0      0.0      0.0      0.0 orderStatus
		   2m22s        0           16.0           23.0     15.7     19.9     21.0     21.0 payment
		   2m22s        0            1.0            2.3     11.5     11.5     11.5     11.5 stockLevel
		: signal: killed
	cluster.go:1688,cdc.go:223,cdc.go:530,test.go:1237: unexpected status: failed

@cockroach-teamcity cockroach-teamcity added this to the 19.1 milestone Apr 21, 2019
@cockroach-teamcity cockroach-teamcity added C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. labels Apr 21, 2019
@danhhz danhhz self-assigned this Apr 22, 2019
@danhhz
Copy link
Contributor

danhhz commented Apr 22, 2019

This was on 2.1, which is almost certainly missing a bunch of backports for surviving sink chaos

@danhhz danhhz changed the title roachtest: cdc/sink-chaos/rangefeed=false failed roachtest: cdc/sink-chaos/rangefeed=false failed (skipped on release-19.1) Apr 23, 2019
danhhz added a commit to danhhz/cockroach that referenced this issue May 13, 2019
Also make the crdb-chaos test more lenient to recovery times to work
around the flakes we're seeing. This thread should probably get pulled
at some point so leaving cockroachdb#36879 open to track it.

Closes cockroachdb#36905
Closes cockroachdb#36979

Release note: None
craig bot pushed a commit that referenced this issue May 13, 2019
37498: roachtest: unskip cdc/{crdb,sink}-chaos on 19.1 r=tbg a=danhhz

Also make the crdb-chaos test more lenient to recovery times to work
around the flakes we're seeing. This thread should probably get pulled
at some point so leaving #36879 open to track it.

Closes #36905
Closes #36979

Release note: None

Co-authored-by: Daniel Harrison <daniel.harrison@gmail.com>
@craig craig bot closed this as completed in #37498 May 13, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants