Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ccl/logictestccl: TestCCLLogic failed under stress #38864

Closed
cockroach-teamcity opened this issue Jul 14, 2019 · 10 comments · Fixed by #39168
Closed

ccl/logictestccl: TestCCLLogic failed under stress #38864

cockroach-teamcity opened this issue Jul 14, 2019 · 10 comments · Fixed by #39168
Assignees
@solongordon
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Milestone
19.2

Comments

@cockroach-teamcity
Copy link
Member

SHA: https://github.com/cockroachdb/cockroach/commits/02d52a6aa42740d03b696ed1c20a21ff9007568a

Parameters:

TAGS=
GOFLAGS=-parallel=4

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=TestCCLLogic PKG=github.com/cockroachdb/cockroach/pkg/ccl/logictestccl TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1388009&tab=buildLog

=== RUN   TestCCLLogic
--- FAIL: TestCCLLogic (3.92s)
    test_log_scope.go:77: test logs captured to: /tmp/logTestCCLLogic501749576
    test_log_scope.go:58: use -show-logs to present logs inline
=== RUN   TestCCLLogic/5node-dist-opt/zone
=== PAUSE TestCCLLogic/5node-dist-opt/zone
=== CONT  TestCCLLogic/5node-dist-opt/zone
--- done: testdata/logic_test/zone with config 5node-dist-opt: 16 tests, 1 failures
--- total: 293 tests, 0 failures
test logs left over in: /tmp/logTestCCLLogic501749576
        --- FAIL: TestCCLLogic/5node-dist-opt/zone (0.80s)
            logic.go:2338: 
                 
                testdata/logic_test/zone:131: EXPLAIN (OPT, CATALOG) SELECT * FROM t
                expected:
                    TABLE t
                     ├── k int not null
                     ├── v string
                     ├── INDEX primary
                     │    ├── k int not null
                     │    └── ZONE
                     │         └── constraints: [+region=test,+dc=dc1]
                     ├── INDEX secondary
                     │    ├── k int not null
                     │    ├── v string (storing)
                     │    └── ZONE
                     │         └── constraints: [+region=test,+dc=dc2]
                     └── INDEX tertiary
                          ├── k int not null
                          ├── v string (storing)
                          └── ZONE
                               └── constraints: [+region=test,+dc=dc3]
                    scan t
                    
                but found (query options: "") :
                    TABLE t
                     ├── k int not null
                     ├── v string
                     ├── INDEX primary
                     │    ├── k int not null
                     │    └── ZONE
                     │         └── constraints: [+region=test,+dc=dc1]
                     ├── INDEX secondary
                     │    ├── k int not null
                     │    ├── v string (storing)
                     │    └── ZONE
                     │         └── constraints: [+region=test,+dc=dc1]
                     └── INDEX tertiary
                          ├── k int not null
                          ├── v string (storing)
                          └── ZONE
                               └── constraints: [+region=test,+dc=dc3]
                    scan t
                    
                
            logic.go:2370: 
                testdata/logic_test/zone:153: error while processing
            logic.go:2371: testdata/logic_test/zone:153: too many errors encountered, skipping the rest of the input
=== RUN   TestCCLLogic/5node-dist-opt
    --- FAIL: TestCCLLogic/5node-dist-opt (0.00s)
@cockroach-teamcity cockroach-teamcity added this to the 19.2 milestone Jul 14, 2019
@cockroach-teamcity cockroach-teamcity added C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Jul 14, 2019
@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/e3b454452c5cb251958e2e5d91b1854a478614c9

Parameters:

TAGS=
GOFLAGS=-parallel=4

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=TestCCLLogic PKG=github.com/cockroachdb/cockroach/pkg/ccl/logictestccl TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1389565&tab=buildLog

=== RUN   TestCCLLogic
--- FAIL: TestCCLLogic (4.07s)
    test_log_scope.go:77: test logs captured to: /tmp/logTestCCLLogic935431270
    test_log_scope.go:58: use -show-logs to present logs inline
=== RUN   TestCCLLogic/5node-dist-opt/zone
=== PAUSE TestCCLLogic/5node-dist-opt/zone
=== CONT  TestCCLLogic/5node-dist-opt/zone
--- done: testdata/logic_test/zone with config 5node-dist-opt: 16 tests, 1 failures
--- total: 293 tests, 0 failures
test logs left over in: /tmp/logTestCCLLogic935431270
        --- FAIL: TestCCLLogic/5node-dist-opt/zone (0.84s)
            logic.go:2338: 
                 
                testdata/logic_test/zone:131: EXPLAIN (OPT, CATALOG) SELECT * FROM t
                expected:
                    TABLE t
                     ├── k int not null
                     ├── v string
                     ├── INDEX primary
                     │    ├── k int not null
                     │    └── ZONE
                     │         └── constraints: [+region=test,+dc=dc1]
                     ├── INDEX secondary
                     │    ├── k int not null
                     │    ├── v string (storing)
                     │    └── ZONE
                     │         └── constraints: [+region=test,+dc=dc2]
                     └── INDEX tertiary
                          ├── k int not null
                          ├── v string (storing)
                          └── ZONE
                               └── constraints: [+region=test,+dc=dc3]
                    scan t
                    
                but found (query options: "") :
                    TABLE t
                     ├── k int not null
                     ├── v string
                     ├── INDEX primary
                     │    ├── k int not null
                     │    └── ZONE
                     │         └── constraints: [+region=test,+dc=dc1]
                     ├── INDEX secondary
                     │    ├── k int not null
                     │    ├── v string (storing)
                     │    └── ZONE
                     │         └── constraints: [+region=test,+dc=dc1]
                     └── INDEX tertiary
                          ├── k int not null
                          ├── v string (storing)
                          └── ZONE
                               └── constraints: [+region=test,+dc=dc3]
                    scan t
                    
                
            logic.go:2370: 
                testdata/logic_test/zone:153: error while processing
            logic.go:2371: testdata/logic_test/zone:153: too many errors encountered, skipping the rest of the input
=== RUN   TestCCLLogic/5node-dist-opt
    --- FAIL: TestCCLLogic/5node-dist-opt (0.00s)

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/90fc142c8b2bab94f5049e8331e409be74947873

Parameters:

TAGS=
GOFLAGS=-parallel=4

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=TestCCLLogic PKG=github.com/cockroachdb/cockroach/pkg/ccl/logictestccl TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1391659&tab=buildLog

=== RUN   TestCCLLogic
--- FAIL: TestCCLLogic (3.51s)
    test_log_scope.go:77: test logs captured to: /tmp/logTestCCLLogic435790470
    test_log_scope.go:58: use -show-logs to present logs inline
=== RUN   TestCCLLogic/5node-dist-opt/zone
=== PAUSE TestCCLLogic/5node-dist-opt/zone
=== CONT  TestCCLLogic/5node-dist-opt/zone
--- done: testdata/logic_test/zone with config 5node-dist-opt: 16 tests, 1 failures
--- total: 293 tests, 0 failures
test logs left over in: /tmp/logTestCCLLogic435790470
        --- FAIL: TestCCLLogic/5node-dist-opt/zone (0.67s)
            logic.go:2338: 
                 
                testdata/logic_test/zone:131: EXPLAIN (OPT, CATALOG) SELECT * FROM t
                expected:
                    TABLE t
                     ├── k int not null
                     ├── v string
                     ├── INDEX primary
                     │    ├── k int not null
                     │    └── ZONE
                     │         └── constraints: [+region=test,+dc=dc1]
                     ├── INDEX secondary
                     │    ├── k int not null
                     │    ├── v string (storing)
                     │    └── ZONE
                     │         └── constraints: [+region=test,+dc=dc2]
                     └── INDEX tertiary
                          ├── k int not null
                          ├── v string (storing)
                          └── ZONE
                               └── constraints: [+region=test,+dc=dc3]
                    scan t
                    
                but found (query options: "") :
                    TABLE t
                     ├── k int not null
                     ├── v string
                     ├── INDEX primary
                     │    ├── k int not null
                     │    └── ZONE
                     │         └── constraints: [+region=test,+dc=dc1]
                     ├── INDEX secondary
                     │    ├── k int not null
                     │    ├── v string (storing)
                     │    └── ZONE
                     │         └── constraints: [+region=test,+dc=dc1]
                     └── INDEX tertiary
                          ├── k int not null
                          ├── v string (storing)
                          └── ZONE
                               └── constraints: [+region=test,+dc=dc3]
                    scan t
                    
                
            logic.go:2370: 
                testdata/logic_test/zone:153: error while processing
            logic.go:2371: testdata/logic_test/zone:153: too many errors encountered, skipping the rest of the input
=== RUN   TestCCLLogic/5node-dist-opt
    --- FAIL: TestCCLLogic/5node-dist-opt (0.00s)

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/6d644dfb6d38c3ed2e54a548d909950f6b982490

Parameters:

TAGS=
GOFLAGS=-parallel=4

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=TestCCLLogic PKG=github.com/cockroachdb/cockroach/pkg/ccl/logictestccl TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1394068&tab=buildLog

=== RUN   TestCCLLogic
--- FAIL: TestCCLLogic (3.48s)
    test_log_scope.go:77: test logs captured to: /tmp/logTestCCLLogic189973975
    test_log_scope.go:58: use -show-logs to present logs inline
=== RUN   TestCCLLogic/5node-dist-opt/zone
=== PAUSE TestCCLLogic/5node-dist-opt/zone
=== CONT  TestCCLLogic/5node-dist-opt/zone
--- done: testdata/logic_test/zone with config 5node-dist-opt: 16 tests, 1 failures
--- total: 293 tests, 0 failures
test logs left over in: /tmp/logTestCCLLogic189973975
        --- FAIL: TestCCLLogic/5node-dist-opt/zone (0.72s)
            logic.go:2338: 
                 
                testdata/logic_test/zone:131: EXPLAIN (OPT, CATALOG) SELECT * FROM t
                expected:
                    TABLE t
                     ├── k int not null
                     ├── v string
                     ├── INDEX primary
                     │    ├── k int not null
                     │    └── ZONE
                     │         └── constraints: [+region=test,+dc=dc1]
                     ├── INDEX secondary
                     │    ├── k int not null
                     │    ├── v string (storing)
                     │    └── ZONE
                     │         └── constraints: [+region=test,+dc=dc2]
                     └── INDEX tertiary
                          ├── k int not null
                          ├── v string (storing)
                          └── ZONE
                               └── constraints: [+region=test,+dc=dc3]
                    scan t
                    
                but found (query options: "") :
                    TABLE t
                     ├── k int not null
                     ├── v string
                     ├── INDEX primary
                     │    ├── k int not null
                     │    └── ZONE
                     │         └── constraints: [+region=test,+dc=dc1]
                     ├── INDEX secondary
                     │    ├── k int not null
                     │    ├── v string (storing)
                     │    └── ZONE
                     │         └── constraints: [+region=test,+dc=dc1]
                     └── INDEX tertiary
                          ├── k int not null
                          ├── v string (storing)
                          └── ZONE
                               └── constraints: [+region=test,+dc=dc3]
                    scan t
                    
                
            logic.go:2370: 
                testdata/logic_test/zone:153: error while processing
            logic.go:2371: testdata/logic_test/zone:153: too many errors encountered, skipping the rest of the input
=== RUN   TestCCLLogic/5node-dist-opt
    --- FAIL: TestCCLLogic/5node-dist-opt (0.00s)

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/7dab0dcfd37c389af357c302c073b9611b5ada25

Parameters:

TAGS=
GOFLAGS=-parallel=4

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=TestCCLLogic PKG=github.com/cockroachdb/cockroach/pkg/ccl/logictestccl TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1398252&tab=buildLog

=== RUN   TestCCLLogic
--- FAIL: TestCCLLogic (3.51s)
    test_log_scope.go:77: test logs captured to: /tmp/logTestCCLLogic157301111
    test_log_scope.go:58: use -show-logs to present logs inline
=== RUN   TestCCLLogic/5node-dist-opt/zone
=== PAUSE TestCCLLogic/5node-dist-opt/zone
=== CONT  TestCCLLogic/5node-dist-opt/zone
--- done: testdata/logic_test/zone with config 5node-dist-opt: 16 tests, 1 failures
--- total: 293 tests, 0 failures
test logs left over in: /tmp/logTestCCLLogic157301111
        --- FAIL: TestCCLLogic/5node-dist-opt/zone (0.75s)
            logic.go:2338: 
                 
                testdata/logic_test/zone:131: EXPLAIN (OPT, CATALOG) SELECT * FROM t
                expected:
                    TABLE t
                     ├── k int not null
                     ├── v string
                     ├── INDEX primary
                     │    ├── k int not null
                     │    └── ZONE
                     │         └── constraints: [+region=test,+dc=dc1]
                     ├── INDEX secondary
                     │    ├── k int not null
                     │    ├── v string (storing)
                     │    └── ZONE
                     │         └── constraints: [+region=test,+dc=dc2]
                     └── INDEX tertiary
                          ├── k int not null
                          ├── v string (storing)
                          └── ZONE
                               └── constraints: [+region=test,+dc=dc3]
                    scan t
                    
                but found (query options: "") :
                    TABLE t
                     ├── k int not null
                     ├── v string
                     ├── INDEX primary
                     │    ├── k int not null
                     │    └── ZONE
                     │         └── constraints: [+region=test,+dc=dc1]
                     ├── INDEX secondary
                     │    ├── k int not null
                     │    ├── v string (storing)
                     │    └── ZONE
                     │         └── constraints: [+region=test,+dc=dc1]
                     └── INDEX tertiary
                          ├── k int not null
                          ├── v string (storing)
                          └── ZONE
                               └── constraints: [+region=test,+dc=dc3]
                    scan t
                    
                
            logic.go:2370: 
                testdata/logic_test/zone:153: error while processing
            logic.go:2371: testdata/logic_test/zone:153: too many errors encountered, skipping the rest of the input
=== RUN   TestCCLLogic/5node-dist-opt
    --- FAIL: TestCCLLogic/5node-dist-opt (0.00s)

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/86eab2ff0a1a4c2d9b5f7e7a45deda74c98c6c37

Parameters:

TAGS=
GOFLAGS=-parallel=4

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=TestCCLLogic PKG=github.com/cockroachdb/cockroach/pkg/ccl/logictestccl TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1402622&tab=buildLog

=== RUN   TestCCLLogic
--- FAIL: TestCCLLogic (3.69s)
    test_log_scope.go:77: test logs captured to: /tmp/logTestCCLLogic202392037
    test_log_scope.go:58: use -show-logs to present logs inline
=== RUN   TestCCLLogic/5node-dist-opt/zone
=== PAUSE TestCCLLogic/5node-dist-opt/zone
=== CONT  TestCCLLogic/5node-dist-opt/zone
--- done: testdata/logic_test/zone with config 5node-dist-opt: 16 tests, 1 failures
--- total: 293 tests, 0 failures
test logs left over in: /tmp/logTestCCLLogic202392037
        --- FAIL: TestCCLLogic/5node-dist-opt/zone (0.81s)
            logic.go:2338: 
                 
                testdata/logic_test/zone:131: EXPLAIN (OPT, CATALOG) SELECT * FROM t
                expected:
                    TABLE t
                     ├── k int not null
                     ├── v string
                     ├── INDEX primary
                     │    ├── k int not null
                     │    └── ZONE
                     │         └── constraints: [+region=test,+dc=dc1]
                     ├── INDEX secondary
                     │    ├── k int not null
                     │    ├── v string (storing)
                     │    └── ZONE
                     │         └── constraints: [+region=test,+dc=dc2]
                     └── INDEX tertiary
                          ├── k int not null
                          ├── v string (storing)
                          └── ZONE
                               └── constraints: [+region=test,+dc=dc3]
                    scan t
                    
                but found (query options: "") :
                    TABLE t
                     ├── k int not null
                     ├── v string
                     ├── INDEX primary
                     │    ├── k int not null
                     │    └── ZONE
                     │         └── constraints: [+region=test,+dc=dc1]
                     ├── INDEX secondary
                     │    ├── k int not null
                     │    ├── v string (storing)
                     │    └── ZONE
                     │         └── constraints: [+region=test,+dc=dc1]
                     └── INDEX tertiary
                          ├── k int not null
                          ├── v string (storing)
                          └── ZONE
                               └── constraints: [+region=test,+dc=dc3]
                    scan t
                    
                
            logic.go:2370: 
                testdata/logic_test/zone:153: error while processing
            logic.go:2371: testdata/logic_test/zone:153: too many errors encountered, skipping the rest of the input
=== RUN   TestCCLLogic/5node-dist-opt
    --- FAIL: TestCCLLogic/5node-dist-opt (0.00s)

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/ff04012ed2d2c0c8e30e4de106ca0a350bca8c3e

Parameters:

TAGS=
GOFLAGS=-parallel=4

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=TestCCLLogic PKG=github.com/cockroachdb/cockroach/pkg/ccl/logictestccl TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1404935&tab=buildLog

=== RUN   TestCCLLogic
--- FAIL: TestCCLLogic (3.39s)
    test_log_scope.go:77: test logs captured to: /tmp/logTestCCLLogic207829185
    test_log_scope.go:58: use -show-logs to present logs inline
=== RUN   TestCCLLogic/5node-dist-opt
    --- FAIL: TestCCLLogic/5node-dist-opt (0.00s)
=== RUN   TestCCLLogic/5node-dist-opt/zone
=== PAUSE TestCCLLogic/5node-dist-opt/zone
=== CONT  TestCCLLogic/5node-dist-opt/zone
--- done: testdata/logic_test/zone with config 5node-dist-opt: 16 tests, 1 failures
--- total: 293 tests, 0 failures
test logs left over in: /tmp/logTestCCLLogic207829185
        --- FAIL: TestCCLLogic/5node-dist-opt/zone (0.78s)
            logic.go:2365: 
                 
                testdata/logic_test/zone:131: EXPLAIN (OPT, CATALOG) SELECT * FROM t
                expected:
                    TABLE t
                     ├── k int not null
                     ├── v string
                     ├── INDEX primary
                     │    ├── k int not null
                     │    └── ZONE
                     │         └── constraints: [+region=test,+dc=dc1]
                     ├── INDEX secondary
                     │    ├── k int not null
                     │    ├── v string (storing)
                     │    └── ZONE
                     │         └── constraints: [+region=test,+dc=dc2]
                     └── INDEX tertiary
                          ├── k int not null
                          ├── v string (storing)
                          └── ZONE
                               └── constraints: [+region=test,+dc=dc3]
                    scan t
                    
                but found (query options: "") :
                    TABLE t
                     ├── k int not null
                     ├── v string
                     ├── INDEX primary
                     │    ├── k int not null
                     │    └── ZONE
                     │         └── constraints: [+region=test,+dc=dc1]
                     ├── INDEX secondary
                     │    ├── k int not null
                     │    ├── v string (storing)
                     │    └── ZONE
                     │         └── constraints: [+region=test,+dc=dc1]
                     └── INDEX tertiary
                          ├── k int not null
                          ├── v string (storing)
                          └── ZONE
                               └── constraints: [+region=test,+dc=dc3]
                    scan t
                    
                
            logic.go:2397: 
                testdata/logic_test/zone:153: error while processing
            logic.go:2398: testdata/logic_test/zone:153: too many errors encountered, skipping the rest of the input

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/28bb1e92852177708440f5796e8c2bf21c0429ef

Parameters:

TAGS=
GOFLAGS=-parallel=4

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=TestCCLLogic PKG=github.com/cockroachdb/cockroach/pkg/ccl/logictestccl TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1406994&tab=buildLog

=== RUN   TestCCLLogic
--- FAIL: TestCCLLogic (4.05s)
    test_log_scope.go:77: test logs captured to: /tmp/logTestCCLLogic300534004
    test_log_scope.go:58: use -show-logs to present logs inline
=== RUN   TestCCLLogic/5node-dist-opt/zone
=== PAUSE TestCCLLogic/5node-dist-opt/zone
=== CONT  TestCCLLogic/5node-dist-opt/zone
--- done: testdata/logic_test/zone with config 5node-dist-opt: 16 tests, 1 failures
--- total: 341 tests, 0 failures
test logs left over in: /tmp/logTestCCLLogic300534004
        --- FAIL: TestCCLLogic/5node-dist-opt/zone (0.69s)
            logic.go:2365: 
                 
                testdata/logic_test/zone:131: EXPLAIN (OPT, CATALOG) SELECT * FROM t
                expected:
                    TABLE t
                     ├── k int not null
                     ├── v string
                     ├── INDEX primary
                     │    ├── k int not null
                     │    └── ZONE
                     │         └── constraints: [+region=test,+dc=dc1]
                     ├── INDEX secondary
                     │    ├── k int not null
                     │    ├── v string (storing)
                     │    └── ZONE
                     │         └── constraints: [+region=test,+dc=dc2]
                     └── INDEX tertiary
                          ├── k int not null
                          ├── v string (storing)
                          └── ZONE
                               └── constraints: [+region=test,+dc=dc3]
                    scan t
                    
                but found (query options: "") :
                    TABLE t
                     ├── k int not null
                     ├── v string
                     ├── INDEX primary
                     │    ├── k int not null
                     │    └── ZONE
                     │         └── constraints: [+region=test,+dc=dc1]
                     ├── INDEX secondary
                     │    ├── k int not null
                     │    ├── v string (storing)
                     │    └── ZONE
                     │         └── constraints: [+region=test,+dc=dc1]
                     └── INDEX tertiary
                          ├── k int not null
                          ├── v string (storing)
                          └── ZONE
                               └── constraints: [+region=test,+dc=dc3]
                    scan t
                    
                
            logic.go:2397: 
                testdata/logic_test/zone:153: error while processing
            logic.go:2398: testdata/logic_test/zone:153: too many errors encountered, skipping the rest of the input
=== RUN   TestCCLLogic/5node-dist-opt
    --- FAIL: TestCCLLogic/5node-dist-opt (0.00s)

@jordanlewis jordanlewis assigned solongordon and unassigned mberhault Jul 26, 2019
@jordanlewis
Copy link
Member

@solongordon / @rohany could you take a look at this? Looks like the index secondary is not getting constrained to dc2 some of the time, but only under stress. Maybe it's just a race condition and it takes a little while to apply the constraint, I'm not too sure, but investigating this to at least cause the test to stop failing would be good.

@rohany rohany mentioned this issue Jul 26, 2019
Closed
@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/65055d6c16bf9386d8c4f4f9cd23e0a848814dc9

Parameters:

TAGS=
GOFLAGS=-parallel=4

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=TestCCLLogic PKG=github.com/cockroachdb/cockroach/pkg/ccl/logictestccl TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1411255&tab=buildLog

=== RUN   TestCCLLogic
--- FAIL: TestCCLLogic (4.36s)
    test_log_scope.go:77: test logs captured to: /tmp/logTestCCLLogic504738078
    test_log_scope.go:58: use -show-logs to present logs inline
=== RUN   TestCCLLogic/5node-dist-opt/zone
=== PAUSE TestCCLLogic/5node-dist-opt/zone
=== CONT  TestCCLLogic/5node-dist-opt/zone
--- done: testdata/logic_test/zone with config 5node-dist-opt: 16 tests, 1 failures
--- total: 341 tests, 0 failures
test logs left over in: /tmp/logTestCCLLogic504738078
        --- FAIL: TestCCLLogic/5node-dist-opt/zone (0.77s)
            logic.go:2365: 
                 
                testdata/logic_test/zone:131: EXPLAIN (OPT, CATALOG) SELECT * FROM t
                expected:
                    TABLE t
                     ├── k int not null
                     ├── v string
                     ├── INDEX primary
                     │    ├── k int not null
                     │    └── ZONE
                     │         └── constraints: [+region=test,+dc=dc1]
                     ├── INDEX secondary
                     │    ├── k int not null
                     │    ├── v string (storing)
                     │    └── ZONE
                     │         └── constraints: [+region=test,+dc=dc2]
                     └── INDEX tertiary
                          ├── k int not null
                          ├── v string (storing)
                          └── ZONE
                               └── constraints: [+region=test,+dc=dc3]
                    scan t
                    
                but found (query options: "") :
                    TABLE t
                     ├── k int not null
                     ├── v string
                     ├── INDEX primary
                     │    ├── k int not null
                     │    └── ZONE
                     │         └── constraints: [+region=test,+dc=dc1]
                     ├── INDEX secondary
                     │    ├── k int not null
                     │    ├── v string (storing)
                     │    └── ZONE
                     │         └── constraints: [+region=test,+dc=dc1]
                     └── INDEX tertiary
                          ├── k int not null
                          ├── v string (storing)
                          └── ZONE
                               └── constraints: [+region=test,+dc=dc3]
                    scan t
                    
                
            logic.go:2397: 
                testdata/logic_test/zone:153: error while processing
            logic.go:2398: testdata/logic_test/zone:153: too many errors encountered, skipping the rest of the input
=== RUN   TestCCLLogic/5node-dist-opt
    --- FAIL: TestCCLLogic/5node-dist-opt (0.00s)

solongordon added a commit to solongordon/cockroach that referenced this issue Jul 30, 2019
@solongordon
logictestccl: fix zone failures under stress
98b302f 
The 'zone' logic test was failing under stress due to a race between
gossiping zone config changes and running explain plans. As suggested by
an existing comment, I added the "retry" option to more tests to account
for this potential race.

Fixes cockroachdb#38864

Release note: None
@solongordon solongordon mentioned this issue Jul 30, 2019
Merged
@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/da56c792e968574b8f1d9ef3fdb45d56a530221a

Parameters:

TAGS=
GOFLAGS=-parallel=4

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=TestCCLLogic PKG=github.com/cockroachdb/cockroach/pkg/ccl/logictestccl TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1415627&tab=buildLog

=== RUN   TestCCLLogic
--- FAIL: TestCCLLogic (4.82s)
    test_log_scope.go:77: test logs captured to: /tmp/logTestCCLLogic860484529
    test_log_scope.go:58: use -show-logs to present logs inline
=== RUN   TestCCLLogic/5node-dist-opt/zone
=== PAUSE TestCCLLogic/5node-dist-opt/zone
=== CONT  TestCCLLogic/5node-dist-opt/zone
--- done: testdata/logic_test/zone with config 5node-dist-opt: 16 tests, 1 failures
--- total: 363 tests, 0 failures
test logs left over in: /tmp/logTestCCLLogic860484529
        --- FAIL: TestCCLLogic/5node-dist-opt/zone (0.78s)
            logic.go:2365: 
                 
                testdata/logic_test/zone:131: EXPLAIN (OPT, CATALOG) SELECT * FROM t
                expected:
                    TABLE t
                     ├── k int not null
                     ├── v string
                     ├── INDEX primary
                     │    ├── k int not null
                     │    └── ZONE
                     │         └── constraints: [+region=test,+dc=dc1]
                     ├── INDEX secondary
                     │    ├── k int not null
                     │    ├── v string (storing)
                     │    └── ZONE
                     │         └── constraints: [+region=test,+dc=dc2]
                     └── INDEX tertiary
                          ├── k int not null
                          ├── v string (storing)
                          └── ZONE
                               └── constraints: [+region=test,+dc=dc3]
                    scan t
                    
                but found (query options: "") :
                    TABLE t
                     ├── k int not null
                     ├── v string
                     ├── INDEX primary
                     │    ├── k int not null
                     │    └── ZONE
                     │         └── constraints: [+region=test,+dc=dc1]
                     ├── INDEX secondary
                     │    ├── k int not null
                     │    ├── v string (storing)
                     │    └── ZONE
                     │         └── constraints: [+region=test,+dc=dc1]
                     └── INDEX tertiary
                          ├── k int not null
                          ├── v string (storing)
                          └── ZONE
                               └── constraints: [+region=test,+dc=dc3]
                    scan t
                    
                
            logic.go:2397: 
                testdata/logic_test/zone:153: error while processing
            logic.go:2398: testdata/logic_test/zone:153: too many errors encountered, skipping the rest of the input
=== RUN   TestCCLLogic/5node-dist-opt
    --- FAIL: TestCCLLogic/5node-dist-opt (0.00s)

craig bot pushed a commit that referenced this issue Aug 1, 2019
@solongordon
Merge #39168
8282250 
39168: logictestccl: fix zone failures under stress r=solongordon a=solongordon

The 'zone' logic test was failing under stress due to a race between
gossiping zone config changes and running explain plans. As suggested by
an existing comment, I added the "retry" option to more tests to account
for this potential race.

Fixes #38864

Release note: None

Co-authored-by: Solon Gordon <solon@cockroachlabs.com>
@craig craig bot closed this as completed in #39168 Aug 1, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@solongordon solongordon

Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Milestone
19.2
Development

Successfully merging a pull request may close this issue.

logictestccl: fix zone failures under stress solongordon/cockroach
4 participants
@jordanlewis @solongordon @mberhault @cockroach-teamcity