Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: tpchbench/tpch/nodes=3/cpu=4/sf=1 failed #40461

Closed
cockroach-teamcity opened this issue Sep 4, 2019 · 2 comments · Fixed by #40495
Closed

roachtest: tpchbench/tpch/nodes=3/cpu=4/sf=1 failed #40461

cockroach-teamcity opened this issue Sep 4, 2019 · 2 comments · Fixed by #40495
Assignees
@rohany
Labels
C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot.
Milestone
19.2

Comments

@cockroach-teamcity
Copy link
Member

SHA: https://github.com/cockroachdb/cockroach/commits/96b1500e20575ee5c609a00857c78c918078c99b

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpchbench/tpch/nodes=3/cpu=4/sf=1 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1465459&tab=buildLog

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190904-1465459/tpchbench/tpch/nodes=3/cpu=4/sf=1/run_1
	tpchbench.go:123,cluster.go:2091,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1567578879-16-n4cpu4:4 -- ./workload run querybench --db=tpch --concurrency=1 --query-file=tpch --num-runs=3 --max-ops=63 --vectorize= {pgurl:1-3} --histograms=perf/stats.json --histograms-max-latency=8m20s returned:
		stderr:
		
		stdout:
		0 AND l_shipmode IN ('AIR', 'AIR REG') AND l_shipinstruct = 'DELIVER IN PERSON') OR ( p_partkey = l_partkey AND p_brand = 'Brand#34' AND p_container IN ('LG CASE', 'LG BOX', 'LG PACK', 'LG PKG') AND l_quantity >= 20 AND l_quantity <= 20 + 10 AND p_size BETWEEN 1 AND 15 AND l_shipmode IN ('AIR', 'AIR REG') AND l_shipinstruct = 'DELIVER IN PERSON')
		  412.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 19: SELECT s_name, s_address FROM supplier, nation WHERE s_suppkey IN ( SELECT ps_suppkey FROM partsupp WHERE ps_partkey IN ( SELECT p_partkey FROM part WHERE p_name LIKE 'forest%') AND ps_availqty > ( SELECT 0.5 * sum(l_quantity) FROM lineitem WHERE l_partkey = ps_partkey AND l_suppkey = ps_suppkey AND l_shipdate >= DATE '1994-01-01' AND l_shipdate < DATE '1994-01-01' + INTERVAL '1' YEAR)) AND s_nationkey = n_nationkey AND n_name = 'CANADA' ORDER BY s_name
		Error: pq: internal error: unexpected error from the vectorized runtime: runtime error: index out of range
		Error:  exit status 1
		: exit status 1
@cockroach-teamcity cockroach-teamcity added C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. labels Sep 4, 2019
@cockroach-teamcity cockroach-teamcity added this to the 19.2 milestone Sep 4, 2019
@jordanlewis
Copy link
Member 

pq: internal error: unexpected error from the vectorized runtime: runtime error: index out of range
DETAIL: stack trace:
/go/src/github.com/cockroachdb/cockroach/pkg/sql/exec/execerror/error.go:70: func1()
/usr/local/go/src/runtime/panic.go:522: gopanic()
/usr/local/go/src/runtime/panic.go:44: panicindex()
/go/src/github.com/cockroachdb/cockroach/pkg/sql/exec/cast.eg.go:2968: Next()
/go/src/github.com/cockroachdb/cockroach/pkg/sql/exec/projection_ops.eg.go:106150: Next()
/go/src/github.com/cockroachdb/cockroach/pkg/sql/exec/simple_project.go:93: Next()
/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/materializer.go:144: next()
/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/materializer.go:135: nextAdapter()
/go/src/github.com/cockroachdb/cockroach/pkg/sql/exec/execerror/error.go:91: CatchVectorizedRuntimeError()
/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/materializer.go:179: Next()
/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/base.go:171: Run()
/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/processors.go:793: Run()
/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/flow.go:584: Run()
/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:337: Run()
/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:880: planAndRunSubquery()
/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:784: PlanAndRunSubqueries()
/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:807: execWithDistSQLEngine()
/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:708: dispatchToExecutionEngine()
/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:417: execStmtInOpenState()
/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:98: execStmt()
/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1238: execCmd()
/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1167: run()
/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:444: ServeConn()
/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:584: func1()
/usr/local/go/src/runtime/asm_amd64.s:1337: goexit()

HINT: You have encountered an unexpected error.

Please check the public issue tracker to check whether this problem is
already tracked. If you cannot find it there, please report the error
with details by creating a new issue.

If you would rather not post publicly, please contact us directly
using the support form.

We appreciate your feedback.

@rohany rohany mentioned this issue Sep 4, 2019
Merged
@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/c3f447650742706c52354f629b30bb64e5a6959c

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpchbench/tpch/nodes=3/cpu=4/sf=1 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1468445&tab=buildLog

The test failed on branch=provisional_201909041806_v19.2.0-beta.20190910, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190905-1468445/tpchbench/tpch/nodes=3/cpu=4/sf=1/run_1
	tpchbench.go:123,cluster.go:2091,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1567652906-15-n4cpu4:4 -- ./workload run querybench --db=tpch --concurrency=1 --query-file=tpch --num-runs=3 --max-ops=63 --vectorize= {pgurl:1-3} --histograms=perf/stats.json --histograms-max-latency=8m20s returned:
		stderr:
		
		stdout:
		0 AND l_shipmode IN ('AIR', 'AIR REG') AND l_shipinstruct = 'DELIVER IN PERSON') OR ( p_partkey = l_partkey AND p_brand = 'Brand#34' AND p_container IN ('LG CASE', 'LG BOX', 'LG PACK', 'LG PKG') AND l_quantity >= 20 AND l_quantity <= 20 + 10 AND p_size BETWEEN 1 AND 15 AND l_shipmode IN ('AIR', 'AIR REG') AND l_shipinstruct = 'DELIVER IN PERSON')
		 1573.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 19: SELECT s_name, s_address FROM supplier, nation WHERE s_suppkey IN ( SELECT ps_suppkey FROM partsupp WHERE ps_partkey IN ( SELECT p_partkey FROM part WHERE p_name LIKE 'forest%') AND ps_availqty > ( SELECT 0.5 * sum(l_quantity) FROM lineitem WHERE l_partkey = ps_partkey AND l_suppkey = ps_suppkey AND l_shipdate >= DATE '1994-01-01' AND l_shipdate < DATE '1994-01-01' + INTERVAL '1' YEAR)) AND s_nationkey = n_nationkey AND n_name = 'CANADA' ORDER BY s_name
		Error: pq: internal error: unexpected error from the vectorized runtime: runtime error: index out of range
		Error:  exit status 1
		: exit status 1

rohany added a commit to rohany/cockroach that referenced this issue Sep 5, 2019
@rohany
exec: Fix indexing panic in cast
66dd5e0 
The cast operator could panic sometimes due to mismatched
column sizes in a batch. Re-slicing the column to the length
of the batch should ensure that this cannot happen anymore.

Fixes cockroachdb#40461.

Release note: None
@rohany rohany mentioned this issue Sep 5, 2019
Closed
craig bot pushed a commit that referenced this issue Sep 5, 2019
@rohany
Merge #40495
0fa30e5 
40495: exec: Bug fixes in vectorized casts r=rohany a=rohany

* Fixed a bug where the appropriate setters and getters were not being
generated for the casts.

* Fixed a bug where columns were not getting sliced, causing column
length mismatches.

Fixes #40461.

Release note: None

Co-authored-by: Rohan Yadav <rohany@alumni.cmu.edu>
@craig craig bot closed this as completed in 8bb4562 Sep 5, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rohany rohany

Labels
C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot.
Projects
None yet
Milestone
19.2
3 participants
@jordanlewis @rohany @cockroach-teamcity