Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

flaky test TestCTEIterationMemTracker #54379

Closed
hawkingrei opened this issue Jul 2, 2024 · 4 comments · Fixed by #54380, #54400 or #54523
Closed

flaky test TestCTEIterationMemTracker #54379

hawkingrei opened this issue Jul 2, 2024 · 4 comments · Fixed by #54380, #54400 or #54523
Assignees
@guo-shaoge
Labels
affects-5.4 This bug affects the 5.4.x(LTS) versions. affects-6.1 This bug affects the 6.1.x(LTS) versions. affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. affects-8.1 This bug affects the 8.1.x(LTS) versions. component/executor component/test severity/major sig/execution SIG execution type/bug The issue is confirmed as a bug.

Comments

@hawkingrei
Copy link
Member

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

2. What did you expect to see? (Required)

3. What did you see instead (Required)

    testkit.go:274: 
        	Error Trace:	pkg/testkit/testkit.go:274
        	            				pkg/testkit/testkit.go:199
        	            				pkg/testkit/testkit.go:174
        	            				pkg/executor/cte_test.go:227
        	Error:      	Received unexpected error:
        	            	assert row container spill disk failed
        	            	github.com/pingcap/tidb/pkg/util.GetRecoverError
        	            		pkg/util/util.go:305
        	            	github.com/pingcap/tidb/pkg/executor.(*cteProducer).computeRecursivePart.func1
        	            		pkg/executor/cte.go:416
        	            	runtime.gopanic
        	            		GOROOT/src/runtime/panic.go:914
        	            	github.com/pingcap/tidb/pkg/executor.(*cteProducer).computeRecursivePart
        	            		pkg/executor/cte.go:450
        	            	github.com/pingcap/tidb/pkg/executor.(*cteProducer).produce
        	            		pkg/executor/cte.go:366
        	            	github.com/pingcap/tidb/pkg/executor.(*CTEExec).Next
        	            		pkg/executor/cte.go:113
        	            	github.com/pingcap/tidb/pkg/executor/internal/exec.Next
        	            		pkg/executor/internal/exec/executor.go:410
        	            	github.com/pingcap/tidb/pkg/executor.(*ExplainExec).executeAnalyzeExec
        	            		pkg/executor/explain.go:130
        	            	github.com/pingcap/tidb/pkg/executor.(*ExplainExec).generateExplainInfo
        	            		pkg/executor/explain.go:149
        	            	github.com/pingcap/tidb/pkg/executor.(*ExplainExec).Next
        	            		pkg/executor/explain.go:76
        	            	github.com/pingcap/tidb/pkg/executor/internal/exec.Next
        	            		pkg/executor/internal/exec/executor.go:410
        	            	github.com/pingcap/tidb/pkg/executor.(*ExecStmt).next
        	            		pkg/executor/adapter.go:1250
        	            	github.com/pingcap/tidb/pkg/executor.(*recordSet).Next
        	            		pkg/executor/adapter.go:175
        	            	github.com/pingcap/tidb/pkg/session.GetRows4Test
        	            		pkg/session/tidb.go:346
        	            	github.com/pingcap/tidb/pkg/session.ResultSetToStringSlice
        	            		pkg/session/tidb.go:364
        	            	github.com/pingcap/tidb/pkg/testkit.(*TestKit).ResultSetToResultWithCtx
        	            		pkg/testkit/testkit.go:273
        	            	github.com/pingcap/tidb/pkg/testkit.(*TestKit).MustQueryWithContext
        	            		pkg/testkit/testkit.go:199
        	            	github.com/pingcap/tidb/pkg/testkit.(*TestKit).MustQuery
        	            		pkg/testkit/testkit.go:174
        	            	github.com/pingcap/tidb/pkg/executor_test.TestCTEIterationMemTracker
        	            		pkg/executor/cte_test.go:227
        	            	testing.tRunner
        	            		GOROOT/src/testing/testing.go:1595
        	            	runtime.goexit
        	            		src/runtime/asm_amd64.s:1650
        	Test:       	TestCTEIterationMemTracker
        	Messages:   	sql:explain analyze with recursive cte1 as (select c1 from t1 union all select c1 + 1 c1 from cte1 where c1 < 5000) select * from cte1, args:[]

4. What is your TiDB version? (Required)
@hawkingrei hawkingrei added the type/bug The issue is confirmed as a bug. label Jul 2, 2024
@guo-shaoge guo-shaoge mentioned this issue Jul 2, 2024
Merged
13 tasks
@ti-chi-bot ti-chi-bot bot closed this as completed in 7fea27e Jul 2, 2024
@guo-shaoge guo-shaoge mentioned this issue Jul 3, 2024
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Jul 3, 2024
@guo-shaoge
executor: fix cte mem tracker failpoint condition (#54400)
2b6595d 
close #54379
@lance6716
Copy link
Contributor

not fixed

https://do.pingcap.net/jenkins/blue/organizations/jenkins/pingcap%2Ftidb%2Fghpr_unit_test/detail/ghpr_unit_test/12842/pipeline

    testkit.go:274: 
        	Error Trace:	pkg/testkit/testkit.go:274
        	            				pkg/testkit/testkit.go:199
        	            				pkg/testkit/testkit.go:174
        	            				pkg/executor/test/cte/cte_test.go:227
        	Error:      	Received unexpected error:
        	            	assert row container spill disk failed
        	            	github.com/pingcap/tidb/pkg/util.GetRecoverError
        	            		pkg/util/util.go:305
        	            	github.com/pingcap/tidb/pkg/executor.(*cteProducer).computeRecursivePart.func1
        	            		pkg/executor/cte.go:417
        	            	runtime.gopanic
        	            		GOROOT/src/runtime/panic.go:914
        	            	github.com/pingcap/tidb/pkg/executor.(*cteProducer).computeRecursivePart
        	            		pkg/executor/cte.go:449
        	            	github.com/pingcap/tidb/pkg/executor.(*cteProducer).produce
        	            		pkg/executor/cte.go:367
        	            	github.com/pingcap/tidb/pkg/executor.(*CTEExec).Next
        	            		pkg/executor/cte.go:114
        	            	github.com/pingcap/tidb/pkg/executor/internal/exec.Next
        	            		pkg/executor/internal/exec/executor.go:410
        	            	github.com/pingcap/tidb/pkg/executor.(*ExplainExec).executeAnalyzeExec
        	            		pkg/executor/explain.go:130
        	            	github.com/pingcap/tidb/pkg/executor.(*ExplainExec).generateExplainInfo
        	            		pkg/executor/explain.go:149
        	            	github.com/pingcap/tidb/pkg/executor.(*ExplainExec).Next
        	            		pkg/executor/explain.go:76
        	            	github.com/pingcap/tidb/pkg/executor/internal/exec.Next
        	            		pkg/executor/internal/exec/executor.go:410
        	            	github.com/pingcap/tidb/pkg/executor.(*ExecStmt).next
        	            		pkg/executor/adapter.go:1241
        	            	github.com/pingcap/tidb/pkg/executor.(*recordSet).Next
        	            		pkg/executor/adapter.go:169
        	            	github.com/pingcap/tidb/pkg/session.GetRows4Test
        	            		pkg/session/tidb.go:346
        	            	github.com/pingcap/tidb/pkg/session.ResultSetToStringSlice
        	            		pkg/session/tidb.go:364
        	            	github.com/pingcap/tidb/pkg/testkit.(*TestKit).ResultSetToResultWithCtx
        	            		pkg/testkit/testkit.go:273
        	            	github.com/pingcap/tidb/pkg/testkit.(*TestKit).MustQueryWithContext
        	            		pkg/testkit/testkit.go:199
        	            	github.com/pingcap/tidb/pkg/testkit.(*TestKit).MustQuery
        	            		pkg/testkit/testkit.go:174
        	            	pkg/executor/test/cte/cte_test.TestCTEIterationMemTracker
        	            		pkg/executor/test/cte/cte_test.go:227
        	            	testing.tRunner
        	            		GOROOT/src/testing/testing.go:1595
        	            	runtime.goexit
        	            		src/runtime/asm_amd64.s:1650
        	Test:       	TestCTEIterationMemTracker
        	Messages:   	sql:explain analyze with recursive cte1 as (select c1 from t1 union all select c1 + 1 c1 from cte1 where c1 < 5000) select * from cte1, args:[]

@lance6716 lance6716 reopened this Jul 8, 2024
@Defined2014
Copy link
Contributor

Another link https://do.pingcap.net/jenkins/blue/organizations/jenkins/pingcap%2Ftidb%2Fghpr_unit_test/detail/ghpr_unit_test/12872/pipeline

@Defined2014
Copy link
Contributor

Defined2014 commented Jul 8, 2024 •
edited
Loading 

[2024/07/08 08:09:42.696 +00:00] [INFO] [row_container.go:443] ["memory exceeds quota, spill to disk now."] [consumed=380] [quota=128]
[2024/07/08 08:09:42.697 +00:00] [INFO] [cte.go:765] ["cte iteration info"] ["iterInTbl mem usage"=0] ["iterInTbl disk usage"=24000] ["iterOutTbl mem usage"=8245] ["iterOutTbl disk usage"=0] ["resTbl mem usage"=0] ["resTbl disk usage"=16800000] ["resTbl rows"=700000] ["iteration num"=700] []
    testkit.go:274: 
        	Error Trace:	pkg/testkit/testkit.go:274

        	            				pkg/testkit/testkit.go:199

        	            				pkg/testkit/testkit.go:174

        	            				pkg/executor/test/cte/cte_test.go:227

        	Error:      	Received unexpected error:

        	            	assert row container spill disk failed

        	            	github.com/pingcap/tidb/pkg/util.GetRecoverError

        	            		pkg/util/util.go:305

        	            	github.com/pingcap/tidb/pkg/executor.(*cteProducer).computeRecursivePart.func1

        	            		pkg/executor/cte.go:417

        	            	runtime.gopanic

        	            		GOROOT/src/runtime/panic.go:914

        	            	github.com/pingcap/tidb/pkg/executor.(*cteProducer).computeRecursivePart

        	            		pkg/executor/cte.go:449

        	            	github.com/pingcap/tidb/pkg/executor.(*cteProducer).produce

        	            		pkg/executor/cte.go:367

        	            	github.com/pingcap/tidb/pkg/executor.(*CTEExec).Next

        	            		pkg/executor/cte.go:114

        	            	github.com/pingcap/tidb/pkg/executor/internal/exec.Next

        	            		pkg/executor/internal/exec/executor.go:410

        	            	github.com/pingcap/tidb/pkg/executor.(*ExplainExec).executeAnalyzeExec

        	            		pkg/executor/explain.go:130

        	            	github.com/pingcap/tidb/pkg/executor.(*ExplainExec).generateExplainInfo

        	            		pkg/executor/explain.go:149

        	            	github.com/pingcap/tidb/pkg/executor.(*ExplainExec).Next

        	            		pkg/executor/explain.go:76

        	            	github.com/pingcap/tidb/pkg/executor/internal/exec.Next

        	            		pkg/executor/internal/exec/executor.go:410

        	            	github.com/pingcap/tidb/pkg/executor.(*ExecStmt).next

        	            		pkg/executor/adapter.go:1241

        	            	github.com/pingcap/tidb/pkg/executor.(*recordSet).Next

        	            		pkg/executor/adapter.go:169

        	            	github.com/pingcap/tidb/pkg/session.GetRows4Test

        	            		pkg/session/tidb.go:346

        	            	github.com/pingcap/tidb/pkg/session.ResultSetToStringSlice

        	            		pkg/session/tidb.go:364

        	            	github.com/pingcap/tidb/pkg/testkit.(*TestKit).ResultSetToResultWithCtx

        	            		pkg/testkit/testkit.go:273

        	            	github.com/pingcap/tidb/pkg/testkit.(*TestKit).MustQueryWithContext

        	            		pkg/testkit/testkit.go:199

        	            	github.com/pingcap/tidb/pkg/testkit.(*TestKit).MustQuery

        	            		pkg/testkit/testkit.go:174

        	            	pkg/executor/test/cte/cte_test.TestCTEIterationMemTracker

        	            		pkg/executor/test/cte/cte_test.go:227

        	            	testing.tRunner

        	            		GOROOT/src/testing/testing.go:1595

        	            	runtime.goexit

        	            		src/runtime/asm_amd64.s:1650

        	Test:       	TestCTEIterationMemTracker

        	Messages:   	sql:explain analyze with recursive cte1 as (select c1 from t1 union all select c1 + 1 c1 from cte1 where c1 < 5000) select * from cte1, args:[]

@tiancaiamao tiancaiamao mentioned this issue Jul 9, 2024
Merged
13 tasks
@tiancaiamao tiancaiamao added sig/execution SIG execution component/executor and removed sig/sql-infra SIG: SQL Infra labels Jul 9, 2024
ti-chi-bot bot pushed a commit that referenced this issue Jul 9, 2024
@guo-shaoge
executor: fix cte mem tracker failpoint condition again (#54523)
887637b 
close #54379
@ti-chi-bot ti-chi-bot added affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-8.1 This bug affects the 8.1.x(LTS) versions. affects-6.1 This bug affects the 6.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. affects-5.4 This bug affects the 5.4.x(LTS) versions. labels Jul 18, 2024
This was referenced Jul 18, 2024
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Merged
Closed
Closed
Merged
Merged
Open
Closed
Closed
Merged
Merged
Open
ti-chi-bot bot pushed a commit that referenced this issue Jul 21, 2024
@ti-chi-bot
executor: fix cte mem tracker failpoint condition again (#54523) (#54732
ac79ace 
)

close #54379
ti-chi-bot bot pushed a commit that referenced this issue Jul 22, 2024
@ti-chi-bot
executor: fix cte mem tracker failpoint condition again (#54523) (#54729
e07d004 
)

close #54379
ti-chi-bot bot pushed a commit that referenced this issue Jul 22, 2024
@ti-chi-bot
executor: fix cte mem tracker failpoint condition again (#54523) (#54764
b426095 
)

close #54379
ti-chi-bot bot pushed a commit that referenced this issue Jul 22, 2024
@ti-chi-bot
executor: fix cte mem tracker failpoint condition again (#54523) (#54725
b364bf4 
)

close #54379
ti-chi-bot bot pushed a commit that referenced this issue Jul 22, 2024
@ti-chi-bot
executor: fix cte mem tracker failpoint condition again (#54523) (#54762
1d204c6 
)

close #54379
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@guo-shaoge guo-shaoge

Labels
affects-5.4 This bug affects the 5.4.x(LTS) versions. affects-6.1 This bug affects the 6.1.x(LTS) versions. affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. affects-8.1 This bug affects the 8.1.x(LTS) versions. component/executor component/test severity/major sig/execution SIG execution type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
7 participants
@tiancaiamao @lance6716 @hawkingrei @jebter @guo-shaoge @Defined2014 @ti-chi-bot