Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CTEExec should not be deadlocked with defer function after panic action triggered #49096

Closed
AilinKid opened this issue Dec 1, 2023 · 1 comment · Fixed by #49192
Closed

CTEExec should not be deadlocked with defer function after panic action triggered #49096

AilinKid opened this issue Dec 1, 2023 · 1 comment · Fixed by #49192
Assignees
@AilinKid
Labels
affects-5.4 This bug affects the 5.4.x(LTS) versions. affects-6.1 This bug affects the 6.1.x(LTS) versions. affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. report/customer Customers have encountered this bug. severity/major sig/execution SIG execution type/bug The issue is confirmed as a bug.

Comments

@AilinKid
Copy link
Contributor

AilinKid commented Dec 1, 2023

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

sync.runtime_SemacquireMutex(0x0?, 0x25?, 0xc0bfd74ca8?)
	/usr/local/go/src/runtime/sema.go:77 +0x25
sync.(*Mutex).lockSlow(0xc12cf36108)
	/usr/local/go/src/sync/mutex.go:171 +0x165
sync.(*Mutex).Lock(...)
	/usr/local/go/src/sync/mutex.go:90
github.com/pingcap/tidb/util/cteutil.(*StorageRC).Lock(0x0?)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/cteutil/storage.go:211 +0x33
github.com/pingcap/tidb/executor.resetCTEStorageMap({0x55508f8, 0xc06d4b2a00})
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/adapter.go:1459 +0x159
github.com/pingcap/tidb/executor.(*ExecStmt).handleNoDelay.func1()
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/adapter.go:751 +0x5d
panic({0x4581a60, 0xc66ac508c0})
	/usr/local/go/src/runtime/panic.go:884 +0x212
github.com/pingcap/tidb/util/memory.(*PanicOnExceed).Action(0xc158421040, 0xc11ca39ae0)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/memory/action.go:148 +0x11f
github.com/pingcap/tidb/util/memory.(*Tracker).Consume.func2(0x46d8260?, 0xc000713f80?)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/memory/tracker.go:452 +0xff
github.com/pingcap/tidb/util/memory.(*Tracker).Consume(0x46d8080?, 0x448b2812c)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/memory/tracker.go:464 +0x2c2
github.com/pingcap/tidb/util/memory.(*Tracker).remove(0xc0465905a0, 0xc125a28500)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/memory/tracker.go:355 +0x105
github.com/pingcap/tidb/util/memory.(*Tracker).Detach(0xc125a28500)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/memory/tracker.go:327 +0x1b6
github.com/pingcap/tidb/executor.(*hashRowContainer).Close(0xc001371600?)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/hash_table.go:506 +0x9e
github.com/pingcap/tidb/parser/terror.Call(0xc0d801ba40?)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:306 +0x31
github.com/pingcap/tidb/executor.(*HashJoinExec).Close(0xc0923a2d00)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/join.go:174 +0x265
github.com/pingcap/tidb/executor.(*baseExecutor).Close(0xc06e6bcee8?)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/executor.go:214 +0x7c
github.com/pingcap/tidb/executor.(*ProjectionExec).Close(0xc07e332ea0)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/projection.go:334 +0x1d4
github.com/pingcap/tidb/executor.(*ShuffleExec).Close(0xc185c93420)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/shuffle.go:190 +0x35b
github.com/pingcap/tidb/executor.(*baseExecutor).Close(0x54e0e50?)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/executor.go:214 +0x7c
github.com/pingcap/tidb/executor.(*SelectionExec).Close(0xc185c935e0?)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/executor.go:1572 +0x65
github.com/pingcap/tidb/executor.(*ShuffleExec).Close(0xc185c93340)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/shuffle.go:190 +0x35b
github.com/pingcap/tidb/executor.(*baseExecutor).Close(0x18?)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/executor.go:214 +0x7c
github.com/pingcap/tidb/executor.(*SelectionExec).Close(0xc185c937a0?)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/executor.go:1572 +0x65
github.com/pingcap/tidb/executor.(*baseExecutor).Close(0xc06e6bcee8?)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/executor.go:214 +0x7c
github.com/pingcap/tidb/executor.(*ProjectionExec).Close(0xc07e333c20)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/projection.go:334 +0x1d4
github.com/pingcap/tidb/executor.(*cteProducer).closeProducer(0xc0190ff100)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/cte.go:218 +0x2e
github.com/pingcap/tidb/executor.(*CTEExec).Close(0xc13d0df4a0)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/cte.go:119 +0x52
github.com/pingcap/tidb/executor.(*baseExecutor).Close(0x18?)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/executor.go:214 +0x7c
github.com/pingcap/tidb/executor.(*SelectionExec).Close(0xc185c93880?)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/executor.go:1572 +0x65
github.com/pingcap/tidb/executor.(*baseExecutor).Close(0xc06e6bcee8?)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/executor.go:214 +0x7c
github.com/pingcap/tidb/executor.(*ProjectionExec).Close(0xc0f1326240)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/projection.go:334 +0x1d4
github.com/pingcap/tidb/executor.(*baseExecutor).Close(0xc0913c3f20?)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/executor.go:214 +0x7c
github.com/pingcap/tidb/executor.(*HashAggExec).Close(0xc1624f9b00)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/aggregate.go:292 +0x2e5
github.com/pingcap/tidb/executor.(*baseExecutor).Close(0xc06e6bcee8?)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/executor.go:214 +0x7c
github.com/pingcap/tidb/executor.(*ProjectionExec).Close(0xc0f1326480)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/projection.go:334 +0x1d4
github.com/pingcap/tidb/executor.(*baseExecutor).Close(0xc0ad1f7aa0?)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/executor.go:214 +0x7c
github.com/pingcap/tidb/executor.(*HashJoinExec).Close(0xc0923a3520)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/join.go:194 +0x4da
github.com/pingcap/tidb/executor.(*baseExecutor).Close(0xc0928aa000?)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/executor.go:214 +0x7c
github.com/pingcap/tidb/executor.(*HashAggExec).Close(0xc10b278000)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/aggregate.go:292 +0x2e5
github.com/pingcap/tidb/executor.(*baseExecutor).Close(0xc06e6bcee8?)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/executor.go:214 +0x7c
github.com/pingcap/tidb/executor.(*ProjectionExec).Close(0xc0f1327b00)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/projection.go:334 +0x1d4
github.com/pingcap/tidb/executor.(*UnionExec).Close(0xc1471f2480)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/executor.go:1936 +0x122
github.com/pingcap/tidb/executor.(*baseExecutor).Close(0xc08f63d5c0?)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/executor.go:214 +0x7c
github.com/pingcap/tidb/executor.(*HashAggExec).Close(0xc10b2786c0)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/aggregate.go:292 +0x2e5
github.com/pingcap/tidb/executor.(*ReplaceExec).Close(0xc1cd83c530)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/replace.go:51 +0x136
github.com/pingcap/tidb/executor.(*ExecStmt).handleNoDelayExecutor.func1()
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/adapter.go:909 +0x2b
github.com/pingcap/tidb/executor.(*ExecStmt).handleNoDelayExecutor(0xc076e92e10, {0x54dfe90, 0xc03a63c450}, {0x54e4c20?, 0xc1cd83c530})
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/adapter.go:929 +0x60e
github.com/pingcap/tidb/executor.(*ExecStmt).handleNoDelay(0xc076e92e10, {0x54dfe90, 0xc03a63c450}, {0x54e4c20?, 0xc1cd83c530?}, 0x0)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/adapter.go:776 +0x26d
github.com/pingcap/tidb/executor.(*ExecStmt).Exec(0xc076e92e10, {0x54dfe90, 0xc03a63c450})
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/adapter.go:571 +0xbe5
github.com/pingcap/tidb/session.runStmt({0x54dfe90, 0xc15d68c930}, 0xc06d4b2a00, {0x54f1d60, 0xc076e92e10?})
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:2373 +0x565
github.com/pingcap/tidb/session.(*session).ExecuteStmt(0xc06d4b2a00, {0x54dfe90, 0xc15d68c750}, {0x54f9330?, 0xc0ccc02600})
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:2237 +0xab8
github.com/pingcap/tidb/server.(*TiDBContext).ExecuteStmt(0xc06e6bcd20, {0x54dfe90, 0xc15d68c750}, {0x54f9330?, 0xc0ccc02600})
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/driver_tidb.go:252 +0xaf
github.com/pingcap/tidb/server.(*clientConn).handleStmt(0xc06e386180, {0x54dfde8, 0xc0adf786c0}, {0x54f9330, 0xc0ccc02600}, {0x7e04638, 0x0, 0x0}, 0x1)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:2122 +0x167
github.com/pingcap/tidb/server.(*clientConn).handleQuery(0xc06e386180, {0x54dfde8, 0xc0adf786c0}, {0xc12e792801, 0x1606})
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1972 +0x93d
github.com/pingcap/tidb/server.(*clientConn).dispatch(0xc06e386180, {0x54dfe90?, 0xc05bb17500?}, {0xc12e792800, 0x1607, 0x1607})
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1403 +0x1005
github.com/pingcap/tidb/server.(*clientConn).Run(0xc06e386180, {0x54dfe90, 0xc05bb17500})
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1152 +0x24e
github.com/pingcap/tidb/server.(*Server).onConn(0xc002dcf8c0, 0xc06e386180)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:648 +0xac7
created by github.com/pingcap/tidb/server.(*Server).startNetworkListener
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:483 +0x72c

2. What did you expect to see? (Required)

closeCTE should be conflict with defer function after panic triggered

3. What did you see instead (Required)

dead lock

4. What is your TiDB version? (Required)

master
@AilinKid AilinKid added type/bug The issue is confirmed as a bug. sig/execution SIG execution labels Dec 1, 2023
@AilinKid AilinKid mentioned this issue Dec 5, 2023
Merged
13 tasks
@AilinKid AilinKid added affects-5.4 This bug affects the 5.4.x(LTS) versions. affects-6.1 This bug affects the 6.1.x(LTS) versions. affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. and removed may-affects-5.4 This bug maybe affects 5.4.x versions. may-affects-6.1 may-affects-6.5 may-affects-7.1 may-affects-7.5 labels Dec 5, 2023
@AilinKid AilinKid self-assigned this Dec 5, 2023
ti-chi-bot bot pushed a commit that referenced this issue Dec 5, 2023
@AilinKid
executor: fix deadlock in dml statement with cte when oom panic actio…
0c7659c 
…n was triggered (#49192)

close #49096
This was referenced Dec 5, 2023
Closed
Closed
Merged
Merged
Merged
ti-chi-bot bot pushed a commit that referenced this issue Dec 6, 2023
@ti-chi-bot
executor: fix deadlock in dml statement with cte when oom panic actio…
003d0d4 
…n was triggered (#49192) (#49202)

close #49096
ti-chi-bot bot pushed a commit that referenced this issue Dec 8, 2023
@ti-chi-bot
executor: fix deadlock in dml statement with cte when oom panic actio…
bb5be54 
…n was triggered (#49192) (#49201)

close #49096
ti-chi-bot bot pushed a commit that referenced this issue Dec 20, 2023
@ti-chi-bot
executor: fix deadlock in dml statement with cte when oom panic actio…
5d1665d 
…n was triggered (#49192) (#49203)

close #49096
@seiya-annie
Copy link

/found customer

@ti-chi-bot ti-chi-bot bot added the report/customer Customers have encountered this bug. label Jun 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@AilinKid AilinKid

Labels
affects-5.4 This bug affects the 5.4.x(LTS) versions. affects-6.1 This bug affects the 6.1.x(LTS) versions. affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. report/customer Customers have encountered this bug. severity/major sig/execution SIG execution type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

executor: fix deadlock in dml statement with cte when oom panic action was triggered AilinKid/tidb
3 participants
@XuHuaiyu @AilinKid @seiya-annie