tidb panic for "IndexMergeProcessWorker: runtime error: invalid memory address or nil pointer dereference"

[seiya-annie]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

steps please see in :
utf-go-clustered-index-panic.log

2. What did you expect to see? (Required)

no panic

3. What did you see instead (Required)

[2023/08/10 20:54:12.065 +00:00] [ERROR] [index_merge_reader.go:884] ["IndexMergeProcessWorker: runtime error: invalid memory address or nil pointer dereference"] [conn=2636120476]
[2023/08/10 20:54:12.065 +00:00] [ERROR] [misc.go:91] ["panic in the recoverable goroutine"] [r="\"invalid memory address or nil pointer dereference\""] ["stack trace"="[github.com/pingcap/tidb/util.WithRecovery.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/misc.go:93\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:914\nruntime.panicmem\n\t/usr/local/go/src/runtime/panic.go:261\nruntime.sigpanic\n\t/usr/local/go/src/runtime](http://github.com/pingcap/tidb/util.WithRecovery.func1/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/misc.go:93/nruntime.gopanic/n/t/usr/local/go/src/runtime/panic.go:914/nruntime.panicmem/n/t/usr/local/go/src/runtime/panic.go:261/nruntime.sigpanic/n/t/usr/local/go/src/runtime)

4. What is your TiDB version? (Required)

master

[seiya-annie]

This panic occurs probabilistically

[seiya-annie]

tidb.log.gz

[XuHuaiyu]

[2023/08/10 20:54:12.065 +00:00] [ERROR] [index_merge_reader.go:884] ["IndexMergeProcessWorker: runtime error: invalid memory address or nil pointer dereference"] [conn=2636120476]
[2023/08/10 20:54:12.065 +00:00] [ERROR] [misc.go:91] ["panic in the recoverable goroutine"] [r="\"invalid memory address or nil pointer dereference\""] ["stack trace"="github.com/pingcap/tidb/util.WithRecovery.func1
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/misc.go:93
runtime.gopanic
	/usr/local/go/src/runtime/panic.go:914
runtime.panicmem
	/usr/local/go/src/runtime/panic.go:261
runtime.sigpanic
	/usr/local/go/src/runtime/signal_unix.go:861
github.com/pingcap/tidb/util/chunk.(*Chunk).GetRow
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/chunk/chunk.go:357
github.com/pingcap/tidb/executor.handleHeap.Less
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/index_merge_reader.go:948
container/heap.up
	/usr/local/go/src/container/heap/heap.go:92
container/heap.Push
	/usr/local/go/src/container/heap/heap.go:53
github.com/pingcap/tidb/executor.(*indexMergeProcessWorker).fetchLoopUnionWithOrderBy
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/index_merge_reader.go:1066
github.com/pingcap/tidb/executor.(*IndexMergeReaderExecutor).startIndexMergeProcessWorker.func1.1
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/index_merge_reader.go:308
github.com/pingcap/tidb/util.WithRecovery
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/misc.go:96
github.com/pingcap/tidb/executor.(*IndexMergeReaderExecutor).startIndexMergeProcessWorker.func1
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/index_merge_reader.go:300"]
[2023/08/10 20:54:12.066 +00:00] [INFO] [conn.go:1082] ["command dispatched failed"] [conn=2636120476] [connInfo="id:2636120476, addr:10.233.96.131:35656 status:10, collation:utf8mb4_general_ci, user:root"] [command=Query] [status="inTxn:0, autocommit:1"] [sql="select /*+  stream_agg() */ avg(   col_208 ) aggCol from (select  /*+ use_index_merge( tbl_34 ) */ * from tbl_34 where not( tbl_34.col_207 in ( '' , '' , '' , '' ) ) or not( tbl_34.col_208 in ( 16095109 , 3208480 , 12732339 ) ) and not( tbl_34.col_207 in ( '' , '' , '' , '' ) ) and not( tbl_34.col_208 between 3590883 and 10132169 ) order by col_207,col_208  ) ordered_tbl  order by aggCol "] [txn_mode=PESSIMISTIC] [timestamp=443469228157173763] [err="IndexMergeProcessWorker: runtime error: invalid memory address or nil pointer dereference
github.com/pingcap/tidb/executor.(*IndexMergeReaderExecutor).startIndexMergeProcessWorker.func1.handleWorkerPanic.func2
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/index_merge_reader.go:883
github.com/pingcap/tidb/util.WithRecovery.func1
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/misc.go:88
runtime.gopanic
	/usr/local/go/src/runtime/panic.go:914
runtime.panicmem
	/usr/local/go/src/runtime/panic.go:261
runtime.sigpanic
	/usr/local/go/src/runtime/signal_unix.go:861
github.com/pingcap/tidb/util/chunk.(*Chunk).GetRow
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/chunk/chunk.go:357
github.com/pingcap/tidb/executor.handleHeap.Less
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/index_merge_reader.go:948
container/heap.up
	/usr/local/go/src/container/heap/heap.go:92
container/heap.Push
	/usr/local/go/src/container/heap/heap.go:53
github.com/pingcap/tidb/executor.(*indexMergeProcessWorker).fetchLoopUnionWithOrderBy
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/index_merge_reader.go:1066
github.com/pingcap/tidb/executor.(*IndexMergeReaderExecutor).startIndexMergeProcessWorker.func1.1
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/index_merge_reader.go:308
github.com/pingcap/tidb/util.WithRecovery
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/misc.go:96
github.com/pingcap/tidb/executor.(*IndexMergeReaderExecutor).startIndexMergeProcessWorker.func1
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/index_merge_reader.go:300
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1650"]

[XuHuaiyu]

Can not reproduce the problem till now

[wshwsh12]

reproduce step:

use test;
drop table t;
set tidb_index_lookup_size = 1024;
create table t(a int, b int, c int, index idx1(a, c), index idx2(b, c));
insert into t values(1, 1, 1), (1, 2, -1), (2, 1, -2);
insert into t(a,b,c) select a,b,c from t;
insert into t(a,b,c) select a,b,c from t;
insert into t(a,b,c) select a,b,c from t;
insert into t(a,b,c) select a,b,c from t;
insert into t(a,b,c) select a,b,c from t;
insert into t(a,b,c) select a,b,c from t;
insert into t(a,b,c) select a,b,c from t;
insert into t(a,b,c) select a,b,c from t;
insert into t(a,b,c) select a,b,c from t;
insert into t values(1, 1, 1), (1, 2, -1), (2, 1, -2);
select /*+ USE_INDEX_MERGE(t, idx1, idx2) */ * from t where a = 1 or b = 1 order by c limit 1025;

[seiya-annie]

tidb.log.gz

[Defined2014]

github.com/pingcap/tidb/kv.(*HandleMap).Get
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/kv/key.go:420
github.com/pingcap/tidb/executor.(*indexMergeTableScanWorker).executeTask
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/index_merge_reader.go:1793
github.com/pingcap/tidb/executor.(*indexMergeTableScanWorker).pickAndExecTask
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/index_merge_reader.go:1698
github.com/pingcap/tidb/executor.(*IndexMergeReaderExecutor).startIndexMergeTableScanWorker.func1.1
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/index_merge_reader.go:762
github.com/pingcap/tidb/util.WithRecovery
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/misc.go:96
github.com/pingcap/tidb/executor.(*IndexMergeReaderExecutor).startIndexMergeTableScanWorker.func1
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/index_merge_reader.go:756
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1650"]

It looks like task.indexOrder is empty. And for previous one, the task.idxRows is empty, may be caused by conflict with task struct.

[Defined2014]

CREATE TABLE `tbl_39` (
  `col_239` year(4) NOT NULL DEFAULT '2009',
  PRIMARY KEY (`col_239`) /*T![clustered_index] CLUSTERED */,
  KEY `idx_222` (`col_239`),
  UNIQUE KEY `idx_223` (`col_239`),
  KEY `idx_224` (`col_239`),
  KEY `idx_225` (`col_239`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin;
insert into tbl_39 values (1994),(1995),(1996),(1997);
select  /*+ use_index_merge( tbl_39) */ col_239  from tbl_39 where not( tbl_39.col_239 not in ( '1994' ) ) and      tbl_39.col_239 not in ( '2004' , '2010' , '2010' )     or not( tbl_39.col_239 <= '1996' )     and not( tbl_39.col_239 between '2026' and '2011' ) order by tbl_39.col_239 limit 382;

Seems the plan for index_merge has keep_order = true without ByItems which is unexpected.