Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Panic in split table t index stmts #50177

Closed
Defined2014 opened this issue Jan 8, 2024 · 3 comments · Fixed by #50250
Closed

Panic in split table t index stmts #50177

Defined2014 opened this issue Jan 8, 2024 · 3 comments · Fixed by #50250
Labels
component/test severity/moderate sig/sql-infra SIG: SQL Infra type/bug The issue is confirmed as a bug.

Comments

@Defined2014
Copy link
Contributor

Defined2014 commented Jan 8, 2024 •
edited
Loading

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

https://do.pingcap.net/jenkins/blue/organizations/jenkins/pingcap%2Ftidb%2Fghpr_check/detail/ghpr_check/1090/pipeline

time="2024-01-08T13:42:27+08:00" level=error msg="run test [planner/core/integration_partition] err: sql:split table tlist index a between (2) and (15) regions 10;: run \"split table tlist index a between (2) and (15) regions 10;\" at line 237 err Error 1105 (HY000): runtime error: index out of range [30] with length 30"

Related log

[2024/01/08 13:40:22.890 +08:00] [INFO] [split_region.go:85] ["split batch regions request"] ["split key count"=1] ["batch count"=1] ["first batch, region ID"=12822] ["first split key"=748000000000004263]
[2024/01/08 13:40:22.890 +08:00] [INFO] [region_request.go:1532] ["send request meet region error without retry"] [req-ts=0] [req-type=SplitRegion] [region="{ region id: 12822, ver: 1, confVer: 1 }"] [replica-read-type=leader] [stale-read=false] [request-sender="{rpcError:<nil>,replicaSelector: replicaSelector{selectorStateStr: accessKnownLeader, cacheRegionIsValid: false, replicaStatus: [peer: 12823, store: 1, isEpochStale: false, attempts: 1, replica-epoch: 0, store-epoch: 0, store-state: resolved, store-liveness-state: reachable]}}"] [retry-times=0] [total-backoff-ms=0] [total-backoff-times=0] [max-exec-timeout-ms=30000] [total-region-errors=12823-epoch_not_match:1]
[2024/01/08 13:40:22.890 +08:00] [INFO] [split_region.go:187] ["batch split regions complete"] ["batch region ID"=12822] ["first at"=7480000000000042615f698000000000000002] ["first new region left"="{Id:12822 StartKey:7480000000000042ff6100000000000000f8 EndKey:7480000000000042ff615f698000000000ff0000010380000000ff000000034fccccccffcccccc0000000000fa RegionEpoch:{ConfVer:1 Version:3} Peers:[id:12823 store_id:1 ] EncryptionMeta:<nil> IsInFlashback:false FlashbackStartTs:0}"] ["new region count"=31]
[2024/01/08 13:40:22.890 +08:00] [INFO] [split_region.go:242] ["start scatter region"] [regionID=12822]
[2024/01/08 13:40:22.890 +08:00] [INFO] [split_region.go:199] ["batch split regions, scatter region complete"] ["batch region ID"=12822] [at=7480000000000042615f698000000000000002] ["new region left"="{Id:12822 StartKey:7480000000000042ff6100000000000000f8 EndKey:7480000000000042ff615f698000000000ff0000010380000000ff000000034fccccccffcccccc0000000000fa RegionEpoch:{ConfVer:1 Version:3} Peers:[id:12823 store_id:1 ] EncryptionMeta:<nil> IsInFlashback:false FlashbackStartTs:0}"]
[2024/01/08 13:40:22.890 +08:00] [INFO] [split_region.go:242] ["start scatter region"] [regionID=12826]
[2024/01/08 13:40:22.890 +08:00] [INFO] [split_region.go:199] ["batch split regions, scatter region complete"] ["batch region ID"=12822] [at=7480000000000042615f6980000000000000010380000000000000034fcccccccccccc] ["new region left"="{Id:12826 StartKey:7480000000000042ff615f698000000000ff0000010380000000ff000000034fccccccffcccccc0000000000fa EndKey:7480000000000042ff615f698000000000ff0000010380000000ff000000049c999999ff9999980000000000fa RegionEpoch:{ConfVer:1 Version:1} Peers:[id:12827 store_id:1 ] EncryptionMeta:<nil> IsInFlashback:false FlashbackStartTs:0}"]
[2024/01/08 13:40:22.890 +08:00] [INFO] [split_region.go:242] ["start scatter region"] [regionID=12828]
[2024/01/08 13:40:22.890 +08:00] [INFO] [split_region.go:199] ["batch split regions, scatter region complete"] ["batch region ID"=12822] [at=7480000000000042615f6980000000000000010380000000000000049c999999999998] ["new region left"="{Id:12828 StartKey:7480000000000042ff615f698000000000ff0000010380000000ff000000049c999999ff9999980000000000fa EndKey:7480000000000042ff615f698000000000ff0000010380000000ff00000005e9666666ff6666640000000000fa RegionEpoch:{ConfVer:1 Version:1} Peers:[id:12829 store_id:1 ] EncryptionMeta:<nil> IsInFlashback:false FlashbackStartTs:0}"]
[2024/01/08 13:40:22.890 +08:00] [INFO] [split_region.go:242] ["start scatter region"] [regionID=12830]
[2024/01/08 13:40:22.890 +08:00] [INFO] [split_region.go:199] ["batch split regions, scatter region complete"] ["batch region ID"=12822] [at=7480000000000042615f698000000000000001038000000000000005e9666666666664] ["new region left"="{Id:12830 StartKey:7480000000000042ff615f698000000000ff0000010380000000ff00000005e9666666ff6666640000000000fa EndKey:7480000000000042ff615f698000000000ff0000010380000000ff0000000736333333ff3333300000000000fa RegionEpoch:{ConfVer:1 Version:1} Peers:[id:12831 store_id:1 ] EncryptionMeta:<nil> IsInFlashback:false FlashbackStartTs:0}"]
[2024/01/08 13:40:22.890 +08:00] [INFO] [split_region.go:242] ["start scatter region"] [regionID=12832]
[2024/01/08 13:40:22.890 +08:00] [INFO] [split_region.go:199] ["batch split regions, scatter region complete"] ["batch region ID"=12822] [at=7480000000000042615f69800000000000000103800000000000000736333333333330] ["new region left"="{Id:12832 StartKey:7480000000000042ff615f698000000000ff0000010380000000ff0000000736333333ff3333300000000000fa EndKey:7480000000000042ff615f698000000000ff0000010380000000ff0000000882fffffffffffffc0000000000fa RegionEpoch:{ConfVer:1 Version:1} Peers:[id:12833 store_id:1 ] EncryptionMeta:<nil> IsInFlashback:false FlashbackStartTs:0}"]
[2024/01/08 13:40:22.891 +08:00] [INFO] [split_region.go:242] ["start scatter region"] [regionID=12834]
[2024/01/08 13:40:22.891 +08:00] [INFO] [split_region.go:199] ["batch split regions, scatter region complete"] ["batch region ID"=12822] [at=7480000000000042615f69800000000000000103800000000000000882fffffffffffc] ["new region left"="{Id:12834 StartKey:7480000000000042ff615f698000000000ff0000010380000000ff0000000882fffffffffffffc0000000000fa EndKey:7480000000000042ff615f698000000000ff0000010380000000ff00000009cfccccccffccccc80000000000fa RegionEpoch:{ConfVer:1 Version:1} Peers:[id:12835 store_id:1 ] EncryptionMeta:<nil> IsInFlashback:false FlashbackStartTs:0}"]
[2024/01/08 13:40:22.891 +08:00] [INFO] [split_region.go:242] ["start scatter region"] [regionID=12836]
[2024/01/08 13:40:22.891 +08:00] [INFO] [split_region.go:199] ["batch split regions, scatter region complete"] ["batch region ID"=12822] [at=7480000000000042615f698000000000000001038000000000000009cfccccccccccc8] ["new region left"="{Id:12836 StartKey:7480000000000042ff615f698000000000ff0000010380000000ff00000009cfccccccffccccc80000000000fa EndKey:7480000000000042ff615f698000000000ff0000010380000000ff0000000b1c999999ff9999940000000000fa RegionEpoch:{ConfVer:1 Version:1} Peers:[id:12837 store_id:1 ] EncryptionMeta:<nil> IsInFlashback:false FlashbackStartTs:0}"]
[2024/01/08 13:40:22.891 +08:00] [INFO] [split_region.go:242] ["start scatter region"] [regionID=12838]
[2024/01/08 13:40:22.891 +08:00] [INFO] [split_region.go:199] ["batch split regions, scatter region complete"] ["batch region ID"=12822] [at=7480000000000042615f69800000000000000103800000000000000b1c999999999994] ["new region left"="{Id:12838 StartKey:7480000000000042ff615f698000000000ff0000010380000000ff0000000b1c999999ff9999940000000000fa EndKey:7480000000000042ff615f698000000000ff0000010380000000ff0000000c69666666ff6666600000000000fa RegionEpoch:{ConfVer:1 Version:1} Peers:[id:12839 store_id:1 ] EncryptionMeta:<nil> IsInFlashback:false FlashbackStartTs:0}"]
[2024/01/08 13:40:22.891 +08:00] [INFO] [split_region.go:242] ["start scatter region"] [regionID=12840]
[2024/01/08 13:40:22.891 +08:00] [INFO] [split_region.go:199] ["batch split regions, scatter region complete"] ["batch region ID"=12822] [at=7480000000000042615f69800000000000000103800000000000000c69666666666660] ["new region left"="{Id:12840 StartKey:7480000000000042ff615f698000000000ff0000010380000000ff0000000c69666666ff6666600000000000fa EndKey:7480000000000042ff615f698000000000ff0000010380000000ff0000000db6333333ff33332c0000000000fa RegionEpoch:{ConfVer:1 Version:1} Peers:[id:12841 store_id:1 ] EncryptionMeta:<nil> IsInFlashback:false FlashbackStartTs:0}"]
[2024/01/08 13:40:22.891 +08:00] [INFO] [split_region.go:242] ["start scatter region"] [regionID=12842]
[2024/01/08 13:40:22.891 +08:00] [INFO] [split_region.go:199] ["batch split regions, scatter region complete"] ["batch region ID"=12822] [at=7480000000000042615f69800000000000000103800000000000000db633333333332c] ["new region left"="{Id:12842 StartKey:7480000000000042ff615f698000000000ff0000010380000000ff0000000db6333333ff33332c0000000000fa EndKey:7480000000000042ff615f698000000000ff0000020000000000fa RegionEpoch:{ConfVer:1 Version:1} Peers:[id:12843 store_id:1 ] EncryptionMeta:<nil> IsInFlashback:false FlashbackStartTs:0}"]
[2024/01/08 13:40:22.891 +08:00] [INFO] [split_region.go:242] ["start scatter region"] [regionID=12844]
[2024/01/08 13:40:22.891 +08:00] [INFO] [split_region.go:199] ["batch split regions, scatter region complete"] ["batch region ID"=12822] [at=7480000000000042625f698000000000000002] ["new region left"="{Id:12844 StartKey:7480000000000042ff615f698000000000ff0000020000000000fa EndKey:7480000000000042ff6200000000000000f8 RegionEpoch:{ConfVer:1 Version:1} Peers:[id:12845 store_id:1 ] EncryptionMeta:<nil> IsInFlashback:false FlashbackStartTs:0}"]
[2024/01/08 13:40:22.891 +08:00] [INFO] [split_region.go:242] ["start scatter region"] [regionID=12824]
[2024/01/08 13:40:22.891 +08:00] [INFO] [split_region.go:199] ["batch split regions, scatter region complete"] ["batch region ID"=12822] [at=7480000000000042625f6980000000000000010380000000000000034fcccccccccccc] ["new region left"="{Id:12824 StartKey:7480000000000042ff6200000000000000f8 EndKey:7480000000000042ff625f698000000000ff0000010380000000ff000000034fccccccffcccccc0000000000fa RegionEpoch:{ConfVer:1 Version:2} Peers:[id:12825 store_id:1 ] EncryptionMeta:<nil> IsInFlashback:false FlashbackStartTs:0}"]
[2024/01/08 13:40:22.891 +08:00] [INFO] [split_region.go:242] ["start scatter region"] [regionID=12846]
[2024/01/08 13:40:22.891 +08:00] [INFO] [split_region.go:199] ["batch split regions, scatter region complete"] ["batch region ID"=12822] [at=7480000000000042625f6980000000000000010380000000000000049c999999999998] ["new region left"="{Id:12846 StartKey:7480000000000042ff625f698000000000ff0000010380000000ff000000034fccccccffcccccc0000000000fa EndKey:7480000000000042ff625f698000000000ff0000010380000000ff000000049c999999ff9999980000000000fa RegionEpoch:{ConfVer:1 Version:1} Peers:[id:12847 store_id:1 ] EncryptionMeta:<nil> IsInFlashback:false FlashbackStartTs:0}"]
[2024/01/08 13:40:22.891 +08:00] [INFO] [split_region.go:242] ["start scatter region"] [regionID=12848]
[2024/01/08 13:40:22.891 +08:00] [INFO] [split_region.go:199] ["batch split regions, scatter region complete"] ["batch region ID"=12822] [at=7480000000000042625f698000000000000001038000000000000005e9666666666664] ["new region left"="{Id:12848 StartKey:7480000000000042ff625f698000000000ff0000010380000000ff000000049c999999ff9999980000000000fa EndKey:7480000000000042ff625f698000000000ff0000010380000000ff00000005e9666666ff6666640000000000fa RegionEpoch:{ConfVer:1 Version:1} Peers:[id:12849 store_id:1 ] EncryptionMeta:<nil> IsInFlashback:false FlashbackStartTs:0}"]
[2024/01/08 13:40:22.891 +08:00] [INFO] [split_region.go:242] ["start scatter region"] [regionID=12850]
[2024/01/08 13:40:22.891 +08:00] [INFO] [split_region.go:199] ["batch split regions, scatter region complete"] ["batch region ID"=12822] [at=7480000000000042625f69800000000000000103800000000000000736333333333330] ["new region left"="{Id:12850 StartKey:7480000000000042ff625f698000000000ff0000010380000000ff00000005e9666666ff6666640000000000fa EndKey:7480000000000042ff625f698000000000ff0000010380000000ff0000000736333333ff3333300000000000fa RegionEpoch:{ConfVer:1 Version:1} Peers:[id:12851 store_id:1 ] EncryptionMeta:<nil> IsInFlashback:false FlashbackStartTs:0}"]
[2024/01/08 13:40:22.891 +08:00] [INFO] [split_region.go:242] ["start scatter region"] [regionID=12852]
[2024/01/08 13:40:22.891 +08:00] [INFO] [split_region.go:199] ["batch split regions, scatter region complete"] ["batch region ID"=12822] [at=7480000000000042625f69800000000000000103800000000000000882fffffffffffc] ["new region left"="{Id:12852 StartKey:7480000000000042ff625f698000000000ff0000010380000000ff0000000736333333ff3333300000000000fa EndKey:7480000000000042ff625f698000000000ff0000010380000000ff0000000882fffffffffffffc0000000000fa RegionEpoch:{ConfVer:1 Version:1} Peers:[id:12853 store_id:1 ] EncryptionMeta:<nil> IsInFlashback:false FlashbackStartTs:0}"]
[2024/01/08 13:40:22.891 +08:00] [INFO] [split_region.go:242] ["start scatter region"] [regionID=12854]
[2024/01/08 13:40:22.891 +08:00] [INFO] [split_region.go:199] ["batch split regions, scatter region complete"] ["batch region ID"=12822] [at=7480000000000042625f698000000000000001038000000000000009cfccccccccccc8] ["new region left"="{Id:12854 StartKey:7480000000000042ff625f698000000000ff0000010380000000ff0000000882fffffffffffffc0000000000fa EndKey:7480000000000042ff625f698000000000ff0000010380000000ff00000009cfccccccffccccc80000000000fa RegionEpoch:{ConfVer:1 Version:1} Peers:[id:12855 store_id:1 ] EncryptionMeta:<nil> IsInFlashback:false FlashbackStartTs:0}"]
[2024/01/08 13:40:22.891 +08:00] [INFO] [split_region.go:242] ["start scatter region"] [regionID=12856]
[2024/01/08 13:40:22.891 +08:00] [INFO] [split_region.go:199] ["batch split regions, scatter region complete"] ["batch region ID"=12822] [at=7480000000000042625f69800000000000000103800000000000000b1c999999999994] ["new region left"="{Id:12856 StartKey:7480000000000042ff625f698000000000ff0000010380000000ff00000009cfccccccffccccc80000000000fa EndKey:7480000000000042ff625f698000000000ff0000010380000000ff0000000b1c999999ff9999940000000000fa RegionEpoch:{ConfVer:1 Version:1} Peers:[id:12857 store_id:1 ] EncryptionMeta:<nil> IsInFlashback:false FlashbackStartTs:0}"]
[2024/01/08 13:40:22.891 +08:00] [INFO] [split_region.go:242] ["start scatter region"] [regionID=12858]
[2024/01/08 13:40:22.891 +08:00] [INFO] [split_region.go:199] ["batch split regions, scatter region complete"] ["batch region ID"=12822] [at=7480000000000042625f69800000000000000103800000000000000c69666666666660] ["new region left"="{Id:12858 StartKey:7480000000000042ff625f698000000000ff0000010380000000ff0000000b1c999999ff9999940000000000fa EndKey:7480000000000042ff625f698000000000ff0000010380000000ff0000000c69666666ff6666600000000000fa RegionEpoch:{ConfVer:1 Version:1} Peers:[id:12859 store_id:1 ] EncryptionMeta:<nil> IsInFlashback:false FlashbackStartTs:0}"]
[2024/01/08 13:40:22.891 +08:00] [INFO] [split_region.go:242] ["start scatter region"] [regionID=12860]
[2024/01/08 13:40:22.891 +08:00] [INFO] [split_region.go:199] ["batch split regions, scatter region complete"] ["batch region ID"=12822] [at=7480000000000042625f69800000000000000103800000000000000db633333333332c] ["new region left"="{Id:12860 StartKey:7480000000000042ff625f698000000000ff0000010380000000ff0000000c69666666ff6666600000000000fa EndKey:7480000000000042ff625f698000000000ff0000010380000000ff0000000db6333333ff33332c0000000000fa RegionEpoch:{ConfVer:1 Version:1} Peers:[id:12861 store_id:1 ] EncryptionMeta:<nil> IsInFlashback:false FlashbackStartTs:0}"]
[2024/01/08 13:40:22.891 +08:00] [INFO] [split_region.go:242] ["start scatter region"] [regionID=12862]
[2024/01/08 13:40:22.891 +08:00] [INFO] [split_region.go:199] ["batch split regions, scatter region complete"] ["batch region ID"=12822] [at=7480000000000042635f698000000000000002] ["new region left"="{Id:12862 StartKey:7480000000000042ff625f698000000000ff0000010380000000ff0000000db6333333ff33332c0000000000fa EndKey:7480000000000042ff625f698000000000ff0000020000000000fa RegionEpoch:{ConfVer:1 Version:1} Peers:[id:12863 store_id:1 ] EncryptionMeta:<nil> IsInFlashback:false FlashbackStartTs:0}"]
[2024/01/08 13:40:22.891 +08:00] [INFO] [split_region.go:242] ["start scatter region"] [regionID=12864]
[2024/01/08 13:40:22.891 +08:00] [INFO] [split_region.go:199] ["batch split regions, scatter region complete"] ["batch region ID"=12822] [at=7480000000000042635f6980000000000000010380000000000000034fcccccccccccc] ["new region left"="{Id:12864 StartKey:7480000000000042ff625f698000000000ff0000020000000000fa EndKey:7480000000000042ff635f698000000000ff0000010380000000ff000000034fccccccffcccccc0000000000fa RegionEpoch:{ConfVer:1 Version:1} Peers:[id:12865 store_id:1 ] EncryptionMeta:<nil> IsInFlashback:false FlashbackStartTs:0}"]
[2024/01/08 13:40:22.891 +08:00] [INFO] [split_region.go:242] ["start scatter region"] [regionID=12866]
[2024/01/08 13:40:22.891 +08:00] [INFO] [split_region.go:199] ["batch split regions, scatter region complete"] ["batch region ID"=12822] [at=7480000000000042635f6980000000000000010380000000000000049c999999999998] ["new region left"="{Id:12866 StartKey:7480000000000042ff635f698000000000ff0000010380000000ff000000034fccccccffcccccc0000000000fa EndKey:7480000000000042ff635f698000000000ff0000010380000000ff000000049c999999ff9999980000000000fa RegionEpoch:{ConfVer:1 Version:1} Peers:[id:12867 store_id:1 ] EncryptionMeta:<nil> IsInFlashback:false FlashbackStartTs:0}"]
[2024/01/08 13:40:22.891 +08:00] [INFO] [split_region.go:242] ["start scatter region"] [regionID=12868]
[2024/01/08 13:40:22.891 +08:00] [INFO] [split_region.go:199] ["batch split regions, scatter region complete"] ["batch region ID"=12822] [at=7480000000000042635f698000000000000001038000000000000005e9666666666664] ["new region left"="{Id:12868 StartKey:7480000000000042ff635f698000000000ff0000010380000000ff000000049c999999ff9999980000000000fa EndKey:7480000000000042ff635f698000000000ff0000010380000000ff00000005e9666666ff6666640000000000fa RegionEpoch:{ConfVer:1 Version:1} Peers:[id:12869 store_id:1 ] EncryptionMeta:<nil> IsInFlashback:false FlashbackStartTs:0}"]
[2024/01/08 13:40:22.891 +08:00] [INFO] [split_region.go:242] ["start scatter region"] [regionID=12870]
[2024/01/08 13:40:22.891 +08:00] [INFO] [split_region.go:199] ["batch split regions, scatter region complete"] ["batch region ID"=12822] [at=7480000000000042635f69800000000000000103800000000000000736333333333330] ["new region left"="{Id:12870 StartKey:7480000000000042ff635f698000000000ff0000010380000000ff00000005e9666666ff6666640000000000fa EndKey:7480000000000042ff635f698000000000ff0000010380000000ff0000000736333333ff3333300000000000fa RegionEpoch:{ConfVer:1 Version:1} Peers:[id:12871 store_id:1 ] EncryptionMeta:<nil> IsInFlashback:false FlashbackStartTs:0}"]
[2024/01/08 13:40:22.891 +08:00] [INFO] [split_region.go:242] ["start scatter region"] [regionID=12872]
[2024/01/08 13:40:22.891 +08:00] [INFO] [split_region.go:199] ["batch split regions, scatter region complete"] ["batch region ID"=12822] [at=7480000000000042635f69800000000000000103800000000000000882fffffffffffc] ["new region left"="{Id:12872 StartKey:7480000000000042ff635f698000000000ff0000010380000000ff0000000736333333ff3333300000000000fa EndKey:7480000000000042ff635f698000000000ff0000010380000000ff0000000882fffffffffffffc0000000000fa RegionEpoch:{ConfVer:1 Version:1} Peers:[id:12873 store_id:1 ] EncryptionMeta:<nil> IsInFlashback:false FlashbackStartTs:0}"]
[2024/01/08 13:40:22.891 +08:00] [INFO] [split_region.go:242] ["start scatter region"] [regionID=12874]
[2024/01/08 13:40:22.891 +08:00] [INFO] [split_region.go:199] ["batch split regions, scatter region complete"] ["batch region ID"=12822] [at=7480000000000042635f698000000000000001038000000000000009cfccccccccccc8] ["new region left"="{Id:12874 StartKey:7480000000000042ff635f698000000000ff0000010380000000ff0000000882fffffffffffffc0000000000fa EndKey:7480000000000042ff635f698000000000ff0000010380000000ff00000009cfccccccffccccc80000000000fa RegionEpoch:{ConfVer:1 Version:1} Peers:[id:12875 store_id:1 ] EncryptionMeta:<nil> IsInFlashback:false FlashbackStartTs:0}"]
[2024/01/08 13:40:22.891 +08:00] [INFO] [split_region.go:242] ["start scatter region"] [regionID=12876]
[2024/01/08 13:40:22.891 +08:00] [INFO] [split_region.go:199] ["batch split regions, scatter region complete"] ["batch region ID"=12822] [at=7480000000000042635f69800000000000000103800000000000000b1c999999999994] ["new region left"="{Id:12876 StartKey:7480000000000042ff635f698000000000ff0000010380000000ff00000009cfccccccffccccc80000000000fa EndKey:7480000000000042ff635f698000000000ff0000010380000000ff0000000b1c999999ff9999940000000000fa RegionEpoch:{ConfVer:1 Version:1} Peers:[id:12877 store_id:1 ] EncryptionMeta:<nil> IsInFlashback:false FlashbackStartTs:0}"]
[2024/01/08 13:40:22.891 +08:00] [INFO] [split_region.go:242] ["start scatter region"] [regionID=12878]
[2024/01/08 13:40:22.891 +08:00] [INFO] [split_region.go:199] ["batch split regions, scatter region complete"] ["batch region ID"=12822] [at=7480000000000042635f69800000000000000103800000000000000c69666666666660] ["new region left"="{Id:12878 StartKey:7480000000000042ff635f698000000000ff0000010380000000ff0000000b1c999999ff9999940000000000fa EndKey:7480000000000042ff635f698000000000ff0000010380000000ff0000000c69666666ff6666600000000000fa RegionEpoch:{ConfVer:1 Version:1} Peers:[id:12879 store_id:1 ] EncryptionMeta:<nil> IsInFlashback:false FlashbackStartTs:0}"]
[2024/01/08 13:40:22.891 +08:00] [INFO] [split_region.go:242] ["start scatter region"] [regionID=12880]
[2024/01/08 13:40:22.891 +08:00] [INFO] [split_region.go:199] ["batch split regions, scatter region complete"] ["batch region ID"=12822] [at=7480000000000042635f69800000000000000103800000000000000db633333333332c] ["new region left"="{Id:12880 StartKey:7480000000000042ff635f698000000000ff0000010380000000ff0000000c69666666ff6666600000000000fa EndKey:7480000000000042ff635f698000000000ff0000010380000000ff0000000db6333333ff33332c0000000000fa RegionEpoch:{ConfVer:1 Version:1} Peers:[id:12881 store_id:1 ] EncryptionMeta:<nil> IsInFlashback:false FlashbackStartTs:0}"]
[2024/01/08 13:40:22.891 +08:00] [INFO] [split_region.go:242] ["start scatter region"] [regionID=12882]
[2024/01/08 13:40:22.891 +08:00] [INFO] [conn.go:1155] ["command dispatched failed"] [conn=2098834] [session_alias=] [connInfo="id:2098834, addr:127.0.0.1:46176 status:10, collation:utf8mb4_general_ci, user:root"] [command=Query] [status="inTxn:0, autocommit:1"] [sql="split table tlist index a between (2) and (15) regions 10;"] [txn_mode=PESSIMISTIC] [timestamp=0] [err="runtime error: index out of range [30] with length 30"]

2. What did you expect to see? (Required)

No panic

3. What did you see instead (Required)

panic

4. What is your TiDB version? (Required)

test_logs.tar.gz
@Defined2014 Defined2014 added type/bug The issue is confirmed as a bug. component/test severity/moderate labels Jan 8, 2024
@Defined2014
Copy link
Contributor Author

Defined2014 commented Jan 8, 2024 •
edited
Loading

Although no stack in log, I think it's panic on https://github.com/tikv/client-go/blob/c7d29aafa7073f64555532d0d10f19714bc8e6c2/tikv/split_region.go#L201 .

The returned region count is greatter than given keys numbers.

@Defined2014
Copy link
Contributor Author

Defined2014 commented Jan 8, 2024 •
edited
Loading

Seems we combined two regions (regionID=12824 and regionID=12822) in one batch and it also passed epoch check in unistore.

@jebter jebter added the sig/sql-infra SIG: SQL Infra label Jan 9, 2024
@Defined2014
Copy link
Contributor Author

Defined2014 commented Jan 9, 2024 •
edited
Loading

I think the problem is like

  1. We check the regionCtx and it is correct. Then unlock the MockRegionManager.
  2. Another goroutine split the region, update region epoch.
  3. Call newRegions, err := rm.splitKeys(splitKeys) got one more newRegions.

Modify code like this

diff --git a/pkg/store/mockstore/unistore/tikv/mock_region.go b/pkg/store/mockstore/unistore/tikv/mock_region.go
index 709d4c10f2..cf209603f5 100644
--- a/pkg/store/mockstore/unistore/tikv/mock_region.go
+++ b/pkg/store/mockstore/unistore/tikv/mock_region.go
@@ -403,6 +403,13 @@ func (rm *MockRegionManager) SplitRegion(req *kvrpcpb.SplitRegionRequest) *kvrpc
        }
        slices.SortFunc(splitKeys, bytes.Compare)

+       if len(splitKeys) < 3 {
+               time.Sleep(time.Second * 1)
+       }
+       if len(splitKeys) > 10 {
+               time.Sleep(time.Second * 2)
+       }
+
        newRegions, err := rm.splitKeys(splitKeys)
        if err != nil {
                return &kvrpcpb.SplitRegionResponse{RegionError: &errorpb.Error{Message: err.Error()}}

and got the same panic message.

@Defined2014 Defined2014 mentioned this issue Jan 10, 2024
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Jan 10, 2024
@Defined2014
store: let MockRegionManager check region epoch and split region in…
eaab02f 
… same lock (#50250)

close #50177
@bb7133 bb7133 mentioned this issue Mar 27, 2024
Merged
18 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
component/test severity/moderate sig/sql-infra SIG: SQL Infra type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

store: let MockRegionManager check region epoch and split region in same lock Defined2014/tidb
2 participants
@jebter @Defined2014