*: add split index region syntax #10203

crazycs520 · 2019-04-19T14:47:12Z

What problem does this PR solve?

PR: #10138 support pre-split table region. But it will only split 1 region for per index. Then the index region may still be the hot region and the write bottleneck.

What is changed and how it works?

This PR support split index region by SQL. Syntax like below:

SPLIT TABLE table_name INDEX index_name BY (val1...), (val2...),(val3...) ...

example

SPLIT INDEX t1 INDEX idx1 (100), (1000), (10000);   
//range is : 0~100, 100 ~1000, 1000~ 10000 ， 10000 ~ maxInt64

split table t1 idx2 ("a", "2000-01-01 00:00:01"),  ("b", "2019-04-17 14:26:19"),  ("c", "");  
// range is: minIndexValue ~ ("a", "2000-01-01 00:00:01"),
("a", "2000-01-01 00:00:01") ~ ("b", "2019-04-17 14:26:19"),
("b", "2019-04-17 14:26:19") ~ ("c", ""),
("c", "") ~ maxIndexValue.

Please merge #10138 firstly.
Related parser PR: pingcap/parser#297

Check List

Tests

Adding test.

Code changes

Has exported function/method change

Side effects

Related changes

…region

…ndex

nolouch

LGTM

nolouch · 2019-04-25T12:55:52Z

executor/split.go

+	regionIDs := make([]uint64, 0, len(e.valueLists))
+	index := tables.NewIndex(e.table.Meta().ID, e.table.Meta(), e.indexInfo)
+	for _, values := range e.valueLists {
+		idxKey, _, err := index.GenIndexKey(e.ctx.GetSessionVars().StmtCtx, values, 1, nil)


1 means min handle id?

yes, Maybe 0 is better?

you can use a constant, better than hard code!

@winkyao
I think 1 isn't always the min handle. If a table has a PKIsHandle column, then the handle may be a negative value.
So I think we need to use math.MinInt64.

@zimulala The key is encoding in binary format, so negative value is always greater than a positive value. @nolouch Right?

Here is a test:
0 encode to byte is: [128 0 0 0 0 0 0 0]
math.MinInt64 encode to byte is: [0 0 0 0 0 0 0 0]

so, math.MinInt64 is better.

store/tikv/split_region.go

planner/core/planbuilder.go

…cate err

…ync split region.

codecov · 2019-04-28T02:28:09Z

Codecov Report

Merging #10203 into master will decrease coverage by 0.0206%.
The diff coverage is 63.5135%.

@@               Coverage Diff                @@
##             master     #10203        +/-   ##
================================================
- Coverage   77.6772%   77.6566%   -0.0206%     
================================================
  Files           411        412         +1     
  Lines         85460      85511        +51     
================================================
+ Hits          66383      66405        +22     
- Misses        14119      14141        +22     
- Partials       4958       4965         +7

zimulala · 2019-04-29T07:28:32Z

planner/core/planbuilder.go

+		for j, valueItem := range valuesItem {
+			x, ok := valueItem.(*driver.ValueExpr)
+			if !ok {
+				return nil, errors.Trace(errors.New("expect constant values"))


Could we remove this trace? Other places don't use it.
And we can print the value to the error message.

zimulala · 2019-04-29T08:00:08Z

executor/split.go

+	regionIDs := make([]uint64, 0, len(e.valueLists))
+	index := tables.NewIndex(e.table.Meta().ID, e.table.Meta(), e.indexInfo)
+	for _, values := range e.valueLists {
+		idxKey, _, err := index.GenIndexKey(e.ctx.GetSessionVars().StmtCtx, values, 1, nil)


@winkyao
I think 1 isn't always the min handle. If a table has a PKIsHandle column, then the handle may be a negative value.
So I think we need to use math.MinInt64.

executor/executor.go

…ndex

planner/core/planbuilder.go

winkyao

LGTM

executor/split.go

zimulala · 2019-05-06T09:14:14Z

executor/split.go

+		err := s.WaitScatterRegionFinish(regionID)
+		if err != nil {
+			logutil.Logger(context.Background()).Warn("wait scatter region failed",
+				zap.Uint64("regionID", regionID),


Do we need to add the table and index information to this log？

go.mod

…ndex

crazycs520 · 2019-05-06T09:37:21Z

/run-all-tests

zimulala

LGTM

crazycs520 added 16 commits April 12, 2019 13:55

init

5c613ad

add scatter

2bc50c8

fix test

a88ea9b

only split region for positive num

3145795

add PreSplitRegions option

2fccfab

add retry for scatter region

77f0a96

refine comment

189335d

Merge branch 'master' into splite-region

03783c0

remove blank line

b517f03

Merge branch 'master' into splite-region

f96d8d3

update go.mod for pd

55cd7be

add wait scatter finish

eb27dfb

add table option: wait_split_finish

a70696f

Merge branch 'master' of https://github.com/pingcap/tidb into splite-…

df45ac4

…region

split and scatter region first, wait scatter finish together

e362d68

*: add split table index syntax

918b183

crazycs520 added the type/usability label Apr 19, 2019

crazycs520 requested a review from nolouch April 19, 2019 14:50

wait scatter finish when create table

87ec41a

winkyao added the component/DDL-need-LGT3 label Apr 22, 2019

crazycs520 added 2 commits April 25, 2019 09:23

Merge branch 'master' of https://github.com/pingcap/tidb into split_i…

58c4a9e

…ndex

refine code

6cfd51e

crazycs520 added sig/execution SIG execution and removed component/DDL-need-LGT3 labels Apr 25, 2019

crazycs520 added 2 commits April 25, 2019 09:33

Merge branch 'master' of https://github.com/pingcap/tidb into split_i…

f1b22a3

…ndex

add comment

952b870

nolouch reviewed Apr 25, 2019

View reviewed changes

zimulala reviewed Apr 26, 2019

View reviewed changes

store/tikv/split_region.go Outdated Show resolved Hide resolved

winkyao reviewed Apr 26, 2019

View reviewed changes

planner/core/planbuilder.go Outdated Show resolved Hide resolved

winkyao reviewed Apr 26, 2019

View reviewed changes

planner/core/planbuilder.go Outdated Show resolved Hide resolved

crazycs520 added 3 commits April 26, 2019 22:01

address comment and add test for convert value should not ignore trun…

38a057d

…cate err

Merge branch 'master' into split_index

53ccd4f

use session variable tidb_wait_table_split_finish to chose sync or as…

b44db0b

…ync split region.

update go mod

15c9514

zimulala reviewed Apr 29, 2019

View reviewed changes

winkyao reviewed May 6, 2019

View reviewed changes

executor/executor.go Show resolved Hide resolved

crazycs520 added 2 commits May 6, 2019 10:27

address comment

650c428

Merge branch 'master' of https://github.com/pingcap/tidb into split_i…

77ca0d1

…ndex

winkyao reviewed May 6, 2019

View reviewed changes

planner/core/planbuilder.go Outdated Show resolved Hide resolved

winkyao reviewed May 6, 2019

View reviewed changes

planner/core/planbuilder.go Outdated Show resolved Hide resolved

address comment

1d2e9c9

winkyao reviewed May 6, 2019

View reviewed changes

zimulala reviewed May 6, 2019

View reviewed changes

zimulala added the status/LGT1 Indicates that a PR has LGTM 1. label May 6, 2019

zimulala reviewed May 6, 2019

View reviewed changes

go.mod Outdated Show resolved Hide resolved

crazycs520 added 2 commits May 6, 2019 17:27

address comment and update go.mod

44016a5

Merge branch 'master' of https://github.com/pingcap/tidb into split_i…

652e95c

…ndex

zimulala approved these changes May 6, 2019

View reviewed changes

zimulala added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels May 6, 2019

crazycs520 merged commit 7ecb315 into pingcap:master May 6, 2019

crazycs520 added a commit to crazycs520/tidb that referenced this pull request Jun 19, 2019

*: add split index region syntax (pingcap#10203)

19f29ef

crazycs520 mentioned this pull request Jun 19, 2019

*: add split index region syntax (#10203) #10865

Merged

zz-jason pushed a commit that referenced this pull request Jun 19, 2019

*: add split index region syntax (#10203) (#10865)

97ebef8

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

*: add split index region syntax #10203

*: add split index region syntax #10203

crazycs520 commented Apr 19, 2019 •

edited

Loading

nolouch left a comment

nolouch Apr 25, 2019

crazycs520 Apr 26, 2019

winkyao Apr 29, 2019

zimulala Apr 29, 2019

winkyao May 6, 2019

crazycs520 May 6, 2019 •

edited

Loading

codecov bot commented Apr 28, 2019 •

edited

Loading

zimulala Apr 29, 2019

crazycs520 May 6, 2019

zimulala Apr 29, 2019

winkyao left a comment

zimulala May 6, 2019

crazycs520 May 6, 2019

crazycs520 commented May 6, 2019

zimulala left a comment

*: add split index region syntax #10203

*: add split index region syntax #10203

Conversation

crazycs520 commented Apr 19, 2019 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

nolouch left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

crazycs520 May 6, 2019 • edited Loading

Choose a reason for hiding this comment

codecov bot commented Apr 28, 2019 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

winkyao left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

crazycs520 commented May 6, 2019

zimulala left a comment

Choose a reason for hiding this comment

crazycs520 commented Apr 19, 2019 •

edited

Loading

crazycs520 May 6, 2019 •

edited

Loading

codecov bot commented Apr 28, 2019 •

edited

Loading