ddl: speed up the operation of "create table" #6861

zimulala · 2018-06-19T10:49:00Z

What have you changed? (mandatory)

Handle table split operations asynchronously.

What are the type of the changes (mandatory)?

Improvement

Does this PR affect documentation (docs/docs-cn) update? (optional)

No.

Benchmark result if necessary (optional)

Using TiKV
The test code as follows：

start := time.Now()
	for j := 0; j < 1000; j++ {
		sql = fmt.Sprintf("create table db%d.t%d(a int, b int, c int, d varchar(255), e varchar(255))", i, j)
		if _, err = db.Exec(sql); err != nil {
			log.Errorf("create table err %v", err)
		}
	}
	log.Infof("exec sqls take time %v", time.Since(start))

use the original code, the result as follows:
2018/05/18 17:37:16 tidb_create.go:35: [info] exec sql use db0
2018/05/18 17:41:09 tidb_create.go:44: [info] exec sqls take time 3m53.493631103s
2018/05/18 17:41:09 tidb_create.go:35: [info] exec sql use db1
2018/05/18 17:45:03 tidb_create.go:44: [info] exec sqls take time 3m53.914944715s
2018/05/18 17:45:03 tidb_create.go:35: [info] exec sql use db2
2018/05/18 17:48:57 tidb_create.go:44: [info] exec sqls take time 3m53.528751212s
2018/05/18 17:48:57 tidb_create.go:35: [info] exec sql use db3
2018/05/18 17:52:56 tidb_create.go:44: [info] exec sqls take time 3m59.223324563s
2018/05/18 17:52:56 tidb_create.go:35: [info] exec sql use db4
2018/05/18 17:56:56 tidb_create.go:44: [info] exec sqls take time 3m59.867498659s
2018/05/18 17:56:56 tidb_create.go:35: [info] exec sql use db5
2018/05/18 18:01:01 tidb_create.go:44: [info] exec sqls take time 4m5.327908179s
2018/05/18 18:01:01 tidb_create.go:35: [info] exec sql use db6
2018/05/18 18:05:09 tidb_create.go:44: [info] exec sqls take time 4m7.886515515s
2018/05/18 18:05:09 tidb_create.go:35: [info] exec sql use db7
2018/05/18 18:09:15 tidb_create.go:44: [info] exec sqls take time 4m5.60381973s
2018/05/18 18:09:15 tidb_create.go:35: [info] exec sql use db8
2018/05/18 18:13:32 tidb_create.go:44: [info] exec sqls take time 4m16.640030126s

use the new code, the result as follows:
2018/05/18 18:42:03 tidb_create.go:35: [info] exec sql use db0
2018/05/18 18:44:08 tidb_create.go:44: [info] exec sqls take time 2m4.89636175s
2018/05/18 18:44:08 tidb_create.go:35: [info] exec sql use db1
2018/05/18 18:46:14 tidb_create.go:44: [info] exec sqls take time 2m5.840172046s
2018/05/18 18:46:14 tidb_create.go:35: [info] exec sql use db2
2018/05/18 18:48:22 tidb_create.go:44: [info] exec sqls take time 2m7.847079946s
2018/05/18 18:48:22 tidb_create.go:35: [info] exec sql use db3
2018/05/18 18:50:32 tidb_create.go:44: [info] exec sqls take time 2m9.464254105s
2018/05/18 18:50:32 tidb_create.go:35: [info] exec sql use db4
2018/05/18 18:52:42 tidb_create.go:44: [info] exec sqls take time 2m10.56319663s
2018/05/18 18:52:43 tidb_create.go:35: [info] exec sql use db5
2018/05/18 18:54:57 tidb_create.go:44: [info] exec sqls take time 2m14.954220944s
2018/05/18 18:54:58 tidb_create.go:35: [info] exec sql use db6
2018/05/18 18:57:14 tidb_create.go:44: [info] exec sqls take time 2m16.526956612s
2018/05/18 18:57:14 tidb_create.go:35: [info] exec sql use db7
2018/05/18 18:59:33 tidb_create.go:44: [info] exec sqls take time 2m18.566134241s
2018/05/18 18:59:33 tidb_create.go:35: [info] exec sql use db8
2018/05/18 19:02:33 tidb_create.go:44: [info] exec sqls take time 3m0.43806444s

jackysp · 2018-06-19T11:23:28Z

/run-all-tests

alivxxx · 2018-06-20T02:56:07Z

ddl/table.go

@@ -130,19 +127,23 @@ func onDropTable(t *meta.Meta, job *model.Job) (ver int64, _ error) {
 	return ver, errors.Trace(err)
 }

-func splitTableRegion(store kv.Storage, tableID int64) error {
+type splitableStore interface {


Why add it?

I don't add it, I only move it from splitTableRegion to here.

But it still appears in splitTableRegion.

Should we remove the code at line 135~137?

shenli · 2018-06-20T09:19:51Z

ddl/table.go

 	}
 	tableStartKey := tablecodec.GenTablePrefix(tableID)
 	if err := s.SplitRegion(tableStartKey); err != nil {
-		return errors.Trace(err)
+		// It will be automatically splitting by TiKV later.


s/splitting/splited/

zimulala · 2018-06-22T02:07:03Z

PTAL @lamxTyler @shenli

shenli · 2018-06-22T02:21:07Z

LGTM

shenli · 2018-06-22T02:21:38Z

@winkyao @lamxTyler PTAL

alivxxx

LGTM

coocood

LGTM

zimulala added 3 commits May 21, 2018 11:05

ddl: update create table

9852c0d

Merge branch 'master' into zimuxia/create-table

1c73a79

ddl: clean up

0c942f9

zimulala added the DDL-need-LGT3 label Jun 19, 2018

jackysp added the all-tests-passed label Jun 19, 2018

alivxxx reviewed Jun 20, 2018

View reviewed changes

shenli reviewed Jun 20, 2018

View reviewed changes

ddl: address comments

0a34f49

Merge branch 'master' into create-table

68f8fa4

shenli added the status/LGT1 Indicates that a PR has LGTM 1. label Jun 22, 2018

alivxxx reviewed Jun 22, 2018

View reviewed changes

alivxxx added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Jun 22, 2018

coocood approved these changes Jun 25, 2018

View reviewed changes

coocood added status/LGT3 The PR has already had 3 LGTM. and removed status/LGT2 Indicates that a PR has LGTM 2. labels Jun 25, 2018

coocood and others added 4 commits June 25, 2018 10:55

Merge branch 'master' into create-table

618dfe0

Merge branch 'master' into create-table

a6e2cd3

Merge branch 'master' into create-table

44e9e03

Merge branch 'master' into create-table

c21d0c6

coocood merged commit 6edbf58 into pingcap:master Jun 25, 2018

zimulala deleted the create-table branch June 26, 2018 12:59

zimulala mentioned this pull request Sep 17, 2019

executor: fix scatter region timeout issues and "show processlist" display issues #12057

Merged

you06 added the sig/sql-infra SIG: SQL Infra label Mar 4, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ddl: speed up the operation of "create table" #6861

ddl: speed up the operation of "create table" #6861

zimulala commented Jun 19, 2018

jackysp commented Jun 19, 2018

alivxxx Jun 20, 2018

zimulala Jun 20, 2018

alivxxx Jun 20, 2018

shenli Jun 20, 2018

shenli Jun 20, 2018

zimulala commented Jun 22, 2018

shenli commented Jun 22, 2018

shenli commented Jun 22, 2018

alivxxx left a comment

coocood left a comment

ddl: speed up the operation of "create table" #6861

ddl: speed up the operation of "create table" #6861

Conversation

zimulala commented Jun 19, 2018

What have you changed? (mandatory)

What are the type of the changes (mandatory)?

Does this PR affect documentation (docs/docs-cn) update? (optional)

Benchmark result if necessary (optional)

jackysp commented Jun 19, 2018

alivxxx Jun 20, 2018

Choose a reason for hiding this comment

zimulala Jun 20, 2018

Choose a reason for hiding this comment

alivxxx Jun 20, 2018

Choose a reason for hiding this comment

shenli Jun 20, 2018

Choose a reason for hiding this comment

shenli Jun 20, 2018

Choose a reason for hiding this comment

zimulala commented Jun 22, 2018

shenli commented Jun 22, 2018

shenli commented Jun 22, 2018

alivxxx left a comment

Choose a reason for hiding this comment

coocood left a comment

Choose a reason for hiding this comment