table: avoid redundant encode in insert/update/delete. #11532

crazycs520 · 2019-07-30T12:54:32Z

What problem does this PR solve?

avoid redundant encode in AddRecord.

Benchmark AddRecord function

-- create table with 10 timestamp column.
create table t (c1 timestamp, c2 timestamp .... c10 timestamp);

-- benchmark AddRecord function. The test function was below.

-- before this pr.
AddRecord              7.8s/100w ops

-- this pr
AddRecord              6.0s/100w ops

benchmark insert

Use prepare to batch insert 50w rows to table, which table contains 60 columns, 20 int columns, 20 timestamp columns, 20 varchar(50) columns.

-- before this pr
insert 50w rows spend 42s.

-- this pr
insert 50w rows spend 45s.

Before This PR

This PR

benchmark update

update t_wide set c0=c0+1 where c0>0          and c0<20000;
update t_wide set c0=c0+1 where c0>20000 and c0<40000;
update t_wide set c0=c0+1 where c0>40000 and c0<60000;
. . .

-- before this pr
update 50w rows spend 34s.

-- this pr
update 50w rows spend 41s.

Before This PR

This PR

benchmark delete

set @@tidb_batch_delete=1;
set @@tidb_dml_batch_size=2000;

-- before this pr
delete 50w rows spend 16s.

-- this pr
delete 50w rows spend 13s.

Before This PR

This PR

What is changed and how it works?

Check List

Tests

No test. Tiny refactor.
benchmark AddRecord function code.

// Add below test in table/tables/tables_test.go and run `go test -check.f "TestAddRecordForBench"`
func (ts *testSuite) TestAddRecordForBench(c *C) {
	ctx := context.Background()
	_, err := ts.se.Execute(ctx, "drop table if exists test.t")
	c.Assert(err, IsNil)

	benchNum := 1000000
	colNum := 10
	colTp := "timestamp"
	colValue := types.Time{Time: types.FromGoTime(time.Now()), Type: mysql.TypeTimestamp, Fsp: 6}

	sql := "CREATE TABLE test.t ("
	for i := 0; i < colNum; i++ {
		if i > 0 {
			sql += ", "
		}
		sql += fmt.Sprintf("c%d %s", i, colTp)
	}
	sql += ")"

	_, err = ts.se.Execute(ctx, sql)
	c.Assert(err, IsNil)
	tb, err := ts.dom.InfoSchema().TableByName(model.NewCIStr("test"), model.NewCIStr("t"))
	c.Assert(err, IsNil)

	values := make([]interface{}, 0, colNum)
	for i := 0; i < colNum; i++ {
		values = append(values, colValue)
	}

	sctx := ts.se
	c.Assert(sctx.NewTxn(ctx), IsNil)
	txn, err := sctx.Txn(true)
	c.Assert(err, IsNil)
	start := time.Now()
	row := types.MakeDatums(values...)
	for i := 0; i < benchNum; i++ {
		if i%100000 == 0 {
			c.Assert(txn.Rollback(), IsNil)
			c.Assert(sctx.NewTxn(ctx), IsNil)
			txn, err = sctx.Txn(true)
			c.Assert(err, IsNil)
		}
		_, err = tb.AddRecord(sctx, row)
		c.Assert(err, IsNil)
	}
	fmt.Printf("\n\nbench add recored : %v\n\n", time.Since(start).Seconds())
	c.Assert(txn.Rollback(), IsNil)
}

Code changes

Side effects

Related changes

crazycs520 · 2019-07-30T13:01:44Z

/run-all-tests

crazycs520 · 2019-07-30T13:01:56Z

@eurekaka PTAL

codecov · 2019-07-30T13:06:25Z

Codecov Report

Merging #11532 into master will decrease coverage by 0.5225%.
The diff coverage is 93.75%.

@@               Coverage Diff               @@
##             master    #11532        +/-   ##
===============================================
- Coverage   81.7785%   81.256%   -0.5226%     
===============================================
  Files           426       426                
  Lines         94345     92035      -2310     
===============================================
- Hits          77154     74784      -2370     
- Misses        11788     11872        +84     
+ Partials       5403      5379        -24

table/tables/tables.go

tablecodec/tablecodec.go

crazycs520 · 2019-08-02T07:38:35Z

@eurekaka PTAL

crazycs520 · 2019-08-02T07:40:10Z

/run-all-tests

tablecodec/tablecodec.go

…nto opt-add-record

table/tables/tables.go

…nto opt-add-record

crazycs520 · 2019-08-06T09:21:13Z

/run-all-tests

eurekaka

LGTM

table/tables/tables.go

crazycs520 · 2019-08-06T10:56:07Z

/run-all-tests

jackysp · 2019-08-06T10:58:44Z

util/codec/codec.go

@@ -201,6 +237,21 @@ func encodeSignedInt(b []byte, v int64, comparable bool) []byte {
 	return b
 }

+var varIntSize = []int64{64, 8192, 1048576, 134217728, 17179869184, 2199023255552, 281474976710656, 36028797018963968, 4611686018427387904}


Is it possible to use some bit shifts?

great, thanks.done.

jackysp · 2019-08-06T10:58:53Z

util/codec/codec.go

@@ -212,6 +263,17 @@ func encodeUnsignedInt(b []byte, v uint64, comparable bool) []byte {
 	return b
 }

+var varUintSize = []uint64{128, 16384, 2097152, 268435456, 34359738368, 4398046511104, 562949953421312, 72057594037927936, 9223372036854775808}


great, thanks.done.

jackysp

rest LGTM

crazycs520 · 2019-08-06T13:12:27Z

/run-all-tests

crazycs520 · 2019-08-06T13:19:54Z

/run-unit-test

crazycs520 added 3 commits July 30, 2019 18:06

table: remove redundant encode in AddRecord

6cbf433

add column size for generated column

6da2bce

add comment

786520f

crazycs520 added the type/enhancement The issue or PR belongs to an enhancement. label Jul 30, 2019

Merge branch 'master' into opt-add-record

9535048

jackysp reviewed Aug 2, 2019

View reviewed changes

table/tables/tables.go Outdated Show resolved Hide resolved

jackysp reviewed Aug 2, 2019

View reviewed changes

tablecodec/tablecodec.go Outdated Show resolved Hide resolved

address comment

e6f0b8c

Merge branch 'master' into opt-add-record

1e58342

bb7133 reviewed Aug 2, 2019

View reviewed changes

tablecodec/tablecodec.go Outdated Show resolved Hide resolved

Merge branch 'master' into opt-add-record

ef1c1dc

sre-bot mentioned this pull request Aug 5, 2019

util: update the minimum version of tikv information #11608

Merged

crazycs520 added 4 commits August 5, 2019 16:51

add comment

95c174c

Merge branch 'master' into opt-add-record

7355cbe

Merge branch 'opt-add-record' of https://github.com/crazycs520/tidb i…

dbc7f01

…nto opt-add-record

Merge branch 'master' into opt-add-record

033fefb

jackysp reviewed Aug 6, 2019

View reviewed changes

table/tables/tables.go Outdated Show resolved Hide resolved

crazycs520 added the status/WIP label Aug 6, 2019

crazycs520 added 3 commits August 6, 2019 17:18

add estimate value size function to collect column value size

09dbef7

Merge branch 'opt-add-record' of https://github.com/crazycs520/tidb i…

bd24f91

…nto opt-add-record

Merge branch 'master' into opt-add-record

f478d9c

avoid update and delete redundant encode logic

5ce9c20

eurekaka reviewed Aug 6, 2019

View reviewed changes

table/tables/tables.go Outdated Show resolved Hide resolved

eurekaka added the status/LGT1 Indicates that a PR has LGTM 1. label Aug 6, 2019

crazycs520 removed the status/WIP label Aug 6, 2019

jackysp reviewed Aug 6, 2019

View reviewed changes

address comment

0cfe15c

crazycs520 changed the title ~~table: avoid redundant encode in AddRecord~~ table: avoid redundant encode in insert/update/delete. Aug 6, 2019

jackysp added the status/all tests passed label Aug 7, 2019

jackysp approved these changes Aug 7, 2019

View reviewed changes

jackysp merged commit 0dc9106 into pingcap:master Aug 7, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

table: avoid redundant encode in insert/update/delete. #11532

table: avoid redundant encode in insert/update/delete. #11532

crazycs520 commented Jul 30, 2019 •

edited

Loading

crazycs520 commented Jul 30, 2019

crazycs520 commented Jul 30, 2019

codecov bot commented Jul 30, 2019 •

edited

Loading

crazycs520 commented Aug 2, 2019

crazycs520 commented Aug 2, 2019

crazycs520 commented Aug 6, 2019

eurekaka left a comment

crazycs520 commented Aug 6, 2019

jackysp Aug 6, 2019

crazycs520 Aug 6, 2019

jackysp Aug 6, 2019

crazycs520 Aug 6, 2019

jackysp left a comment

crazycs520 commented Aug 6, 2019

crazycs520 commented Aug 6, 2019

table: avoid redundant encode in insert/update/delete. #11532

table: avoid redundant encode in insert/update/delete. #11532

Conversation

crazycs520 commented Jul 30, 2019 • edited Loading

What problem does this PR solve?

Benchmark AddRecord function

benchmark insert

benchmark update

benchmark delete

What is changed and how it works?

Check List

crazycs520 commented Jul 30, 2019

crazycs520 commented Jul 30, 2019

codecov bot commented Jul 30, 2019 • edited Loading

Codecov Report

crazycs520 commented Aug 2, 2019

crazycs520 commented Aug 2, 2019

crazycs520 commented Aug 6, 2019

eurekaka left a comment

Choose a reason for hiding this comment

crazycs520 commented Aug 6, 2019

jackysp Aug 6, 2019

Choose a reason for hiding this comment

crazycs520 Aug 6, 2019

Choose a reason for hiding this comment

jackysp Aug 6, 2019

Choose a reason for hiding this comment

crazycs520 Aug 6, 2019

Choose a reason for hiding this comment

jackysp left a comment

Choose a reason for hiding this comment

crazycs520 commented Aug 6, 2019

crazycs520 commented Aug 6, 2019

crazycs520 commented Jul 30, 2019 •

edited

Loading

codecov bot commented Jul 30, 2019 •

edited

Loading