*: use chunk grow for simple executor #7540

lysu · 2018-08-29T11:58:18Z

What problem does this PR solve?

ref issue #7077 and base on PR #7473

in this PR we start to make executor use chunk grow.

What is changed and how it works?

There are several small modification in this PR.

modify ChunkList use chunk grow, more bigger chunk node as more chunk node inserted
Prepare, Deallocate, Simple, Set, Insert, Dual, Sort, MaxOneRow, Update, Delete, TopN, Limit use NoDataCap for exec result chunk
CancelDDLJobsExec, ShowDDLJobQueriesExec, ShowDDLJobsExec, SelectionExec, TableScanExec, ExplainExec, ShowExec, UnionScanExec check batch size use chunk.Capacity() instead of baseExecutor.maxChunkSize
use GrowAndReset to replace Reset in some executors
replace some chunk.NewChunkWithCapacity(which will be depercated soon) with chunk.New

This PR doesn't modify join/agg/distsql/sort operators which are the most sophisticated. We will handle them in next PR.

Check List

Tests

Unit test
Integration test

Code changes

Refactor

Side effects

No

Related changes

No for this PR.

This change is

lysu · 2018-08-29T13:12:50Z

/run-all-tests

alivxxx · 2018-09-05T09:30:59Z

executor/builder.go

@@ -420,8 +421,9 @@ func (b *executorBuilder) buildLimit(v *plan.PhysicalLimit) Executor {
 		b.err = errors.Trace(b.err)
 		return nil
 	}
+	n := int(mathutil.MinUint64(v.Count, math.MaxInt64))


Why math.MaxInt64?

zz-jason · 2018-09-10T03:33:32Z

distsql/distsql_test.go

@@ -69,7 +69,7 @@ func (s *testSuite) TestSelectNormal(c *C) {
 	response.Fetch(context.TODO())

 	// Test Next.
-	chk := chunk.NewChunkWithCapacity(colTypes, 32)
+	chk := chunk.New(colTypes, 32, 32*3)


zz-jason · 2018-09-10T03:43:43Z

executor/executor.go

@@ -102,7 +103,7 @@ func (e *baseExecutor) Schema() *expression.Schema {

 // newChunk creates a new chunk to buffer current executor's result.
 func (e *baseExecutor) newChunk() *chunk.Chunk {


It seems that after this patch, newChunk should only be called once in the parent operator?

how about renaming it?

@lysu how about renaming it to newFirstChk()

zz-jason · 2018-09-10T03:45:40Z

executor/executor.go

@@ -133,6 +135,16 @@ func newBaseExecutor(ctx sessionctx.Context, schema *expression.Schema, id strin
 	return e
 }

+func (e baseExecutor) withInitCap(initCap int) (ret baseExecutor) {


how about:

s/withInitCap/setInitCap/

don't return baseExecutor, because this object takes a lot of memory

withXX and return is common chain builder pattern, I test that go compiler can inline but still do return alloc and copy as you said , so we back to ugly way 🤣

zz-jason · 2018-09-10T04:27:48Z

util/chunk/chunk.go

@@ -38,6 +38,7 @@ type Chunk struct {
 // Capacity constants.
 const (
 	InitialCapacity = 32
+	NoDataChunkCap  = -1


how about:

ZeroCapacity = 0

0 is ok, but it will still alloc columns and field num of column.. use magic -1 we can alloc less.

considered there is a InitialCapacity, I think ZeroCapacity is more suitable.

zz-jason · 2018-09-10T04:28:39Z

@XuHuaiyu PTAL

alivxxx · 2018-09-10T06:31:49Z

session/session.go

@@ -553,7 +553,7 @@ func (s *session) ExecRestrictedSQL(sctx sessionctx.Context, sql string) ([]chun
 	)
 	// Execute all recordset, take out the first one as result.
 	for i, rs := range recordSets {
-		tmp, err := drainRecordSet(ctx, rs)
+		tmp, err := drainRecordSet(ctx, sctx, rs)


Maybe it is better to use se here because it is the session that is used to execute the sql.

zz-jason · 2018-09-10T09:24:54Z

executor/executor.go

+	e.initCap = initCap
+}
+
+func (e *baseExecutor) setMaxChunkSize(maxChunkSize int) {


can be removed?

zz-jason · 2018-09-10T09:25:52Z

executor/executor.go

@@ -133,6 +135,14 @@ func newBaseExecutor(ctx sessionctx.Context, schema *expression.Schema, id strin
 	return e
 }

+func (e *baseExecutor) setInitCap(initCap int) {


maybe these kind of setters can be removed?

the function is not exported to be used as an API.

the function logic is too simple.

coocood · 2018-09-14T06:22:09Z

executor/executor.go

@@ -121,6 +122,7 @@ func newBaseExecutor(ctx sessionctx.Context, schema *expression.Schema, id strin
 		ctx:          ctx,
 		id:           id,
 		schema:       schema,
+		initCap:      ctx.GetSessionVars().MaxChunkSize,


Should set to math.Min(chunk.InitialCapacity, ctx.GetSessionVars().MaxChunkSize)?

In this PR, we can not do that, because JOIN relate modification will in next PR which will make grow real work.

then we will make initCap be configurable and check constrait in set config stage.

tiancaiamao · 2018-09-17T09:06:41Z

privilege/privileges/cache.go

@@ -296,6 +293,10 @@ func (p *MySQLPrivilege) loadTable(sctx sessionctx.Context, sql string,
 				return errors.Trace(err)
 			}
 		}
+		// NOTE: decodeTableRow decodes data from a chunk Row, that is a shallow copy.


Please read the NOTE once again! @lysu

yes, in here, use Renew will always create a new chunk just like old logic 😆

lysu · 2018-09-25T08:53:18Z

PTAL @XuHuaiyu

lysu · 2018-09-26T12:11:22Z

fping @XuHuaiyu

tiancaiamao · 2018-09-26T12:46:52Z

LGTM

zz-jason · 2018-09-26T13:25:28Z

util/chunk/chunk.go

@@ -38,6 +38,7 @@ type Chunk struct {
 // Capacity constants.
 const (
 	InitialCapacity = 32
+	ZeroCapacity    = -1


ZeroCapacity = 0?

zz-jason

LGTM

zz-jason · 2018-09-26T13:34:39Z

/run-all-tests

lysu · 2018-09-26T13:34:49Z

/run-all-tests

lysu added type/enhancement The issue or PR belongs to an enhancement. status/WIP sig/execution SIG execution labels Aug 29, 2018

lysu removed the status/WIP label Sep 3, 2018

alivxxx reviewed Sep 5, 2018

View reviewed changes

zz-jason reviewed Sep 10, 2018

View reviewed changes

alivxxx reviewed Sep 10, 2018

View reviewed changes

zz-jason reviewed Sep 10, 2018

View reviewed changes

coocood reviewed Sep 14, 2018

View reviewed changes

tiancaiamao reviewed Sep 17, 2018

View reviewed changes

lysu added 10 commits September 25, 2018 16:38

executor,chunk: add initCap to executor and grow chunk list

f7b778e

*: modify "simple" executor first.

2e3d555

*: use grow then reset.

bd6e630

*: use maxChunkSize in sessionVars as possible

2e99831

rebase and fix PhySort no-longer Count'

6eb8058

address comment

4a076a9

address comment

ea5f4ec

address comment

f7a85c5

address comment

cf3502c

fix rebase problem

4efa618

lysu assigned XuHuaiyu Sep 25, 2018

tiancaiamao added the status/LGT1 Indicates that a PR has LGTM 1. label Sep 26, 2018

address comment

03029d2

zz-jason reviewed Sep 26, 2018

View reviewed changes

address comment

bbb107f

zz-jason approved these changes Sep 26, 2018

View reviewed changes

Merge branch 'master' into dev-simple-executor-chunk

5d5b349

zz-jason added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Sep 26, 2018

zz-jason merged commit 05b37de into pingcap:master Sep 27, 2018

zz-jason added the status/all tests passed label Sep 27, 2018

lysu deleted the dev-simple-executor-chunk branch September 27, 2018 03:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

*: use chunk grow for simple executor #7540

*: use chunk grow for simple executor #7540

lysu commented Aug 29, 2018 •

edited by shenli

Loading

lysu commented Aug 29, 2018

alivxxx Sep 5, 2018

zz-jason Sep 10, 2018

zz-jason Sep 10, 2018

lysu Sep 10, 2018

zz-jason Sep 10, 2018

zz-jason Sep 26, 2018

zz-jason Sep 10, 2018

lysu Sep 10, 2018

zz-jason Sep 10, 2018

lysu Sep 10, 2018

zz-jason Sep 26, 2018

zz-jason commented Sep 10, 2018

alivxxx Sep 10, 2018

zz-jason Sep 10, 2018

zz-jason Sep 10, 2018

coocood Sep 14, 2018

lysu Sep 25, 2018 •

edited

Loading

tiancaiamao Sep 17, 2018

lysu Sep 17, 2018

tiancaiamao Sep 21, 2018

lysu commented Sep 25, 2018

lysu commented Sep 26, 2018

tiancaiamao commented Sep 26, 2018

zz-jason Sep 26, 2018

zz-jason left a comment

zz-jason commented Sep 26, 2018

lysu commented Sep 26, 2018

		@@ -102,7 +103,7 @@ func (e baseExecutor) Schema() expression.Schema {

		// newChunk creates a new chunk to buffer current executor's result.
		func (e baseExecutor) newChunk() chunk.Chunk {

*: use chunk grow for simple executor #7540

*: use chunk grow for simple executor #7540

Conversation

lysu commented Aug 29, 2018 • edited by shenli Loading

What problem does this PR solve?

What is changed and how it works?

Check List

lysu commented Aug 29, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zz-jason commented Sep 10, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lysu Sep 25, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lysu commented Sep 25, 2018

lysu commented Sep 26, 2018

tiancaiamao commented Sep 26, 2018

Choose a reason for hiding this comment

zz-jason left a comment

Choose a reason for hiding this comment

zz-jason commented Sep 26, 2018

lysu commented Sep 26, 2018

lysu commented Aug 29, 2018 •

edited by shenli

Loading

lysu Sep 25, 2018 •

edited

Loading