Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

executor: control Chunk size for TableReader&IndexReader&IndexLookup #9452

Merged
merged 20 commits into from
Apr 1, 2019
Merged

executor: control Chunk size for TableReader&IndexReader&IndexLookup #9452

merged 20 commits into from
Apr 1, 2019

Conversation

qw4990
Copy link
Contributor

@qw4990 qw4990 commented Feb 25, 2019

What problem does this PR solve?

Control the number of rows in chunks returned by TableReader & IndexReader & IndexLookup.
Following up #9398, this PR is a subtask of #9166.

What is changed and how it works?

  1. introduce requiredRowsSelectResult to mock SelectResult.
  2. introduce selectWithRuntimeStats into TableReader & IndexReader & IndexLookup & dataReaderBuilder to create mock SelectResult.
  3. add some unit tests.

Check List

Tests

  • Unit test

@qw4990 qw4990 added the sig/execution SIG execution label Feb 25, 2019
@qw4990 qw4990 force-pushed the chunkSizeControl_TableReader/IndexReader/LookupReader branch from a227726 to 25cb1f3 Compare February 26, 2019 05:48
@qw4990 qw4990 requested a review from alivxxx February 26, 2019 05:54
@qw4990 qw4990 force-pushed the chunkSizeControl_TableReader/IndexReader/LookupReader branch 2 times, most recently from 60d43ea to f54a4f6 Compare February 27, 2019 07:58
@cwen0
Copy link
Member

cwen0 commented Feb 27, 2019

/rebuild

@qw4990 qw4990 force-pushed the chunkSizeControl_TableReader/IndexReader/LookupReader branch from f54a4f6 to 5b431e8 Compare February 27, 2019 11:46
@qw4990
Copy link
Contributor Author

qw4990 commented Feb 27, 2019

All CI problems have been solved, PTAL~

@codecov-io
Copy link

codecov-io commented Feb 27, 2019 •
edited by codecov bot
Loading

Codecov Report

Merging #9452 into master will decrease coverage by 0.0857%.
The diff coverage is 89.6551%.

@@               Coverage Diff                @@
##             master      #9452        +/-   ##
================================================
- Coverage   77.5575%   77.4718%   -0.0858%     
================================================
  Files           403        403                
  Lines         81827      81782        -45     
================================================
- Hits          63463      63358       -105     
- Misses        13663      13705        +42     
- Partials       4701       4719        +18

@qw4990 qw4990 requested a review from XuHuaiyu February 28, 2019 06:29
@qw4990 qw4990 mentioned this pull request Feb 28, 2019
Merged
lysu
lysu reviewed Mar 1, 2019
View reviewed changes
executor/distsql.go Outdated Show resolved Hide resolved
@qw4990 qw4990 force-pushed the chunkSizeControl_TableReader/IndexReader/LookupReader branch from 5b431e8 to afb7528 Compare March 4, 2019 05:59
zz-jason
zz-jason reviewed Mar 4, 2019
View reviewed changes
executor/distsql.go Outdated Show resolved Hide resolved
winoros
winoros reviewed Mar 4, 2019
View reviewed changes
executor/table_reader.go Outdated Show resolved Hide resolved
@qw4990 qw4990 force-pushed the chunkSizeControl_TableReader/IndexReader/LookupReader branch from a4835e0 to 5f253d4 Compare March 4, 2019 12:43
zz-jason
zz-jason reviewed Mar 4, 2019
View reviewed changes
executor/distsql.go Outdated Show resolved Hide resolved
@qw4990 qw4990 force-pushed the chunkSizeControl_TableReader/IndexReader/LookupReader branch from 5f253d4 to 21ec706 Compare March 5, 2019 07:04
@qw4990 qw4990 mentioned this pull request Mar 8, 2019
Merged
@qw4990
Copy link
Contributor Author

qw4990 commented Mar 11, 2019

IndexLookUp has a high level property of pipeline breaker, since all work what it does is processed asynchronously in the background, so it’s hard to control its status and determine a correct requiredRows value to its children.

How about we don't let IndexLookUp push requiredRows down to its children and just use maxChunkSize as before, like what we do in IndexLookupJoinExec and HashJoinExec(#9614), otherwise it will make this executor more complex and tricky.

@zz-jason @lysu PTAL~

@qw4990 qw4990 force-pushed the chunkSizeControl_TableReader/IndexReader/LookupReader branch from 21ec706 to ed0c0fb Compare March 27, 2019 08:30
@qw4990
Copy link
Contributor Author

qw4990 commented Mar 27, 2019

According to your comments, IndexLookupSize and the double extend batch size strategy remain.
PTAL @zz-jason @lysu

zz-jason
zz-jason reviewed Mar 27, 2019
View reviewed changes
Copy link
Member

@zz-jason zz-jason left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

executor/distsql.go Show resolved Hide resolved
lysu
lysu reviewed Mar 28, 2019
View reviewed changes
executor/distsql.go Outdated Show resolved Hide resolved
@qw4990 qw4990 force-pushed the chunkSizeControl_TableReader/IndexReader/LookupReader branch from 2cda04c to 4179218 Compare March 28, 2019 05:56
lysu
lysu reviewed Mar 29, 2019
View reviewed changes
Copy link
Contributor

@lysu lysu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rest LGTM

executor/distsql.go Show resolved Hide resolved
@zz-jason zz-jason added the status/LGT1 Indicates that a PR has LGTM 1. label Mar 30, 2019
lysu
lysu approved these changes Apr 1, 2019
View reviewed changes
Copy link
Contributor

@lysu lysu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zz-jason
Copy link
Member

zz-jason commented Apr 1, 2019

/run-all-tests

@zz-jason zz-jason added status/LGT2 Indicates that a PR has LGTM 2. type/enhancement The issue or PR belongs to an enhancement. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Apr 1, 2019
@zz-jason zz-jason merged commit 435a081 into pingcap:master Apr 1, 2019
qw4990 added a commit to qw4990/tidb that referenced this pull request Apr 16, 2019
@qw4990
executor: control Chunk size for TableReader&IndexReader&IndexLookup (p…
ba25965 
…ingcap#9452)
@qw4990 qw4990 mentioned this pull request Apr 16, 2019
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@winoros winoros winoros left review comments

@zz-jason zz-jason zz-jason left review comments

@lysu lysu lysu approved these changes

@alivxxx alivxxx Awaiting requested review from alivxxx

@XuHuaiyu XuHuaiyu Awaiting requested review from XuHuaiyu

Assignees
No one assigned
Labels
sig/execution SIG execution status/LGT2 Indicates that a PR has LGTM 2. type/enhancement The issue or PR belongs to an enhancement.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@qw4990 @cwen0 @codecov-io @zz-jason @lysu @winoros