Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

executor: add batch copy to inner join, left and right outer join. #7493

Merged
merged 23 commits into from
Sep 5, 2018
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
0cc4ab9
batch copy init
crazycs520 Aug 25, 2018
2592034
add comment
crazycs520 Aug 28, 2018
265dd58
add batch copy test and benchmark
crazycs520 Aug 28, 2018
39cde9c
refine code
crazycs520 Aug 28, 2018
7b19c1c
fix bugs
crazycs520 Aug 28, 2018
e312fa4
Merge branch 'master' of https://github.com/pingcap/tidb into only-ba…
crazycs520 Aug 28, 2018
4022038
Merge branch 'master' of https://github.com/pingcap/tidb into only-ba…
crazycs520 Aug 29, 2018
9b6ce0f
Merge branch 'master' of https://github.com/pingcap/tidb into only-ba…
crazycs520 Sep 2, 2018
315c1fa
refactor code and comment
crazycs520 Sep 3, 2018
37902a2
refine code
crazycs520 Sep 3, 2018
c4556b4
address comment
crazycs520 Sep 3, 2018
cefae06
refactor code and comment
crazycs520 Sep 3, 2018
9fe55b3
refactor code and comment
crazycs520 Sep 4, 2018
91366cd
address comment
crazycs520 Sep 4, 2018
d3f8e85
address comment
crazycs520 Sep 4, 2018
026be7f
Merge branch 'master' of https://github.com/pingcap/tidb into only-ba…
crazycs520 Sep 4, 2018
50a698d
address comment
crazycs520 Sep 4, 2018
28018f9
address comment
crazycs520 Sep 4, 2018
2d04676
Merge branch 'master' into only-batch-copy
zz-jason Sep 4, 2018
e47fb7c
Merge branch 'master' into only-batch-copy
zz-jason Sep 5, 2018
3208738
refine comments
crazycs520 Sep 5, 2018
cd6eda1
Merge branch 'master' of https://github.com/pingcap/tidb into only-ba…
crazycs520 Sep 5, 2018
cedcd0e
Merge branch 'only-batch-copy' of https://github.com/crazycs520/tidb …
crazycs520 Sep 5, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion executor/joiner.go
Original file line number Diff line number Diff line change
Expand Up @@ -164,7 +164,7 @@ func (j *baseJoiner) filter(input, output *chunk.Chunk, outerColsLen int) (bool,
if err != nil {
return false, errors.Trace(err)
}
// batch copy selected row to output chunk
// Batch copies selected rows to output chunk.
innerColOffset, outerColOffset := 0, input.NumCols()-outerColsLen
if !j.outerIsRight {
innerColOffset, outerColOffset = outerColsLen, 0
Expand Down
1 change: 1 addition & 0 deletions util/chunk/chunk_util.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ package chunk

// CopySelectedJoinRows copies the selected joined rows from the source Chunk
// to the destination Chunk.
// Return true if at least one joined row was selected.
//
// NOTE: All the outer rows in the source Chunk should be the same.
func CopySelectedJoinRows(src *Chunk, innerColOffset, outerColOffset int, selected []bool, dst *Chunk) bool {
Expand Down
31 changes: 17 additions & 14 deletions util/chunk/column.go
Original file line number Diff line number Diff line change
Expand Up @@ -72,40 +72,43 @@ func (c *column) isNull(rowIdx int) bool {
return nullByte&(1<<(uint(rowIdx)&7)) == 0
}

func (c *column) appendNullBitmap(on bool) {
func (c *column) appendNullBitmap(notNull bool) {
idx := c.length >> 3
if idx >= len(c.nullBitmap) {
c.nullBitmap = append(c.nullBitmap, 0)
}
if on {
if notNull {
pos := uint(c.length) & 7
c.nullBitmap[idx] |= byte(1 << pos)
} else {
c.nullCount++
}
}

func (c *column) appendMultiSameNullBitmap(on bool, num int) {
// appendMultiSameNullBitmap appends multiple same bit value to `nullBitMap`.
// notNull mean not not null.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/ mean/ means
remove redundant not

// num mean appends `num` bit value to `nullBitMap`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

num means the number of bits that should be appended.

func (c *column) appendMultiSameNullBitmap(notNull bool, num int) {
numNewBytes := ((c.length + num + 7) >> 3) - len(c.nullBitmap)
b := byte(0)
if on {
if notNull {
b = 0xff
}
for i := 0; i < numNewBytes; i++ {
c.nullBitmap = append(c.nullBitmap, b)
}
if on {
// 1. Set all the higher 8-'numOldBits' bits in the last old byte to 1.
numOldBits := uint(c.length % 8)
bitMask := byte(^((1 << numOldBits) - 1))
c.nullBitmap[c.length/8] |= bitMask
// 2. Set all the higher 'numRedundantBits' bits in the last new byte to 0.
numRedundantBits := uint(len(c.nullBitmap)*8 - c.length - num)
bitMask = byte(1<<(8-numRedundantBits)) - 1
c.nullBitmap[len(c.nullBitmap)-1] &= bitMask
} else {
if !notNull {
c.nullCount += num
Copy link
Member

@zz-jason zz-jason Sep 3, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we also need to set all the existing bits to zero even if on is set to false, because this Chunk maybe is truncated, and the null bitmap is not reset in that scenario.

return
}
// 1. Set all the higher 8-'numOldBits' bits in the last old byte to 1.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Set all the remained bits in the last slot of old c.numBitMap to 1.
  2. Set all the redundant bits in the last slot of new c.numBitMap to 0.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wait, I ask @CaitinChen

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@crazycs520 I prefer "the remaining bits"

numOldBits := uint(c.length % 8)
bitMask := byte(^((1 << numOldBits) - 1))
c.nullBitmap[c.length/8] |= bitMask
// 2. Set all the higher 'numRedundantBits' bits in the last new byte to 0.
numRedundantBits := uint(len(c.nullBitmap)*8 - c.length - num)
bitMask = byte(1<<(8-numRedundantBits)) - 1
c.nullBitmap[len(c.nullBitmap)-1] &= bitMask
}

func (c *column) appendNull() {
Expand Down