Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

util: refine chunk.SwapColumn to rebuild the column reference #7841

Merged
merged 7 commits into from
Oct 9, 2018

Conversation

XuHuaiyu
Copy link
Contributor

@XuHuaiyu XuHuaiyu commented Oct 8, 2018

What problem does this PR solve?

If there exist some columns refer to one column in a chunk, we need to rebuild
the reference when call chunk.SwapColumn.

Take an image as an example,
2018-10-08 6 49 58

If we call chk2.Reset(), the access to chk1 may cause unexpected panic.

What is changed and how it works?

Check whether exist columns refer to the column to be swapped, if so, record
the index of these columns and rebuild the reference after swapping.

Check List

Tests

  • Unit test

To be added.

Code changes

  • Has exported function/method change

Side effects
none

Related changes

  • Need to cherry-pick to the release branch
    Need to cherry-pick to release 2.0, release 2.1.

@XuHuaiyu XuHuaiyu added type/bugfix This PR fixes a bug. sig/execution SIG execution labels Oct 8, 2018
// If there exists columns refer to the column to be swapped, we need to
// re-build the reference.
refColsIdx := make([]int, 0, len(c.columns)-colIdx)
for i := colIdx + 1; i < len(c.columns); i++ {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we also need to check the column between 0 and colIdx-1

@zz-jason
Copy link
Member

zz-jason commented Oct 8, 2018

please add some tests

@XuHuaiyu
Copy link
Contributor Author

XuHuaiyu commented Oct 8, 2018

PTAL @zz-jason @eurekaka @winoros

Copy link
Member

@winoros winoros left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

need we maintain the reference information in chunk' structure instead of building every time swapping the column?

@XuHuaiyu
Copy link
Contributor Author

XuHuaiyu commented Oct 8, 2018

@winoros
I think keeping the current chunk structure unchanged may be better,
since the check and rebuild process will not take much negative effect on the performance.

@XuHuaiyu
Copy link
Contributor Author

XuHuaiyu commented Oct 8, 2018

/run-all-tests

// If there exists columns refer to the column to be swapped, we need to
// re-build the reference.
refColsIdx := make([]int, 0, len(c.columns))
for i := colIdx; i < len(c.columns); i++ {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why don't check 0 - colIdx?
What if column[0] refer to column[1]? Or this situation doesn't happen?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@crazycs520
I updated the code.
column[i] refers to column[j] will never happen now where i < j.

chk2.SwapColumn(2, chk2, 0)
c.Assert(chk1.columns[0] == chk1.columns[1], check.IsTrue)
c.Assert(chk1.columns[0] == chk2.columns[0], check.IsFalse)
c.Assert(chk2.columns[0] == chk2.columns[1], check.IsTrue)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Delete redundant code.
How about this:

	checkRef := func () {
		c.Assert(chk1.columns[0] == chk1.columns[1], check.IsTrue)
		c.Assert(chk1.columns[0] == chk2.columns[0], check.IsFalse)
		c.Assert(chk2.columns[0] == chk2.columns[1], check.IsTrue)
	}
	checkRef();
	chk1.SwapColumn(0, chk2, 0)
	checkRef();
	chk1.SwapColumn(0, chk2, 1)
	checkRef();
	chk2.SwapColumn(1, chk2, 0)
	checkRef();
	chk2.SwapColumn(1, chk2, 1)
	checkRef();
	chk2.SwapColumn(1, chk2, 2)
	checkRef();
	chk2.SwapColumn(2, chk2, 0)
	checkRef();

@XuHuaiyu
Copy link
Contributor Author

XuHuaiyu commented Oct 9, 2018

  1. The column reference will only be built in ProjectionExec now, which will only use chunk.SwapColumn to changes the original data of the input chunk. So we need to handle this problem in SwapColumn.

  2. We do not need to concern the column reference thing in other functions now, because that the output chunk of ProjectionExec is only readable for the parent executors except for ProjectionExec.

Copy link
Contributor

@crazycs520 crazycs520 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@crazycs520 crazycs520 added the status/LGT1 Indicates that a PR has LGTM 1. label Oct 9, 2018
Copy link
Contributor

@eurekaka eurekaka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@eurekaka
Copy link
Contributor

eurekaka commented Oct 9, 2018

/run-all-tests

@eurekaka eurekaka added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Oct 9, 2018
@eurekaka
Copy link
Contributor

eurekaka commented Oct 9, 2018

/run-all-tests

@XuHuaiyu XuHuaiyu merged commit 7623899 into pingcap:master Oct 9, 2018
@XuHuaiyu XuHuaiyu deleted the projection_makeref branch October 9, 2018 06:27
XuHuaiyu added a commit to XuHuaiyu/tidb that referenced this pull request Oct 9, 2018
XuHuaiyu added a commit to XuHuaiyu/tidb that referenced this pull request Oct 9, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
sig/execution SIG execution status/LGT2 Indicates that a PR has LGTM 2. type/bugfix This PR fixes a bug.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants