plan: push down constant filters over join properly #9848

eurekaka · 2019-03-21T07:56:05Z

What problem does this PR solve?

Fix #9843

What is changed and how it works?

For filter expression which does not contain columns:

if the condition is filter condition, push it down to both children of join, whatever the join type is;
if the condition is join condition, push it down to inner child of join if the join is outer join, or both children if the join is not outer join;

Check List

Tests

Unit test
Integration test

Code changes

N/A

Side effects

N/A

Related changes

Need to cherry-pick to the release branch

ngaut · 2019-03-22T02:17:12Z

/run-all-tests

codecov · 2019-03-25T10:14:23Z

Codecov Report

❗ No coverage uploaded for pull request base (master@3faafb2). Click here to learn what that means.
The diff coverage is 61.9047%.

@@             Coverage Diff             @@
##             master      #9848   +/-   ##
===========================================
  Coverage          ?   77.2275%           
===========================================
  Files             ?        405           
  Lines             ?      81682           
  Branches          ?          0           
===========================================
  Hits              ?      63081           
  Misses            ?      13926           
  Partials          ?       4675

eurekaka · 2019-03-25T10:15:49Z

/run-all-tests

eurekaka · 2019-03-25T13:25:32Z

/run-all-tests

eurekaka · 2019-03-25T13:34:54Z

/rebuild

eurekaka · 2019-03-25T13:39:11Z

/run-all-tests

eurekaka · 2019-03-27T03:11:50Z

cmd/explaintest/r/explain_easy.result

+      │ └─Selection_40	2.40	cop	eq(3, test.t.a)
+      │   └─IndexScan_39	3.00	cop	table:s, index:b, range:[3,3], keep order:false
+      └─TableReader_48	4.00	root	data:Selection_47
+        └─Selection_47	4.00	cop	eq(3, test.t.a)


For reviewers: the row count of Selection_47 and TableReader_48 is wrong, but this should be a separate issue and I would fix it in another PR later.

eurekaka · 2019-03-27T03:13:23Z

planner/core/logical_plan_test.go

@@ -906,7 +906,7 @@ func (s *testPlanSuite) TestJoinReOrder(c *C) {
 		},
 		{
 			sql:  "select * from t o where o.b in (select t3.c from t t1, t t2, t t3 where t1.a = t3.a and t2.a = t3.a and t2.a = o.a and t1.a = 1)",
-			best: "Apply{DataScan(o)->Join{Join{DataScan(t3)->DataScan(t1)}->DataScan(t2)}->Projection}->Projection",
+			best: "Apply{DataScan(o)->Join{Join{DataScan(t1)->DataScan(t2)}->DataScan(t3)}->Projection}->Projection",


@winoros Please confirm this join order change does not effect correctness.

THe join order changed since some filter pushed before but now is not.

winoros · 2019-03-27T03:37:46Z

planner/core/logical_plan_builder.go

@@ -180,6 +180,33 @@ func (b *PlanBuilder) buildResultSetNode(node ast.ResultSetNode) (p LogicalPlan,
 	}
 }

+// pushDownConstExpr checks if the condition is from filter condition, if true, push it down to both


What does filter here mean?

To differentiate the where conditions from join conditions, I use filter condition here. I can change it to another name if it is misleading.

So it should be the conditions exclude the join key conditions? If so, can deriveLeft || deriveRight match its meaning?

extractOnCondition is called by PredicatePushDown and atttachOnConds, in PredicatePushDown, the input conditions are treated as filter conditions, and at least deriveLeft or deriveRight is true; in attachOnConds, the input conditions are join conditions, and deriveLeft and deriveRight are both false, so it should be correct?

planner/core/logical_plan_builder.go

zz-jason

LGTM

alivxxx

LGTM

eurekaka added type/bugfix This PR fixes a bug. sig/planner SIG: Planner labels Mar 21, 2019

eurekaka changed the title ~~plan: keep constant filters in OtherConditions of join~~ plan: push down constant filters over join properly Mar 25, 2019

eurekaka force-pushed the extract_on branch from 3d769b9 to a8c9abb Compare March 25, 2019 13:19

eurekaka marked this pull request as ready for review March 25, 2019 13:32

eurekaka added the status/all tests passed label Mar 27, 2019

eurekaka commented Mar 27, 2019

View reviewed changes

eurekaka requested review from winoros, zz-jason and alivxxx March 27, 2019 03:14

winoros reviewed Mar 27, 2019

View reviewed changes

alivxxx reviewed Mar 27, 2019

View reviewed changes

planner/core/logical_plan_builder.go Outdated Show resolved Hide resolved

eurekaka requested review from winoros and alivxxx March 27, 2019 09:46

alivxxx reviewed Mar 29, 2019

View reviewed changes

planner/core/logical_plan_builder.go Show resolved Hide resolved

plan: keep constant filters in OtherConditions of join

6cca2a7

zz-jason reviewed Apr 4, 2019

View reviewed changes

zz-jason added the status/LGT1 Indicates that a PR has LGTM 1. label Apr 4, 2019

alivxxx previously approved these changes Apr 4, 2019

View reviewed changes

alivxxx added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Apr 4, 2019

eurekaka dismissed alivxxx’s stale review via 6cca2a7 April 4, 2019 07:08

eurekaka force-pushed the extract_on branch from 61a6a8b to 6cca2a7 Compare April 4, 2019 07:08

Merge branch 'master' into extract_on

a65d66d

winoros approved these changes Apr 4, 2019

View reviewed changes

Merge branch 'master' into extract_on

d3f4923

eurekaka merged commit 8800ec1 into pingcap:master Apr 4, 2019

eurekaka deleted the extract_on branch April 4, 2019 07:50

eurekaka added a commit to eurekaka/tidb that referenced this pull request Apr 4, 2019

plan: push down constant filters over join properly (pingcap#9848)

b1f17d1

eurekaka mentioned this pull request Apr 4, 2019

plan: push down constant filters over join properly (#9848) #10047

Merged

eurekaka added a commit to eurekaka/tidb that referenced this pull request Apr 4, 2019

plan: push down constant filters over join properly (pingcap#9848)

33ce21e

eurekaka mentioned this pull request Apr 4, 2019

plan: push down constant filters over join properly (#9848) #10049

Merged

ngaut pushed a commit that referenced this pull request Apr 8, 2019

plan: push down constant filters over join properly (#9848) (#10047)

6641576

zz-jason pushed a commit that referenced this pull request Apr 8, 2019

plan: push down constant filters over join properly (#9848) (#10049)

0f97e0b

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

plan: push down constant filters over join properly #9848

plan: push down constant filters over join properly #9848

eurekaka commented Mar 21, 2019 •

edited

Loading

ngaut commented Mar 22, 2019

codecov bot commented Mar 25, 2019 •

edited

Loading

eurekaka commented Mar 25, 2019

eurekaka commented Mar 25, 2019

eurekaka commented Mar 25, 2019

eurekaka commented Mar 25, 2019

eurekaka Mar 27, 2019 •

edited

Loading

eurekaka Mar 27, 2019

winoros Mar 27, 2019 •

edited

Loading

winoros Mar 27, 2019

eurekaka Mar 27, 2019

winoros Mar 28, 2019

eurekaka Apr 2, 2019

zz-jason left a comment

alivxxx left a comment

plan: push down constant filters over join properly #9848

plan: push down constant filters over join properly #9848

Conversation

eurekaka commented Mar 21, 2019 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

ngaut commented Mar 22, 2019

codecov bot commented Mar 25, 2019 • edited Loading

Codecov Report

eurekaka commented Mar 25, 2019

eurekaka commented Mar 25, 2019

eurekaka commented Mar 25, 2019

eurekaka commented Mar 25, 2019

eurekaka Mar 27, 2019 • edited Loading

Choose a reason for hiding this comment

eurekaka Mar 27, 2019

Choose a reason for hiding this comment

winoros Mar 27, 2019 • edited Loading

Choose a reason for hiding this comment

winoros Mar 27, 2019

Choose a reason for hiding this comment

eurekaka Mar 27, 2019

Choose a reason for hiding this comment

winoros Mar 28, 2019

Choose a reason for hiding this comment

eurekaka Apr 2, 2019

Choose a reason for hiding this comment

zz-jason left a comment

Choose a reason for hiding this comment

alivxxx left a comment

Choose a reason for hiding this comment

eurekaka commented Mar 21, 2019 •

edited

Loading

codecov bot commented Mar 25, 2019 •

edited

Loading

eurekaka Mar 27, 2019 •

edited

Loading

winoros Mar 27, 2019 •

edited

Loading