[Feat](nereids) support pull up predicate from set operator #39450
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
|
run buildall |
TPC-H: Total hot run time: 38093 ms |
TPC-DS: Total hot run time: 189870 ms |
ClickBench: Total hot run time: 31.37 s |
feiniaofeiafei
force-pushed
the
pull_up_from_set_op
branch
from
62de7b3 to
d2e8ab6
Compare
|
run buildall |
TPC-H: Total hot run time: 38116 ms |
TPC-DS: Total hot run time: 195770 ms |
ClickBench: Total hot run time: 31.75 s |
xzj7019
reviewed
Aug 16, 2024
| @@ -422,7 +422,10 @@ public class Rewriter extends AbstractBatchJobExecutor { | |||
| topic("eliminate", | |||
| // SORT_PRUNING should be applied after mergeLimit | |||
| custom(RuleType.ELIMINATE_SORT, EliminateSort::new), | |||
| bottomUp(new EliminateEmptyRelation()) | |||
| bottomUp(new EliminateEmptyRelation()), | |||
| topDown(new PushProjectIntoUnion()), | |||
|
run buildall |
TPC-H: Total hot run time: 38044 ms |
|
run buildall |
TPC-H: Total hot run time: 38182 ms |
TPC-DS: Total hot run time: 196994 ms |
ClickBench: Total hot run time: 31.05 s |
|
run buildall |
TPC-H: Total hot run time: 38062 ms |
TPC-DS: Total hot run time: 195919 ms |
ClickBench: Total hot run time: 31.17 s |
|
run p0 |
|
run buildall |
TPC-H: Total hot run time: 38427 ms |
TPC-DS: Total hot run time: 190409 ms |
ClickBench: Total hot run time: 32.2 s |
|
run p0 |
|
run cloud_p1 |
|
run buildall |
TPC-H: Total hot run time: 38629 ms |
morrySnow
force-pushed
the
pull_up_from_set_op
branch
from
f0f82a8 to
13b6815
Compare
|
run buildall |
TPC-H: Total hot run time: 38189 ms |
TPC-DS: Total hot run time: 187839 ms |
ClickBench: Total hot run time: 32.74 s |
|
run p0 |
|
run buildall |
TPC-H: Total hot run time: 38127 ms |
TPC-DS: Total hot run time: 193481 ms |
ClickBench: Total hot run time: 31.92 s |
|
run cloud_p0 |
morrySnow
approved these changes
Sep 5, 2024
feiniaofeiafei
added a commit
to feiniaofeiafei/doris
that referenced
this pull request
Oct 15, 2024
…9450) This pr complete the support scope of PullUpPredicate: select a,b from test1 where a<1 union all select a,b from test2 where a<1; pull up filter a<1 select a,b from test1 where a<1 intersect all select a,b from test2 where b<1; pull up filter a<1 and b<1 select a,b from test1 where a<1 except all select a,b from test2 where b<1; pull up filter a<1 select 1 a, 'bbb' b union (all) select 2,'aa'; pull up filter a in(1,2) and b in ('bbb','aa') And support infer predicate for except and intersect: select a,b from test1 where a<1 except all select a,b from test2 where b<1; -> select a,b from test1 where a<1 except all select a,b from test2 where b<1 and a<1; select a,b from test1 where a<1 intersect all select a,b from test2 where b<1; -> select a,b from test1 where a<1 and b < 1 intersect all select a,b from test2 where b<1 and a < 1;
924060929
pushed a commit
that referenced
this pull request
Oct 24, 2024
…41908) This pr complete the support scope of PullUpPredicate: select a,b from test1 where a<1 union all select a,b from test2 where a<1; pull up filter a<1 select a,b from test1 where a<1 intersect all select a,b from test2 where b<1; pull up filter a<1 and b<1 select a,b from test1 where a<1 except all select a,b from test2 where b<1; pull up filter a<1 select 1 a, 'bbb' b union (all) select 2,'aa'; pull up filter a in(1,2) and b in ('bbb','aa') And support infer predicate for except and intersect: select a,b from test1 where a<1 except all select a,b from test2 where b<1; -> select a,b from test1 where a<1 except all select a,b from test2 where b<1 and a<1; select a,b from test1 where a<1 intersect all select a,b from test2 where b<1; -> select a,b from test1 where a<1 and b < 1 intersect all select a,b from test2 where b<1 and a < 1;
924060929
pushed a commit
that referenced
this pull request
Oct 24, 2024
…on all (#41613) (#41909) introduce by #31811 and #39450 ```sql select count(1) from(select 3, 6 union all select 1, 3) t ``` wrong LogicalUnion plan: ```sql LogicalUnion( qualifier=ALL, outputs=[3#6], regularChildrenOutputs=[], constantExprsList=[[], []], hasPushedFilter=false ``` this sql will report error in explain, because the logical union outputs has a slot, but the logical union has no child and has a empty constantExprList, which is wrong set in column prune. this pr fixes it by consider when require columns is empty and keep the min slot and min slot corresponding const expressions.
feiniaofeiafei
added a commit
to feiniaofeiafei/doris
that referenced
this pull request
Nov 15, 2024
…9450) This pr complete the support scope of PullUpPredicate: select a,b from test1 where a<1 union all select a,b from test2 where a<1; pull up filter a<1 select a,b from test1 where a<1 intersect all select a,b from test2 where b<1; pull up filter a<1 and b<1 select a,b from test1 where a<1 except all select a,b from test2 where b<1; pull up filter a<1 select 1 a, 'bbb' b union (all) select 2,'aa'; pull up filter a in(1,2) and b in ('bbb','aa') And support infer predicate for except and intersect: select a,b from test1 where a<1 except all select a,b from test2 where b<1; -> select a,b from test1 where a<1 except all select a,b from test2 where b<1 and a<1; select a,b from test1 where a<1 intersect all select a,b from test2 where b<1; -> select a,b from test1 where a<1 and b < 1 intersect all select a,b from test2 where b<1 and a < 1;
feiniaofeiafei
added a commit
to feiniaofeiafei/doris
that referenced
this pull request
Nov 21, 2024
…9450) This pr complete the support scope of PullUpPredicate: select a,b from test1 where a<1 union all select a,b from test2 where a<1; pull up filter a<1 select a,b from test1 where a<1 intersect all select a,b from test2 where b<1; pull up filter a<1 and b<1 select a,b from test1 where a<1 except all select a,b from test2 where b<1; pull up filter a<1 select 1 a, 'bbb' b union (all) select 2,'aa'; pull up filter a in(1,2) and b in ('bbb','aa') And support infer predicate for except and intersect: select a,b from test1 where a<1 except all select a,b from test2 where b<1; -> select a,b from test1 where a<1 except all select a,b from test2 where b<1 and a<1; select a,b from test1 where a<1 intersect all select a,b from test2 where b<1; -> select a,b from test1 where a<1 and b < 1 intersect all select a,b from test2 where b<1 and a < 1;
feiniaofeiafei
added a commit
to feiniaofeiafei/doris
that referenced
this pull request
Nov 21, 2024
…9450) This pr complete the support scope of PullUpPredicate: select a,b from test1 where a<1 union all select a,b from test2 where a<1; pull up filter a<1 select a,b from test1 where a<1 intersect all select a,b from test2 where b<1; pull up filter a<1 and b<1 select a,b from test1 where a<1 except all select a,b from test2 where b<1; pull up filter a<1 select 1 a, 'bbb' b union (all) select 2,'aa'; pull up filter a in(1,2) and b in ('bbb','aa') And support infer predicate for except and intersect: select a,b from test1 where a<1 except all select a,b from test2 where b<1; -> select a,b from test1 where a<1 except all select a,b from test2 where b<1 and a<1; select a,b from test1 where a<1 intersect all select a,b from test2 where b<1; -> select a,b from test1 where a<1 and b < 1 intersect all select a,b from test2 where b<1 and a < 1;
feiniaofeiafei
added a commit
to feiniaofeiafei/doris
that referenced
this pull request
Nov 21, 2024
…9450) This pr complete the support scope of PullUpPredicate: select a,b from test1 where a<1 union all select a,b from test2 where a<1; pull up filter a<1 select a,b from test1 where a<1 intersect all select a,b from test2 where b<1; pull up filter a<1 and b<1 select a,b from test1 where a<1 except all select a,b from test2 where b<1; pull up filter a<1 select 1 a, 'bbb' b union (all) select 2,'aa'; pull up filter a in(1,2) and b in ('bbb','aa') And support infer predicate for except and intersect: select a,b from test1 where a<1 except all select a,b from test2 where b<1; -> select a,b from test1 where a<1 except all select a,b from test2 where b<1 and a<1; select a,b from test1 where a<1 intersect all select a,b from test2 where b<1; -> select a,b from test1 where a<1 and b < 1 intersect all select a,b from test2 where b<1 and a < 1;
morrySnow
pushed a commit
that referenced
this pull request
Dec 3, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pr complete the support scope of PullUpPredicate:
select a,b from test1 where a<1 union all select a,b from test2 where a<1; pull up filter a<1
select a,b from test1 where a<1 intersect all select a,b from test2 where b<1; pull up filter a<1 and b<1
select a,b from test1 where a<1 except all select a,b from test2 where b<1; pull up filter a<1
select 1 a, 'bbb' b union (all) select 2,'aa'; pull up filter a in(1,2) and b in ('bbb','aa')
And support infer predicate for except and intersect:
select a,b from test1 where a<1 except all select a,b from test2 where b<1;
->
select a,b from test1 where a<1 except all select a,b from test2 where b<1 and a<1;
select a,b from test1 where a<1 intersect all select a,b from test2 where b<1;
->
select a,b from test1 where a<1 and b < 1 intersect all select a,b from test2 where b<1 and a < 1;