[Fix](ScanNode) Move the finalize phase of ScanNode to after the end of the Physical Translate phase. #37565
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
|
run buildall |
1 similar comment
|
run buildall |
TPC-H: Total hot run time: 39605 ms |
TPC-DS: Total hot run time: 174419 ms |
ClickBench: Total hot run time: 30.66 s |
BePPPower
force-pushed
the
fixTrinoConnectorProject
branch
from
353870a to
d494308
Compare
|
run buildall |
TPC-H: Total hot run time: 40033 ms |
TPC-DS: Total hot run time: 173219 ms |
ClickBench: Total hot run time: 31.53 s |
|
run p0 |
BePPPower
changed the title
|
PR approved by at least one committer and no changes requested. |
github-actions
bot
added
the
approved
|
PR approved by anyone and no changes requested. |
morrySnow
reviewed
Jul 17, 2024
fe/fe-core/src/main/java/org/apache/doris/planner/OlapScanNode.java
Outdated
Show resolved
Hide resolved
github-actions
bot
removed
the
approved
|
run buildall |
TPC-H: Total hot run time: 39516 ms |
TPC-DS: Total hot run time: 174444 ms |
ClickBench: Total hot run time: 30.61 s |
morrySnow
approved these changes
Jul 19, 2024
|
PR approved by at least one committer and no changes requested. |
github-actions
bot
added
the
approved
dataroaring
pushed a commit
that referenced
this pull request
Jul 24, 2024
…of the Physical Translate phase. (#37565) ## Proposed changes Currently, Doris first obtains splits and then performs projection. After column pruning, it calls `updateRequiredSlots` to update the scanRange information. However, the Trino connector's column pruning pushdown needs to be completed before obtaining splits. Therefore, we move the finalize phase of `ScanNode` to after the end of the `Physical Translate` phase, so that `createScanRangeLocations` can use the final columns which have been pruning.
BePPPower
added a commit
to BePPPower/doris
that referenced
this pull request
Jul 31, 2024
…of the Physical Translate phase. (apache#37565) Currently, Doris first obtains splits and then performs projection. After column pruning, it calls `updateRequiredSlots` to update the scanRange information. However, the Trino connector's column pruning pushdown needs to be completed before obtaining splits. Therefore, we move the finalize phase of `ScanNode` to after the end of the `Physical Translate` phase, so that `createScanRangeLocations` can use the final columns which have been pruning.
yiguolei
pushed a commit
that referenced
this pull request
Aug 5, 2024
…of the Physical Translate phase (#38604) bp: #37565 Currently, Doris first obtains splits and then performs projection. After column pruning, it calls `updateRequiredSlots` to update the scanRange information. However, the Trino connector's column pruning pushdown needs to be completed before obtaining splits. Therefore, we move the finalize phase of `ScanNode` to after the end of the `Physical Translate` phase, so that `createScanRangeLocations` can use the final columns which have been pruning. ## Proposed changes Issue Number: close #xxx <!--Describe your changes.-->
morningman
pushed a commit
that referenced
this pull request
Aug 12, 2024
In #37565, due to the change in the calling order of finalize, the final generated Plan will be missing the PREDICATES that have been pushed down in Jdbc. Although this behavior is correct, before perfectly handling the push down of various PREDICATES, we need to keep all conjuncts to ensure that we can still filter data normally when the data returned by Jdbc is a superset.
wyxxxcat
pushed a commit
to wyxxxcat/doris
that referenced
this pull request
Aug 14, 2024
…he#39180) In apache#37565, due to the change in the calling order of finalize, the final generated Plan will be missing the PREDICATES that have been pushed down in Jdbc. Although this behavior is correct, before perfectly handling the push down of various PREDICATES, we need to keep all conjuncts to ensure that we can still filter data normally when the data returned by Jdbc is a superset.
zy-kkk
added a commit
to zy-kkk/doris
that referenced
this pull request
Aug 15, 2024
…he#39180) In apache#37565, due to the change in the calling order of finalize, the final generated Plan will be missing the PREDICATES that have been pushed down in Jdbc. Although this behavior is correct, before perfectly handling the push down of various PREDICATES, we need to keep all conjuncts to ensure that we can still filter data normally when the data returned by Jdbc is a superset.
zy-kkk
added a commit
that referenced
this pull request
Aug 16, 2024
…bcScan (#39407) pick (#39180) In #37565, due to the change in the calling order of finalize, the final generated Plan will be missing the PREDICATES that have been pushed down in Jdbc. Although this behavior is correct, before perfectly handling the push down of various PREDICATES, we need to keep all conjuncts to ensure that we can still filter data normally when the data returned by Jdbc is a superset.
dataroaring
pushed a commit
that referenced
this pull request
Aug 17, 2024
In #37565, due to the change in the calling order of finalize, the final generated Plan will be missing the PREDICATES that have been pushed down in Jdbc. Although this behavior is correct, before perfectly handling the push down of various PREDICATES, we need to keep all conjuncts to ensure that we can still filter data normally when the data returned by Jdbc is a superset.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed changes
Currently, Doris first obtains splits and then performs projection. After column pruning, it calls
updateRequiredSlotsto update the scanRange information. However, the Trino connector's column pruning pushdown needs to be completed before obtaining splits.Therefore, we move the finalize phase of
ScanNodeto after the end of thePhysical Translatephase, so thatcreateScanRangeLocationscan use the final columns which have been pruning.