Allow reordering of joins in Planner::planRegularMatch
#4625
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
royi-luo
force-pushed
the
royi/more-plan-improvements
branch
from
085efd6 to
0ed3fae
Compare
royi-luo
commented
Dec 12, 2024
| @@ -1,5 +1,5 @@ | |||
| -DATASET CSV lsqb-sf01 | |||
| -BUFFER_POOL_SIZE 1073741824 | |||
| -BUFFER_POOL_SIZE 4294967296 | |||
There was a problem hiding this comment.
For q3 we overestimate the cardinality of the cycle query MATCH (person1)-[:Person_knows_Person]-(person2)-[:Person_knows_Person]-(person3)-[:Person_knows_Person]-(person1) so when we join it with the remaining part of the query we reorder the cycle query to be on the probe side. So q3 runs quite a bit slower and goes over the old buffer pool limit.
There was a problem hiding this comment.
The plan (the problematic join is the outer/leftmost one):
┌────────────────────────────┐
│┌──────────────────────────┐│
││ Logical Plan ││
│└──────────────────────────┘│
└────────────────────────────┘
┌────────────────────────────┐
│ PROJECTION │
│ ---------------------- │
│ ---------------------- │
│ Cardinality: 1 │
└─────────────┬──────────────┘
┌─────────────┴──────────────┐
│ AGGREGATE │
│ ---------------------- │
│ ---------------------- │
│ Cardinality: 1 │
└─────────────┬──────────────┘
┌─────────────┴──────────────┐
│ PROJECTION │
│ ---------------------- │
│ ---------------------- │
│ Cardinality: 119 │
└─────────────┬──────────────┘
┌─────────────┴──────────────┐
│ HASH_JOIN │
│ ---------------------- │
│ ---------------------- │──────────────────────────────────────────────┐
│ Cardinality: 119 │ │
└─────────────┬──────────────┘ │
┌─────────────┴──────────────┐ ┌─────────────┴──────────────┐
│ HASH_JOIN │ │ FLATTEN │
│ ---------------------- │ │ ---------------------- │
│ ---------------------- │───────────────┐ │ ---------------------- │
│ Cardinality: 1473696 │ │ │ Cardinality: 398749 │
└─────────────┬──────────────┘ │ └─────────────┬──────────────┘
┌─────────────┴──────────────┐ ┌─────────────┴──────────────┐ ┌─────────────┴──────────────┐
│ FLATTEN │ │ PROJECTION │ │ PROJECTION │
│ ---------------------- │ │ ---------------------- │ │ ---------------------- │
│ ---------------------- │ │ ---------------------- │ │ ---------------------- │
│ Cardinality: 22015143 │ │ Cardinality: 18135 │ │ Cardinality: 398749 │
└─────────────┬──────────────┘ └─────────────┬──────────────┘ └─────────────┬──────────────┘
┌─────────────┴──────────────┐ ┌─────────────┴──────────────┐ ┌─────────────┴──────────────┐
│ FLATTEN │ │ EXTEND │ │ HASH_JOIN │
│ ---------------------- │ │ ---------------------- │ │ ---------------------- │
│ ---------------------- │ │ (person1)-[]-(person2) │ │ ---------------------- │─────────────────────────────── ──────────────────────────────────────────────┐
│ Cardinality: 2063730 │ │ ---------------------- │ │ Cardinality: 398749 │ │
│ │ │ Cardinality: 18135 │ │ │ │
└─────────────┬──────────────┘ └─────────────┬──────────────┘ └─────────────┬──────────────┘ │
┌─────────────┴──────────────┐ ┌─────────────┴──────────────┐ ┌─────────────┴──────────────┐ ┌─────────────┴──────────────┐
│ EXTEND │ │ SCAN_NODE_TABLE │ │ HASH_JOIN │ │ PROJECTION │
│ ---------------------- │ │ ---------------------- │ │ ---------------------- │ │ ---------------------- │
│ (person3)-[]-(person1) │ │ Tables: person1._ID │ │ ---------------------- │ │ ---------------------- │
│ ---------------------- │ │ Properties : │ │ Cardinality: 26036 │──────────────────────────────────────────────┬─────────────── │ Cardinality: 1700 │
│ Cardinality: 193457 │ │ ---------------------- │ │ │ │ │ │
│ │ │ Cardinality: 1700 │ │ │ │ │ │
└─────────────┬──────────────┘ └────────────────────────────┘ └─────────────┬──────────────┘ │ └─────────────┬──────────────┘
┌─────────────┴──────────────┐ ┌─────────────┴──────────────┐ ┌─────────────┴──────────────┐ ┌─────────────┴──────────────┐
│ EXTEND │ │ FLATTEN │ │ PROJECTION │ │ HASH_JOIN │
│ ---------------------- │ │ ---------------------- │ │ ---------------------- │ │ ---------------------- │
│ (person3)-[]-(person2) │ │ ---------------------- │ │ ---------------------- │ │ ---------------------- │───────────────┐
│ ---------------------- │ │ Cardinality: 1700 │ │ Cardinality: 1700 │ │ Cardinality: 1700 │ │
│ Cardinality: 18135 │ │ │ │ │ │ │ │
└─────────────┬──────────────┘ └─────────────┬──────────────┘ └─────────────┬──────────────┘ └─────────────┬──────────────┘ │
┌─────────────┴──────────────┐ ┌─────────────┴──────────────┐ ┌─────────────┴──────────────┐ ┌─────────────┴──────────────┐ ┌─────────────┴──────────────┐
│ SCAN_NODE_TABLE │ │ HASH_JOIN │ │ HASH_JOIN │ │ FLATTEN │ │ EXTEND │
│ ---------------------- │ │ ---------------------- │ │ ---------------------- │ │ ---------------------- │ │ ---------------------- │
│ Tables: person3._ID │ │ ---------------------- │ │ ---------------------- │ │ ---------------------- │ │ (city3)-[]->(country) │
│ Properties : │ │ Cardinality: 1700 │───────────────┐ │ Cardinality: 1700 │───────────────┐ │ Cardinality: 1700 │ │ ---------------------- │
│ ---------------------- │ │ │ │ │ │ │ │ │ │ Cardinality: 1343 │
│ Cardinality: 1700 │ │ │ │ │ │ │ │ │ │ │
└────────────────────────────┘ └─────────────┬──────────────┘ │ └─────────────┬──────────────┘ │ └─────────────┬──────────────┘ └─────────────┬──────────────┘
┌─────────────┴──────────────┐ ┌─────────────┴──────────────┐ ┌─────────────┴──────────────┐ ┌─────────────┴──────────────┐ ┌─────────────┴──────────────┐ ┌─────────────┴──────────────┐
│ FLATTEN │ │ EXTEND │ │ FLATTEN │ │ EXTEND │ │ EXTEND │ │ SCAN_NODE_TABLE │
│ ---------------------- │ │ ---------------------- │ │ ---------------------- │ │ ---------------------- │ │ ---------------------- │ │ ---------------------- │
│ ---------------------- │ │ (city1)-[]->(country) │ │ ---------------------- │ │ (city2)-[]->(country) │ │ (person3)-[]->(city3) │ │ Tables: city3._ID │
│ Cardinality: 1700 │ │ ---------------------- │ │ Cardinality: 1700 │ │ ---------------------- │ │ ---------------------- │ │ Properties : │
│ │ │ Cardinality: 1343 │ │ │ │ Cardinality: 1343 │ │ Cardinality: 1700 │ │ ---------------------- │
│ │ │ │ │ │ │ │ │ │ │ Cardinality: 1343 │
└─────────────┬──────────────┘ └─────────────┬──────────────┘ └─────────────┬──────────────┘ └─────────────┬──────────────┘ └─────────────┬──────────────┘ └────────────────────────────┘
┌─────────────┴──────────────┐ ┌─────────────┴──────────────┐ ┌─────────────┴──────────────┐ ┌─────────────┴──────────────┐ ┌─────────────┴──────────────┐
│ EXTEND │ │ SCAN_NODE_TABLE │ │ EXTEND │ │ SCAN_NODE_TABLE │ │ SCAN_NODE_TABLE │
│ ---------------------- │ │ ---------------------- │ │ ---------------------- │ │ ---------------------- │ │ ---------------------- │
│ (person1)-[]->(city1) │ │ Tables: city1._ID │ │ (person2)-[]->(city2) │ │ Tables: city2._ID │ │ Tables: person3._ID │
│ ---------------------- │ │ Properties : │ │ ---------------------- │ │ Properties : │ │ Properties : │
│ Cardinality: 1700 │ │ ---------------------- │ │ Cardinality: 1700 │ │ ---------------------- │ │ ---------------------- │
│ │ │ Cardinality: 1343 │ │ │ │ Cardinality: 1343 │ │ Cardinality: 1700 │
└─────────────┬──────────────┘ └────────────────────────────┘ └─────────────┬──────────────┘ └────────────────────────────┘ └────────────────────────────┘
┌─────────────┴──────────────┐ ┌─────────────┴──────────────┐
│ SCAN_NODE_TABLE │ │ SCAN_NODE_TABLE │
│ ---------------------- │ │ ---------------------- │
│ Tables: person1._ID │ │ Tables: person2._ID │
│ Properties : │ │ Properties : │
│ ---------------------- │ │ ---------------------- │
│ Cardinality: 1700 │ │ Cardinality: 1700 │
└────────────────────────────┘ └────────────────────────────┘
Temp disable reorder for hash join Re-enable hash-join reorder Undo test change Correctly populate schema cardinality multiplier in cardinality multiplier Temp raise buffer pool size for lsqb test
royi-luo
force-pushed
the
royi/more-plan-improvements
branch
from
0ed3fae to
a2cbcbe
Compare
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #4625 +/- ##
==========================================
+ Coverage 87.46% 87.47% +0.01%
==========================================
Files 1369 1371 +2
Lines 57888 57951 +63
Branches 7206 7211 +5
==========================================
+ Hits 50632 50693 +61
- Misses 7088 7090 +2
Partials 168 168 ☔ View full report in Codecov by Sentry. |
Benchmark ResultMaster commit hash:
|
Benchmark ResultMaster commit hash:
|
Benchmark ResultMaster commit hash:
|
Benchmark ResultMaster commit hash:
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
For multi-part queries in cases where there is no dependency between the inner and outer plans we can choose to reorder the join if it produces a plan with lower cost.
Problems
I'm running into an potential bug for test
e2e_test_agg~multi_query_part.AggMultiQueryPartsubcaseGroupByMultiQueryTest4:To reproduce this make sure that the comparison is
planRegularMatchJoinOrder()isestimateJoinCostFunc(leftPlan, rightPlan) < estimateJoinCostFunc(rightPlan, leftPlan)(with<not<=as the comparison operator).On the new branch the output is only producing a single tuple
2021-06-30|2|1instead of the three expected tuples.Update: the test passes with plan optimizations disabled (more specifically with the unnecessary join remover disabled). It also passes if I force flattens on the probe-side of hash joins (so I suspect the problem has something to do with flattening).
Logical Plan (unnecessary join remover disabled)
Logical Plan (all optimizations enabled)
Contributor agreement