-
Notifications
You must be signed in to change notification settings - Fork 5.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
expression: a constraint propagate framework mainly for partition pruning #7643
Conversation
What's mean by |
Such as Our current constant propagate can't infer the following, because it doesn't propagate conditions from
|
Why not put this after the current And since this algo is about CNF why not break when finding some part is false. |
@tiancaiamao The code patch is too large, could you please provide a brief introduction about how this algorithm works? Or maybe a proposal? |
1 similar comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why not put this after the current
ConstantPropagation
, so you don't need to consider propagate the in and eq conditions.
And we can make the optimization more flexible.
For example, we may extract new filters fromExtractFiltersFromDNFs
, and the new extracted filter can be evaluated by this algo. If you combine the propagating and evaluating, it's hard to placeExtractFiltersFromDNFs
to its right place.And since this algo is about CNF why not break when finding some part is false.
+1
Besides, we propagate filters over outer join now, for this situation, we need to differentiate join conditions and filter conditions, conditions from left child schema and right child schema, it looks hard to fit this into fixPoint
. I prefer using this framework for expression simplification as mentioned in PR description, e.g, a > 1 and a > 2 => a > 2
.
My concern stated above has not been addressed yet, i.e, where to use this framework? if we are going to use it to substitute current constant propagation, then the problems we mentioned above should be addressed? if this is just for partition pruning, then LGTM now and I am going to shut up. |
Sorry for the late reply, I'm far too busy recently.
In table partition pruning, of course, that's also the primary reason it's written. If you'll address the problem for non-equal condition and monotonic function using current constant propagate algorithm (or give me a promise to do it in the foreseeable future), I'd like to close this PR and wait for your solution. Otherwise, I have to do it myself. If the proposed algorithm is general enough, I'd like to try to substitute current constant propagation to make the code base simpler. If it's not or, due to time limitation, it could still works with current algorithm, they are not conflict (correct me if I am wrong). |
So LGTM.
That would be great. |
2 similar comments
@tiancaiamao If we treat |
And I prefer to change the name from |
Sounds not bad.
Then I'll push you to do that. @zz-jason |
90dc749
to
582469c
Compare
I remove the require-LGTM3 tag, which I think is not reasonable @zz-jason |
Friendly ping @zz-jason |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
/run-all-tests |
What problem does this PR solve?
For partition pruning, we will have the partition expression such as
to_days(x)
,we have the constant range like:
When the user send a query, such as
... where x > 632
The final partition pruning conditions could be:
If some conditions eval to constant false, that partition should be prune.
So, in-equal conditions and some simple function should be handled in the partition pruning process.
Ref #7516
I need a constraint propagate framework to solve that problem for table partition.
What is changed and how it works?
The comment in the code describes an algorithm in detail.
Partition pruning will use the output of current constant propagate result, and continue to propagate more constraints to see whether a partition could be pruned.
Check List
Tests
Side effects
This change is