Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

expression: constraint propagate for '>' and monotonous function #8640

Merged
merged 12 commits into from
Jan 4, 2019
Merged

expression: constraint propagate for '>' and monotonous function #8640

merged 12 commits into from
Jan 4, 2019

Conversation

tiancaiamao
Copy link
Contributor

@tiancaiamao tiancaiamao commented Dec 10, 2018 •
edited
Loading

What problem does this PR solve?

Propagate some thing like:

TO_DAYS(col) < TO_DAYS('2001-01-01') && col > '2009-04-03'  => false

What is changed and how it works?

Add a propagate rule ruleColumnGTConst leverage this PR #7643

Some simple rules:

	col > c1, col > c2, c1 > c2 => col > c1
        col > c1, col < c2, c1 >= c2 => false

A more complicate rule:

	col > c1, f(col) < c2, f is monotonous, f(c1) <= c2 => false

Prove:

        col > c1, f is monotonous => f(col) > f(c1)
        f(col) > f(c1), f(col) < c2, f(c1) >= c2 => false

Check List

Tests

  • Unit test

Related changes

Ref #7516

This change is Reviewable

@tiancaiamao
Copy link
Contributor Author

PTAL @eurekaka @zz-jason @winoros

winoros
winoros reviewed Dec 11, 2018
View reviewed changes
expression/constraint_propagation.go Outdated Show resolved Hide resolved
expression/constraint_propagation.go Outdated Show resolved Hide resolved
expression/constraint_propagation.go Outdated Show resolved Hide resolved
expression/constraint_propagation.go Outdated Show resolved Hide resolved
expression/constraint_propagation.go Show resolved Hide resolved
@tiancaiamao
Copy link
Contributor Author

PTAL @winoros

@tiancaiamao tiancaiamao mentioned this pull request Dec 29, 2018
Merged
@tiancaiamao
Copy link
Contributor Author

PTAL @winoros

@tiancaiamao
Copy link
Contributor Author

Tiny update. I remove this rule, it's not useful for partition pruning and adds code complexity

col > c1, col > c2, c1 > c2 => col > c1

PTAL @winoros

winoros
winoros reviewed Jan 2, 2019
View reviewed changes
Copy link
Member

@winoros winoros left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@winoros winoros added the status/LGT1 Indicates that a PR has LGTM 1. label Jan 2, 2019
@tiancaiamao
Copy link
Contributor Author

PTAL @eurekaka @zz-jason

@tiancaiamao
Copy link
Contributor Author

PTAL @eurekaka @zz-jason

eurekaka
eurekaka reviewed Jan 3, 2019
View reviewed changes
expression/constraint_propagation.go
con1, ok = f1.GetArgs()[1].(*Constant)
if !ok {
return
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't consider const op col?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are only two rules that really useful to partition pruning:

  1. col < const
  2. col >= const

Why?

create table ... partition by col 
p1 values less than const1
p2 values less than const2
...

Let's take a look at p2, the filter expressions are:

col >= const1 and col < const2

The final conditions would be:

col >= const1 and col < const2 and query

So the problem become that whether we can propagate:

col >= const1 and query => false
col < const2 and query => false

ruleColumnOPConst means use col op const as rule, perfectly cover the above two.

You'll find the PR can't handle the const op col.
It doesn't matter, we can add another rule that transform
const op col to col op const

Then you'll find the rule order matters.

Not a serious problem, we can control the order by
newConstraintSolver(ruleTransformConstOPColumn, ruleColumnOPConst)

I'll make pruning for select ... where to_days(c) > '2018-01-04' work, and improve some cases later.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll make pruning for select ... where to_days(c) > '2018-01-04' work, and improve some cases later.

Thant would be great.

zz-jason
zz-jason reviewed Jan 4, 2019
View reviewed changes
expression/constant_test.go
@@ -32,12 +32,16 @@ var _ = Suite(&testExpressionSuite{})
type testExpressionSuite struct{}

func newColumn(id int) *Column {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how about:
s/newColumn/newInt64Column/
s/newColumnWithType/newColumn/

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are so many newColumn in this file https://github.com/pingcap/tidb/blob/3cf386b5ca2300a0e0cebeae64d58eac9fcbff6c/expression/constant_test.go
If the function signature changes, it bring about ~50 (lines) irrelevant changes to this PR.

And this is _test.go, in this file newColumn indicates newInt64Column

@zz-jason zz-jason requested a review from alivxxx January 4, 2019 04:58
eurekaka
eurekaka reviewed Jan 4, 2019
View reviewed changes
Copy link
Contributor

@eurekaka eurekaka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@eurekaka eurekaka added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Jan 4, 2019
@eurekaka
Copy link
Contributor

eurekaka commented Jan 4, 2019

/run-all-tests

@codecov-io
Copy link

codecov-io commented Jan 4, 2019 •
edited
Loading

Codecov Report

❗ No coverage uploaded for pull request base (master@e0b30fe). Click here to learn what that means.
The diff coverage is 62.22%.

Impacted file tree graph

@@            Coverage Diff            @@
##             master    #8640   +/-   ##
=========================================
  Coverage          ?   67.52%           
=========================================
  Files             ?      363           
  Lines             ?    75073           
  Branches          ?        0           
=========================================
  Hits              ?    50693           
  Misses            ?    19905           
  Partials          ?     4475
Impacted Files Coverage Δ
expression/constraint_propagation.go 74.67% <62.22%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e0b30fe...c2e6940. Read the comment docs.

@eurekaka eurekaka added type/enhancement The issue or PR belongs to an enhancement. status/all tests passed labels Jan 4, 2019
@tiancaiamao tiancaiamao merged commit 3708a83 into pingcap:master Jan 4, 2019
@tiancaiamao tiancaiamao deleted the propagate-pruning branch January 4, 2019 06:06
zimulala added a commit that referenced this pull request Jan 4, 2019
@zimulala
Revert "expression: constraint propagate for '>' and monotonous funct…
9b30cde 
…ion (#8640)"

This reverts commit 3708a83.
@tiancaiamao tiancaiamao mentioned this pull request Feb 7, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@winoros winoros winoros left review comments

@zz-jason zz-jason zz-jason left review comments

@eurekaka eurekaka eurekaka approved these changes

@alivxxx alivxxx Awaiting requested review from alivxxx

Assignees
No one assigned
Labels
component/expression status/LGT2 Indicates that a PR has LGTM 2. type/enhancement The issue or PR belongs to an enhancement.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@tiancaiamao @eurekaka @codecov-io @zz-jason @winoros