Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

planner: fix bug that aggregate push down may generate wrong plan for outer joins #34468

Merged
merged 4 commits into from
May 13, 2022
Merged

planner: fix bug that aggregate push down may generate wrong plan for outer joins #34468

merged 4 commits into from
May 13, 2022

Conversation

fixdb
Copy link
Contributor

@fixdb fixdb commented May 9, 2022

What problem does this PR solve?

Issue Number: close #34465

Problem Summary:
When we push down aggregate past outer join, the top count aggregate assumed
that the partial count aggregate always return non-null value. But in case of
outer joins when the aggregate is pushed down to the null generating side of
the outer join, the input for the final aggregate may be nullable.

Before this patch, the final count aggregate always assume its input is not
nullable, hence during aggregate elimination, the optimizer rewrote aggregate
into a plain column of null-generating side of outer join, which led to the
wrong result.

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

None

@ti-chi-bot
Copy link
Member

ti-chi-bot commented May 9, 2022 •
edited
Loading

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • Reminiscent
  • qw4990

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added release-note-none Denotes a PR that doesn't merit a release note. do-not-merge/needs-triage-completed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels May 9, 2022
@sre-bot
Copy link
Contributor

sre-bot commented May 9, 2022 •
edited
Loading

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/a588d147f8b952297ebd7e7262bd50be0be2cdba

@qw4990 qw4990 self-requested a review May 9, 2022 12:06
@qw4990
Copy link
Contributor

qw4990 commented May 12, 2022

We should format these files by using gofmt -w {file-path}:

[2022-05-10T06:38:28.319Z] planner/core/integration_test.go
[2022-05-10T06:38:28.319Z] planner/core/rule_aggregation_push_down.go
[2022-05-10T06:38:28.319Z] planner/core/integration_test.go
[2022-05-10T06:38:28.319Z] planner/core/rule_aggregation_push_down.go
[2022-05-10T06:38:28.319Z] make: *** [fmt] Error 1

qw4990
qw4990 approved these changes May 12, 2022
View reviewed changes
Copy link
Contributor

@qw4990 qw4990 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label May 12, 2022
@fixdb fixdb force-pushed the agg_pushdown_outerjoin branch from 768743b to 33cb6b4 Compare May 13, 2022 06:02
@fixdb
planner: fix bug that aggregate push down may generate wrong plan for…
6a3ff41 
… outer joins

When we push down aggregate past outer join, the top count aggregate assumed
that the partial count aggregate always return non-null value. But in case of
outer joins when the aggregate is pushed down to the null generating side of
the outer join, the input for the final aggregate may be nullable.

Before this patch, the final count aggregate always assume its input is not
nullable, hence during aggregate elimimation, the optimizer rewrote aggregate
into a plain column of null-generating side of outer join, which led to the
wrong result.

Close pingcap#34465
@fixdb fixdb force-pushed the agg_pushdown_outerjoin branch from 33cb6b4 to 6a3ff41 Compare May 13, 2022 07:24
@fixdb
Copy link
Contributor Author

fixdb commented May 13, 2022

/run-all-tests

@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels May 13, 2022
@Reminiscent Reminiscent added affects-5.0 This bug affects 5.0.x versions. affects-5.1 This bug affects 5.1.x versions. affects-5.2 This bug affects 5.2.x versions. affects-5.3 This bug affects 5.3.x versions. affects-5.4 This bug affects the 5.4.x(LTS) versions. affects-6.0 and removed do-not-merge/needs-triage-completed labels May 13, 2022
@Reminiscent
Copy link
Contributor

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 6a3ff41

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label May 13, 2022
@ti-srebot ti-srebot mentioned this pull request May 13, 2022
Merged
12 tasks
@ti-srebot
Copy link
Contributor

cherry pick to release-5.1 in PR #34646

@ti-srebot ti-srebot mentioned this pull request May 13, 2022
Merged
12 tasks
@ti-srebot
Copy link
Contributor

cherry pick to release-5.2 in PR #34647

@ti-srebot ti-srebot mentioned this pull request May 13, 2022
Merged
12 tasks
@ti-srebot
Copy link
Contributor

cherry pick to release-5.3 in PR #34648

@ti-srebot ti-srebot mentioned this pull request May 13, 2022
Merged
12 tasks
@ti-srebot
Copy link
Contributor

cherry pick to release-5.4 in PR #34649

@ti-srebot
Copy link
Contributor

cherry pick to release-6.0 failed

@sre-bot
Copy link
Contributor

sre-bot commented May 13, 2022

TiDB MergeCI notify

🔴 Bad News! New failing [1] after this pr merged.
These new failed integration tests seem to be caused by the current PR, please try to fix these new failed integration tests, thanks!

CI Name Result Duration Compare with Parent commit
idc-jenkins-ci-tidb/integration-ddl-test 🟥 failed 1, success 5, total 6 7 min 56 sec New failing
idc-jenkins-ci/integration-cdc-test 🟢 all 34 tests passed 23 min Existing passed
idc-jenkins-ci-tidb/integration-common-test 🟢 all 11 tests passed 8 min 55 sec Existing passed
idc-jenkins-ci-tidb/sqllogic-test-1 🟢 all 26 tests passed 5 min 53 sec Existing passed
idc-jenkins-ci-tidb/sqllogic-test-2 🟢 all 28 tests passed 5 min 46 sec Existing passed
idc-jenkins-ci-tidb/common-test 🟢 all 12 tests passed 5 min 41 sec Existing passed
idc-jenkins-ci-tidb/tics-test 🟢 all 1 tests passed 4 min 54 sec Existing passed
idc-jenkins-ci-tidb/mybatis-test 🟢 all 1 tests passed 4 min 47 sec Existing passed
idc-jenkins-ci-tidb/integration-compatibility-test 🟢 all 1 tests passed 4 min 43 sec Existing passed
idc-jenkins-ci-tidb/plugin-test 🟢 build success, plugin test success 4min Existing passed

fixdb added a commit to ti-srebot/tidb that referenced this pull request May 17, 2022
@fixdb
cherry pick pingcap#34468 to release-5.4
92216d1 
Signed-off-by: ti-srebot <ti-srebot@pingcap.com>
fixdb added a commit to ti-srebot/tidb that referenced this pull request May 17, 2022
@fixdb
cherry pick pingcap#34468 to release-5.0
75cd123 
Signed-off-by: ti-srebot <ti-srebot@pingcap.com>
fixdb added a commit to ti-srebot/tidb that referenced this pull request May 17, 2022
@fixdb
cherry pick pingcap#34468 to release-5.1
3160d71 
Signed-off-by: ti-srebot <ti-srebot@pingcap.com>
fixdb added a commit to ti-srebot/tidb that referenced this pull request May 17, 2022
@fixdb
cherry pick pingcap#34468 to release-5.2
82ec1d7 
Signed-off-by: ti-srebot <ti-srebot@pingcap.com>
fixdb added a commit to ti-srebot/tidb that referenced this pull request May 17, 2022
@fixdb
cherry pick pingcap#34468 to release-5.3
bf6fb35 
Signed-off-by: ti-srebot <ti-srebot@pingcap.com>
@fixdb
Copy link
Contributor Author

fixdb commented May 17, 2022

/cherry-pick release-6.0

1 similar comment
@qw4990
Copy link
Contributor

qw4990 commented May 17, 2022

/cherry-pick release-6.0

fixdb added a commit to fixdb/tidb that referenced this pull request May 28, 2022
@fixdb
planner: fix bug that aggregate push down may generate wrong plan for…
1d3bfb3 
… outer joins (pingcap#34468)

close pingcap#34465

(cherry picked from commit 43984d8)
ti-chi-bot pushed a commit that referenced this pull request Jun 14, 2022
@ti-srebot
planner: fix bug that aggregate push down may generate wrong plan for…
70ac9ef 
… outer joins (#34468) (#34648)

close #34465
ti-chi-bot pushed a commit that referenced this pull request Jun 22, 2022
@ti-srebot
planner: fix bug that aggregate push down may generate wrong plan for…
b67cbf9 
… outer joins (#34468) (#34649)

close #34465
ti-chi-bot pushed a commit that referenced this pull request Jun 23, 2022
@ti-srebot
planner: fix bug that aggregate push down may generate wrong plan for…
3279c08 
… outer joins (#34468) (#34647)

close #34465
ti-chi-bot pushed a commit that referenced this pull request Jul 5, 2022
@ti-srebot
planner: fix bug that aggregate push down may generate wrong plan for…
ba76bb4 
… outer joins (#34468) (#34645)

close #34465
ti-chi-bot pushed a commit that referenced this pull request Jul 18, 2022
@ti-srebot
planner: fix bug that aggregate push down may generate wrong plan for…
1c57ffd 
… outer joins (#34468) (#34646)

close #34465
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Reminiscent Reminiscent Reminiscent approved these changes

@qw4990 qw4990 qw4990 approved these changes

Assignees
No one assigned
Labels
affects-5.0 This bug affects 5.0.x versions. affects-5.1 This bug affects 5.1.x versions. affects-5.2 This bug affects 5.2.x versions. affects-5.3 This bug affects 5.3.x versions. affects-5.4 This bug affects the 5.4.x(LTS) versions. affects-6.0 needs-cherry-pick-release-5.0 needs-cherry-pick-release-5.1 needs-cherry-pick-release-5.2 needs-cherry-pick-release-5.3 Type: Need cherry pick to release-5.3 needs-cherry-pick-release-5.4 Should cherry pick this PR to release-5.4 branch. release-note-none Denotes a PR that doesn't merit a release note. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

mpp: wrong planner behaviors when tidb_opt_agg_push_down and tidb_enforce_mpp enabled
6 participants
@fixdb @ti-chi-bot @sre-bot @qw4990 @Reminiscent @ti-srebot