Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[bugfix](inverted index) temporary disable skip reading column data if it match inverted index #18065

Merged
merged 1 commit into from
Mar 27, 2023
Merged

[bugfix](inverted index) temporary disable skip reading column data if it match inverted index #18065

merged 1 commit into from
Mar 27, 2023

Conversation

xiaokang
Copy link
Contributor

Proposed changes

Issue Number: close #xxx

Problem summary

The optimization for skip reading column data if it match inverted index and only used in WHERE clause may get wrong result for complex SQL.

This PR temporary disable the optimization and later PRs will resolve the problem fundamentality.

Checklist(Required)

  • Does it affect the original behavior
  • Has unit tests been added
  • Has document been added or modified
  • Does it need to update dependencies
  • Is this PR support rollback (If NO, please explain WHY)

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@xiaokang
Copy link
Contributor Author

run buildall

@hello-stephen
Copy link
Contributor

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 35.52 seconds
stream load tsv: 459 seconds loaded 74807831229 Bytes, about 155 MB/s
stream load json: 24 seconds loaded 2358488459 Bytes, about 93 MB/s
stream load orc: 74 seconds loaded 1101869774 Bytes, about 14 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230323182521_clickbench_pr_119840.html

Tanya-W
Tanya-W approved these changes Mar 27, 2023
View reviewed changes
Copy link
Contributor

@Tanya-W Tanya-W left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

qidaye
qidaye approved these changes Mar 27, 2023
View reviewed changes
Copy link
Contributor

@qidaye qidaye left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 27, 2023
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@qidaye qidaye merged commit d1f34a3 into apache:master Mar 27, 2023
luwei16 pushed a commit to luwei16/incubator-doris that referenced this pull request Apr 7, 2023
@Tanya-W @xiaokang
Pick "[bugfix](inverted index)temporary disable skip read column data…
89fbfa2 
… if it match inverted index (apache#18065)" (apache#1545)

The optimization for skip reading column data if it match inverted index and only used in WHERE clause may get wrong result for complex SQL.

This PR temporary disable the optimization and later PRs will resolve the problem fundamentality.

Fix http://jira.flywheels.com:8090/browse/CORE-1739 

commit d1f34a3
Author: Kang <kxiao.tiger@gmail.com>
Date:   Mon Mar 27 11:29:42 2023 +0800

    [bugfix](inverted index)temporary disable skip read column data if it match inverted index (apache#18065)

    The optimization for skip reading column data if it match inverted index and only used in WHERE clause may get wrong result for complex SQL.

    This PR temporary disable the optimization and later PRs will resolve the problem fundamentality.

Co-authored-by: Kang <kxiao.tiger@gmail.com>
luwei16 pushed a commit to luwei16/incubator-doris that referenced this pull request Apr 7, 2023
@xiaokang
[cherry-pick] Pick doris topn 18121, jsonb 18065 17430, inverted inde…
de2b520 
…x 17352 (apache#1565)


commit 4e1e0ce
Author: Kang <kxiao.tiger@gmail.com>
Date:   Fri Mar 31 10:01:34 2023 +0800

    [bugfix](topn) fix topn optimzation wrong result for NULL values (apache#18121)
    
    1. add PassNullPredicate to fix topn wrong result for NULL values
    2. refactor RuntimePredicate to avoid using TCondition
    3. refactor using ordering_exprs in fe and vsort_node


Author: Kang <kxiao.tiger@gmail.com>
Date:   Mon Mar 27 11:29:42 2023 +0800

    [bugfix](inverted index)temporary disable skip read column data if it match inverted index (apache#18065)
    
    The optimization for skip reading column data if it match inverted index and only used in WHERE clause may get wrong result for complex SQL.
    
    This PR temporary disable the optimization and later PRs will resolve the problem fundamentality.


Author: Kang <kxiao.tiger@gmail.com>
Date:   Wed Mar 8 14:18:26 2023 +0800

    [bugfix](jsonb) Fix create mv using jsonb key cause be crash (apache#17430)


Author: Kang <kxiao.tiger@gmail.com>
Date:   Fri Mar 3 21:00:49 2023 +0800

    [improvement](inverted index)use reference to avoid bitmap copy for performance (apache#17352)
    
    Query runtime is reduced from 10s to 1s for a MATCH query that match 40 million rows out of 44 million rows.
gnehil pushed a commit to gnehil/doris that referenced this pull request Apr 21, 2023
@xiaokang @gnehil
[bugfix](inverted index)temporary disable skip read column data if it…
c91bc7c 
… match inverted index (apache#18065)

The optimization for skip reading column data if it match inverted index and only used in WHERE clause may get wrong result for complex SQL.

This PR temporary disable the optimization and later PRs will resolve the problem fundamentality.
xiaokang added a commit to xiaokang/doris that referenced this pull request May 6, 2023
@xiaokang
Revert "[bugfix](inverted index)temporary disable skip read column da…
425b4a6 
…ta if it match inverted index (apache#18065)"

This reverts commit d1f34a3.
SWJTU-ZhangLei pushed a commit to SWJTU-ZhangLei/incubator-doris that referenced this pull request Jul 25, 2023
@Tanya-W @xiaokang @gavinchou
Pick "[bugfix](inverted index)temporary disable skip read column data…
cf9d9af 
… if it match inverted index (apache#18065)" (apache#1545)

The optimization for skip reading column data if it match inverted index and only used in WHERE clause may get wrong result for complex SQL.

This PR temporary disable the optimization and later PRs will resolve the problem fundamentality.

Fix http://jira.flywheels.com:8090/browse/CORE-1739 

commit d1f34a3
Author: Kang <kxiao.tiger@gmail.com>
Date:   Mon Mar 27 11:29:42 2023 +0800

    [bugfix](inverted index)temporary disable skip read column data if it match inverted index (apache#18065)

    The optimization for skip reading column data if it match inverted index and only used in WHERE clause may get wrong result for complex SQL.

    This PR temporary disable the optimization and later PRs will resolve the problem fundamentality.

Co-authored-by: Kang <kxiao.tiger@gmail.com>
SWJTU-ZhangLei pushed a commit to SWJTU-ZhangLei/incubator-doris that referenced this pull request Jul 25, 2023
@xiaokang @gavinchou
[cherry-pick] Pick doris topn 18121, jsonb 18065 17430, inverted inde…
220317c 
…x 17352 (apache#1565)

commit 4e1e0ce
Author: Kang <kxiao.tiger@gmail.com>
Date:   Fri Mar 31 10:01:34 2023 +0800

    [bugfix](topn) fix topn optimzation wrong result for NULL values (apache#18121)

    1. add PassNullPredicate to fix topn wrong result for NULL values
    2. refactor RuntimePredicate to avoid using TCondition
    3. refactor using ordering_exprs in fe and vsort_node

Author: Kang <kxiao.tiger@gmail.com>
Date:   Mon Mar 27 11:29:42 2023 +0800

    [bugfix](inverted index)temporary disable skip read column data if it match inverted index (apache#18065)

    The optimization for skip reading column data if it match inverted index and only used in WHERE clause may get wrong result for complex SQL.

    This PR temporary disable the optimization and later PRs will resolve the problem fundamentality.

Author: Kang <kxiao.tiger@gmail.com>
Date:   Wed Mar 8 14:18:26 2023 +0800

    [bugfix](jsonb) Fix create mv using jsonb key cause be crash (apache#17430)

Author: Kang <kxiao.tiger@gmail.com>
Date:   Fri Mar 3 21:00:49 2023 +0800

    [improvement](inverted index)use reference to avoid bitmap copy for performance (apache#17352)

    Query runtime is reduced from 10s to 1s for a MATCH query that match 40 million rows out of 44 million rows.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@qidaye qidaye qidaye approved these changes

@Tanya-W Tanya-W Tanya-W approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@xiaokang @hello-stephen @qidaye @Tanya-W