Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bug: slow-SQL-Q4-when subquery use exists sql on 10GB data spend 40 mininus and happen OOM on 100GB data #330

Closed
1 task done
adofsauron opened this issue Jul 28, 2022 · 0 comments · Fixed by #369
Closed
1 task done

bug: slow-SQL-Q4-when subquery use exists sql on 10GB data spend 40 mininus and happen OOM on 100GB data #330

adofsauron opened this issue Jul 28, 2022 · 0 comments · Fixed by #369
Assignees
@adofsauron
Labels
A-bug Something isn't working
Milestone
StoneDB_5.7_v1.0.0

Comments

@adofsauron
Copy link
Collaborator

Describe the problem

  1. use tpch data
  2. exec follow sql

select o_orderpriority, count(*) as order_count from orders where o_orderdate >= date '1993-07-01' and o_orderdate < date '1993-07-01' + interval '3' month and exists ( select * from lineitem where l_orderkey = o_orderkey and l_commitdate < l_receiptdate ) group by o_orderpriority order by o_orderpriority;

Expected behavior

No response

How To Reproduce

No response

Environment

No response

Are you interested in submitting a PR to solve the problem?

  • Yes, I will!
@adofsauron adofsauron added the A-bug Something isn't working label Jul 28, 2022
@adofsauron adofsauron changed the title bug: bug: slow-SQL-Q4-when subquery use exists sql on 10GB data spend 40 mininus and happen OOM on 100GB data bug: slow-SQL-Q4-when subquery use exists sql on 10GB data spend 40 mininus and happen OOM on 100GB data Jul 28, 2022
@adofsauron adofsauron mentioned this issue Aug 5, 2022
Merged
15 tasks
@isredstar isredstar added this to the stonedb_5.7_v1.0.0 milestone Aug 6, 2022
@mergify mergify bot closed this as completed in #369 Aug 9, 2022
mergify bot added a commit that referenced this issue Aug 9, 2022
@adofsauron @mergify
fix stonedb for optimize exists subquery and deep memory copy #369 (#330
cd93ce1 
, #329, #340, #332) (#369)

* fix stonedb for optimize exists subquery and deep memcopy copy

* Update parameterized_filter.cpp

fix ParameterizedFilter deep copy mind memory leak

* fix subquery when this is in sql

* fix code readability and defensive

* use shallow memory when paral hash join by class DimensionGroupMultiMaterialized

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
xiaoguangye pushed a commit to xiaoguangye/stonedb that referenced this issue Aug 11, 2022
@leizzboy @xiaoguangye
fix bugs of stoneatom#309 and stoneatom#310 (stoneatom#313)
6069120 
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

fix(mtr):mtr result stonedb change to tianmu(stoneatom#359) (stoneatom#368)

* fix(mtr):mysqlpump*.test sys_stoendb to sys_tianmu

fix(mtr):fix main result stonedb to tianmu

fix(mtr):mv innodb_fts.ngram_1 to test/innodb_fts

fix(mtr):fix mtr funcs_1 stonedb to tianmu

fix(mtr):fix innodb mtr result stonedb change to tianmu

fix(mtr):fix perfschema mtr result stonedb change to tianmu

* fix(mtr):fix mtr stoendb change to tianmu

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

fix(tianmu): fix group_concat_max_len warning (stoneatom#338) (stoneatom#363)

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

docs(stonedb): update docs and fix bugs (stoneatom#373)

* docs(Quick Deployment and Compiling Methods): update docs

Make the document more user-friendly for new users

* fix(docs): Let the official website pictures can be adaptive

* Update compile-using-ubuntu2004.md

In the install make step, add the go to directory：cd make-3.82

* Update compile-using-redhat7.md

In the install make step, add the go to directory：cd make-3.82

* Update compile-using-centos7.md

In the install make step, add the go to directory：cd make-3.82

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: haitaoguan <105625912+haitaoguan@users.noreply.github.com>

feat(doc):doc add innodb backup to stonedb demonstrate（stoneatom#353） (stoneatom#355)

* feat(doc):doc add innodb backup to stonedb demonstrate

* fix(doc):fix mydumper and mysqldump doc

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

feat(innodb): To support compiling innodb as C++ 17(stoneatom#375) (stoneatom#376)

To support innodb is compiled as C++ 17. For now, tianmu is compiling as C++17,
and C++17 has became mainstream of C++ standard. Therefore, now, we star to support
to compile Innodb as C++17 in stonedb.

This is backport from MySQL, which commit id is: 1a8a111d8f855a31d0aeffc8f02309b2b82dd410 and bug# is #32907274

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

fix stonedb for optimize exists subquery and deep memory copy stoneatom#369 (stoneatom#330, stoneatom#329, stoneatom#340, stoneatom#332) (stoneatom#369)

* fix stonedb for optimize exists subquery and deep memcopy copy

* Update parameterized_filter.cpp

fix ParameterizedFilter deep copy mind memory leak

* fix subquery when this is in sql

* fix code readability and defensive

* use shallow memory when paral hash join by class DimensionGroupMultiMaterialized

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

feat(stonedb): To support gcc 9 and adds `WITH_ROCKSDB` and `WITH_MARISA` to specify where (stoneatom#377)

rocksdb and marisa installed, respectively.

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

fix(mtr): Add test cases for NULL value select on UNIQUE constraint(stoneatom#281) (stoneatom#379)

* fix(core): Double prepare on lex unit for cts with union(stoneatom#226)

fix(core): MTR + Double prepare on lex unit for ctas with union(stoneatom#226)

* fix(mtr): Add test cases for NULL value select on UNIQUE constraint(stoneatom#281)

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

fix bug stoneatom#342 (stoneatom#346)

Co-authored-by: dfx <duanfuxiang>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

Doc(stonedb): Added extensions deccriptions in topics related to backup and recovery for O&M Guide(stoneatom#360) (stoneatom#361)

* updated some code snippets in Install RocksDB step

Signed-off-by: xuejiao-joy <821682573@qq.com>

* some description here

Signed-off-by: xuejiao-joy <821682573@qq.com>

Co-authored-by: Yuting <34496563+Nliver@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

fix(mtr): Add more ctas queries for the same bug-fix from code-analysis perspectives(stoneatom#226)
xiaoguangye added a commit to xiaoguangye/stonedb that referenced this issue Aug 11, 2022
@xiaoguangye
fix(core): Double prepare on lex unit for cts with union(stoneatom#226)
444f014 
fix(core): MTR + Double prepare on lex unit for ctas with union(stoneatom#226)

rich doc to fix  env dependence "not found" (stoneatom#316)

fix(mtr): make the mtr passed (stoneatom#315) (stoneatom#317)

[summary]
How to run:
./mysql-test-run.pl --suite=stonedb --nowarnings --force --nocheck-testcases

1. test cases below are failed, we ignored them first:
issue410
issue415
issue433
issue446
issue515
load.test
2. Sys variables next are not set correctly, they should be added to `/mysql-test/include/default_mysqld.cnf`.
stonedb_insert_delayed = 0
stonedb_ini_allowmysqlquerypath=1

3.check-testcases: A new MTR option to enforce strict cleanup:
https://dev.mysql.com/blog-archive/fail-check-testcases-a-new-mtr-option-to-enforce-strict-cleanup

In `mtr` comment:
```
Some options that control enabling a feature for normal test runs,
can be turned off by prepending 'no' to the option, e.g. --notimer.
This applies to reorder, timer, check-testcases and warnings.
```
Currently we turn off it first to make mtr run success.

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

feat(stonedb): update the copyright date time (stoneatom#321) (stoneatom#324)

fix(core): fix crash when executing sql. (stoneatom#327) (stoneatom#328)

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

feat(index): Unified class member variable name with google style (stoneatom#311) (stoneatom#312)

[summary]
Class member name will be like:

class TableInfo {
  ...
 private:
  std::string table_name_;  // OK - underscore at end.
  static Pool<TableInfo>* pool_;  // OK.
};

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

fix(mtr):move character_set to test.character_set, fix main mtr_test and innodb mtr_test(stoneatom#277) (stoneatom#335)

* fix(mtr):fix innodb mtr and mv some test to test/innodb

* fix(mtr):move character_set to test/main/

fix(mtr):mv main mtr character_set test to test/main/cahracter_set

fix(mtr):mv myisam error mtr-test to test/main/myisam

fix(mtr):add optimizer_switch to include/default_mysqld.cnf

* fix(mtr):fix main mtr optimizer_switch amd mv some test to test/main

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

fix(mtr): Add more ctas queries for the same bug-fix from code-analysis perspectives(stoneatom#226)

fix bugs of stoneatom#309 and stoneatom#310 (stoneatom#313)

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

fix(core): Double prepare on lex unit for cts with union(stoneatom#226) (stoneatom#365)

fix(core): MTR + Double prepare on lex unit for ctas with union(stoneatom#226)

fix(mtr):mtr result stonedb change to tianmu(stoneatom#359) (stoneatom#368)

* fix(mtr):mysqlpump*.test sys_stoendb to sys_tianmu

fix(mtr):fix main result stonedb to tianmu

fix(mtr):mv innodb_fts.ngram_1 to test/innodb_fts

fix(mtr):fix mtr funcs_1 stonedb to tianmu

fix(mtr):fix innodb mtr result stonedb change to tianmu

fix(mtr):fix perfschema mtr result stonedb change to tianmu

* fix(mtr):fix mtr stoendb change to tianmu

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

fix(tianmu): fix group_concat_max_len warning (stoneatom#338) (stoneatom#363)

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

docs(stonedb): update docs and fix bugs (stoneatom#373)

* docs(Quick Deployment and Compiling Methods): update docs

Make the document more user-friendly for new users

* fix(docs): Let the official website pictures can be adaptive

* Update compile-using-ubuntu2004.md

In the install make step, add the go to directory：cd make-3.82

* Update compile-using-redhat7.md

In the install make step, add the go to directory：cd make-3.82

* Update compile-using-centos7.md

In the install make step, add the go to directory：cd make-3.82

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: haitaoguan <105625912+haitaoguan@users.noreply.github.com>

feat(doc):doc add innodb backup to stonedb demonstrate（stoneatom#353） (stoneatom#355)

* feat(doc):doc add innodb backup to stonedb demonstrate

* fix(doc):fix mydumper and mysqldump doc

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

feat(innodb): To support compiling innodb as C++ 17(stoneatom#375) (stoneatom#376)

To support innodb is compiled as C++ 17. For now, tianmu is compiling as C++17,
and C++17 has became mainstream of C++ standard. Therefore, now, we star to support
to compile Innodb as C++17 in stonedb.

This is backport from MySQL, which commit id is: 1a8a111d8f855a31d0aeffc8f02309b2b82dd410 and bug# is #32907274

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

fix stonedb for optimize exists subquery and deep memory copy stoneatom#369 (stoneatom#330, stoneatom#329, stoneatom#340, stoneatom#332) (stoneatom#369)

* fix stonedb for optimize exists subquery and deep memcopy copy

* Update parameterized_filter.cpp

fix ParameterizedFilter deep copy mind memory leak

* fix subquery when this is in sql

* fix code readability and defensive

* use shallow memory when paral hash join by class DimensionGroupMultiMaterialized

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

feat(stonedb): To support gcc 9 and adds `WITH_ROCKSDB` and `WITH_MARISA` to specify where (stoneatom#377)

rocksdb and marisa installed, respectively.

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

fix(mtr): Add test cases for NULL value select on UNIQUE constraint(stoneatom#281) (stoneatom#379)

* fix(core): Double prepare on lex unit for cts with union(stoneatom#226)

fix(core): MTR + Double prepare on lex unit for ctas with union(stoneatom#226)

* fix(mtr): Add test cases for NULL value select on UNIQUE constraint(stoneatom#281)

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

fix bug stoneatom#342 (stoneatom#346)

Co-authored-by: dfx <duanfuxiang>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

Doc(stonedb): Added extensions deccriptions in topics related to backup and recovery for O&M Guide(stoneatom#360) (stoneatom#361)

* updated some code snippets in Install RocksDB step

Signed-off-by: xuejiao-joy <821682573@qq.com>

* some description here

Signed-off-by: xuejiao-joy <821682573@qq.com>

Co-authored-by: Yuting <34496563+Nliver@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

fix(mtr): Add more ctas queries for the same bug-fix from code-analysis perspectives(stoneatom#226)
@xiaoguangye xiaoguangye mentioned this issue Aug 11, 2022
Closed
15 tasks
@adofsauron adofsauron mentioned this issue Sep 29, 2022
Closed
15 tasks
@adofsauron adofsauron mentioned this issue Oct 8, 2022
Closed
15 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@adofsauron adofsauron

Labels
A-bug Something isn't working
Projects
None yet
Milestone
StoneDB_5.7_v1.0.0
2 participants
@adofsauron @isredstar