Skip to content
This repository has been archived by the owner on Sep 18, 2023. It is now read-only.

ColumnarWSCG further optimization #17

Closed
3 of 4 tasks
xuechendi opened this issue Jan 7, 2021 · 0 comments · Fixed by #18 or #47
Closed
3 of 4 tasks

ColumnarWSCG further optimization #17

xuechendi opened this issue Jan 7, 2021 · 0 comments · Fixed by #18 or #47
Labels
feature

Comments

@xuechendi
Copy link
Collaborator

xuechendi commented Jan 7, 2021 •
edited
Loading

We noticed that in Q72 native WSCG is still slower than vanilla spark, some ideas below may improve performance.

  • postpone GetValue in WSCG when value is required.
  • BHJ build : we may decide hashmap initial size according to build table size
  • BHJ Get: optimize hashmap probe and add a fast cache
  • SMJ optimization: TBD
@zhouyuan zhouyuan mentioned this issue Jan 8, 2021
Merged
@xuechendi xuechendi mentioned this issue Jan 18, 2021
Merged
zhouyuan pushed a commit to zhouyuan/native-sql-engine that referenced this issue Mar 11, 2021
@zhztheplayer
[ADS-16] Upgrade Arrow version to 3.0.0 (oap-project#17)
1cc5531 
Closes oap-project#16
zhouyuan added a commit that referenced this issue Mar 12, 2021
@zhouyuan @jerrychenhf
@zhixingheyi-tian @zhztheplayer @rui-mo
[NSE-86] Merge Arrow Data Source (#152)
da6d1ee 
* [ARROW-DATA-SOURCE-1] Reorganize the source code for the new repository organization

* [ARROW-DATA-SOURCE-1] Make compiling succeed

* [ADS-2] Fix RAM usage CI (#3)

* Update tpch.yml

* Update report_ram_log.yml

* Update report_ram_log.yml

* Delete github-ci-fix

* Delete github-ci-fix2

* [Arrow-Data-Source-4]Add mkdocs.yml and update docs

* [ADS-6]  Add utility methods to check leaked Allocators/MemoryPools (#7)

Close #6

* [NSE-51] Update ArrowWritableColumnVector

* [ADS-9][Parquet] Parquet data source not replaced by default (#10)

* [ADS-9][Parquet] Parquet data source not replaced by default

* Code style

* [ADS-13] Validate metric TaskMetrics.peakExecutionMemory for native SQL engine (#14)

Closes #13

* [ADS-11]Modify title check and automatic link to Issues for PRs (#12)

* [ADS-16] Upgrade Arrow version to 3.0.0 (#17)

Closes #16

* Initialize new repo

* Move arrow data source files to arrow-data-source directory

* move native sql files to native-sql-engine folder

* [NSE-86] Add root pom.xml; Remove native-sql-engine/core/ArrowWritableColumnVector.java (#88)

* fix github actions

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

* fix building & CI

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

Co-authored-by: Chen Haifeng <haifeng.chen@intel.com>
Co-authored-by: zhixingheyi-tian  <xiangxiang.shen@intel.com>
Co-authored-by: Hongze Zhang <mailtozhz@126.com>
Co-authored-by: HongW2019 <hong2.wang@intel.com>
Co-authored-by: Hongze Zhang <hongze.zhang@intel.com>
Co-authored-by: Rui Mo <rui.mo@intel.com>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
feature
Projects
None yet
Milestone
No milestone
2 participants
@xuechendi @zhixingheyi-tian