Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[NativeIO] Native io misc improvements #190

Merged
merged 3 commits into from
Mar 30, 2023
Merged

[NativeIO] Native io misc improvements #190

merged 3 commits into from
Mar 30, 2023

Conversation

dmetasoul01
Copy link
Contributor

@dmetasoul01 dmetasoul01 commented Mar 30, 2023 •
edited
Loading

  1. Improve MOR reader performance by 30% by reducing elements in min heap during merge
  2. Catch exceptions in java callback
  3. Add micro batch capability for NativeParquetScan

Close #178

@dmetasoul01 dmetasoul01 added enhancement New feature or request spark spark support into lakesoul native-io labels Mar 30, 2023
@dmetasoul01 dmetasoul01 requested a review from Ceng23333 March 30, 2023 09:25
@dmetasoul01 dmetasoul01 self-assigned this Mar 30, 2023
@dmetasoul01 dmetasoul01 merged commit f65a2f6 into main Mar 30, 2023
@dmetasoul01 dmetasoul01 deleted the native_io_misc branch March 30, 2023 11:24
Ceng23333 pushed a commit that referenced this pull request Apr 21, 2023
@dmetasoul01 @dmetasoul-opensource @Ceng23333
[NativeIO] Native io misc improvements (#190)
e9711e0 
* improve mor read by avoiding adding too many batch into heap

* catch exception in native io read java callback

* add stream capability to native parquet scan

---------

Co-authored-by: dmetasoul01 <opensource@dmetasoul.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Ceng23333 Ceng23333 Ceng23333 approved these changes

Assignees

@dmetasoul01 dmetasoul01

Labels
enhancement New feature or request native-io spark spark support into lakesoul
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[NativeIO][Spark] Enable native io by default
3 participants
@dmetasoul01 @Ceng23333 @dmetasoul-opensource