Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Flink/Rust] Adjust rolling file logic to reduce memory usage during write #426

Merged
merged 2 commits into from
Jan 19, 2024
Merged

[Flink/Rust] Adjust rolling file logic to reduce memory usage during write #426

merged 2 commits into from
Jan 19, 2024

Conversation

xuchen-plus
Copy link
Contributor

  1. Roll part file for every 5 million rows by default;
  2. Fix error not returned when sorting writer failed.

@dmetasoul01
adjust rolling file logic to reduce memory usage during write
1aa2ae0 
Signed-off-by: chenxu <chenxu@dmetasoul.com>
@xuchen-plus xuchen-plus added bug Something isn't working enhancement New feature or request flink flink support into lakesoul native-io labels Jan 19, 2024
@xuchen-plus xuchen-plus requested a review from Ceng23333 January 19, 2024 08:03
@dmetasoul01
fix parameters not used from flink-conf.yaml
eb506f9 
Signed-off-by: chenxu <chenxu@dmetasoul.com>
@xuchen-plus xuchen-plus merged commit 58600b6 into lakesoul-io:main Jan 19, 2024
17 checks passed
@xuchen-plus xuchen-plus deleted the optimize_flink_memory branch January 19, 2024 09:55
mag1c1an1 pushed a commit to mag1c1an1/LakeSoul that referenced this pull request Jan 20, 2024
@xuchen-plus @dmetasoul01 @mag1c1an1
[Flink/Rust] Adjust rolling file logic to reduce memory usage during …
517c000 
…write (lakesoul-io#426)

* adjust rolling file logic to reduce memory usage during write

Signed-off-by: chenxu <chenxu@dmetasoul.com>

* fix parameters not used from flink-conf.yaml

Signed-off-by: chenxu <chenxu@dmetasoul.com>

---------

Signed-off-by: chenxu <chenxu@dmetasoul.com>
Co-authored-by: chenxu <chenxu@dmetasoul.com>
xuchen-plus added a commit that referenced this pull request Jan 29, 2024
@xuchen-plus @dmetasoul01
[Flink/Rust] Adjust rolling file logic to reduce memory usage during …
49eec84 
…write (#426)

* adjust rolling file logic to reduce memory usage during write

Signed-off-by: chenxu <chenxu@dmetasoul.com>

* fix parameters not used from flink-conf.yaml

Signed-off-by: chenxu <chenxu@dmetasoul.com>

---------

Signed-off-by: chenxu <chenxu@dmetasoul.com>
Co-authored-by: chenxu <chenxu@dmetasoul.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Ceng23333 Ceng23333 Awaiting requested review from Ceng23333

Assignees
No one assigned
Labels
bug Something isn't working enhancement New feature or request flink flink support into lakesoul native-io
Projects
2.5.0 Release
Status: Done
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@xuchen-plus @dmetasoul01