feat: buffered parquet writer #1263
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## develop #1263 +/- ##
===========================================
- Coverage 85.88% 85.63% -0.25%
===========================================
Files 500 500
Lines 75697 75667 -30
===========================================
- Hits 65013 64799 -214
- Misses 10684 10868 +184 |
v0y4g3r
force-pushed
the
feat/streaming-parquet-writer
branch
from
7349924 to
2898e88
Compare
|
It's impressive! |
waynexia
reviewed
Mar 31, 2023
killme2008
reviewed
Mar 31, 2023
…to buffer before flush to underlying storage
v0y4g3r
force-pushed
the
feat/streaming-parquet-writer
branch
from
ad76340 to
c5a2f8a
Compare
killme2008
approved these changes
Apr 1, 2023
This was referenced Apr 1, 2023
paomian
pushed a commit
to paomian/greptimedb
that referenced
this pull request
Oct 19, 2023
* wip: use * rebase develop * chore: fix typos * feat: replace export parquet writer with buffered writer * fix: some cr comments * feat: add sst_write_buffer_size config item to config how many bytes to buffer before flush to underlying storage * chore: reabse onto develop
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I hereby agree to the terms of the GreptimeDB CLA
What's changed and what's your intention?
This PR proposed a buffered parquet writer to avoid writing whole parquet file into memory and flush to underlying storage at a time, which is caused by the gap between ArrowWriter which requires a
std::io::Writeand opendal'sWriter, which implementsfutures::AsyncWriteonly.This PR can help to reduce the memory consumption during flush/compaction.
Given an experiment that writes 1,000,000,000 rows into parquet file (target file size 1.2GB), original code uses up to 1.5GB ram while
BufferedWriteruses at most 270MB.Original
ParquetWriterwhich used to export table data is also replaced byBufferedWriter. This change can write table content to single file instead of previous approach that exports into several fragments while significantly improves export performance to reduce cost time to nearly 60%. Additionally, it can smooth the memory footprint as demonstrated belowKnown issue
This solution requires opendal's backend support
WriterAPI to continously append data to the end of object. This API is supported by fs and s3 backend, but not yet by aliyun oss (which I'm working on).Checklist
Refer to a related PR or issue link (optional)