Use aircompressor codecs #3575
Merged
Use aircompressor codecs #3575
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fixes deephaven#3470 Verified deephaven#2495 and deephaven#2569 still work, as well as LZO compressed parquet files
devinrsmith
added
parquet
devinrsmith
changed the title
3 tasks
devinrsmith
changed the title
devinrsmith
requested review from
chipkent,
jmao-denver and
rcaudy
as code owners
|
Nightlies all pass. |
chipkent
approved these changes
Mar 24, 2023
abaranec
approved these changes
Mar 31, 2023
|
I'll get these all run on M1 before merge. |
|
I've verified on M1 that the new tests pass, and have also verified reading large parquet files in GZIP, LZO, LZ4, SNAPPY, and ZSTD. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Updates to use the aircompressor ZSTD codec, which does not suffer from the same off-heap leak that our own implementation does. Fixes #3470
Additionally, this PR more explicitly configures the order of codecs using
org.apache.hadoop.conf.Configuration, and ensures we use the aircompressor GZIP, LZO, and LZ4 codecs. The structure has also been updated to give us explicit control over mapping customorg.apache.hadoop.io.compress.CompressionCodecto the appropriateorg.apache.parquet.hadoop.metadata.CompressionCodecName.Verified #2495 and #2569 still work, as well as LZO compressed parquet files