-
Notifications
You must be signed in to change notification settings - Fork 79
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Prevent parquet from closing zstd streams on finalization #2586
Prevent parquet from closing zstd streams on finalization #2586
Conversation
*/ | ||
public class ZstdCodec extends ZstandardCodec { | ||
@Override | ||
public CompressionInputStream createInputStream(InputStream stream) throws IOException { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think all of these changes belong in a differently-licensed module. Probably ParquetHadoop
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Or we can make this module apache as well? (If it makes more sense to put into ParquetHadoop b/c they are intimately related, then I agree that might be best.)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I moved the code for the moment anyway, but if we end up adding much more in this compression stuff I think it could be nice to open that up, offer as a different option than the upstream CompressionCodecFactory(s).
...uetHadoop/src/main/java/io/deephaven/parquet/compress/codec/zstd/ZstdDecompressorStream.java
Outdated
Show resolved
Hide resolved
…c/zstd/ZstdDecompressorStream.java Co-authored-by: Ryan Caudy <rcaudy@gmail.com>
Fixes #2569