[Go][Parquet] Allow registering new codecs for parquet #40113
Comments
zhouyan
added a commit
to zhouyan/arrow
that referenced
this issue
Feb 20, 2024
zhouyan
added a commit
to zhouyan/arrow
that referenced
this issue
Feb 20, 2024
zeroshade
pushed a commit
that referenced
this issue
Feb 20, 2024
This is to allow addition/overwrite of custom codec implementation This allows other modules to provide alternative implementations for the compression algorithms, such as using libdeflate for Gzip, or CGO version of ZSTD. In addition, it allows others to supply codecs that cannot be easily supported by this library such as LZO due to license reasons or LZ4. ### Rationale for this change See #40113 ### What changes are included in this PR? A new RegisterCodec function added ### Are these changes tested? yes ### Are there any user-facing changes? It's an addition more targeted towards library writers. * Closes: #40113 Authored-by: Yan Zhou <zhouyan@me.com> Signed-off-by: Matt Topol <zotthewizard@gmail.com>
zanmato1984
pushed a commit
to zanmato1984/arrow
that referenced
this issue
Feb 28, 2024
This is to allow addition/overwrite of custom codec implementation This allows other modules to provide alternative implementations for the compression algorithms, such as using libdeflate for Gzip, or CGO version of ZSTD. In addition, it allows others to supply codecs that cannot be easily supported by this library such as LZO due to license reasons or LZ4. ### Rationale for this change See apache#40113 ### What changes are included in this PR? A new RegisterCodec function added ### Are these changes tested? yes ### Are there any user-facing changes? It's an addition more targeted towards library writers. * Closes: apache#40113 Authored-by: Yan Zhou <zhouyan@me.com> Signed-off-by: Matt Topol <zotthewizard@gmail.com>
thisisnic
pushed a commit
to thisisnic/arrow
that referenced
this issue
Mar 8, 2024
This is to allow addition/overwrite of custom codec implementation This allows other modules to provide alternative implementations for the compression algorithms, such as using libdeflate for Gzip, or CGO version of ZSTD. In addition, it allows others to supply codecs that cannot be easily supported by this library such as LZO due to license reasons or LZ4. ### Rationale for this change See apache#40113 ### What changes are included in this PR? A new RegisterCodec function added ### Are these changes tested? yes ### Are there any user-facing changes? It's an addition more targeted towards library writers. * Closes: apache#40113 Authored-by: Yan Zhou <zhouyan@me.com> Signed-off-by: Matt Topol <zotthewizard@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the enhancement requested
Current Parquet implementation in Go does not support all codec defined in the standard, some due to licensing issues, which is fine. And the choice of implementation is also not necessarily the best for every use cases. For example, I would like to be able to use libdeflate for GZIP codec or CGO binding of ZSTD codec. Instead of updating the implementation in the main repo, I think it is better to allow user to supply their own. For example, CGO may not be suitable for all use cases and hence may not be suitable as the default implementation.
Component(s)
Go, Parquet
The text was updated successfully, but these errors were encountered: