Skip to content

Commit

Permalink
GH-40113 [Go][Parquet] New RegisterCodec function (#40114)
Browse files Browse the repository at this point in the history
This is to allow addition/overwrite of custom codec implementation

This allows other modules to provide alternative implementations for the compression algorithms, such as using libdeflate for Gzip, or CGO version of ZSTD.

In addition, it allows others to supply codecs that cannot be easily supported by this library such as LZO due to license reasons or LZ4.

### Rationale for this change

See #40113

### What changes are included in this PR?

A new RegisterCodec function added 

### Are these changes tested?

yes

### Are there any user-facing changes?

It's an addition more targeted towards library writers. 

* Closes: #40113

Authored-by: Yan Zhou <zhouyan@me.com>
Signed-off-by: Matt Topol <zotthewizard@gmail.com>
  • Loading branch information
zhouyan authored Feb 20, 2024
1 parent a690088 commit 47f15b0
Show file tree
Hide file tree
Showing 5 changed files with 24 additions and 4 deletions.
2 changes: 1 addition & 1 deletion go/parquet/compress/brotli.go
Original file line number Diff line number Diff line change
Expand Up @@ -110,5 +110,5 @@ func (brotliCodec) NewWriterLevel(w io.Writer, level int) (io.WriteCloser, error
}

func init() {
codecs[Codecs.Brotli] = brotliCodec{}
RegisterCodec(Codecs.Brotli, brotliCodec{})
}
20 changes: 20 additions & 0 deletions go/parquet/compress/compress.go
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,26 @@ type Codec interface {

var codecs = map[Compression]Codec{}

// RegisterCodec adds or overrides a codec implementation for a given compression algorithm.
// The intended use case is within the init() section of a package. For example,
//
// // inside a custom codec package, say czstd
//
// func init() {
// RegisterCodec(compress.Codecs.Zstd, czstdCodec{})
// }
//
// type czstdCodec struct{} // implementing Codec interface using CGO based ZSTD wrapper
//
// And user of the custom codec can import the above package like below,
//
// package main
//
// import _ "package/path/to/czstd"
func RegisterCodec(compression Compression, codec Codec) {
codecs[compression] = codec
}

type nocodec struct{}

func (nocodec) NewReader(r io.Reader) io.ReadCloser {
Expand Down
2 changes: 1 addition & 1 deletion go/parquet/compress/gzip.go
Original file line number Diff line number Diff line change
Expand Up @@ -93,5 +93,5 @@ func (gzipCodec) NewWriterLevel(w io.Writer, level int) (io.WriteCloser, error)
}

func init() {
codecs[Codecs.Gzip] = gzipCodec{}
RegisterCodec(Codecs.Gzip, gzipCodec{})
}
2 changes: 1 addition & 1 deletion go/parquet/compress/snappy.go
Original file line number Diff line number Diff line change
Expand Up @@ -57,5 +57,5 @@ func (s snappyCodec) NewWriterLevel(w io.Writer, _ int) (io.WriteCloser, error)
}

func init() {
codecs[Codecs.Snappy] = snappyCodec{}
RegisterCodec(Codecs.Snappy, snappyCodec{})
}
2 changes: 1 addition & 1 deletion go/parquet/compress/zstd.go
Original file line number Diff line number Diff line change
Expand Up @@ -108,5 +108,5 @@ func (zstdCodec) CompressBound(len int64) int64 {
}

func init() {
codecs[Codecs.Zstd] = zstdCodec{}
RegisterCodec(Codecs.Zstd, zstdCodec{})
}

0 comments on commit 47f15b0

Please sign in to comment.