-
Notifications
You must be signed in to change notification settings - Fork 153
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Consider adding option to use cgo zstd implementation #482
Comments
Thanks for the excellent investigation work- I will try to take a look early next week. |
I think this is something worth trying to land, and it would be good to keep both implementations around at least to begin with. I hope it doesn't make cross-compilation too difficult. |
Allows to select cgo implementation of zstd if compiled in. The cgo implementation is more performant than pure go implementation, see buchgr#482 for actual benchmarks.
Allows to select cgo implementation of zstd if compiled in. The cgo implementation is more performant than pure go implementation, see buchgr#482 for actual benchmarks.
Allows to select cgo implementation of zstd if compiled in. The cgo implementation is more performant than pure go implementation, see buchgr#482 for actual benchmarks.
Added WIP implementation: #485 For now results for downloading files are promising (40% overhead of enabling zstd instead of 160%), but looks like overhead of uploading files is too high in the bazel itself - the PR reduced it to 310% from 490%, but those numbers are far from expected based on raw zstd benchmarks. Investigating. |
Marked #485 as ready for review, the overhead of compression is actually expected. |
Allows to select cgo implementation of zstd if compiled in. The cgo implementation is more performant than pure go implementation, see buchgr#482 for actual benchmarks.
The cgo implementation may be more performant than the pure go implementation, see buchgr#482 for benchmarks.
The cgo implementation may be more performant than the pure go implementation, see buchgr#482 for benchmarks.
The cgo implementation may be more performant than the pure go implementation, see buchgr#482 for benchmarks.
When checking performance of bazel-remote noticed that most of the time is spent in zstd compression.
Created a benchmark that shows that cgo implementation of zstd is more performant (2x-3x compression, 4x-6x decompression) : https://github.com/glukasiknuro/go-zstd-benchmarks
I have created a proof of concept implementation using go zstd implementation of bazel-remote in 591894a, and created a load test that confirms the benefits of using cgo implementation are significant:
https://github.com/glukasiknuro/go-zstd-benchmarks/tree/master/bazel-remote-load-test
I will try to suggest a patch that will allow to select cgo implementation of zstd to bazel-remote if compiled in CGO mode.
Also, some more details about performance of cgo implementation in klauspost/compress#444
The text was updated successfully, but these errors were encountered: