Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use faster gzip package #982

Merged
merged 2 commits into from
Sep 12, 2022
Merged

Conversation

klauspost
Copy link
Contributor

@klauspost klauspost commented Sep 6, 2022

Before:

BenchmarkCompression/gzip/compress-32                 81          13573535 ns/op           8.77 MB/s       10115 B/op          1 allocs/op
BenchmarkCompression/gzip/decompress-32              580           2023225 ns/op         959.10 MB/s        7563 B/op         53 allocs/op

  gzip:         93.86%

After:

BenchmarkCompression/gzip/compress-32                224           5340019 ns/op          24.42 MB/s        4838 B/op          1 allocs/op
BenchmarkCompression/gzip/decompress-32              692           1713771 ns/op        1132.28 MB/s         566 B/op         34 allocs/op

  gzip:         93.28%

Pretty typical scenario for default settings, 2x faster, sometimes at a slight compression loss. This provides a much better "default" trade-off than the stdlib.

For people regretting the 0.6% loss the higher compression levels can be used.

Edit: Next release will bring compression to 93.43% due to klauspost/compress#656

Decompression typically has even bigger margin, but it depends on the input.

It also provides much better handling of incompressible data:

BenchmarkCompression/gzip/compress-32                 36          31021311 ns/op          62.57 MB/s       22738 B/op          1 allocs/op
-> 
BenchmarkCompression/gzip/compress-32               3555            311620 ns/op        6227.58 MB/s          17 B/op          1 allocs/op

Before:
```
BenchmarkCompression/gzip/compress-32                 81          13573535 ns/op           8.77 MB/s       10115 B/op          1 allocs/op
BenchmarkCompression/gzip/decompress-32              580           2023225 ns/op         959.10 MB/s        7563 B/op         53 allocs/op

  gzip:         93.86%
```

After:

```
BenchmarkCompression/gzip/compress-32                224           5340019 ns/op          24.42 MB/s        4838 B/op          1 allocs/op
BenchmarkCompression/gzip/decompress-32              692           1713771 ns/op        1132.28 MB/s         566 B/op         34 allocs/op

  gzip:         93.28%
````

Pretty typical scenario for default settings, 2-3x faster, sometimes at a slight compression loss. This provides a much better "default" trade-off than the stdlib.

For people regretting the 0.6% loss the higher compression levels can be used.

Decompression typically has even bigger margin, but it depends on the input.
@dominicbarnes
Copy link
Contributor

@klauspost it looks like CircleCI had some sort of trouble executing the run. Oddly enough, we can't seem to get any of the UI to work for this build, would you mind throwing on an empty commit or something to try and trigger once again?

@klauspost
Copy link
Contributor Author

@dominicbarnes Tried an innocent change. Same result. Seems like a permission issue and you need to approve me.

@achille-roussel
Copy link
Contributor

I don't understand why CircleCI is bugging us on this PR (and don't have time to waste to figure it out), it runs fine for other external contributors.

I ran the tests locally and everything is looking good, let's get this merged 👍

Thanks for your contribution @klauspost !

@achille-roussel achille-roussel merged commit d4b89e7 into segmentio:main Sep 12, 2022
@klauspost klauspost deleted the faster-gzip-pkg branch September 13, 2022 07:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants