deflate: Faster/efficient low level encoders #105
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Refactor level 1+2 to skip more checks, but do intermediate hashes and extend matches backwards. Same or better compression at same or better speed.
Mostly avoiding bounds checks.
…riting' into deflate-skip-lower-levels
# Conflicts: # flate/deflate.go
Fix level 6 problem.
klauspost
changed the title
Closed
Instead add a token emitter that can emit arbitrary lengths.
cyphar
added a commit
to cyphar/umoci
that referenced
this pull request
Jun 18, 2020
We haven't done a dependency bump in quite a while, and there are a fair few fixes and improvements we should get into umoci before the next release. % go get -u go: github.com/apex/log upgrade => v1.4.0 go: github.com/cpuguy83/go-md2man/v2 upgrade => v2.0.0 go: github.com/golang/protobuf upgrade => v1.4.2 go: github.com/klauspost/compress upgrade => v1.10.9 go: github.com/klauspost/cpuid upgrade => v1.3.0 go: github.com/klauspost/pgzip upgrade => v1.2.4 go: github.com/konsorten/go-windows-terminal-sequences upgrade => v1.0.3 go: github.com/opencontainers/go-digest upgrade => v1.0.0 go: github.com/opencontainers/runtime-spec upgrade => v1.0.2 go: github.com/pkg/errors upgrade => v0.9.1 go: github.com/sirupsen/logrus upgrade => v1.6.0 go: github.com/vbatts/go-mtree upgrade => v0.5.0 go: golang.org/x/crypto upgrade => v0.0.0-20200604202706-70a84ac30bf9 go: golang.org/x/net upgrade => v0.0.0-20200602114024-627f9648deb9 go: golang.org/x/sys upgrade => v0.0.0-20200615200032-f1bc736245b1 go: google.golang.org/protobuf upgrade => v1.24.0 However there are two issues with this update: * We cannot update github.com/urfave/cli to anything later than v1.22.1 because of a bug when it comes to StringSliceFlag parsing that we hit in CI[1] -- hence the new excludes block. * Updating github.com/klauspost/compress to anything later than v1.8.6 causes us to generate different gzip-compressed blobs due to an optimisation in their compression[2]. Since this is generally a good change to have, we have to update our CI so that it works with the newest version (even if it's sub-optimal to generate different bytes between versions). [1]: urfave/cli#1152 [2]: klauspost/compress#105 Signed-off-by: Aleksa Sarai <asarai@suse.de>
cyphar
added a commit
to cyphar/umoci
that referenced
this pull request
Jun 18, 2020
We haven't done a dependency bump in quite a while, and there are a fair few fixes and improvements we should get into umoci before the next release. % go get -u go: github.com/apex/log upgrade => v1.4.0 go: github.com/cpuguy83/go-md2man/v2 upgrade => v2.0.0 go: github.com/golang/protobuf upgrade => v1.4.2 go: github.com/klauspost/compress upgrade => v1.10.9 go: github.com/klauspost/cpuid upgrade => v1.3.0 go: github.com/klauspost/pgzip upgrade => v1.2.4 go: github.com/konsorten/go-windows-terminal-sequences upgrade => v1.0.3 go: github.com/opencontainers/go-digest upgrade => v1.0.0 go: github.com/opencontainers/runtime-spec upgrade => v1.0.2 go: github.com/pkg/errors upgrade => v0.9.1 go: github.com/sirupsen/logrus upgrade => v1.6.0 go: github.com/vbatts/go-mtree upgrade => v0.5.0 go: golang.org/x/crypto upgrade => v0.0.0-20200604202706-70a84ac30bf9 go: golang.org/x/net upgrade => v0.0.0-20200602114024-627f9648deb9 go: golang.org/x/sys upgrade => v0.0.0-20200615200032-f1bc736245b1 go: google.golang.org/protobuf upgrade => v1.24.0 However there are two issues with this update: * We cannot update github.com/urfave/cli to anything later than v1.22.1 because of a bug when it comes to StringSliceFlag parsing that we hit in CI[1] -- hence the new excludes block. * Updating github.com/klauspost/compress to anything later than v1.8.6 causes us to generate different gzip-compressed blobs due to an optimisation in their compression[2]. Since this is generally a good change to have, we have to update our CI so that it works with the newest version (even if it's sub-optimal to generate different bytes between versions). [1]: urfave/cli#1152 [2]: klauspost/compress#105 Signed-off-by: Aleksa Sarai <asarai@suse.de>
cyphar
added a commit
to cyphar/umoci
that referenced
this pull request
Jun 18, 2020
We haven't done a dependency bump in quite a while, and there are a fair few fixes and improvements we should get into umoci before the next release. % go get -u go: github.com/apex/log upgrade => v1.4.0 go: github.com/cpuguy83/go-md2man/v2 upgrade => v2.0.0 go: github.com/golang/protobuf upgrade => v1.4.2 go: github.com/klauspost/compress upgrade => v1.10.9 go: github.com/klauspost/cpuid upgrade => v1.3.0 go: github.com/klauspost/pgzip upgrade => v1.2.4 go: github.com/konsorten/go-windows-terminal-sequences upgrade => v1.0.3 go: github.com/opencontainers/go-digest upgrade => v1.0.0 go: github.com/opencontainers/runtime-spec upgrade => v1.0.2 go: github.com/pkg/errors upgrade => v0.9.1 go: github.com/sirupsen/logrus upgrade => v1.6.0 go: github.com/vbatts/go-mtree upgrade => v0.5.0 go: golang.org/x/crypto upgrade => v0.0.0-20200604202706-70a84ac30bf9 go: golang.org/x/net upgrade => v0.0.0-20200602114024-627f9648deb9 go: golang.org/x/sys upgrade => v0.0.0-20200615200032-f1bc736245b1 go: google.golang.org/protobuf upgrade => v1.24.0 However there are two issues with this update: * We cannot update github.com/urfave/cli to anything later than v1.22.1 because of a bug when it comes to StringSliceFlag parsing that we hit in CI[1] -- hence the new excludes block. * Updating github.com/klauspost/compress to anything later than v1.8.6 causes us to generate different gzip-compressed blobs due to an optimisation in their compression[2]. Since this is generally a good change to have, we have to update our CI so that it works with the newest version (even if it's sub-optimal to generate different bytes between versions). [1]: urfave/cli#1152 [2]: klauspost/compress#105 Signed-off-by: Aleksa Sarai <asarai@suse.de>
cyphar
added a commit
to cyphar/umoci
that referenced
this pull request
Jun 18, 2020
We haven't done a dependency bump in quite a while, and there are a fair few fixes and improvements we should get into umoci before the next release. % go get -u go: github.com/apex/log upgrade => v1.4.0 go: github.com/cpuguy83/go-md2man/v2 upgrade => v2.0.0 go: github.com/golang/protobuf upgrade => v1.4.2 go: github.com/klauspost/compress upgrade => v1.10.9 go: github.com/klauspost/cpuid upgrade => v1.3.0 go: github.com/klauspost/pgzip upgrade => v1.2.4 go: github.com/konsorten/go-windows-terminal-sequences upgrade => v1.0.3 go: github.com/opencontainers/go-digest upgrade => v1.0.0 go: github.com/opencontainers/runtime-spec upgrade => v1.0.2 go: github.com/pkg/errors upgrade => v0.9.1 go: github.com/sirupsen/logrus upgrade => v1.6.0 go: github.com/vbatts/go-mtree upgrade => v0.5.0 go: golang.org/x/crypto upgrade => v0.0.0-20200604202706-70a84ac30bf9 go: golang.org/x/net upgrade => v0.0.0-20200602114024-627f9648deb9 go: golang.org/x/sys upgrade => v0.0.0-20200615200032-f1bc736245b1 go: google.golang.org/protobuf upgrade => v1.24.0 However there are two issues with this update: * We cannot update github.com/urfave/cli to anything later than v1.22.1 because of a bug when it comes to StringSliceFlag parsing that we hit in CI[1] -- hence the new excludes block. * Updating github.com/klauspost/compress to anything later than v1.8.6 causes us to generate different gzip-compressed blobs due to an optimisation in their compression[2]. Since this is generally a good change to have, we have to update our CI so that it works with the newest version (even if it's sub-optimal to generate different bytes between versions). [1]: urfave/cli#1152 [2]: klauspost/compress#105 Signed-off-by: Aleksa Sarai <asarai@suse.de>
cyphar
added a commit
to cyphar/umoci
that referenced
this pull request
Jun 19, 2020
We haven't done a dependency bump in quite a while, and there are a fair few fixes and improvements we should get into umoci before the next release. % go get -u go: github.com/apex/log upgrade => v1.4.0 go: github.com/cpuguy83/go-md2man/v2 upgrade => v2.0.0 go: github.com/golang/protobuf upgrade => v1.4.2 go: github.com/klauspost/compress upgrade => v1.10.9 go: github.com/klauspost/cpuid upgrade => v1.3.0 go: github.com/klauspost/pgzip upgrade => v1.2.4 go: github.com/konsorten/go-windows-terminal-sequences upgrade => v1.0.3 go: github.com/opencontainers/go-digest upgrade => v1.0.0 go: github.com/opencontainers/runtime-spec upgrade => v1.0.2 go: github.com/pkg/errors upgrade => v0.9.1 go: github.com/sirupsen/logrus upgrade => v1.6.0 go: github.com/vbatts/go-mtree upgrade => v0.5.0 go: golang.org/x/crypto upgrade => v0.0.0-20200604202706-70a84ac30bf9 go: golang.org/x/net upgrade => v0.0.0-20200602114024-627f9648deb9 go: golang.org/x/sys upgrade => v0.0.0-20200615200032-f1bc736245b1 go: google.golang.org/protobuf upgrade => v1.24.0 However there are two issues with this update: * We cannot update github.com/urfave/cli to anything later than v1.22.1 because of a bug when it comes to StringSliceFlag parsing that we hit in CI[1] -- hence the new excludes block. * Updating github.com/klauspost/compress to anything later than v1.8.6 causes us to generate different gzip-compressed blobs due to an optimisation in their compression[2]. Since this is generally a good change to have, we have to update our CI so that it works with the newest version (even if it's sub-optimal to generate different bytes between versions). [1]: urfave/cli#1152 [2]: klauspost/compress#105 Signed-off-by: Aleksa Sarai <asarai@suse.de>
cyphar
added a commit
to cyphar/umoci
that referenced
this pull request
Jun 19, 2020
We haven't done a dependency bump in quite a while, and there are a fair few fixes and improvements we should get into umoci before the next release. % go get -u go: github.com/apex/log upgrade => v1.4.0 go: github.com/cpuguy83/go-md2man/v2 upgrade => v2.0.0 go: github.com/golang/protobuf upgrade => v1.4.2 go: github.com/klauspost/compress upgrade => v1.10.9 go: github.com/klauspost/cpuid upgrade => v1.3.0 go: github.com/klauspost/pgzip upgrade => v1.2.4 go: github.com/konsorten/go-windows-terminal-sequences upgrade => v1.0.3 go: github.com/opencontainers/go-digest upgrade => v1.0.0 go: github.com/opencontainers/runtime-spec upgrade => v1.0.2 go: github.com/pkg/errors upgrade => v0.9.1 go: github.com/sirupsen/logrus upgrade => v1.6.0 go: github.com/vbatts/go-mtree upgrade => v0.5.0 go: golang.org/x/crypto upgrade => v0.0.0-20200604202706-70a84ac30bf9 go: golang.org/x/net upgrade => v0.0.0-20200602114024-627f9648deb9 go: golang.org/x/sys upgrade => v0.0.0-20200615200032-f1bc736245b1 go: google.golang.org/protobuf upgrade => v1.24.0 However there are two issues with this update: * We cannot update github.com/urfave/cli to anything later than v1.22.1 because of a bug when it comes to StringSliceFlag parsing that we hit in CI[1] -- hence the new excludes block. * Updating github.com/klauspost/compress to anything later than v1.8.6 causes us to generate different gzip-compressed blobs due to an optimisation in their compression[2]. Since this is generally a good change to have, we have to update our CI so that it works with the newest version (even if it's sub-optimal to generate different bytes between versions). [1]: urfave/cli#1152 [2]: klauspost/compress#105 Signed-off-by: Aleksa Sarai <asarai@suse.de>
cyphar
added a commit
to cyphar/umoci
that referenced
this pull request
Jun 19, 2020
We haven't done a dependency bump in quite a while, and there are a fair few fixes and improvements we should get into umoci before the next release. % go get -u go: github.com/apex/log upgrade => v1.4.0 go: github.com/cpuguy83/go-md2man/v2 upgrade => v2.0.0 go: github.com/golang/protobuf upgrade => v1.4.2 go: github.com/klauspost/compress upgrade => v1.10.9 go: github.com/klauspost/cpuid upgrade => v1.3.0 go: github.com/klauspost/pgzip upgrade => v1.2.4 go: github.com/konsorten/go-windows-terminal-sequences upgrade => v1.0.3 go: github.com/opencontainers/go-digest upgrade => v1.0.0 go: github.com/opencontainers/runtime-spec upgrade => v1.0.2 go: github.com/pkg/errors upgrade => v0.9.1 go: github.com/sirupsen/logrus upgrade => v1.6.0 go: github.com/vbatts/go-mtree upgrade => v0.5.0 go: golang.org/x/crypto upgrade => v0.0.0-20200604202706-70a84ac30bf9 go: golang.org/x/net upgrade => v0.0.0-20200602114024-627f9648deb9 go: golang.org/x/sys upgrade => v0.0.0-20200615200032-f1bc736245b1 go: google.golang.org/protobuf upgrade => v1.24.0 However there are two issues with this update: * We cannot update github.com/urfave/cli to anything later than v1.22.1 because of a bug when it comes to StringSliceFlag parsing that we hit in CI[1] -- hence the new excludes block. * Updating github.com/klauspost/compress to anything later than v1.8.6 causes us to generate different gzip-compressed blobs due to an optimisation in their compression[2]. Since this is generally a good change to have, we have to update our CI so that it works with the newest version (even if it's sub-optimal to generate different bytes between versions). [1]: urfave/cli#1152 [2]: klauspost/compress#105 Signed-off-by: Aleksa Sarai <asarai@suse.de>
cyphar
added a commit
to cyphar/umoci
that referenced
this pull request
Jun 19, 2020
We haven't done a dependency bump in quite a while, and there are a fair few fixes and improvements we should get into umoci before the next release. % go get -u go: github.com/apex/log upgrade => v1.4.0 go: github.com/cpuguy83/go-md2man/v2 upgrade => v2.0.0 go: github.com/golang/protobuf upgrade => v1.4.2 go: github.com/klauspost/compress upgrade => v1.10.9 go: github.com/klauspost/cpuid upgrade => v1.3.0 go: github.com/klauspost/pgzip upgrade => v1.2.4 go: github.com/konsorten/go-windows-terminal-sequences upgrade => v1.0.3 go: github.com/opencontainers/go-digest upgrade => v1.0.0 go: github.com/opencontainers/runtime-spec upgrade => v1.0.2 go: github.com/pkg/errors upgrade => v0.9.1 go: github.com/sirupsen/logrus upgrade => v1.6.0 go: github.com/vbatts/go-mtree upgrade => v0.5.0 go: golang.org/x/crypto upgrade => v0.0.0-20200604202706-70a84ac30bf9 go: golang.org/x/net upgrade => v0.0.0-20200602114024-627f9648deb9 go: golang.org/x/sys upgrade => v0.0.0-20200615200032-f1bc736245b1 go: google.golang.org/protobuf upgrade => v1.24.0 However there are three issues with this update: * We cannot update github.com/urfave/cli to anything later than v1.22.1 because of a bug when it comes to StringSliceFlag parsing that we hit in CI[1] -- hence the new excludes block. * Updating github.com/klauspost/compress to anything later than v1.8.6 causes us to generate different gzip-compressed blobs due to an optimisation in their compression[2]. Since this is generally a good change to have, we have to update our CI so that it works with the newest version (even if it's sub-optimal to generate different bytes between versions). * Updating github.com/cpuguy83/go-md2man to v2 caused issues with our "go get" invocation. It turns out we were silently adding go-md2man (v1) to our go.mod file each time we ran a Travis build, and the switch to v2 uncovered this issue. This is easily fixed by setting GO111MODULE=off when doing 'go get' in .travis.yml. [1]: urfave/cli#1152 [2]: klauspost/compress#105 Signed-off-by: Aleksa Sarai <asarai@suse.de> asd Signed-off-by: Aleksa Sarai <asarai@suse.de>
cyphar
added a commit
to cyphar/umoci
that referenced
this pull request
Jun 19, 2020
We haven't done a dependency bump in quite a while, and there are a fair few fixes and improvements we should get into umoci before the next release. % go get -u go: github.com/apex/log upgrade => v1.4.0 go: github.com/cpuguy83/go-md2man/v2 upgrade => v2.0.0 go: github.com/golang/protobuf upgrade => v1.4.2 go: github.com/klauspost/compress upgrade => v1.10.9 go: github.com/klauspost/cpuid upgrade => v1.3.0 go: github.com/klauspost/pgzip upgrade => v1.2.4 go: github.com/konsorten/go-windows-terminal-sequences upgrade => v1.0.3 go: github.com/opencontainers/go-digest upgrade => v1.0.0 go: github.com/opencontainers/runtime-spec upgrade => v1.0.2 go: github.com/pkg/errors upgrade => v0.9.1 go: github.com/sirupsen/logrus upgrade => v1.6.0 go: github.com/vbatts/go-mtree upgrade => v0.5.0 go: golang.org/x/crypto upgrade => v0.0.0-20200604202706-70a84ac30bf9 go: golang.org/x/net upgrade => v0.0.0-20200602114024-627f9648deb9 go: golang.org/x/sys upgrade => v0.0.0-20200615200032-f1bc736245b1 go: google.golang.org/protobuf upgrade => v1.24.0 However there are three issues with this update: * We cannot update github.com/urfave/cli to anything later than v1.22.1 because of a bug when it comes to StringSliceFlag parsing that we hit in CI[1] -- hence the new excludes block. * Updating github.com/klauspost/compress to anything later than v1.8.6 causes us to generate different gzip-compressed blobs due to an optimisation in their compression[2]. Since this is generally a good change to have, we have to update our CI so that it works with the newest version (even if it's sub-optimal to generate different bytes between versions). * Updating github.com/cpuguy83/go-md2man to v2 caused issues with our "go get" invocation. It turns out we were silently adding go-md2man (v1) to our go.mod file each time we ran a Travis build, and the switch to v2 uncovered this issue. This is easily fixed by setting GO111MODULE=off when doing 'go get' in .travis.yml. [1]: urfave/cli#1152 [2]: klauspost/compress#105 Signed-off-by: Aleksa Sarai <asarai@suse.de>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Refactor level 1-6 to skip more checks, but do intermediate hashes and extend matches backwards.
Added possibility to reuse Huffman tables across blocks.
Same or better compression at same or better speed. Removes assembler since it provides little gain.
Decompression should also be faster by this change.
Comparisons before / after / standard library:
Single files: