*: switch from "compress/gzip" to more optimal library

One of the largest resource hogs in umoci is compression-related, and it
turns out[1] that Go's ordinary gzip implementation is nowhere near as
fast as other modern gzip implementations. In order to help our users
get more efficient builds, switch to a "pure-Go" implementation which
has x64-specific optimisations. We cannot use zlib wrappers like [2]
because of issues with "os/user" and static compilation (so we wouldn't
be able to release binaries).

This very simplified benchmark shows the positive difference when
switching libraries (using "tensorflow/tensorflow:latest" as the image
base):

  % time umoci unpack --image tensorflow:latest bundle # before
  39.03user 7.58system 0:45.16elapsed
  % time umoci unpack --image tensorflow:latest bundle # after
  40.89user 7.99system 0:34.89elapsed

But the real benefit is when it comes to compression and repacking (in
this benchmark the changes were installing FireFox and doing a repo
refresh):

  % time umoci repack --image tensorflow:latest bundle # before
  78.54user 13.71system 1:26.66elapsed
  % time umoci repack --image tensorflow:latest bundle # after
  51.14user 3.25system 0:30.30elapsed

That's almost 3x faster, just by having a more optimised compression
library!

[1]: golang/go#23154
[2]: https://github.com/vitessio/vitess/tree/v2.2/go/cgzip

Signed-off-by: Aleksa Sarai <asarai@suse.de>

CHANGELOG.md

@@ -27,6 +27,12 @@ and this project adheres to [Semantic Versioning](http://semver.org/).
   this ordering is not recommended by the spec -- nor is it required). This is
   an extension of openSUSE/umoci#229 that was missed during review.
   openSUSE/umoci#232
+- `umoci unpack` and `umoci repack` now make use of a far more optimised `gzip`
+  compression library. In some benchmarks this has resulted in `umoci repack`
+  speedups of up to 3x (though of course, you should do your own benchmarks).
+  `umoci unpack` unfortunately doesn't have as significant of a performance
+  improvement, due to the nature of `gzip` decompression (in future we may
+  switch to `zlib` wrappers). openSUSE/umoci#225 openSUSE/umoci#233
 
 ## [0.4.0] - 2018-03-10
 ### Added

mutate/mutate.go

@@ -23,11 +23,12 @@
 package mutate
 
 import (
-	"compress/gzip"
 	"io"
 	"reflect"
+	"runtime"
 	"time"
 
+	gzip "github.com/klauspost/pgzip"
 	"github.com/openSUSE/umoci/oci/cas"
 	"github.com/openSUSE/umoci/oci/casext"
 	"github.com/opencontainers/go-digest"
@@ -231,6 +232,9 @@ func (m *Mutator) add(ctx context.Context, reader io.Reader) (digest.Digest, int
 
 	gzw := gzip.NewWriter(pipeWriter)
 	defer gzw.Close()
+	if err := gzw.SetConcurrency(256<<10, 2*runtime.NumCPU()); err != nil {
+		return "", -1, errors.Wrapf(err, "set concurrency level to %v blocks", 2*runtime.NumCPU())
+	}
 	go func() {
 		_, err := io.Copy(gzw, hashReader)
 		if err != nil {

oci/layer/unpack.go

@@ -19,7 +19,6 @@ package layer
 
 import (
 	"archive/tar"
-	"compress/gzip"
 	// Import is necessary for go-digest.
 	_ "crypto/sha256"
 	"fmt"
@@ -32,6 +31,7 @@ import (
 	"time"
 
 	"github.com/apex/log"
+	gzip "github.com/klauspost/pgzip"
 	"github.com/openSUSE/umoci/oci/cas"
 	"github.com/openSUSE/umoci/oci/casext"
 	iconv "github.com/openSUSE/umoci/oci/config/convert"

vendor.conf

@@ -18,6 +18,10 @@ github.com/apex/log 941dea75d3ebfbdd905a5d8b7b232965c5e5c684
   github.com/mattn/go-colorable v0.0.9
   github.com/mattn/go-isatty v0.0.3
 github.com/golang/protobuf v1.1.0
+github.com/klauspost/pgzip v1.1
+  github.com/klauspost/compress v1.4.0
+  github.com/klauspost/crc32 v1.1
+  github.com/klauspost/cpuid v1.1
 
 # Only used for our tests.
 github.com/mohae/deepcopy 491d3605edfb866af34a48075bd4355ac1bf46ca