Skip to content

Commit

Permalink
pkg/zstdchunked: new zstd variant zstd:chunked
Browse files Browse the repository at this point in the history
add a new custom variant of the zstd compression that permits to
retrieve each file separately.

The idea is based on CRFS and its stargz format for having seekable
and indexable tarballs.

One disadvantage of the stargz format is that a custom file is added
to the tarball to store the layer metadata, and the metadata file is
part of the image itself.  Clients that are not aware of the stargz
format will propagate the metadata file inside of the containers.

The zstd compression supports embeddeding additional data as part of
the stream that the zstd decompressor will ignore (skippable frame),
so the issue above with CRFS can be solved directly within the zstd
compression format.

Beside this minor advantage, zstd is much faster and compresses better
than gzip, so take this opportunity to push the zstd format further.

The zstd compression is supported by the OCI image specs since August
2019: opencontainers/image-spec#788 and has
been supported by containers/image since then.

Clients that are not aware of the zstd:chunked format, won't notice
any difference when handling a blob that uses the variant.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
  • Loading branch information
giuseppe committed Dec 1, 2020
1 parent a3a28d1 commit fe91364
Show file tree
Hide file tree
Showing 11 changed files with 1,695 additions and 0 deletions.
1 change: 1 addition & 0 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ require (
github.com/BurntSushi/toml v0.3.1
github.com/Microsoft/go-winio v0.4.15
github.com/Microsoft/hcsshim v0.8.9
github.com/cyphar/filepath-securejoin v0.2.2
github.com/docker/go-units v0.4.0
github.com/hashicorp/go-multierror v1.1.0
github.com/klauspost/compress v1.11.3
Expand Down
Loading

0 comments on commit fe91364

Please sign in to comment.