Support zstd (non-chunked) conversion #2530
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ktock
reviewed
Sep 24, 2023
pkg/imgutil/converter/zstd.go
Outdated
| } | ||
|
|
||
| pr, pw := io.Pipe() | ||
| enc, err := zstd.NewWriter(pw, zstd.WithEncoderLevel(zstd.EncoderLevel(options.ZstdCompressionLevel))) |
There was a problem hiding this comment.
Can we use EncoderLevelFromZstd? https://pkg.go.dev/github.com/klauspost/compress/zstd#EncoderLevelFromZstd
ktock
reviewed
Sep 25, 2023
Comment on lines
+63
to
+66
| // #region zstd flags | ||
| imageConvertCommand.Flags().Bool("zstd", false, "Convert legacy tar(.gz) layers to zstd. Should be used in conjunction with '--oci'") | ||
| imageConvertCommand.Flags().Int("zstd-compression-level", 3, "zstd compression level") | ||
| // #endregion |
There was a problem hiding this comment.
Could you add docs to /docs/command-reference.md?
ktock
approved these changes
Sep 25, 2023
docs/command-reference.md
Outdated
| @@ -878,6 +878,8 @@ Flags: | |||
| - `--estargz-min-chunk-size=<SIZE>` : The minimal number of bytes of data must be written in one gzip stream (requires stargz-snapshotter >= v0.13.0). Useful for creating a smaller eStargz image (refer to [`./stargz.md`](./stargz.md) for details). | |||
| - `--estargz-external-toc` : Separate TOC JSON into another image (called \"TOC image\"). The name of TOC image is the original + \"-esgztoc\" suffix. Both eStargz and the TOC image should be pushed to the same registry. (requires stargz-snapshotter >= v0.13.0) Useful for creating a smaller eStargz image (refer to [`./stargz.md`](./stargz.md) for details). :warning: This flag is experimental and subject to change. | |||
| - `--estargz-keep-diff-id`: Convert to esgz without changing diffID (cannot be used in conjunction with '--estargz-record-in'. must be specified with '--estargz-external-toc') | |||
| - `--zstd` : Use zstd compression instead of gzip (a.k.a zstd). Should be used in conjunction with '--oci' | |||
Signed-off-by: Henry Wang <henwang@amazon.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds non-chunked zstd image conversion to
nerdctl.nerdctltoday already supports converting images tozstdchunkedformat for lazy-loading. Traditional non-chunkedzstdformat is usually smaller than the chunked equivalent, and faster to convert. This is desirable for the non lazy-loading scenarios.The following is the image size comparison between gzip, zstd and ztdchunked formats. Both
zstdandzstdchunkedused the default level 3 compression level.