-
Notifications
You must be signed in to change notification settings - Fork 527
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for zstd compression #1487
Add support for zstd compression #1487
Conversation
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
40ece4f
to
8bac013
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wow, thanks for this! I've just taken a quick look so far but it looks great to me.
It looks like there are some conflicts preventing CI from running. Can you rebase and we can see what the tests have to say? |
Squashed commits: [6563fc8] Run ./hack/update-codegen.sh [c86a013] Fix linting issues [353dd69] Remove redundant break statements [a06b4cc] Fix some comments [fb0678f] Expose compression algorithms enum [3321d0c] Update mediaType according to compression for tarball layers (cherry picked from commit 56d4df5) [5124d2c] Fix unit tests (cherry picked from commit 7ce08d1) [2958f30] Optimize compression detection (cherry picked from commit 0a3437f) [c211a61] Add support for zstd-compressedd layers (cherry picked from commit d8fb0f3) [0b9a996] Add zstd compression utilities (cherry picked from commit c645201) [7dd71ab] Add layer types with zstd compression (cherry picked from commit 328a14b) Signed-off-by: Lavrenti Frobeen <lavrenti@northflank.com>
8bac013
to
879537e
Compare
Thanks for the feedback @imjasonh. I've removed the |
That seems sufficient honestly. If this is something we expect to expose via a flag in cc @jonjohnsonjr in case he has thoughts |
Codecov Report
@@ Coverage Diff @@
## main #1487 +/- ##
==========================================
- Coverage 73.19% 73.12% -0.08%
==========================================
Files 115 117 +2
Lines 8812 8940 +128
==========================================
+ Hits 6450 6537 +87
- Misses 1712 1743 +31
- Partials 650 660 +10
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. |
pkg/v1/tarball/layer.go
Outdated
if compression == ZStd { | ||
l.mediaType = types.OCILayerZStd | ||
} else { | ||
l.mediaType = types.DockerLayer |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would expect WithCompression(None)
to set l.mediaType
to DockerUncompressedLayer
and produce uncompressed layers. WDYT?
I am also OK with punting on this for now, returning an error or documenting that None is invalid.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That makes sense, but I'm not quite sure how easy this would be to implement. My understanding is that the decision whether layers are produced with or without compression is made by consumers of the Layer interface when they access the data via .Compressed()
or .Uncompressed()
. Hence I'd have to modify all the code sites that use .Compressed()
to make WithCompression(None)
work, right?
I was thinking about alternatively just returning uncompressed data from the .Compressed()
method when WithCompression(None)
is set. But I felt that would conflict with semantics of the .Compressed()
method which may be a bit confusing. Then again, this PR assumes the consumers of the .Compressed()
method are agnostic to the compression format anyway, so one could argue that no compression can still be regarded "some kind of very inefficient compression format".
If you agree with this idea, I would implement it that way. Otherwise, I think the best option might be to drop WithCompression(None)
altogether for now.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
But I felt that would conflict with semantics of the .Compressed() method which may be a bit confusing.
Yeah that's fair. Having both the Compressed
and Uncompressed
methods is a design mistake. It would be better if we deprecated Uncompressed
and renamed Compressed
, but it's too late for that. I am fine with dropping this for now, but I think we want to return an error if we don't want to support it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I modified the WithCompression
method to ignore calls the compression.None
argument and use gzip
instead. Additionally it will print this warning:
Compression type 'none' is not supported for tarball layers; using gzip compression.
pkg/v1/tarball/layer.go
Outdated
if compression == ZStd { | ||
l.mediaType = types.OCILayerZStd | ||
} else { | ||
l.mediaType = types.DockerLayer |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
But I felt that would conflict with semantics of the .Compressed() method which may be a bit confusing.
Yeah that's fair. Having both the Compressed
and Uncompressed
methods is a design mistake. It would be better if we deprecated Uncompressed
and renamed Compressed
, but it's too late for that. I am fine with dropping this for now, but I think we want to return an error if we don't want to support it.
pkg/v1/tarball/layer.go
Outdated
func WithCompression(compression LayerCompression) LayerOption { | ||
return func(l *layer) { | ||
if compression == ZStd { | ||
l.mediaType = types.OCILayerZStd |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This conflicts a bit with layer.WithMediaType
. I might just drop the else
here? We could also warn or error or document this fact.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for all the feedback so far.
I removed the mediaType override for both zstd and gzip in order to be consistent.
If WithCompression(compression.ZStd)
is used, then the user should also set WithMediaType(types.OCILayerZStd)
. I've updated the documentation for WithCompression
to reflect this. When the layer is created via LayerFromOpener
, the method will check whether the layer's mediaType
matches its compression
and print a warning if not. What do you think about this approach?
…" compression for tarball layer
859aa0c
to
7c8c437
Compare
I've extended the unit tests so that they cover the new zstd logic. I noticed that the presubmit check fails for some reason, but I can't seem to figure out what the issue is. Based on the logs from one of the recent runs I thought that I'd have to run Apart from getting the presubmit check to pass, is there anything else that needs to be done before this can be merged? |
There are a handful of submodules here (sorry) -- you might need to
|
Oh yes, I totally overlooked those. I ran |
@imjasonh @jonjohnsonjr Sorry for being annoying, but could you re-trigger the tests please? I'd really like to get this merged so that I can add zstd support to kaniko next. |
Not annoying at all! |
Awesome, thanks for triggering the tests! It looks like all checks have passed this time. Can we merge the PR now @imjasonh? |
Thanks for sticking with this -- I'd like to play with it a bit before cutting a release, so I might break the API in small ways, but it looks pretty close to what we want. |
… and compression level We want to make the layer compression in kaniko configurable, so we have added two optional command line arguments “--compression” and “--compression-level”. The former allows the user to specify a compression algorithm (zstd, gzip or none) and the latter can be used to specify the compression level. Depending on the selected compression algorithm and level we modify the set of layerOptions that are used to create tarball layers in `push.go` and `build.go`. The actual implementation of the zstd support can be found in our fork of the go-containerregistry package for which we have filed this PR: google/go-containerregistry#1487 The changes should be fully backwards compatible. (cherry picked from commit 7c8b681)
…ompression level We want to make the layer compression in kaniko configurable, so we have added two optional command line arguments “--compression” and “--compression-level”. The former allows the user to specify a compression algorithm (zstd, gzip) and the latter can be used to specify the compression level. Depending on the selected compression algorithm and level we modify the set of layerOptions that are used to create tarball layers in `push.go` and `build.go`. The actual implementation of the zstd support can be found in our fork of the go-containerregistry package for which we have filed this PR: google/go-containerregistry#1487 The changes should be fully backwards compatible.
…ompression level We want to make the layer compression in kaniko configurable, so we have added two optional command line arguments “--compression” and “--compression-level”. The former allows the user to specify a compression algorithm (zstd, gzip) and the latter can be used to specify the compression level. Depending on the selected compression algorithm and level we modify the set of layerOptions that are used to create tarball layers in `push.go` and `build.go`. The actual implementation of the zstd support can be found in our fork of the go-containerregistry package for which we have filed this PR: google/go-containerregistry#1487 The changes should be fully backwards compatible.
…ompression level We want to make the layer compression in kaniko configurable, so we have added two optional command line arguments “--compression” and “--compression-level”. The former allows the user to specify a compression algorithm (zstd, gzip) and the latter can be used to specify the compression level. Depending on the selected compression algorithm and level we modify the set of layerOptions that are used to create tarball layers in `push.go` and `build.go`. The actual implementation of the zstd support can be found in our fork of the go-containerregistry package for which we have filed this PR: google/go-containerregistry#1487 The changes should be fully backwards compatible.
…ompression level We want to make the layer compression in kaniko configurable, so we have added two optional command line arguments “--compression” and “--compression-level”. The former allows the user to specify a compression algorithm (zstd, gzip) and the latter can be used to specify the compression level. Depending on the selected compression algorithm and level we modify the set of layerOptions that are used to create tarball layers in `push.go` and `build.go`. The actual implementation of the zstd support can be found in our fork of the go-containerregistry package for which we have filed this PR: google/go-containerregistry#1487 The changes should be fully backwards compatible.
* Add support for configurable compression algorithm (gzip, zstd) and compression level We want to make the layer compression in kaniko configurable, so we have added two optional command line arguments “--compression” and “--compression-level”. The former allows the user to specify a compression algorithm (zstd, gzip) and the latter can be used to specify the compression level. Depending on the selected compression algorithm and level we modify the set of layerOptions that are used to create tarball layers in `push.go` and `build.go`. The actual implementation of the zstd support can be found in our fork of the go-containerregistry package for which we have filed this PR: google/go-containerregistry#1487 The changes should be fully backwards compatible. * Restrict inputs for compression flag to gzip and zstd This change will ensure that users can only specify supported compression algorithms (`zstd`, `gzip`) to the `--compression` flag. * Fix incorrect type for switch statements on config.Compression
[![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [github.com/google/go-containerregistry](https://github.com/google/go-containerregistry) | require | minor | `v0.11.0` -> `v0.15.2` | --- ### Release Notes <details> <summary>google/go-containerregistry</summary> ### [`v0.15.2`](https://github.com/google/go-containerregistry/releases/tag/v0.15.2) [Compare Source](https://github.com/google/go-containerregistry/compare/v0.15.1...v0.15.2) #### What's Changed - Make 403 non-fatal for manifest existence checks by [@​jonjohnsonjr](https://github.com/jonjohnsonjr) in [google/go-containerregistry#1691 - Do not reuse pushers for pullers by [@​jonjohnsonjr](https://github.com/jonjohnsonjr) in [google/go-containerregistry#1701 **Full Changelog**: google/go-containerregistry@v0.15.1...v0.15.2 ### [`v0.15.1`](https://github.com/google/go-containerregistry/releases/tag/v0.15.1) [Compare Source](https://github.com/google/go-containerregistry/compare/v0.15.0...v0.15.1) #### Changelog - [`e2620e5`](https://github.com/google/go-containerregistry/commit/e2620e56) Actually retry retryable status codes ([#​1618](https://github.com/google/go-containerregistry/issues/1618)) - [`afd15f1`](https://github.com/google/go-containerregistry/commit/afd15f14) Add --all-tags flag to crane cp ([#​1682](https://github.com/google/go-containerregistry/issues/1682)) - [`69d1a19`](https://github.com/google/go-containerregistry/commit/69d1a197) Add `mutate` `--ports` option to set the exposed ports ([#​1677](https://github.com/google/go-containerregistry/issues/1677)) - [`65e78dc`](https://github.com/google/go-containerregistry/commit/65e78dc8) Add partial.Manifests for lazy index access ([#​1631](https://github.com/google/go-containerregistry/issues/1631)) - [`3228a60`](https://github.com/google/go-containerregistry/commit/3228a601) Add ppc64le to .goreleaser.yml ([#​1680](https://github.com/google/go-containerregistry/issues/1680)) - [`0b12f56`](https://github.com/google/go-containerregistry/commit/0b12f56b) Add ppc64le to all binaries ([#​1688](https://github.com/google/go-containerregistry/issues/1688)) - [`d958444`](https://github.com/google/go-containerregistry/commit/d9584448) Add remote.Descriptor.Schema1() ([#​1626](https://github.com/google/go-containerregistry/issues/1626)) - [`07c767c`](https://github.com/google/go-containerregistry/commit/07c767c7) Add remote.Puller ([#​1644](https://github.com/google/go-containerregistry/issues/1644)) - [`005bb71`](https://github.com/google/go-containerregistry/commit/005bb719) Add remote.Reuse for Pusher/Puller ([#​1672](https://github.com/google/go-containerregistry/issues/1672)) - [`21ac1b2`](https://github.com/google/go-containerregistry/commit/21ac1b24) Adding `mutate` `--workdir` option to set the working directory ([#​1615](https://github.com/google/go-containerregistry/issues/1615)) - [`0962e29`](https://github.com/google/go-containerregistry/commit/0962e296) Allow remote config layers to be lazy fetched ([#​1634](https://github.com/google/go-containerregistry/issues/1634)) - [`53189d3`](https://github.com/google/go-containerregistry/commit/53189d33) Bump actions/setup-go from 3 to 4 ([#​1602](https://github.com/google/go-containerregistry/issues/1602)) - [`54e3f49`](https://github.com/google/go-containerregistry/commit/54e3f49e) Bump actions/stale from 7 to 8 ([#​1616](https://github.com/google/go-containerregistry/issues/1616)) - [`07eb440`](https://github.com/google/go-containerregistry/commit/07eb440c) Bump codecov/codecov-action from 3.1.1 to 3.1.2 ([#​1650](https://github.com/google/go-containerregistry/issues/1650)) - [`58bd35b`](https://github.com/google/go-containerregistry/commit/58bd35bc) Bump codecov/codecov-action from 3.1.2 to 3.1.3 ([#​1668](https://github.com/google/go-containerregistry/issues/1668)) - [`e055961`](https://github.com/google/go-containerregistry/commit/e055961a) Bump peter-evans/create-pull-request from 4 to 5 ([#​1642](https://github.com/google/go-containerregistry/issues/1642)) - [`b8d1c0a`](https://github.com/google/go-containerregistry/commit/b8d1c0a1) Bump slsa-framework/slsa-verifier from 2.0.1 to 2.1.0 ([#​1621](https://github.com/google/go-containerregistry/issues/1621)) - [`375fb61`](https://github.com/google/go-containerregistry/commit/375fb61c) Bump slsa-framework/slsa-verifier from 2.1.0 to 2.2.0 ([#​1649](https://github.com/google/go-containerregistry/issues/1649)) - [`9aa45a1`](https://github.com/google/go-containerregistry/commit/9aa45a1a) Change return type of remote.Referrers ([#​1652](https://github.com/google/go-containerregistry/issues/1652)) - [`2ccd41c`](https://github.com/google/go-containerregistry/commit/2ccd41c4) Cleanup: Switch the debug image to `cgr.dev/chainguard/busybox` ([#​1638](https://github.com/google/go-containerregistry/issues/1638)) - [`93be9c4`](https://github.com/google/go-containerregistry/commit/93be9c42) Don't export whiteouts for single layers ([#​1629](https://github.com/google/go-containerregistry/issues/1629)) - [`b7c6e9d`](https://github.com/google/go-containerregistry/commit/b7c6e9dc) Fall back to puller if reusing pusher fails ([#​1676](https://github.com/google/go-containerregistry/issues/1676)) - [`370e8a5`](https://github.com/google/go-containerregistry/commit/370e8a5f) Fix a few lint issues ([#​1684](https://github.com/google/go-containerregistry/issues/1684)) - [`bc990d6`](https://github.com/google/go-containerregistry/commit/bc990d6b) Fix fetching referrers error handling ([#​1648](https://github.com/google/go-containerregistry/issues/1648)) - [`aee00b1`](https://github.com/google/go-containerregistry/commit/aee00b15) Fix race in mutate ([#​1627](https://github.com/google/go-containerregistry/issues/1627)) - [`4a79e94`](https://github.com/google/go-containerregistry/commit/4a79e94b) Fix race in stream ([#​1632](https://github.com/google/go-containerregistry/issues/1632)) - [`6743ec9`](https://github.com/google/go-containerregistry/commit/6743ec96) Implement remote.Pusher ([#​1633](https://github.com/google/go-containerregistry/issues/1633)) - [`3120ba5`](https://github.com/google/go-containerregistry/commit/3120ba58) Keep order of env in crane mutate ([#​1683](https://github.com/google/go-containerregistry/issues/1683)) - [`6ac92e8`](https://github.com/google/go-containerregistry/commit/6ac92e8a) Refactor fetcher, writer, and progress ([#​1625](https://github.com/google/go-containerregistry/issues/1625)) - [`249d7e1`](https://github.com/google/go-containerregistry/commit/249d7e1f) Refresh authn.DefaultKeychain creds every 5 min ([#​1624](https://github.com/google/go-containerregistry/issues/1624)) - [`ad695c0`](https://github.com/google/go-containerregistry/commit/ad695c04) Replace deprecated command with environment file ([#​1667](https://github.com/google/go-containerregistry/issues/1667)) - [`ed5c185`](https://github.com/google/go-containerregistry/commit/ed5c185d) Retry net.ErrClosed ([#​1637](https://github.com/google/go-containerregistry/issues/1637)) - [`43710a9`](https://github.com/google/go-containerregistry/commit/43710a91) Retry without mount if auth fails ([#​1681](https://github.com/google/go-containerregistry/issues/1681)) - [`2435320`](https://github.com/google/go-containerregistry/commit/24353209) Revert "Cleanup: Switch the debug image to `cgr.dev/chainguard/busybox` ([#​1638](https://github.com/google/go-containerregistry/issues/1638))" ([#​1641](https://github.com/google/go-containerregistry/issues/1641)) - [`27a6ad6`](https://github.com/google/go-containerregistry/commit/27a6ad62) Revert "authn: Add NewConfigKeychain to load a config from explicit path ([#​1603](https://github.com/google/go-containerregistry/issues/1603))" ([#​1664](https://github.com/google/go-containerregistry/issues/1664)) - [`a34235c`](https://github.com/google/go-containerregistry/commit/a34235c8) Support `Warning` header aggregation and reporting in `crane` ([#​1604](https://github.com/google/go-containerregistry/issues/1604)) - [`01bbd53`](https://github.com/google/go-containerregistry/commit/01bbd539) Update release.yml ([#​1601](https://github.com/google/go-containerregistry/issues/1601)) - [`df518f9`](https://github.com/google/go-containerregistry/commit/df518f95) add crane auth logout ([#​1589](https://github.com/google/go-containerregistry/issues/1589)) - [`5438948`](https://github.com/google/go-containerregistry/commit/54389485) add registry.Repo("foo") ([#​1671](https://github.com/google/go-containerregistry/issues/1671)) - [`3706061`](https://github.com/google/go-containerregistry/commit/3706061a) allow pkg/v1/random to accept a RNG source ([#​1675](https://github.com/google/go-containerregistry/issues/1675)) - [`1cb7e13`](https://github.com/google/go-containerregistry/commit/1cb7e133) authn: Add NewConfigKeychain to load a config from explicit path ([#​1603](https://github.com/google/go-containerregistry/issues/1603)) - [`348cd86`](https://github.com/google/go-containerregistry/commit/348cd866) bump deps using ./hack/bump-deps.sh ([#​1659](https://github.com/google/go-containerregistry/issues/1659)) - [`691004b`](https://github.com/google/go-containerregistry/commit/691004b4) bump deps using ./hack/bump-deps.sh ([#​1685](https://github.com/google/go-containerregistry/issues/1685)) - [`df72a9a`](https://github.com/google/go-containerregistry/commit/df72a9a9) crane push: print pushed ref@digest to stdout ([#​1663](https://github.com/google/go-containerregistry/issues/1663)) - [`0577676`](https://github.com/google/go-containerregistry/commit/0577676c) crane: ignore buildx attestations in flatten ([#​1630](https://github.com/google/go-containerregistry/issues/1630)) - [`217318c`](https://github.com/google/go-containerregistry/commit/217318c3) deprecate estargz ([#​1660](https://github.com/google/go-containerregistry/issues/1660)) - [`9f68710`](https://github.com/google/go-containerregistry/commit/9f68710b) feat(daemon): generate config file from Docker Engine API ([#​1130](https://github.com/google/go-containerregistry/issues/1130)) - [`52d59d1`](https://github.com/google/go-containerregistry/commit/52d59d1b) feat(remote): make retryStatusCodes configurable ([#​1635](https://github.com/google/go-containerregistry/issues/1635)) - [`6f96bba`](https://github.com/google/go-containerregistry/commit/6f96bba2) gcrane: Use page size of 10,000 for googley things ([#​1645](https://github.com/google/go-containerregistry/issues/1645)) - [`0f2db49`](https://github.com/google/go-containerregistry/commit/0f2db492) release: use ko 0.13.0 to build images ([#​1607](https://github.com/google/go-containerregistry/issues/1607)) - [`d64f9e0`](https://github.com/google/go-containerregistry/commit/d64f9e0e) remove time from random.Image history ([#​1678](https://github.com/google/go-containerregistry/issues/1678)) - [`46488f7`](https://github.com/google/go-containerregistry/commit/46488f7d) retry HTTP 499 errors by default ([#​1612](https://github.com/google/go-containerregistry/issues/1612)) - [`ace7be7`](https://github.com/google/go-containerregistry/commit/ace7be76) update GCR quotas ([#​1619](https://github.com/google/go-containerregistry/issues/1619)) - [`d1c4e9f`](https://github.com/google/go-containerregistry/commit/d1c4e9ff) verify provenance ([#​1611](https://github.com/google/go-containerregistry/issues/1611)) ##### Container Images https://gcr.io/go-containerregistry/crane:v0.15.1 https://gcr.io/go-containerregistry/gcrane:v0.15.1 For example: docker pull gcr.io/go-containerregistry/crane:v0.15.1 docker pull gcr.io/go-containerregistry/gcrane:v0.15.1 ### [`v0.15.0`](https://github.com/google/go-containerregistry/compare/v0.14.0...v0.15.0) [Compare Source](https://github.com/google/go-containerregistry/compare/v0.14.0...v0.15.0) ### [`v0.14.0`](https://github.com/google/go-containerregistry/releases/tag/v0.14.0) [Compare Source](https://github.com/google/go-containerregistry/compare/v0.13.0...v0.14.0) #### Changelog - [`9306eba`](https://github.com/google/go-containerregistry/commit/9306ebad) Allow crane edit to generate non-image artifacts ([#​1545](https://github.com/google/go-containerregistry/issues/1545)) - [`de35f0f`](https://github.com/google/go-containerregistry/commit/de35f0f7) Allow setting Content-Type in crane edit manifest ([#​1551](https://github.com/google/go-containerregistry/issues/1551)) - [`4b081f8`](https://github.com/google/go-containerregistry/commit/4b081f80) Avoid v1.Manifest in crane edit config ([#​1583](https://github.com/google/go-containerregistry/issues/1583)) - [`1cfe1fc`](https://github.com/google/go-containerregistry/commit/1cfe1fc2) Bump aws-actions/configure-aws-credentials from 1.7.0 to 2.0.0 ([#​1593](https://github.com/google/go-containerregistry/issues/1593)) - [`da1008f`](https://github.com/google/go-containerregistry/commit/da1008fb) Bump golangci/golangci-lint-action from 3.3.1 to 3.4.0 ([#​1548](https://github.com/google/go-containerregistry/issues/1548)) - [`86be45f`](https://github.com/google/go-containerregistry/commit/86be45fb) Bump goreleaser/goreleaser-action from 4.1.0 to 4.1.1 ([#​1547](https://github.com/google/go-containerregistry/issues/1547)) - [`62f183e`](https://github.com/google/go-containerregistry/commit/62f183e5) Bump goreleaser/goreleaser-action from 4.1.1 to 4.2.0 ([#​1556](https://github.com/google/go-containerregistry/issues/1556)) - [`1b8dc2b`](https://github.com/google/go-containerregistry/commit/1b8dc2ba) Bump slsa-framework/slsa-github-generator from 1.2.2 to 1.5.0 ([#​1580](https://github.com/google/go-containerregistry/issues/1580)) - [`11843ba`](https://github.com/google/go-containerregistry/commit/11843ba2) Enforce proper sha256 usage ([#​1544](https://github.com/google/go-containerregistry/issues/1544)) - [`2ceebaa`](https://github.com/google/go-containerregistry/commit/2ceebaaf) Implement crane index subcommand ([#​1561](https://github.com/google/go-containerregistry/issues/1561)) - [`9f42e02`](https://github.com/google/go-containerregistry/commit/9f42e028) Set mediaType for empty.ImageIndex in RawManifest ([#​1562](https://github.com/google/go-containerregistry/issues/1562)) - [`759b19f`](https://github.com/google/go-containerregistry/commit/759b19f7) Support artifactType, for images whose config.mediaType is not a config ([#​1541](https://github.com/google/go-containerregistry/issues/1541)) - [`b3c23b4`](https://github.com/google/go-containerregistry/commit/b3c23b4c) Support for OCI 1.1+ referrers via API ([#​1546](https://github.com/google/go-containerregistry/issues/1546)) - [`061ee6b`](https://github.com/google/go-containerregistry/commit/061ee6bf) Support for OCI 1.1+ referrers via fallback tag ([#​1543](https://github.com/google/go-containerregistry/issues/1543)) - [`6770304`](https://github.com/google/go-containerregistry/commit/67703048) Update descriptor "data" field (when valid) during "crane edit config" ([#​1584](https://github.com/google/go-containerregistry/issues/1584)) - [`76bac93`](https://github.com/google/go-containerregistry/commit/76bac933) Update release.yml ([#​1540](https://github.com/google/go-containerregistry/issues/1540)) - [`eb7d746`](https://github.com/google/go-containerregistry/commit/eb7d746c) authn: also read mount secrets ([#​1560](https://github.com/google/go-containerregistry/issues/1560)) - [`e94d408`](https://github.com/google/go-containerregistry/commit/e94d4089) bump deps using ./hack/bump-deps.sh ([#​1592](https://github.com/google/go-containerregistry/issues/1592)) - [`4e95ae2`](https://github.com/google/go-containerregistry/commit/4e95ae2b) crane: add --flatten for index append ([#​1566](https://github.com/google/go-containerregistry/issues/1566)) - [`ff810c1`](https://github.com/google/go-containerregistry/commit/ff810c18) crane: add serve subcommand ([#​1586](https://github.com/google/go-containerregistry/issues/1586)) - [`8ea5e0e`](https://github.com/google/go-containerregistry/commit/8ea5e0e8) crane: support --omit-digest-tags in crane ls ([#​1528](https://github.com/google/go-containerregistry/issues/1528)) - [`824efc7`](https://github.com/google/go-containerregistry/commit/824efc77) fix(mutate): also set timestamps only present in some formats ([#​1550](https://github.com/google/go-containerregistry/issues/1550)) - [`e04520b`](https://github.com/google/go-containerregistry/commit/e04520bc) fix: Fix the crane release url and add more steps ([#​1532](https://github.com/google/go-containerregistry/issues/1532)) - [`d872232`](https://github.com/google/go-containerregistry/commit/d8722327) hash: use generic instantiation ([#​1538](https://github.com/google/go-containerregistry/issues/1538)) - [`57f010d`](https://github.com/google/go-containerregistry/commit/57f010d2) replace manual slsa-verifier installation with action ([#​1585](https://github.com/google/go-containerregistry/issues/1585)) - [`9cd098e`](https://github.com/google/go-containerregistry/commit/9cd098e3) skip tls verification if default transport is used with insecure option ([#​1559](https://github.com/google/go-containerregistry/issues/1559)) - [`3624968`](https://github.com/google/go-containerregistry/commit/36249683) tarball: pass imageToTags ([#​1563](https://github.com/google/go-containerregistry/issues/1563)) ##### Container Images https://gcr.io/go-containerregistry/crane:v0.14.0 https://gcr.io/go-containerregistry/gcrane:v0.14.0 For example: docker pull gcr.io/go-containerregistry/crane:v0.14.0 docker pull gcr.io/go-containerregistry/gcrane:v0.14.0 ### [`v0.13.0`](https://github.com/google/go-containerregistry/releases/tag/v0.13.0) [Compare Source](https://github.com/google/go-containerregistry/compare/v0.12.1...v0.13.0) #### What's Changed - Bump slsa-framework/slsa-github-generator to 1.2.2 by [@​jonjohnsonjr](https://github.com/jonjohnsonjr) in [google/go-containerregistry#1489 - Features: Allow eliding `serviceaccount` lookups. by [@​mattmoor](https://github.com/mattmoor) in [google/go-containerregistry#1490 - Bump golangci/golangci-lint-action from 3.3.0 to 3.3.1 by [@​dependabot](https://github.com/dependabot) in [google/go-containerregistry#1491 - add source archive checksum into the checksums.txt by [@​developer-guy](https://github.com/developer-guy) in [google/go-containerregistry#1492 - Fix calculating tarball size when duplicated layers exist by [@​tydra-wang](https://github.com/tydra-wang) in [google/go-containerregistry#1495 - Add support for zstd compression by [@​LFrobeen](https://github.com/LFrobeen) in [google/go-containerregistry#1487 - docs: pull latest instead of debug by [@​AndrewCharlesHay](https://github.com/AndrewCharlesHay) in [google/go-containerregistry#1497 - Make credential warning slightly more accurate by [@​jonjohnsonjr](https://github.com/jonjohnsonjr) in [google/go-containerregistry#1499 - Make unit tests substantially faster by [@​jonjohnsonjr](https://github.com/jonjohnsonjr) in [google/go-containerregistry#1498 - Use the default retry predicate in transport by [@​jonjohnsonjr](https://github.com/jonjohnsonjr) in [google/go-containerregistry#1502 - Revert "docs: pull latest instead of debug ([#​1497](https://github.com/google/go-containerregistry/issues/1497))" by [@​jonjohnsonjr](https://github.com/jonjohnsonjr) in [google/go-containerregistry#1504 - Update Arch Linux install instructions by [@​kpcyrd](https://github.com/kpcyrd) in [google/go-containerregistry#1508 - Fix various lints by [@​jonjohnsonjr](https://github.com/jonjohnsonjr) in [google/go-containerregistry#1507 - Fix missing doc comment by [@​jonjohnsonjr](https://github.com/jonjohnsonjr) in [google/go-containerregistry#1509 - Treat empty registry config as anonymous by [@​lcarva](https://github.com/lcarva) in [google/go-containerregistry#1512 - Bump goreleaser/goreleaser-action from 3.2.0 to 4.1.0 by [@​dependabot](https://github.com/dependabot) in [google/go-containerregistry#1511 - Bump actions/stale from 6 to 7 by [@​dependabot](https://github.com/dependabot) in [google/go-containerregistry#1519 - Race http fallback ping by [@​jonjohnsonjr](https://github.com/jonjohnsonjr) in [google/go-containerregistry#1521 - FIX mutate.Time not respecting history by [@​miguelvalerio](https://github.com/miguelvalerio) in [google/go-containerregistry#1520 - test: use `T.TempDir` to create temporary test directory by [@​Juneezee](https://github.com/Juneezee) in [google/go-containerregistry#1522 - crane: add digest --full by [@​imjasonh](https://github.com/imjasonh) in [google/go-containerregistry#1524 - Hack around DockerHub plugin scope handling by [@​jonjohnsonjr](https://github.com/jonjohnsonjr) in [google/go-containerregistry#1527 - crane: support --full-ref for crane ls by [@​imjasonh](https://github.com/imjasonh) in [google/go-containerregistry#1525 - Revert plugin scope hack by [@​jonjohnsonjr](https://github.com/jonjohnsonjr) in [google/go-containerregistry#1531 - clarify crane download readme by [@​dtanner](https://github.com/dtanner) in [google/go-containerregistry#1533 #### New Contributors - [@​LFrobeen](https://github.com/LFrobeen) made their first contribution in [google/go-containerregistry#1487 - [@​AndrewCharlesHay](https://github.com/AndrewCharlesHay) made their first contribution in [google/go-containerregistry#1497 - [@​kpcyrd](https://github.com/kpcyrd) made their first contribution in [google/go-containerregistry#1508 - [@​miguelvalerio](https://github.com/miguelvalerio) made their first contribution in [google/go-containerregistry#1520 - [@​Juneezee](https://github.com/Juneezee) made their first contribution in [google/go-containerregistry#1522 - [@​dtanner](https://github.com/dtanner) made their first contribution in [google/go-containerregistry#1533 **Full Changelog**: google/go-containerregistry@v0.12.1...v0.13.0 ### [`v0.12.1`](https://github.com/google/go-containerregistry/releases/tag/v0.12.1) [Compare Source](https://github.com/google/go-containerregistry/compare/v0.12.0...v0.12.1) #### Changelog - [`426de7d`](https://github.com/google/go-containerregistry/commit/426de7d4) Bump golangci/golangci-lint-action from 3.2.0 to 3.3.0 ([#​1475](https://github.com/google/go-containerregistry/issues/1475)) - [`6442b02`](https://github.com/google/go-containerregistry/commit/6442b027) Bump goreleaser/goreleaser-action from 3.1.0 to 3.2.0 ([#​1476](https://github.com/google/go-containerregistry/issues/1476)) - [`76ae819`](https://github.com/google/go-containerregistry/commit/76ae8192) Fix context.DeadlineExceeded comparison ([#​1488](https://github.com/google/go-containerregistry/issues/1488)) - [`1711cef`](https://github.com/google/go-containerregistry/commit/1711cefd) Fix missing body.Close() in bearer auth ([#​1482](https://github.com/google/go-containerregistry/issues/1482)) - [`02f47e1`](https://github.com/google/go-containerregistry/commit/02f47e13) bump version of slsa generator ([#​1468](https://github.com/google/go-containerregistry/issues/1468)) - [`353a117`](https://github.com/google/go-containerregistry/commit/353a1176) crane: add catalog argument use annotation ([#​1473](https://github.com/google/go-containerregistry/issues/1473)) - [`a0cca8a`](https://github.com/google/go-containerregistry/commit/a0cca8a0) k8schain: Log and proceed if secret or SA are not found ([#​1472](https://github.com/google/go-containerregistry/issues/1472)) ##### Container Images https://gcr.io/go-containerregistry/crane:v0.12.1 https://gcr.io/go-containerregistry/gcrane:v0.12.1 For example: docker pull gcr.io/go-containerregistry/crane:v0.12.1 docker pull gcr.io/go-containerregistry/gcrane:v0.12.1 ### [`v0.12.0`](https://github.com/google/go-containerregistry/releases/tag/v0.12.0) [Compare Source](https://github.com/google/go-containerregistry/compare/v0.11.0...v0.12.0) #### Changelog - [`9b4fdd5`](https://github.com/google/go-containerregistry/commit/9b4fdd50) Bump actions/setup-go from 2 to 3 ([#​1463](https://github.com/google/go-containerregistry/issues/1463)) - [`7268da0`](https://github.com/google/go-containerregistry/commit/7268da01) Bump actions/stale from 5 to 6 ([#​1452](https://github.com/google/go-containerregistry/issues/1452)) - [`7196cf3`](https://github.com/google/go-containerregistry/commit/7196cf3d) Bump aws-actions/configure-aws-credentials from 1.6.1 to 1.7.0 ([#​1424](https://github.com/google/go-containerregistry/issues/1424)) - [`8eae069`](https://github.com/google/go-containerregistry/commit/8eae069f) Bump codecov/codecov-action from 3.1.0 to 3.1.1 ([#​1453](https://github.com/google/go-containerregistry/issues/1453)) - [`969699e`](https://github.com/google/go-containerregistry/commit/969699e7) Bump deps using ./hack/bump-deps.sh ([#​1467](https://github.com/google/go-containerregistry/issues/1467)) - [`c1f9836`](https://github.com/google/go-containerregistry/commit/c1f9836a) Bump opencontainers/image-spec ([#​1423](https://github.com/google/go-containerregistry/issues/1423)) - [`49cdb8b`](https://github.com/google/go-containerregistry/commit/49cdb8b4) Correct usage of authn.NewKeychainFromHelper in docs ([#​1419](https://github.com/google/go-containerregistry/issues/1419)) - [`3ba4c51`](https://github.com/google/go-containerregistry/commit/3ba4c511) Fix tar PAX format handling ([#​1414](https://github.com/google/go-containerregistry/issues/1414)) - [`24a1c33`](https://github.com/google/go-containerregistry/commit/24a1c335) Ignore docker config if it's a directory ([#​1420](https://github.com/google/go-containerregistry/issues/1420)) - [`a0f6687`](https://github.com/google/go-containerregistry/commit/a0f66878) Make ErrBadName checkable via errors.Is() ([#​1462](https://github.com/google/go-containerregistry/issues/1462)) - [`19e3eff`](https://github.com/google/go-containerregistry/commit/19e3eff7) Retry ECONNRESET errors ([#​1415](https://github.com/google/go-containerregistry/issues/1415)) - [`5749ee6`](https://github.com/google/go-containerregistry/commit/5749ee68) Support the platform specific authentication of krane in "auth get" command ([#​1413](https://github.com/google/go-containerregistry/issues/1413)) - [`e3b94c7`](https://github.com/google/go-containerregistry/commit/e3b94c7e) allow remote.DefaultTransport to be overridden by an http.RoundTripper ([#​1449](https://github.com/google/go-containerregistry/issues/1449)) - [`f981b4c`](https://github.com/google/go-containerregistry/commit/f981b4c0) deps: update goreleaser-action for bug ([#​1444](https://github.com/google/go-containerregistry/issues/1444)) - [`771a9b4`](https://github.com/google/go-containerregistry/commit/771a9b44) e2e: pull and export stdin and stdout ([#​1436](https://github.com/google/go-containerregistry/issues/1436)) - [`87b3a79`](https://github.com/google/go-containerregistry/commit/87b3a796) feat: Add krane to release archive ([#​1443](https://github.com/google/go-containerregistry/issues/1443)) - [`2859a0d`](https://github.com/google/go-containerregistry/commit/2859a0d0) feat: generate slsa provenance on github release artifacts ([#​1438](https://github.com/google/go-containerregistry/issues/1438)) - [`9a5c14a`](https://github.com/google/go-containerregistry/commit/9a5c14ad) fix crane's root.go after DefaultTransport change ([#​1450](https://github.com/google/go-containerregistry/issues/1450)) - [`2b54510`](https://github.com/google/go-containerregistry/commit/2b54510b) fix: consider base image media type when appending layers ([#​1437](https://github.com/google/go-containerregistry/issues/1437)) - [`d3ed408`](https://github.com/google/go-containerregistry/commit/d3ed4089) registry: implement blob deletion ([#​1432](https://github.com/google/go-containerregistry/issues/1432)) - [`3413eb6`](https://github.com/google/go-containerregistry/commit/3413eb6c) registry: implement pagination ([#​1430](https://github.com/google/go-containerregistry/issues/1430)) - [`e2d575c`](https://github.com/google/go-containerregistry/commit/e2d575cf) update crane installation instructions and release verification ([#​1440](https://github.com/google/go-containerregistry/issues/1440)) ##### Container Images https://gcr.io/go-containerregistry/crane:v0.12.0 https://gcr.io/go-containerregistry/gcrane:v0.12.0 For example: docker pull gcr.io/go-containerregistry/crane:v0.12.0 docker pull gcr.io/go-containerregistry/gcrane:v0.12.0 </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://app.renovatebot.com/dashboard#github/kairos-io/provider-kairos). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNS44LjMiLCJ1cGRhdGVkSW5WZXIiOiIzNS44Ny4xIiwidGFyZ2V0QnJhbmNoIjoibWFpbiJ9--> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
* Add support for configurable compression algorithm (gzip, zstd) and compression level We want to make the layer compression in kaniko configurable, so we have added two optional command line arguments “--compression” and “--compression-level”. The former allows the user to specify a compression algorithm (zstd, gzip) and the latter can be used to specify the compression level. Depending on the selected compression algorithm and level we modify the set of layerOptions that are used to create tarball layers in `push.go` and `build.go`. The actual implementation of the zstd support can be found in our fork of the go-containerregistry package for which we have filed this PR: google/go-containerregistry#1487 The changes should be fully backwards compatible. * Restrict inputs for compression flag to gzip and zstd This change will ensure that users can only specify supported compression algorithms (`zstd`, `gzip`) to the `--compression` flag. * Fix incorrect type for switch statements on config.Compression
This PR adds support for zstandard compression to the tarball layer implementation. It adds the
application/vnd.oci.image.layer.v1.tar+zstd media
type, aLayerCompression
type and some extra logic in the tarball package. Moreover, this PR modifies pkg/v1/partial/compressed.go so that it is able to unpack zstd compressed layers.When creating a tarball layer, one can specify the compression algorithm using
WithCompression
and the compression level withWithCompressionLevel
. By default it will still use gzip compression.The changes should be fully backwards compatible.
To give some more context:
We are adding zstd support to Northflank CI/CD & Images. We have built support for multiple build engines including Buildkit, Kaniko and CNBPs. Buildkit already supports zstd, but Kaniko does not. Adding zstd support to
go-containerregistry
will in turn allow Kaniko to build images with zstd compression which outperformed gzip in both performance and compression ratio in our tests.