Fail without pushing when `crane optimize` misses files #883

mattmoor · 2020-12-22T16:34:43Z

No description provided.

mattmoor · 2020-12-22T16:42:45Z

Sample failing output (nothing pushed):

# crane optimize gcr.io/distroless/static:nonroot ghcr.io/mattmoor/distroless/static:nonroot --prioritize /a/b --prioritize /c/d
2020/12/22 08:40:52 Optimizing from gcr.io/distroless/static:nonroot to ghcr.io/mattmoor/distroless/static:nonroot
2020/12/22 08:40:52 No matching credentials were found for "gcr.io/distroless/static", falling back on anonymous
2020/12/22 08:40:56 failed to optimize index: the following prioritized files were missing from all images: [/a/b /c/d]

Sample successful output:

# crane optimize gcr.io/distroless/static:nonroot ghcr.io/mattmoor/distroless/static:nonroot --prioritize /etc/passwd
2020/12/22 08:41:34 Optimizing from gcr.io/distroless/static:nonroot to ghcr.io/mattmoor/distroless/static:nonroot
2020/12/22 08:41:34 No matching credentials were found for "gcr.io/distroless/static", falling back on anonymous
2020/12/22 08:41:38 existing manifest: sha256:96fd34242ad0be6d50809b9f4a2b2fdfd79e8fb4fbef36c8905e53816bd7a69d
2020/12/22 08:41:38 existing manifest: sha256:4ca465ad088e114d4c207f268cb0cfd59ff1feae16d1d8ca4e7c4857863d533e
2020/12/22 08:41:38 existing manifest: sha256:93fd2b8ecf0d102094930acef84d064253e2c9cc307354f4dbb89e1e7d35915c
2020/12/22 08:41:38 existing manifest: sha256:83564ecf2de20e3ab36aad46ab320574a54a9d149a952b0f9534a516a2707815
2020/12/22 08:41:39 existing manifest: sha256:0bd13356efab3bf5162bd2e6f074d65205c99f101155711eb68d0495773316fc
2020/12/22 08:41:39 ghcr.io/mattmoor/distroless/static:nonroot: digest: sha256:e09730b49128208050b41a6e42991ea64716af1b4ae17bb697971c31c767965d size: 1165

mattmoor · 2020-12-22T16:56:46Z

@ktock thanks for the new option 🤩

mattmoor · 2020-12-22T17:54:44Z

I am trying to pull in containerd/stargz-snapshotter#225 to fix the Windows leg 🤞

mattmoor · 2020-12-22T19:07:31Z

Alright, looks like this needs: containerd/stargz-snapshotter#225 to land.

codecov-io · 2020-12-23T02:14:37Z

Codecov Report

Merging #883 (958010b) into master (c95d7bd) will decrease coverage by 0.23%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##           master     #883      +/-   ##
==========================================
- Coverage   72.76%   72.52%   -0.24%     
==========================================
  Files         108      108              
  Lines        4655     4670      +15     
==========================================
  Hits         3387     3387              
- Misses        758      773      +15     
  Partials      510      510

Impacted Files	Coverage Δ
pkg/crane/optimize.go	`0.00% <0.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c95d7bd...958010b. Read the comment docs.

mattmoor · 2020-12-23T02:42:01Z

@imjasonh @jonjohnsonjr this is RFAL (upstream merged, and this is rebased)

imjasonh · 2020-12-23T15:39:41Z

pkg/crane/optimize.go

 		}
+		missingFromImage = missingFromImage.Intersection(sets.NewString(missingFromLayer...))


If the presence of missing files is considered an error, why not just return an error if this set is non-empty? This makes the method a bit easier to use, since you don't have to remember to check for returned missing files.

Because they might be missing from one image, but not the index.

We only error before pushing because it is only at that point that we have no more places we might find the file.

Interesting. For an index, this only fails if a prioritized file is not found in any of the images. That makes some sense to me, but isn't super intuitive...

Maybe worth logging a warning or requiring a flag that says "yeah I know, these files might not be in every image".

My original change only warned for missing files and I found the output easily got lost in the push output, so I opted instead for making it fatal (and not pushing the image(s)).

I do think it would be interesting to add a verbose mode that displays:

Which files are found in which layers

Which images are missing which files

... might be useful, but this feels more diagnostic than something I'd want always-on...

TBH I don't know how common different files are for different architectures. I know for muti-arch kaniko (b/c of how Bazel names outputs) I ended up making the key binary /kaniko/executor_{arch} because it was the only way to get multiple binaries out of Bazel for the same Go 🙃

I suspect that if this takes off, then we're going to want to have significantly more diagnostic tooling around this (e.g. inspecting the TOC of a layer, which files are prioritized?)

I suspect that if this takes off, then we're going to want to have significantly more diagnostic tooling around this (e.g. inspecting the TOC of a layer, which files are prioritized?)

Maybe an entire estargz subcommand? 😉

It's hidden, so we can always do that later 🤷

jonjohnsonjr · 2020-12-23T17:51:32Z

pkg/crane/optimize.go

@@ -27,11 +27,12 @@ import (
 	"github.com/google/go-containerregistry/pkg/v1/remote"
 	"github.com/google/go-containerregistry/pkg/v1/tarball"
 	"github.com/google/go-containerregistry/pkg/v1/types"
+	"k8s.io/apimachinery/pkg/util/sets"


Not super happy about depending on this.

It's fine :P

jonjohnsonjr · 2020-12-23T17:54:07Z

pkg/crane/optimize.go

 		}
+		missingFromImage = missingFromImage.Intersection(sets.NewString(missingFromLayer...))


Interesting. For an index, this only fails if a prioritized file is not found in any of the images. That makes some sense to me, but isn't super intuitive...

Maybe worth logging a warning or requiring a flag that says "yeah I know, these files might not be in every image".

jonjohnsonjr

I'm still not happy about the set thing but it lgtm. When we isolate k8schain (move it or give it its own go.mod), I'll remove this.

mattmoor assigned jonjohnsonjr Dec 22, 2020

mattmoor force-pushed the missing-files branch from 39dc2bc to 063da5a Compare December 22, 2020 16:41

Warn when crane optimize misses files

4625b8a

mattmoor force-pushed the missing-files branch from 063da5a to 4625b8a Compare December 22, 2020 16:43

mattmoor changed the title ~~Warn when crane optimize misses files~~ Fail without pushing when crane optimize misses files Dec 22, 2020

mattmoor force-pushed the missing-files branch from ef9f50d to ace3d93 Compare December 22, 2020 17:55

mattmoor force-pushed the missing-files branch from ace3d93 to 445cffd Compare December 23, 2020 02:14

Try my windows fix

958010b

mattmoor force-pushed the missing-files branch from 445cffd to 958010b Compare December 23, 2020 02:15

imjasonh reviewed Dec 23, 2020

View reviewed changes

jonjohnsonjr reviewed Dec 23, 2020

View reviewed changes

jonjohnsonjr approved these changes Dec 23, 2020

View reviewed changes

mattmoor merged commit 54efceb into google:master Dec 23, 2020

mattmoor deleted the missing-files branch December 23, 2020 20:01

dependabot bot mentioned this pull request Mar 8, 2021

Bump github.com/google/go-containerregistry from 0.3.0 to 0.4.1 phantooom/minikube#6

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fail without pushing when `crane optimize` misses files #883

Fail without pushing when `crane optimize` misses files #883

mattmoor commented Dec 22, 2020

mattmoor commented Dec 22, 2020

mattmoor commented Dec 22, 2020

mattmoor commented Dec 22, 2020

mattmoor commented Dec 22, 2020

codecov-io commented Dec 23, 2020 •

edited

Loading

mattmoor commented Dec 23, 2020

imjasonh Dec 23, 2020

mattmoor Dec 23, 2020

jonjohnsonjr Dec 23, 2020

mattmoor Dec 23, 2020

jonjohnsonjr Dec 23, 2020

mattmoor Dec 23, 2020

jonjohnsonjr Dec 23, 2020

mattmoor Dec 23, 2020

jonjohnsonjr Dec 23, 2020

jonjohnsonjr left a comment

		}
		missingFromImage = missingFromImage.Intersection(sets.NewString(missingFromLayer...))

Fail without pushing when crane optimize misses files #883

Fail without pushing when crane optimize misses files #883

Conversation

mattmoor commented Dec 22, 2020

mattmoor commented Dec 22, 2020

mattmoor commented Dec 22, 2020

mattmoor commented Dec 22, 2020

mattmoor commented Dec 22, 2020

codecov-io commented Dec 23, 2020 • edited Loading

Codecov Report

mattmoor commented Dec 23, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonjohnsonjr left a comment

Choose a reason for hiding this comment

Fail without pushing when `crane optimize` misses files #883

Fail without pushing when `crane optimize` misses files #883

codecov-io commented Dec 23, 2020 •

edited

Loading