added support for overriding manifest annotations #1056

developer-guy · 2021-06-17T22:06:32Z

Signed-off-by: Batuhan Apaydın batuhan.apaydin@trendyol.com

Hello, this PR will add overriding manifest annotations support to crane's mutate command, it will work like the following picture:

codecov-commenter · 2021-06-18T07:43:20Z

Codecov Report

Merging #1056 (46ce3c3) into main (5f53e4e) will decrease coverage by 0.11%.
The diff coverage is 0.00%.

❗ Current head 46ce3c3 differs from pull request most recent head 50d14fe. Consider uploading reports for the commit 50d14fe to get more accurate results

@@            Coverage Diff             @@
##             main    #1056      +/-   ##
==========================================
- Coverage   75.46%   75.34%   -0.12%     
==========================================
  Files         107      107              
  Lines        5087     5095       +8     
==========================================
  Hits         3839     3839              
- Misses        703      711       +8     
  Partials      545      545

Impacted Files	Coverage Δ
pkg/v1/mutate/mutate.go	`69.04% <0.00%> (-3.46%)`	⬇️
pkg/crane/get.go	`100.00% <0.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5f53e4e...50d14fe. Read the comment docs.

imjasonh

Thanks for this change! 🎉

cmd/crane/cmd/mutate.go

imjasonh · 2021-06-18T12:17:49Z

pkg/v1/mutate/mutate.go

+	m, err := base.Manifest()
+	if err != nil {
+		return nil, err
+	}


Can we m.DeepCopy here instead, as early as possible and before any changes?

thank you so much for this review, but I tried to do that but it didn't work for me, I updated the code with the following content, what did I do wrong?

mn := m.DeepCopy() if mn.Annotations == nil { mn.Annotations = map[string]string{} } for k, v := range annotations { mn.Annotations[k] = v } image := &image{ base: base, manifest: mn, }

pkg/v1/mutate/mutate_test.go

imjasonh · 2021-06-18T12:25:51Z

pkg/v1/mutate/mutate_test.go

+		t.Fatalf("failed to mutate a config: %v", err)
+	}
+
+	if manifestsAreEqual(t, source, result) {


Does this indicate that the source was mutated in the process? I would expect source to be unmodified and the modified result to be returned.

pkg/v1/mutate/mutate_test.go

jonjohnsonjr · 2021-06-18T16:20:56Z

I'll add some context from #281 we talked about how this would work for both image and manifest lists. I don't think you need to solve this immediately, but being able to annotate just the top-level manifest vs recursing and annotating every manifest is an interesting problem.

pkg/v1/mutate/mutate.go

pkg/v1/mutate/mutate_test.go

pkg/v1/mutate/mutate.go

jonjohnsonjr · 2021-06-18T20:17:56Z

pkg/v1/mutate/mutate.go

@@ -113,6 +113,29 @@ func Config(base v1.Image, cfg v1.Config) (v1.Image, error) {
 	return ConfigFile(base, cf)
 }

+// IndexAnnotations mutates the provided v1.Image to have the provided annotations
+func IndexAnnotations(base v1.Image, annotations map[string]string) (v1.Image, error) {


Sorry, my comment wasn't clear, just call this Annotations

jonjohnsonjr · 2021-06-18T21:28:24Z

Oh yeah, you'll want to do this lazily. That manifest field in image is the final result of computing everything, so we need to move this logic into compute():

$ git diff
diff --git a/pkg/v1/mutate/image.go b/pkg/v1/mutate/image.go
index 14b9c70a..23635f2b 100644
--- a/pkg/v1/mutate/image.go
+++ b/pkg/v1/mutate/image.go
@@ -30,12 +30,13 @@ type image struct {
        base v1.Image
        adds []Addendum
 
-       computed   bool
-       configFile *v1.ConfigFile
-       manifest   *v1.Manifest
-       mediaType  *types.MediaType
-       diffIDMap  map[v1.Hash]v1.Layer
-       digestMap  map[v1.Hash]v1.Layer
+       computed    bool
+       configFile  *v1.ConfigFile
+       manifest    *v1.Manifest
+       annotations map[string]string
+       mediaType   *types.MediaType
+       diffIDMap   map[v1.Hash]v1.Layer
+       digestMap   map[v1.Hash]v1.Layer
 }
 
 var _ v1.Image = (*image)(nil)
@@ -138,6 +139,15 @@ func (i *image) compute() error {
                        manifest.MediaType = *i.mediaType
                }
        }
+       if i.annotations != nil {
+               if manifest.Annotations == nil {
+                       manifest.Annotations = map[string]string{}
+               }
+
+               for k, v := range i.annotations {
+                       manifest.Annotations[k] = v
+               }
+       }
 
        i.configFile = configFile
        i.manifest = manifest
diff --git a/pkg/v1/mutate/mutate.go b/pkg/v1/mutate/mutate.go
index 7615b323..70df4e46 100644
--- a/pkg/v1/mutate/mutate.go
+++ b/pkg/v1/mutate/mutate.go
@@ -115,22 +115,9 @@ func Config(base v1.Image, cfg v1.Config) (v1.Image, error) {
 
 // Annotations mutates the provided v1.Image to have the provided annotations
 func Annotations(base v1.Image, annotations map[string]string) (v1.Image, error) {
-       m, err := base.Manifest()
-       if err != nil {
-               return nil, err
-       }
-
-       if m.Annotations == nil {
-               m.Annotations = map[string]string{}
-       }
-
-       for k, v := range annotations {
-               m.Annotations[k] = v
-       }
-
        image := &image{
-               base:     base,
-               manifest: m.DeepCopy(),
+               base:        base,
+               annotations: annotations,
        }
 
        return image, nil

One thing I'm a little worried about is that if you run this command on a manifest list, I think it will turn that manifest list into an image, because we just call crane.Pull. We don't want to do that...

Give me one second, I want to implement something that will let us check if something is an image.

jonjohnsonjr · 2021-06-18T21:53:59Z

Give me one second, I want to implement something that will let us check if something is an image.

#1057

developer-guy · 2021-06-18T23:39:49Z

Oh yeah, you'll want to do this lazily. That manifest field in image is the final result of computing everything, so we need to move this logic into compute():

$ git diff
diff --git a/pkg/v1/mutate/image.go b/pkg/v1/mutate/image.go
index 14b9c70a..23635f2b 100644
--- a/pkg/v1/mutate/image.go
+++ b/pkg/v1/mutate/image.go
@@ -30,12 +30,13 @@ type image struct {
        base v1.Image
        adds []Addendum
 
-       computed   bool
-       configFile *v1.ConfigFile
-       manifest   *v1.Manifest
-       mediaType  *types.MediaType
-       diffIDMap  map[v1.Hash]v1.Layer
-       digestMap  map[v1.Hash]v1.Layer
+       computed    bool
+       configFile  *v1.ConfigFile
+       manifest    *v1.Manifest
+       annotations map[string]string
+       mediaType   *types.MediaType
+       diffIDMap   map[v1.Hash]v1.Layer
+       digestMap   map[v1.Hash]v1.Layer
 }
 
 var _ v1.Image = (*image)(nil)
@@ -138,6 +139,15 @@ func (i *image) compute() error {
                        manifest.MediaType = *i.mediaType
                }
        }
+       if i.annotations != nil {
+               if manifest.Annotations == nil {
+                       manifest.Annotations = map[string]string{}
+               }
+
+               for k, v := range i.annotations {
+                       manifest.Annotations[k] = v
+               }
+       }
 
        i.configFile = configFile
        i.manifest = manifest
diff --git a/pkg/v1/mutate/mutate.go b/pkg/v1/mutate/mutate.go
index 7615b323..70df4e46 100644
--- a/pkg/v1/mutate/mutate.go
+++ b/pkg/v1/mutate/mutate.go
@@ -115,22 +115,9 @@ func Config(base v1.Image, cfg v1.Config) (v1.Image, error) {
 
 // Annotations mutates the provided v1.Image to have the provided annotations
 func Annotations(base v1.Image, annotations map[string]string) (v1.Image, error) {
-       m, err := base.Manifest()
-       if err != nil {
-               return nil, err
-       }
-
-       if m.Annotations == nil {
-               m.Annotations = map[string]string{}
-       }
-
-       for k, v := range annotations {
-               m.Annotations[k] = v
-       }
-
        image := &image{
-               base:     base,
-               manifest: m.DeepCopy(),
+               base:        base,
+               annotations: annotations,
        }
 
        return image, nil

One thing I'm a little worried about is that if you run this command on a manifest list, I think it will turn that manifest list into an image, because we just call crane.Pull. We don't want to do that...

Give me one second, I want to implement something that will let us check if something is an image.

thank you so much for this, I really appreciate it, I updated the PR according to your feedback. 🙋🏻‍♂️😋

developer-guy · 2021-06-18T23:40:40Z

Give me one second, I want to implement something that will let us check if something is an image.

#1057

where should we do this validation, would you like to help me if you don't mind? 🙏 @jonjohnsonjr

jonjohnsonjr · 2021-06-21T19:41:25Z

where should we do this validation

So at the beginning of the command, before we crane.Pull, we want to check to make sure we're not trying to mutate annotations on an index, because that won't work... Something like:

if len(anntns) != 0 {
  desc, err := crane.Head(ref, *options...)
  if err != nil {
    log.Fatalf("checking %s: %v", ref, err)
  }
  if desc.MediaType.IsIndex() {
    log.Fatalf("mutating annotations on an index is not yet supported")
  }
}

developer-guy · 2021-06-21T21:48:34Z

where should we do this validation

So at the beginning of the command, before we crane.Pull, we want to check to make sure we're not trying to mutate annotations on an index, because that won't work... Something like:
if len(anntns) != 0 {
  desc, err := crane.Head(ref, *options...)
  if err != nil {
    log.Fatalf("checking %s: %v", ref, err)
  }
  if desc.MediaType.IsIndex() {
    log.Fatalf("mutating annotations on an index is not yet supported")
  }
}

done @jonjohnsonjr in the commit de9902f but I don't understand the necessity of doing this, can you please tell me why we are doing this, and what is the index thing actually, thank you 🙏

jonjohnsonjr · 2021-06-21T22:35:02Z

can you please tell me why we are doing this, and what is the index thing actually, thank you

Sure, let's look at ubuntu:latest on DockerHub:

https://explore.ggcr.dev/?image=ubuntu@sha256:aba80b77e27148d99c034a987e7da3a287ed455390352663418c0f2ed40417fe

This is a multi-platform image, AKA an image index, which I often shorten to just "index". Per the spec, an image index can have annotations, so it would be reasonable to expect calling crane mutate -a "hello=world" ubuntu to work and add an annotation to the multi-platform image.

However, this will not work! It would instead add the annotation to one of it's children -- the linux/amd64 image:

https://explore.ggcr.dev/?image=index.docker.io/library/ubuntu@sha256:376209074d481dca0a9cf4282710cd30a9e7ff402dea8261acdaaf57a18971dd&mt=application/vnd.docker.distribution.manifest.v2+json

Why? Because crane.Pull returns a v1.Image, which means it has to pick one of the child images to pull. There's a --platform flag and crane.WithPlatform option for specifying which platform you should pick.

After annotating the child image, it would update the ubuntu:latest tag to point to that child instead of the multi-platform image, which would essentially break any other platforms trying to pull it.

We want to avoid that last part, so if we encounter something that isn't an image, we'll just abort. In the future, we could add something like mutate.IndexAnnotations or something to support annotating multi-platform images. This is awful because we bolted on multi-platform stuff after the fact, so it doesn't work that well with the type system. We've had some discussion around how to make this better (#835) but I think we'll need to redesign everything and bump to a v2 to improve things, really :/

jonjohnsonjr · 2021-06-21T23:02:45Z

Looks like the linter is mad, need to run gofmt.

developer-guy · 2021-06-22T08:26:49Z

we'll just abort

can you please tell me why we are doing this, and what is the index thing actually, thank you

Sure, let's look at ubuntu:latest on DockerHub:

https://explore.ggcr.dev/?image=ubuntu@sha256:aba80b77e27148d99c034a987e7da3a287ed455390352663418c0f2ed40417fe

This is a multi-platform image, AKA an image index, which I often shorten to just "index". Per the spec, an image index can have annotations, so it would be reasonable to expect calling crane mutate -a "hello=world" ubuntu to work and add an annotation to the multi-platform image.

However, this will not work! It would instead add the annotation to one of it's children -- the linux/amd64 image:

https://explore.ggcr.dev/?image=index.docker.io/library/ubuntu@sha256:376209074d481dca0a9cf4282710cd30a9e7ff402dea8261acdaaf57a18971dd&mt=application/vnd.docker.distribution.manifest.v2+json

Why? Because crane.Pull returns a v1.Image, which means it has to pick one of the child images to pull. There's a --platform flag and crane.WithPlatform option for specifying which platform you should pick.

After annotating the child image, it would update the ubuntu:latest tag to point to that child instead of the multi-platform image, which would essentially break any other platforms trying to pull it.

We want to avoid that last part, so if we encounter something that isn't an image, we'll just abort. In the future, we could add something like mutate.IndexAnnotations or something to support annotating multi-platform images. This is awful because we bolted on multi-platform stuff after the fact, so it doesn't work that well with the type system. We've had some discussion around how to make this better (#835) but I think we'll need to redesign everything and bump to a v2 to improve things, really :/

Omg, thank you so much for this detailed explanation, I understand now 👍

developer-guy · 2021-06-22T08:27:21Z

Looks like the linter is mad, need to run gofmt.

fixed, thank you so much 👍

Signed-off-by: Batuhan Apaydın <batuhan.apaydin@trendyol.com> Co-Authored-by: Jon Johnson <jonjohnson@google.com> - improvements based on feedbacks - check to make sure we're not trying to mutate annotations on an index

jonjohnsonjr · 2021-06-22T17:59:29Z

Thanks!

developer-guy force-pushed the main branch from 4116c47 to 46ce3c3 Compare June 17, 2021 22:20

jonjohnsonjr assigned imjasonh Jun 18, 2021

developer-guy force-pushed the main branch from 46ce3c3 to 7139382 Compare June 18, 2021 07:43

developer-guy force-pushed the main branch from 7139382 to 2e9aeba Compare June 18, 2021 12:08

developer-guy changed the title ~~overriding manifest annotations support~~ added support for overriding manifest annotations Jun 18, 2021

imjasonh reviewed Jun 18, 2021

View reviewed changes

jonjohnsonjr reviewed Jun 18, 2021

View reviewed changes

pkg/v1/mutate/mutate.go Outdated Show resolved Hide resolved

pkg/v1/mutate/mutate_test.go Outdated Show resolved Hide resolved

pkg/v1/mutate/mutate.go Outdated Show resolved Hide resolved

pkg/v1/mutate/mutate.go Outdated Show resolved Hide resolved

developer-guy requested review from imjasonh and jonjohnsonjr June 18, 2021 20:13

jonjohnsonjr reviewed Jun 18, 2021

View reviewed changes

developer-guy force-pushed the main branch from 9e619a8 to 6b1b7d7 Compare June 18, 2021 20:27

developer-guy requested a review from jonjohnsonjr June 18, 2021 20:28

developer-guy force-pushed the main branch from 6b1b7d7 to 7ea91f6 Compare June 18, 2021 23:38

developer-guy force-pushed the main branch from 7ea91f6 to 8d519aa Compare June 21, 2021 21:47

developer-guy force-pushed the main branch 2 times, most recently from d5e8970 to 50d14fe Compare June 22, 2021 08:22

feat(mutate): added support for overriding manifest annotations

f99b6a8

Signed-off-by: Batuhan Apaydın <batuhan.apaydin@trendyol.com> Co-Authored-by: Jon Johnson <jonjohnson@google.com> - improvements based on feedbacks - check to make sure we're not trying to mutate annotations on an index

developer-guy force-pushed the main branch from 50d14fe to f99b6a8 Compare June 22, 2021 09:09

jonjohnsonjr approved these changes Jun 22, 2021

View reviewed changes

jonjohnsonjr merged commit 92e9e85 into google:main Jun 22, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

added support for overriding manifest annotations #1056

added support for overriding manifest annotations #1056

developer-guy commented Jun 17, 2021

codecov-commenter commented Jun 18, 2021 •

edited

Loading

imjasonh left a comment

imjasonh Jun 18, 2021

developer-guy Jun 18, 2021

imjasonh Jun 18, 2021

jonjohnsonjr commented Jun 18, 2021

jonjohnsonjr Jun 18, 2021

developer-guy Jun 18, 2021

jonjohnsonjr commented Jun 18, 2021

jonjohnsonjr commented Jun 18, 2021

developer-guy commented Jun 18, 2021

developer-guy commented Jun 18, 2021 •

edited

Loading

jonjohnsonjr commented Jun 21, 2021

developer-guy commented Jun 21, 2021 •

edited

Loading

jonjohnsonjr commented Jun 21, 2021

jonjohnsonjr commented Jun 21, 2021

developer-guy commented Jun 22, 2021 •

edited

Loading

developer-guy commented Jun 22, 2021

jonjohnsonjr commented Jun 22, 2021

added support for overriding manifest annotations #1056

added support for overriding manifest annotations #1056

Conversation

developer-guy commented Jun 17, 2021

codecov-commenter commented Jun 18, 2021 • edited Loading

Codecov Report

imjasonh left a comment

Choose a reason for hiding this comment

imjasonh Jun 18, 2021

Choose a reason for hiding this comment

developer-guy Jun 18, 2021

Choose a reason for hiding this comment

imjasonh Jun 18, 2021

Choose a reason for hiding this comment

jonjohnsonjr commented Jun 18, 2021

jonjohnsonjr Jun 18, 2021

Choose a reason for hiding this comment

developer-guy Jun 18, 2021

Choose a reason for hiding this comment

jonjohnsonjr commented Jun 18, 2021

jonjohnsonjr commented Jun 18, 2021

developer-guy commented Jun 18, 2021

developer-guy commented Jun 18, 2021 • edited Loading

jonjohnsonjr commented Jun 21, 2021

developer-guy commented Jun 21, 2021 • edited Loading

jonjohnsonjr commented Jun 21, 2021

jonjohnsonjr commented Jun 21, 2021

developer-guy commented Jun 22, 2021 • edited Loading

developer-guy commented Jun 22, 2021

jonjohnsonjr commented Jun 22, 2021

codecov-commenter commented Jun 18, 2021 •

edited

Loading

developer-guy commented Jun 18, 2021 •

edited

Loading

developer-guy commented Jun 21, 2021 •

edited

Loading

developer-guy commented Jun 22, 2021 •

edited

Loading