add to layout replacedescriptor and removedescriptor

[deitch]

Fixes #842

adds the following to v1/layout:

• RemoveDescriptors(matcher match.Matcher) - removes from index.json any descriptors that match
• ReplaceDescriptor(desc v1.Descriptor, matcher match.Matcher) - equivalent of RemoveDescriptors(matcher) followed by AppendDescriptor(desc)
• ReplaceImage(image v1.Image, matcher match.Matcher) - equivalent of RemoveDescriptors(matcher) followed by AppendImage(image)
• ReplaceIndex(index v1.ImageIndex, matcher match.Matcher) - equivalent of RemoveDescriptors(matcher) followed by AppendIndex(index)

None of these does garbage collection in the blobs dir, which is a completely separate topic, and should be handled under separate cover.

[jonjohnsonjr]

I like Remove and Replace as primitives, but these should probably be part of the mutate package. Ideally, everything that does mutations is in the mutate package... but layout has a couple of exceptions already. The Append stuff was an oversight on my part that I wish I could walk back. The WriteFile etc. enable some things that would be otherwise impossible, so that was a concession of purity. For things like this PR (everything is generic except the source and the sink), the calling should look something like:

layout.Write(path, mutate.Remove(layout.FromPath(path), matcher)) and
layout.Write(path, mutate.AppendManifests(mutate.Remove(layout.FromPath(path), matcher), addendum)).

In the same way we have mutate.AppendManifests, we should have mutate.RemoveManifest[s]. You can just shove a Matcher in here and use that in compute().

If we get cute with type-casting, we could just have mutate.Remove that acts on both v1.Image and v1.ImageIndex, but we could also do mutate.RemoveManifests and mutate.RemoveLayers.

For your replace implementation, I would really expect:

Layout([A, B, C]).ReplaceImage(D, Matcher(B)) == Layout([A, D, C]) and also

Layout([A, A]).ReplaceImage(B, Matcher(A)) == Layout([B, B]),

but these are currently just equivalent to Append(Remove(Matcher(...), X).

If just Remove then Append is your desired behavior, how would you feel about adding Remove to mutate, then doing the remove && append in your client code? I'd be okay with a "replace" for mutate if it did in-place and not just remove + append.

[deitch]

The Append stuff was an oversight on my part that I wish I could walk back.

Is it really? Given that Append*() does:

1. Write the Image/Index contents to the layout dir
2. Append the descriptor to the root index
3. Write that index to the layout dir as an updated index.json

how would you place that in any other package?

If we get cute with type-casting, we could just have mutate.Remove that acts on both v1.Image and v1.ImageIndex, but we could also do mutate.RemoveManifests and mutate.RemoveLayers.

Please not, or at least not yet. The need for distinct Image and ImageIndex in various places is an issue throughout, as you raised. I would rather be consistent here with what we do elsewhere, and if we find a way to unify it, then do it everywhere at once.

For things like this PR (everything is generic except the source and the sink), the calling should look something like:

layout.Write(path, mutate.Remove(layout.FromPath(path), matcher)) and
layout.Write(path, mutate.AppendManifests(mutate.Remove(layout.FromPath(path), matcher), addendum)).

I really don't understand (which leaves me concerned others might not as well). I fully get that I can open a layout and get a Path and I can AppendImage or AppendIndex, etc.

The above is complex to understand, especially if I already have a Path, that I want to modify.

If just Remove then Append is your desired behavior, how would you feel about adding Remove to mutate, then doing the remove && append in your client code? I'd be okay with a "replace" for mutate if it did in-place and not just remove + append.

I have no objection to adding Remove and Replace to mutate, but then what goes in here that makes it easy to understand.

What do I, as a newbie user of this library, call to remove or replace a manifest from a Path?

With this PR - or a variant - I can do:

p, err := layout.FromPath(dir) 
// OR
p, err := layout.Write(dir, empty.Index)
// and then
p.ReplaceDescriptor(desc, someMatcher)

but now you are saying Replace*() goes in mutate, so I end up with:

p, err := layout.FromPath(dir)
// OR
p, err := layout.Write(dir, empty.Index)
// and then
err = p.WriteImage(img)
index, err := p.ImageIndex()
// generate the descriptor from img, redoing everything that is at https://github.com/google/go-containerregistry/blob/3c3abfb82ca884fb7cc6db0efe77264f1c817f1f/pkg/v1/layout/write.go#L42-L69
desc := generateDescriptorFromImage(img)
index, err = mutate.ReplaceDescriptor(desc, someMatcher)
err = p.WriteIndex(index) // this is a function that does not exist now

That is a lot messier and a lot more work than the original

If you say, you want to keep the conveniences in layout, just move some of the core logic into mutate, I can see that, but that will get messy with the Option as well.

[deitch]

I think I figured out what you meant @jonjohnsonjr . Take a look at it now. All of the mutation is done in mutate, while the convenience functions wrap them in layout.

[codecov-io]

Codecov Report

Merging #848 (f336189) into master (d29eb11) will decrease coverage by 0.08%.
The diff coverage is 56.86%.

@@            Coverage Diff             @@
##           master     #848      +/-   ##
==========================================
- Coverage   74.33%   74.24%   -0.09%     
==========================================
  Files         107      107              
  Lines        4523     4598      +75     
==========================================
+ Hits         3362     3414      +52     
- Misses        661      675      +14     
- Partials      500      509       +9     

Impacted Files	Coverage Δ
pkg/v1/layout/write.go	47.53% <44.44%> (-0.89%)	⬇️
pkg/v1/mutate/index.go	82.82% <83.33%> (+5.05%)	⬆️
pkg/v1/mutate/mutate.go	72.50% <100.00%> (+0.52%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d29eb11...3af4f8f. Read the comment docs.

[deitch]

OK, we have #841 in, which makes using this easier. Any other comments on this?

[jonjohnsonjr]

Use partial.Descriptor here, for some bad reasons and some good reasons.

[deitch]

I don't understand

[jonjohnsonjr]

Sure, sorry I should have provided more details. It's two things:

1. This is ~9 lines of code to produce a v1.Descriptor for an artifact, and we've consolidated that into a partial.Descriptor method, just for brevity.
2. There are situations where you have some artifact, say a v1.Image, which was originally referenced by a v1.ImageIndex descriptor that contained platform information. If you just reconstruct the v1.Descriptor as you're doing here, we lose that information. Thus, artifacts can optionally implement Descriptor() (v1.Descriptor, error) to return the original v1.Descriptor by which they were referenced. This is especially interesting for foreign layers. See here and Add partial.Descriptor #654.

It unfortunately needs to live in partial because the way we wrap things in partial means we shadow the wrapped artifact, but fortunately it makes sense to live there anyway.

[jonjohnsonjr]

Oh, I see that I bungled the multiline comment thing in GitHub. This comment should apply to lines 139-158.

[jonjohnsonjr]

I really don't understand (which leaves me concerned others might not as well).

I agree this is a problem, which maybe we can solve with docs and examples, but the "model" for how this library works will be hard to change without a big, breaking overhaul.

think I figured out what you meant

I think we're roughly on the same page, but if there's anything from this comment that I still need to address, let me know.

[deitch]

I think we're roughly on the same page

I didn't understand the three comments above. Can you clarify?

[deitch]

OK, now I think I got what you meant. See the latest.

[jonjohnsonjr]

Nice, I think this is a lot cleaner -- with tests this would lgtm.

[deitch]

OK, lots of tests. Back at you @jonjohnsonjr

[deitch]

And rebased

[jonjohnsonjr]

Small suggestion to dedupe some code now that ReplaceDescriptor is removed, otherwise lgtm, feel free to push back if I'm missing something with my suggestion.

[deitch]

Oh, yeah, I like that. Nice and clean.

[deitch]

Updated with your suggestions @jonjohnsonjr ; back at you.

[jonjohnsonjr]

Okay, fine, one last thing before I merge this. Sorry... thank you for your patience.

We could defer this to be a little bit lazier (happens in compute instead of here) so that we don't return an error here, which would make using it a little easier (one less error check, matching AppendManifests).

This would allow for:

	ii = mutate.AppendManifests(mutate.RemoveManifests(ii, matcher), add)

Not a huge deal, but something I often find frustrating when using this library is that I am constantly checking for errors for things that don't really need to return an error, and I wish I could go back and fix that.

[deitch]

I understand what you are trying to do. If this only returns a v1.ImageIndex and not (v1.ImageIndex, error), then you have the option to chain them.

I just don't see how.

The source for this is:

// RemoveManifests removes any descriptors that match the match.Matcher.
func RemoveManifests(base v1.ImageIndex, matcher match.Matcher) (v1.ImageIndex, error) {
	ii, err := base.IndexManifest()
	if err != nil {
		return base, err
	}

	// create a list without any existing ones
	manifests := []v1.Descriptor{}
	for _, manifest := range ii.Manifests {
		if !matcher(manifest) {
			manifests = append(manifests, manifest)
		}
	}
	return &index{
		base:     base,
		replaces: manifests,
	}, nil
}

The only potential error is from getting base.IndexManifest(), but it could return an error. How would we handle it?

[jonjohnsonjr]

but it could return an error. How would we handle it?

Drop index.replaces, for an index.removeMatcher and set it to the match.Matcher we pass in, then instead of doing:

        manifests := manifest.Manifests
	if len(i.replaces) > 0 {
		manifests = i.replaces
	}

in index.compute, do:

	manifests := []v1.Descriptor
        for _, desc := range manifest.Manifests {
            if !i.removeMatcher(desc) {
                manifests = append(manifests, desc)
            }
        }

We're basically just deferring that IndexManifest() call until compute() is called.

RemoveManifests just becomes:

// RemoveManifests removes any descriptors that match the match.Matcher.
func RemoveManifests(base v1.ImageIndex, matcher match.Matcher) v1.ImageIndex {
	return &index{
		base:     base,
		removeMatcher: matcher,
	}
}

[deitch]

I think I got what you had in mind. I added it as another commit, so I can roll back if that is not it. Take a look and comment here. If it is, I will squash the two commits.

[jonjohnsonjr]

This is just unused now, right? We can drop index.replaces.

[deitch]

We can, but having figured it out how to work it, maybe we should just leave it?

[jonjohnsonjr]

Having an unreachable/untested path is a bit strange to me, I'd just drop it.

[deitch]

OK done.

[jonjohnsonjr]

I don't think this needs to be a list (yet?). In theory, we may be able to optimize some things in the future to do introspection and directly modify these fields instead of returning an entirely new mutate.index each time, but right now this can only be a single matcher, I think.

[deitch]

It probably can, but it parallels adds, for a clean API, and I made it similarly variadic. Let's just leave it.

[jonjohnsonjr]

Eh... I'm not sure.

I do like that this mirrors AddManifests, but I think making this variadic is going to be a source of confusion. There's more than one way to compose matchers, and this interface forces the operator to only ever be "or".

You can achieve the same thing now with multiple calls to RemoveManifests, which is definitely less efficient, but you can also just implement a match.Matcher that matches multiple predicates.

Having RemoveManifests accept multiple matchers encodes some matcher composition within mutate, and I'd rather that live in match if we need it.

Of course, you know that I will advocate for adding match.Or and match.And to solve this, but we're not there yet :P

[deitch]

OK done. And rebased.

[jonjohnsonjr]

Thanks :)