match package with matcher and utils to filter v1.Descriptors; v1.Platform.Equals utility

[deitch]

Starts to address #821

• creates a ManifestMatcher and utilities that can be used to find manifests that match the provided matcher
• provides convenience matcher utilities to match by annotation, media-types, name (specific annotation), platform
• provides a FindManifests that takes a v1.ImageIndex and returns the manifests in it that match. This is useful in its own right, but will be even more useful later for tree walking.

Some questions and follow-ons:

• is this the right location for it? partial was suggested but seems, well, off. This is a utilities package, but mostly seems to have internal utils, although it still works better than partial. I think this could handle being renamed to util since v1/v1util is a mouthful.
• while this is really simple, it doesn't provide a good option for depth. E.g. we might want in the future to be able to walk down and filter based on depth. I don't think that matters much now because (a) this is just the manifests, while the walker can handle depth; (b) this is simple and clean; (b) we haven't proven we have a real use case for infinite depth. Theoretically, and index can include an index can include (ad infinitum), but in practice we only ever see index->manifest or, in the case of v1.layout, index->index->manifest. That can be handled separately.

Once this is good, we can start to add the next parts.

cc @jonjohnsonjr

[deitch]

Oh, notice with the MediaTypeMatcher, which will allow you to do what you wanted with matching just the index and manifest blobs, and skip the rest. :-)

[imjasonh]

This seems to overlap a bit with #797 -- I don't feel strongly about either approach, but I don't think both should exist.

Re: naming, I don't really love v1/partial or v1/v1util. If we expect there to be a number of provided filters, and we expect them to be used for walking a graph of manifests, perhaps these should go in their own package, e.g., v1/filter or v1/walk?

[deitch]

Hey @imjasonh 👋🏻

@jonjohnsonjr and I had a really long talk yesterday. He pointed out #797, as well as other things, see #821, as well as the ability to get the right image resolved and such for layout like we do for remote.

Agreed totally on where to put it. @jonjohnsonjr asked that I just put it up there in some package, and then we can use the PR to discuss where it should do (as we are doing).

As for which one? 🤷‍♂️

This includes FindManifests, which eventually will be used like remote.Get. To recap some of our Hangouts chat:

For remote registry, my entrypoint is an image ref string, which always returns a descriptor, either an index or manifest. That descriptor is captured as remote.Descriptor, on which I can call ImageIndex() and get it (if it is an index) or Image(). If it is an image, I get the v1.Image represented by the manifest, and on which I can call Config() and Layers() to get readers; if it is an index, it figures out the right manifest based on platform, and then gets me the right v1.Image.

Essentially, remote.Get() gets me a great starting point, remote.Descriptor, through which I can get everything I need.

For layout, it isn't that simple. There is no direct mapping of string ref -> root (index or manifest). If my index.json has the right annotations on one of the manifests, then it will work.

What we are looking for, then, is, something like:

p, err := layout.Path(root)
desc, err := p.Get(matcher) // or maybe GetDescriptor(p, matcher) 
desc.ImageIndex() // just like remote
desc.Image()          // just like remote

Does that help?

[deitch]

I have no idea what that one failed test is complaining about.

[jonjohnsonjr]

I have no idea what that one failed test is complaining about.

We went from an indirect dependency on opencontainers/image-spec to a direct dependency. If you run ./hack/update-codegen.sh it should update it for you, but basically our go.mod is out of sync.

[deitch]

Got it. Ran and updated and pushed.

[jonjohnsonjr]

Re: naming, I don't really love v1/partial or v1/v1util. If we expect there to be a number of provided filters, and we expect them to be used for walking a graph of manifests, perhaps these should go in their own package, e.g., v1/filter or v1/walk?

I'm a little torn here because this is useful both as a helper for packages implementing an ImageIndex (argument for partial) and as a new API for library consumers interacting with an ImageIndex (argument for new package?).

Given your precedent in #797, I'd lean partial, but I hate that package and wish we could get rid of it, honestly.

[jonjohnsonjr]

Started typing out "what if we wanted to match multiple names?" but it might be better to have an API for composing matchers... e.g.:

And(Or(NameMatcher("foo"), NameMatcher("bar")), Or(PlatformMatcher(arm), PlatformMatcher(arm64)))

Which would match anything images with name "foo" or "bar" that also have platform "arm" or "arm64".

func Or(matchers ...ManifestMatcher) ManifestMatcher {
  return func(desc v1.Descriptor) bool {
    for _, m := range matchers {
      if m(desc) {
        return true
      }
    }
    return false
  }
}

func And(matchers ...ManifestMatcher) ManifestMatcher {
  return func(desc v1.Descriptor) bool {
    for _, m := range matchers {
      if !m(desc) {
        return false
      }
    }
    return true
  }
}

Now we've got a whole language, and we maybe want a new package for this craziness, e.g.:

match.And(
  match.Or(match.Name("foo"), match.Name("bar")),
  match.Or(match.Platform(arm), match.Platform(arm64))
)

[deitch]

That is pretty complex. I suspect for the vast majority of use cases, people will be looking just for one name. This is the closest thing to remote.Get() I can think of. How about if we start with the simple matchers, and then create composites later?

[jonjohnsonjr]

Agreed, we shouldn't start here, but if we want to end up here, this is a good argument for pulling out matchers into a separate matching-only package so to make names a little less verbose and avoid polluting other package namespaces.

[deitch]

If that is the approach, I can pull it out into a matcher package. I am a little concerned about getting into a circular imports loop (matcher importing v1 importing matcher), but I think we will be ok.

Let me know.

[jonjohnsonjr]

matcher importing v1 importing matcher

This is why v1 is (almost) strictly structs and interfaces. There's some stuff for JSON [de]serialization in there as well, but it should be as small as possible and have as few dependencies as possible. Any real behavior should go in a different package.

[deitch]

Given your precedent in #797, I'd lean partial, but I hate that package and wish we could get rid of it, honestly.

So let's stay away from it.

new API for library consumers interacting with an ImageIndex (argument for new package?).

Could just go in a utilities package, but isn't that what v1util is?

[codecov-io]

Codecov Report

Merging #823 (fd68802) into master (b004c8a) will increase coverage by 0.17%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #823      +/-   ##
==========================================
+ Coverage   74.67%   74.85%   +0.17%     
==========================================
  Files         103      105       +2     
  Lines        4348     4379      +31     
==========================================
+ Hits         3247     3278      +31     
  Misses        619      619              
  Partials      482      482              

Impacted Files	Coverage Δ
pkg/v1/match/match.go	100.00% <100.00%> (ø)
pkg/v1/platform.go	100.00% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b004c8a...fd68802. Read the comment docs.

[jonjohnsonjr]

ManifestMatcher as a name is too specific, we can reuse this type to match any descriptor, which can reference arbitrary things. We could just call this a Matcher.

If we end up in a package called match or matcher or whatever, we may want to simply call this thing a Predicate to avoid stuttering.

[deitch]

we can reuse this type to match any descriptor, which can reference arbitrary things

That is a good point. Within Index, we are passing it through the descriptors in the manifests, but it can be for anything.

If we end up in a package called match or matcher or whatever, we may want to simply call this thing a Predicate to avoid stuttering.

Are we moving it to match or matcher package? I am happy to do it, we just need to decide.

Predicate

Bit of a mouthful. I am not sure people will understand it at first glance, or know to look for it. It is, well, a "matcher". I know it stutters, but it still is easier to understand than "predicate".

[jonjohnsonjr]

Are we moving it to match or matcher package?

I'd prefer match, simply because it's monosyllabic and shorter to type, but I'm open to arguments against it.

Bit of a mouthful.

Yeah that's a good point. I'd be fine with match.Matcher as well... predicate just feels so right, you know? 😉

I'll leave it up to you, since you'll be doing the heavy lifting. Matcher is two letters shorter as well, so it's roughly a tie in my vague, inexpressible, gut-feeling-based name-scoring belief system.

[deitch]

😂

Those of us with engineering or mathematical training and/or experience tend to belittle those gut feelings; it is a mistake on our part. Guts count.

In line with what you wrote earlier, this really is not a manifest matcher, but a descriptor matcher. So I put it in match package as Descriptor, but that feels wrong. It is a Descriptor matcher. But then it gets ugly: match.DescriptorMatcher? Of course, we might have an IndexMatcher and a ManifestMatcher and even BlobMatcher in the future. Thoughts?

[jonjohnsonjr]

Here's what I'd propose:

• Add a v1.Platform.Equals method.
• Create a new match package, probably pkg/v1/match.
  • Rename v1util.ManifestMatcher to match.Predicate (or match.Matcher).
  • Rename all the matchers from v1util.FooMatcher to match.Foo.
  • Rename v1util.MediaTypeMatcher to match.MediaTypes (note pluralization).
• Put FindManifests in partial (for now).
  • There's precedent for similar functions.
  • We will probably want to refactor a bunch of other packages to use it.
  • I'm not completely convinced this is the final API, and partial being a grab bag of stuff like this makes me feel like it's okay for it to live here for a bit, since it's immediately and obviously useful.

I imagine we'll start with something like these, maybe in partial:

	one	many	recursive
Image	FindLayer	FindLayers	WalkLayers
Index	FindManifest	FindManifests	WalkManifests

And end up with something like this, not sure where:

	one	many	recursive
generic	Find	Filter	Walk

We may want to just skip to the generic bits and do some type-checking (like in remote.MultiWrite) to parse the descriptors appropriately.

Maybe we should rename FindManifests to FilterManifests?

[deitch]

Responding to your suggestions:

Add a v1.Platform.Equals method.

Done.

Create a new match package, probably pkg/v1/match.

Done.

Rename v1util.ManifestMatcher to match.Predicate (or match.Matcher).
Rename all the matchers from v1util.FooMatcher to match.Foo.
Rename v1util.MediaTypeMatcher to match.MediaTypes (note pluralization).

I renamed it to Descriptor, so it is match.Descriptor, which (surprise) providers a matcher for v1.Descriptor. We might also have a matcher on other things.

My problem with it is that, on the one hand, it isn't a descriptor, but rather a matcher for a v1.Descriptor, so the name match.Descriptor is not quite right. On the other hand, match.DescriptorMatcher stutters.

In line with that, the other funcs are now match.DescriptorName(), match.DescriptorAnnotation(), match.DescriptorPlatform(), match.DescriptorMediaTypes()

Happy for some better naming suggestions. We could have sub-packages under match, so you get pkg/v1/match/descriptor.Matcher, pkg/v1/match/descriptor.NameMatcher, etc. I am not sure if that is too deep, and all of the matchers should be in the same match package?

Actually, this makes me realize that the package probably should be matcher and not match? Would pkg/v1/matcher.Descriptor (or pkg/v1/matcher/descriptor.Matcher) make more sense?

[deitch]

Put FindManifests in partial (for now).

You don't like it in partial, @imjasonh doesn't like it in partial, I don't like it in partial... why put it there?

[deitch]

Maybe we should rename FindManifests to FilterManifests?

The functionality isn't quite right for it. "Filter" almost always implies, give in a list, get back a (possibly abbreviated) list of the same kind. Here we are doing more; give in an index, get back a filtered list. It is a convenience function with more than just filter.

[deitch]

I imagine we'll start with something like these, maybe in partial:
And end up with something like this, not sure where

I imagine these make sense in v1util; isn't that what it is there for?

Actually, I would be quite happy to get this PR to the right state, merge it in, and then do the others in another run.

I only added FindManifests here as a functional use case, to see how we would use it. Of course, we should get it right.

I don't think the recursive ones work.

Image

WalkLayers() has no meaning. Once I have a manifest, layers cannot have children; they just "are". So FindLayer() and FindLayers() make sense, but WalkLayers() doesn't.

Index

WalkManifests(); what will I do with it? The goal, given an Index, would be to get all children (index or image) or descendants that match the matcher criteria.

• For children, we have FindManifest() and FindManifests(), which return just the manifest embedded in the index itself.
• For resolving image children (like remote.Descriptor.Image()), I would use Image(), which, under the covers, uses FindManifest().
• For resolving index children, I am not sure; we don't have a parallel in remote.Descriptor, since on a remote, an index never has children that aren't an image (even though, as you pointed out, it could).

I am struggling to find the recursive use case. I definitely see it in the other area (outside of this PR), where I want to WriteIndex() or AppendIndex() and want to filter out which blobs I write. But as a standalone? I don't see it.

[jonjohnsonjr]

This response got longer than I expected, so it might be somewhat incoherent. If I missed anything or you still disagree about something or have more questions, keep poking at it until we find consensus.

I renamed it to Descriptor ...

I think this makes sense if there are other things that we intend to match against, but given that Descriptors are intentionally a generic pointer to anything, and used throughout every artifact, I think it's fine to assume we only care about descriptors, especially for the sake of the API being cleaner. Interested in your thoughts here if you have other things in mind that would make sense for matchers.

matcher and not match

I really think those two letters will make a difference if we implement the stupid composition DSL I proposed, which I think is very likely something I'll eventually do.

I really like how e.g. match.MediaTypes reads, because I want it to match these media types.

The stutter on match.Matcher is kind of nice because it makes it stand out more.

I'm mostly using https://blog.golang.org/package-names as a guide here, which makes me lean towards just match and just match.Name etc. I think that's much cleaner to read, but again I'd listen to arguments.

You don't like it in partial, @imjasonh doesn't like it in partial, I don't like it in partial... why put it there?

I don't think there's a better place for it right now, but it feels silly to create a new package just yet. Putting this in partial lets us starting using it and experiment with it before we actually commit to interfaces. It's not so much "this is bad and partial is bad so let's put it in partial", but "this makes the most sense in partial and we don't care about polluting the partial API, so we aren't going to commit to too much by just adding one more convenience function".

The functionality isn't quite right for it. "Filter" almost always implies, give in a list, get back a (possibly abbreviated) list of the same kind. Here we are doing more; give in an index, get back a filtered list. It is a convenience function with more than just filter.

Yeah good point, I'd agree with that. Filter(v1.ImageIndex) v1.ImageIndex and Filter([]v1.Descriptor) []v1.Descriptor would make sense, but not mixing.

I imagine these make sense in v1util; isn't that what it is there for?

Eh, kind of. If you look at what's in v1util vs partial:

v1util is more similar to ioutil where it's operating on bytes/readers/etc and bridging io stuff to v1 structs.

partial is higher level, dealing with the actual business logic stuff for implementing the v1 interfaces and enforcing logical invariants.

Actually, I would be quite happy to get this PR to the right state, merge it in, and then do the others in another run.

Yeah, this is definitely my intention. I just want to think forward to other applications so that we can consider this API with more context. I want to merge whatever we're happy with as soon as we're happy with it so we can start using it.

I'm happy to merge parts of this we agree on (at this point, I think platform.Equals is the only uncontroversial change?) if you want to split them off into separate PRs, then we can keep discussing the rest.

WalkLayers() has no meaning. Once I have a manifest, layers cannot have children; they just "are". So FindLayer() and FindLayers() make sense, but WalkLayers() doesn't.

In practice that's usually true, but there's really nothing preventing someone from referencing another manifest or index from a layers descriptor: https://github.com/opencontainers/image-spec/blob/master/manifest.md

Perhaps the behavior when encountering something like that is undefined, but people often [ab]use the registry for all kinds of stuff, so I don't know that it's always safe to assume a layers descriptor is a leaf from an interface perspective. For most implementations, yeah it's probably safe to assume that.

For resolving index children, I am not sure; we don't have a parallel in remote.Descriptor, since on a remote, an index never has children that aren't an image (even though, as you pointed out, it could).

One thing I've suggested to a bunch of people (not sure if anyone actually did it) for air-gapped environments (etc.) was to just pull everything down into an OCI image layout, ship that via physical media, and re-hydrate a destination registry by just uploading the entire index. If you also include the index.json by pushing it as an index to the registry, you get this incredible added benefit of a single reference entrypoint to a DAG that encompasses all of the images in a given release (or what have you). Then you can just think of your entire registry as a single artifact that is portable via a format that is defined by a standards body. It's pretty nice.

I suggested this to the CNAB folks early on, and I think they didn't understand what I was saying because they (paraphrasing) suggested I was confused and this was unrelated, but I am pretty sure they have since adopted a similar approach.

Also, "never" is a pretty strong word, here's an index I just pushed to make my point:

$ crane manifest us-west1-docker.pkg.dev/jonjohnson-test/test/experiments | jq .
{
  "manifests": [
    {
      "digest": "sha256:8414aa82208bc4c2761dc149df67e25c6b8a9380e5d8c4e7b5c84ca2d04bb244",
      "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
      "size": 1854,
      "annotations": {
        "source": "debian"
      }
    },
    {
      "digest": "sha256:a9286defaba7b3a519d585ba0e37d0b2cbee74ebfe590960b0b1d6a5e97d1e1d",
      "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
      "size": 2080,
      "annotations": {
        "source": "busybox"
      }
    }
  ],
  "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
  "schemaVersion": 2
}

I am struggling to find the recursive use case. I definitely see it in the other area (outside of this PR), where I want to WriteIndex() or AppendIndex() and want to filter out which blobs I write. But as a standalone? I don't see it.

Certainly this is outside the scope of this PR, but in terms of discussing APIs I think it's an important use case to consider.

One thing that jumps to mind is identifying artifacts that are affected by a CVE or something. Imagine that we implement ko-build/ko#221 and add version information as annotations to an image. If there were a bug in ko and we wanted to identify affected images, it would be nice to Walk every descriptor in an OCI layout bundle or registry and flag any with the bad version.

Similarly, imagine one of our base images has a CVE, and we want to find any images with a specific bad layer in an OCI layout bundle. A simple Walk with a matcher on a digest would do it in a couple lines.

These are a bit contrived but they're not that far-fetched.

[deitch]

I am not going to try to hit all of this right now. I will do the following:

• we are in agreement on v1.Platform.Equals()
• I think we are in agreement on the Descriptor matchers in a match package. I can accept your argument that we are matching on descriptors, and match.Matcher and match.MediaTypes and match.Name all make sense.

So I will update this PR to restructure just those, and pull out the FindManifest. That will avoid the discussion of where it should go and what it should be named. As soon as this one is in (restructured, and without that), we can open a new PR on that, figure out where it goes and what it should be named.

[deitch]

there's really nothing preventing someone from referencing another manifest or index from a layers descriptor

Is that really legitimate? I don't read the spec that way. A layer is a leaf. Sure, people abuse it, but that is not our problem as an implementation. I don't know of a single registry that would support it, or containerd. Sure, you could place a manifest or index in there, but nothing would read it and walk the tree down.

[deitch]

These are a bit contrived but they're not that far-fetched.

No, they aren't that far-fetched. Still and all, I would think you would want something like:

Walk(desc v1.Descriptor, Handler handlerFunc)

that just walks down the tree. It looks for children on media-types that have children. I can see why you might do the same for layers, but I would just look at something and say, "you are an index, you have manifest children that I need to read; whether or not they have children depends on the manifest's media-type", following by, "you are a manifest, you have children that I need to read, but, no, they don't have children."

It certainly is simpler to start, and nothing prevents us from expanding it in the future.

[deitch]

OK, updated to just include:

• v1.Platform.Equal()
• everything in match, which has been renamed to, e.g. match.Matcher, match.Name, match.MediaTypes, match.Annotations, match.Platform

Removed the FindManifest for a separate PR once this is in.

If we are in agreement on the above, I can add some simple unit tests for each of the ~5 funcs we added. We need agreement that, subject to those tests, this is ready.

[jonjohnsonjr]

Still and all, I would think you would want something like ...

Oh we might have just been speaking past each other. I agree we would want a signature like:

Walk(desc v1.Descriptor, Handler handlerFunc) error

NOT this:

Walk(root ThingWithChildren, match.Matcher) []v1.Descriptor

Is that really legitimate? I don't read the spec that way. A layer is a leaf. Sure, people abuse it, but that is not our problem as an implementation. I don't know of a single registry that would support it, or containerd. Sure, you could place a manifest or index in there, but nothing would read it and walk the tree down.

I mostly agree with you. If we ever encountered something I described, we would almost certainly do the wrong thing or just return an error, but in the future that might change if we ever can agree on a better format than gzipped tarballs and json for representing filesystems.

[jonjohnsonjr]

If we are in agreement on the above, I can add some simple unit tests for each of the ~5 funcs we added. We need agreement that, subject to those tests, this is ready.

SGTM. This looks good (assuming tests and happy linters).

My last pedantic nit will be that the godoc comments should end with a period/full stop (or really any sentence-terminating punctuation mark, if you're feeling festive).

[deitch]

And now you have unit tests for everything added in this PR.

[jonjohnsonjr]

A few small nits but otherwise lgtm, thanks :)

[deitch]

Resolved everything except for the test package naming, to be discussed, but not such a big deal. Back at you @jonjohnsonjr

[jonjohnsonjr]

Code LGTM, do you mind updating the PR and commit titles to reflect that you've added the match package and platform.Equals? Just for my sake in the future when scanning logs.

Thanks for sticking through all this back and forth :)

[deitch]

Code LGTM, do you mind updating the PR and commit titles to reflect that you've added the match package and platform.Equals? Just for my sake in the future when scanning logs.

Sure, doing it right now. 30 seconds and it will be done.

[deitch]

Urgh, I thought I fixed it, but the wrong changes are on the wrong commits. Time for some git-foo. Hang on another minute.

[deitch]

Now I got it. Back at you @jonjohnsonjr

Thanks for sticking through all this back and forth :)

All good. I ma enjoying the conversations.

Once this is in, I will open a PR with one or more of the Find*() and move your and my comments over to it, and then we can resolve that part.

[jonjohnsonjr]

Thanks!

[deitch]

By the way, if I didn't say it before @jonjohnsonjr, this is pretty nice:

One thing I've suggested to a bunch of people (not sure if anyone actually did it) for air-gapped environments (etc.) was to just pull everything down into an OCI image layout, ship that via physical media, and re-hydrate a destination registry by just uploading the entire index. If you also include the index.json by pushing it as an index to the registry, you get this incredible added benefit of a single reference entrypoint to a DAG that encompasses all of the images in a given release (or what have you). Then you can just think of your entire registry as a single artifact that is portable via a format that is defined by a standards body. It's pretty nice.