Test Caes for Interop Scenarios and Comparison Grid #84
Conversation
Signed-off-by: Steve Lasker <stevenlasker@hotmail.com>
Signed-off-by: Steve Lasker <stevenlasker@hotmail.com>
|
SVG's on this are rendering strange with Firefox on Linux |
There was a problem hiding this comment.
Overall, this comes across as a product pitch to use ORAS Artifacts, and not really focusing on the scenarios to solve and what the alternatives are. Maybe that's appropriate for the oras-project, which is why I was pushing to have this discussion in a more neutral location so we could evaluate options to solve the end user issue without biasing towards any one vendor.
|
|
||
| ### Scenarios 4: Support for Tag Locking (Immutable Tags) | ||
|
|
||
| Many registries support the ability to lock a tag to a given digest. ([acr: lock an image by tag](https://docs.microsoft.com/en-us/azure/container-registry/container-registry-image-lock#lock-an-image-by-tag), [ecr: Image tag mutability](https://docs.aws.amazon.com/AmazonECR/latest/userguide/image-tag-mutability.html)]). This pattern precludes the ability to use a alias manifest (change the tag to an index) to represent a level of indirection for a given tag. |
There was a problem hiding this comment.
This pattern precludes the ability to use a alias manifest (change the tag to an index)
This seems to be making a claim rather than describing the requirements of the scenario. As a counter, you can push an index with signature/SBOM data already attached, but depending on the implementation you may not be able to later extend it. That may be a feature for those that want immutable tags.
There was a problem hiding this comment.
The scenario accounts for registries that support tag locking, where a tag is locked to a specific digest.
ORAS Artifact reference types provide for adding references after an artifact is pushed. If the tag is locked to a digest, how would you push a new manifest (index) that links a signature and image together, using the same tag?
There was a problem hiding this comment.
+1 to @sudo-bmitch. I think we can collapse non-mutating tags and Tag Locking since this is all about customer not being able to modify their tags.
|
|
||
| ### Scenario 1: Non mutating tag/digest | ||
|
|
||
| The development team uses tags or digests for their container references in their service deployment documents. Adding a signature, SBoM, scan result, attestation or other reference must not change the tag or digest references. |
There was a problem hiding this comment.
Immutable tags is a duplicate of scenario 4. Immutable digests is the definition of a digest.
There was a problem hiding this comment.
I was trying to find the right granularity, so some fallback proposals may account for a subset. This was one of the first requirements of Notary v2, so it simply covers the baseline here. All fallback solutions should be able to implement this as a baseline.
Scenario 4 accounts for proposals that may want to redirect the tag. Tag aliases is a viable option, with the limitation that it can't account for tag locking. So, these are separate scenarios to track.
|
|
||
| ### Scenario 6: Multi-blob Support | ||
|
|
||
| [OCI Artifacts][oci-artifacts] leverages the generalized distribution design for an array of `blobs`. This pattern enables flexibility for artifacts to persist a small single file to multiple multi-gigabyte files across multiple blobs. Multiple blobs enables several scenarios, including concurrent downloading, or blob contents to be placed across different directories. Reference types should benefit from this flexibility, and not be restricted to an assumption that a reference type may only have a single blob. |
There was a problem hiding this comment.
I'm not seeing the problem being solved by this one as an end user. Is the problem multi-blob support or are we trying to say a goal is network efficiency?
There was a problem hiding this comment.
Different artifact types implement multiple blobs (layers). For example, Helm uses two layers of the OCI Image manifest to implement the chart and another layer for provenance. The runtime image format uses multiple blobs to provide concurrent network downloads and registry de-duping. The OCI Distribution spec does a great job by enabling this generic capability that should be maintained.
A particular fallback proposal may not be capable of supporting this capability, while others may. This is why its captured for a full comparison of proposals.
There was a problem hiding this comment.
In the cases described here, multiple blobs isn't really the problem being solved, it's the solution being proposed to the problems. E.g. Helm needs a way to package their chart and associated provenance. How they do that should be up to the various proposed solutions rather than specifying that we must have multiple blob support in every solution.
There was a problem hiding this comment.
Helm uses multiple blobs today, and there's value in how the image spec handles this. Which is what we carried forward in the ORAS artifact manifest.
So, the test case here captures the ability for someone to interop (round trip data) stored with oras artifact manifest into a 1.1 registry.
Helm is an example of how the value was used. Which, I'd like to maintain as ORAS Artifact manifest is a superset of capabilities. The list of test cases doesn't mean all proposed solutions must support this, but it is a tick of comparisons to show which do, and which don't.
There was a problem hiding this comment.
If a proposal solves the Helm use case without directly supporting multiple blobs in an artifact, do we not give them a check?
There was a problem hiding this comment.
Would it require the artifact author, helm in this case, to make a change?
Are we trying to fairly compare proposals, or just give everyone a trophy?
The lis is meant to give a fair comparison that maintains the vast list of capabilities. Should we impose changes on all the artifact authors, the registry libraries or the comparatively few registry products and projects?
There was a problem hiding this comment.
Should we impose changes on all the artifact authors, the registry libraries or the comparatively few registry products and projects?
I would guess there are even fewer registry libraries than registry implementations.
If we could put the onus of improving the experience for both clients and registries on the libraries, I think that would be a win (and I say this as a library maintainer, who would have that onus!)
There was a problem hiding this comment.
I hope we can abstract much of this in the registry libraries. And, I'm not suggesting all creative ideas shouldn't be explored. I'm just trying to capture a true list of measurements, so we give credence to the proposal that meets the most, with the least pain.
The thing this list, and the offline feedback helped me realize was: we do need a prioritized list to identify which are the blockers.
This is why I first changed the title from fallback to scenarios, then test cases.
Fallback suggests one way, down conversion. Like taking a 4k video and reducing it down to something you can stream on your phone. We really need round-tripping, with no data loss. That's the P0, core scenario. If the query apis need to be implemented differently in various registry clients, if a 1.1 registry needs polluted tags, those are experiences that are reasonable. It enables the user to have their data round-tripped, no loss, and if they really want a better experience, upgrade.
Just like older cars having the cassette adapters. If you want a nice integrated experience, upgrade. If not, you can still listen to the music, with a bit of a messy experience.
|
|
||
| ### Scenario 12: Filtered Referrers by Annotation | ||
|
|
||
| As registries are used to store multiple signatures or attestations, these are persisted as the same `artifactType`. To provide generic (non-artifact specific knowledge) referrer results, the registry MAY be capable of supporting filtering by `artifactType`, and named `annotations`. Example: `registry.acme-rockets.io/v2/net-monitor/_oras/artifacts/referrers?digest={$DIGEST}&artifactType={$ARTIFACTTYPE}&annotation={$ANNOTATION}`. |
There was a problem hiding this comment.
This also seems to be discussing features that don't exist in fallback support scenarios.
There was a problem hiding this comment.
That may be true, which is why it's important to list, so we have a full view of what a particular fallback may support, compared to the ORAS Artifact manifest scenarios.
| ### Scenario 13: Multi-Region Push Contention | ||
|
|
||
| Several registries support geo-replicated instances, where a reference type may be pushed from different nodes to different replicas. A build system may push to `registry.acme-rockets.io/net-monitor` from an east us region, while a security scanner may be load balanced to operate in west us region and the UK SBoM policy validation service may operate in a west UK region. While an argument may be made these workflows may be adequately time spaced to avoid contention, diagnosing these types of problems are frustrating to users, costly to track and implies a lack of trust in the system when they do surface. Most registries that implement replication are based on eventual consistency, as registries maintain immutable content. | ||
| The design of independent reference artifacts avoids contention and race condition as all individual artifacts have eventual consistency, where the registry manages the index of references external to the documents and blobs uploaded. |
There was a problem hiding this comment.
This seems to be a recommendation for a specific solution.
There was a problem hiding this comment.
This one was a bit tricky to write. When discussing possible solutions, such as the eTag proposal, I think folks were thinking about the contention of a single region, where two concurrent requests to the same endpoint (region) can be resolved. In a distributed replicated scenario, we need to think about eventual consistency. This isn't just a feature of one registry, but several different registry providers.
The ORAS Artifacts design is based on each artifact being an individual set of content the registry indexes. These are designs that serve this scenario that we just need to think about when comparing fallback options.
|
|
||
| ### Scenario 14: OCI Index References | ||
|
|
||
| A signature, attestation, or possibly platform aggregated SBoM may be associated with an OCI Index. While it's debateable if a single SBoM _should_ be used to represent multiple architectures, as opposed to each architecture has its own SBoM, the reference type design should be flexible for other reference types that may be more applicable. For example, [CNAB](https://cnab.io)s are independent artifacts that happen to use OCI index as their representation. |
There was a problem hiding this comment.
An index can be used for more than only multi-platform images.
There was a problem hiding this comment.
Agreed, which is why I captured the CNAB scenario. Did you read that we shouldn't compare designs that would support an OCI index?
|
|
||
| ### Scenario 15: Tag Listing Clarity | ||
|
|
||
| The ORAS Artifact manifest enables a graph of artifacts to be represented through a top level tag. This maintains existing automated and human workflows where a tag represents the root artifact the user and/or workflow depend upon. |
There was a problem hiding this comment.
First sentence is another product pitch.
There was a problem hiding this comment.
It's not a vendor product, but a CNCF Project that we're evaluating fallback options. What's the concern?
|
|
||
| ## Comparison of Options | ||
|
|
||
| A matrix of designs that support each requirement. The options fall into two buckets: |
There was a problem hiding this comment.
I think we need more than a 1 sentence explanation of each option. I'd include enough description here to understand how each works at a high level without needing to follow links. Probably dedicate a full section of options since that's the other axis of the later table and we included a detailed description of each scenario above.
There was a problem hiding this comment.
Completely agree. Open to suggestions on how we implement this. The grid would capture the summary, and there are likely several sentences for how a proposal may, may not, or may support with a workaround to the workflow or client implementation.
I'm thinking each proposal has a separate doc that can iterate independently, and we link to that doc from the grid. This way we have a summary, with details.
Ugh, the font, positioning conpat fun. I’ll convert to pngs |
Signed-off-by: Steve Lasker <stevenlasker@hotmail.com>
…cts-spec into ref-type-requirements
|
Pulling this discussion out of the line item comments:
Scenarios to me should cover a problem that a user is facing. And as a user, none of my problems today involve ORAS Artifacts. My goals are to push signatures, SBOMs, helm charts, etc, a registry without breaking my other workflows. And some of those registries may be managed by other groups, possibly by a 3rd party SaaS, where I have no ability to change the registry implementation. So I'm bothered to see the notary project's issue #112 moved from an implementation neutral place to solving it here where the answers are all biased to start from "ORAS Artifacts are the solution". It really feels like we're poisoning the well on this, especially since it presupposes one of many possible outcomes from the OCI Reference Types Working Group. |
|
Everything you've outlined is why we're discussing how we enable fallback in ORAS so the user doesn't have to think about the details. We want to enable an abstract way to get the benefits of a registry that has rich support for references, with interop of registries that don't. A user should be able to move between those with some level of fidelity. The graph the ability for each to enable the scenarios. Notary v2 has a dependency on ORAS and ORAS Artifacts to manage registry interactions. How is this any different than cosign taking a dependency on sigstore or crane? ORAS is vendor-neutral and the underpinnings registry interactions for Notary. Why is this a problem? As the OCI Reference working group proceeds, we can consider the outcome. Months ago, it was agreed we'd proceed with ORAS in parallel so Notary V2 can make progress. These are parallel efforts, where ORAS Artifacts is an implementation that's taking feedback, based on use. |
Just going to clarify this point - cosign depends only on the existing, 1.0 version of the OCI Image and Distribution specifications, with no extensions. |
My desire in looking at fall back solutions is how to handle registries where ORAS Artifacts today is not an option. So the productive path forward to me is to start from the view that there's a registry without any of this API support, assume that we don't have any of the possible solutions that could result from the Reference Type WG, and evaluate possible solutions to the problem. Then once we decide what makes the most sense, if you want to implement that in ORAS Artifacts, great. But starting the question where you already have the answer you want to push is concerning to me, both as a user, and as someone that would like to interoperate with other code that won't be adding an ORAS dependency. Cosign is a good example, because I don't recall them pushing that I need to implement go-containerregistry to support their signing spec, it's only to compile their specific implementation. And since they stuck to fairly standard OCI primitives, I've been able to write my own tooling that pulls and copies their signatures with the image across registries/repositories, none of which required that I look at crane code to understand their spec. |
Cosign uses "hacks", your words. The proposal here are ways to avoid those hacks and bring the APIs forward to support the scenarios outlined. The comparison here is cosign depends on new sigstore services and uses crane for registry operations.
ORAS Artifacts is a spec for how to store reference types, with all the scenarios captured here. |
Just correcting this point, not attempting to weigh on on the rest of this - cosign does not depend on any sigstore services. It can use some optionally, but basic container signing and verification depends only on the existing, 1.0 version of the OCI Image and Distribution specifications, with no extensions. Cosign does not depend on crane. |
Signed-off-by: Steve Lasker <stevenlasker@hotmail.com>
Signed-off-by: Steve Lasker <stevenlasker@hotmail.com>
|
Hey folks - Does it make sense to schedule a time to talk through these scenarios? I've taken a read myself and have several questions and I personally would benefit from a walk through from Steve on the intent and goals he had in mind so that I can correctly frame the context while pondering. I seldom see documents/proposals of this length and on topics that I don't have any prior art as a frame of reference hard to parse without at least having a Q&A with the author. Just a suggestion and I'm happy to do the leg work getting a time setup. The goals of the call would be to walk through the doc with Steve and get the context and intent. Then offer any feedback on how the document to be better structured so that the information is correctly conveyed. From there we can break back out into the comments. |
|
I'm going to be frank here - that kind of meeting sounds great, but should that just happen in the Reference Types WG? Why have a separate effort and meeting here? As far as I can tell it's the same topic and mostly the same people. |
|
+1 to a meeting, and possibly even bringing this up in the context of the OCI Reference Types WG. If we do that, I'd love to get alignment between us all on this topic first, then bring it to that wider group as a united front, to get another round of feedback from them. Otherwise we risk distracting that goal and audience even more with this topic. |
SteveLasker
changed the title
|
Okay. I will work on getting a meeting scheduled. There's no reason why we can't continue discussing these scenarios in the context of the ORAS project. |
| @@ -0,0 +1,159 @@ | |||
| # Reference Type Requirements | |||
There was a problem hiding this comment.
Recommend Retitling this to Interoperability Requirements or Referrers Interoperability Requirements to keep it consistent with the ORAS project terminilogy.
| @@ -0,0 +1,159 @@ | |||
| # Reference Type Requirements | |||
|
|
|||
| The ability to distribute and consume supply chain artifacts has driven a new set of requirements for adding information within an OCI Registry. [OCI Artifacts](https://github.com/opencontainers/artifacts/) enabled new, independent artifacts. [ORAS Artifacts](README.md) enables supply chain scenarios such as signatures, systems bill of materials (SBoM), security scan results and attestations. | |||
There was a problem hiding this comment.
I think we can remove this section since it only restates the project goals and why we need the artifacts spec. To reduce vebiage maybe we could just start with the need for interoperability and jump to the remaining sections.
|
|
||
| The [ORAS Artifacts spec](https://github.com/oras-project/artifacts-spec/) accounts for these new reference types, however it does require support from registries to implement the new [artifact manifest](artifact-manifest.md), and the new [referrers api](manifest-referrers-api.md) to discover artifacts that refer to a given digest and/or tag. | ||
|
|
||
| To account for registries that have not yet implemented ORAS Artifacts, a fallback design will be provided. The assumption is the fallback will have some tradeoffs, as a fallback that implements the full set of requirements would question why a new manifest and referrers api would be required. |
There was a problem hiding this comment.
Maybe retitle fallback to interop?
|
|
||
| To account for registries that have not yet implemented ORAS Artifacts, a fallback design will be provided. The assumption is the fallback will have some tradeoffs, as a fallback that implements the full set of requirements would question why a new manifest and referrers api would be required. | ||
|
|
||
| The following captures the scenarios as a comparison for different proposals. The proposals should account for zero changes to registries that implement the [OCI Distribution-spec 1.1 ](https://github.com/opencontainers/distribution-spec/releases/tag/v1.0.1) to the full [ORAS Artifacts spec](README.md). Based on a set of proposals, a fallback design will be chosen, documented so ORAS Artifacts may be promoted to/from OCI 1.1 registries to registries that implement the ORAS Artifacts spec. |
There was a problem hiding this comment.
Recommend making this the first requirement. To me zero change sounds quite strong.
For e.g. ORAS clients should be able to pull/push/discover artifacts with current registry operators that align with distribution spec 1.1 or higher.
|
|
||
| As reference: ORAS artifact reference type scenarios are covered [here](scenarios.md). | ||
|
|
||
| ### Scenario 1: Non mutating tag/digest |
There was a problem hiding this comment.
Digests cannot change so would recommend removing the term digest from here.
|
|
||
| ### Scenario 2: Pushing Independent Reference Types | ||
|
|
||
| The `net-monitor:v1` image is pushed to the registry as part of a build system. A [detached signature is pushed](./scenarios.md#notary-v2-signatures) attesting to where the image was built. The detached signature is pulled, independently from the image to validate the signature without having to pull the container image. |
There was a problem hiding this comment.
attesting to where the image was built. This makes claims about the signature which might be better to avoid here.
|
|
||
| ### Scenarios 4: Support for Tag Locking (Immutable Tags) | ||
|
|
||
| Many registries support the ability to lock a tag to a given digest. ([acr: lock an image by tag](https://docs.microsoft.com/en-us/azure/container-registry/container-registry-image-lock#lock-an-image-by-tag), [ecr: Image tag mutability](https://docs.aws.amazon.com/AmazonECR/latest/userguide/image-tag-mutability.html)]). This pattern precludes the ability to use a alias manifest (change the tag to an index) to represent a level of indirection for a given tag. |
There was a problem hiding this comment.
+1 to @sudo-bmitch. I think we can collapse non-mutating tags and Tag Locking since this is all about customer not being able to modify their tags.
| An artifact may have periodic scans, multiple signatures, and multiple attestations. As the artifact is built in the dev registry, it may be promoted through staging to production. It may be made public for others to consume. At each stage, a set of reference types will exist, which build up over time. As an artifact is promoted, the consumer may only care about the most recent scan result, specific signatures, or a specific set of attestations. To promote the subset of the graph across registries, each reference must be independent, allowing a filtered graph to be promoted. | ||
|
|
||
| This requirement is highlighted in the fallback scenarios as it makes using a single manifest to aggregate multiple references a challenge. If a single manifest (or OCI Index) is used to represent multiple artifacts, how is the subset promoted? Are new manifests/indexes expected to be re-created? If so, how does the graph of references maintain their `subject` reference as a digest? | ||
|
|
There was a problem hiding this comment.
OCI index is an implementation detail. Recommend scoping this section to just the requirement for now and I think the first paragraph capture is well.
|
|
||
| This requirement is highlighted in the fallback scenarios as it makes using a single manifest to aggregate multiple references a challenge. If a single manifest (or OCI Index) is used to represent multiple artifacts, how is the subset promoted? Are new manifests/indexes expected to be re-created? If so, how does the graph of references maintain their `subject` reference as a digest? | ||
|
|
||
| ### Scenario 6: Multi-blob Support |
There was a problem hiding this comment.
I'm not following this at this point. Probably need to align with a user story here. Multi-blob is an implementation detai.
There was a problem hiding this comment.
If an artifact author implements ORAS Artifact and takes advantage of the multi-blob support, how would we compare two proposals? One may club the blobs into a single manifest, while another keeps them separate? If multiple blobs can be intermixed for 1.1 interop, and separable to meet filtered promotion or merging, it could still meet the multi-blob support scenario.
|
|
||
| This requirement is highlighted in the fallback scenarios as some proposals have implemented an aggregate of artifacts in a single manifest. For example, as additional signatures are added, each signature is persisted as a different blob within the same manifest. This design precludes scenarios 1-5. It may be a reasonable choice for fallback support, but highlighted as a delta between a degraded fallback and an optimal, long-term experience. | ||
|
|
||
| ### Scenario 7: Annotation Only Support |
There was a problem hiding this comment.
Can we describe this a bit more? What is the user story as to how the client might use this kind of artifact?
There was a problem hiding this comment.
Users may wish to add additional metadata to existing artifacts, such as an expiration date
I was trying to balance the abstract, but a concrete example. In this example, I can add some metadata to the net-monitor:v1 image, after the image was pushed. It leaks implementation detail that I want to avoid having to push a blob, just to get a simple name:value pair added to the net-monitor:v1 image. Is that a good enough example? How would you re-word that?
SteveLasker
changed the title
From sigstore/cosign docs:
|
|
Not sure the point you're trying to make here. Crane is a CLI tool, go-containerregistry is a library. cosign uses the go-containerregistry library to interact with OCI registries, like many other things: https://github.com/justincormack/sign-index/blob/main/pkg/signing/signing.go#L15 |
This is a category error. Using a different client library is not the same as forking the protocol. |
|
cosign uses go-container registry (crane as the cli) for working with a registry. @jonjohnsonjr what fork of a protocol are you referring to? |
Still not sure what this part means. |
The question above was, (paraphrased): "why does This is no different than There's value in the factoring. |
A list of scenarios to consider for various fallback options, supporting OCI 1.1 based registries.