-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feedback on mediaTypes for storing OPA bundles as OCI images #1413
Comments
This looks good to me. For the data file mimetypes, would it make sense to distinguish between JSON and YAML (OPA supports both)? There shouldn't be any issue using the OPA logo/icon in the registry. /cc @caniszczyk just to double check. |
I think the convention here is use subtypes. SO you can say "this is data" and "this is formatted in YAML". eg.
|
it’s fine
On Thu, May 9, 2019 at 6:48 AM Gareth Rushgrove ***@***.***> wrote:
I think the conversion here is use subtypes. SO you can say "this is data"
and "this is formatted in YAML". eg.
application/vnd.cncf.openpolicyagent.data.v1+json
application/vnd.cncf.openpolicyagent.data.v1+yaml
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1413 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAAPSIJJD2XOCALVZK3TNULPUQFRLANCNFSM4HLICY4A>
.
--
Cheers,
Chris Aniszczyk
http://aniszczyk.org
+1 512 961 6719
|
Hey Gareth, et all. Great to see new scenarios. |
Thanks for the feedback Steve. Looks like we had the same thing for the top level config. Modified versions of the layer suggestions below:
This assumes a single layer per file, mainly to facilitate easy uploading with oras push localhost:5000/my-bundle-bundle:latest \
.manifest:application/vnd.cncf.openpolicyagent.manifest.layer.v1.json \
helpers.rego:application/vnd.cncf.openpolicyagent.rego.layer.v1.rego \
deny.rego:application/vnd.cncf.openpolicyagent.policy.layer.v1.rego \
samples.json:application/vnd.cncf.openpolicyagent.data.layer.v1.json \ I assume that later on also using @jdolitsky I'd love your feedback on whether that makes sense? I'd imagine writing a custom uploader with some domain knowledge using oras as a library, but looking to punt that to a little later. |
Ahh, I see the confusion. You don't actually push a manifest by referencing a file. By pushing the oras push localhost:5000/my-bundle-bundle:latest \
--manifest-config ./config.json:application/vnd.cncf.openpolicyagent.config.v1+json \
helpers.rego:application/vnd.cncf.openpolicyagent.rego.layer.v1+rego \
deny.rego:application/vnd.cncf.openpolicyagent.policy.layer.v1+rego \
samples.json:application/vnd.cncf.openpolicyagent.data.layer.v1+json If you don't need a So, you don't actually need the
|
There are two manifests :) One is the OCI manifest for the bundle as a whole, which I'd missed out from the oras command example. The other is the
Ah, the first draft of your blog post had eg. application/vnd.cncf.helm.values.layer.v3.yaml so I thought this had changed. But it's the right way round in the final published post. Swapping back. Updates for both of those below:
And usage with Oras: oras push localhost:5000/my-bundle-bundle:latest \
--manifest-config ./config.json:application/vnd.cncf.openpolicyagent.config.v1+json \
.manifest:application/vnd.cncf.openpolicyagent.manifest.layer.v1+json \
helpers.rego:application/vnd.cncf.openpolicyagent.policy.layer.v1+rego \
deny.rego:application/vnd.cncf.openpolicyagent.policy.layer.v1+rego \
samples.json:application/vnd.cncf.openpolicyagent.data.layer.v1+json \ |
The world of manifests. This makes sense. |
@garethr your table of mediatypes + ORAS command in your last comment looks good to me, although seems the command has one extra media type than the table? It could maybe be the case that some bundle contain files/layers that others don't, and that's ok if the client knows how to handle it. For some hints of using ORAS as a Go lib, you can take a look at the docs. We are still working on putting together a bunch of examples for copy-pasta goodness. The following is how you might push an OPA bundle based on your spec: package main
import (
"context"
"fmt"
"io/ioutil"
"github.com/containerd/containerd/remotes/docker"
"github.com/deislabs/oras/pkg/content"
"github.com/deislabs/oras/pkg/oras"
ocispec "github.com/opencontainers/image-spec/specs-go/v1"
)
const (
// Remote reference
// Command used to run a local test registry on port 5000:
// docker run --rm -it -p 5000:5000 registry
RemoteRef = "localhost:5000/opa/mybundle:1.0.0"
// OPA-specific file names
// Command used to create dummy files created for test purposes:
// for f in ".manifest" "helpers.rego" "deny.rego" "samples.json"; do echo "hello world" > $f; done
OpenPolicyAgentManifestFileName = ".manifest"
OpenPolicyAgentRegoFileName = "helpers.rego"
OpenPolicyAgentPolicyFileName = "deny.rego"
OpenPolicyAgentDataFileName = "samples.json"
// OPA-specific media types
OpenPolicyAgentConfigMediaType = "application/vnd.cncf.openpolicyagent.config.v1+json"
OpenPolicyAgentManifestLayerMediaType = "application/vnd.cncf.openpolicyagent.manifest.layer.v1+json"
OpenPolicyAgentRegoLayerMediaType = "application/vnd.cncf.openpolicyagent.rego.layer.v1+rego"
OpenPolicyAgentPolicyLayerMediaType = "application/vnd.cncf.openpolicyagent.policy.layer.v1+rego"
OpenPolicyAgentDataLayerMediaType = "application/vnd.cncf.openpolicyagent.data.layer.v1+json"
)
func check(e error) {
if e != nil {
panic(e)
}
}
func getFileContents(fileName string) []byte {
b, err := ioutil.ReadFile(fileName)
check(err)
return b
}
func main() {
// Setup
var contents []byte
var layer ocispec.Descriptor
var layers []ocispec.Descriptor
resolver := docker.NewResolver(docker.ResolverOptions{})
memoryStore := content.NewMemoryStore()
// Add layers
contents = getFileContents(OpenPolicyAgentManifestFileName)
layer = memoryStore.Add(OpenPolicyAgentManifestFileName, OpenPolicyAgentManifestLayerMediaType, contents)
layers = append(layers, layer)
contents = getFileContents(OpenPolicyAgentRegoFileName)
layer = memoryStore.Add(OpenPolicyAgentRegoFileName, OpenPolicyAgentRegoLayerMediaType, contents)
layers = append(layers, layer)
contents = getFileContents(OpenPolicyAgentPolicyFileName)
layer = memoryStore.Add(OpenPolicyAgentPolicyFileName, OpenPolicyAgentPolicyLayerMediaType, contents)
layers = append(layers, layer)
contents = getFileContents(OpenPolicyAgentDataFileName)
layer = memoryStore.Add(OpenPolicyAgentDataFileName, OpenPolicyAgentDataLayerMediaType, contents)
layers = append(layers, layer)
// Push
fmt.Printf("Pushing OPA bundle to %s...\n", RemoteRef)
extraOpts := []oras.PushOpt{oras.WithConfigMediaType(OpenPolicyAgentConfigMediaType)}
manifest, err := oras.Push(context.Background(), resolver, RemoteRef, memoryStore, layers, extraOpts...)
check(err)
fmt.Printf("Pushed OPA bundle to %s with digest %s\n", RemoteRef, manifest.Digest)
} and the resulting manifest: {
"schemaVersion": 2,
"config": {
"mediaType": "application/vnd.cncf.openpolicyagent.config.v1+json",
"digest": "sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a",
"size": 2
},
"layers": [
{
"mediaType": "application/vnd.cncf.openpolicyagent.manifest.layer.v1+json",
"digest": "sha256:a948904f2f0f479b8f8197694b30184b0d2ed1c1cd2a1ec0fb85d299a192a447",
"size": 12,
"annotations": {
"org.opencontainers.image.title": ".manifest"
}
},
{
"mediaType": "application/vnd.cncf.openpolicyagent.rego.layer.v1+rego",
"digest": "sha256:a948904f2f0f479b8f8197694b30184b0d2ed1c1cd2a1ec0fb85d299a192a447",
"size": 12,
"annotations": {
"org.opencontainers.image.title": "helpers.rego"
}
},
{
"mediaType": "application/vnd.cncf.openpolicyagent.policy.layer.v1+rego",
"digest": "sha256:a948904f2f0f479b8f8197694b30184b0d2ed1c1cd2a1ec0fb85d299a192a447",
"size": 12,
"annotations": {
"org.opencontainers.image.title": "deny.rego"
}
},
{
"mediaType": "application/vnd.cncf.openpolicyagent.data.layer.v1+json",
"digest": "sha256:a948904f2f0f479b8f8197694b30184b0d2ed1c1cd2a1ec0fb85d299a192a447",
"size": 12,
"annotations": {
"org.opencontainers.image.title": "samples.json"
}
}
]
} Hope this helps! It's interesting to see this concept being brought to other projects. |
@jdolitsky thanks, the extra type was a typo from a previous version. Fixed now. Thanks for the code, I'll use that as the starting point. I have the pull side already implemented. |
I've submitted the top level media type to the ACR registry Azure/acr#225. I'll close this issue when that gets merged. Thanks everyone. |
This topic needs to be brought to another forum, but there is probably a
more appropriate long term place to store these “well known” media types
(OCI?)
…On Tue, May 14, 2019 at 1:33 AM Gareth Rushgrove ***@***.***> wrote:
I've submitted the top level media type to the ACR registry Azure/acr#225
<Azure/acr#225>. I'll close this issue when that
gets merged. Thanks everyone.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1413?email_source=notifications&email_token=AADACFRF5YME7XDX6R6L6M3PVJMMPA5CNFSM4HLICY4KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVKNBXQ#issuecomment-492097758>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AADACFQVE6T57IHD4VYBJILPVJMMPANCNFSM4HLICY4A>
.
|
I'm going to close this since everyone is on board. If you think there is anything we should do in OPA around this, feel free to file more issues. Cheers! |
Just commenting on the location for mediaTypes. Steve |
As discussed briefly with @tsandall. Posting for visibility and to widen the discussion.
I've been hacking on https://github.com/instrumenta/conftest, which uses OPA/rego but presents an interface for local unit testing of configuration. I'll be talking at KubeCon in a few weeks about this and why I think it's useful.
One thing I've added recently is the ability to share rego files on OCI registries. Basically you can do the following and download existing rules or other bits.
conftest pull instrumenta.azurecr.io/kubernetes-helpers
That spun off work I was doing with @SteveLasker (product manager for Azure Container Registry at Microsoft) before I left Docker. Basically better support in registries for other types of content than just Docker images. Steve has a proposal up at:
https://github.com/SteveLasker/RegistryArtifactTypes/blob/master/mediaTypes.md
The rationale for sharing things via OCI images is described in this blog post https://stevelasker.blog/2019/01/25/cloud-native-artifact-stores-evolve-from-container-registries/. But in short, everyone already has one (whether cloud provider, public/private, self-hosted, geo-replicated, etc.)
As part of the proposal content can have a mimetype. This allows for more intelligent clients and interesting possibilities on the registries. As a very simple example, the registry could show the OPA icon when you view an OCI image which contains a rego bundle.
I'm proposing the following mimetypes for OPA content:
application/vnd.cncf.openpolicyagent.config.v1
application/vnd.cncf.openpolicyagent.manifest.v1
application/vnd.cncf.openpolicyagent.rego.v1
application/vnd.cncf.openpolicyagent.data.v1
This broadly follows the current bundle format, just packaged as an OCI image rather than a flat tar file. https://www.openpolicyagent.org/docs/latest/bundles/
It should be possible to unpack an OCI bundle and get back the folder structure. The different mediaTypes simply mean the content is separated into individual layers in to the OCI image.
Sharing OPA bundles as OCI images may be useful outside conftest, at which point I'm happy to rip out that code to somewhere else.
Do the above media types look sensible to folks?
The text was updated successfully, but these errors were encountered: