Declarative installation support

[glyn]

Problem

CNAB has been criticised from a security standpoint since installing a bundle requires the bundle’s invocation image (i.e. imperative application code) to run with elevated privileges (that is, higher privileges than the bundle's application code needs).

CNAB aims to support a diversity of cloud environments. For such generality, imperative installation code is needed since a standard installer cannot support arbitrary environments. This is no worse than running an installer with elevated privileges.

However, systems such as kubernetes allow purely declarative installation and do not force application code to run with elevated privileges. So the criticism is fair in the context of kubernetes. There should be a way to install kubernetes applications, packaged as CNABs, without necessarily giving application code elevated privileges.

Two Kubernetes examples will make the advantages of purely declarative installation over an executable invocation image clearer:

• Suppose an application needs a namespace to be created, but the application’s controllers do not need that privilege. Then to install the product using CNAB, the invocation image needs to be allowed to create namespaces. This is the point: CNAB forces us to give application code elevated privileges. In contrast, a namespace can be declared and created by deploying a namespace declaration, e.g. using kubectl, and no part of the application needs to be given elevated privileges.

• Declarative installation has another advantage. It is possible for tools to examine declarations and see what will happen before giving the application access to a cluster. For example, if the application has plain YAML declarations, kapp deploy will list the proposed changes to the cluster and ask the user to confirm that these are acceptable. This could even be extended to analysing the RBAC permissions being granted.

Possible solutions

A basic requirement is to be able to include declarations in a CNAB. This can be achieved by using custom extensions in the bundle manifest.

Instead of using an invocation image to install the bundle, a separate installer will need to be used. The installer can be obtained and verified out of band. Trust can be built around particular installers.
The installer to be used could be provided in the CNAB's documentation, but it may be better for the CNAB to include a more formal description of the installer with details such as name, download, version, and perhaps a cryptographic digest.

Such a CNAB would not declare any invocation images. In general, a CNAB would declare an installer if and only if it did not declare any invocation images.

Other potential solutions were mooted, e.g. by @radu-matei and @jlegrone , in the CNAB Community Meeting on 2 October 2019.

[squillace]

I think also a document describing the general problem might help illuminate where the issues are and what can and should be done:

1. That this is a general software problem, not a CNAB issue, or a container issue. You CAN inspect a kube manifest, for example, but it installs stuff you may not have or be able to inspect. Same issue. Kube has RBAC tools that mitigate; but that requires you to use them correctly, and so on.
2. There is definitely an issue with having to give creation-level permissions to a bundle and the bundle taking those permissions and doing something else with them. But, again, this is the same problem that "software" has unless you have inspected your code all the way down.
3. The solutions lie, including the one above, in out-of-band trust mechanisms. Those include: trusted bundles, trusted base images, policies that prevent public bundles, TUF+In-toto like trust chains, the above solution, and even carving out special mechanisms for platforms like Kubernetes that we expect to be critically important.

The goal, however, is not to argue that CNAB is right for you. The goal is to be transparent and helpful about software problems related to trusting a specific image about which you know nothing and for which you have no out-of-band mechanism to establish trust.

What do we think? I think this is a good idea and orthogonal to any specific solution in this PR. Should I open an issue for that doc separate from this?

[glyn]

What do we think? I think this is a good idea and orthogonal to any specific solution in this PR. Should I open an issue for that doc separate from this?

Yes, let's limit the scope of this issue to declarative installation support (with kubernetes as the principal driver of requirements).

[jchesterpivotal]

By "declarative" do we mean:

1. Users would give a target state and a trusted component (or components) creates a plan and then applies the plan?
2. Users would have a limited palette of fixed actions, which they form into a plan themselves, handing over to be applied by a trusted component?

In both cases there's presumably some trusted component or (ideally) components that carry out parts of a plan, what's left is who does the planning.

These remind me a bit of other tools and categories of tools:

• Cloud deployment and lifecycle tools
  • BOSH, which has declared dependencies and workload placement, but which performs some planning itself
  • Terraform, which explicitly splits planning from applying
• Container orchestrators
  • Kubernetes, which maintains a blackboard that kubelets subscribe and react to individually
  • Diego, which has a specific mechanism for assembling Turing-incomplete plans from a fixed palette (download this, run that etc) and transmitting them to Cells
• Configuration management tools
  • Chef, Puppet etc, which have various degrees of listing steps vs providing ways to define goal states and libraries of sensors and actuators which can detect and move towards goal states
  • Cfengine, which has a "promise theory" concept and in which multiple application is intended to converge

And so on. Installation complexity will be conserved, so most of what is left is how much worker the CNAB packager has to do to adapt a system. In Cloud Foundry land, BOSH is enormously robust, powerful and reproducible, but requires as sacrifice that packagers provide highly prescribed "releases" with very limited space for custom code. This made the barrier to entry quite high and so, in practice, very little was ever packaged for BOSH. But going the other way, I'd be concerned at being too flexible, a la configuration management tools. A very large palette all but guarantees that planning and verification is an intractable problem.

[trishankatdatadog]

Yes, agree with @jlegrone that signing and verifying invocation images with CNAB Security will go a long way...

[glyn]

By "declarative" do we mean:

1. Users would give a target state and a trusted component (or components) creates a plan and then applies the plan?

Yes, that's what I have in mind. More specifically, I'd be surprised if the solution didn't exploit an existing format such as kubernetes YAML files, Helm charts, etc.

[glyn]

Yes, agree with @jlegrone that signing and verifying invocation images with CNAB Security will go a long way...

That can indeed help, but please think of this issue as being "belt and braces" ("belt and suspenders" for Americans) in addition to CNAB security.

[astrieanna]

I'd like to suggest a potential approach to adding support for declarative, invocation-image-free bundles. I think this should implemented as an extension, since there are open questions on the specifics. Extensions can change essentially anything about the rest of the bundle since only CNAB runtimes that recognize the required extension should operate on the bundle. This gives a lot of room for experimentation about what the right changes are.

One Installer

Because the bundle will not include an invocation image, the CNAB runtime will need to know what executable to run instead. The bundle will need to indicate which tool it requires, either via a string or a map that contains, say, a name string and a version string. This approach is helpful for the human installing bundles because they can invest in trusting a single installer tool (e.g. helm) and then install many bundles with a clear understanding of what they will do via reading plain-text helm charts rather than arbitrary container images.

For the simplicity of the CNAB runtime, these installer tools could be packaged as images, the same way that invocation images are today. I will call these "installation images" in the rest of this comment. Before installing (or running any other action on) a bundle that requires an installation image, the CNAB runtime would need to separately be configured to know what image to run when the bundle would normal use an invocation image (i.e. a string/version to image mapping). The CNAB runtime does not need to know more about the interface between bundles and installation images. Because all the actual coordination is between an installer image and the bundle, the CNAB runtime maintains its goal of generic support (and can easily support many installer tools).

There are two parts to what I expect an installation's contract to provide:

1. What version of the tool(s) does it provide? (e.g. helm version)
2. How does it translate from bundle parameters/credentials to tool inputs? (e.g. helm variables)

On the second point, a key part of the interface for an installer image is where the bundle should place its helm templates, input values, and credentials (and what outputs it should expect from the installer). For example, all parameters might need to be placed in files in a specific directory, allowing the invocation image to use the file names as helm variable names, and the kubeconfig to apply those templates to a cluster might need to be at another specified path.

How the static files are packaged into the bundle

Because the invocation image currently servers two purposes -- a place to put the run executable and a place to put any other files needed -- when the run tool is pulled into the more re-usable installer image, a new way to include static files (e.g. helm charts) in a bundle is required.

One proposal from the meeting was to add it to the list of “images” in the bundle. Another option would be to add a separate section of files, which would want both a reference to the file and a path in the invocation image to place it at. For thin bundles, these could be URLs where the files can be downloaded from (i.e. via an http GET), but that opens up new config requirements for the CNAB runtime if the files are to be pulled from authenticated sources. For thick bundles, the files need to be stored in the tar in some clearly defined way.

Problems with one installer

If a CNAB bundle would require a sequence of tools -- such as if you want to install 3 OSS components to create a single whole, and one of them has chosen a different templating technology than the others -- then that bundle may require a different installation image. Having a number of installation images to trust that is the product of the number of templating tools is a burden on users. Or a single installation image could handle applying multiple tools -- such as by storing the different template formats at different paths and adding a static config file to orchestrate the sequencing. But this makes the installation image more complex.

Many installers

If we want to pull the complexity of what tool to apply in what order up to the CNAB runtime level (as part of its support for this extension), then we could have a section for listing the installation steps, each specifying an installation image and which static files and parameters should apply. This seems likely to incentivize small, simple installation images, but pulls more complexity into the CNAB runtime's implementation. It might make it easier for human users to understand what a bundle is doing because they can see the full list of high-level steps in the bundle itself.

Conclusion

I think an extension would give sufficient freedom to experiment with this format. I'd like it to eventually be considered for inclusion in the main spec the way that dependencies should someday be. The main concern I have for making this easy for CNAB runtimes to adopt is the bundled static files (since otherwise it's just about mapping a string to a separately configured installation image). Bundling static files into thick bundles requires defining the format for that and creating a tool to make those bundles, and supporting static files requires the CNAB runtimes to know how to fetch and/or store them. Making the files into OCI images would largely defeat the purpose of removing the invocation image to create declarative bundles (but it crossed my mind because then you could store them in a docker registry, which a CNAB runtime already knows how to do).

[jeremyrickard]

TL;DR: Working on Porter has made me envision CNAB.next as being a manifest that says something like:

• what trusted installers will I use (and how)
• what OCI artifacts will I references (Helm Charts, TF modules/configs, Docker Images, trusted installers themselves)

@glyn @astrieanna our work with Porter has been making us think of something along the lines of your installation images thinking. The approach we've taken so far has been to define those installers (we've called them mixins) and how they're used fairly explictly (but within the constraints set by CNAB).

They report back what version to install currently in the form of additions to the invocation image build, but that's some what an artifact of living within the context of CNAB. Then, they're used by declaring what things to use with them (static files like templates are currently placed in the invocation image and referenced, again because of CNAB envelope, but ends up looking something like (this is an excerpt):

mixins:
  - helm
  - terraform

install: 
  - terraform: 
      backendConfig: 
        access_key: "{{bundle.credentials.do_spaces_key}}"
        key: "{{ bundle.name }}.tfstate"
        secret_key: "{{bundle.credentials.do_spaces_secret}}"
      description: "Create DO PostgreSQL With Terraform"
      manifests: tf/digital-ocean
      outputs: 
        - connString
  - helm: 
      chart: charts/spring-music
      description: "Deploy Spring Music with Helm"
      name: "{{bundle.parameters.helm_release}}"
      namespace: "{{bundle.parameters.namespace}}"
      replace: true
      set: 
        deploy.image: "{{bundle.images.webapp.repository}}@{{bundle.images.webapp.digest}}"

I don't think that's what CNAB looks like in a next iteration, but I think it can be informing of what it looks like. When I envision what an evolution of CNAB might look like without the invocation image, it looks something like that to me. I think the declarative installers need to know how to consume stuff (params,manifests) and return stuff (outputs) and the runtime tools know how to provide orchestration between those, since things are defined in a standard way (the JSON Schema stuff seems to go a long way, but probably not all the way). Those installers can be built by trusted sources (i.e. HashiCorp can make and distribute a TF installer, Kubernetes folks could provide a kubectl based installer). Right now, our Mixins look a lot like this, they're essentially wrappers on top of some underlying official tool.

I think the use of OCI registries could be used for storing static artifacts, assuming tools (maybe the installer images, or probably the cnab runtime things) provide a mechanism for inspecting them. The "images" section of the current bundle could be changed to "artifacts" and you could reference a Helm chart in a registry or some CloudFormation stuff or some Kubernetes manifests, and whatever cnab runtime tool you've decided to use could provide some sort of higher level "inspect" command for the bundle and show you the contents of the OCI artifacts referenced in a given bundle (or even some sort of rendering of them if they're template things that take params).

$ mcguffin inspect bundlexyz:0.1.0 

name: CoolCorp Automated Delivery
description: CAD delivery mechanism

helm:
 ---
# Source: spring-music/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: spring-music
  labels:
    app: spring-music
spec:
  type: "LoadBalancer" 
  ports:
  - name: http
    protocol: TCP
    port: 80
    targetPort: 8080
  selector:
    app: spring-music
---
# Source: spring-music/templates/deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: spring-music
spec:
  selector:
    matchLabels:
      app: spring-music
  replicas: 1
  template:
    metadata:
      labels:
        app: spring-music
    spec:
      containers:
      - name: spring-music
        image: "jeremyrickard/spring-music:latest"
        imagePullPolicy: Always
        resources:
          requests:
            memory: "512Mi"
            cpu: "1.0"
          limits:
            memory: "1024Mi"
            cpu: "2.0"
        ports:
        - containerPort: 8080
        env:
        - name: RUNTIME
          value: kubernetes
        - name: SPRING_PROFILES_ACTIVE
          value: "default"
        - name: SPRING_DATASOURCE_URL
          value: "jdbc:postgresql://:/?sslmode=require&user=&password="

terraform:
digitalocean_database_cluster.music: 
  database:     "" => "<computed>"
  engine:       "" => "pg"
  host:         "" => "<computed>"
  name:         "" => "jrrportertest"
  node_count:   "" => "1"
  password:     "<sensitive>" => "<sensitive>"
  port:         "" => "<computed>"
  private_host: "" => "<computed>"
  private_uri:  "<sensitive>" => "<sensitive>"
  region:       "" => "nyc3"
  size:         "" => "db-s-1vcpu-1gb"
  uri:          "<sensitive>" => "<sensitive>"
  urn:          "" => "<computed>"
  user:         "" => "<computed>"
  version:      "" => "10"

Obviously, tools could provide whatever good UX on that they want, but the standard artifact definition and defined installer tools could make it pretty easy to determine what's going to happen.

Maybe the artifacts could even be to store/distribute the trusted installers.

installers: { 
  "terraform" : {
       "signature" : "somehash",
       "artifact" : "hashicorp.gcr.io/terraform:0.12"
  }, 
  "kubectl" : {
       "signature" : "somehash",
       "artifact" : "k8s.gcr.io/kubectl:v1.15.3"
    },
   "helm" : {
       "signature" : "somehash",
       "artifact" : "helm/helm:v3.0"
   }
} 

Anyone's installer could choose to fetch and install those from registries, or they could out of band install them. They could also fail if say, there isn't an installer for given platform. I think it makes the concept of a "thick bundle" still fairly viable too. Having worked in highly regulated environments with air gapped networks, it would still solve the problem we had burning CD-Rs or DVD-rs with all the stuff for a given release that we needed to build and ship (and then discover oops we missed tool X). Also allows people to write artisinal "installers" for some custom in-house thing and publish it to their private repos as needed.

[trishankatdatadog]

It is important to note that when using multiple CNAB registries, CNAB Security guarantees, at best, the end-to-end provenance of bundles. It cannot guarantee that developers of bundles are not malicious or have not been compromised. The issue of trust is similar to adding third-party repositories on Linux distributions: you basically trust the developers at the end of the day. Therefore, you might need to take extra precaution when choosing to trust additional registries.

PS: @glyn totally did not make me say this.

[glyn]

Thanks @trishankatdatadog. So one way to think about the need for "belt and braces" is when there are multiple CNAB registries and where a customer really wants the function of a particular bundle but isn't sure whether to trust the provider of the bundle completely. In that situation, declarative installation should make it possible to introspect the properties of the bundle and work out in advance what permissions it needs to function.

[jchesterpivotal]

@glyn I agree that knowing the permissions in advance is a worthy goal.

How will we know we have achieved that goal? I feel like the space of candidate solutions is vast, so it would be interesting to get a tighter grip on the mischief.

[astrieanna]

+1 to storing static files as OCI Artifacts

@jeremyrickard I like the idea of using OCI Artifacts to store the non-image files needed by declarative bundles. (I hadn't realized that you could store non-image things in an OCI registry.) I'm happy with saying that the CNAB runtime should then be able to display those files for the user's inspection; actually templating anything else into place is probably an installer image action.

-1 to storing installer binaries as OCI Artifacts

I don't know that storing installer binaries as artifacts would work. Each binary has its own interface; how does the CNAB runtime know what to do with them? Where does the top-level coordination come from? (e.g. run helm template then put the output into kubectl apply)

I like installation images better because the CNAB runtime wouldn't need to do anything really new beyond mapping bundles to images that are provided separately. And I'd like CNAB runtimes to support both as easily as possible, so that bundle developers can move from a declarative bundle to a standard one if they need the much greater flexibility of a custom invocation image.

Installation images could also include extra actions (like a preview that shows the templated files as outputs without installing anything) -- and different installation images would likely have different custom actions, depending on what that tool does. I really want it to be easy for users to add support for more installers without any changes to the CNAB runtime or some complex configuration language to learn (vs. literally writing the same run script you'd write for your invocation image anyway).

Knowing permissions in advance

Is having the complete set of YAML you want to kubectl apply sufficient to determine what k8s permissions you would need to actually apply it? Is there existing tooling for helping users figure out what permissions a given set of YAML requires (or what it is giving itself, if it creates its own roles?)
(I'm not 100% certain that I've interpreted what you all mean by 'permissions` in this part of the conversation, so please correct me if my guess was wrong.)

[carolynvs]

I don't know that storing installer binaries as artifacts would work. Each binary has its own interface; how does the CNAB runtime know what to do with them? Where does the top-level coordination come from? (e.g. run helm template then put the output into kubectl apply)

Jeremy was sharing how Porter works, which is by using an adapter between CNAB's runtime and native tools, such as helm or terraform, which Porter calls "mixins". Mixins are binaries that conform to a standard interface and handle adapting between that YAML he pasted earlier and properly executing the native binary (e.g. helm/terraform). The mixin is responsible translating the yaml into a CLI command, receiving CNAB parameters, credentials, and outputs from previous steps. It also can generate bundle outputs as part of the mixin interface.

Right now Porter distributes these mixin binaries itself, but it could just as easily be distributed via an OCI registry since they do all use a standard interface that could be used generically by a runtime.

This is Porter's big "secret" really. Porter injects a tiny workflow engine into every bundle built by Porter to orchestrate running the declarative bundle using these mixins. 😀

[jlegrone]

I'll start with a TLDR:

The extensibility of invocation images is at the core of what I think makes CNAB a worthwhile investment. Attempting to adopt "declarative installers" as described would introduce many of the very problems CNAB was originally designed to sidestep, while the issues discussed here may already have solutions within the bounds of the spec as it is today.

---

In the discussion so far I've seen a few objectives that I've tried to narrow down:

1. Declarative way to specify the deployment tool, its version, and how the runtime should pull/execute it
2. Runtime that handles binary authorization for deployment tool execution
3. Runtime that scopes credentials provided to deployment tools appropriately for each installation
4. Runtime that can output a "plan" of what will be changed on the next upgrade

I'll add to that list one other thing I'd like to see:

5. Encourage bundle portability and innovation within the CNAB community

Notice that none of these actually has much to do with whether the deployment tool is a host binary or a container image. Let's look at each objective and see if there is a viable way to achieve it within the spec as it stands today.

1. Declarative way to specify the deployment tool, its version, and how the runtime should pull/execute it

This is really already at the heart of the CNAB spec. Invocation images have a standard way of accepting input parameters and producing outputs, and how to pull and run container images is already well defined. Each of these aspects of invocation images would need to be solved again, by every runtime implementation, if we added a new type of deployment tool definition.

2. Runtime that handles binary authorization for deployment tool execution

Running arbitrary images and giving them credentials to do things in your environments is definitely a valid concern. However, the spec does not prevent a CNAB runtime from leveraging established solutions for trusting the origin and content of container images in order to only run invocation images that have been built by trusted parties and/or gone through required processes like image vulnerability scanning.

In fact, we can go beyond only trusting only our deployment tool, and verify the whole bundle. The security spec is an attempt to formalize this, and I think any deficiencies in this model should be addressed there.

As the ecosystem grows, I believe several standard invocation images for various platforms will naturally emerge. Looking further out, "supported distributions" from the likes of Bitnami and Redhat will probably become available as well. A whitelist of these curated invocation images might suffice for many use cases and would be simple to implement from a runtime perspective.

3. Runtime that scopes credentials provided to deployment tools appropriately for each installation

There should be a way to install kubernetes applications, packaged as CNABs, without necessarily giving application code elevated privileges.

I'm was a little puzzled by this point, because so far I haven't written or seen any invocation images that do this. And at Datadog we now deploy lots of applications to Kubernetes with CNAB 😄.

A CNAB runtime may provide to an invocation image whichever credentials it sees fit, and those credentials need not be passed by the invocation image to the application workload. As CNAB matures I think we'll see this continue to evolve, but it is entirely possible to implement a runtime that provides short-lived credentials that are scoped appropriately to what the bundle requires in order to deploy.

4. Runtime that can output a "plan" of what will be changed on the next upgrade

This is the idea behind "well-known" custom actions (specifically io.cnab.dry-run). Again, this is not a feature that is blocked by the design of invocation images.

5. Encourage bundle portability and innovation within the CNAB community

The fact that anyone can publish an invocation image to a public registry and make it available to the world without coordinating with every CNAB runtime implementation to get their deployment tooling supported is, in my opinion, at the core of what makes CNAB a worthy investment. Need custom lifecycle hooks for your Kubernetes deployments? Write an invocation image. Need to work with some old ksonnet config? Same idea.

Going back to point 2, it is perfectly valid for CNAB runtimes to only allow execution of a curated set of invocation images. But I think we'd be doing ourselves a disservice by expanding the spec to require implementations to have knowledge of the contracts for n deployment tools targeting p platforms, since in reality a given runtime will only be able to implement support for a subset of those tools. Keeping the spec small is key to having bundles that are portable between runtimes.

[trishankatdatadog]

In fact, we can go beyond only trusting only our deployment tool, and verify the whole bundle. The security spec is an attempt to formalize this, and I think any deficiencies in this model should be addressed there.

I agree: CNAB-Sec should be able to give guarantees on the authenticity, integrity, and even provenance of your bundle. If something is missing, please let us know.

[jchesterpivotal]

A nitpick:

Running arbitrary images and giving them credentials to do things in your environments is definitely a valid concern. However, the spec does not prevent a CNAB runtime from leveraging established solutions for trusting the origin and content of container images in order to only run invocation images that have been built by trusted parties and/or gone through required processes like image vulnerability scanning.

Validating the provenance of an image defends against injection of malicious code somewhere in the supply chain. But it doesn't prevent injections by insider threats or, much more likely, bugs that create vulnerabilities.

Putting it another way: strong provenance guarantees is one design defense that increases confidence. Another defense to increase confidence is minimal permissions. Solving for provenance is not solving for minimal permissions. It only gives me someone to blame for mistakes or attacks made within the scope of the minimal permissions which were granted.

[jlegrone]

Validating the provenance of an image defends against injection of malicious code somewhere in the supply chain. But it doesn't prevent injections by insider threats or, much more likely, bugs that create vulnerabilities.

Right, and this is where a curated set of invocation images is useful. Runtimes can enforce that any bundles they install use a whitelisted invocation image, or even swap out the provided invocation image for a preferred one if the contents of the bundle are known. This could be handled via a custom extension.

Solving for provenance is not solving for minimal permissions.

I never meant to imply that these are the same. I'm not sure I understand how this proposal allows limiting deployment tool permissions beyond what is already possible via invocation images though.

[glyn]

The runtime whitelisting, or swapping out, the invocation image is certainly one approach. The runtime should be able to use the (e.g. kubernetes) declarations stored (somehow, TBD) in the bundle to determine the minimal permissions necessary for the invocation image to install the bundle.

This works fine if a suitable runtime is used. There is a risk that someone will unwittingly use an unsuitable runtime and run an invocation image they weren't supposed to, but that would probably concern only the most security conscious organisations.

That said, I'm not sure how attractive a solution this would be compared to a bundle declaring that it needs to be installed using a well-known and trusted installer (instead of an invocation image). At the very least, the optics are rather different. Maybe any users would care to comment?

[squillace]

This is a wonderful thread, frankly. As it's changed directions slightly, I wanted to re-summarize one of the OP's objectives:

However, systems such as kubernetes allow purely declarative installation and do not force application code to run with elevated privileges. So the criticism is fair in the context of kubernetes. There should be a way to install kubernetes applications, packaged as CNABs, without necessarily giving application code elevated privileges. -- #285 (comment)

I would restate this objective as, given that (at least) one critical target runtime has its own deployment system, does CNAB need to be modified in order to deploy using that runtime only, or can this be done now? (Several reasons, mostly revolving around having to pass deployment credentials in order to invoke the inner deployment tools, are then discussed for why this is an objective, but let's start merely with the objective....)

@glyn, I'll ask you first if I am properly recapitulating the example objective using the case of K8s?
If I have it right, I am not really sure whether the issue is whether the CNAB specification prevents the construction of a runtime that does precisely this, or whether the relevant question is whether any of the current runtimes publicly available (docker-app, duffle, or porter) implement a way to do this.

Are we discussing the former, or the latter? Both are good conversations, methinks, but they are different conversations. As for the former, I think you can build a runtime for CNAB that does this right now, given that the types of invocationImages are effectively unlimited, which means that a CNAB runtime could easily use custom metadata to determine whether an image was merely a package of manifests -- using K8s as an example -- and extracting the manifests from the package, turn around and invoke kubectl on them in the manner described. That runtime could also perform any number of verification steps, including live mfa on the invocation process rights, for example.

(As an aside, I'm fairly sure that a kubectl plugin could be written that merely retrieved, verified, and examined the bundle, extracting the resources and applying them with the current user's creds, achieving precisely what your OP wanted. And it could certainly build, sign, and push such bundles as well.)

[glyn]

@squillace no, the objective is to install kubernetes applications, packaged as CNABs, without necessarily giving application code (particularly invocation images) elevated privileges.

The question you raise is whether the objective can be achieved without a spec change.

The earlier discussion proposed one solution: using a well known (docker/OCI) invocation image as a way of indicating that the bundle can be deployed using a corresponding out of band deployment tool. I don't think this is a good solution for the reasons I pointed out here.

You propose another solution: using a special type of invocation image to denote that a corresponding out of band deployment tool should be used. Unfortunately, the invocation images section of the CNAB spec is pretty prescriptive, for example:

An invocation image is composed of the following:

• A file system hierarchy following a defined pattern [...]

The run tool MUST be located at the path /cnab/app/run. It MUST be executable. It MUST react to the CNAB_ACTION provided to it.

It's quite clear that the spec intends invocation images to be file systems containing executable code and executable installation code packaged in the bundle is precisely what we are trying to avoid in this issue.

Also, the way invocation images are referenced is in line with the intention that they should be file system images:

The image field MUST give a path-like or URI-like representation of the location of the image.

The contentDigest field MUST contain a digest, in OCI format, to be used to compute the integrity of the image.

So, if we were to broaden the concept of invocation image to allow for purely declarative installation artefacts, we would need to make some spec changes.

[jbeda]

Putting in my two cents here as we look to figure out what tool chain we want to standardize and support for VMware Tanzu (including Pivotal).

Our primary target is Kubernetes. One of the goals for CNAB is to be flexible enough to target many environments. The extensible invocation images are the core of that. This works against the goals of having a clear trusted tool chain that is delivered separately from the application code.

I’d love to see a defined subset of CNAB that is nothing more than Plain Old Data. Having a way to collect a set of artifacts and upload/download those to registries (with both fat and skinny bundles) is enormously useful regardless of how those bundles are used. I’d love to see a more evolved (and signed, traceable, portable, verifiable) tar file format.

From there we can build different types of experiences — these are not exclusive:

• A purely declarative install mechanism. All necessary artifacts (with metadata about those artifacts) would be included in the bundle. External tools would be required to interpret and use those artifacts to do... something. It could be installing something but it could also be other things over time.
• Move the install tool chain into a container (or set of containers) that are left as a suggestion in the metadata on how to use the artifacts. It would be up to the user to use (or not use) those containers.
• Have everything bundled into the install image much as it is today. That image would is opaque and must be used to install the application as some of the required artifacts may be baked into that image.

For what it is worth, we are interested in the first couple of options but I really don’t want to support or encourage our users to do the third.

Some concrete concerns with the third option:

• As stated earlier, the install image would most times be given elevated privileges. It could then use those privileges without any sort of verification. There isn’t enough metadata to even know how to downscope those privileges. Sure, we can sign these, but that is what we said about ActiveX.
• There is no way to break out the install manifests to modify those. In the case where the key/value params are not enough to adapt an application to a use case the user is left with no easy options to extract the data. With helm, at least, you can fork/modify the helm chart to your use case as the “source” is available. The opaqueness of the install instructions defeats some of the may ways to adapt systems like kustomize.
• There is no opportunity to know more about the images themselves and how they are built. Let’s say that, instead of using a Dockerfile, the application is built using build packs. Being able to rebase those build packs to account for security fixes or adapt/modify the runtime (change which JDK you are using?) is impossible without more metadata about the artifacts and more transparency about how those artifacts can be modified/adapted.

While having something super generic is powerful (You can do anything!) it is also dangerous (You can do anything!). I much prefer something that can be introspected and understood as data vs. a black box that runs with elevated privs.

[glyn]

The sheaf prototype gives an idea of the kind of thinking inside VMware.

[technosophos]

Maybe this is a question with an obvious answer, but... why not just use Helm 3 charts? They already achieve all of this, hand Helm 3 even includes support for storing objects in OCI registries. There's no requirement to use the templating features of Helm, artifacts are trivially easy to inspect, and it seems to achieve what Sheaf does already.

[voor]

Maybe this is a question with an obvious answer, but... why not just use Helm 3 charts? They already achieve all of this, hand Helm 3 even includes support for storing objects in OCI registries. There's no requirement to use the templating features of Helm, artifacts are trivially easy to inspect, and it seems to achieve what Sheaf does already.

When evaluating Helm 3 for certain use cases the lack of a rigorous way to declare images specifically for image relocation created some roadblocks if I recall correctly -- I know there's some work in that area to try and rectify helm/helm#7154 (comment)

[jlegrone]

I've made an attempt to provide a more detailed description of some of the solutions mentioned in #285 (comment) via issues #337 and #338. While these are certainly incomplete, I hope they demonstrate that CNAB extensions are a viable mechanism to give runtimes the visibility they need in order to apply layers of security beyond signature verification.

[chris-crone]

This is a great conversation, thank you all who have contributed so far.

One of the goals of CNAB is to avoid exposing tool builders and users to the complexities of the application definition and deployment. That being said, if one is packaging and deploying only simple applications (i.e.: those defined by Kubernetes manifests), there is value in being able to introspect these manifests. Given this goal of CNAB, I would expect to see any wording for how to do this (like that in #337 or #338, thanks @jlegrone!) in a non-normative section of the specification.

If you expect to control both the CNAB bundle authoring tool and the CNAB runtime tool, you could implement custom actions to output the Kubernetes manifest and then deploy it from the runtime tool or use something like #338. This would give both the image storage mechanisms of CNAB and a declarative manifest.

Tackling the elevated privileges issue raised by @jbeda:

As stated earlier, the install image would most times be given elevated privileges. It could then use those privileges without any sort of verification.

One way to mitigate this is to have ephemeral credentials that last a single CNAB action duration with clear scoping (like that in #337). This allows auditing what the action does, scoping the credentials down to the minimum required, and ensures that the credentials cannot be reused.

[glyn]

Maybe this is a question with an obvious answer, but... why not just use Helm 3 charts? They already achieve all of this, hand Helm 3 even includes support for storing objects in OCI registries. There's no requirement to use the templating features of Helm, artifacts are trivially easy to inspect, and it seems to achieve what Sheaf does already.

When riff tried Helm, we ran into problems in the way CRDs are handled, especially when it came to upgrading CRDs (given that we have multiple optional components, each with CRDs).

[glyn]

After discussion in the CNAB Community Call on 18 March 202, it appears that making invocation images optional bifurcates the spec and makes interoperability more difficult. Keeping invocation images doesn't achieve the goals of this issue. Closing.