Bitnami SealedSecrets (query custom resources)

[brandonkal]

Problem description

I've run into some trouble managing SealedSecrets with Pulumi. This is a dump of that issue.

Briefly, SealedSecrets are Secrets asymmetrically encrypted for the controller in the kubernetes cluster. A SealedSecret maps 1:1 with a native Kubernetes Secret that is unsealed in cluster.

The SealedSecret itself has the appropriate events that should be queryable (i.e. failed to decrypt, unsealed, etc) the controller then creates a Secret.

I ran into this because a later resource (an ingress) referenced a secret by its string name. The secret was never created on kubernetes because the Bitnami sealedsecret controller rejected the sealedsecret. So the ingress was unhealthy but the pulumi program failed to detect that. I imagine the issue would be the same if the secret came from a Jetstack cert-manager controller as well.

It's not entirely clear what the best approach would be in pulumi's current state. I know pulumi checks the health of created resources, so one solution would be to extend this to created kubernetes custom resources. The SealedSecret itself reports error events, so ideally there should be an easy way to query for this or to somehow reference the expected output of a custom resources controller action.

For example, a means in which I could extend the pulumi-kubernetes SDK to create a pulumi.CustomResource (this gets confusing) that:

1. Applies a SealedSecret definition with kubernetes.yaml.ConfigFile
2. Watches that kubernetes object for a success or failure
3. And returns a the type for a standard secret object.

Errors & Logs

None

Affected product version(s)

latest

Reproducing the issue

import * as k8s from '@pulumi/kubernetes'
import * as kx from '@pulumi/kubernetesx'
import * as pulumi from '@pulumi/pulumi'

// Create SealedSecret for TLS
const tlsCert = new k8s.yaml.ConfigFile('dev-local-tls', {
  file: '../01-identity/dex/sealed-dev-local-tls.yaml',
})

// The SealedSecrets controller will decrypt the above and generate a
// dev-local-tls secret in the identity namespace

// In this example this should fail as the sealed-secret will fail to decrypt...
const secret = tlsCert.getResource(
  'v1/Secret',
  'identity',
  'dev-local-tls'
)
// this is probably returning undefined

const next = new k8s.core.v1.Service('should-wait', {}, { dependsOn: secret })

To test, the kubernetes cluster will need SealedSecrets Controller installed. To create the referenced yaml file, generate a sealed secret with a tls key pair (or any random data) and then change the namespace. Because SealedSecrets are sealed for a specific namespace, changing it is enough to make the encryptedData block invalid. We want the unseal to fail.

Suggestions for a fix

• An ingress managed by pulumi that references a non-existant secret for tls will fail silently. One of the reasons I enjoy Pulumi so much is because it helps surface these errors. Unfortunately that is not the case for the ingress
• It should be easy to watch a CustomResources that report errors (i.e. a sealedsecret failed to unseal, or a Jetstack cert-manager Certificate that failed to be created, etc) and stop a pulumi program.
• Add a way to describe expected outputs. i.e. if a sealedsecret is applied, I expect a secret almost immediately.

[omidraha]

Any approach or update around this?

[lblackstone]

See pulumi/pulumi#1691 and #1260 for additional discussion

[blampe]

@brandonkal @jnatherley @omidraha #1260 has been implemented and will help with this. It's not released yet but see my comment here if you'd like to try using the alpha release.

Adding an annotation pulumi.com/waitFor: condition=Synced to your SealedSecret resource (in sealed-dev-local-tls.yaml) will cause Pulumi to wait for the secret to be unwrapped by the controller. Additionally if the controller rejects the secret it will now be logged -- #3135 has an example of this exact use case.

I'd also recommend using v2 ConfigFile (or a CustomResourceDefinition) since Pulumi doesn't respect readiness as well with the v1 ConfigFile. That would allow you to simplify your example to just dependsOn: tlsCert.