RFC: Add a way to decrypt files from a source before kustomize

[toshokan]

Currently, users specify that resources from a source need decryption by annotating the Kustomization that references that source.

This works well when the files that have to be decrypted are Kubernetes Secret manifests with data or stringData encrypted, or plain text files (useful for kustomize SecretGenerator), but not for all other types of files.

In particular, while fluxcd/kustomize-controller#328 (and some of the work that came after it) was a great help for specific types of secrets, it doesn't help with others.

In fluxcd/flux2#745, the comments suggest (and I can confirm) that we still cannot decrypt .env files. There is some code for handling the case of plain text files that generate a single key in a secret here https://github.com/fluxcd/kustomize-controller/blob/8eb204106335d13d423e8b6c55e733a0634e26b0/controllers/kustomization_decryptor.go#L133, but it doesn't for for .env-like files because they don't get encrypted the same way, extra sops_-prefixed keys are generated in the file and end up as keys in the secret, so values cannot be encrypted independently.

The code there could be changed to try to load the secret data map as a sops-encrypted file, but that would also be fragile and probably not do the right thing, because a secret might be generated from keys coming from different encrypted sources.

Instead, if we add an option for decrypting everything (or everything in a subtree, or everything matching a regex, etc) in a source directly, before kustomize, these problems disappear, and we could use SecretsGenerator and other such facilities without any special handling because all the decryption was already handled at a lower "layer".

It would also allow us to successfully encrypt and decrypt manifests without metadata (or encrypt some of the metadata), which some users have been requesting for a while.

What do other contributors and users think? If this is a direction the project wants to go in, I'd be happy to contribute the feature.

[stefanprodan]

We can't decrypt secrets in source-controller as this would expose secrets on disk, source-controller keeps a cache of manifests locally.

[toshokan]

As long as this is opt-in behaviour with a disclaimer, I think many people would be happy to have that option, especially if they can choose exactly which file patterns or subtrees get decrypted early.

[toshokan]

@stefanprodan What are your thoughts on having this be opt-in behaviour with a warning?
Maybe another option is that sources with this option enabled do not get cached?

[hiddeco]

Maybe another option is that sources with this option enabled do not get cached?

This is not a viable option as it is a distributed cache, and without it, controllers would not be able to make use of the resources.

In addition, introducing decryption as an opt-in option would add a lot of things to the source API and introduce a set of dependencies that are not introduced lightly, making this a no-go for me personally.

[stefanprodan]

We've added support for .env files in fluxcd/kustomize-controller#463