validate.sh error "sops: Additional property sops is not allowed"

[MxNxPx]

Hello,
First of all, this repo has been very helpful!!!

Reason for opening this issue is that when using sops to store encrypted secrets in a repo with a similar structure as this, the validate.sh exits with non-zero exit code and these errors:

INFO - Validating kustomization ./sops/staging/
WARN - Set to ignore missing schemas
WARN - stdin contains an invalid Secret (kube-system.foo-secret) - sops: Additional property sops is not allowed
WARN - stdin contains an invalid Secret (kube-system.bar-secret) - sops: Additional property sops is not allowed
$ echo $?
1

I believe it is due to the nature of a sops encrypted secret having the sops: section while being of kind: Secret and not matching the standard k8s Secret spec:

apiVersion: v1
data:
    address: ENC[AES256_GCM,data:************,type:str]
kind: Secret
metadata:
    creationTimestamp: null
    name: foo-secret
    namespace: kube-system
sops:
    kms: []
    gcp_kms: []
    azure_kv:
        - vault_url: https://kvname.vault.azure.net
          name: keyname
          version: ************
          created_at: "2021-12-09T14:00:56Z"
          enc: ************
    hc_vault: []
    age: []
    lastmodified: "2021-12-09T14:00:59Z"
    mac: ENC[AES256_GCM,data:************,type:str]
    pgp: []
    encrypted_regex: ^(data|stringData)$
    version: 3.7.1

To work around this, we updated the script as such so kubeval could skip validation of Secrets:

# mirror kustomize-controller build options
kustomize_flags="--load-restrictor=LoadRestrictionsNone --reorder=legacy"
kustomize_config="kustomization.yaml"
kubeval_flags="--skip-kinds=Secret"

echo "INFO - Validating kustomize overlays"
find . -type f -name $kustomize_config -print0 | while IFS= read -r -d $'\0' file;
  do
    echo "INFO - Validating kustomization ${file/%$kustomize_config}"
    kustomize build "${file/%$kustomize_config}" $kustomize_flags | \
      kubeval --ignore-missing-schemas --strict --additional-schema-locations=file:///tmp/flux-crd-schemas $kubeval_flags
    if [[ ${PIPESTATUS[0]} != 0 ]]; then
      exit 1
    fi
done

Would like to seek out other possible ways folks think this to address could be addressed.

Thanks!

[kingdonb]

The validate script could be clearer about what it does, but I've read it somewhat carefully to understand the details, and I found that it validates basically three things:

1. each OpenAPI schema that it downloads must be valid
2. each cluster in the clusters folder is assumed to be a cluster root and must be valid
3. every directory in the repo with a kustomization.yaml file is assumed to be in use by some cluster, and must be valid

That leaves an opening that we should call out explicitly in the docs, for how to structure your repo so that secrets are kept in a way that they are not validated, by kubeval which cannot validate them before they are decrypted:

https://github.com/kingdonb/bootstrap-repo/tree/staging/secrets/cluster-api

This directory simply doesn't include a kustomization.yaml file and it is hosted outside of clusters. That's how to avoid validation. Use a Flux kustomization like this flux-kustomization.yaml to bring them into your cluster. By keeping secrets separate from the rest of the manifests, you make it easier to tell at PR-review time whether someone is engaging in funny business or not.

The biggest risk of keeping secrets encrypted in the repo is that someone will be able to decrypt them who should not have been able to, into a namespace where they can read secrets (when they should not have access to this secret or decryption key.) So PRs to "secrets" repo can be closely monitored, perhaps through codeowners, to ensure that keys are used responsibly by all project members. At a certain point you have to go, "hey, write access to the repo means you are trusted" if you are giving people access to decryption keys through Flux, but it's a small thing that might help, and it works with the validation script. 👍

[TheEdgeOfRage]

If anybody stumbles across this, since the validate.sh script was changed to use kubeconform instead of kubeval, the correct flag for skipping secrets now is -skip=Secret

[kingdonb]

I think we do recommend users keep secrets in a repo, so they can take advantage of Kustomize Secret generators.

We have some docs about how that can be used with SOPS - this -skip=Secret should either be mentioned in the current repo somewhere alongside validate.sh, or in this doc near here where we combine both tools, (or perhaps in both places):

https://fluxcd.io/flux/guides/helmreleases/#refer-to-values-in-secret-generated-with-kustomize-and-sops

But we are a bit light on prescriptive guidance about where secrets are kept, especially considering that solutions like KeyVault and KMS can be used that keep the secret entirely out-of-band and csi secrets (where you cannot take advantage of the secretGenerator anymore)

If you want to add that -skip=Secret somewhere in the doc here, I think that PR would get merged fast 👍

I'll open a separate issue for the other doc, I have some ideas about how we can expand on the SOPS guide and separate cloud-provider guides for using KMS. The user might get the idea these solutions are exclusive of each other, if we don't show a specific example that covers the use of all these in concert at the same time.