Separate user-defined secrets, to allow SealedSecrets to work for SSO, webhooks, private repos etc.

[aloysius-lim]

Summary

The argocd-secret Secret should be divided into auto-generated secrets and user-defined ones, such as secrets for SSO, webhooks and private repositories.

Motivation

I would like to use SealedSecrets to manage the secrets for ArgoCD. I have defined a SealedSecret for these secrets. But ArgoCD generates certain secrets during install and creates the argocd-secret Secret resource. Because this Secret resource is not managed by SealedSecrets, the SealedSecret that I defined cannot be applied to argocd-secrets. Even if it did, it would overwrite the secrets that were generated by ArgoCD.

Steps to reproduce this scenario:

1. Create a SealedSecret with namespace argocd and name argocd-secret, containing encrypted user-defined secrets. For example:

   # my-sealed-secret.yaml
   apiVersion: bitnami.com/v1alpha1
   kind: SealedSecret
   metadata:
     creationTimestamp: null
     name: argocd-secret
     namespace: argocd
   spec:
     encryptedData:
       githubAppPrivateKey: "..."
       oidc.clientID: "..."
       oidc.clientSecret: "..."
       webhook.github.secret: "..."
     template:
       data: null
       metadata:
         creationTimestamp: null
         name: argocd-secret
         namespace: argocd

2. Create the corresponding patch for argocd-cm ConfigMap for SSO and private repository credentials (not shown here for brevity).

3. Create a kustomization.yaml file to install everything.

   apiVersion: kustomize.config.k8s.io/v1beta1
   kind: Kustomization
   namespace: argocd
   
   resources:
   - github.com/argoproj/argo-cd/manifests/cluster-install?ref=tags/v2.0.4
   - my-sealed-secret.yaml
   
   patchesStrategicMerge:
   - argocd-cm.yaml

4. Install.

   kubectl apply -n argocd -k .

5. Inspect SealedSecret logs

   kubectl logs -f -n kube-system deployments/sealed-secrets-controller
   ...
   2021/07/05 16:29:14 Updating argocd/argocd-secret
   2021/07/05 16:29:14 Error updating argocd/argocd-secret, will retry: failed update: Resource "argocd-secret" already exists and is not managed by SealedSecret
   2021/07/05 16:29:14 Event(v1.ObjectReference{Kind:"SealedSecret", Namespace:"argocd", Name:"argocd-secret", UID:"f9193ada-3230-45c4-9593-55ead2ac16ef", APIVersion:"bitnami.com/v1alpha1", ResourceVersion:"15680", FieldPath:""}): type: 'Warning' reason: 'ErrUpdateFailed' Resource "argocd-secret" already exists and is not managed by SealedSecret

SealedSecrets does not overwrite the existing argocd-secret Secret resource, which is the expected and safe behavior. However, there is no way to merge the two into one. A peek at the argocd-server logs confirms that the secrets are not injected:

kubectl logs -f -n argocd deployments/argocd-server
...
time="2021-07-05T16:29:39Z" level=warning msg="config referenced '$oidc.clientSecret', but key does not exist in secret"
time="2021-07-05T16:29:39Z" level=warning msg="config referenced '$oidc.clientID', but key does not exist in secret"

Proposal

1. Allow user-defined secrets to be defined in another Secret resource named argocd-secret-user. This allows secrets management tools like SealedSecrets to inject user-defined secrets.
2. Allow argocd-secret-user to also define the secrets that are sometimes auto-generated during install, including tls.crt, tls.key, admin.password, admin.passwordMtime, and server.secretkey. If any of these are defined in argocd-secret-user, they take precedence and override the values in argocd-secret.

[jannfis]

I think this goes in the same direction as #6460

[jannfis]

As a workaround (since you're already using Kustomize), you can use the following patch for argocd-secret to allow sealed-secrets to unseal an SealedSecret resource to an existing argocd-secret:

[
  {
    "op": "add",
    "path": "/metadata/annotations",
    "value": {"sealedsecrets.bitnami.com/managed": "true"}
  },
  {
    "op": "remove",
    "path": "/metadata/labels/app.kubernetes.io~1part-of"
  }
]

You can specify this in your kustomization.yaml as follows:

patches:
  - path: patch-secret.json
    target:
      version: v1
      kind: Secret
      name: argocd-secret

Of course, you'd need to pull in server.secretkey and others into your SealedSecret (you can use the values created by Argo CD for this).

[aloysius-lim]

Thank you for the suggestion, @jannfis. That is pretty much what I did to work around the issue. I created a bash script to perform these steps outside of ArgoCD. Once the values created by ArgoCD are merged with my user-defined secrets, everything is checked into Git and works well from there on.

[alexmt]

As of v2.1 Argo CD allows referencing secret values from other secrets in the same namespace where argocd installed: https://argo-cd.readthedocs.io/en/stable/operator-manual/user-management/#alternative

[aloysius-lim]

That's perfect for SSO. Will that also work for webhook secrets (webhook.github.secret) and if so, how? The documentation is not clear on whether the new functionality works for other secrets apart from user management.

[paul-civitas]

@alexmt that's a good feature, but so long as you're putting in that kinda ${} via helm or some other kind of source control, you still need to worry about whatever thing you're using to set the argocd-secret blanking out the server.secretkey

see: #10393

The better solution is to get stuff that's programmatically generated by argo, far far away from stuff that you're asking users to configure.

[HariSekhon]

+1

This is important to reduce risk when fiddling with secrets and also as more people move to Sealed Secrets or External Secrets for things like Azure AD SSO:

https://github.com/HariSekhon/Kubernetes-configs/blob/master/argocd/base/argocd-azure-ad-external-secret.yaml