Question about this pattern and ordered application set

[alex-treebeard]

Hi @csantanapr, thanks a lot for this org -- I'm trying out the gitops bridge pattern for bootstrapping Kubeflow (see tf module, and fork of your control plane.

I can see that this stuff is quite bleeding edge so I'm trying to find practical workarounds in places.

One of the first issues I've noticed is that I'd like these applicationsets to sync in order (i.e. so that istio webhook works before we try creating pods in the mesh). If sync waves doesn't work reliably I may enforce dependencies with terraform instead.

Are you aware of any inherent argo limitations here? or do you have any other pointers to those interested in feeding terraform outputs into K8s systems in this way?

Thanks again!

edit:

Another thing I've been wondering is: What is the benefit of passing cluster variables to argo via the cluster secret, as opposed to having some helm-based App of Apps, where cluster variables can be passed to child apps via helm values?

[csantanapr]

Hi @alex-treebeard thank you opening the issue with your question

On the ordering for applicationset is something we are working with @christianh814

Adding syncwaves it each application set in order but from one application set to the other it doesn't wait for the app to be healhty. This is something we plan to address with progresive sync on each application set.
We need to test if this would address the sync wave health on the applicationset, without progresive sync applicationsets don't have statues healthy.

Right now I have a repo I"m experimenting and having syncwave and increasing the delay from 2s to 30s is enough time to guaranteed that the addon/helmchart is ready before starting the next one

  argocd_values = <<-EOT
    controller:
      env:
        - name: ARGOCD_SYNC_WAVE_DELAY
          value: '30'
    EOT

And adding the sync wave annotation

(base) ➜  gitops grep "argocd.argoproj.io/sync-wave" -r .  -B 4                                                                                                            kind-kind
./bootstrap/addons/oss/addons-metrics-server-appset.yaml-kind: ApplicationSet
./bootstrap/addons/oss/addons-metrics-server-appset.yaml-metadata:
./bootstrap/addons/oss/addons-metrics-server-appset.yaml-  name: addons-metrics-server
./bootstrap/addons/oss/addons-metrics-server-appset.yaml-  annotations:
./bootstrap/addons/oss/addons-metrics-server-appset.yaml:    argocd.argoproj.io/sync-wave: "5"
--
./bootstrap/addons/oss/addons-argo-cd-appset.yaml-kind: ApplicationSet
./bootstrap/addons/oss/addons-argo-cd-appset.yaml-metadata:
./bootstrap/addons/oss/addons-argo-cd-appset.yaml-  name: addons-argocd
./bootstrap/addons/oss/addons-argo-cd-appset.yaml-  annotations:
./bootstrap/addons/oss/addons-argo-cd-appset.yaml:    argocd.argoproj.io/sync-wave: "4"
--
./bootstrap/addons/oss/addons-argo-events-appset.yaml-kind: ApplicationSet
./bootstrap/addons/oss/addons-argo-events-appset.yaml-metadata:
./bootstrap/addons/oss/addons-argo-events-appset.yaml-  name: addons-argo-events
./bootstrap/addons/oss/addons-argo-events-appset.yaml-  annotations:
./bootstrap/addons/oss/addons-argo-events-appset.yaml:    argocd.argoproj.io/sync-wave: "6"
--
./bootstrap/addons/oss/addons-argo-workflows-appset.yaml-kind: ApplicationSet
./bootstrap/addons/oss/addons-argo-workflows-appset.yaml-metadata:
./bootstrap/addons/oss/addons-argo-workflows-appset.yaml-  name: addons-argo-workflows
./bootstrap/addons/oss/addons-argo-workflows-appset.yaml-  annotations:
./bootstrap/addons/oss/addons-argo-workflows-appset.yaml:    argocd.argoproj.io/sync-wave: "2"
--
./bootstrap/addons/aws/addons-aws-load-balancer-controller-appset.yaml-kind: ApplicationSet
./bootstrap/addons/aws/addons-aws-load-balancer-controller-appset.yaml-metadata:
./bootstrap/addons/aws/addons-aws-load-balancer-controller-appset.yaml-  name: addons-aws-load-balancer-controller
./bootstrap/addons/aws/addons-aws-load-balancer-controller-appset.yaml-  annotations:
./bootstrap/addons/aws/addons-aws-load-balancer-controller-appset.yaml:    argocd.argoproj.io/sync-wave: "-1"
--
./bootstrap/addons/aws/addons-aws-oss-cluster-autoscaler-appset.yaml-kind: ApplicationSet
./bootstrap/addons/aws/addons-aws-oss-cluster-autoscaler-appset.yaml-metadata:
./bootstrap/addons/aws/addons-aws-oss-cluster-autoscaler-appset.yaml-  name: addons-aws-cluster-autoscaler
./bootstrap/addons/aws/addons-aws-oss-cluster-autoscaler-appset.yaml-  annotations:
./bootstrap/addons/aws/addons-aws-oss-cluster-autoscaler-appset.yaml:    argocd.argoproj.io/sync-wave: "3"
--
./bootstrap/addons/aws/addons-aws-oss-karpenter-appset.yaml-kind: ApplicationSet
./bootstrap/addons/aws/addons-aws-oss-karpenter-appset.yaml-metadata:
./bootstrap/addons/aws/addons-aws-oss-karpenter-appset.yaml-  name: addons-aws-karpenter
./bootstrap/addons/aws/addons-aws-oss-karpenter-appset.yaml-  annotations:
./bootstrap/addons/aws/addons-aws-oss-karpenter-appset.yaml:    argocd.argoproj.io/sync-wave: "1"
--
./bootstrap/workloads/apps-appset.yaml-metadata:
./bootstrap/workloads/apps-appset.yaml-  name: apps
./bootstrap/workloads/apps-appset.yaml-  namespace: argocd
./bootstrap/workloads/apps-appset.yaml-  annotations:
./bootstrap/workloads/apps-appset.yaml:    argocd.argoproj.io/sync-wave: "101"
--
./hack/exclude/oss/addons-argo-cd-appset.yaml-kind: ApplicationSet
./hack/exclude/oss/addons-argo-cd-appset.yaml-metadata:
./hack/exclude/oss/addons-argo-cd-appset.yaml-  name: addons-argocd
./hack/exclude/oss/addons-argo-cd-appset.yaml-  annotations:
./hack/exclude/oss/addons-argo-cd-appset.yaml:    argocd.argoproj.io/sync-wave: "4"
--
./hack/exclude/oss/addons-argo-workflows-appset.yaml-kind: ApplicationSet
./hack/exclude/oss/addons-argo-workflows-appset.yaml-metadata:
./hack/exclude/oss/addons-argo-workflows-appset.yaml-  name: addons-argo-workflows
./hack/exclude/oss/addons-argo-workflows-appset.yaml-  annotations:
./hack/exclude/oss/addons-argo-workflows-appset.yaml:    argocd.argoproj.io/sync-wave: "2"
--
./hack/exclude/aws/addons-aws-load-balancer-controller-appset.yaml-kind: ApplicationSet
./hack/exclude/aws/addons-aws-load-balancer-controller-appset.yaml-metadata:
./hack/exclude/aws/addons-aws-load-balancer-controller-appset.yaml-  name: addons-aws-load-balancer-controller
./hack/exclude/aws/addons-aws-load-balancer-controller-appset.yaml-  annotations:
./hack/exclude/aws/addons-aws-load-balancer-controller-appset.yaml:    argocd.argoproj.io/sync-wave: "-1"
--
./hack/exclude/aws/addons-aws-oss-cluster-autoscaler-appset.yaml-kind: ApplicationSet
./hack/exclude/aws/addons-aws-oss-cluster-autoscaler-appset.yaml-metadata:
./hack/exclude/aws/addons-aws-oss-cluster-autoscaler-appset.yaml-  name: addons-aws-cluster-autoscaler
./hack/exclude/aws/addons-aws-oss-cluster-autoscaler-appset.yaml-  annotations:
./hack/exclude/aws/addons-aws-oss-cluster-autoscaler-appset.yaml:    argocd.argoproj.io/sync-wave: "3"
--
./hack/exclude/aws/addons-aws-oss-karpenter-appset.yaml-kind: ApplicationSet
./hack/exclude/aws/addons-aws-oss-karpenter-appset.yaml-metadata:
./hack/exclude/aws/addons-aws-oss-karpenter-appset.yaml-  name: addons-aws-karpenter
./hack/exclude/aws/addons-aws-oss-karpenter-appset.yaml-  annotations:
./hack/exclude/aws/addons-aws-oss-karpenter-appset.yaml:    argocd.argoproj.io/sync-wave: "1"

[csantanapr]

About your comment

Another thing I've been wondering is: What is the benefit of passing cluster variables to argo via the cluster secret, as opposed to having some helm-based App of Apps, where cluster variables can be passed to child apps via helm values?

The design of the pattern is to make the template git repo work out of the box for centralize hub-spoke, that you have one argocd deploying to multiple clusters.

With App of Apps you loose a lot of benefits of using ApplicationSets, we are kind of doing App or Apps but with ApplicationSet in the middle. This way the team starts to think on using ApplicationSets from day 1.

In the backlog there is a request to create a helm chart that renders the applicationsets this would gitops-bridge-dev/gitops-bridge-argocd-control-plane-template#40

[csantanapr]

We have a Slack channel in CNCF argo-cd-gitops-bridge if you want to join to discuss more

[csantanapr]

There is a Issue and PR in the works that adds the dependsOn on Applications that should help with ordering for apps (ie across clusters)
argoproj/argo-cd#7437
argoproj/argo-cd#15280

[csantanapr]

About passing thru cluster secret vs. helm boostrap, there are benefits that you can put this data in cluster usig argo api-server, or other IaC (Pulumi, Crossplane, etc) don't have to worry on passing the values to a helm chart (this will make everyone have their own version of app or apps helm chart) and just put the minimum metadata on the cluster without worrying how the bootstrap is done, they responsibility as IaC is to create cluster and create cluster secret, that's it, for spoke cluters there is no helm chart to boostrap, just registering a remote cluster all the addons start deploying to the remote cluster (not running argo)

[alex-treebeard]

thanks for this clarification...joined slack!