ApplicationSet: Support for heterogeneous source types in application set with generators

[TalhaNaeem1]

Summary

I used argoCD application-set to deploy my helm charts from one source and my k8s manifests from another source. But there is no support for multiple sources in application-set in argoproj.io/v1alpha1 API version.

Motivation

To deploy different application sourced from different points, like helm charts from some repo at bitnami and customize packages from my own private repo.

Proposal

Multiple sources must be support in applicationset so that we can deploy multiple application by getting deployment files from different sources.

[TalhaNaeem1]

My use is as follows:
Source code is in a Gitlab repo and I am using argoCD to deploy my different applications in different clusters. For that, I am using Application-Set in ArgoCD along with generators.
In my git repo, I have some helm chart packages and some simple kubernetes manifests, and the RepoURLs for both are different as we have to provide URL of helm package registry of the gitlab repo. I tried to use generators in application set to implement this logic, but It did not support multi-source for application. While in the source section, we can provide only one RepoURL(<>) as shown below:

  template:
    metadata:
      name: '{{name}}'
    spec:
      project: default
      source:
        repoURL: '<<URL>>
        targetRevision: '{{targetRevision}}'
        path: '{{path}}'
      
       
      destination:
        server: https://kubernetes.default.svc
        namespace: argocd

[crenshaw-dev]

Do you want a single Application created by ApplicationSet to support multiple sources? Or do you just need the ApplicationSet to be able to create Applications, each with their own source type?

In other words, are you looking for this:

  template:
    metadata:
      name: '{{name}}'
    spec:
      project: default
      sources:
        - repoURL: '<<URL>>
          targetRevision: '{{targetRevision}}'
          path: '{{path}}'
        - repoURL: '<<URL2>>
          targetRevision: '{{targetRevision}}'
          path: '{{path}}'
      destination:
        server: https://kubernetes.default.svc
        namespace: argocd

Or this:

  template:
    metadata:
      name: '{{name}}'
    spec:
      project: default
      source:
        {{ if somecondition then }}
        repoURL: '<<URL>>
        targetRevision: '{{targetRevision}}'
        path: '{{path}}'
        {{ else }}
        repoURL: '<<URL2>>
        targetRevision: '{{targetRevision}}'
        path: '{{path}}'
        {{ end }}
      destination:
        server: https://kubernetes.default.svc
        namespace: argocd

[TalhaNaeem1]

I have two different source repos (GitLab), one having a number of K8s manifests and the other having a number of helm charts (from package registry). I have to deploy all of these through one application-set.

apiVersion: [argoproj.io/v1alpha1](http://argoproj.io/v1alpha1)
kind: ApplicationSet
metadata:
  name: alldeployments
  namespace: argocd
spec:
  generators:  
    - list:
        elements:
        - repoURL: https://gitlab.com/cadsfs/csa.git
          targetRevision: demo
          path: devops/k8s-manifests/demo/frontend         
          name: front
        - chart: cadence-server
          targetRevision: 0.1.4 
          repoURL:  https://gitlab.com/api/v4/projects/33509869/packages/helm/stable
          name: 'cs'
        - chart: edge-stack
          targetRevision: 7.3.4 
          repoURL:  https://gitlab.com/api/v4/projects/33509869/packages/helm/stable
          name: 'es'
        - chart: cadence-worker-service
          targetRevision: 0.1.5
          repoURL:  https://gitlab.com/api/v4/projects/33509869/packages/helm/stable
          name: 'cws'

**Here under template.spec.source, I want to get my k8s manifests first and then helm charts one by one. KEEPING IN MIND THAT WE WILL HAVE THE PARAMETER chart FOR HELM, AND path FOR K8S MANIFESTS.

template:
  metadata:
    name: '{{name}}'
  spec:
    project: default
    source:
      repoURL: '{{repoURL}}' 
      targetRevision: '{{targetRevision}}'
      path: '{{path}}'
      chart: '{{chart}}'
     
    destination:
      server: https://kubernetes.default.svc/
      namespace: argocd

[TalhaNaeem1]

Do you want a single Application created by ApplicationSet to support multiple sources? Or do you just need the ApplicationSet to be able to create Applications, each with their own source type?

In other words, are you looking for this:

  template:
    metadata:
      name: '{{name}}'
    spec:
      project: default
      sources:
        - repoURL: '<<URL>>
          targetRevision: '{{targetRevision}}'
          path: '{{path}}'
        - repoURL: '<<URL2>>
          targetRevision: '{{targetRevision}}'
          path: '{{path}}'
      destination:
        server: https://kubernetes.default.svc
        namespace: argocd

Or this:

  template:
    metadata:
      name: '{{name}}'
    spec:
      project: default
      source:
        {{ if somecondition then }}
        repoURL: '<<URL>>
        targetRevision: '{{targetRevision}}'
        path: '{{path}}'
        {{ else }}
        repoURL: '<<URL2>>
        targetRevision: '{{targetRevision}}'
        path: '{{path}}'
        {{ end }}
      destination:
        server: https://kubernetes.default.svc
        namespace: argocd

Yes close to the second scenario, but the list under source section is to be generated by a generator, first I tried to use matrix generator for that, later on, I concluded to use a list generator by writing all my sources within it.
@crenshaw-dev I have elaborated my use case in the latest comment as well. Please check. Thanks in advance.

[reddymh]

@crenshaw-dev argocd applicationset supports multiple source (multiple repository/helm chart) and one destination to deploy?

[crenshaw-dev]

@reddymh I think what's being described here is multi-source Applications which is currently a work in progress.

[joebowbeer]

See #2789 (comment)

[tback]

I think the issue number for multi source applications is #677

[timblaktu]

@crenshaw-dev FWIW we also have need for your second use case: ApplicationSet encapsulating Applications of heterogeneous types: specifically source.helm and also sources that refer to "normal" Kubernetes manifests.

After researching the relevant issues I am now proposing to my team that we workaround this limitation by defining Kustomize overlays as a wrapper around our helm charts, removing our need for homogeneity at the ApplicationSet level.

Does anyone know of known issues with this approach?

We would initially be doing Zero modification to the underlying charts.

[crenshaw-dev]

I think this will solve the need for heterogeneous types: #11567

[crenshaw-dev]

Just realized there's a workaround: don't set directory, helm, kustomize, or plugin in the Application's spec.source field.

Instead use the .argocd-source.yaml override file in the Application's source repository: https://argo-cd.readthedocs.io/en/latest/user-guide/parameters/#store-overrides-in-git

It's not perfect, but might unblock some use cases.

[qtheya]

Here is one more case when ApplicationSet needs multiple sources for valueFiles.
Some public chart (doesn't matter which one) has value field for sensitive data (i.e database password). So it is necessary to have possibility to set 1st source of actually helm chart and 2nd for encrypted (i.e sops) values.enc.yaml with sensitive data (database password) in private repo

[r-arek]

@qtheya Yeah, this case is problematic for me also. I'm using applicationsets widely in my setup (mostly because of generators). It works great when I provide path to my own helm chart and values files - I have different values files for each environment, for example:

...
    spec:
      source:
        repoURL: 'https://github.com/my-argo-repo.git'
        targetRevision: HEAD
        path: charts/app
        helm:
          valueFiles: 
            - values-{{metadata.labels.env}}.yaml

And It would be great to be able to do the same for external helm charts maintained by community.

[jastBytes]

Is this in active development? Since Apps can have multiple sources it should be possible to have that on AppSets too right?

[vl-kp]

i integrated helm+git plugin in order to solve my problem plus git+ssh in valuesFile, but unfortunate, require hard refresh for the content in values file change...

[crenshaw-dev]

"Multiple sources" are supported (i.e. the spec.template.spec.sources field.

Heterogeneous source types are not yet supported. An app must use the same source type for all sources (i.e. either all Helm or all Kustomize).

There's an open PR which would solve the problem, but in a hacky way. I'm going to get with the PR author and brainstorm a less hacky solution. But that'll be 2.9, I won't have time before 2.8.

[wofr]

I'm using argocd-server: v2.6.7+5bcd846
I do have an ApplicationSet where I would like to use "Multiple sources"

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
.....
  template:
    spec:
      project: test
      source:
       - repoURL: git@github.com:****
         targetRevision: "{{repoBranch}}"
         path: chart
         helm:
           valueFiles:
            - secrets+age-import:///helm-secrets-private-keys/key.txt?../environmentconfigs/{{environmentcfgpath}}/globalsecrets.yaml
            - $val/environmentconfigs/{{environmentcfgpath}}/globalvalues.yaml
       - repoURL: git@github.com:**********
         targetRevision: "{{configBranch}}"
         ref: val

but I do get the following error when applying it

argocd appset create .\apps.yaml --upsert
time="2023-07-06T15:10:28+02:00" level=warning msg="Failed to invoke grpc call. Use flag --grpc-web in grpc calls. To avoid this warning message, use flag --grpc-web."
time="2023-07-06T15:10:28+02:00" level=fatal msg="error reading applicationset from file .\\apps.yaml: error unmarshalling appset: json: cannot unmarshal array into Go struct field ApplicationSpec.spec.template.spec.source of type v1alpha1.ApplicationSource"

If I define just one source it works fine. I do not know exactly what my problem is because I thougth wit ARgocd >2.6 this should work.

 spec:
      project: test
      source:
         repoURL: git@github.com:****
         targetRevision: "{{repoBranch}}"
         path: chart
         helm:
           valueFiles:
            - secrets+age-import:///helm-secrets-private-keys/key.txt?../environmentconfigs/{{environmentcfgpath}}/globalsecrets.yaml
            - $val/environmentconfigs/{{environmentcfgpath}}/globalvalues.yaml

Any idea what my problem could be?

[grahamprimm]

I'm having the same issue as above on version 2.6.7.

[crenshaw-dev]

@wofr change the source field to sources. They're two different fields.

[grahamprimm]

Mine is set to sources and i'm getting this error:
one or more objects failed to apply, reason: ApplicationSet.argoproj.io "test-cert-manager" is invalid: spec.template.spec.source: Required value

AppSet:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: test-cert-manager
  namespace: argocd
  labels:
    app.kubernetes.io/component: cert-manager
spec:
  generators:
    - list:
        elements:
        - cluster: cluster-name
          url: https://cluster-url
  template:
    metadata:
      name: 'cert-manager-{{cluster}}'
      labels:
        app.kubernetes.io/component: cert-manager
        app.kubernetes.io/name: 'cert-manager-{{cluster}}'
    spec:
      project: 'ds.{{cluster}}'
      sources:
      - repoURL: 'https://gitlab/services.git'
        targetRevision: master
        ref: values
      - chart: cert-manager
        repoURL: https://registry/cert-manager
        targetRevision: v1.9.1
        helm:
          valueFiles:
          - $values/cert-manager/values.yaml
      destination:
        server: '{{url}}'
        namespace: cert-manager
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
        - CreateNamespace=true

[crenshaw-dev]

@grahamprimm make sure your ApplicationSet CRD is up to date. source used to be a required field, but now it's not.

[grahamprimm]

That worked, thanks for the help!