Skip to content

Commit

Permalink
feat: Multiple sources for applications (argoproj#2789) (argoproj#10432)
Browse files Browse the repository at this point in the history 
* feat: support multiple sources for application

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

remove debug logging and unwanted code

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

fix lint and unit test errors

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

fix lint and unit test errors

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

* fix bug introduced after rebase

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

executed make codegen

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

* rebase with master

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

remove unwanted logging

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

fix ci failures

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

* fix index out of bounds error

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

* ui fixes

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>
Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>
Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

* add revisions to SyncOperation for rollback

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

* change Source to *ApplicationSource in ApplicationSpec

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

* fix env variable read logic for ValueFiles

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

* update multiple sources doc

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

* add repository lock and checkout target revision

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

fix codegen

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

* checkout all sources before generating manifest

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

generate mock reposerverclient

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

* update logic for returning ManifestResponse to avoid nil pointer issues

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

* fix nil reference and key mismatch bugs; add more logs (argoproj#6)

* fix nil reference and key mismatch bugs; add more logs
* remove temporary comment
* addressed the lint failure and added chart to RefTargeRevisionMapping
* normalize git repo (argoproj#7)
* do not leak lock releases
* prevent deadlock
* allow spec update
* move settings fetch outside loop
* cache busing
* return err instead of logging it
* no caching in test
* fix cache key marshaling

Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com>

Rebase with master

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

* update grpc field numbers

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

* path resolution tests (argoproj#12)

Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com>

do things in better ways

Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com>

consolidate

Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com>

Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com>
Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

* add regex check for value of source.ref

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

* add webhook tests

Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com>

change Source to *ApplicationSource in ApplicationSpec

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

address PR comments

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

rebase with master

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

Retrigger CI pipeline

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

rebased with master

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

* fix env variable read logic for ValueFiles

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

Address PR comments

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

* add repository lock and checkout target revision

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

fix codegen

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

* checkout all sources before generating manifest

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

* generate mock reposerverclient

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

* address comments

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

* update logic for returning ManifestResponse to avoid nil pointer issues

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

* fix nil reference and key mismatch bugs; add more logs (argoproj#6)

* fix nil reference and key mismatch bugs; add more logs
* remove temporary comment
* addressed the lint failure and added chart to RefTargeRevisionMapping
* normalize git repo (argoproj#7)
* do not leak lock releases
* prevent deadlock
* allow spec update
* move settings fetch outside loop
* cache busing
* return err instead of logging it
* no caching in test
* fix cache key marshaling

Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com>

Rebase with master

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

* update grpc field numbers

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

* add regex check for value of source.ref

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

* Rebase with master

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

* Added unit tests (argoproj#15)

* add unit tests 1
* fix lint

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

* fix application parameters tab, rebased UI changes, tests

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

* More tests (argoproj#16)

* more tests

Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com>

fix lint error

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

Test get ref sources (argoproj#17)

* test GetRefSources

Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com>

* fix lint

Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com>

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>
Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com>
Co-authored-by: ishitasequeira <ishiseq29@gmail.com>

GenerateManifests test (argoproj#18)

* GenerateManifests test

Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com>

Fix broken tests (argoproj#19)

* fix broken tests

Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com>

Symlink test (argoproj#20)

* check referenced sources for out-of-bounds symlinks

Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com>

* unlock the values file repo before doing a symlink check (argoproj#22)

Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com>

* multi source docs (argoproj#21)

* multi source docs

Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com>

* fix warning title

Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com>

* clarify

Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com>

* clarify

Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com>

Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com>

* add e2e tests for multiple sources and fix UI lint (argoproj#23)

* add e2e tests for multiple sources and fix UI lint

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

* add auto-sync and hard refresh to e2e tests

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

* change refresh type to RefreshTypeNormal for e2e

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

* update e2e testcase with helm data

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

* add TestMultiSourceAppWithSourceOverride

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

* add missing yaml file

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

* rebase with master

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

* rebase with master

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

* fix lint

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

Signed-off-by: ishitasequeira <ishiseq29@gmail.com>
Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>
Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com>
Co-authored-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com>
Signed-off-by: schakrad <chakradari.sindhu@gmail.com>
  • Loading branch information
@ishitasequeira @crenshaw-dev @schakrad
2 people authored and schakrad committed Mar 14, 2023
1 parent e36ee5e commit a928e3b
Show file tree
Hide file tree
Showing 90 changed files with 27,950 additions and 4,186 deletions.
28 changes: 14 additions & 14 deletions applicationset/controllers/applicationset_controller_test.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -684,7 +684,7 @@ func TestCreateOrUpdateInCluster(t *testing.T) {
Template: argov1alpha1.ApplicationSetTemplate{
Spec: argov1alpha1.ApplicationSpec{
Project: "project",
Source: argov1alpha1.ApplicationSource{Path: "path", TargetRevision: "revision", RepoURL: "repoURL"},
Source: &argov1alpha1.ApplicationSource{Path: "path", TargetRevision: "revision", RepoURL: "repoURL"},
Destination: argov1alpha1.ApplicationDestination{Server: "server", Namespace: "namespace"},
},
},
Expand Down Expand Up @@ -721,7 +721,7 @@ func TestCreateOrUpdateInCluster(t *testing.T) {
},
Spec: argov1alpha1.ApplicationSpec{
Project: "project",
Source: argov1alpha1.ApplicationSource{Path: "path", TargetRevision: "revision", RepoURL: "repoURL"},
Source: &argov1alpha1.ApplicationSource{Path: "path", TargetRevision: "revision", RepoURL: "repoURL"},
Destination: argov1alpha1.ApplicationDestination{Server: "server", Namespace: "namespace"},
},
},
Expand All @@ -741,7 +741,7 @@ func TestCreateOrUpdateInCluster(t *testing.T) {
},
Spec: argov1alpha1.ApplicationSpec{
Project: "project",
Source: argov1alpha1.ApplicationSource{Path: "path", TargetRevision: "revision", RepoURL: "repoURL"},
Source: &argov1alpha1.ApplicationSource{Path: "path", TargetRevision: "revision", RepoURL: "repoURL"},
Destination: argov1alpha1.ApplicationDestination{Server: "server", Namespace: "namespace"},
},
Status: argov1alpha1.ApplicationStatus{
Expand Down Expand Up @@ -918,7 +918,7 @@ func TestRemoveFinalizerOnInvalidDestination_FinalizerTypes(t *testing.T) {
},
Spec: argov1alpha1.ApplicationSpec{
Project: "project",
Source: argov1alpha1.ApplicationSource{Path: "path", TargetRevision: "revision", RepoURL: "repoURL"},
Source: &argov1alpha1.ApplicationSource{Path: "path", TargetRevision: "revision", RepoURL: "repoURL"},
// Destination is always invalid, for this test:
Destination: argov1alpha1.ApplicationDestination{Name: "my-cluster", Namespace: "namespace"},
},
Expand Down Expand Up @@ -1080,7 +1080,7 @@ func TestRemoveFinalizerOnInvalidDestination_DestinationTypes(t *testing.T) {
},
Spec: argov1alpha1.ApplicationSpec{
Project: "project",
Source: argov1alpha1.ApplicationSource{Path: "path", TargetRevision: "revision", RepoURL: "repoURL"},
Source: &argov1alpha1.ApplicationSource{Path: "path", TargetRevision: "revision", RepoURL: "repoURL"},
Destination: c.destinationField,
},
}
Expand Down Expand Up @@ -1589,7 +1589,7 @@ func TestValidateGeneratedApplications(t *testing.T) {
ObjectMeta: metav1.ObjectMeta{},
Spec: argov1alpha1.ApplicationSpec{
Project: "default",
Source: argov1alpha1.ApplicationSource{
Source: &argov1alpha1.ApplicationSource{
RepoURL: "https://url",
Path: "/",
TargetRevision: "HEAD",
Expand All @@ -1612,7 +1612,7 @@ func TestValidateGeneratedApplications(t *testing.T) {
ObjectMeta: metav1.ObjectMeta{},
Spec: argov1alpha1.ApplicationSpec{
Project: "default",
Source: argov1alpha1.ApplicationSource{
Source: &argov1alpha1.ApplicationSource{
RepoURL: "https://url",
Path: "/",
TargetRevision: "HEAD",
Expand All @@ -1636,7 +1636,7 @@ func TestValidateGeneratedApplications(t *testing.T) {
ObjectMeta: metav1.ObjectMeta{},
Spec: argov1alpha1.ApplicationSpec{
Project: "DOES-NOT-EXIST",
Source: argov1alpha1.ApplicationSource{
Source: &argov1alpha1.ApplicationSource{
RepoURL: "https://url",
Path: "/",
TargetRevision: "HEAD",
Expand All @@ -1659,7 +1659,7 @@ func TestValidateGeneratedApplications(t *testing.T) {
ObjectMeta: metav1.ObjectMeta{},
Spec: argov1alpha1.ApplicationSpec{
Project: "default",
Source: argov1alpha1.ApplicationSource{
Source: &argov1alpha1.ApplicationSource{
RepoURL: "https://url",
Path: "/",
TargetRevision: "HEAD",
Expand All @@ -1682,7 +1682,7 @@ func TestValidateGeneratedApplications(t *testing.T) {
ObjectMeta: metav1.ObjectMeta{},
Spec: argov1alpha1.ApplicationSpec{
Project: "default",
Source: argov1alpha1.ApplicationSource{
Source: &argov1alpha1.ApplicationSource{
RepoURL: "https://url",
Path: "/",
TargetRevision: "HEAD",
Expand Down Expand Up @@ -1811,7 +1811,7 @@ func TestReconcilerValidationErrorBehaviour(t *testing.T) {
Namespace: "argocd",
},
Spec: argov1alpha1.ApplicationSpec{
Source: argov1alpha1.ApplicationSource{RepoURL: "https://github.com/argoproj/argocd-example-apps", Path: "guestbook"},
Source: &argov1alpha1.ApplicationSource{RepoURL: "https://github.com/argoproj/argocd-example-apps", Path: "guestbook"},
Project: "default",
Destination: argov1alpha1.ApplicationDestination{Server: "{{.url}}"},
},
Expand Down Expand Up @@ -1954,7 +1954,7 @@ func TestGenerateAppsUsingPullRequestGenerator(t *testing.T) {
},
},
Spec: argov1alpha1.ApplicationSpec{
Source: argov1alpha1.ApplicationSource{
Source: &argov1alpha1.ApplicationSource{
RepoURL: "https://testurl/testRepo",
TargetRevision: "{{.head_short_sha}}",
},
Expand All @@ -1973,7 +1973,7 @@ func TestGenerateAppsUsingPullRequestGenerator(t *testing.T) {
},
},
Spec: v1alpha1.ApplicationSpec{
Source: v1alpha1.ApplicationSource{
Source: &v1alpha1.ApplicationSource{
RepoURL: "https://testurl/testRepo",
TargetRevision: "089d92cb",
},
Expand Down Expand Up @@ -2113,7 +2113,7 @@ func TestPolicies(t *testing.T) {
},
},
Spec: argov1alpha1.ApplicationSpec{
Source: argov1alpha1.ApplicationSource{RepoURL: "https://github.com/argoproj/argocd-example-apps", Path: "guestbook"},
Source: &argov1alpha1.ApplicationSource{RepoURL: "https://github.com/argoproj/argocd-example-apps", Path: "guestbook"},
Project: "default",
Destination: argov1alpha1.ApplicationDestination{Server: "https://kubernetes.default.svc"},
},
Expand Down
6 changes: 3 additions & 3 deletions applicationset/utils/utils_test.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ func TestRenderTemplateParams(t *testing.T) {
Namespace: "default",
},
Spec: argoappsv1.ApplicationSpec{
Source: argoappsv1.ApplicationSource{
Source: &argoappsv1.ApplicationSource{
Path: "",
RepoURL: "",
TargetRevision: "",
Expand Down Expand Up @@ -220,7 +220,7 @@ func TestRenderTemplateParamsGoTemplate(t *testing.T) {
Namespace: "default",
},
Spec: argoappsv1.ApplicationSpec{
Source: argoappsv1.ApplicationSource{
Source: &argoappsv1.ApplicationSource{
Path: "",
RepoURL: "",
TargetRevision: "",
Expand Down Expand Up @@ -511,7 +511,7 @@ func TestRenderTemplateParamsFinalizers(t *testing.T) {

emptyApplication := &argoappsv1.Application{
Spec: argoappsv1.ApplicationSpec{
Source: argoappsv1.ApplicationSource{
Source: &argoappsv1.ApplicationSource{
Path: "",
RepoURL: "",
TargetRevision: "",
Expand Down
74 changes: 74 additions & 0 deletions assets/swagger.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5831,6 +5831,10 @@
"plugin": {
"$ref": "#/definitions/v1alpha1ApplicationSourcePlugin"
},
"ref": {
"description": "Ref is reference to another source within sources field. This field will not be used if used with a `source` tag.",
"type": "string"
},
"repoURL": {
"type": "string",
"title": "RepoURL is the URL to the repository (Git or Helm) that contains the application manifests"
Expand Down Expand Up @@ -6068,6 +6072,13 @@
"source": {
"$ref": "#/definitions/v1alpha1ApplicationSource"
},
"sources": {
"type": "array",
"title": "Sources is a reference to the location of the application's manifests or chart",
"items": {
"$ref": "#/definitions/v1alpha1ApplicationSource"
}
},
"syncPolicy": {
"$ref": "#/definitions/v1alpha1SyncPolicy"
}
Expand Down Expand Up @@ -6118,6 +6129,13 @@
"type": "string",
"title": "SourceType specifies the type of this application"
},
"sourceTypes": {
"type": "array",
"title": "SourceTypes specifies the type of the sources included in the application",
"items": {
"type": "string"
}
},
"summary": {
"$ref": "#/definitions/v1alpha1ApplicationSummary"
},
Expand Down Expand Up @@ -6415,6 +6433,13 @@
},
"source": {
"$ref": "#/definitions/v1alpha1ApplicationSource"
},
"sources": {
"type": "array",
"title": "Sources is a reference to the application's multiple sources used for comparison",
"items": {
"$ref": "#/definitions/v1alpha1ApplicationSource"
}
}
}
},
Expand Down Expand Up @@ -7757,8 +7782,22 @@
"type": "string",
"title": "Revision holds the revision the sync was performed against"
},
"revisions": {
"type": "array",
"title": "Revisions holds the revision of each source in sources field the sync was performed against",
"items": {
"type": "string"
}
},
"source": {
"$ref": "#/definitions/v1alpha1ApplicationSource"
},
"sources": {
"type": "array",
"title": "Sources is a reference to the application sources used for the sync operation",
"items": {
"$ref": "#/definitions/v1alpha1ApplicationSource"
}
}
}
},
Expand Down Expand Up @@ -8059,9 +8098,23 @@
"description": "Revision is the revision (Git) or chart version (Helm) which to sync the application to\nIf omitted, will use the revision specified in app spec.",
"type": "string"
},
"revisions": {
"description": "Revisions is the list of revision (Git) or chart version (Helm) which to sync each source in sources field for the application to\nIf omitted, will use the revision specified in app spec.",
"type": "array",
"items": {
"type": "string"
}
},
"source": {
"$ref": "#/definitions/v1alpha1ApplicationSource"
},
"sources": {
"type": "array",
"title": "Sources overrides the source definition set in the application.\nThis is typically set in a Rollback operation and is nil during a Sync operation",
"items": {
"$ref": "#/definitions/v1alpha1ApplicationSource"
}
},
"syncOptions": {
"type": "array",
"title": "SyncOptions provide per-sync sync-options, e.g. Validate=false",
Expand Down Expand Up @@ -8107,8 +8160,22 @@
"type": "string",
"title": "Revision holds the revision this sync operation was performed to"
},
"revisions": {
"type": "array",
"title": "Revisions holds the revision this sync operation was performed for respective indexed source in sources field",
"items": {
"type": "string"
}
},
"source": {
"$ref": "#/definitions/v1alpha1ApplicationSource"
},
"sources": {
"type": "array",
"title": "Source records the application source information of the sync, used for comparing auto-sync",
"items": {
"$ref": "#/definitions/v1alpha1ApplicationSource"
}
}
}
},
Expand Down Expand Up @@ -8163,6 +8230,13 @@
"type": "string",
"title": "Revision contains information about the revision the comparison has been performed to"
},
"revisions": {
"type": "array",
"title": "Revisions contains information about the revisions of multiple sources the comparison has been performed to",
"items": {
"type": "string"
}
},
"status": {
"type": "string",
"title": "Status is the sync state of the comparison"
Expand Down
7 changes: 6 additions & 1 deletion cmd/argocd/commands/admin/app.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -401,7 +401,12 @@ func reconcileApplications(
return nil, err
}

res := appStateManager.CompareAppState(&app, proj, app.Spec.Source.TargetRevision, app.Spec.Source, false, false, nil)
sources := make([]v1alpha1.ApplicationSource, 0)
revisions := make([]string, 0)
sources = append(sources, app.Spec.GetSource())
revisions = append(revisions, app.Spec.GetSource().TargetRevision)

res := appStateManager.CompareAppState(&app, proj, revisions, sources, false, false, nil, false)
items = append(items, appReconcileResult{
Name: app.Name,
Conditions: app.Status.Conditions,
Expand Down
1 change: 1 addition & 0 deletions cmd/argocd/commands/admin/app_test.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,7 @@ func TestGetReconcileResults_Refresh(t *testing.T) {
Namespace: "default",
},
Spec: v1alpha1.ApplicationSpec{
Source: &v1alpha1.ApplicationSource{},
Project: "default",
Destination: v1alpha1.ApplicationDestination{
Server: v1alpha1.KubernetesInternalAPIServerAddr,
Expand Down
Loading

0 comments on commit a928e3b

Please sign in to comment.