Adding new pattern for Crossplane Argocd GitOps #173

ajpaws · 2024-05-15T11:01:20Z

Issue #, if available:

Description of changes:

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

elamaran11

@ajpaws Thankyou for working on this and quick PR. There are few alignments to the approach we have to make :

First the arrangment is not aligning to the theme of the repo. You should have one bin ts file for this pattern which invokes the class in the lib file to run the pattern. Please check other patterns like SecureIngress Pattern which we worked for another blog.
I understand you most of the code i shared from my partner work but we cant reuse it as is, we need to refactor it work with patterns repo. We done need this file aws-addon-clusters-stack.ts for instance. Plus we need one pipeline which deploys management cluster and all child cluster and those pipeline code should be in lib folder like multi account monitoring pattern.
So basically most of the work you have in bin folder should move to lib folder refactored.
See if you have any opportunities to minimize the number of custom addons.
Also fix all GH Warnings and errors.
Overall great progress, we have some work to do. Lets connect if you have questions.

elamaran11

@ajpaws Awesome work. Love the idea of adding Pod Identity Addon via ClusterAuth We are very near to finish line. One minor feedback that i mentioned on bin file. Also lets add the architecture diagram of the blog optionally to make this complete.

bin/crossplane-argocd-gitops.ts

shapirov103

@ajpaws great work, a lot of effort. I went through the first round of review and let's fix the issues, I also posted a couple of questions. I will review again after you are done, and we will validate it as well. I realize there is a dependency on the workloads repo PR.

lib/crossplane-argocd-gitops/common/construct-utils.ts

lib/crossplane-argocd-gitops/custom-addons/crossplane-k8s-provider-addon.ts

shapirov103 · 2024-06-24T17:38:42Z

lib/crossplane-argocd-gitops/custom-addons/upbound-crossplane-addon.ts

@@ -0,0 +1,89 @@
+import 'source-map-support/register';


@elamaran11 is there a rationale why this addon is not in the blueprints repo? We have upbound addon there, it may be confusing to the customers how to reconcile this with the one in the blueprints repo. I assume we should deprecate or just replace the upbound addon in the blueprints.

@shapirov103 The upbound addon in Blueprints is EKS Addon and they are no longer supporting it. I asked the team but no response so the Ubound addon on the bluepeints should be depracated or this should be updated there. I think for now lets depracate upbound addon for upcoming release. I will plan to work to move this later to blueprints.

It is already in 1.15, let's replace it here, - too much code as it stands for the pattern.

Not addressed?

@ajpaws This comment?

I believe we are deprecating upbound addon in Blueprints is EKS Addon so no changed made on this. I would recommend go ahead with this for now since anyway we have code for all other upbound addons at one place in custom addons folder.

lib/crossplane-argocd-gitops/management-cluster-builder.ts

lib/crossplane-argocd-gitops/multi-cluster-pipeline.ts

docs/patterns/crosplane-argocd-gitops.md

lib/crossplane-argocd-gitops/custom-addons/upbound-crossplane-addon.ts

elamaran11

@ajpaws Fix this for GH Action to run.

lib/crossplane-argocd-gitops/multi-cluster-pipeline.ts

elamaran11 · 2024-07-13T10:52:47Z

@shapirov103 Could you please check again and let me know if this is good to merge.

shapirov103

@ajpaws please see my additional comments. Thank you for addressing my previous comments, I see some were deferred and that is ok.

I left a bunch of minor comments, but there a few important. Happy to discuss if needed. I will focus on the workloads PR.

@elamaran11 FYI ^^

docs/patterns/crossplane-argocd-gitops.md

lib/crossplane-argocd-gitops/multi-cluster-pipeline.ts

lib/crossplane-argocd-gitops/custom-addons/crossplane-k8s-provider-addon.ts

lib/crossplane-argocd-gitops/custom-addons/mgmt-role-teams.ts

shapirov103 · 2024-07-16T19:09:25Z

lib/crossplane-argocd-gitops/custom-addons/upbound-crossplane-addon.ts

@@ -0,0 +1,89 @@
+import 'source-map-support/register';


It is already in 1.15, let's replace it here, - too much code as it stands for the pattern.

lib/crossplane-argocd-gitops/multi-cluster-options.ts

…atterns

ajpaws · 2024-08-30T04:42:08Z

@shapirov103 Could you please check again and let me know if this is good to merge.

@ajpaws Thankyou for working on this and quick PR. There are few alignments to the approach we have to make :

First the arrangment is not aligning to the theme of the repo. You should have one bin ts file for this pattern which invokes the class in the lib file to run the pattern. Please check other patterns like SecureIngress Pattern which we worked for another blog.

I understand you most of the code i shared from my partner work but we cant reuse it as is, we need to refactor it work with patterns repo. We done need this file aws-addon-clusters-stack.ts for instance. Plus we need one pipeline which deploys management cluster and all child cluster and those pipeline code should be in lib folder like multi account monitoring pattern.

So basically most of the work you have in bin folder should move to lib folder refactored.

See if you have any opportunities to minimize the number of custom addons.

Also fix all GH Warnings and errors.
Overall great progress, we have some work to do. Lets connect if you have questions.

done. made all changes

shapirov103

Overall looks good to me, pending resolution of feedback and functional verification by @elamaran11

docs/patterns/crossplane-argocd-gitops.md

shapirov103 · 2024-08-30T14:30:37Z

docs/patterns/crossplane-argocd-gitops.md

+```
+make build
+make list
+make pattern aws-addon-clusters deploy


Is that the correct name? should not it be crossplace-argocd-gitops?

shapirov103 · 2024-08-30T14:49:41Z

lib/crossplane-argocd-gitops/custom-addons/upbound-crossplane-addon.ts

@@ -0,0 +1,89 @@
+import 'source-map-support/register';


Not addressed?

elamaran11 · 2024-08-30T17:29:30Z

@ajpaws Please pull from main to get past the Markdown error,.

merging changes from main to fix Markdown errors

…atterns

ajpaws · 2024-09-04T06:08:17Z

@ajpaws Please pull from main to get past the Markdown error,.

done

elamaran11 · 2024-09-10T14:00:58Z

docs/patterns/crossplane-argocd-gitops.md

+
+```shell
+export ACCOUNT_ID=$(aws sts get-caller-identity --output text --query Account)
+export AWS_REGION=$(curl -s 169.254.169.254/latest/dynamic/instance-identity/document | jq -r '.region')


This is not accurate for mac based deployments. May work from Cloud9. I think you should ask the readers to hardcode ther egion.

elamaran11 · 2024-09-10T14:14:27Z

lib/crossplane-argocd-gitops/multi-cluster-pipeline.ts

+
+        const gitProps = {
+            owner :'aws-samples',
+            secretName : props.gitHubSecret ?? 'cdk_blueprints_gitops_github_secret',


Secret you are creating is export CDK_REPO_AWS_SECRET_NAME="cdk_blueprints_github_secret" but the code has cdk_blueprints_gitops_github_secret

elamaran11 · 2024-09-10T20:52:07Z

@ajpaws Cleanup Instructions are missing

elamaran11 · 2024-09-10T21:30:59Z

docs/patterns/crossplane-argocd-gitops.md

+The example command looks like below.
+
+```shell
+aws eks update-kubeconfig --name mgmt-cluster --region us-west-2 --role-arn arn:aws:iam::ACCOUNT_ID:role/mgmt-cluster-stage-mgmt-c-managementclusterAccessRo-XYSC5PKL8WnA


This is not correct

elamaran11 · 2024-09-10T21:34:37Z

docs/patterns/crossplane-argocd-gitops.md

+The example command looks like below.
+
+```shell
+aws eks update-kubeconfig --name workload-amd-1-29-blueprint --region us-west-2 --role-arn arn:aws:iam::ACCOUNT_ID:role/eks-workload-connector-role


This is in correct too. You should basically have a command to query outputs from stack and grab or from console.

elamaran11 · 2024-09-10T21:34:53Z

docs/patterns/crossplane-argocd-gitops.md

+The example command looks like below.
+
+```shell
+aws eks update-kubeconfig --name workload-arm-1-29-blueprint --region us-west-2 --role-arn arn:aws:iam::$ACCOUNT_ID:role/eks-workload-connector-role


This is in correct too. You should basically have a command to query outputs from stack and grab or from console.

elamaran11 · 2024-09-10T21:35:11Z

docs/patterns/crossplane-argocd-gitops.md

+The output will look like below.
+
+```shell
+Added new context arn:aws:eks:us-west-2:ACCOUNT_ID:cluster/workload-amd-1-29-blueprint to /Users/jalawala/.kube/config


Output can be removed.

elamaran11 · 2024-09-10T21:35:15Z

docs/patterns/crossplane-argocd-gitops.md

+The output will look like below.
+
+```shell
+Added new context arn:aws:eks:us-west-2:ACCOUNT_ID:cluster/workload-arm-1-29-blueprint to /Users/jalawala/.kube/config


Output can be removed.

elamaran11 · 2024-09-10T21:35:22Z

docs/patterns/crossplane-argocd-gitops.md

+The output will look like below.
+
+```shell
+Updated context arn:aws:eks:us-west-2:ACCOUNT_ID:cluster/mgmt-cluster in /Users/<user_name>/.kube/config


Output can be removed.

elamaran11 · 2024-09-10T21:36:42Z

docs/patterns/crossplane-argocd-gitops.md

+
+## Test 
+
+### Install the ArgoCD ALI


elamaran11 · 2024-09-10T21:38:39Z

docs/patterns/crossplane-argocd-gitops.md

+Get the ArgoCD Admin password using below command.
+
+```shell
+kubectl --context $MANAGEMENT_CLUSTER_CONTEXT  -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo


I dont see this secret, so i cant get to Argo Server. See what is missing

❯ kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo
─╯
Error from server (NotFound): secrets "argocd-initial-admin-secret" not found

elamaran11 · 2024-09-10T21:44:38Z

docs/patterns/crossplane-argocd-gitops.md

+Add EKS Cluster to ArgoCD.
+
+```shell
+argocd cluster add $MANAGEMENT_CLUSTER_CONTEXT


Fails

─╯ WARNING: This will create a service account `argocd-manager` on the cluster referenced by context `arn:aws:eks:us-west-2:940019131157:cluster/mgmt-cluster` with full cluster level privileges. Do you want to continue [y/N]? y INFO[0003] ServiceAccount "argocd-manager" created in namespace "kube-system" INFO[0003] ClusterRole "argocd-manager-role" created INFO[0003] ClusterRoleBinding "argocd-manager-role-binding" created INFO[0009] Created bearer token secret for ServiceAccount "argocd-manager" FATA[0009] Argo CD server address unspecified```

When i describe the pod. i see secret failures. Though i have github-token in same region.

Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 43m default-scheduler Successfully assigned argocd/blueprints-addon-argocd-server-666866878b-92qgh to ip-10-0-189-12.us-west-2.compute.internal Warning FailedMount 6s (x29 over 43m) kubelet MountVolume.SetUp failed for volume "blueprints-secret-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod argocd/blueprints-addon-argocd-server-666866878b-92qgh, err: rpc error: code = Unknown desc = Failed to fetch secret from all regions: github-token

elamaran11 · 2024-09-11T12:05:47Z

docs/patterns/crossplane-argocd-gitops.md

+Run the below command to get the list of EKS add-on Objects deployed in the Management Cluster.
+
+```shell
+kubectl  --context $MANAGEMENT_CLUSTER_CONTEXT get add-ons.eks.aws.upbound.io


This command should be kubectl --context $MANAGEMENT_CLUSTER_CONTEXT get addons.eks.aws.upbound.io

Adding new pattern for Crossplane Argocd GitOps

4eb131d

elamaran11 requested changes May 15, 2024

View reviewed changes

ajpaws added 3 commits June 3, 2024 17:04

fixing review comments for cleanup

81ac009

adding error handling during stack creation

a5d7aa3

changing workloads path for argocd

26c4fd6

elamaran11 reviewed Jun 11, 2024

View reviewed changes

bin/crossplane-argocd-gitops.ts Outdated Show resolved Hide resolved

cleanup and add arch diagram

7a6a63a

shapirov103 reviewed Jun 24, 2024

View reviewed changes

elamaran11 reviewed Jun 25, 2024

View reviewed changes

lib/crossplane-argocd-gitops/custom-addons/upbound-crossplane-addon.ts Outdated Show resolved Hide resolved

ajpaws added 4 commits June 25, 2024 12:26

fixing review comments

ffe0608

passing variables values via gitops approach from argoCD

a7f97ae

fixing doc errors

60d940e

add EKS Provider Addon and cleanup

ba54f88

elamaran11 reviewed Jun 26, 2024

View reviewed changes

lib/crossplane-argocd-gitops/multi-cluster-pipeline.ts Outdated Show resolved Hide resolved

fixing typos

41331fa

shapirov103 reviewed Jul 16, 2024

View reviewed changes

ajpaws added 13 commits July 22, 2024 15:05

fixing review comments

00b0337

automating iam role creation

fd069ae

testing the stack

32eb8da

fixing pipeline errors

ee94079

fixing deployment issues

32c457e

fixing few deployment issues

896e62f

adding custom iam role creator resource

c3e1995

testing custom iam role creator resource

80a8634

fixing duplicate role creation

f8f9442

testing custom iam role provider resource

26e0dca

fixing duplicate iam role creation

02aec06

clean up

89f43f4

cleanup

41bdaaf

ajpaws added 6 commits August 27, 2024 10:00

creating dedicated IRSA role for EKS Provider

2e5d709

Merge branch 'main' of https://github.com/ajpaws/cdk-eks-blueprints-p…

8443c72

…atterns

creating dedicated iam role for eks provider pod

7bdb3db

automating secret creation

f68efa8

automating argocd secret creation

152fc70

cleanup and doc update

8ae1ea7

shapirov103 reviewed Aug 30, 2024

View reviewed changes

ajpaws and others added 3 commits September 4, 2024 11:34

Merge pull request #1 from aws-samples/main

0dcda7a

merging changes from main to fix Markdown errors

update README

58aaad5

Merge branch 'main' of https://github.com/ajpaws/cdk-eks-blueprints-p…

311efbb

…atterns

fixing README Issues

fd22534

elamaran11 reviewed Sep 10, 2024

View reviewed changes

ajpaws and others added 4 commits September 10, 2024 21:57

update README

e58d132

Update multi-cluster-pipeline.ts

1f3f4bb

Update multi-cluster-pipeline.ts

2902111

Update multi-cluster-pipeline.ts

55d345b

elamaran11 requested changes Sep 10, 2024

View reviewed changes

removing secret store secret

d0d6615

elamaran11 requested changes Sep 11, 2024

View reviewed changes

ajpaws added 5 commits September 11, 2024 20:25

fixing IRSA issues

b18f5e7

testing IRSA Issues

e796464

testing IRSA roles

c878437

testing with new mgmt cluster

ee3b16f

re-creating mgmt cluster

1c6d95b

Adding new pattern for Crossplane Argocd GitOps #173

Are you sure you want to change the base?

Adding new pattern for Crossplane Argocd GitOps #173

Conversation

ajpaws commented May 15, 2024

elamaran11 left a comment • edited Loading

Choose a reason for hiding this comment

elamaran11 left a comment

Choose a reason for hiding this comment

shapirov103 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elamaran11 left a comment

Choose a reason for hiding this comment

elamaran11 commented Jul 13, 2024

shapirov103 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ajpaws commented Aug 30, 2024

shapirov103 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elamaran11 commented Aug 30, 2024

ajpaws commented Sep 4, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elamaran11 commented Sep 10, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elamaran11 left a comment •

edited

Loading