WIP Add delete script task

[liyanwei93]

Add delete script task:

• check-object-prereqs job: After Installation CR deployed successfully, check whether Crossplane has been deployed by common services.
• Object: Create a deployment containing a delete script, which will not be controlled by Gitops.
• The check-object-prereqs job and Object will be deleted by Argocd when the argocd application is deleted. The left deployment will check the Installation CR for a deletionTimestamp, and if so will start the delete script.
• Add finalizers to avoid deleting deployments when argocd deleting AIManager namespace. When the script finishes running, it removes these "finalizers"
• The uninstall process is often blocked by installation CR because it cannot be removed successfully, which is the biggest problem facing now. （Even if run the delete script manually）

[morningspace]

@liyanwei93 I did a round of review and added some comments. Generally, I can understand this is a very hacky resolution, which can be working as an experimental feature, but is error-prone if that goes to be an official feature.

Essentially, we are simulating what a Kubernetes controller or an operator is supposed to do, but using a very tricky way.

After considering this carefully, I think we can leave it as an experimental feature for now. Long termly, we should seek ways to create a real operator which has to be co-located w/ the deployment w/ additional workload and footprint, or using seed cluster to remotely control it by pipeline as example.

cc @gyliu513

[morningspace]

This comment seems misleading.

[morningspace]

Suggested change

	echo "Waitting to be deleted......"
	echo "Waiting to be deleted ..."

[morningspace]

Suggested change

	echo "INFO: Crossplane kubernetes provider not available, waiting."
	echo "INFO: Crossplane provider kubernetes not available, waiting ..."

[morningspace]

More thoughts on the long term solution:

The clean up job is essentially very similar to a regular procedure, so that can be nicely modeled as a pipeline task, which can be triggered either manually or automatically. It's easy to encapsulate the clean up shell script into a Tekton task.

The drawback of the above idea is that it is a one-time off job, which can not be used to regularly monitoring the system for an uninstall need. This goes to 2nd option which is a real Kubernetes controller that has reconciliation logic to keep monitoring the system status. As an example, I think maybe the Crossplane provider ansible is a good choice in this case, because it is working as a controller inside a Kubernetes cluster w/ all the native Kubernetes behavior including the reconciliation, as well as the ability to call Ansible roles and playbooks remotely, e.g.: some shell scripts wrapped as Ansible playbook and hosted in a remote git repo. This can nicely handle our case.

[gyliu513]

Opened a github issue for argoCD at argoproj/argo-cd#9351 to discuss post uninstall

[gyliu513]

@morningspace @liyanwei93 as we are using helm chart, can we leverage helm chart hooks for uninstall https://helm.sh/docs/topics/charts_hooks/#writing-a-hook ?

[morningspace]

Some similar discussions here at: argoproj/argo-cd#7575, as I remember, we couldn't rely on helm hook when combing w/ Argo CD, because there's no such mapping between helm hook and Argo CD sync phase for now, which means the hook will not be triggered when we delete Argo application.