A way to do something in hook after delete the Argo Application

[morningspace]

Summary

It looks there's no way to do some post delete work when I delete an Argo Application, e.g.: to clean up something.

Motivation

For example, I have a set of Kubernetes resources for OLM, including Subscription, certain Custom Resources, etc. that can be synced by Argo CD to deploy on target cluster. During deploying, OLM will auto-generate a resource called ClusterServiceVersion, which will trigger the launch of operator, then operator provision the actual workloads.

When I delete the Argo Application to uninstall the existing deployments. All resources will be dropped on target cluster except the ClusterServiceVersion and the operator, which is not a clean uninstall to me.

Proposal

In such a case, do we have an option that allows customized clean up work defined somewhere such as in a hook, so that I can delete the ClusterServiceVersion resource and the operator inside the hook.

Helm has post-delete hook, but there's no equivalent in Argo CD.

[omgapuppy]

Probably not the neatest solution, but a workaround could be using PostSync hooks with some logic of identifying the nuked app. This would only work if the app-of-apps pattern is used and the parent application contains the hook definition. Native functionality would be nice as it'd greatly reduce complexity.

[morningspace]

@omgapuppy Could you please share a bit more details on how the workaround works? I'd like to have a try before it can be natively supported :-) Thanks!

a workaround could be using PostSync hooks with some logic of identifying the nuked app

[morningspace]

Oh, are you saying that people may do the clean up work in hook in the next round of sync? Should it be the PreSync instead of PostSync? i.e.: to clean up before start the sync?

[seansund]

Has any more thought been given to this issue? Ideally, when an app is removed from the GitOps repo and subsequently deleted from the cluster by ArgoCD there should be no trace remaining of the app. There are a number of resources that do not clean themselves up properly when they are deleted and would need some additional help. Helm has the post-delete hook for this purpose.

A PostSync hook could possibly work but as @omgapuppy points out, you need to add extra logic to determine the context of the sync. Generally, it would be nice if the Hooks had some/any context information available to them (perhaps through injected environment variables)

[morningspace]

@seansund I wonder if the PostSync hook will be triggered when I delete an Argo Application. (i.e.: to uninstall Argo Application). I thought sync hooks are used for something install, not uninstall, right?

Another thing that I'm worried about is that, typically happening in a scenario such as operator who has k8 custom resource to be watched. When in uninstall, a delete of Argo Application, might trigger the deletion of the k8 custom resource and the operator at the same time, and in some cases, the operator will be deleted prior to the k8 custom resource, this will lead to the k8 custom resource remained and never be deleted.

[seansund]

@morningspace I was assuming the PostSync hook would be called at the end of a sync operation, whether create or delete. However, it doesn't look like PostSync hooks are called at all when the application is deleted. On some level it makes sense because there isn't an application anymore for the sync to run against.

[nneram]

Hi, I'm looking for a way to remove some finalizer in resources after that the operator has been deleted by argocd and didn't got the time to remove it. I can't use helm webhook because I'm using a plugin to apply my app so I tried PostSync et PreSync, and none of the two are working for a deletion...
So I would be happy to hear from you if you have a solution for this, and I continue my research to find a way !

[morningspace]

@nneram Having the same problem here. I'm really thinking that there needs to be a builtin feature in Argo CD that supports this use case, the deletion of the resources, although I guess it's not something that gets high attention at the moment.

[nneram]

#9296 (comment)
This can can maybe help you, it solved my problem 👍🏻

[rbabyuk-vs]

I was surprised that none of helm 'delete' hooks are supported, there is a need on my side to do extra clean-up when deleting the app, intentially I was about to create k8s job and execute it automatically if the argocd app is being deleted.

[mohammedshirine]

We are tussling with similar use case but with applicationSets. I am looking for a nice argo driven clean up hook points to run a little clean up for following use cases.

• We use PR generator to create preview envs with configurations living in an ephemeral pr branch. This branch has to go away once PR is merged (when application is removed).
• We use createNamespace sync option. ArgoCD does not remove the namespaces it creates.

We would treasure a postDelete hook for running these kind of clean up jobs.

An alternative option is to run extensive github actions pipelines to compute the post delete trigger. This is exhaustive as there are multiple repositories participating in PR generation and kind of takes away the whole simplicity of using argo.

Are there any thing we can do to get out of this situation ?

[lamebear]

Similar to what @mohammedshirine is doing, we use Argo to create vClusters via a Helm chart for various environments.

• We have a Job annotated with helm.sh/hook: post-install that creates cluster credential secrets for Argo.
• We have a Jobs annotated with helm.sh/hook: pre-delete that will delete any applications deployed to the vCluster and then delete the cluster credential secret.

Since Argo does not run the Job's annotated with helm.sh/hook: pre-delete, deletion of the cluster application can get stuck and the cluster credentials do not get removed.

[RammusXu]

In our case, we are trying to manage a service lifecycle by Helm. So it needs to implement in helm.sh/hook: post-delete

[dyasny]

Same here, need a way to do cleanup after the app is deleted. Was expecting helm hooks to work, and was very surprised when they didn't.

[mel1nn]

Same problem for me with the uninstallation of longhorn. A job needs to be run before deleting the resources (https://github.com/longhorn/charts/blob/master/charts/longhorn/templates/uninstall-job.yaml). This one has a helm.sh/hook: pre-delete annotation but it is not launched during the application deletion. I do not understand why Argo CD does not support this kind of hooks.

[mpluhar]

Same issue here. It is not great to have to work around the lack of pre/post delete hooks.

[markercm]

We have a similar use-case where it would be helpful to have a PostDelete hook implemented for additional cleanup activities.

[winterrobert]

Same here - an post delete hook seems a really good fit to the Pull Request Generator to dynamically create and destroy preview environments.

[bravosierrasierra]

Same here: can not delete environment database without database CRD solution like CrossPlane.

[omgapuppy]

Same here: can not delete environment database without database CRD solution like CrossPlane.

This feels out of scope for Argo CD, and precisely why Crossplane et al exist 🤔

[bravosierrasierra]

Same here: can not delete environment database without database CRD solution like CrossPlane.

This feels out of scope for Argo CD, and precisely why Crossplane et al exist 🤔

but why argocd application finalizer can not throw kubernetes job on application delete to call user finish tasks? Or push argo events? Applicationset functional without cleanup logic seems hardly incompleted :(

[dyasny]

We ended up dropping argocd altogether, and this was one of the main reasons.

[BadLiveware]

Having an issue that would get at least bearable with this capability, an app containing mongodb community operator, deleting the app causes the mongodbcommunity sub-resources to get deleted, the operator then tries to reconcile and deploy them again getting us stuck in an infinite loop of argocd trying to delete, mongodb trying to deploy. Only solution I have found is to force delete the mongodbcommunity resource.

[chary1112004]

We face the same situation where we would like to run job to delete resource like kafka topic and kafka user if tenant is deleted

[bravosierrasierra]

My solution: helm chart with Application without Argo hooks and helm hooks with my jobs. Applicationset without applicationset :)

[chary1112004]

My solution: helm chart with Application without Argo hooks and helm hooks with my jobs. Applicationset without applicationset :)

Hi @bravosierrasierra,

I also tried with helm chart with helm hook in the job but it seems job is not running.

apiVersion: batch/v1
kind: Job
metadata:
  generateName: app-post-delete-notification-
  annotations:
    "helm.sh/hook": post-delete
spec:
  template:
    spec:
      containers:
        - name: post-delete-notification
          image: curlimages/curl
          command:
            - "curl"
            - "-X"
            - "GET"
            - "http://url"
      restartPolicy: Never
  backoffLimit: 2

[bravosierrasierra]

i'm talking about additional simple helm chart for deploy from inside pipeline: helm release with argo application manifest and helm hooks. Also we can check argo app status with cli: argocd app wait

[chary1112004]

i'm talking about additional simple helm chart for deploy from inside pipeline: helm release with argo application manifest and helm hooks. Also we can check argo app status with cli: argocd app wait

thank you for the feedback. Since we would like to not deploy things from gitlab pipeline then that is reason why we move to argocd job. So I think we still need this feature from argocd. Thanks!

[deanb-everc]

Same here - we are using the Pull Request generator in order to create our dynamic ENVs and we need to create some Kafka topics upon creation and delete them upon deletion of the application, will really help us out.

[vl-kp]

We have the same issue that needs to do some cleanup after ephemeral argocd application is deleted