Operator is removed before ressource when application is deleted causing deletion deadlock

[nneram]

Describe the bug

I have an application that deploys an operator for Keycloak that creates and sync resources. This operator handle custom resources deletion with finalizer.

My problem is when I want to delete the application, the operator and the resources are deleted in the same time, so no one is removing finalizer from the resources and they stay. As a result, the application is looping in Deletion state. When you remove the finalizer by hand it delete the application well.

As you can see, the 3 resources have a finalize and k8s waits for it to be clean deleted by the operator which has been removed from the cluster.

I used both foreground and background deletion and also try to put annotation on the operator:

argocd.argoproj.io/sync-options: PruneLast=true

These solutions didn't make it work.

To Reproduce

Create an application that deploys an operator, custom resources definition and some resources that have deletion handles by operator. (Using finalizer)
Then request a deletion of this application; It will stay in deletion state while resources will be waiting for a clean removing from the operator.

Expected behavior

Wait for the operator to delete resources that are managed by it or delete operator at the last.

Version

argocd: v2.3.3+07ac038
  BuildDate: 2022-03-30T01:46:59Z
  GitCommit: 07ac038a8f97a93b401e824550f0505400a8c84e
  GitTreeState: clean
  GoVersion: go1.17.6
  Compiler: gc
  Platform: linux/amd64
WARN[0000] Failed to invoke grpc call. Use flag --grpc-web in grpc calls. To avoid this warning message, use flag --grpc-web.
argocd-server: v2.2.5+8f981cc
  BuildDate: 2022-02-05T01:33:25Z
  GitCommit: 8f981ccfcf942a9eb00bc466649f8499ba0455f5
  GitTreeState: clean
  GoVersion: go1.16.11
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: v0.13.1
  Kustomize Version: v4.2.0 2021-06-30T22:49:26Z
  Helm Version: v3.7.1+g1d11fcb
  Kubectl Version: v0.22.2
  Jsonnet Version: v0.17.0

[nneram]

#7575

[alexec]

Could you try to work-around this using sync waves? I.e. put the controller into an earlier wave than the resource. When an app is deleted, resources should be delete in the reverse sync wave order. That should mean the controller is delete after the resource it needs to finalizer.

[nneram]

Hi @alexec, 😃

Indeed using sync waves annotation seems to work 👍🏻

If I choose the operator to be deleted at the end, it will be applied first and wait for other resources to be deployed and never become healthy so I was also stuck in first sync-wave.

I added an annotation on custom resources so they can be deleted earlier:
Custom resources:

metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "1" 

This work fine, thank you for you help 💯

[jgato]

I had similar issue, but in my case, the ArgoCD APP contains a Namespace and some other resources. So, when deleting, the NS deletion has to wait for the other objects to be deleted. Actually, one of them would take even some minutes.
Using sync-waves seems not be working, I have objects with waves higher than the NS. I would expect the NS to be deleted the last.
We dont delete the APP, we delete all the objects inside. And the NS is deleted immediately, the same than all the other objects.
The objects inside the APP are generated by a Kustomization generator, would this be changing the way waves are managed?