Last Sync Result: Sync failed with multi level root-app-of-apps & sync-waves

[cconnert]

Checklist:

• [x ] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [~] I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

We have a self-managed, multi (2) level app-of-app setup which utilizes sync-waves for initial bootstrapping. More ore less one can thinks of:

Root-App-Of-Apps->Intermediated-App-Of-Apps->Leaf-Apps

As pointed out the Root-App-Of-Apps is a self-managed ArgoCD. In addition to the Intermediated-App-Of-Apps it also manages several base apps.

So to be able to work with sync-waves we added a custom health check, as the base code got removed 3781.

Now what happens is when a Leaf-Apps fails to deploy, it gets degraded. The health state gets properly propagated upwards the App-Of-App chain and the corresponding Intermediated-App-Of-Apps and the Root-App-Of-Apps get degraded too.

The interesting part is actually the operationState (Last Sync Result). The failing Leaf-Apps is synced ok (aka. operationState.phase: Succeeded). The corresponding Intermediated-App-Of-App also gets synced ok. BUT the Root-App-Of-Apps fails with: "one or more synchronization tasks completed unsuccessfully" (aka. operationState.phase: Failed).

Once I remove the degraded Intermediated-App-Of-App the operationState of the Root-App-Of-Apps succeeds. The same thing happens once I remove the lua health script.

What I noticed is that the status that the Root-App-Of-Apps operationState.syncResult[name==Root-App-Of-Apps].hookPhase: Failed.

To Reproduce

Basically the description should provided valid steps to reproduce the issue. Otherwise please get back to me so I can enhance the description or provide more input.

Expected behavior

I would expect that the Root-App-Of-Apps operationState succeeds, as it got synced ok.

Screenshots

Root-App-Of-Apps:

Intermediated-App-Of-App

Leaf-App

Version

{
    "Version": "v2.4.6+a48bca0",
    "BuildDate": "2022-07-12T22:31:17Z",
    "GitCommit": "a48bca03c79b6d63be0c34d6094831bc6916b3bc",
    "GitTreeState": "clean",
    "GoVersion": "go1.18.3",
    "Compiler": "gc",
    "Platform": "linux/amd64",
    "KustomizeVersion": "v4.4.1 2021-11-11T23:36:27Z",
    "HelmVersion": "v3.8.1+g5cb9af4",
    "KubectlVersion": "v0.23.1",
    "JsonnetVersion": "v0.18.0"
}

[adberger]

@cconnert We have a similar use-case.
How did you resolve your issue?
Are you able to share your custom health check lua script?

[cconnert]

Hi @adberger,

I had to check as I raised this issue quiet some time ago. Back then I closed it i favor of #10550 proposal