Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime error: invalid memory address or nil pointer dereference #14098

Open
2 of 3 tasks
farcaller opened this issue Jun 16, 2023 · 20 comments
Open
2 of 3 tasks

runtime error: invalid memory address or nil pointer dereference #14098

farcaller opened this issue Jun 16, 2023 · 20 comments
Labels
bug Something isn't working component:core Syncing, diffing, cluster state cache type:bug

Comments

@farcaller
Copy link
Contributor

Checklist:

  • I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
  • I've included steps to reproduce the bug.
  • I've pasted the output of argocd version.

Describe the bug

At some point the sync stops working and the sync status reports runtime error: invalid memory address or nil pointer dereference.

The following stacktrace can be observed:

Recovered from panic: runtime error: invalid memory address or nil pointer dereference
goroutine 206 [running]:
runtime/debug.Stack()
	/usr/local/go/src/runtime/debug/stack.go:24 +0x64
github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).processAppRefreshQueueItem.func1()
	/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:1321 +0x50
panic({0x29cb4e0, 0x5cdd2f0})
	/usr/local/go/src/runtime/panic.go:890 +0x260
github.com/argoproj/gitops-engine/pkg/diff.structuredMergeDiff(0x400cfeb5f0)
	/go/pkg/mod/github.com/argoproj/gitops-engine@v0.7.1-0.20230512020822-b4dd8b8c3976/pkg/diff/diff.go:153 +0x90
github.com/argoproj/gitops-engine/pkg/diff.StructuredMergeDiff(0x40040f34c0?, 0x400cfeb8f0?, 0x5?, {0x2f85024?, 0x412c2f8?})
	/go/pkg/mod/github.com/argoproj/gitops-engine@v0.7.1-0.20230512020822-b4dd8b8c3976/pkg/diff/diff.go:133 +0x4c
github.com/argoproj/gitops-engine/pkg/diff.Diff(0x40040f3360, 0x40040f3368, {0x400cfeb8f0, 0x5, 0x5})
	/go/pkg/mod/github.com/argoproj/gitops-engine@v0.7.1-0.20230512020822-b4dd8b8c3976/pkg/diff/diff.go:101 +0x180
github.com/argoproj/gitops-engine/pkg/diff.DiffArray({0x40040f3378, 0x1, 0x0?}, {0x40040f3370, 0x1?, 0x0?}, {0x400cfeb8f0, 0x5, 0x5})
	/go/pkg/mod/github.com/argoproj/gitops-engine@v0.7.1-0.20230512020822-b4dd8b8c3976/pkg/diff/diff.go:646 +0x11c
github.com/argoproj/argo-cd/v2/util/argo/diff.StateDiffs({0x40040f3358?, 0x4022d3bad0?, 0x2fa68f0?}, {0x40040f3348?, 0x4022fb12d0?, 0xa?}, {0x413b8b8, 0x401e9fcc60?})
	/go/src/github.com/argoproj/argo-cd/util/argo/diff/diff.go:266 +0x3ec
github.com/argoproj/argo-cd/v2/controller.(*appStateManager).CompareAppState(0x400087d130, 0x40011b5c00, 0x400917f8c0, {0x400cbc5060, 0x1, 0x1}, {0x4005a7b960?, 0x1, 0x1}, 0x0, ...)
	/go/src/github.com/argoproj/argo-cd/controller/state.go:556 +0x24cc
github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).processAppRefreshQueueItem(0x40007ebd40)
	/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:1428 +0xd00
github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).Run.func3()
	/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:732 +0x2c
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x3a22000000006172?)
	/go/pkg/mod/k8s.io/apimachinery@v0.24.2/pkg/util/wait/wait.go:155 +0x40
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x7b3a226e6f697461?, {0x40fd540, 0x40010b0d20}, 0x1, 0x40000bfb00)
	/go/pkg/mod/k8s.io/apimachinery@v0.24.2/pkg/util/wait/wait.go:156 +0x90
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x7463657269643a66?, 0x3b9aca00, 0x0, 0x6a?, 0x7d7b3a222e227b3a?)
	/go/pkg/mod/k8s.io/apimachinery@v0.24.2/pkg/util/wait/wait.go:133 +0x80
k8s.io/apimachinery/pkg/util/wait.Until(0x66222c7d7b3a2268?, 0x4c52556f7065723a?, 0x3a66222c7d7b3a22?)
	/go/pkg/mod/k8s.io/apimachinery@v0.24.2/pkg/util/wait/wait.go:90 +0x28
created by github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).Run
	/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:731 +0x59c

I'm pretty much reporting it only because nil pointer dereference is a bad thing(tm) and it seems like I triggered some very non-trivial race condition. Looking through the code, I don't have an immediate clue of what could have gone wrong there.

To Reproduce

I wasn't able to repo properly. It was 100% reproduceable on the broken instance but went away after the restart (and was fine since). Just before the issue happened I manually triggered a sync for a bunch of projects.

Expected behavior

Screenshots

image

Version

2.7.3
@farcaller farcaller added the bug Something isn't working label Jun 16, 2023
@crenshaw-dev
Copy link
Member

Looks like this line in gitops-engine: https://github.com/argoproj/gitops-engine/blob/b4dd8b8c3976d03f9928a393c2447830a3ef7449/pkg/diff/diff.go#L153

@farcaller
Copy link
Contributor Author

I can reproduce it pretty much weekly now, with the same stacktrace. If any debugging info would be useful, I can annotate some code around with extra logging.

@crenshaw-dev
Copy link
Member

I think either pt or p.live must be nil. Wanna add this?

if pt == nil || p.live == nil {
    log.Errorf("pt: %v, p.live: %v, gvk: %v", pt, p.live, gvk)
}

It would at least give us a start.

@Amr-Aly
Copy link

Amr-Aly commented Aug 4, 2023

Facing the same issue on v2.7.7, any workaround for this?

@farcaller
Copy link
Contributor Author

It looks like I figured where this one is coming from.

I had argo-cd manage a VolumeSnapshotClass, specifically snapshot.storage.k8s.io/v1beta1 of it. But I'm running v1.27.3 now, which actually removed the v1beta1 of it now that it's stable.

Argocd fails to diff it because live's empty at that version.

A simple fix my side was to use the proper GVK for the resource, but practically, it should fail in a more user-friendly way up the chain.

@zswanson
Copy link

Hit this in 2.8.0 - the issue seemed to stem from a PodDisruptionBudget from the Kyverno chart, which used a cluster capabilities function to decide if it was creating the new v1 or deprecated v1beta1 PDB kind. Had to remove the offending PDB from the clusters manually before I could get a sync to work.

@blakebarnett
Copy link
Contributor

We see this quite often also. Currently our workaround is to invalidate the cache for the affected cluster. We're running crossplane so this GVK issue would probably be more extreme than usual.

ArgoCD: v2.6.7
k8s: v1.24.15 (EKS)

@bravosierrasierra
Copy link

my problem was api version change: HorizontalPodAutoscaler/v2beta1 to HorizontalPodAutoscaler/v2

time="2023-10-17T12:08:21Z" level=info msg="Adding resource result, status: 'SyncFailed', phase: '', message: 'the server could not find the requested resource'" application=argocd/app-backend-feature-test kind=HorizontalPodAutoscaler name=app-hpa namespace=backend-feature-test phase=Sync syncId=00005-RAPRf

message in gui after application recreate.
autoscaling/v2beta1/HorizontalPodAutoscaler
backend-feature-test
celery-hpa
SyncFailed
the server could not find the requested resource
autoscaling/v2beta1/HorizontalPodAutoscaler
backend-feature-test
app-hpa
SyncFailed
the server could not find the requested resource

contrintuitive problem :(

@ivan-cai
Copy link

We see this quite often also. Currently our workaround is to invalidate the cache for the affected cluster. We're running crossplane so this GVK issue would probably be more extreme than usual.

ArgoCD: v2.6.7 k8s: v1.24.15 (EKS)

@blakebarnett how to invalidate the cache?

@blakebarnett
Copy link
Contributor

Settings -> Clusters -> (click cluster) -> Invalidate Cache (top-left)

@ivan-cai
Copy link

Settings -> Clusters -> (click cluster) -> Invalidate Cache (top-left)

@blakebarnett Thx!

@jgwest jgwest mentioned this issue Mar 28, 2024
Open
3 tasks
@prein
Copy link

prein commented Apr 10, 2024 •
edited
Loading

Settings -> Clusters -> (click cluster) -> Invalidate Cache (top-left)

Is there a way to do it using kubectl? In my UI when I click cluster I get an error
image
and in a popup
image

EDIT. Restarting argocd-application-controller pod is the way

@prein
Copy link

prein commented Apr 10, 2024

Observing this in 2.8.1

@MohammedNoureldin
Copy link

MohammedNoureldin commented May 22, 2024 •
edited
Loading

What to do if the the issue persists, and the solution above don't help?

runtime error: invalid memory address or nil pointer dereference

I invalidated the cache, removed the cluster. I even deployed the whole cluster (the remote cluster as I use ApplicationSet), still I see the same issue.

I addition I removed all pods of ArgoCD, but nothing helped.

@bergner
Copy link

bergner commented Jul 3, 2024

I've had exactly this problem recently in one of my ArgoCD apps in ArgoCD v2.10.11+27e49f8. The Invalidate Cache workaround mentioned in this ticket worked for me. One of my colleagues had the same issue but in one our other ArgoCD apps on his local Openshift CRC environment so it isn't specific to one of our apps. This is the stack trace I got:

Recovered from panic: runtime error: invalid memory address or nil pointer dereference
goroutine 258 [running]:
runtime/debug.Stack()
	/usr/local/go/src/runtime/debug/stack.go:24 +0x5e
github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).processAppRefreshQueueItem.func1()
	/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:1470 +0x54
panic({0x3666200?, 0x71a59c0?})
	/usr/local/go/src/runtime/panic.go:920 +0x270
github.com/argoproj/gitops-engine/pkg/diff.structuredMergeDiff(0xc003d37570)
	/go/pkg/mod/github.com/argoproj/gitops-engine@v0.7.1-0.20240416142647-fbecbb86e412/pkg/diff/diff.go:321 +0xdc
github.com/argoproj/gitops-engine/pkg/diff.StructuredMergeDiff(0xc002703a90?, 0xc003d378a8?, 0x8?, {0x3d8c1a0?, 0xc00a4ba820?})
	/go/pkg/mod/github.com/argoproj/gitops-engine@v0.7.1-0.20240416142647-fbecbb86e412/pkg/diff/diff.go:301 +0x52
github.com/argoproj/gitops-engine/pkg/diff.Diff(0xc002703900, 0xc002703910, {0xc003d378a8, 0x8, 0x8})
	/go/pkg/mod/github.com/argoproj/gitops-engine@v0.7.1-0.20240416142647-fbecbb86e412/pkg/diff/diff.go:110 +0x268
github.com/argoproj/gitops-engine/pkg/diff.DiffArray({0xc00800bf40, 0x5, 0x5080660?}, {0xc00800bf00, 0x5?, 0xc0054a5de8?}, {0xc003d378a8, 0x8, 0x8})
	/go/pkg/mod/github.com/argoproj/gitops-engine@v0.7.1-0.20240416142647-fbecbb86e412/pkg/diff/diff.go:814 +0x132
github.com/argoproj/argo-cd/v2/util/argo/diff.StateDiffs({0xc00800bec0?, 0xc005b3abc0?, 0x3ddf11c?}, {0xc00800be80?, 0xc0081da2a0?, 0xa?}, {0x50da4a0, 0xc008dc4480?})
	/go/src/github.com/argoproj/argo-cd/util/argo/diff/diff.go:318 +0x74b
github.com/argoproj/argo-cd/v2/controller.(*appStateManager).CompareAppState(0xc000edf500, 0xc001594000, 0xc007052240, {0xc002033c20, 0x2, 0x2}, {0xc005cc6460?, 0x2, 0x2}, 0x0, ...)
	/go/src/github.com/argoproj/argo-cd/controller/state.go:651 +0x3e1f
github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).processAppRefreshQueueItem(0xc000c56700)
	/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:1581 +0x1223
github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).Run.func3()
	/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:819 +0x25
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
	/go/pkg/mod/k8s.io/apimachinery@v0.26.11/pkg/util/wait/wait.go:157 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0?, {0x5082fc0, 0xc00109bc50}, 0x1, 0xc0008b0060)
	/go/pkg/mod/k8s.io/apimachinery@v0.26.11/pkg/util/wait/wait.go:158 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x3b9aca00, 0x0, 0x0?, 0x0?)
	/go/pkg/mod/k8s.io/apimachinery@v0.26.11/pkg/util/wait/wait.go:135 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(0x0?, 0x0?, 0x0?)
	/go/pkg/mod/k8s.io/apimachinery@v0.26.11/pkg/util/wait/wait.go:92 +0x1e
created by github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).Run in goroutine 95
	/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:818 +0x7ce

@alexmt alexmt added component:core Syncing, diffing, cluster state cache type:bug labels Jul 3, 2024
@crenshaw-dev
Copy link
Member

As of the latest gitops-engine commit, this error is at least handled instead of panicking:

unable to resolve parseableType for GroupVersionKind

Of course that doesn't resolve the underlying problem...

@messiahUA
Copy link

argocd v2.10.7

faced this when deploying clickhouse operator installation:

Recovered from panic: runtime error: invalid memory address or nil pointer dereference
goroutine 181 [running]:
runtime/debug.Stack()
	/usr/local/go/src/runtime/debug/stack.go:24 +0x5e
github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).processAppRefreshQueueItem.func1()
	/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:1466 +0x54
panic({0x3657b60?, 0x7187d60?})
	/usr/local/go/src/runtime/panic.go:920 +0x270
k8s.io/apimachinery/pkg/util/managedfields.(*GvkParser).Type(...)
	/go/pkg/mod/k8s.io/apimachinery@v0.26.11/pkg/util/managedfields/gvkparser.go:43
github.com/argoproj/gitops-engine/pkg/diff.removeWebhookMutation(0xc0488d75a8, 0xc02afd6120, 0x0, {0x50b2420?, 0x3d7b488?})
	/go/pkg/mod/github.com/argoproj/gitops-engine@v0.7.1-0.20240122213038-792124280fcc/pkg/diff/diff.go:210 +0xfd
github.com/argoproj/gitops-engine/pkg/diff.serverSideDiff(0xc031863fc8, 0xc02afd6120, {0xc00f4d38b0, 0x8, 0x8})
	/go/pkg/mod/github.com/argoproj/gitops-engine@v0.7.1-0.20240122213038-792124280fcc/pkg/diff/diff.go:176 +0x2bd
github.com/argoproj/gitops-engine/pkg/diff.ServerSideDiff(0xc02afd6120?, 0xc00f4d38b0?, {0xc00f4d38b0?, 0x8?, 0xc00f4be050?})
	/go/pkg/mod/github.com/argoproj/gitops-engine@v0.7.1-0.20240122213038-792124280fcc/pkg/diff/diff.go:138 +0x2a
github.com/argoproj/gitops-engine/pkg/diff.Diff(0xc02cb783e0, 0xc02cb783e8, {0xc00f4d38b0, 0x8, 0x8})
	/go/pkg/mod/github.com/argoproj/gitops-engine@v0.7.1-0.20240122213038-792124280fcc/pkg/diff/diff.go:88 +0x1d6
github.com/argoproj/gitops-engine/pkg/diff.DiffArray({0xc00e1ae680, 0xa, 0xc0562c97f8?}, {0xc00e1ae600, 0xa?, 0xc0562c9860?}, {0xc00f4d38b0, 0x8, 0x8})
	/go/pkg/mod/github.com/argoproj/gitops-engine@v0.7.1-0.20240122213038-792124280fcc/pkg/diff/diff.go:814 +0x132
github.com/argoproj/argo-cd/v2/util/argo/diff.StateDiffs({0xc029162730?, 0xc001052060?, 0x1e?}, {0xc02b7ff700?, 0xc0448910a0?, 0xa?}, {0x50c63f0, 0xc005486000?})
	/go/src/github.com/argoproj/argo-cd/util/argo/diff/diff.go:310 +0x725
github.com/argoproj/argo-cd/v2/controller.(*appStateManager).CompareAppState(0xc000a9ab60, 0xc007df7c00, 0xc000b37440, {0xc043af9150, 0x1, 0x1}, {0xc043adfb90?, 0x1, 0x1}, 0x0, ...)
	/go/src/github.com/argoproj/argo-cd/controller/state.go:649 +0x3dfa
github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).processAppRefreshQueueItem(0xc000d4b500)
	/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:1577 +0x1223
github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).Run.func3()
	/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:815 +0x25
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
	/go/pkg/mod/k8s.io/apimachinery@v0.26.11/pkg/util/wait/wait.go:157 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0?, {0x5070260, 0xc0012bff20}, 0x1, 0xc0003fa3c0)
	/go/pkg/mod/k8s.io/apimachinery@v0.26.11/pkg/util/wait/wait.go:158 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x3b9aca00, 0x0, 0x0?, 0x0?)
	/go/pkg/mod/k8s.io/apimachinery@v0.26.11/pkg/util/wait/wait.go:135 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(0x0?, 0x0?, 0x0?)
	/go/pkg/mod/k8s.io/apimachinery@v0.26.11/pkg/util/wait/wait.go:92 +0x1e
created by github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).Run in goroutine 78
	/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:814 +0x7ce

maybe related to server-side diff

restarting or clearing cache doesn't help

@seth-acuitymd
Copy link

seth-acuitymd commented Sep 28, 2024 •
edited
Loading

Was hit by this today, I've invalidated caches, basically restarted/zero-ed out all the ArgoCD components, can't seem to get things to figure itself out, if anyone had any other things that worked, would love to hear em! 🙏

Edit a day later 😄

We were running 2.11.2 and I ended up bumping us up to 2.11.5, not confident that was the fix but I did see a few chore PRs to ArgoCD in recent releases that mention the gitops-engine update! We're a small enough env that the version update is relatively easy!

@Sayrus
Copy link

Sayrus commented Oct 3, 2024

Running into this (2.11.3):

goroutine 308 [running]:
runtime/debug.Stack()
\t/usr/local/go/src/runtime/debug/stack.go:24 +0x5e
github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).processAppRefreshQueueItem.func1()
\t/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:1475 +0x54
panic({0x36a5000?, 0x71fcb20?})
\t/usr/local/go/src/runtime/panic.go:920 +0x270
k8s.io/apimachinery/pkg/util/managedfields.(*GvkParser).Type(...)
\t/go/pkg/mod/k8s.io/apimachinery@v0.26.11/pkg/util/managedfields/gvkparser.go:43
github.com/argoproj/gitops-engine/pkg/diff.removeWebhookMutation(0xc0307712d8, 0xc035df5e58, 0x0, {0x5106b00?, 0x3dc10cb?})
\t/go/pkg/mod/github.com/argoproj/gitops-engine@v0.7.1-0.20240416142647-fbecbb86e412/pkg/diff/diff.go:210 +0xfd
github.com/argoproj/gitops-engine/pkg/diff.serverSideDiff(0xc035df5e18, 0xc035df5e58, {0xc009937898, 0x8, 0x8})
\t/go/pkg/mod/github.com/argoproj/gitops-engine@v0.7.1-0.20240416142647-fbecbb86e412/pkg/diff/diff.go:176 +0x2bd
github.com/argoproj/gitops-engine/pkg/diff.ServerSideDiff(0xc035df5e58?, 0xc009937898?, {0xc009937898?, 0x8?, 0xc034667e50?})
\t/go/pkg/mod/github.com/argoproj/gitops-engine@v0.7.1-0.20240416142647-fbecbb86e412/pkg/diff/diff.go:138 +0x2a
github.com/argoproj/gitops-engine/pkg/diff.Diff(0xc035df5bd8, 0xc035df5be0, {0xc009937898, 0x8, 0x8})
\t/go/pkg/mod/github.com/argoproj/gitops-engine@v0.7.1-0.20240416142647-fbecbb86e412/pkg/diff/diff.go:88 +0x1d6
github.com/argoproj/gitops-engine/pkg/diff.DiffArray({0xc02bc31200, 0x10, 0x410805?}, {0xc02bc31180, 0x10?, 0x58?}, {0xc009937898, 0x8, 0x8})
\t/go/pkg/mod/github.com/argoproj/gitops-engine@v0.7.1-0.20240416142647-fbecbb86e412/pkg/diff/diff.go:814 +0x132
github.com/argoproj/argo-cd/v2/util/argo/diff.StateDiffs({0xc02bc30e00?, 0xc00373a5f0?, 0x48?}, {0xc02bc30d80?, 0xc0249daa10?, 0x8?}, {0x511bd40, 0xc04c0b18c0?})
\t/go/src/github.com/argoproj/argo-cd/util/argo/diff/diff.go:318 +0x74b
github.com/argoproj/argo-cd/v2/controller.(*appStateManager).CompareAppState(0xc0002c1ea0, 0xc0501d1000, 0xc04df96480, {0xc043285980, 0x5, 0x8}, {0xc01751dc00?, 0x5, 0x8}, 0x0, ...)
\t/go/src/github.com/argoproj/argo-cd/controller/state.go:684 +0x3e1f
github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).processAppRefreshQueueItem(0xc000afddc0)
\t/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:1586 +0x1223
github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).Run.func3()
\t/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:824 +0x25
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
\t/go/pkg/mod/k8s.io/apimachinery@v0.26.11/pkg/util/wait/wait.go:157 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x2273757461747322?, {0x50c4660, 0xc005581800}, 0x1, 0xc0009a8f60)
\t/go/pkg/mod/k8s.io/apimachinery@v0.26.11/pkg/util/wait/wait.go:158 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x6f692e73386b2e6e?, 0x3b9aca00, 0x0, 0x75?, 0x223a22646e696b22?)
\t/go/pkg/mod/k8s.io/apimachinery@v0.26.11/pkg/util/wait/wait.go:135 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(0x223a226567617373?, 0x7272657473756c63?, 0x636162722e656c6f?)
\t/go/pkg/mod/k8s.io/apimachinery@v0.26.11/pkg/util/wait/wait.go:92 +0x1e
created by github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).Run in goroutine 101
\t/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:823 +0x89b

Happened after enabling IncludeMutationWebhook=true as a workaround for #17568

@Sayrus
Copy link

Sayrus commented Oct 3, 2024

Was hit by this today, I've invalidated caches, basically restarted/zero-ed out all the ArgoCD components, can't seem to get things to figure itself out, if anyone had any other things that worked, would love to hear em! 🙏

Edit a day later 😄

We were running 2.11.2 and I ended up bumping us up to 2.11.5, not confident that was the fix but I did see a few chore PRs to ArgoCD in recent releases that mention the gitops-engine update! We're a small enough env that the version update is relatively easy!

Upgrading to 2.11.5 with the new gitops-engine version solved the issue. We moved onto 2.11.9 after that and the behavior didn't reappear so far.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working component:core Syncing, diffing, cluster state cache type:bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests