ArgoCD Notifications Controller - Permission issue

[anjuls]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

I am trying to use argocd notifications to send the slack notifications. But I see a frequent error in notifications pod.

argocd-notifications-controller-646c65dbb-99grj argocd-notifications-controller E1018 18:00:21.234089       7 reflector.go:140] pkg/mod/k8s.io/client-go@v0.26.4/tools/cache/reflector.go:169: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: failed to list applications: applications.argoproj.io is forbidden: User "system:serviceaccount:argocd:argocd-notifications-controller" cannot list resource "applications" in API group "argoproj.io" at the cluster scope
^C

To Reproduce

Current version I have.

INFO[0000] ArgoCD Notifications Controller is starting   built="2023-10-18T15:17:56Z" commit=9e0e8d5e8a055ccc93b0bfbedcfa2eee91aaf5d3 namespace=argocd version=v2.9.0+9e0e8d5

Expected behavior

There should not be any error and notification should be sent on Slack.

Screenshots

Version

argocd version
argocd: v2.9.0+9e0e8d5
  BuildDate: 2023-10-18T15:17:56Z
  GitCommit: 9e0e8d5e8a055ccc93b0bfbedcfa2eee91aaf5d3
  GitTreeState: clean
  GoVersion: go1.21.3
  Compiler: gc
  Platform: linux/amd64

[anjuls]

solved by creating clusterrole and clusterrolebinding.
https://github.com/argoproj/argo-cd/pull/15702/files

[motoki317]

I have also encountered into this after upgrading from v2.8.4 to v2.8.5, since /manifests/install.yaml doesn't include the necessary ClusterRole and ClusterRoleBinding which are now required by the notifications-controller.
I think these roles should have been included in the all-in-one install yaml files in #15702.

[enys]

Confirming this also happens via https://github.com/argoproj/argo-helm/tree/main/charts/argo-cd with the upgrade to Helm release argo-cd to v5.48.0

[mayzhang2000]

This should have been fixed by https://github.com/argoproj/argo-cd/pull/16057/files.

[davidmendezph]

Updated yesterday with helm chart directly from the main branch and still same issue.

time="2023-10-30T22:20:27Z" level=info msg="ArgoCD Notifications Controller is starting" built="2023-10-27T23:36:30Z" commit=85025e1dcb683b192ea3599de0b0a196d64c94a7 namespace=argocd version=v2.8.5+85025e1

Edit: Using argo-helm version not manifest.

Fixed by argoproj/argo-helm#2315

[motoki317]

This should have been fixed by https://github.com/argoproj/argo-cd/pull/16057/files.

I think technically yes, #16057 fixes the notifications controller by listing Application resources in its installed namespace when application-namespaces is not configured (the default behavior).
But it would be convenient to include the ClusterRole in cluster-wide installation script in case user wants to use Applications in any namespace by configuring application-namespaces, after installing with install.yaml.
That's what I did in #16153. argoproj/argo-helm#2315 is a similar fix but for the helm chart.

edit: I got it, this part states that users need to apply extra manifests if they want to use Applications in any namespace, but I think that's easy to miss.

[crenshaw-dev]

@davidmendezph the fix has been merged but still needs to be released. It'll be in 2.8.6. I'm working through some CI issues, but should get that cut today.

@motoki317 let's stick with adding the cluster role as an opt-in for now. If you'd like to update the docs to be more clear, that would be appreciated!

[andrii-korotkov-verkada]

ArgoCD versions 2.10 and below have reached EOL. Can you upgrade and let us know if the issue is still present, please?