ApplicationSet's Work Queue Related Metrics Missing After Upgrade

[hcelaloner]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

We were following the work queue of applicationset controller using the metrics generated by the controller-runtime library. We upgraded our ArgoCD setup from v2.3.12 to v2.5.5 and we observed that metrics related to the work queue are no more generated in /metrics endpoint.

For instance, /metrics endpoint of applicationset controller was returning workqueue_depth metric in v2.3.12. When we upgraded to v2.5.5 it stopped returning workqueue_depth metric.

We suspect that init function located at https://github.com/argoproj/argo-cd/blob/master/controller/metrics/workqueue.go#L70 interfere with the init function located at https://github.com/kubernetes-sigs/controller-runtime/blob/master/pkg/metrics/workqueue.go#L91-L99. Since it is imported with the following https://github.com/argoproj/argo-cd/blob/master/cmd/main.go#L9 import statement.

We built a custom image by commenting out init function located at https://github.com/argoproj/argo-cd/blob/master/controller/metrics/workqueue.go#L70. When we deployed it, we again observed the workqueue_depth metric from /metrics endpoint of applicationset controller. However, of course, this is not a solution, I guess it just shows the cause of the problem.

To Reproduce

For instance, you can use the following steps to reproduce it,

• Install ArgoCD v2.3.3 using https://raw.githubusercontent.com/argoproj/argo-cd/v2.3.12/manifests/install.yaml
• Enable metrics endpoint of applicationset controller
• Send a request to /metrics endpoint of applicationset controller and observe that work queue related metrics are generated
• Install latest ArgoCD using https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
• Send a request to /metrics endpoint of applicationset controller and observe that work queue related metrics are not generated

Expected behavior

Being able to monitor all work queue metrics generated by controller-runtime library.

Version

argocd: v2.5.8+bbe870f.dirty
  BuildDate: 2023-01-25T18:33:35Z
  GitCommit: bbe870ff5904dd1cebeba6c5dcb7129ce7c2b5e2
  GitTreeState: dirty
  GoVersion: go1.19.5
  Compiler: gc
  Platform: darwin/amd64
argocd-server: v2.5.5+fc3eaec

cc: @ieud @mstfymrtc @erkanzileli @peacecwz

[jaideepr97]

hi @hcelaloner
The line you pointed to seems to have come from this PR #8318, which appears to have been made to solve the same issue you're describing here 🤔

[hcelaloner]

@jaideepr97 yes that is a little bit confusing but this change seems for the application controller. I guess when applicationset controller code is merged into this repo or since it is already merged into this repo, this code also affected applicationset controller. Since it is also imported somehow (probably because cmd/main.go imports), its init function also called for applicationset controller and this somehow seems to break workqueu metrics provided by controller-runtime in the applicationset controller.

[rumstead]

I also see the metric missing in 2.4.* as noted here.

[rumstead]

hi @hcelaloner The line you pointed to seems to have come from this PR #8318, which appears to have been made to solve the same issue you're describing here 🤔

application controller vs applicationset controller

[taliastocks]

This issue also appears that it needs to be reopened: #7682

The metric is missing in the Application Controller as of:

{
    "Version": "v2.9.1+58b04e5",
    "BuildDate": "2023-11-14T15:08:20Z",
    "GitCommit": "58b04e5e11d007b0518853029ff7612c24a2eb35",
    "GitTreeState": "clean",
    "GoVersion": "go1.21.3",
    "Compiler": "gc",
    "Platform": "linux/amd64",
    "KustomizeVersion": "v5.2.1 2023-10-19T20:13:51Z",
    "HelmVersion": "v3.12.1+gf32a527",
    "KubectlVersion": "v0.24.2",
    "JsonnetVersion": "v0.20.0"
}

[jujubetsz]

Can confirm that the metric is missing in 2.9.3 and v2.10.0-rc1+d919606 too