Application controller should export workqueue depth metrics

[Ailsa-Wu]

Summary

Application controller should export workqueue depth metrics

Motivation

Queuing delay and queue length are important information to debug control plane performance in large scale applications(3k+).

We use one ArgoCD instance to manage about 3K applications. And the number is still increasing, followed by the performance downgrade, for example it would take more time to trigger the auto sync or it can't refresh the resources status and keep with live status.

We have tried multi method to enhance performance.

1. scale up the replicas of ArgoCD server and repo-server with the HA guide
2. increase the --status-processors to 200 and --operation-processors to 100 refer to the doc
3. configure server replicas with the HPA.

But unfortunately we still need about 20s to list application and update project role, e.g.

And after we switch to HA mode, the reconcile performance looks worse.

Not sure if I am understanding this right, any response would be appreciated🙏.

Proposal

Application controller export the workqueue depth metrics.

[alexmt]

Hello @Ailsa-Wu ,

The controller workqueue depth metric is already available. You might use the following query to get it: sum(workqueue_depth{name=~"app_.*"}) .

However, the API response slowness is not related to the controller. It is possible that you are affected by this issue #4296 . The issue has a fix already and will be available in v2.2-rc1 that will be published later today. I would appreciate it if you give it a try and let us know if performance improved in your environment.

[Ailsa-Wu]

Hi @alexmt , thanks for your response. Actually, I have tried to get the metric by accessing the endpoint http://localhost:8082/metrics . But I could not find this metric workqueue_depth. If there were some issues?🧐

kubectl port-forward svc/argocd-application-controller-metrics -n argocd 8082:8082

Our ArgoCD version is v2.0.5, and we still need about 20s to access /api/v1/applications, even if we use local user admin.

[sidewinder12s]

@jessesuen @alexmt I also don't think I see workqueue_depth on version 2.1.5 anymore.

[yeya24]

@alexmt @jessesuen I don't think this works as well. We only registers workqueue metrics to the registry inside component-base library and we don't register it when serving /metrics endpoint. We need to register workqueue metrics to controller's own registry

[yeya24]

I just created #8318 to fix this.

[taliastocks]

This appears to be an issue again as of (at least) the 2.9 release.

[jujubetsz]

Can confirm that after the upgrade to v2.9 the workqueue_depth is no longer available.

[joshuabezaleel]

+1. the workqueue_depth also disappeard on our side.