-
Notifications
You must be signed in to change notification settings - Fork 5.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Application controller should export workqueue depth metrics #7682
Comments
Hello @Ailsa-Wu , The controller workqueue depth metric is already available. You might use the following query to get it: However, the API response slowness is not related to the controller. It is possible that you are affected by this issue #4296 . The issue has a fix already and will be available in v2.2-rc1 that will be published later today. I would appreciate it if you give it a try and let us know if performance improved in your environment. |
Hi @alexmt , thanks for your response. Actually, I have tried to get the metric by accessing the endpoint kubectl port-forward svc/argocd-application-controller-metrics -n argocd 8082:8082 Our ArgoCD version is v2.0.5, and we still need about 20s to access |
@jessesuen @alexmt I also don't think I see workqueue_depth on version 2.1.5 anymore. |
@alexmt @jessesuen I don't think this works as well. We only registers workqueue metrics to the registry inside |
I just created #8318 to fix this. |
This appears to be an issue again as of (at least) the 2.9 release. |
Can confirm that after the upgrade to |
Summary
Application controller should export workqueue depth metrics
Motivation
Queuing delay and queue length are important information to debug control plane performance in large scale applications(3k+).
We use one ArgoCD instance to manage about 3K applications. And the number is still increasing, followed by the performance downgrade, for example it would take more time to trigger the auto sync or it can't refresh the resources status and keep with live status.
We have tried multi method to enhance performance.
--status-processors
to 200 and--operation-processors
to 100 refer to the docBut unfortunately we still need about 20s to list application and update project role, e.g.
And after we switch to HA mode, the reconcile performance looks worse.
![image](https://user-images.githubusercontent.com/66725688/141255515-8044ff3a-6169-46bf-a9ec-87911510047a.png)
Not sure if I am understanding this right, any response would be appreciated🙏.
Proposal
Application controller export the workqueue depth metrics.
The text was updated successfully, but these errors were encountered: