chore: add argocd_app_refresh_total counter metric #17178
Conversation
Signed-off-by: Javier Solana Huertas <javier.solana@cabify.com>
Signed-off-by: Javier Solana Huertas <javier.solana@cabify.com>
| obj, exists, err := ctrl.appInformer.GetIndexer().GetByKey(key) | ||
| app, ok := obj.(*appv1.Application) | ||
| if exists && err == nil && ok { | ||
| ctrl.metricsServer.IncRefresh(app, compareWith.String()) |
There was a problem hiding this comment.
I am wondering what additional value this will add?
The controller has metrics (#17013) on the refreshQueue that will give almost the same information as this metrics.
Additionally, the refresh/reconciliation metric is already provided.
Maybe adding the compare_with to the existing metrics would help?
There was a problem hiding this comment.
I am wondering what additional value this will add?
The controller has metrics (#17013) on the refreshQueue that will give almost the same information as this metrics.
Right now, It is not clear how to monitor and diagnose application controller performance issues eg related with "Refresh requested by object updated" (including orphan resources). The dashboard provided maybe is not up-to-date and has a lot of panels.
Eg: I am calculating the average in seconds application reconciliation with the next promQL:
sum(rate(argocd_app_reconcile_sum{app_kubernetes_io_instance=~"$namespace",kubernetes_namespace=~"$namespace", app_kubernetes_io_name=~"$component"}[5m])) by (app_kubernetes_io_instance, app_kubernetes_io_name) / sum(rate(argocd_app_reconcile_count{app_kubernetes_io_instance=~"$namespace",kubernetes_namespace=~"$namespace", app_kubernetes_io_name=~"$component"}[5m]))by (app_kubernetes_io_instance, app_kubernetes_io_name) > 0
The times returned are in order of minutes
I can see high numbers related to applicationset related to workqueue_unfinished_work_seconds and memory consumption in application controller metrics.
I was assuming the root cause is non desirable refresh requested but I have no clear metric to visualize it and diagnose the root of the problem
How is the way to observe the application controller performance using refreshQueue metrics?
Additionally, the refresh/reconciliation metric is already provided.
Refresh requested is not the same than reconciliation, no?
It consumes a lot of memory and require manual intervention to fix unknown status apps
We started detecting a lot of Refresh requested by object updated and thats why I suggested to improve the total refresh requested including the compare_with to identify those from orphan resources.
There was a problem hiding this comment.
Based on the code path, this will be almost the same as the reconciliation. Reconciliation will be increased less often than your new metric because a lot of refreshRequest that does not require a reconciliation will be discarded.
Afaik, refreshRequest that does not require a reconciliation (reconciliation here is what happens when you click refresh in the ui, when an app resources has changed, when timeout expires, when orphan resource in the same namespace changes, etc) have a very low impact on performance.
I think this metrics is a duplication of
workqueue_adds_total{name="app_reconciliation_queue"}, with a cardinality for each app * reconciliation_level.
For the reconciliation, there is already argocd_app_reconcile histogram, but it does not have the app * reconciliation_level cardinality.
argocd_app_reconcile_bucket{dest_server="https://kubernetes.default.svc",namespace="argocd",le="0.25"} 61
argocd_app_reconcile_bucket{dest_server="https://kubernetes.default.svc",namespace="argocd",le="0.5"} 124
argocd_app_reconcile_bucket{dest_server="https://kubernetes.default.svc",namespace="argocd",le="1"} 124
argocd_app_reconcile_bucket{dest_server="https://kubernetes.default.svc",namespace="argocd",le="2"} 246
argocd_app_reconcile_bucket{dest_server="https://kubernetes.default.svc",namespace="argocd",le="4"} 252
argocd_app_reconcile_bucket{dest_server="https://kubernetes.default.svc",namespace="argocd",le="8"} 315
argocd_app_reconcile_bucket{dest_server="https://kubernetes.default.svc",namespace="argocd",le="16"} 315
argocd_app_reconcile_bucket{dest_server="https://kubernetes.default.svc",namespace="argocd",le="+Inf"} 315
argocd_app_reconcile_sum{dest_server="https://kubernetes.default.svc",namespace="argocd"} 537.5461727870005
argocd_app_reconcile_count{dest_server="https://kubernetes.default.svc",namespace="argocd"} 315
I am not sure adding a metric with app * reconciliation_level cardinality is a good idea....
There was a problem hiding this comment.
Hi! After apply your fix, honestly I think this PR doesnt make sense and can be closed.
jsolana
changed the title
There was a problem hiding this comment.
This PR needs to be redesigned to avoid high cardinality in the new metric. @agaudreault provided a suggestion to reuse existing metrics to achieve similar results. Please consider.
| @@ -229,6 +240,11 @@ func (m *MetricsServer) IncSync(app *argoappv1.Application, state *argoappv1.Ope | |||
| m.syncCounter.WithLabelValues(app.Namespace, app.Name, app.Spec.GetProject(), app.Spec.Destination.Server, string(state.Phase)).Inc() | |||
| } | |||
|
|
|||
| // IncRefresh increments the refresh counter for an application | |||
| func (m *MetricsServer) IncRefresh(app *argoappv1.Application, compareWithStr string) { | |||
| m.refreshCounter.WithLabelValues(app.Namespace, app.Name, app.Spec.GetProject(), app.Spec.Destination.Server, compareWithStr).Inc() | |||
There was a problem hiding this comment.
We have Argo CD instances handling thousands of apps in hundreds of namespaces. Adding all those labels in the new refreshCounter metric will cause a huge boost in cardinality which could lead to serious issues in Prometheus.
More on the subject: https://blog.fourninecloud.com/understanding-cardinality-in-prometheus-monitoring-96ad082b6398
There was a problem hiding this comment.
Hi!
After applying this fix, I think this PR can be closed.
|
Closing due to #17178 (comment) |
Closes: #17122
Introduce a new count metric in the application controller to track the number of refresh requests, named
argocd_app_refresh_total.Labels:
Checklist: