-
Notifications
You must be signed in to change notification settings - Fork 5.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
chore: add argocd_app_refresh_total counter metric #17178
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -28,6 +28,7 @@ import ( | |
type MetricsServer struct { | ||
*http.Server | ||
syncCounter *prometheus.CounterVec | ||
refreshCounter *prometheus.CounterVec | ||
kubectlExecCounter *prometheus.CounterVec | ||
kubectlExecPendingGauge *prometheus.GaugeVec | ||
k8sRequestCounter *prometheus.CounterVec | ||
|
@@ -90,6 +91,14 @@ var ( | |
append(descAppDefaultLabels, "dest_server", "phase"), | ||
) | ||
|
||
refreshCounter = prometheus.NewCounterVec( | ||
prometheus.CounterOpts{ | ||
Name: "argocd_app_refresh_total", | ||
Help: "Number of application refreshes.", | ||
}, | ||
append(descAppDefaultLabels, "dest_server", "compare_with"), | ||
) | ||
|
||
k8sRequestCounter = prometheus.NewCounterVec( | ||
prometheus.CounterOpts{ | ||
Name: "argocd_app_k8s_request_total", | ||
|
@@ -171,6 +180,7 @@ func NewMetricsServer(addr string, appLister applister.ApplicationLister, appFil | |
healthz.ServeHealthCheck(mux, healthCheck) | ||
|
||
registry.MustRegister(syncCounter) | ||
registry.MustRegister(refreshCounter) | ||
registry.MustRegister(k8sRequestCounter) | ||
registry.MustRegister(kubectlExecCounter) | ||
registry.MustRegister(kubectlExecPendingGauge) | ||
|
@@ -186,6 +196,7 @@ func NewMetricsServer(addr string, appLister applister.ApplicationLister, appFil | |
Handler: mux, | ||
}, | ||
syncCounter: syncCounter, | ||
refreshCounter: refreshCounter, | ||
k8sRequestCounter: k8sRequestCounter, | ||
kubectlExecCounter: kubectlExecCounter, | ||
kubectlExecPendingGauge: kubectlExecPendingGauge, | ||
|
@@ -229,6 +240,11 @@ func (m *MetricsServer) IncSync(app *argoappv1.Application, state *argoappv1.Ope | |
m.syncCounter.WithLabelValues(app.Namespace, app.Name, app.Spec.GetProject(), app.Spec.Destination.Server, string(state.Phase)).Inc() | ||
} | ||
|
||
// IncRefresh increments the refresh counter for an application | ||
func (m *MetricsServer) IncRefresh(app *argoappv1.Application, compareWithStr string) { | ||
m.refreshCounter.WithLabelValues(app.Namespace, app.Name, app.Spec.GetProject(), app.Spec.Destination.Server, compareWithStr).Inc() | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We have Argo CD instances handling thousands of apps in hundreds of namespaces. Adding all those labels in the new More on the subject: https://blog.fourninecloud.com/understanding-cardinality-in-prometheus-monitoring-96ad082b6398 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Hi! There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thank you for confirming. |
||
} | ||
|
||
func (m *MetricsServer) IncKubectlExec(command string) { | ||
m.kubectlExecCounter.WithLabelValues(m.hostname, command).Inc() | ||
} | ||
|
@@ -288,6 +304,7 @@ func (m *MetricsServer) SetExpiration(cacheExpiration time.Duration) error { | |
_, err := m.cron.AddFunc(fmt.Sprintf("@every %s", cacheExpiration), func() { | ||
log.Infof("Reset Prometheus metrics based on existing expiration '%v'", cacheExpiration) | ||
m.syncCounter.Reset() | ||
m.refreshCounter.Reset() | ||
m.kubectlExecCounter.Reset() | ||
m.kubectlExecPendingGauge.Reset() | ||
m.k8sRequestCounter.Reset() | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am wondering what additional value this will add?
The controller has metrics (#17013) on the refreshQueue that will give almost the same information as this metrics.
Additionally, the refresh/reconciliation metric is already provided.
Maybe adding the compare_with to the existing metrics would help?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Right now, It is not clear how to monitor and diagnose application controller performance issues eg related with "Refresh requested by object updated" (including orphan resources). The dashboard provided maybe is not up-to-date and has a lot of panels.
Eg: I am calculating the average in seconds application reconciliation with the next promQL:
The times returned are in order of minutes
I can see high numbers related to applicationset related to
workqueue_unfinished_work_seconds
and memory consumption in application controller metrics.I was assuming the root cause is non desirable refresh requested but I have no clear metric to visualize it and diagnose the root of the problem
How is the way to observe the application controller performance using refreshQueue metrics?
Refresh requested is not the same than reconciliation, no?
It consumes a lot of memory and require manual intervention to fix unknown status apps
We started detecting a lot of Refresh requested by object updated and thats why I suggested to improve the total refresh requested including the compare_with to identify those from orphan resources.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Based on the code path, this will be almost the same as the reconciliation. Reconciliation will be increased less often than your new metric because a lot of refreshRequest that does not require a reconciliation will be discarded.
Afaik, refreshRequest that does not require a reconciliation (reconciliation here is what happens when you click refresh in the ui, when an app resources has changed, when timeout expires, when orphan resource in the same namespace changes, etc) have a very low impact on performance.
I think this metrics is a duplication of
workqueue_adds_total{name="app_reconciliation_queue"}
, with a cardinality for each app * reconciliation_level.For the reconciliation, there is already
argocd_app_reconcile
histogram, but it does not have the app * reconciliation_level cardinality.I am not sure adding a metric with app * reconciliation_level cardinality is a good idea....
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi! After apply your fix, honestly I think this PR doesnt make sense and can be closed.