You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Reducing the feedback loop: In self-service environments managing dozens of applications, it's common for applications to be installed and updated autonomously, sometimes resulting in unexpected behaviors where certain resources or controllers begin interfering with the application controller due to updates to untracked resources or orphaned resources.
Identifying these situations and addressing them can be time-consuming. Often, changing the log level to Debug is necessary to trace the source of these refresh requests, which prolongs the time needed to resolve performance issues in production environments. Entering into sync loops can also affect other applications, causing them to become stuck in an Unknown or OutOfSync status.
As a temporary workaround, one can configure the application controller to log at the Debug level indefinitely, but this can lead to verbose logs and increased consumption of I/O resources.
Given the significance of this issue, ensuring good observability is crucial for proactive anticipation and the application of reconciliation optimizations before system degradation occurs.
Proposal
Introduce a new count metric in the application controller to track the number of refresh requests, named argocd_app_refresh_total.
Labels: namespace, server, is_managed (to identify whether the object updated triggering the refresh is an orphan or not), application name.
Update the metric documentation to include the new metric here.
Additionally, considering the relevance of orphan monitoring, it might be beneficial to relocate the orphan monitoring documentation under the operator manual instead of the user manual.
The text was updated successfully, but these errors were encountered:
Summary
Enhance observability of refresh operations.
Related to: #8192
Motivation
Reducing the feedback loop: In self-service environments managing dozens of applications, it's common for applications to be installed and updated autonomously, sometimes resulting in unexpected behaviors where certain resources or controllers begin interfering with the application controller due to updates to untracked resources or orphaned resources.
Identifying these situations and addressing them can be time-consuming. Often, changing the log level to Debug is necessary to trace the source of these refresh requests, which prolongs the time needed to resolve performance issues in production environments. Entering into sync loops can also affect other applications, causing them to become stuck in an Unknown or OutOfSync status.
As a temporary workaround, one can configure the application controller to log at the
Debug
level indefinitely, but this can lead to verbose logs and increased consumption of I/O resources.Given the significance of this issue, ensuring good observability is crucial for proactive anticipation and the application of reconciliation optimizations before system degradation occurs.
Proposal
Introduce a new count metric in the application controller to track the number of refresh requests, named
argocd_app_refresh_total
.Labels: namespace, server, is_managed (to identify whether the object updated triggering the refresh is an orphan or not), application name.
Additionally, considering the relevance of orphan monitoring, it might be beneficial to relocate the orphan monitoring documentation under the operator manual instead of the user manual.
The text was updated successfully, but these errors were encountered: