Improve refreshes observability

[jsolana]

Summary

Enhance observability of refresh operations.

Related to: #8192

Motivation

Reducing the feedback loop: In self-service environments managing dozens of applications, it's common for applications to be installed and updated autonomously, sometimes resulting in unexpected behaviors where certain resources or controllers begin interfering with the application controller due to updates to untracked resources or orphaned resources.

Identifying these situations and addressing them can be time-consuming. Often, changing the log level to Debug is necessary to trace the source of these refresh requests, which prolongs the time needed to resolve performance issues in production environments. Entering into sync loops can also affect other applications, causing them to become stuck in an Unknown or OutOfSync status.

As a temporary workaround, one can configure the application controller to log at the Debug level indefinitely, but this can lead to verbose logs and increased consumption of I/O resources.

Given the significance of this issue, ensuring good observability is crucial for proactive anticipation and the application of reconciliation optimizations before system degradation occurs.

Proposal

Introduce a new count metric in the application controller to track the number of refresh requests, named argocd_app_refresh_total.

Labels: namespace, server, is_managed (to identify whether the object updated triggering the refresh is an orphan or not), application name.

• Update the metric documentation to include the new metric here.
• It's suggested to add a warning annotation about the performance impact when using orphan monitoring and provide a link to the reconciliation documentation.
• grafana dashboards

Additionally, considering the relevance of orphan monitoring, it might be beneficial to relocate the orphan monitoring documentation under the operator manual instead of the user manual.

[jsolana]

cc: @crenshaw-dev

[jsolana]

Instead of is_managed as label, maybe we can use compare_with. Refresh requested by orphan resources updated would have ComparisonWithNothing values.

Wdyt?