controller histogram buckets configuration

[terinjokes]

Metrics are created for the controller's workqueue, using Prometheus's default buckets. Unfortunately, the default buckets are poorly chosen for event processing.

The default buckets are tailored to broadly measure the response time (in seconds) of a network service.

This can easily result in metrics that are extremely coarse. In a controller I was working on today, every single reconcile was faster than the smallest bucket:

controller_runtime_reconcile_time_seconds_bucket{controller="application",le="0.005"} 31
controller_runtime_reconcile_time_seconds_bucket{controller="application",le="0.01"} 31
controller_runtime_reconcile_time_seconds_bucket{controller="application",le="0.025"} 31
controller_runtime_reconcile_time_seconds_bucket{controller="application",le="0.05"} 31
controller_runtime_reconcile_time_seconds_bucket{controller="application",le="0.1"} 31
controller_runtime_reconcile_time_seconds_bucket{controller="application",le="0.25"} 31
controller_runtime_reconcile_time_seconds_bucket{controller="application",le="0.5"} 31
controller_runtime_reconcile_time_seconds_bucket{controller="application",le="1"} 31
controller_runtime_reconcile_time_seconds_bucket{controller="application",le="2.5"} 31
controller_runtime_reconcile_time_seconds_bucket{controller="application",le="5"} 31
controller_runtime_reconcile_time_seconds_bucket{controller="application",le="10"} 31
controller_runtime_reconcile_time_seconds_bucket{controller="application",le="+Inf"} 31

It is possible to change the default buckets by modifying DefBuckets in init, as my init will be called after the DefBuckets variable has been initialized, but before the controllermetrics package init. But this is a very heavy handed brush, changing the defaults of all histograms.

I propose two paths forward:

1. Allow the user to pass their own metrics to the controller, with the current collectors used if the none are provided.
2. Alternatively, move the current package outside of the internal package, This will allow users to call Unregister and then assign a replacement.

My preference would be the first option, as the change to the system is would be more easily understood.

[mengqiy]

The 1st option makes sense to me.

[DirectXMan12]

I think we can probably choose some better buckets, too. This smae discussion is being had right now in the main k/k repo ;-).

[DirectXMan12]

It seems to me like exposing the exact mechanics of how we observe metrics is exposing an implementation detail as public API (at least, that's my gut feeling), particularly since the only way to actually change the bucket definition would be to expose the individual metrics at a API level. That would mean that we'd be tied to a particular set of metrics with a particular interface for measuring them (and it would mean that only a major revision could change/remove metrics, but it's unclear if that's a bad thing). If we ever wanted to switch to, say, contextual metrics measurement, we couldn't without a major version rev, even if we didn't actually change the metrics exposed.

My instinct here is to fix our buckets, and wait for more evidence that a "one-size-fits-all" solution won't work here, especially since we have little evidence that our currently buckets are correct for many cases. The Kubernetes SLO for request latency against the API server is 1s, IIRC, and that's on the high end. This bucket selection seems to be poor in that case. k/k is discussing the same thing right now in the switch from summaries to buckets.

[mengqiy]

Ref: kubernetes/kubernetes#63750 and kubernetes/kubernetes#67476 are the upstream issue and PR.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[DirectXMan12]

/remove-lifecycle stale

looks like kube has fixed this upstream, so we can follow suit

/good-first-issue

[k8s-ci-robot]

@DirectXMan12:
This request has been marked as suitable for new contributors.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-good-first-issue command.

In response to this:

/remove-lifecycle stale

looks like kube has fixed this upstream, so we can follow suit

/good-first-issue

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[vincepri]

/priority awaiting-more-evidence

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[irajdeep]

Hi @DirectXMan12
We also have a use-case where we need more time duration buckets (> 10secs) for controller_runtime_reconcile_time_seconds_bucket histogram.
Should we add the buckets to controller-runtime metrics in accordance with the upstream changes done in this PR: kubernetes/kubernetes#73638. ?