Add metrics for config file changes

[mrueg]

What this PR does / why we need it:
Provides metrics for config file updates when viper picks up a new config.

This uses code pieces from prometheus/alertmanager in https://github.com/prometheus/alertmanager/blob/main/config/coordinator.go#LL56C26-L56C26 licensed under Apache-2.0.

kube_state_metrics_config_hash 4.0061079457904e+13 kube_state_metrics_config_last_reload_success_timestamp_seconds 1.6697483049487052e+09 kube_state_metrics_config_last_reload_successful 1

How does this change affect the cardinality of KSM: (increases, decreases or does not change cardinality)
Adds three metric series to the ksm exporter, not the kubernetes exporter.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #1893

[rexagod]

These metrics will be very useful, thanks Manuel!

[logicalhan]

I'd prefer kube_state_metrics_last_config_reload_successful

[mrueg]

I tend to agree, in future we might have other config files as well though, that's why I added the infix.
Another config we might want to create metrics for once the hot reload for it is in place as well:
kube_state_metrics_customresourcestateconfig_last_reload_successful

Alternatively we could drop it from the name and use:

kube_state_metrics_last_reload_successful{type="config"}
kube_state_metrics_last_reload_successful{type="customresourcestateconfig"}

[fpetkovski]

I'd vote for using labels, seems to reduce verbosity and users can also monitor all config updates through a single metric.

[CatherineF-dev]

How about kube_state_metrics_last_config_uptime?
So that we can easily know whether there are some changes to config recently.

It equals to current_timestamp - kube_state_metrics_config_last_reload_success_timestamp_seconds

[mrueg]

I prefer to keep it as it, as a timestamp over a counter has a couple of advantages:

• few static values inside the TSDB for this metric
• Resets can also be detected if the timestamp changes
• You never have the same counter twice.
  See also: https://prometheus.io/docs/practices/instrumentation/#timestamps-not-time-since

[mrueg]

# HELP kube_state_metrics_config_hash Hash of the currently loaded configuration.
# TYPE kube_state_metrics_config_hash gauge
kube_state_metrics_config_hash{filename="config.yml",type="config"} 2.76423981094048e+14
# HELP kube_state_metrics_last_config_reload_success_timestamp_seconds Timestamp of the last successful configuration reload.
# TYPE kube_state_metrics_last_config_reload_success_timestamp_seconds gauge
kube_state_metrics_last_config_reload_success_timestamp_seconds{filename="config.yml",type="config"} 1.669900259628547e+09
# HELP kube_state_metrics_last_config_reload_successful Whether the last configuration reload attempt was successful.
# TYPE kube_state_metrics_last_config_reload_successful gauge
kube_state_metrics_last_config_reload_successful{filename="config.yml",type="config"} 1

So this is how metrics currently look like. Any thoughts on the config hash? We could make the value static (=1) and add it as a label instead.

[rexagod]

Seems fine to me to keep it as the value.

[logicalhan]

/triage accepted

[fpetkovski]

/lgtm
/hold

[rexagod]

/lgtm
/hold
/cc @dgrisonnet @logicalhan @fpetkovski

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: fpetkovski, mrueg, rexagod

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [fpetkovski,mrueg]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[mrueg]

/hold cancel

Let's proceed here