Add support for multi cluster alert rule queries

[jaygridley]

What is missing?

A Bunch of Prometheus alert rules do not support multi-cluster setup. For example KubeAPITerminatedRequests should group metric by cluster, ie. sum by (cluster) (rate(apiserver_request_terminations_total{job="apiserver"}[10m])) / ( sum by (cluster) (rate(apiserver_request_total{job="apiserver"}[10m])) + sum by (cluster) (rate(apiserver_request_terminations_total{job="apiserver"}[10m])) ) > 0.20.

List of alerts lacking multi-cluster support:

• KubeCPUQuotaOvercommit
• KubeMemoryQuotaOvercommit
• KubeStateMetricsListErrors
• KubeStateMetricsShardingMismatch
• KubeStateMetricsShardsMissing
• KubeStateMetricsWatchErrors
• PrometheusOperatorListErrors
• PrometheusOperatorNotReady
• PrometheusOperatorReconcileErrors
• PrometheusOperatorWatchErrors
• TargetDown

Why do we need it?

Environment

• kube-prometheus version:

  8.4.1

Anything else we need to know?:

Other alerts already do support this, ie. https://github.com/prometheus-operator/kube-prometheus/blob/main/manifests/kubernetesControlPlane-prometheusRule.yaml#L551

[jkroepke]

Having the same issue. I guess we have to contribute by own?

[jkroepke]

For

KubeStateMetricsListErrors
KubeStateMetricsShardingMismatch
KubeStateMetricsShardsMissing
KubeStateMetricsWatchErrors

See: kubernetes/kube-state-metrics#2058 then it can be implemented as well here. too