Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Reduce apiserver metrics cardinality and extraneous labels #596

Merged
merged 1 commit into from
Dec 9, 2019
Merged

Reduce apiserver metrics cardinality and extraneous labels #596

merged 1 commit into from
Dec 9, 2019

Conversation

dghubble
Copy link
Member

@dghubble dghubble commented Dec 9, 2019

  • Stop mapping node labels to targets discovered via Kubernetes nodes (e.g. etcd, kubelet, cadvisor). It is rarely useful to store node labels (e.g. kubernetes.io/os=linux) on these metrics
  • kube-apiserver's apiserver_request_duration_seconds_bucket metric has a high cardinality that includes labels for the API group, verb, scope, resource, and component for each object type, including for each CRD. This one metric has ~10k time series in a typical cluster (btw 10-40% of total)
  • Removing the apiserver request duration outright would make latency alerts a NoOp and break a Grafana apiserver panel. Instead, drop series that have a "group" label. Effectively, only request durations for core Kubernetes APIs will be kept (e.g. cardinality won't grow with each CRD added). This reduces the metric to ~2k unique series

@dghubble
Reduce apiserver metrics cardinality and extraneous labels
178afe4 
* Stop mapping node labels to targets discovered via Kubernetes
nodes (e.g. etcd, kubelet, cadvisor). It is rarely useful to
store node labels (e.g. kubernetes.io/os=linux) on these metrics
* kube-apiserver's apiserver_request_duration_seconds_bucket metric
has a high cardinality that includes labels for the API group, verb,
scope, resource, and component for each object type, including for
each CRD. This one metric has ~10k time series in a typical cluster
(btw 10-40% of total)
* Removing the apiserver request duration outright would make latency
alerts a NoOp and break a Grafana apiserver panel. Instead, drop series
that have a "group" label. Effectively, only request durations for
core Kubernetes APIs will be kept (e.g. cardinality won't grow with
each CRD added). This reduces the metric to ~2k unique series
@dghubble dghubble force-pushed the reduce-apiserver-time-series branch from 6ab676f to 178afe4 Compare December 9, 2019 06:48
@dghubble dghubble merged commit 178afe4 into master Dec 9, 2019
@dghubble dghubble deleted the reduce-apiserver-time-series branch December 9, 2019 07:03
@dghubble dghubble mentioned this pull request Sep 11, 2020
Closed
dghubble added a commit that referenced this pull request Sep 13, 2020
@dghubble
Drop Kubelet labelmap relabel for node_name
7a57b38 
* Originally, Kubelet and CAdvisor metrics used a labelmap
relabel to add Kubernetes SD node labels onto timeseries
* With #596 that
relabel was dropped since node labels aren't usually that
valuable. `__meta_kubernetes_node_name` was retained but
the field name is empty
* Favor just using Prometheus server-side `instance` in
queries that require some node identifier for aggregation
or debugging

Fix #823
@dghubble dghubble mentioned this pull request Sep 13, 2020
Merged
dghubble added a commit that referenced this pull request Sep 13, 2020
@dghubble
Drop Kubelet labelmap relabel for node_name
eb093af 
* Originally, Kubelet and CAdvisor metrics used a labelmap
relabel to add Kubernetes SD node labels onto timeseries
* With #596 that
relabel was dropped since node labels aren't usually that
valuable. `__meta_kubernetes_node_name` was retained but
the field name is empty
* Favor just using Prometheus server-side `instance` in
queries that require some node identifier for aggregation
or debugging

Fix #823
dghubble added a commit that referenced this pull request Sep 13, 2020
@dghubble
Reduce apiserver metrics cardinality of non-core APIs
694e803 
* Reduce `apiserver_request_duration_seconds_count` cardinality
by dropping series for non-core Kubernetes APIs. This is done
to match `apiserver_request_duration_seconds_count` relabeling
* These two relabels must be performed the same way to avoid
affecting new SLO calculations (upcoming)
* See kubernetes-monitoring/kubernetes-mixin#498

Related: #596
dghubble added a commit that referenced this pull request Sep 13, 2020
@dghubble
Reduce apiserver metrics cardinality of non-core APIs
8a0b822 
* Reduce `apiserver_request_duration_seconds_count` cardinality
by dropping series for non-core Kubernetes APIs. This is done
to match `apiserver_request_duration_seconds_count` relabeling
* These two relabels must be performed the same way to avoid
affecting new SLO calculations (upcoming)
* See kubernetes-monitoring/kubernetes-mixin#498

Related: #596
@dghubble dghubble mentioned this pull request Sep 13, 2020
Merged
dghubble added a commit that referenced this pull request Sep 13, 2020
@dghubble
Reduce apiserver metrics cardinality of non-core APIs
979c092 
* Reduce `apiserver_request_duration_seconds_count` cardinality
by dropping series for non-core Kubernetes APIs. This is done
to match `apiserver_request_duration_seconds_count` relabeling
* These two relabels must be performed the same way to avoid
affecting new SLO calculations (upcoming)
* See kubernetes-monitoring/kubernetes-mixin#498

Related: #596
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@dghubble