Adjust and refactor allowing labels to work for Kubernetes labels metrics #1301

lilic · 2020-11-12T12:28:46Z

During performance testing of the v2.0 release, I noticed the new filtering mechanism for labels always goes through all resources even in cases when flag is disabled, this introduces a performance regression. The use cases we have currently to filter labels are for Kubernetes resource labels metrics e.g. kube_pod_labels metrics, and in the future for annotation metrics. We heavily control other labels in metrics and make sure it does always contain a set list of labels and that set list is bound cardinality and does not contain any sensitive information. There are other ways of not exposing metrics, for example with the metric allow and deny list flags in kube-state-metrics as well as with resource allow list flag, as well as with Prometheus own mechanisms of metric relabeling.

In short, this PR introduces a flag called --labels-metric-allow-list which allows Kubernetes labels to be added to the specific resource labels metrics. By default all labels metrics will only have name of resource and namespace in them and all other labels are denied. This flag does not affect any other metrics and thus does not degrade performance. This same pattern and functions can be used for annotation metrics in the future.
Example:
--labels-metric-allow-list=pods=["app","app.kubernetes.io/component"]
will produce a metric the following:

kube_pod_labels{pod="my-pod-name", label_app="myfancy-label", label_app_kubernetes_io_component="myotherfancylabel"namespace="pod-namespace"}

Note PR is WIP, I run this idea already with @brancz and also proposed this in the various issues and PRs. I am looking for actual implementation feedback now.

lilic · 2020-11-12T12:29:58Z

docs/cli-arguments.md

+  -h, --help                              Print Help text
+      --host string                       Host to expose metrics on. (default "0.0.0.0")
+      --kubeconfig string                 Absolute path to the kubeconfig file
+      --labels-metric-allow-list string   By default all labels metric will only have resource name and namespace labels, to allow more labels pass in a list of resource names in their plural form and Kubernetes labels you would like to allow, =namespaces=[k8s-label1,k8s-labeln...],pods[app]...


Its hard to explain the labels-metric-allow-list flag due to the nature of the metric labels and k8s labels, if anyone has a more clear naming, open to suggestions! I have a similar idea for annotations where it would be --annotations-metric-allow-list and you pass the resource and annotations to allow.

I would suggest to change the order and explain first what the flag does, then its default. Maybe something like: Allows to pass a list of additional Kubernetes label keys that will be used in the resource' labels metric. By default the metric contains only name and namespace labels. To include additional labels provide a list of resource names in their plural form and Kubernetes label keys you would like to allow for them

I agree with @mrueg, we should first explain what the flag does, then the default. I like what @mrueg suggested, but I think there should additionally be an example, similar to what is currently at the end of what @lilic suggested in the PR.

lilic · 2020-11-12T12:31:19Z

internal/store/endpoint_test.go

@@ -93,8 +168,11 @@ func TestEndpointStore(t *testing.T) {
 		},
 	}
 	for i, c := range cases {
-		c.Func = generator.ComposeMetricGenFuncs(endpointMetricFamilies)
-		c.Headers = generator.ExtractMetricFamilyHeaders(endpointMetricFamilies)
+		allowLabels := []string{


Diff is strange here, so best expand, essentially this adds a Labels test, with labels we allow for the kube_endpoint_labels metric.

internal/store/mutatingwebhookconfiguration.go

lilic · 2020-11-12T12:39:00Z

Note that I will adjust the unit tests and some other nits I noticed myself after initial thoughts for or against the approach. :)

dgrisonnet

I might have missed it, but are you planning to set the sane defaults in another PR?

dgrisonnet · 2020-11-12T13:04:56Z

docs/cli-arguments.md

+  -h, --help                              Print Help text
+      --host string                       Host to expose metrics on. (default "0.0.0.0")
+      --kubeconfig string                 Absolute path to the kubeconfig file
+      --labels-metric-allow-list string   By default all labels metric will only have resource name and namespace labels, to allow more labels pass in a list of resource names in their plural form and Kubernetes labels you would like to allow, =namespaces=[k8s-label1,k8s-labeln...],pods[app]...


Suggested change

--labels-metric-allow-list string By default all labels metric will only have resource name and namespace labels, to allow more labels pass in a list of resource names in their plural form and Kubernetes labels you would like to allow, =namespaces=[k8s-label1,k8s-labeln...],pods[app]...

--labels-metric-allow-list string By default all labels metric will only have resource name and namespace labels, to allow more labels pass in a list of resource names in their plural form and Kubernetes labels you would like to allow, =namespaces=[k8s-label1,k8s-labeln...],pods=[app]...

internal/store/mutatingwebhookconfiguration.go

internal/store/persistentvolumeclaim.go

internal/store/secret.go

internal/store/service.go

internal/store/statefulset.go

dgrisonnet · 2020-11-12T13:35:07Z

pkg/options/options.go

@@ -97,7 +97,7 @@ func (o *Options) AddFlags() {
 	o.flags.StringVar(&o.Namespace, "pod-namespace", "", "Name of the namespace of the pod specified by --pod. "+autoshardingNotice)
 	o.flags.BoolVarP(&o.Version, "version", "", false, "kube-state-metrics build version information")
 	o.flags.BoolVar(&o.EnableGZIPEncoding, "enable-gzip-encoding", false, "Gzip responses when requested by clients via 'Accept-Encoding: gzip' header.")
-	o.flags.Var(&o.LabelsAllowList, "labels-allow-list", "list of metric names and labels you would like to allow, metric_name=[label1,labeln...],metric_name[]...")
+	o.flags.Var(&o.LabelsAllowList, "labels-metric-allow-list", "By default all labels metric will only have resource name and namespace labels, to allow more labels pass in a list of resource names in their plural form and Kubernetes labels you would like to allow, =namespaces=[k8s-label1,k8s-labeln...],pods[app]...")


Suggested change

o.flags.Var(&o.LabelsAllowList, "labels-metric-allow-list", "By default all labels metric will only have resource name and namespace labels, to allow more labels pass in a list of resource names in their plural form and Kubernetes labels you would like to allow, =namespaces=[k8s-label1,k8s-labeln...],pods[app]...")

o.flags.Var(&o.LabelsAllowList, "labels-metric-allow-list", "By default all labels metric will only have resource name and namespace labels, to allow more labels pass in a list of resource names in their plural form and Kubernetes labels you would like to allow, =namespaces=[k8s-label1,k8s-labeln...],pods=[app]...")

lilic · 2020-11-12T13:48:37Z

I might have missed it, but are you planning to set the sane defaults in another PR?

Thanks for the review, what do you mean by sane defaults, defaults for?

brancz · 2020-11-12T15:17:55Z

Strategy wise this looks like the perfect fit for the problem to me. I think we should go with this.

lilic · 2020-11-13T09:31:15Z

@brancz @dgrisonnet @jw-s please have a look, should be ready for review the details are in the description of the PR, thanks!

lilic · 2020-11-13T09:51:33Z

Since this includes a bug fix for the labels metrics name and namespace, I changed base to release-2.0.

internal/store/builder.go

jw-s · 2020-11-17T08:23:17Z

Great job! Looks good :)

lilic · 2020-11-18T12:51:00Z

.golangci.yml

@@ -18,6 +18,9 @@ linters:
 linters-settings:
  goimports:
    local-prefixes: k8s.io/kube-state-metrics
+  gocyclo:


I will open an issue for this after the PR is merged, to refactor the podMetricFamilies function, note this function did not change, it was just a var before.

brancz

I think this is almost ready to merge. Just a few small things.

Side note: I love the parser for the label flag, and the ability to specify labels as they are in form of the Kubernetes label, I think this is much more ergonomic.

brancz · 2020-11-18T12:59:46Z

docs/cli-arguments.md

+  -h, --help                              Print Help text
+      --host string                       Host to expose metrics on. (default "0.0.0.0")
+      --kubeconfig string                 Absolute path to the kubeconfig file
+      --labels-metric-allow-list string   By default all labels metric will only have resource name and namespace labels, to allow more labels pass in a list of resource names in their plural form and Kubernetes labels you would like to allow, =namespaces=[k8s-label1,k8s-labeln...],pods[app]...


I agree with @mrueg, we should first explain what the flag does, then the default. I like what @mrueg suggested, but I think there should additionally be an example, similar to what is currently at the end of what @lilic suggested in the PR.

internal/store/builder.go

pkg/options/types.go

this uncovered a cyclomatic complexity 127 for podMetricFamilies.

lilic · 2020-11-18T15:01:53Z

docs/cli-arguments.md

+  -h, --help                              Print Help text
+      --host string                       Host to expose metrics on. (default "0.0.0.0")
+      --kubeconfig string                 Absolute path to the kubeconfig file
+      --labels-metric-allow-list string   Allows to pass a list of additional Kubernetes label keys that will be used in the resource' labels metric. By default the metric contains only name and namespace labels. To include additional labels provide a list of resource names in their plural form and Kubernetes label keys you would like to allow for them (Example: '=namespaces=["k8s-label-1","k8s-label-n",...],pods=["app"],...)'


Adjusted the help text, are we happy with flag name?

--labels-metric-allow-list

My thinking for annotation it would be:

--annotations-metric-label-allow-list

So might need a change here as well? 🤔 It gets confusing as its label metrics for label allow list. 🤯

Flag name(s) sounds good to me.

I wonder if at some point, we want to consider reading arguments from a config file for all settings. I don't think we'll hit max cmdline input length, but let's assume someone is very verbose ;)
It might also be easier to define it there in a structured way?

labels-metric-allow-list: - namespaces: - k8s-label1 - k8s-label2 - pods - app

Good point, I think if that happens we can add a flag called --labels-metric-allow-list-configfile that inputs a file, or one for both labels and annotations. Right now the parser makes certain assumptions (as it did before), I would say as we did not have a reading from a file until now I would not add it as part of this PR, but if there is something users face in the future, definitely! (For now its out of scope of this PR, as it just focuses on fixing bug and defining direction)

I agree, I was thinking about a generic --config flag to do configuration for everything kube-state-metrics related.

I see, okay if that is what you need, and you have use case, please feel free to open a feature request w and we can discuss there :)

lilic · 2020-11-18T15:10:30Z

Addressed all comments, let me know if I missed some, ready for another review! Thanks all!

brancz · 2020-11-19T12:27:58Z

Amazing work! 💯

/lgtm
/approve

k8s-ci-robot · 2020-11-19T12:28:17Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: brancz, lilic

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [brancz,lilic]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot requested a review from brancz November 12, 2020 12:29

lilic force-pushed the optimize-filter-labels branch 3 times, most recently from bc02939 to 9bd4b63 Compare November 12, 2020 12:34

lilic commented Nov 12, 2020

View reviewed changes

dgrisonnet reviewed Nov 12, 2020

View reviewed changes

lilic force-pushed the optimize-filter-labels branch from 9bd4b63 to 20016d7 Compare November 13, 2020 09:26

k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Nov 13, 2020

lilic changed the title ~~WIP: Adjust and refactor allowing labels to work for kubernetes labels metrics~~ Adjust and refactor allowing labels to work for kubernetes labels metrics Nov 13, 2020

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 13, 2020

lilic changed the base branch from master to release-2.0 November 13, 2020 09:50

dgrisonnet reviewed Nov 16, 2020

View reviewed changes

internal/store/builder.go Outdated Show resolved Hide resolved

internal/store/builder.go Outdated Show resolved Hide resolved

lilic mentioned this pull request Nov 18, 2020

annotation metrics, what is the future direction? #941

Closed

lilic force-pushed the optimize-filter-labels branch 4 times, most recently from 71cde75 to bdc3fdd Compare November 18, 2020 12:49

lilic commented Nov 18, 2020

View reviewed changes

brancz reviewed Nov 18, 2020

View reviewed changes

lilic added 5 commits November 18, 2020 15:59

internal/store: Rework label allow list for label metrics

0f4c512

go.mod,go.sum

37e834e

vendor

cdad26e

pkg/options/types.go: Adjust Set to parse new flag params

3b23d01

.golangci.yml: Due to refactor of the resources from a var to a function

a551501

this uncovered a cyclomatic complexity 127 for podMetricFamilies.

lilic force-pushed the optimize-filter-labels branch from bdc3fdd to a551501 Compare November 18, 2020 14:59

lilic commented Nov 18, 2020

View reviewed changes

k8s-ci-robot assigned brancz Nov 19, 2020

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 19, 2020

k8s-ci-robot merged commit 6141abc into kubernetes:release-2.0 Nov 19, 2020

lilic deleted the optimize-filter-labels branch November 19, 2020 12:29

lilic changed the title ~~Adjust and refactor allowing labels to work for kubernetes labels metrics~~ Adjust and refactor allowing labels to work for Kubernetes labels metrics Nov 19, 2020

lilic mentioned this pull request Nov 20, 2020

Add feature to disable labels-allow-list #1309

Closed

Mikulas mentioned this pull request Mar 3, 2021

Add wildcard option to metric-labels-allowlist #1403

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adjust and refactor allowing labels to work for Kubernetes labels metrics #1301

Adjust and refactor allowing labels to work for Kubernetes labels metrics #1301

lilic commented Nov 12, 2020

lilic Nov 12, 2020

mrueg Nov 13, 2020

brancz Nov 18, 2020

lilic Nov 12, 2020

lilic commented Nov 12, 2020

dgrisonnet left a comment

dgrisonnet Nov 12, 2020

dgrisonnet Nov 12, 2020

lilic commented Nov 12, 2020

brancz commented Nov 12, 2020

lilic commented Nov 13, 2020

lilic commented Nov 13, 2020

jw-s commented Nov 17, 2020

lilic Nov 18, 2020

brancz left a comment

brancz Nov 18, 2020

lilic Nov 18, 2020

mrueg Nov 18, 2020

lilic Nov 18, 2020 •

edited

Loading

mrueg Nov 19, 2020

lilic Nov 19, 2020

lilic commented Nov 18, 2020

brancz commented Nov 19, 2020

k8s-ci-robot commented Nov 19, 2020

	--labels-metric-allow-list string By default all labels metric will only have resource name and namespace labels, to allow more labels pass in a list of resource names in their plural form and Kubernetes labels you would like to allow, =namespaces=[k8s-label1,k8s-labeln...],pods[app]...
	--labels-metric-allow-list string By default all labels metric will only have resource name and namespace labels, to allow more labels pass in a list of resource names in their plural form and Kubernetes labels you would like to allow, =namespaces=[k8s-label1,k8s-labeln...],pods=[app]...

	o.flags.Var(&o.LabelsAllowList, "labels-metric-allow-list", "By default all labels metric will only have resource name and namespace labels, to allow more labels pass in a list of resource names in their plural form and Kubernetes labels you would like to allow, =namespaces=[k8s-label1,k8s-labeln...],pods[app]...")
	o.flags.Var(&o.LabelsAllowList, "labels-metric-allow-list", "By default all labels metric will only have resource name and namespace labels, to allow more labels pass in a list of resource names in their plural form and Kubernetes labels you would like to allow, =namespaces=[k8s-label1,k8s-labeln...],pods=[app]...")

Adjust and refactor allowing labels to work for Kubernetes labels metrics #1301

Adjust and refactor allowing labels to work for Kubernetes labels metrics #1301

Conversation

lilic commented Nov 12, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lilic commented Nov 12, 2020

dgrisonnet left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lilic commented Nov 12, 2020

brancz commented Nov 12, 2020

lilic commented Nov 13, 2020

lilic commented Nov 13, 2020

jw-s commented Nov 17, 2020

Choose a reason for hiding this comment

brancz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lilic Nov 18, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lilic commented Nov 18, 2020

brancz commented Nov 19, 2020

k8s-ci-robot commented Nov 19, 2020

lilic Nov 18, 2020 •

edited

Loading