[node metrics] add message field to kube_node_status_condition

[daveoy]

What would you like to be added:

the message field from kube_node_status_condition would be helpful to have for visualization and alerting purposes -- especially for custom conditions

Why is this needed:

to enhance alerts based on these conditions or add context to visualizations of node status conditions

Describe the solution you'd like

add a message label to the existing timeseries generated by the generator family function that tracks this metric

Additional context

could it be as simple as adding the desired field as a label here

kube-state-metrics/internal/store/node.go

Lines 478 to 507 in 85762cd

	func createNodeStatusConditionFamilyGenerator() generator.FamilyGenerator {
	return *generator.NewFamilyGeneratorWithStability(
	"kube_node_status_condition",
	"The condition of a cluster node.",
	metric.Gauge,
	basemetrics.STABLE,
	"",
	wrapNodeFunc(func(n *v1.Node) *metric.Family {
	ms := make([]*metric.Metric, len(n.Status.Conditions)*len(conditionStatuses))
	// Collect node conditions and while default to false.
	for i, c := range n.Status.Conditions {
	conditionMetrics := addConditionMetrics(c.Status)
	for j, m := range conditionMetrics {
	metric := m
	metric.LabelKeys = []string{"condition", "status"}
	metric.LabelValues = append([]string{string(c.Type)}, metric.LabelValues...)
	ms[i*len(conditionStatuses)+j] = metric
	}
	}
	return &metric.Family{
	Metrics: ms,
	}
	}),
	)
	}

?

[ricardoapl]

I would be happy to submit a patch to support your use case.

However, I'm afraid such label could have a significant impact on cardinality. It seems to me that conditions DiskPressure, MemoryPressure, PIDPressure, and NetworkUnavailable have a small, bounded set of values for the message field, whereas condition Ready doesn't.

Let's wait for others to comment on this.

[daveoy]

Thanks for that. I'm happy to contribute the patch as well.

Regarding cardinality; Perhaps it could be feature flagged or something?

[dgrisonnet]

I agree with @ricardoapl, message is unbounded so it could cause cardinality explosions if we introduced it. Especially if some variable information such as timestamps are introduced inside of it.

Regarding cardinality; Perhaps it could be feature flagged or something?

I wouldn't be in favor of adding any feature to kube-state-metrics that could harm users clusters, even behind feature flags. In the end we would still be responsible for the cardinality explosions that are bound to happen with such metrics.

[daveoy]

how about if enablement meant automatically adding a labeldrop blacklist that doesn't allow message / reason etc that the user had to repeal:

relabel_configs:
- action: labeldrop
  regex: message
- action: labeldrop
  regex: reason

or perhaps a default regex sort of thing in kubernetes sd objects (serviceMonitor,podMonitor etc) -- and docs for non-kubernetes config use cases? this way users could allow a certain set of regex-matching messages through and contain the chaos?

[dgrisonnet]

/assign @ricardoapl
/triage accepted