Custom-resource-metrics non label conform annotation names

[ulikl]

What happened:

When using the generic include of annotations in config for custom-resource-state , via

labelsFromPath:
       "**": [metadata, annotations]

with Objects having annotation key which contain characters, which are not allowed in Prometheus Metrics labels.
e.g.

  metadata:
    annotations:
      field.cattle.io/creatorId: u-5cl2zk65bl
      ownerBindingsCreated: "true"

They are just exported as is.
i.e.

custom_... {field.cattle.io/creatorId="u-5cl2zk65b", ...} 1

This leads to an "expected equal, got "INVALID"" error, when Prometheus tries to parse these metrics.

What you expected to happen:

I would expect the annotation names to be adapted analog to the standard kube_node_labels metrics,
e.g.

custom_... {field_cattle_io_creatorId="u-5cl2zk65b", ...} 1

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• kube-state-metrics version: 2.8.2
• Kubernetes version (use kubectl version): 1.25.10
• Cloud provider or hardware configuration: rke2
• Other info:

[dashpole]

/assign @rexagod
/triage accepted

@dgrisonnet thinks you might have a PR out to fix this?

[kchopra456]

Checking if there was any movement on this?, we have had this issue for a while now.

[rexagod]

I believe this should've been addressed in v2.9+. @ulikl @kchopra456 Could you specify the KSM version you were experiencing this on?