CustomResourceMetrics info and stateset are now broken in main when scraping with prometheus

[primeroz]

What happened:
before #1930 all metrics type for CustomResourceMetrics were reported as type gauge but now ( i do run a build from main for some testing ) the type defined in the CR is respected and a metrics are reported as type info and stateset

for example

 λ curl localhost:8000/metrics 2>/dev/null | egrep "*_info" | grep TYPE                                                                                                                                            
# TYPE capi_azurecluster_info info 

while those are valid OpenTelemtry types they are not valid Prometheus Types and so Prometheus fail to scrape with invalid metric type "info"

What you expected to happen:

Prometheus be able to scrape those metrics

How to reproduce it (as minimally and precisely as possible):

Run a build of KSM from main with CustomResourceMetrics with type set to Info or StateSet ( i do use the cluster-api example https://github.com/kubernetes-sigs/cluster-api/blob/main/hack/observability/kube-state-metrics/crd-config.yaml )

Anything else we need to know?:

I think it would be great to support both Prometheus and OpenTelemetry , maybe with a field to set the type of metrics to expose and revert to present all metrics as gauges when Prometheus is the intended target

Environment:

• kube-state-metrics version: main
• Kubernetes version (use kubectl version): 1.24.10
• Cloud provider or hardware configuration:
• Other info:

pinging @bavarianbidi for reference

[mrueg]

@primeroz @bavarianbidi thanks for the report! Can you test and confirm if #1974 fixes your issue?

[mrueg]

/triage accepted

[primeroz]

I will test on Monday and report back. Thanks for the quick feedback

[primeroz]

testing a build out of [49af6b3](https://github.com/mrueg/kube-state-metrics/commit/49af6b3945a071ec63de8b2cadaf4d5eec1bdd57) and is looking good. thank you

[Efrat19]

Hi! I see this happening again for the new version 2.11.0 (2.10.1 is fine)
A curl to a ksm pod 10.254.30.65 with image registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.11.0:

curl --header "Accept: application/openmetrics-text" -v http://10.254.30.65:8080/metrics 
*   Trying 10.254.30.65:8080...
* Connected to 10.254.30.65 (10.254.30.65) port 8080
> GET /metrics HTTP/1.1
> Host: 10.254.30.65:8080
> User-Agent: curl/8.7.1
> Accept: application/openmetrics-text
> 
* Request completely sent off
< HTTP/1.1 200 OK
< Content-Type: text/plain; version=0.0.4; charset=utf-8
< Date: Thu, 28 Mar 2024 08:52:44 GMT
< Transfer-Encoding: chunked
....
a_stateset_metric{label=value} 1
....
* Connection #0 to host 10.254.30.65 left intact

The returned content type is text/plain and there's no # EOF char, so prometheus fails to scrape it.

[kallaics]

Hi!

I also recognized a same issue after last upgrade. I got on all of our clusters the invalid metric type "info" message by Prometheus scrape yesterday. I did do a rollback and the older version is fine.

Issue appeared on

• KSM: v2.11.0
• Prometheus version: v2.51.0

Last version works fine on

• KSM: v2.10.1
• Prometheus: v2.50.1

I tested the output of KSM and looks fine by me.

wget -q -O - http://10.244.4.148:8080/metrics

Sample output:

# HELP app_build_info <description>
# TYPE app_build_info info
app_build_info{customresource_group="apps",customresource_kind="Deployment",customresource_version="v1",exported_namespace="<namespace name>",instance="<instance name>"} 1
...

I checked the KSM and the Prometheus release notes, changelog yesterday, but I didn't found anything regarding to this on KSM or Prometheus side.

I found yesterday and I was not sure it is related to the KSM or the Prometheus itself.

[fischerman]

I checked the KSM and the Prometheus release notes, changelog yesterday, but I didn't found anything regarding to this on KSM or Prometheus side.

I also can't find anything in the change log. I can however confirm that only reverting KMS is sufficient. I tested both v2.11.0 and v2.10.1 with Prometheus 2.51.1. So something must have changed in KMS.

@rexagod You mentioned in #2270 (comment) that #2270 should fix this issue, too. From what I understand #2270 only applies to protobuf expositions. We only negotiate text-based formats in Prometheus:

  scrape_protocols:
  - OpenMetricsText1.0.0
  - OpenMetricsText0.0.1
  - PrometheusText0.0.4

Can the two issues still be related?

[rexagod]

Hello, this should be fixed in the latest v2.12.0 release. While the manifests haven't been merged yet, to test things out, you can checkout the latest tag.

PTAL at all three cases below. These are expected results.

┌[rexagod@nebuchadnezzar] [/dev/ttys004] 
└[~/repositories/misc]> curl --header "Accept: application/openmetrics-text" -v localhost:8080/metrics           
*   Trying [::1]:8080...
* Connected to localhost (::1) port 8080
> GET /metrics HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/8.4.0
> Accept: application/openmetrics-text
> 
< HTTP/1.1 200 OK
< Content-Type: application/openmetrics-text; version=0.0.1; charset=utf-8; escaping=values
< Date: Wed, 03 Apr 2024 08:46:31 GMT
< Content-Length: 580
< 
# HELP kube_customresource_foo_info foo_help
# TYPE kube_customresource_foo_info gauge
kube_customresource_foo_info{customresource_group="contoso.com",customresource_kind="MyPlatform",customresource_version="v1alpha1",name="test-dotnet-app"} ...
...
# EOF
* Connection #0 to host localhost left intact

┌[rexagod@nebuchadnezzar] [/dev/ttys004] 
└[~/repositories/misc]> curl --header "Accept: application/vnd.google.protobuf" -v localhost:8080/metrics
*   Trying [::1]:8080...
* Connected to localhost (::1) port 8080
> GET /metrics HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/8.4.0
> Accept: application/vnd.google.protobuf
> 
< HTTP/1.1 200 OK
< Content-Type: text/plain; version=0.0.4; charset=utf-8
< Date: Wed, 03 Apr 2024 08:49:15 GMT
< Content-Length: 574
< 
# HELP kube_customresource_foo_info foo_help
# TYPE kube_customresource_foo_info gauge
kube_customresource_foo_info{customresource_group="contoso.com",customresource_kind="MyPlatform",customresource_version="v1alpha1",name="test-dotnet-app"} ...
...
* Connection #0 to host localhost left intact

┌[rexagod@nebuchadnezzar] [/dev/ttys004] 
└[~/repositories/misc]> curl --header "Accept: application/plain-text" -v localhost:8080/metrics
*   Trying [::1]:8080...
* Connected to localhost (::1) port 8080
> GET /metrics HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/8.4.0
> Accept: application/plain-text
> 
< HTTP/1.1 200 OK
< Content-Type: text/plain; version=0.0.4; charset=utf-8
< Date: Wed, 03 Apr 2024 08:48:52 GMT
< Content-Length: 574
< 
# HELP kube_customresource_foo_info foo_help
# TYPE kube_customresource_foo_info gauge
kube_customresource_foo_info{customresource_group="contoso.com",customresource_kind="MyPlatform",customresource_version="v1alpha1",name="test-dotnet-app"} ...
...
* Connection #0 to host localhost left intact

[rexagod]

Feel free to ping this thread if things are still breaking after v2.12.

[fischerman]

Hi @rexagod, good news first: v2.12 fixes the issue for us and probably most users with a standard Prometheus configuration.

However I could not reproduce your example and the Prometheus exposition format of KMS is broken. This impacts everyone who uses experimental features. I've tested multiple versions of KMS and different exposition formats. For this I used Prometheus (v2.51.1) with the following config:

global:
  scrape_interval: 10s
# scrape_protocols:
# - PrometheusText0.0.4
scrape_configs:
  - job_name: "kms"
    static_configs:
      - targets: ["localhost:8080"]

The default scrape_protocols are:

scrape_protocols:
- OpenMetricsText1.0.0
- OpenMetricsText0.0.1
- PrometheusText0.0.4

Here are the results:

• v2.10.1
  • Prometheus default scrape protocols: KMS returns OM 1.0 + #TYPE info. WORKS
  • Prometheus asks for PrometheusText0.0.4 only: KMS returns text/plain (which means Prometheus text) + #TYPE info. invalid metric type "info"
  • Prometheus asks for PrometheusProto only: KMS returns text/plain (which means Prometheus text) + #TYPE info. invalid metric type "info"
• v2.11.0
  • Prometheus default scrape protocols: KMS returns text/plain (which means Prometheus text) + #TYPE info. invalid metric type "info"
• v2.12.0
  • Prometheus default scrape protocols: KMS returns OM 1.0 + #TYPE info. WORKS but yields a different result then your curl example above.
  • Prometheus asks for PrometheusText0.0.4 only: KMS returns text/plain (which means Prometheus text) + #TYPE info. invalid metric type "info"
  • Prometheus asks for PrometheusProto only: KMS returns text/plain (which means Prometheus text) + #TYPE info. invalid metric type "info"

So as far as I can tell v2.12.0 only works if Prometheus asks for OM. KMS should not return type info if the exposition format is Prometheus (in this case text).

It's weird. It looks a little bit like v2.10.1 and v2.12.0 behave the same, almost like I didn't change the image properly. But I checked three times. I've used registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.12.0.

[rexagod]

@fischerman That's weird. Can you checkout to the latest tag on your local (v2.12.0) and see if this is still reproduced (preferably using curl, so we can be sure any issues arising reside within KSM)? This would also ensure that this wasn't due an image replacement issue.

[kallaics]

I tested on v2.12.0 and I have an issue with custom resource state definitions. It looks like not iterating over the items correctly, when the metric name is same. I suppose the code probably override with the last item instead of append to the result, when the metric name is same.

I have Kustomization, HelmRelease, GitRepository, HelmRepository, Bucket, HelmChart, ImageRepository, ImagePolicy, ImageUpdateAutomation, Provider, Receiver and Alert resources with same metrics name and I just got back one, the HelmRelease.

This definition works well:

metric1
  - resource1

This not works well:

metric1:
  - resource1
  - resource2
  ...
  - resourceX

Same config worked well with version v2.10.1.

[mrueg]

I tested on v2.12.0 and I have an issue with custom resource state definitions. It looks like not iterating over the items correctly, when the metric name is same. I suppose the code probably override with the last item instead of append to the result, when the metric name is same.

I have Kustomization, HelmRelease, GitRepository, HelmRepository, Bucket, HelmChart, ImageRepository, ImagePolicy, ImageUpdateAutomation, Provider, Receiver and Alert resources with same metrics name and I just got back one, the HelmRelease.

This definition works well:

metric1
  - resource1

This not works well:

metric1:
  - resource1
  - resource2
  ...
  - resourceX

Same config worked well with version v2.10.1.

Can you create a new issue and provide a real config? I believe that this is unrelated to this issue.

[kallaics]

I tested on v2.12.0 and I have an issue with custom resource state definitions. It looks like not iterating over the items correctly, when the metric name is same. I suppose the code probably override with the last item instead of append to the result, when the metric name is same.
I have Kustomization, HelmRelease, GitRepository, HelmRepository, Bucket, HelmChart, ImageRepository, ImagePolicy, ImageUpdateAutomation, Provider, Receiver and Alert resources with same metrics name and I just got back one, the HelmRelease.
This definition works well:

metric1
  - resource1

This not works well:

metric1:
  - resource1
  - resource2
  ...
  - resourceX

Same config worked well with version v2.10.1.

Can you create a new issue and provide a real config? I believe that this is unrelated to this issue.

I can create an issue, but it happened on time with it.
It worked with v2.10.1 well.
The Prometheus provided the "invalid metric type" error message in v2.11.0 as I written above (#1973 (comment))
The current state is in my last commit. The configuration not changed since v2.10.1 as I saw it well.
This is the current state.

Issue created #2366

Many thanks for the help.