Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[kube-state-metrics] Scraping target broken after update to v4.30 #3101

Closed
ldunkum opened this issue Mar 7, 2023 · 4 comments
Closed

[kube-state-metrics] Scraping target broken after update to v4.30 #3101

ldunkum opened this issue Mar 7, 2023 · 4 comments
Labels
bug Something isn't working

Comments

@ldunkum
Copy link

ldunkum commented Mar 7, 2023

Describe the bug a clear and concise description of what the bug is.

We use the kube-prometheus-stack chart and in v45.2.0, the kube-state-metrics was updated to v4.30.*.
Since the update, the kube-state-metrics Endpoint is in state down with the following error proto: wrong wireType = 0 for field Metric.
As v4.30 introduces kube-state-metrics v2.8.0 (app version), on the first glance it seems like the bug might be related to the feature introduced in kubernetes/kube-state-metrics#1974. Unfortunately, I'm not quite sure, which exact chart version this is happening in, but I've tried it two weeks ago and today, and get the same error, so it's probably happening in multiple chart versions.

I also opened an issue for the kube-prometheus-stack (#3050) but haven't gotten an answer so far. If it is more fitting there, this issue can be closed, but it has been open without any response for about 2 weeks.

What's your helm version?

v3.11.0 (fluxcd helm-controller)

What's your kubectl version?

Which chart?

kube-state-metrics

What's the chart version?

v4.30.*

What happened?

We updated the kube-prometheus-stack chart version from v45.1.1 to v45.2.0, which resulted in a kube-state-metrics chart update from v4.24.* to v4.30.*.

What you expected to happen?

No response

How to reproduce it?

Update kube-prometheus-stack from v45.1.1 to >= v45.2.0.

Enter the changed values of values.yaml?

resources:
  requests:
    cpu: 10m
    memory: 32Mi
  limits:
    cpu: null
    memory: 128Mi

Enter the command that you execute and failing/misfunctioning.

We use Flux CD, I think it probably runs helm upgrade after updating the CRDs, but I'm not exactly sure.

Anything else we need to know?

No response

@ldunkum ldunkum added the bug Something isn't working label Mar 7, 2023
@ldunkum
Copy link
Author

ldunkum commented Mar 14, 2023

It appears this is an upstream issue that's being discussed in kubernetes/kube-state-metrics#2022.

@CatherineF-dev
Copy link

FYI: it's fixed in v2.8.2

@CatherineF-dev
Copy link

Please try #3137

@CatherineF-dev
Copy link

/close

@mrueg mrueg closed this as completed Mar 22, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants