Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[nri-kubernetes] Update version of kube-state-metrics dependency #1076

Closed
SashaShcherbyna opened this issue May 4, 2023 · 12 comments · Fixed by #1149
Closed

[nri-kubernetes] Update version of kube-state-metrics dependency #1076

SashaShcherbyna opened this issue May 4, 2023 · 12 comments · Fixed by #1149
Labels
bug Categorizes issue or PR as related to a bug. triage/in-progress Issue or PR is in the process of being triaged.

Comments

@SashaShcherbyna
Copy link

Bug description

Kubernetes: 1.26.2-gke.1000
nri-bundle: 5.0.11 which includes kube-state-metrics: version 4.23.0. This cluster version has deprecated API autoscaling.apiserver.k8s.io/v2beta2 looks like kube-state-metrics version 4.23.0 does not know about it, and generate lot of errors:
E0504 14:57:41.765197 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.24.4/tools/cache/reflector.go:167: Failed to watch *v2beta2.HorizontalPodAutoscaler: failed to list *v2beta2.HorizontalPodAutoscaler: the server could not find the requested resource 86 W0504 14:58:12.635910 1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.24.4/tools/cache/reflector.go:167: failed to list *v2beta2.HorizontalPodAutoscaler: the server could not find the requested resource 85 E0504 14:58:12.635943 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.24.4/tools/cache/reflector.go:167: Failed to watch *v2beta2.HorizontalPodAutoscaler: failed to list *v2beta2.HorizontalPodAutoscaler: the server could not find the requested resource 84 W0504 14:59:04.231191 1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.24.4/tools/cache/reflector.go:167: failed to list *v2beta2.HorizontalPodAutoscaler: the server could not find the requested resource

Version of Helm and Kubernetes
Helm: v3.9.3
Kubernetes: 1.26.2-gke.1000 (GCP GKE)

Which chart?

nri-bundle
version: 5.0.11

What happened?

We updated claster to version 1.26.2-gke.1000 and nri-bundle to version 5.0.11, after that i can see lot of errors in pod newrelic-kube-state-metrics
W0504 14:40:45.760497 1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.24.4/tools/cache/reflector.go:167: failed to list *v2beta2.HorizontalPodAutoscaler: the server could not find the requested resource

What you expected to happen?

Update to latest version newrelic-kube-state-metrics in nri-bundle

How to reproduce it?

GKE: 1.26.2 + nri-bundle: 5.0.11
@SashaShcherbyna SashaShcherbyna added bug Categorizes issue or PR as related to a bug. triage/pending Issue or PR is pending for triage and prioritization. labels May 4, 2023
@workato-integration
Copy link

https://issues.newrelic.com/browse/NEWRELIC-8461

@workato-integration
Copy link

Jira CommentId: 201402
Commented by bthaker:

Just FYI: Latest version is: [https://artifacthub.io/packages/helm/newrelic/nri-bundle] 5.0.15

@SashaShcherbyna
Copy link
Author

version 5.0.15 includes kube-state-metrics: version 4.23.0.

@workato-integration
Copy link

Jira CommentId: 204167
Commented by alaiakeith:

Per [https://github.com/kubernetes/kube-state-metrics/issues/1711,] it looks like this was fixed last November in KSM v2.7.0, which is included in [v4.24.0 |https://github.com/prometheus-community/helm-charts/releases/tag/kube-state-metrics-4.24.0]and later of the KSM helm chart. 

@workato-integration
Copy link

Jira CommentId: 204177
Commented by htroisi:

[~bthaker] I'm not sure how this stacks against other planned agent work, but before updating the kube-state-metrics version we deploy as part of the nri-bundle, we'll need to do some testing to make sure the new version works with our integration. 

@workato-integration workato-integration bot changed the title nri-bundle v2beta2.HorizontalPodAutoscaler: the server could not find the requested resource [nri-kubernetes] Update version of kube-state-metrics dependency May 23, 2023
@workato-integration
Copy link

Jira CommentId: 204858
Commented by bthaker:

[~csong] [~jjaramillo] Do we have a manual or automated test for the KSM compatibility check with nri-bundle for every Kubernetes version/release that we support? Any suggestions for the next steps for the ask in this Jira issue?

@marinegreff
Copy link

Is there any update on this? Version 5.0.24 also includes kube-state-metrics:4.23.0 which forbids using nri-bundle on kubernetes 1.26

@davidgit
Copy link
Contributor

hi @marinegreff, we are planning how to proceed with this upgrade because this upgrade requires deprecating support for older k8s versions.

We'll keep you posted

@marinegreff
Copy link

Hi @davidgit , thank you for the update.

I understand the problem, but I just wanted to mention that older k8s versions are being deprecated quickly for cloud-managed clusters (1.23 is already deprecated, 1.24 is already deprecated for AKS and soon on GKE and EKS). We currently face the deprecation of k8s 1.25 by the end of this year, hence my question.
EOL dates for managed clusters:

I'll keep an eye open

@davidgit
Copy link
Contributor

davidgit commented Aug 14, 2023 •
edited

Thanks @marinegreff! We are aware of it and that's one of the main reasons to deprecate its support too

@catt231
Copy link

catt231 commented Aug 31, 2023

Are there any timescales for this?

@davidgit
Copy link
Contributor

Hi @catt231,

Our plan is to address this during the next couple of weeks.

@svetlanabrennan svetlanabrennan added triage/in-progress Issue or PR is in the process of being triaged. and removed triage/pending Issue or PR is pending for triage and prioritization. labels Sep 1, 2023
@juanjjaramillo juanjjaramillo mentioned this issue Sep 7, 2023
Merged
1 task
juanjjaramillo added a commit that referenced this issue Sep 15, 2023
@juanjjaramillo
[nri-bundle] Update KSM version (#1149)
05d6f8f 
<!--
Thank you for contributing to New Relic's Helm charts. Before you submit
this PR we'd like to
make sure you are aware of our technical requirements:

*
https://github.com/newrelic-experimental/helm-charts/blob/master/CONTRIBUTING.md#technical-requirements

For a quick overview across what we will look at reviewing your PR,
please read
our review guidelines:

*
https://github.com/newrelic-experimental/helm-charts/blob/master/REVIEW_GUIDELINES.md

Following our best practices right from the start will accelerate the
review process and
help get your PR merged quicker.

When updates to your PR are requested, please add new commits and do not
squash the
history. This will make it easier to identify new changes. The PR will
be squashed
anyways when it is merged. Thanks.

For fast feedback, please @-mention maintainers that are listed in the
Chart.yaml file.

Please make sure you test your changes before you push them. Once
pushed, a Github Action
will run across your changes and do some initial checks and linting.
These checks run
very quickly. Please check the results. We would like these checks to
pass before we
even continue reviewing your changes.
-->
#### What this PR does / why we need it:
Updates KSM version installed with the bundle, so we can fully support
the latest Kubernetes versions (>v1.23)

#### Which issue this PR fixes
  - fixes #1076 

#### Special notes for your reviewer:

#### Checklist
- [x] Title of the PR starts with chart name (e.g. `[mychartname]`)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Categorizes issue or PR as related to a bug. triage/in-progress Issue or PR is in the process of being triaged.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[nri-bundle] Update KSM version newrelic/helm-charts
5 participants
@davidgit @SashaShcherbyna @catt231 @svetlanabrennan @marinegreff