Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace multiListerWatcher with independent listWatchers per namespace #1499

Merged
merged 5 commits into from
Jul 14, 2021

Conversation

fpetkovski
Copy link
Contributor

@fpetkovski fpetkovski commented Jun 7, 2021

The multiListerWatcher is a composite object encapsulating multiple
ListerWatchers and implements the ListerWatcher interface.
With the current implementation, when an individual lister fails, the
entire List operation fails. This causes no metrics to be shown when KSM
has no permissions to a single namespace.

In addition to this, the multiListerWatcher takes advantage of internal
implementation details if the client-go library by modifiying and
relying on the ResourceVersion metadata field. This introduces a bug
where reconnecting to the API server will break the multiListerWatcher
completely.

This commit replaces the multiListerWatcher with individual
ListerWatchers per each configured namespace, resolving both issues.
It introduces a MetricsWriter interface to writes out metrics from stores,
and implements it in the MetricsStore struct. It also introduces a MultiNamespaceMetricsWriter
struct implementing MetricsWriter to allow writing out metrics from multiple
related stores (stores holding data for objects of the same kind but different namespace).

What this PR does / why we need it:
When KSM has no permissions for resources in at least one namespace, it will not export any metrics on the /metrics endpoint.

Which issue(s) this PR fixes:
Fixes #1413

Fixes a bug where KSM did not export any metrics when it had no permissions for resources in at least one namespace

@k8s-ci-robot k8s-ci-robot requested review from brancz and mrueg June 7, 2021 15:25
@k8s-ci-robot
Copy link
Contributor

Welcome @fpetkovski!

It looks like this is your first PR to kubernetes/kube-state-metrics 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/kube-state-metrics has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jun 7, 2021
pkg/listwatch/listwatch.go Outdated Show resolved Hide resolved
@k8s-ci-robot k8s-ci-robot added do-not-merge/invalid-commit-message Indicates that a PR should not merge because it has an invalid commit message. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jun 7, 2021
@fpetkovski fpetkovski force-pushed the fix-multiwatcher branch 5 times, most recently from 50b9477 to cc47c3b Compare June 7, 2021 15:44
@k8s-ci-robot k8s-ci-robot removed the do-not-merge/invalid-commit-message Indicates that a PR should not merge because it has an invalid commit message. label Jun 7, 2021
@fpetkovski fpetkovski changed the title Fix multiListerWatcher to not not fail on individual namespace errors WIP: Fix multiListerWatcher to not not fail on individual namespace errors Jun 18, 2021
@k8s-ci-robot k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 18, 2021
@k8s-ci-robot k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jun 18, 2021
@fpetkovski fpetkovski force-pushed the fix-multiwatcher branch 4 times, most recently from e071c9f to f77a978 Compare June 21, 2021 11:46
@fpetkovski fpetkovski changed the title WIP: Fix multiListerWatcher to not not fail on individual namespace errors WIP: Replace multiListerWatcher with independent listWatchers per namespace Jun 21, 2021
@fpetkovski fpetkovski force-pushed the fix-multiwatcher branch 3 times, most recently from 9461f61 to cc1d509 Compare June 21, 2021 11:55
@k8s-ci-robot k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 29, 2021
fpetkovski and others added 4 commits June 29, 2021 13:27
The multiListerWatcher is a composite object encapsulating multiple
ListerWatchers and implements the ListerWatcher interface.
With the current implementation, when an individual lister fails, the
entire List operation fails. This causes no metrics to be shown when KSM
has no permissions to a single namespace.

In addition to this, the multiListerWatcher takes advantage of internal
implementation details if the client-go library by modifiying and
relying on the ResourceVersion metadata field. This introduces a bug
where reconnecting to the API server will break the multiListerWatcher
completely.

This commit replaces the multiListerWatcher with individual
ListerWatchers per each configured namespace, resolving both issues.

Signed-off-by: fpetkovski <filip.petkovsky@gmail.com>
Co-authored-by: Damien Grisonnet <damien.grisonnet@epita.fr>
@k8s-ci-robot k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 29, 2021
Co-authored-by: Manuel Rüger <manuel@rueg.eu>
@mrueg
Copy link
Member

mrueg commented Jun 29, 2021

/assign @tariq1890

we might want to backport this to release-2.1 as well.

@brancz
Copy link
Member

brancz commented Jul 14, 2021

Nice work!

/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 14, 2021
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: brancz, dgrisonnet, fpetkovski, mrueg

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 14, 2021
@k8s-ci-robot k8s-ci-robot merged commit 1d61fc1 into kubernetes:master Jul 14, 2021
@brancz
Copy link
Member

brancz commented Jul 14, 2021

I have a suspicion that this might also help with #694

mrueg added a commit to mrueg/kube-state-metrics that referenced this pull request Jul 28, 2021
* [CHANGE] Replace multiListWatcher with independent listWatchers per namespace kubernetes#1499
* [BUGFIX] Use BuilderInterface instead of internal/store.Builder in metricshandler kubernetes#1537
* [BUGFIX] Add WithAllowLabels to public BuilderInterface kubernetes#1514
mrueg added a commit to mrueg/kube-state-metrics that referenced this pull request Jul 28, 2021
* [CHANGE] Replace multiListWatcher with independent listWatchers per namespace kubernetes#1499
* [CHANGE] go.mod: Update and minimize dependencies kubernetes#1529
* [BUGFIX] Use BuilderInterface instead of internal/store.Builder in metricshandler kubernetes#1537
* [BUGFIX] Add WithAllowLabels to public BuilderInterface kubernetes#1514
mrueg added a commit to mrueg/kube-state-metrics that referenced this pull request Jul 28, 2021
* [CHANGE] Replace multiListWatcher with independent listWatchers per namespace kubernetes#1499
* [CHANGE] go.mod: Update and minimize dependencies kubernetes#1529
* [BUGFIX] Use BuilderInterface instead of internal/store.Builder in metricshandler kubernetes#1537
* [BUGFIX] Add WithAllowLabels to public BuilderInterface kubernetes#1514
mrueg added a commit to mrueg/kube-state-metrics that referenced this pull request Jul 28, 2021
* [CHANGE] Replace multiListWatcher with independent listWatchers per namespace kubernetes#1499
* [CHANGE] go.mod: Update and minimize dependencies kubernetes#1529
* [BUGFIX] Use BuilderInterface instead of internal/store.Builder in metricshandler kubernetes#1537
* [BUGFIX] Add WithAllowLabels to public BuilderInterface kubernetes#1514
mrueg added a commit to mrueg/kube-state-metrics that referenced this pull request Jul 28, 2021
* [CHANGE] Replace multiListWatcher with independent listWatchers per namespace kubernetes#1499
* [CHANGE] go.mod: Update and minimize dependencies kubernetes#1529
* [BUGFIX] Use BuilderInterface instead of internal/store.Builder in metricshandler kubernetes#1537
* [BUGFIX] Add WithAllowLabels to public BuilderInterface kubernetes#1514
mrueg added a commit to mrueg/kube-state-metrics that referenced this pull request Jul 28, 2021
* [CHANGE] Replace multiListWatcher with independent listWatchers per namespace kubernetes#1499
* [CHANGE] go.mod: Update and minimize dependencies kubernetes#1529
* [BUGFIX] Use BuilderInterface instead of internal/store.Builder in metricshandler kubernetes#1537
* [BUGFIX] Add WithAllowLabels to public BuilderInterface kubernetes#1514
mrueg added a commit to mrueg/kube-state-metrics that referenced this pull request Jul 29, 2021
* [CHANGE] Replace multiListWatcher with independent listWatchers per namespace kubernetes#1499
* [CHANGE] go.mod: Update and minimize dependencies kubernetes#1529
* [BUGFIX] Use BuilderInterface instead of internal/store.Builder in metricshandler kubernetes#1537
* [BUGFIX] Add WithAllowLabels to public BuilderInterface kubernetes#1514
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

ksm doesn't report any metrics at all if it lacks rights for just 1 subject namespace
6 participants