Argument namespaces-denylist produces an error with v2.10.0

[BernhardGruen]

What happened:

I upgraded kube-state-metrics from 2.9.2 to 2.10.0 on several clusters. On a single cluster it did not work as expected.

This is the error message I got there:

W0908 12:40:15.252401       1 reflector.go:533] pkg/mod/k8s.io/client-go@v0.27.5/tools/cache/reflector.go:231: failed to list *v1.Node: field label not supported: metadata.namespace
E0908 12:40:15.252471       1 reflector.go:148] pkg/mod/k8s.io/client-go@v0.27.5/tools/cache/reflector.go:231: Failed to watch  *v1.Node: failed to list *v1.Node: field label not supported: metadata.namespace
W0908 12:40:39.965420       1 reflector.go:533] pkg/mod/k8s.io/client-go@v0.27.5/tools/cache/reflector.go:231: failed to list *v1.CertificateSigningRequest: field label not supported: metadata.namespace
E0908 12:40:39.965531       1 reflector.go:148] pkg/mod/k8s.io/client-go@v0.27.5/tools/cache/reflector.go:231: Failed to watch *v1.CertificateSigningRequest: failed to list *v1.CertificateSigningRequest: field label not supported: metadata.namespace                               
W0908 12:40:52.773607       1 reflector.go:533] pkg/mod/k8s.io/client-go@v0.27.5/tools/cache/reflector.go:231: failed to list *v1.Namespace: field label not supported: metadata.namespace
E0908 12:40:52.773671       1 reflector.go:148] pkg/mod/k8s.io/client-go@v0.27.5/tools/cache/reflector.go:231: Failed to watch *v1.Namespace: failed to list *v1.Namespace: field label not supported: metadata.namespace

Also on that cluster metrics like kube_node_info do not exist anymore (this correlates to the error message about v1.Node does not have a field label metadata.namespace).

There is a single difference in configuration - the use of --namespaces-denylist=NAMESPACE_TO_EXCLUDE. Therefore I assume that this configuration produces the error. If I remove that configuration option it also works on that exact cluster.

What you expected to happen:

I expect to have metrics like kube_node_info and no exported metrics for namespaces on the denylist.

How to reproduce it (as minimally and precisely as possible):

One just have to add the command line argument --namespaces-denylist=NAMESPACE_TO_EXCLUDE. It does not matter if the namespace exists or not.

Environment:

• kube-state-metrics version: 2.10.0
• Kubernetes version (use kubectl version): 1.27.x (also tested on 1.26.x)
• Cloud provider or hardware configuration: Scaleway, DigitalOcean
• The image was provided by bitnami: docker.io/bitnami/kube-state-metrics:2.10.0-debian-11-r0 (but I don't think this makes a difference)

[mrueg]

Thanks for the report!
The resources you mentioned are non-namespaced resources.
I could track it down to this c3c5528
It looks like my change had some unintended results as I wasn't considering the namespace-denyfilter here.

[mrueg]

/triage accepted
/assign mrueg

[mrueg]

Can you test with #2189 please? @BernhardGruen

[BernhardGruen]

I can confirm that this version runs without an error message and kube_node_info is also present. Thank you for your really fast reaction.

I did not dig any further in the code but aren't there more non-namespaced objects that need that patch too like mutatingwebhookconfiguration, persistentvolume, clusterrole, validatingwebhookconfiguration, volumeattachment.

[mrueg]

Yes that is correct. I wanted to verify that the change is resolving the issue before changing them as well :)

edit: Likely someone else will take that over as I won't be able to do it before my travels.