kube_cronjob_annotations is output even though --metric-annotations-allowlist is not set

[yosshi825]

What happened:

When I set --metric-labels-allowlist=cronjobs=[*], the metrics are also output to kube_cronjob_annotations

What you expected to happen:

• Expected behavior

  # HELP kube_cronjob_annotations Kubernetes annotations converted to Prometheus labels.
  # TYPE kube_cronjob_annotations gauge
  +kube_cronjob_annotations{namespace="kube-system",cronjob="hello"} 1
  # HELP kube_cronjob_labels Kubernetes labels converted to Prometheus labels.
  # TYPE kube_cronjob_labels gauge
  +kube_cronjob_labels{namespace="kube-system",cronjob="hello",label_k8s_app="busybox"} 1

  If --metric-annotations-allowlist is not set, the metrics of kube_x_annotations will not be output.

• Fact

  # HELP kube_cronjob_annotations Kubernetes annotations converted to Prometheus labels.
  # TYPE kube_cronjob_annotations gauge
  -kube_cronjob_annotations{namespace="kube-system",cronjob="hello",label_kubectl_kubernetes_io_last_applied_configuration="{\"apiVersion\":\"batch/v1\",\"kind\":\"CronJob\",\"metadata\":{\"annotations\":{},\"labels\":{\"k8s-app\":\"busybox\"},\"name\":\"hello\",\"namespace\":\"kube-system\"},\"spec\":{\"jobTemplate\":{\"spec\":{\"template\":{\"spec\":{\"containers\":[{\"command\":[\"/bin/sh\",\"-c\",\"date; echo Hello from the Kubernetes cluster\"],\"image\":\"busybox\",\"imagePullPolicy\":\"IfNotPresent\",\"name\":\"hello\"}],\"restartPolicy\":\"OnFailure\"}}}},\"schedule\":\"*/1 * * * *\"}}\n"} 1
  # HELP kube_cronjob_labels Kubernetes labels converted to Prometheus labels.
  # TYPE kube_cronjob_labels gauge
  +kube_cronjob_labels{namespace="kube-system",cronjob="hello",label_k8s_app="busybox"} 1

How to reproduce it (as minimally and precisely as possible):

1. Create a cluster using kind.

   kind: Cluster
   apiVersion: kind.x-k8s.io/v1alpha4
   nodes:
   - role: control-plane
     image: kindest/node:v1.22.1
   - role: worker
     image: kindest/node:v1.22.1
   - role: worker
     image: kindest/node:v1.22.1
   - role: worker
     image: kindest/node:v1.22.1

2. Install kube-state-metrics from https://github.com/kubernetes/kube-state-metrics/tree/master/examples/standard.

3. Apply the following manifest

   apiVersion: batch/v1
   kind: CronJob
   metadata:
     name: hello
     labels: 
       k8s-app: busybox
   spec:
     schedule: "*/1 * * * *"
     jobTemplate:
       spec:
         template:
           spec:
             containers:
             - name: hello
               image: busybox
               imagePullPolicy: IfNotPresent
               command:
               - /bin/sh
               - -c
               - date; echo Hello from the Kubernetes cluster
             restartPolicy: OnFailure

4. Check metrics

   # HELP kube_cronjob_annotations Kubernetes annotations converted to Prometheus labels.
   # TYPE kube_cronjob_annotations gauge
   +kube_cronjob_annotations{namespace="kube-system",cronjob="hello"} 1
   # HELP kube_cronjob_labels Kubernetes labels converted to Prometheus labels.
   # TYPE kube_cronjob_labels gauge
   +kube_cronjob_labels{namespace="kube-system",cronjob="hello"} 1

5. kubectl edit deploy kube-state-metrics

       spec:
         containers:
         -  image: k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.2.0
   +        args:
   +        - --metric-labels-allowlist=cronjobs=[*]
           image: k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.2.0
           imagePullPolicy: IfNotPresent

6. Check metrics

   # HELP kube_cronjob_annotations Kubernetes annotations converted to Prometheus labels.
   # TYPE kube_cronjob_annotations gauge
   -kube_cronjob_annotations{namespace="kube-system",cronjob="hello",label_kubectl_kubernetes_io_last_applied_configuration="{\"apiVersion\":\"batch/v1\",\"kind\":\"CronJob\",\"metadata\":{\"annotations\":{},\"labels\":{\"k8s-app\":\"busybox\"},\"name\":\"hello\",\"namespace\":\"kube-system\"},\"spec\":{\"jobTemplate\":{\"spec\":{\"template\":{\"spec\":{\"containers\":[{\"command\":[\"/bin/sh\",\"-c\",\"date; echo Hello from the Kubernetes cluster\"],\"image\":\"busybox\",\"imagePullPolicy\":\"IfNotPresent\",\"name\":\"hello\"}],\"restartPolicy\":\"OnFailure\"}}}},\"schedule\":\"*/1 * * * *\"}}\n"} 1
   # HELP kube_cronjob_labels Kubernetes labels converted to Prometheus labels.
   # TYPE kube_cronjob_labels gauge
   +kube_cronjob_labels{namespace="kube-system",cronjob="hello",label_k8s_app="busybox"} 1

   Metrics are output to kube_x_annotations even though --metric-annotations-allowlist is not set

Anything else we need to know?:

When I tried with pods = [*], nodes = [*] and deployments = [*], the metrics of kube_x_annotations were not output.
I think the behavior of cronjobs = [*] is strange.

Environment:

• kube-state-metrics version:

  kube-state-metrics:v2.2.0

• Kubernetes version (use kubectl version):

  $ kubectl version                                                                                 
  Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.1", GitCommit:"632ed300f2c34f6d6d15ca4cef3d3c7073412212", GitTreeState:"clean", BuildDate:"2021-08-19T15:38:26Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"darwin/amd64"}
  Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.1", GitCommit:"632ed300f2c34f6d6d15ca4cef3d3c7073412212", GitTreeState:"clean", BuildDate:"2021-08-25T19:54:13Z", GoVersion:"go1.16.7", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration:

  kind

• Other info:

  N/A

[mrueg]

Thanks for your report!
Can you try downgrading apiVersion: batch/v1 to apiVersion: batch/v1beta1?

[yosshi825]

I tried apiVersion: batch / v1beta1 with k8s v1.20

1. Create a cluster using kind.

   kind: Cluster
   apiVersion: kind.x-k8s.io/v1alpha4
   nodes:
   - role: control-plane
     image: kindest/node:v1.20.7
   - role: worker
     image: kindest/node:v1.20.7
   - role: worker
     image: kindest/node:v1.20.7
   - role: worker
     image: kindest/node:v1.20.7

2. Apply the following manifest

   -apiVersion: batch/v1
   +apiVersion: batch/v1beta1
   kind: CronJob
   metadata:
     name: hello
     labels: 
       k8s-app: busybox
       :

   $ kubectl get cronjobs hello -o yaml
   +apiVersion: batch/v1beta1
   kind: CronJob
   metadata:
     annotations:
       kubectl.kubernetes.io/last-applied-configuration: |
         {"apiVersion":"batch/v1beta1","kind":"CronJob","metadata":{"annotations":{},"labels":{"k8s-app":"busybox"},"name":"hello","namespace":"kube-system"},"spec":{"jobTemplate":{"spec":{"template":{"spec":{"containers":[{"command":["/bin/sh","-c","date; echo Hello from the Kubernetes cluster"],"image":"busybox","imagePullPolicy":"IfNotPresent","name":"hello"}],"restartPolicy":"OnFailure"}}}},"schedule":"*/1 * * * *"}}

3. Check metrics

   # HELP kube_cronjob_annotations Kubernetes annotations converted to Prometheus labels.
   # TYPE kube_cronjob_annotations gauge
   -kube_cronjob_annotations{namespace="kube-system",cronjob="hello",label_kubectl_kubernetes_io_last_applied_configuration="{\"apiVersion\":\"batch/v1beta1\",\"kind\":\"CronJob\",\"metadata\":{\"annotations\":{},\"labels\":{\"k8s-app\":\"busybox\"},\"name\":\"hello\",\"namespace\":\"kube-system\"},\"spec\":{\"jobTemplate\":{\"spec\":{\"template\":{\"spec\":{\"containers\":[{\"command\":[\"/bin/sh\",\"-c\",\"date; echo Hello from the Kubernetes cluster\"],\"image\":\"busybox\",\"imagePullPolicy\":\"IfNotPresent\",\"name\":\"hello\"}],\"restartPolicy\":\"OnFailure\"}}}},\"schedule\":\"*/1 * * * *\"}}\n"} 1
   # HELP kube_cronjob_labels Kubernetes labels converted to Prometheus labels.
   # TYPE kube_cronjob_labels gauge
   +kube_cronjob_labels{namespace="kube-system",cronjob="hello",label_k8s_app="busybox"} 1

   Same result as apiVersion: batch / v1

• Kubernetes version

  $ kubectl version
  Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.1", GitCommit:"632ed300f2c34f6d6d15ca4cef3d3c7073412212", GitTreeState:"clean", BuildDate:"2021-08-19T15:38:26Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"darwin/amd64"}
  Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.7", GitCommit:"132a687512d7fb058d0f5890f07d4121b3f0a2e2", GitTreeState:"clean", BuildDate:"2021-05-27T23:27:49Z", GoVersion:"go1.15.12", Compiler:"gc", Platform:"linux/amd64"}
  WARNING: version difference between client (1.22) and server (1.20) exceeds the supported minor version skew of +/-1
  

It's not what I expected, but I'll try again after this PR #1491 is merged.