Metricbeat to support kube-state-metrics version 2.0.0

[jmartin51]

Describe the enhancement:
Metricbeat to support kube-state-metrics version 2.0.0 for kubernetes 1.17/1.18

Describe a specific use case for the enhancement or feature:

• As per our compatibility Matrix metricbeat does support kubernetes 1.17/1.18
  https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-kubernetes.html#_compatibility_28

• As per the kube-state-metrics compatibility matrix, kubernetes 1.17/1.18 only fully supports kube-state-metrics version 2.0.0-alpha:
  https://github.com/kubernetes/kube-state-metrics#compatibility-matrix

• kube-state-metrics version 2.0.0-alpha is not supported by Metricbeat. Metricbeat comes with kube-state-metrics 1.8.0 .

Some clarifications are needed here. Thanks.

[elasticmachine]

Pinging @elastic/integrations-platforms (Team:Platforms)

[rukas]

Is there any update on this?

[exekias]

@ChrsMark this one looks important, we need to check for any breaking change sin ksm 2.0

[eedugon]

FYI I was doing some tests yesterday with Kubernetes 1.19.7 (GKE) + Kube-state-metrics 2.0.0-rc0 (released a few days ago) and Metricbeat 7.10.2 and all the integration working perfectly fine for the following metricsets:

                - module: kubernetes
                  hosts: ["kube-state-metrics:8080"]
                  period: 30s
                  add_metadata: true
                  metricsets:
                    - state_node
                    - state_deployment
                    - state_daemonset
                    - state_replicaset
                    - state_pod
                    - state_container
                    - state_cronjob
                    - state_resourcequota
                    - state_statefulset
                - module: kubernetes
                  metricsets:
                    - event

(I'm not sure if the event module is related with this, but anyway :) )

Apparently no errors in the output:

"state_container":{"events":43,"success":43},"state_daemonset":{"events":8,"success":8},"state_deployment":{"events":8,"success":8},"state_node":{"events":3,"success":3},"state_pod":{"events":28,"success":28},"state_replicaset":{"events":10,"success":10},"state_resourcequota":{"events":91,"success":91},"state_statefulset":{"events":2,"success":2}

[ChrsMark]

Thanks @eedugon ! For clarity, all state_* metricsets collect metrics from kube-state-metrics.

Most probably nothing will break with the new versions. What may happen is that we will don't leverage new metrics that will be exposed or some metrics that we expect will not be there. Indeed we need to find the time to check this one.

[MnrGreg]

Breaking changes in kube-state-metrics v2.0.0 affects state_container and state_node.
Deprecated Metrics were dropped kubernetes/kube-state-metrics#974.

Example of replaced metrics:
state_container:

kube_pod_container_resource_limits_memory_bytes
-->
kube_pod_container_resource_limits{namespace="kube-system",pod="metricbeat-jwgjb",container="metricbeat",node="node1",resource="memory",unit="byte"} 8.05306368e+08

beats/metricbeat/module/kubernetes/state_container/state_container.go

Lines 50 to 53 in e1e6d33

	"kube_pod_container_resource_limits_cpu_cores": p.Metric("cpu.limit.cores"),
	"kube_pod_container_resource_requests_cpu_cores": p.Metric("cpu.request.cores"),
	"kube_pod_container_resource_limits_memory_bytes": p.Metric("memory.limit.bytes"),
	"kube_pod_container_resource_requests_memory_bytes": p.Metric("memory.request.bytes"),

---

state_node:

kube_node_status_capacity_cpu_cores
-->
kube_node_status_capacity{node="node1",resource="cpu",unit="core"} 2

beats/metricbeat/module/kubernetes/state_node/state_node.go

Lines 44 to 49 in e1e6d33

	"kube_node_status_allocatable_pods": p.Metric("pod.allocatable.total"),
	"kube_node_status_capacity_pods": p.Metric("pod.capacity.total"),
	"kube_node_status_capacity_memory_bytes": p.Metric("memory.capacity.bytes"),
	"kube_node_status_allocatable_memory_bytes": p.Metric("memory.allocatable.bytes"),
	"kube_node_status_capacity_cpu_cores": p.Metric("cpu.capacity.cores"),
	"kube_node_status_allocatable_cpu_cores": p.Metric("cpu.allocatable.cores"),

[svonliebenstein]

Great work on the MR @MichaelKatsoulis!

Since Kubernetes 1.19, kube-state-metrics v2.x.x is required to be compatible (https://github.com/kubernetes/kube-state-metrics/#compatibility-matrix). Unfortunately this is currently not fully working with the latest 7.14.1 metricbeat.
Can we expect a 7.14.2 or 7.15 release anytime soon with this feature?

When this is available, it would mean that it's possible to setup a kubernetes cluster with ksm and metricbeat on Kubernetes 1.19 - 1.22.

[MichaelKatsoulis]

Great work on the MR @MichaelKatsoulis!

Thank you @svonliebenstein!

Can we expect a 7.14.2 or 7.15 release anytime soon with this feature?

The support of kube-state-metrics v2.0 will be part of patch release 7.15.1 and new release 7.16.