Add Pod UID to all "kube_pod_*" metrics

[ichekrygin]

Is this a BUG REPORT or FEATURE REQUEST?:

kind feature

What happened:

Currently, only kube_pod_info metric exposes pod UID value.
It is assumed that namespace/pod combination should be always unique at any given point in time.
However, it does not accommodate for cases when pods can be created, deleted, and recreated with the same name over some period of time.
This is typically the case with StatefulSet pods and CRD's auxiliary pods (the pods created by custom resource controllers).
In those cases, it is not uncommon to recreate pods with the previously used names.

This results in interesting and at times incorrect metric values, especially when used in "join" multiple metrics: ON (namespace, pod) prometheus queries.

What you expected to happen:

I propose to add uid label to all kube_pod_.* metrics as it is done in kube_pod_info

How to reproduce it (as minimally and precisely as possible):

The following graph demonstrates the incorrect durations (seconds) value between kube_pod_created and container_start_time_seconds

The green line represents the duration (seconds) and the lines below represent distinct pods according to kube_pod_info (which includes uid value).

We can observe that sometime after 12:28 the old pod was deleted and a new pod (w/ new uid but with the same name) was created shortly after 12:30.
However, the query join did not include uid value (since it does not exist on kube_pod_created) and resulted in incorrect duration computation where
the newer pod's container_start_time_seconds was computed against the old pod's kube_pod_created value.

This issue would have been avoided if we could use join ON (namespace,pod,uid) instead of ON(namespace,pod).

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version):
• Kube-state-metrics image version

[ichekrygin]

If the community/maintainers find this feature request useful, I’d be happy to contribute.

[lilic]

@ichekrygin hey, that sounds like a great idea! Feel free to contribute the changes, if you need any guidence let me know!