kube_pod_container_state_started is not returned for some pods

[AnastasiaBlack]

What happened:
We have multiple pods (jobs) that run for a short period of time (the time from container start to termination varies from 2-3 seconds to a couple of minutes for each pod). We use a query kube_pod_container_state_started{pod="needePodName"} in Prometheus and it appears that it works for some pods and doesn't show the result for others (while other queries, including kube_pod_start_time show the result for all of the pods).

What you expected to happen:
kube_pod_container_state_started shows result for every pod, and not just for some of them.

How to reproduce it (as minimally and precisely as possible):
Run multiple pods with short-time living containers, check that a query kube_pod_container_state_started to prometheus doesn't show the results for some of the pods.

Anything else we need to know?:
If we check the pods with kubectl describe pod [pod_name] - the values we need (container start time) are present. When we make a query with kube_pod_start_time for every pod - we get the expected result (that is the timestamp when the pod started). But now we want to get the timestamps when the Containers were started.

Environment:

• kube-state-metrics version: v.2.0.0
• Kubernetes version (use kubectl version): v.1.19.6

[harjas27]

This metric is captured from the field State.Running.StartedAt in the container status of pod.
In the span of 2-3 seconds, the state of the container changes from Waiting -> Running -> Terminated. Whenever the state is updated, the metrics for that pod are overwritten in the metrics map. Since the state changes to Terminated, the metric is not captured. This might be the reason for the metric not being collected for some of the pods.
As a fix, we can update

kube-state-metrics/internal/store/pod.go

Line 293 in 4009bea

Value: float64((cs.State.Running.StartedAt).Unix()),

to take the value from either cs.State.Running.StartedAt or cs.State.Terminated.StartedAt depending on the state of the container.
@lilic

[AnastasiaBlack]

That would be great!

[AnastasiaBlack]

Hello! Is there any news on this issue? Will it be implemented in the future?