Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ISSUE-1465: Added kube_pod_status_ready_time and kube_pod_status_containers_ready_time #1482

Closed
wants to merge 4 commits into from
Closed

ISSUE-1465: Added kube_pod_status_ready_time and kube_pod_status_containers_ready_time #1482

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
6379aab
Added kube_pod_status_ready_time and kube_pod_status_containers_ready…
sgrzemski May 18, 2021
af5241f
Applied gofmt
sgrzemski May 18, 2021
a0fb2b4
Added tests for kube_pod_status_ready_time and kube_pod_status_contai…
sgrzemski May 18, 2021
1d33b0f
Added two more families, so I had to fix benchamrk tests
sgrzemski May 18, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/pod-metrics.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@
| kube_pod_labels | Gauge | Kubernetes labels converted to Prometheus labels | | `pod`=&lt;pod-name&gt; <br> `namespace`=&lt;pod-namespace&gt; <br> `label_POD_LABEL`=&lt;POD_LABEL&gt; <br> `uid`=&lt;pod-uid&gt; | STABLE |
| kube_pod_status_phase | Gauge | The pods current phase | | `pod`=&lt;pod-name&gt; <br> `namespace`=&lt;pod-namespace&gt; <br> `phase`=&lt;Pending\|Running\|Succeeded\|Failed\|Unknown&gt; <br> `uid`=&lt;pod-uid&gt; | STABLE |
| kube_pod_status_ready | Gauge | Describes whether the pod is ready to serve requests | | `pod`=&lt;pod-name&gt; <br> `namespace`=&lt;pod-namespace&gt; <br> `condition`=&lt;true\|false\|unknown&gt; <br> `uid`=&lt;pod-uid&gt; | STABLE |
| kube_pod_status_ready_time | Gauge | Time when pod passed readiness probes. | seconds | `pod`=&lt;pod-name&gt; <br> `namespace`=&lt;pod-namespace&gt; <br> `uid`=&lt;pod-uid&gt; | STABLE |
| kube_pod_status_containers_ready_time | Gauge | Time when pod containers entered Ready state. | seconds | `pod`=&lt;pod-name&gt; <br> `namespace`=&lt;pod-namespace&gt; <br> `uid`=&lt;pod-uid&gt; | STABLE |
Copy link
Member

@mrueg mrueg May 17, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
| kube_pod_status_containers_ready_time | Gauge | Time when pod containers entered Ready state. | seconds | `pod`=&lt;pod-name&gt; <br> `namespace`=&lt;pod-namespace&gt; <br> `uid`=&lt;pod-uid&gt; | STABLE |
| kube_pod_status_container_ready_time | Gauge | Time when the container of the pod entered Ready state. | seconds | `pod`=&lt;pod-name&gt; <br> `namespace`=&lt;pod-namespace&gt; <br> `uid`=&lt;pod-uid&gt; | STABLE |

Would suggest to use singular here, since we already have kube_pod_container_info and kube_pod_container_status_waiting

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello @mrueg,
The k8s documentation specifies this particular condition as

ContainersReady: all containers in the Pod are ready.

, hence the plural. Due to the fact there can be more than 1 containers in a pod and this condition turns into true only when all pods are ready, I think we should stick with plural. What do you think? If you still want to stick with singular that's fine for me as well, but I found it a little bit confusing.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, good point! I'll defer to @lilic @brancz and @tariq1890 if we want to stick closer to k8s or not

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer keeping to how we have it in kube-state-metrics which is kube_pod_container_info, but up to @brancz and @mrueg .

| kube_pod_status_scheduled | Gauge | Describes the status of the scheduling process for the pod | |`pod`=&lt;pod-name&gt; <br> `namespace`=&lt;pod-namespace&gt; <br> `condition`=&lt;true\|false\|unknown&gt; <br> `uid`=&lt;pod-uid&gt; | STABLE |
| kube_pod_container_info | Gauge | Information about a container in a pod | | `container`=&lt;container-name&gt; <br> `pod`=&lt;pod-name&gt; <br> `namespace`=&lt;pod-namespace&gt; <br> `image`=&lt;image-name&gt; <br> `image_id`=&lt;image-id&gt; <br> `container_id`=&lt;containerid&gt; <br> `uid`=&lt;pod-uid&gt; | STABLE |
| kube_pod_container_status_waiting | Gauge | Describes whether the container is currently in waiting state | | `container`=&lt;container-name&gt; <br> `pod`=&lt;pod-name&gt; <br> `namespace`=&lt;pod-namespace&gt; <br> `uid`=&lt;pod-uid&gt; | STABLE |
Expand Down
54 changes: 54 additions & 0 deletions internal/store/pod.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,8 @@ func podMetricFamilies(allowLabelsList []string) []generator.FamilyGenerator {
createPodStartTimeFamilyGenerator(),
createPodStatusPhaseFamilyGenerator(),
createPodStatusReadyFamilyGenerator(),
createPodStatusReadyTimeFamilyGenerator(),
createPodStatusContainersReadyTimeFamilyGenerator(),
createPodStatusReasonFamilyGenerator(),
createPodStatusScheduledFamilyGenerator(),
createPodStatusScheduledTimeFamilyGenerator(),
Expand Down Expand Up @@ -1362,6 +1364,32 @@ func createPodStartTimeFamilyGenerator() generator.FamilyGenerator {
)
}

func createPodStatusContainersReadyTimeFamilyGenerator() generator.FamilyGenerator {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add a test for both of these, thank you!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added, thank you!

return *generator.NewFamilyGenerator(
"kube_pod_status_containers_ready_time",
"Readiness achieved time in unix timestamp for a pod containers.",
metric.Gauge,
"",
wrapPodFunc(func(p *v1.Pod) *metric.Family {
ms := []*metric.Metric{}

for _, c := range p.Status.Conditions {
if c.Type == v1.ContainersReady {
ms = append(ms, &metric.Metric{
LabelKeys: []string{},
LabelValues: []string{},
Value: float64((c.LastTransitionTime).Unix()),
})
}
}

return &metric.Family{
Metrics: ms,
}
}),
)
}

func createPodStatusPhaseFamilyGenerator() generator.FamilyGenerator {
return *generator.NewFamilyGenerator(
"kube_pod_status_phase",
Expand Down Expand Up @@ -1433,6 +1461,32 @@ func createPodStatusReadyFamilyGenerator() generator.FamilyGenerator {
)
}

func createPodStatusReadyTimeFamilyGenerator() generator.FamilyGenerator {
return *generator.NewFamilyGenerator(
"kube_pod_status_ready_time",
"Readiness achieved time in unix timestamp for a pod.",
metric.Gauge,
"",
wrapPodFunc(func(p *v1.Pod) *metric.Family {
ms := []*metric.Metric{}

for _, c := range p.Status.Conditions {
if c.Type == v1.PodReady {
ms = append(ms, &metric.Metric{
LabelKeys: []string{},
LabelValues: []string{},
Value: float64((c.LastTransitionTime).Unix()),
})
}
}

return &metric.Family{
Metrics: ms,
}
}),
)
}

func createPodStatusReasonFamilyGenerator() generator.FamilyGenerator {
return *generator.NewFamilyGenerator(
"kube_pod_status_reason",
Expand Down
24 changes: 18 additions & 6 deletions internal/store/pod_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1075,13 +1075,19 @@ func TestPodStore(t *testing.T) {
{
Type: v1.PodReady,
Status: v1.ConditionTrue,
LastTransitionTime: metav1.Time{
Time: time.Unix(1501666018, 0),
},
},
},
},
},
Want: `
# HELP kube_pod_status_ready Describes whether the pod is ready to serve requests.
# TYPE kube_pod_status_ready gauge
# HELP kube_pod_status_ready_time Readiness achieved time in unix timestamp for a pod.
# TYPE kube_pod_status_ready gauge
# TYPE kube_pod_status_ready_time gauge
kube_pod_status_ready_time{namespace="ns1",pod="pod1",uid="uid1"} 1.501666018e+09
kube_pod_status_ready{condition="false",namespace="ns1",pod="pod1",uid="uid1"} 0
kube_pod_status_ready{condition="true",namespace="ns1",pod="pod1",uid="uid1"} 1
kube_pod_status_ready{condition="unknown",namespace="ns1",pod="pod1",uid="uid1"} 0
Expand All @@ -1100,16 +1106,22 @@ func TestPodStore(t *testing.T) {
{
Type: v1.PodReady,
Status: v1.ConditionFalse,
LastTransitionTime: metav1.Time{
Time: time.Unix(1501666018, 0),
},
},
},
},
},
Want: `
# HELP kube_pod_status_ready Describes whether the pod is ready to serve requests.
# TYPE kube_pod_status_ready gauge
kube_pod_status_ready{condition="false",namespace="ns2",pod="pod2",uid="uid2"} 1
kube_pod_status_ready{condition="true",namespace="ns2",pod="pod2",uid="uid2"} 0
kube_pod_status_ready{condition="unknown",namespace="ns2",pod="pod2",uid="uid2"} 0
# HELP kube_pod_status_ready_time Readiness achieved time in unix timestamp for a pod.
# TYPE kube_pod_status_ready gauge
# TYPE kube_pod_status_ready_time gauge
kube_pod_status_ready_time{namespace="ns2",pod="pod2",uid="uid2"} 1.501666018e+09
kube_pod_status_ready{condition="false",namespace="ns2",pod="pod2",uid="uid2"} 1
kube_pod_status_ready{condition="true",namespace="ns2",pod="pod2",uid="uid2"} 0
kube_pod_status_ready{condition="unknown",namespace="ns2",pod="pod2",uid="uid2"} 0
`,
MetricNames: []string{"kube_pod_status_ready"},
},
Expand Down Expand Up @@ -1613,7 +1625,7 @@ func BenchmarkPodStore(b *testing.B) {
},
}

expectedFamilies := 50
expectedFamilies := 52
for n := 0; n < b.N; n++ {
families := f(pod)
if len(families) != expectedFamilies {
Expand Down
8 changes: 6 additions & 2 deletions main_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -203,14 +203,16 @@ func TestFullScrapeCycle(t *testing.T) {
# HELP kube_pod_labels Kubernetes labels converted to Prometheus labels.
# HELP kube_pod_overhead_cpu_cores The pod overhead in regards to cpu cores associated with running a pod.
# HELP kube_pod_overhead_memory_bytes The pod overhead in regards to memory associated with running a pod.
# HELP kube_pod_runtimeclass_name_info The runtimeclass associated with the pod.
# HELP kube_pod_owner Information about the Pod's owner.
# HELP kube_pod_restart_policy Describes the restart policy in use by this pod.
# HELP kube_pod_runtimeclass_name_info The runtimeclass associated with the pod.
# HELP kube_pod_spec_volumes_persistentvolumeclaims_info Information about persistentvolumeclaim volumes in a pod.
# HELP kube_pod_spec_volumes_persistentvolumeclaims_readonly Describes whether a persistentvolumeclaim is mounted read only.
# HELP kube_pod_start_time Start time in unix timestamp for a pod.
# HELP kube_pod_status_containers_ready_time Readiness achieved time in unix timestamp for a pod containers.
# HELP kube_pod_status_phase The pods current phase.
# HELP kube_pod_status_ready Describes whether the pod is ready to serve requests.
# HELP kube_pod_status_ready_time Readiness achieved time in unix timestamp for a pod.
# HELP kube_pod_status_reason The pod status reasons
# HELP kube_pod_status_scheduled Describes the status of the scheduling process for the pod.
# HELP kube_pod_status_scheduled_time Unix timestamp when pod moved into scheduled status
Expand Down Expand Up @@ -253,14 +255,16 @@ func TestFullScrapeCycle(t *testing.T) {
# TYPE kube_pod_labels gauge
# TYPE kube_pod_overhead_cpu_cores gauge
# TYPE kube_pod_overhead_memory_bytes gauge
# TYPE kube_pod_runtimeclass_name_info gauge
# TYPE kube_pod_owner gauge
# TYPE kube_pod_restart_policy gauge
# TYPE kube_pod_runtimeclass_name_info gauge
# TYPE kube_pod_spec_volumes_persistentvolumeclaims_info gauge
# TYPE kube_pod_spec_volumes_persistentvolumeclaims_readonly gauge
# TYPE kube_pod_start_time gauge
# TYPE kube_pod_status_containers_ready_time gauge
# TYPE kube_pod_status_phase gauge
# TYPE kube_pod_status_ready gauge
# TYPE kube_pod_status_ready_time gauge
# TYPE kube_pod_status_reason gauge
# TYPE kube_pod_status_scheduled gauge
# TYPE kube_pod_status_scheduled_time gauge
Expand Down