Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Missing kube_inventory metrics when Pod has no valid containers #8473

Closed
Fiery-Fenix opened this issue Nov 25, 2020 · 1 comment
Closed

Missing kube_inventory metrics when Pod has no valid containers #8473

Fiery-Fenix opened this issue Nov 25, 2020 · 1 comment
Labels
area/k8s bug unexpected problem or unintended behavior

Comments

@Fiery-Fenix
Copy link
Contributor

Fiery-Fenix commented Nov 25, 2020

Relevant telegraf.conf:

[[inputs.kube_inventory]]
interval = "30s"
url = "https://kubernetes.default"
namespace = ""
insecure_skip_verify = true

System info:

Telegraf version 1.15.2

Steps to reproduce:

On Kubernetes cluster one of Pods was manually restarted and we didn't receive any metrics that can indicate that restart because telegraf has take metrics directly in the moment when no valid containers were in v1.Pod.Status.ContainerStatuses.
Respective piece of code:

for i, cs := range p.Status.ContainerStatuses {
	c := p.Spec.Containers[i]
	gatherPodContainer(*p.Spec.NodeName, ki, p, *cs, *c, acc)
}

For some valid reasons (for example Network issues) it could be that p.Status.ContainerStatuses is empty and we will not receive any metrics at all until issue will be resolved, as in above example of restarting Pod.

Expected behavior:

kubernetes_pod_container,container_name=telegraf,namespace=default,node_name=ip-172-17-0-2.internal,node_selector_node-role.kubernetes.io/compute=true,pod_name=tick1,state=running,readiness=ready resource_requests_cpu_units=0.1,resource_limits_memory_bytes=524288000,resource_limits_cpu_units=0.5,restarts_total=0i,state_code=0i,state_reason="",resource_requests_memory_bytes=524288000 1547597616000000000
kubernetes_pod_container,container_name=telegraf,namespace=default,node_name=ip-172-17-0-2.internal,node_selector_node-role.kubernetes.io/compute=true,pod_name=tick1,state=unknown,readiness=unready resource_requests_cpu_units=0.1,resource_limits_memory_bytes=524288000,resource_limits_cpu_units=0.5,restarts_total=0i,state_code=3i,state_reason="",resource_requests_memory_bytes=524288000 1547597916000000000
kubernetes_pod_container,container_name=telegraf,namespace=default,node_name=ip-172-17-0-2.internal,node_selector_node-role.kubernetes.io/compute=true,pod_name=tick1,state=unknown,readiness=ready resource_requests_cpu_units=0.1,resource_limits_memory_bytes=524288000,resource_limits_cpu_units=0.5,restarts_total=0i,state_code=0i,state_reason="",resource_requests_memory_bytes=524288000 1547598516000000000

Actual behavior:

kubernetes_pod_container,container_name=telegraf,namespace=default,node_name=ip-172-17-0-2.internal,node_selector_node-role.kubernetes.io/compute=true,pod_name=tick1,state=running,readiness=ready resource_requests_cpu_units=0.1,resource_limits_memory_bytes=524288000,resource_limits_cpu_units=0.5,restarts_total=0i,state_code=0i,state_reason="",resource_requests_memory_bytes=524288000 1547597616000000000
kubernetes_pod_container,container_name=telegraf,namespace=default,node_name=ip-172-17-0-2.internal,node_selector_node-role.kubernetes.io/compute=true,pod_name=tick1,state=unknown,readiness=ready resource_requests_cpu_units=0.1,resource_limits_memory_bytes=524288000,resource_limits_cpu_units=0.5,restarts_total=0i,state_code=0i,state_reason="",resource_requests_memory_bytes=524288000 1547598516000000000

Additional info:

PR: #8472

@Fiery-Fenix Fiery-Fenix added the bug unexpected problem or unintended behavior label Nov 25, 2020
@akrantz01
Copy link
Contributor

Already fixed by #8472.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/k8s bug unexpected problem or unintended behavior
Projects
None yet
Development

No branches or pull requests

3 participants