[k8sclusterreceiver] add pod status reason as a metric

[povilasv]

Component(s)

receiver/k8scluster

Is your feature request related to a problem? Please describe.

I would like to get pod status reason as a metric:

In Kube State metrics there are 5 pod reasons being reported - "Evicted", "NodeAffinity", "NodeLost", "Shutdown", "UnexpectedAdmissionError"

In practice it looks like this:

kube_pod_status_reason{pod="otel-traces-app-59dccdc594-8x28b", reason="Evicted"}
0
kube_pod_status_reason{pod="otel-traces-app-59dccdc594-8x28b", reason="NodeAffinity"}
0
kube_pod_status_reason{pod="otel-traces-app-59dccdc594-8x28b", reason="NodeLost"}
0
kube_pod_status_reason{pod="otel-traces-app-59dccdc594-8x28b", reason="Shutdown"}
0
kube_pod_status_reason{pod="otel-traces-app-59dccdc594-8x28b", reason="UnexpectedAdmissionError"}

K8s API docs describe this field as:

	// A brief CamelCase message indicating details about why the pod is in this state.
	// e.g. 'Evicted'
	// +optional
	Reason string `json:"reason,omitempty" protobuf:"bytes,4,opt,name=reason"`

Example from K8s kubectl get po -o yaml:

  message: 'The node was low on resource: ephemeral-storage. '
  phase: Failed
  qosClass: Burstable
  reason: Evicted
  startTime: "2023-07-21T10:44:19Z"

Any thoughts?

Describe the solution you'd like

There are two possible solutions:

• Add 5 metrics as gauges set to either 0 or 1 (similiar to k8s.node.conditon_ready, k8s.node.condition_pid_pressure)

Alternatively:

• we could encode to numbers, similarly as we do for pod phase:

func phaseToInt(phase corev1.PodPhase) int32 {
	switch phase {
	case corev1.PodPending:
		return 1
	case corev1.PodRunning:
		return 2
	case corev1.PodSucceeded:
		return 3
	case corev1.PodFailed:
		return 4
	case corev1.PodUnknown:
		return 5
	default:
		return 5
	}
}

I can work on this :)

Describe alternatives you've considered

No response

Additional context

No response

[github-actions]

Pinging code owners:

• k8sclusterreceiver: @dmitryax
• receiver/k8scluster: @dmitryax

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[povilasv]

REF #21234

[github-actions]

Pinging code owners for receiver/k8scluster: @dmitryax. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dmitryax]

If we add pod status reason as several metrics, we will also need to change the pod phase metric the same way. I wouldn't like to make this change to pod state metric until we have strict guidelines from the OTel Spec on how to represent this kind of data

If you want the pod state reason added right now, we can add it as one metric for consistency with the pod state and keep it optional (disabled by default) for now. Later we can reconsider it once we bring all the k8s metrics to the Semantic conventions.

But before that, we need to finalize #4367 to be able to add optional metrics at all.

[povilasv]

Created an issue for optional metrics in #24568