-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Duration of kube_pod_container_status_terminated_reason metrics #344
Comments
Prometheus metrics always only reflect the current state. In this case, it reflects whatever the Pod object says, that the Kubernetes API returns. The value for the |
I find it really unlikely for a Prom scrape to catch a pod container while its current state is |
I agree here. Trying to make an alert when pods are OOMKilled. But its only in that state for milliseconds so |
Yeah we can absolutely add a metric that exposes the Last state. |
Anyone working on this? We need it to send alert to the correct team when their app OOMs in production... I could give it a try if no one has planned to add lastState. What would be a good prometheus metrics structure for this?
|
Hey guys, any updates on that? I am trying to implement a very similar alert... 😄 |
Hi guys, no one got this solved? |
You can turn it into a time series and use sum_over_time. To catch if anything has been OOMKilled in the last hour for example: |
+1 Where can we get this information as a durable manner ? |
+1 |
@dingobaby you can join the |
I would love to monitor OOM with |
I would like to add one more vote on this thread. Here are the tasks I'd like to achieve: While last state will help with task #1, is there anything we could do about enabling task #2 via kube-state-metrics as well? |
I guess one could introduce a For anyone wanting to tackle this, I am happy to help. Changes would need to touch |
@benoittgt I was able to get your query to work as expected by just removing the |
I understand that this metrics is keeping track of pods which were terminated and the reason for it, but the metrics store data of how many days? Like the data will be of last two days or last 12 hours?
The text was updated successfully, but these errors were encountered: