-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
kube_cronjob_next_schedule_time has time in the past #500
Comments
kube-state-metrics/pkg/collectors/cronjob.go Lines 154 to 166 in 23d8fc3
This looks to me like a timezone normalization problem. This is tricky as Prometheus only handles unix timestamps and next schedule times need to be calculated based on the actual timezone. To me the
Any idea, how would a container might know about this? |
I poked around a bit and it seems that K8S API is reporting timestamps with UTC timezone designator (the Z at the end). |
Interesting, so you are saying, that the Kubernetes API is not respecting your timezone settings, and as a result kube-state-metrics is using the same "wrong" timezone. That seems plausible, we should open a new issue on Kubernetes in that case. |
I think that kubernetes API works correctly - I don't need the API to keep the timezone, I just need that the status contains correct timestamp. I.e. the job is specified to run at 2AM and the server is in CEST. The job have been started at 2AM CEST. Just the API shows 0AM UTC, which represents the same point in time so there is no problem in it. |
That would mean we need to teach kube-state-metrics the timezone that the cronjob is working off of, which would either mean we need to read it from somewhere (if that information would actually be correct, I don't know) or allow it to be passed as a flag. What do you think? |
Yes, I was thinking about it and got to the same conclusion. |
But this is not all - I am not getting the next schedule time shifted by my timezone offset. The original problem was that I am getting something that looks like last run time. |
I think that's because next schedule time is based on the UTC time, and kube-state-metrics thinks it's in UTC. So if we make sure kube-state-metrics parses the timestamp as UTC but calculates the next time with the configured timezone then we should get the right unix timestamp (I think .. timezone stuff is confusing, we should write proper unit tests for this). |
So I've tried to set timezone in the container:
So now I think this is definitely timezone problem. If we can set the kube-state-metrics's timezone to the same zone that is used by the cluster, the problem is fixed. |
I'm not sure setting the timezone to the cluster's timezone is possible unless you do something rather hacky, like mount the hosts |
True, /etc/timezone is not the way. But consider using TZ variable instead of adding a new CLI flag for this - it is pretty standard, many apps (i.e. date) obey it. |
sure I think we can do both and have the flag default to the value of the TZ environment variable |
Issues go stale after 90d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
Stale issues rot after 30d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
Rotten issues close after 30d of inactivity. Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
@fejta-bot: Closing this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Mapping We still need a better way (maybe TZ env support ?) to specify timezone with kube-state-metrics. |
Facing the same problem. Is there any updates? |
@zerda Hi how did you map the
|
Is this a BUG REPORT or FEATURE REQUEST?:
/kind bug
What happened:
I've got a cron job with:
For this cronjob, current value of kube_cronjob_next_schedule_time metric is 1532397600 (now it is 1532433916, so it is in the past). When formated for my timezone (+0200), the time is 2018-07-24 02:00:00 which is equal to the time the job was run tonight.
What you expected to happen:
kube_cronjob_next_schedule_time should contain time in the future (i.e. 2018-07-25 02:00:00).
How to reproduce it (as minimally and precisely as possible):
It happens to all my cron jobs so I suppose it is enough to create a cronjob and check kube_cronjob_next_schedule_time's value.
Anything else we need to know?:
I've also played with another job, which was created 47 days ago and was suspended since the beginning. I unsuspended it, checked the metric and got 47.75 days in the past.
Also note that the cluster's hosts timezone is set to CEST.
Environment:
kubectl version
): N/AThe text was updated successfully, but these errors were encountered: