You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We have some connectivity problems between our nomad servers and sometimes the leader changed because current leader can't contact other nodes.
After this election, some periodic job are force run whereas they were already launched at the good time.
The particularity is that our nomad servers and clients are in UTC and our nomad periodic job specify a timezone.
func (p *PeriodicConfig) Next(fromTime time.Time) time.Time {
switch p.SpecType {
case PeriodicSpecCron:
if e, err := cronexpr.Parse(p.Spec); err == nil {
return e.Next(fromTime)
}
[...]
}
}
The cron parser does not take care of the configured timezone. So with the job above, the last launch time was in UTC at 2017-06-19 17:00 so the next launch parsed in "0 19 * * *" without timezone will be 2017-06-19 19:00 UTC
When leader changed the next day at 2017-06-20 09:21, computed next launch time is in the past so Nomad runs the job.
I'm just not sure where the launch.Launch come from in restorePeriodicDispatcher
(if it's the timestamp parsed for the last periodic sub-job name, it's in UTC, but if all evaluations were gc, I don't know...)
@cyrilgdn Thanks for the issue and the debugging! I think you are spot on about the restore not taking the timezone into consideration! Will hopefully get this fixed in 0.6.0
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.
Nomad version
Nomad v0.5.6
Issue
We have some connectivity problems between our nomad servers and sometimes the leader changed because current leader can't contact other nodes.
After this election, some periodic job are force run whereas they were already launched at the good time.
The particularity is that our nomad servers and clients are in UTC and our nomad periodic job specify a timezone.
I'm not sure but I think in this call: https://github.com/hashicorp/nomad/blob/master/nomad/leader.go#L297
The function is: https://github.com/hashicorp/nomad/blob/master/nomad/structs/structs.go#L1866
The cron parser does not take care of the configured timezone. So with the job above, the last launch time was in UTC at 2017-06-19 17:00 so the next launch parsed in "0 19 * * *" without timezone will be 2017-06-19 19:00 UTC
When leader changed the next day at 2017-06-20 09:21, computed next launch time is in the past so Nomad runs the job.
I see in other call to Next function that fromTime is convert to the periodic timezone
(e.g.: https://github.com/hashicorp/nomad/blob/master/nomad/periodic.go#L218)
I'm just not sure where the launch.Launch come from in restorePeriodicDispatcher
(if it's the timestamp parsed for the last periodic sub-job name, it's in UTC, but if all evaluations were gc, I don't know...)
Job file
Nomad logs
Multiple servers logs:
The text was updated successfully, but these errors were encountered: