KEP 3140: TimeZone support in CronJob #3143

soltysh · 2022-01-14T16:18:18Z

One-line PR description: TimeZone support in CronJob
Issue link: TimeZone support in CronJob #3140

/assign @ravisantoshgudimetla @janetkuo
/cc @iterion

/sig apps

k8s-ci-robot · 2022-01-14T16:18:21Z

@soltysh: GitHub didn't allow me to request PR reviews from the following users: iterion.

Note that only kubernetes members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

One-line PR description: TimeZone support in CronJob

Issue link: TimeZone support in CronJob #3140

/assign @ravisantoshgudimetla @janetkuo
/cc @iterion

/sig apps

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

soltysh · 2022-01-14T16:28:17Z

/assign @deads2k

sftim · 2022-01-14T17:08:15Z

keps/sig-apps/3140-TimeZone-support-in-CronJob/README.md

+
+### Goals
+
+- Add the field `.spec.timeZone` which allows specifying a valid TimeZone


Is this a name or an offset from UTC?

name - updated

sftim · 2022-01-14T17:09:06Z

keps/sig-apps/3140-TimeZone-support-in-CronJob/README.md

+<!--
+What other approaches did you consider, and why did you rule them out? These do
+not need to be as detailed as the proposal, but should include enough
+information to express the idea and why it was not acceptable.
+-->


Whichever sense of the .spec.timeZone field we use (name or offset from UTC), let's list the other approach as an alternative.

sftim · 2022-01-14T17:09:33Z

keps/sig-apps/3140-TimeZone-support-in-CronJob/kep.yaml

+# The following PRR answers are required at beta release
+metrics:
+  - my_feature_metric


Maybe comment this out?

sftim · 2022-01-14T17:12:07Z

keps/sig-apps/3140-TimeZone-support-in-CronJob/README.md

+The value provided in `TimeZone` field will be validated against the embedded
+golang timezone database.


Does this mean that changes to time zone databases need to be applied to the API server and kube-controller-manager in a particular order?

No need for manual action, just follow the general guideline for updating the cluster: https://kubernetes.io/docs/tasks/administer-cluster/cluster-upgrade/

sftim · 2022-01-14T17:13:45Z

keps/sig-apps/3140-TimeZone-support-in-CronJob/README.md

+The `CronJobSpec` structure is expanded with new `TimeZone` field which allows
+specifying the name of the time zone to be used. Missing or empty value of the
+field indicates the current behavior, which relies on the time zone of the
+kube-controller-manager process.


Suggested change

The `CronJobSpec` structure is expanded with new `TimeZone` field which allows

specifying the name of the time zone to be used. Missing or empty value of the

field indicates the current behavior, which relies on the time zone of the

kube-controller-manager process.

The `.spec` for a CronJob is expanded with new `timeZone` field which allows

specifying the name of the time zone to be used. Missing or empty value of the

field indicates the current behavior, which relies on the time zone of the

kube-controller-manager process.

In the API code, that looks like:

Does the kube-controller-manager populate the .spec.timeZone once it detects a CronJob with no timezone set?

Should it?

no, it uses the default, to maintain backwards compatibility

no, again backwards compatibility

keps/sig-apps/3140-TimeZone-support-in-CronJob/README.md

atiratree · 2022-01-14T22:33:40Z

keps/sig-apps/3140-TimeZone-support-in-CronJob/README.md

+on the time zone of the kube-controller-manager process which is hard for a
+regular user to figure out. Exposing an explicit field for setting a time zone
+will allow CronJob authors to simplify the user experience when creating CronJobs.
+


Would it make sense to also introduce a way to set a global default TZ?

Ah, I had an idea about that.
In my view that is best left for an external controller or for an admission webhook. However, the “external” mechanism could still be official Kubernetes code.

Yes, no global setting, we don't have it anywhere, and I'd leave it to cluster admin to implement, re-use such setting.

sftim · 2022-01-31T15:53:47Z

keps/sig-apps/3140-TimeZone-support-in-CronJob/README.md

+
+## Proposal
+
+Add the field `.spec.timeZone` to the CronJob resource. The cronjob controller


We could make the field a bit more self-documenting

Suggested change

Add the field `.spec.timeZone` to the CronJob resource. The cronjob controller

Add the field `.spec.timeZoneName` to the CronJob resource. The cronjob controller

I think using time zone is in most cases sufficient.

sftim · 2022-01-31T15:55:08Z

keps/sig-apps/3140-TimeZone-support-in-CronJob/README.md

+
+CronJob controller will use non-nil, non-empty value from `TimeZone` field when
+parsing the schedule and during scheduling the next run time. Additionally the
+time zone will be reflected in the `.status.lastSuccessfulTime` and `.status.lastScheduleTime`.


We could also record the applied offset from UTC (from the last invocation) in .status. Maybe after an alpha though?

lastSuccessfulTime is of metav1.Time type which embeds time.Time which already has time zone information embeded in it, so I'm planning to re-use that.

sftim · 2022-01-31T15:56:38Z

keps/sig-apps/3140-TimeZone-support-in-CronJob/README.md

+Why should this KEP _not_ be implemented?
+-->
+
+## Alternatives


I suppose the other other alternative is timezone details in the cron time spec (the approach that got merged by accident).

Rarely, we re-use the current fields, that case you're talking about is more a feature creep not intentional on our side, b/c it exposes implementation details, that were not covered in the original design .

The inline timezone spec - as mentioned in https://web.archive.org/web/20211007074332/https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/ - is not the design we want. I get that. We should nonetheless list it as a ruled-out alternative.

I can PR that in after this merges.

soltysh · 2022-01-31T16:01:30Z

@ravisantoshgudimetla @atiratree @deads2k @janetkuo ptal

ravisantoshgudimetla

/lgtm

Thank you for working on this @soltysh. This will be really helpful for the community.

deads2k · 2022-01-31T19:16:10Z

keps/sig-apps/3140-TimeZone-support-in-CronJob/README.md

+
+When downgrading from a release with this feature, to a release without `TimeZone`
+there are a few cases:
+  1. If TimeZone feature gate was enabled and user specified a TimeZone, the newly


How will you test to be sure the cronjob controller correctly handles going from timezone aware to non-timezone aware?

Manual testing of a cronjob with time zone specified, I can't think of a better approach.

I've mentioned that in the test plan, that we want to enabled with the feature on and off to ensure the controller behaves as expected. As much as possible those tests will be automated in integration, but not everything is possible there.

keps/sig-apps/3140-TimeZone-support-in-CronJob/README.md

deads2k · 2022-01-31T19:18:06Z

keps/sig-apps/3140-TimeZone-support-in-CronJob/README.md

+This section must be completed when targeting beta to a release.
+-->
+
+###### How can a rollout or rollback fail? Can it impact already running workloads?


An upgrade flow can be vulnerable to the enable, disable, enable if you have a lease that is acquired by a new KCM, then an old KCM, then a new KCM. The unit and integration tests need to be robust.

ack, added that here

deads2k · 2022-01-31T19:19:23Z

keps/sig-apps/3140-TimeZone-support-in-CronJob/README.md

+Pick one more of these and delete the rest.
+-->
+
+- [ ] Metrics


While not required for alpha, can you think of what metrics you'll watch? If you're able to fill this in now, people running the alpha will know what information to look for.

We have a pre-existing cronjob_job_creation_skew which ensures that the skew between the schedule and the actual creation is minimal, we could look into that, but that won't give you information about if a cronjob has or not time zone set, although in all honesty from that point of view you only care about timely creation of new jobs.

dims · 2022-02-01T17:40:10Z

@soltysh

can you please add examples of "timezone name"?
https://pkg.go.dev/time/tzdata mentions that the binary size is going to be larger when time/tzdata is imported, do we want to mention that somewhere? ( looking at https://pkg.go.dev/time/tzdata )

deads2k · 2022-02-01T19:47:58Z

PRR is complete for alpha.

/approve

k8s-ci-robot · 2022-02-01T19:48:24Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: deads2k, janetkuo, ravisantoshgudimetla, soltysh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~keps/prod-readiness/OWNERS~~ [deads2k]
~~keps/sig-apps/OWNERS~~ [janetkuo,soltysh]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

soltysh · 2022-02-02T17:12:36Z

@dims

can you please add examples of "timezone name"?

Added link to https://en.wikipedia.org/wiki/List_of_tz_database_time_zones

https://pkg.go.dev/time/tzdata mentions that the binary size is going to be larger when time/tzdata is imported, do we want to mention that somewhere? ( looking at https://pkg.go.dev/time/tzdata )

Added that information, even though the size is negligible since according to my test it's less than 500kB.

Both are in the last commit.

atiratree · 2022-02-02T17:24:09Z

/lgtm

dims · 2022-02-02T17:52:49Z

thanks @soltysh !

BenTheElder · 2022-03-29T18:08:21Z

keps/sig-apps/3140-TimeZone-support-in-CronJob/README.md

+at the time was that introducing such functionality would require cluster operator
+to manually include TimeZone database since golang did not have one.
+Starting from [golang 1.15](https://go.dev/doc/go1.15) `time/tzdata` package can
+be embedded in the binary thus removing the requirement for external database.


I think we should explicitly call out in the KEP intent to embed this package as the approach for keeping tzdata up to date?

(Or else, if we have another approach planned, we should outline it?)

Hmm. If there is a timezone change (eg a new national law), does that trigger a patch release?

The approach I initially put forward was to have Kubernetes itself only aware of offsets from UTC.

The implementing PR appears to be taking the import _ "time/tzdata" approach, operators would have the option to patch their deployments with images or mounts containing a non-embedded on-disk tzdata and go will prefer that.

I'm also not sure that the embedded copy will be used at all in official releases / it's not clear how helpful importing this is, all of our images should be shipping with tzdata on-disk in the image. will x-ref the PR.

EDIT: bringing up here kubernetes/kubernetes#108032 (comment)

I've already responded to that other comment, but I'll leave the justification also here. The intent why we picked this approach, as a safe net was mentioned here in the motivation section. I'll make sure to add some more details during beta reviews to provide more details around expectations and assumptions for the embedded vs the on-system tz database.

BenTheElder · 2022-03-29T21:01:18Z

keps/sig-apps/3140-TimeZone-support-in-CronJob/README.md

+
+This problem can be mitigated with a fresh build of kube-controller-manager with
+updated golang version, but it heavily relies on the go community to keep the
+time zone database up-to-date.


I feel like expectations around this need fleshing out, also as pointed out in this thread kubernetes/kubernetes#108032 (comment) the actual update affecting most users will likely be in the docker images, so we seem to already have confusion around when / where / how this will be updated.

k8s-ci-robot assigned janetkuo and ravisantoshgudimetla Jan 14, 2022

k8s-ci-robot added the sig/apps Categorizes an issue or PR as relevant to SIG Apps. label Jan 14, 2022

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jan 14, 2022

soltysh mentioned this pull request Jan 14, 2022

TimeZone support in CronJob #3140

Closed

13 tasks

k8s-ci-robot added the kind/kep Categorizes KEP tracking issues and PRs modifying the KEP directory label Jan 14, 2022

TimeZone suppot in CronJob

b64b3bf

soltysh force-pushed the cronjob_timezone branch from 0ff8d1b to b64b3bf Compare January 14, 2022 16:26

k8s-ci-robot assigned deads2k Jan 14, 2022

sftim reviewed Jan 14, 2022

View reviewed changes

janetkuo reviewed Jan 14, 2022

View reviewed changes

keps/sig-apps/3140-TimeZone-support-in-CronJob/README.md Outdated Show resolved Hide resolved

janetkuo approved these changes Jan 14, 2022

View reviewed changes

keps/sig-apps/3140-TimeZone-support-in-CronJob/README.md Show resolved Hide resolved

keps/sig-apps/3140-TimeZone-support-in-CronJob/README.md Outdated Show resolved Hide resolved

janetkuo changed the title ~~KEP 3140: TimeZone suppot in CronJob~~ KEP 3140: TimeZone support in CronJob Jan 14, 2022

atiratree reviewed Jan 14, 2022

View reviewed changes

sftim reviewed Jan 31, 2022

View reviewed changes

Address comments

82eb73a

soltysh force-pushed the cronjob_timezone branch from 8ce2edf to 82eb73a Compare January 31, 2022 15:59

ravisantoshgudimetla approved these changes Jan 31, 2022

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 31, 2022

deads2k reviewed Jan 31, 2022

View reviewed changes

keps/sig-apps/3140-TimeZone-support-in-CronJob/README.md Show resolved Hide resolved

deads2k reviewed Jan 31, 2022

View reviewed changes

janetkuo approved these changes Feb 1, 2022

View reviewed changes

Added metric

09a41a1

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 1, 2022

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 1, 2022

Add more clarification about tzdata

ff61819

k8s-ci-robot assigned atiratree Feb 2, 2022

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 2, 2022

k8s-ci-robot merged commit ef0923b into kubernetes:master Feb 2, 2022

k8s-ci-robot added this to the v1.24 milestone Feb 2, 2022

soltysh deleted the cronjob_timezone branch February 2, 2022 17:27

BenTheElder reviewed Mar 29, 2022

View reviewed changes

BenTheElder mentioned this pull request Mar 29, 2022

KEP 3140: TimeZone support for CronJob kubernetes/kubernetes#108032

Merged

BenTheElder reviewed Mar 29, 2022

View reviewed changes

yonatankahana mentioned this pull request Apr 10, 2022

feat: add cronjob injection support (alpha) k8tz/k8tz#17

Merged

sftim mentioned this pull request Feb 21, 2023

Promote CronJob TZ to GA kubernetes/kubernetes#115904

Merged


		### Goals

		- Add the field `.spec.timeZone` which allows specifying a valid TimeZone

		The value provided in `TimeZone` field will be validated against the embedded
		golang timezone database.


		## Proposal

		Add the field `.spec.timeZone` to the CronJob resource. The cronjob controller

	Add the field `.spec.timeZone` to the CronJob resource. The cronjob controller
	Add the field `.spec.timeZoneName` to the CronJob resource. The cronjob controller

KEP 3140: TimeZone support in CronJob #3143

KEP 3140: TimeZone support in CronJob #3143

Conversation

soltysh commented Jan 14, 2022

k8s-ci-robot commented Jan 14, 2022

soltysh commented Jan 14, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

soltysh commented Jan 31, 2022

ravisantoshgudimetla left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dims commented Feb 1, 2022

deads2k commented Feb 1, 2022

k8s-ci-robot commented Feb 1, 2022

soltysh commented Feb 2, 2022

atiratree commented Feb 2, 2022

dims commented Feb 2, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BenTheElder Mar 29, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BenTheElder Mar 29, 2022 •

edited

Loading