Consolidate workload controllers status #2833

ravisantoshgudimetla · 2021-07-22T02:11:19Z

One-line PR description:
Consolidate workload controllers status

Issue link: Consolidate workload controller status #2804

Other comments:

ravisantoshgudimetla · 2021-07-22T02:11:55Z

atiratree · 2021-07-22T21:03:00Z

keps/sig-apps/2804-consolidate-workload-controllers-status/README.md

+
+See the [Kubernetes API Conventions](https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#typical-status-properties) for more information on status conditions
+
+Because of the number of changes that are involved as part of this effort, we are thinking of a phased approach where we introduce these conditions to DaemonSet controller behind a featuregate flag first in one release and then make similar changes to StatefulSet controller. This also needs some code refactoring of existing conditions for Deployment controller.


wrt to Deployments, are we going to copy ReplicaFailure values to a new Failed condition for consistency?

atiratree · 2021-07-22T21:18:37Z

keps/sig-apps/2804-consolidate-workload-controllers-status/README.md

+API type(s): DaemonSet, StatefulSet
+  - Estimated increase in size: 
+    - New field in DaemonSet, StatefulSet spec about 4 bytes
+    - Add new condition in DaemonSet, StatefulSet status


nit: conditions

are we planning on having a shared type for the condition types or is this going to be a naming convention?

can we start using v1.Condition for DaemonSet and StatefulSet ? What about Deployments?

atiratree · 2021-07-22T21:30:01Z

keps/sig-apps/2804-consolidate-workload-controllers-status/README.md

+
+Consider including folks who also work outside the SIG or subproject.
+-->
+We are proposing a new field called `.spec.progressDeadlineSeconds` to both DaemonSet and StatefulSet whose default value will be set to the max value of int32 (i.e. 2147483647) by default, which means "no deadline". In this mode, DaemonSet and StatefulSet controllers will behave exactly like its current behavior but with no `Failed` mode as they're either `Progressing` or `Complete`. This is to ensure backward compatibility with current behavior. We will default to reasonable values for both the controllers in a future release. Since a DaemonSet can make progress no faster than "healthy but not ready nodes", the default value for `progressDeadlineSeconds` can be set to 30 minutes but this value can vary depending on the node count in the cluster. The value for StatefulSet can be longer than 10 minutes since it also involves provisioning storage and binding. This default value can be set to 15 minutes in case of StatefulSet.


please can you elaborate on but this value can vary depending on the node count in the cluster. ?

atiratree · 2021-07-22T21:46:12Z

keps/sig-apps/2804-consolidate-workload-controllers-status/README.md

+
+- All of the replicas associated with the DaemonSet or StatefulSet have been updated to the latest version you've specified, meaning any updates you've requested have been completed.
+- All of the replicas associated with the DaemonSet or StatefulSet are available.
+- No old replicas for the DaemonSet or Stateful are running.


is the behaviour similar to Deployment's Available? What happens when scaling down?

ehashman

/assign
for PRR

keps/sig-apps/2804-consolidate-workload-controllers-status/README.md

ehashman · 2021-09-01T18:40:01Z

keps/sig-apps/2804-consolidate-workload-controllers-status/README.md

+-->
+
+- [x] Feature gate (also fill in values in `kep.yaml`)
+  - Feature gate name: DaemonSetConditions, StatefulSetConditions


Are there two new feature flags? Why?

There should be just one which will apply to all controllers.

I think the idea behind this is to have better granularity for progressDeadlineSeconds in DaemonSet and StatefulSet since this change could be unexpected to some users and they might want to opt in only for one of these.

ehashman · 2021-09-01T18:40:19Z

keps/sig-apps/2804-consolidate-workload-controllers-status/README.md

+    - kube-apiserver
+
+###### Does enabling the feature change any default behavior?
+The default behavior won't change but the new conditions will be additive.


Please elaborate on what this means.

added more details

ehashman · 2021-09-01T18:40:41Z

keps/sig-apps/2804-consolidate-workload-controllers-status/README.md

+
+###### What happens if we reenable the feature if it was previously rolled back?
+
+The DaemonSet, StatefulSet controller sstarts respecting the `progressDeadlineSeconds` again 


Suggested change

The DaemonSet, StatefulSet controller sstarts respecting the `progressDeadlineSeconds` again

The DaemonSet, StatefulSet controller starts respecting the `progressDeadlineSeconds` again.

ehashman · 2021-09-01T18:41:29Z

keps/sig-apps/2804-consolidate-workload-controllers-status/README.md

+- DaemonSet
+- StatefulSet


This seems to be written as though this list could be expanded. Is that a possibility? If not, I think we should be clear about the scope.

This is the initial set of controllers, we do plan to expand and put this as recommendation for all controllers.

added additional workloads

ehashman · 2021-09-01T18:42:53Z

keps/sig-apps/2804-consolidate-workload-controllers-status/README.md

+change are understandable. This may include API specs (though not always
+required) or even code snippets. If there's any ambiguity about HOW your
+proposal will be implemented, this is the place to discuss them.
+-->


Please fill this section out.

ehashman · 2021-09-01T18:43:07Z

keps/sig-apps/2804-consolidate-workload-controllers-status/README.md

+
+### Test Plan
+Unit and E2E tests will be added to cover the
+API validation, behavioral change of DaemonSet and StatefulSet with feature gate enabled and disabled.


What scenarios will be covered?

added more details

ehashman · 2021-09-01T18:44:13Z

keps/sig-apps/2804-consolidate-workload-controllers-status/README.md

+###### Will enabling / using this feature result in any new API calls?
+Yes
+  - Update StatefulSet, DaemonSet status
+  - StatefulSet, DaemonSet controllers make these calls


When and why?

ehashman · 2021-09-01T18:45:17Z

keps/sig-apps/2804-consolidate-workload-controllers-status/README.md

+-->
+- Upgrades
+ When upgrading from a release without this feature, to a release with 
+ `progressDeadlineSeconds`, we will set `progressDeadlineSeconds` to max value of int32 (i.e. 2147483647). This would give users 


I'm not a fan of making the max value have a special meaning... this doesn't seem to be in line with API standards. Are there other possible approaches?

Since the default value for progress deadline will be in 10s of minutes, upgrading to this default value or 0 could break users. Setting this to int max is a safer option as it is effectively an infinity and thus keeps the old behaviour for these workloads.

I suppose there could be some switch for this, but this would bring more complexity and confusion to the users in my opinion.

ehashman · 2021-09-01T18:46:04Z

keps/sig-apps/2804-consolidate-workload-controllers-status/README.md

+
+###### Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs?
+
+No.


This is changing the behaviour of core workload controllers so I think this is a possibility and should be discussed.

Agreed, although I don't expect additional API calls. The goal is to expand the current paths with additional/unified conditions.

@ehashman added additional info

k8s-triage-robot · 2021-12-27T21:29:50Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle stale
Mark this issue or PR as rotten with /lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

atiratree · 2022-01-06T20:31:11Z

/remove-lifecycle stale

soltysh · 2022-01-28T12:47:58Z

keps/sig-apps/2804-consolidate-workload-controllers-status/kep.yaml

@@ -0,0 +1,45 @@
+title: Consolidate Workload controllers life cycle state


Suggested change

title: Consolidate Workload controllers life cycle state

title: Consolidate Workload controllers life cycle status

soltysh · 2022-01-28T12:48:07Z

keps/sig-apps/2804-consolidate-workload-controllers-status/kep.yaml

+# The most recent milestone for which work toward delivery of this KEP has been
+# done. This can be the current (upcoming) milestone, if it is being actively
+# worked on.
+latest-milestone: "v1.23"


Update milestones to 1.24

soltysh · 2022-01-28T12:50:14Z

keps/sig-apps/2804-consolidate-workload-controllers-status/kep.yaml

+# The following PRR answers are required at alpha release
+# List the feature gate name and the components for which it must be enabled
+feature-gates:
+ - name: DaemonSetConditions, StatefulSetConditions


I'd keep a single feature flag, that will be sufficient. We don't need and want that granularity even if the implementation will happen gradually, and not all controllers will be done in the first alpha.

Started using ExtraWorkloadConditions. Please let me know if you can come up with a better name.

ExtendedWorkloadConditions is another alternative, but yours sounds equally well.

that sounds a bit better, changed

soltysh · 2022-01-28T12:50:31Z

keps/sig-apps/2804-consolidate-workload-controllers-status/kep.yaml

+ - name: DaemonSetConditions, StatefulSetConditions
+   components:
+     - kube-controller-manager
+     - kube-apiserver


Why kube-apiserver, I think this will be only on the KCM side.

removed, but unsure. We might need it when downgrading/upgrading for validating conditions?

conditions are not validated

soltysh · 2022-01-28T12:52:21Z

keps/sig-apps/2804-consolidate-workload-controllers-status/README.md

@@ -0,0 +1,912 @@
+<!--


Drop the comments where you don't need them, they make reading the text harder, only left those that will be relevant for beta and GA.

done, should I remove also the empty sections?

I still see some comments you can drop, but that's a minor thing.

dropped other comments

soltysh · 2022-01-28T13:13:29Z

keps/sig-apps/2804-consolidate-workload-controllers-status/README.md

+- No old or mischeduled replicas/pods for the DaemonSet, ReplicaSet or Stateful are running.
+
+#### Failed
+In order to introduce this condition we need to come up with a new field called `.spec.progressDeadlineSeconds` which denotes the number of seconds the controller waits before indicating(in the workload controller status) that the controller progress has stalled.


MinReadySeconds for RS/RC should do that trick.

please see #2833 (comment)

keps/sig-apps/2804-consolidate-workload-controllers-status/README.md

soltysh · 2022-01-28T13:16:04Z

keps/sig-apps/2804-consolidate-workload-controllers-status/README.md

+Kubernetes marks a Job as `waiting` if one of the following conditions is true:
+
+- There are no Pods with phase `Running` and there is at least one Pod with phase `Pending`.
+- The Job is suspended.


IMO Waiting and Suspended are two distinct statuses. The former is awaiting for resources on the cluster to work, the other is explicit desire of the user to stop the work.

yup, makes sense - the running distinction should be enough

soltysh · 2022-01-28T13:16:51Z

keps/sig-apps/2804-consolidate-workload-controllers-status/README.md

+
+Consider including folks who also work outside the SIG or subproject.
+-->
+We are proposing a new field called `.spec.progressDeadlineSeconds` to DaemonSet and StatefulSet whose default value will be set to the max value of int32 (i.e. 2147483647) by default, which means "no deadline".


Since this was mentioned as a non-goal I'd split this into a separate proposal.

ack, I suppose it will be more manageable, but it will split splinter the effort into multiple places. I left most of the progressDeadline content here for the context while acknowledging it will be done in a separate KEP

keps/sig-apps/2804-consolidate-workload-controllers-status/README.md

ehashman · 2022-02-01T00:12:36Z

Waiting for @soltysh's comments to be addressed, then I'll do a final PRR review.

ravisantoshgudimetla

/lgtm

Thank you for working on this @atiratree

k8s-ci-robot · 2022-02-01T20:03:21Z

@ravisantoshgudimetla: you cannot LGTM your own PR.

In response to this:

/lgtm

Thank you for working on this @atiratree

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

ravisantoshgudimetla · 2022-02-01T20:04:36Z

@ehashman - Can you do PRR

soltysh

/lgtm

soltysh

/lgtm

keps/sig-apps/2804-consolidate-workload-controllers-status/README.md

atiratree · 2022-02-03T16:13:56Z

@ehashman thanks. Hopefully resolved, please take a look

ehashman

I have one final question, otherwise this LGTM for PRR.

ehashman · 2022-02-03T18:39:38Z

keps/sig-apps/2804-consolidate-workload-controllers-status/README.md

+In case feature gate is disabled, the workload controllers will remove the stale conditions.
+In case the feature is missing in the workload controllers the conditions will not be removed and become stale.


I would expect the behaviour to be the same if either the feature is not implemented (old version) or the feature gate is disabled. Otherwise, we will have changed default behaviour.

Can you confirm if this is the case? Are stale conditions already removed without this feature, or is this a net new change?

@ehashman Unfortunately we checked the controllers and most of them currently don't do updates of the conditions in the status. So it is not easy to ensure their removal.

I think we don't want removal, as that would change default behaviour.

ehashman

/approve
I think the final PRR clarification can be addressed in a follow-up PR.

k8s-ci-robot · 2022-02-03T21:26:20Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ehashman, ravisantoshgudimetla, soltysh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~keps/prod-readiness/OWNERS~~ [ehashman]
~~keps/sig-apps/OWNERS~~ [soltysh]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ehashman · 2022-02-03T21:27:25Z

Oh this needs an LGTM too

/lgtm

atiratree · 2022-02-03T22:48:11Z

We can take a look at it, but I am not sure about the solution atm. Thanks for the approve

atiratree · 2022-02-04T13:34:20Z

@ehashman following up in #3211

Consolidate workload controllers status

4d0a1df

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Jul 22, 2021

k8s-ci-robot requested review from kow3ns and mattfarina July 22, 2021 02:11

k8s-ci-robot added sig/apps Categorizes an issue or PR as relevant to SIG Apps. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jul 22, 2021

atiratree reviewed Jul 22, 2021

View reviewed changes

ehashman reviewed Sep 1, 2021

View reviewed changes

k8s-ci-robot assigned ehashman Sep 1, 2021

kevindelgado mentioned this pull request Sep 8, 2021

Consolidate workload controller status #2804

Open

4 tasks

fix some PRR feedback

2bca996

atiratree force-pushed the consolidate-status branch from ce32a07 to 3fde61b Compare September 28, 2021 20:39

Consolidate workload controllers status: add new conditions + analysis

83ec184

atiratree force-pushed the consolidate-status branch from 3fde61b to 83ec184 Compare September 28, 2021 20:52

atiratree mentioned this pull request Oct 4, 2021

REQUEST: New membership for atiratree kubernetes/org#3026

Closed

7 tasks

soltysh mentioned this pull request Nov 26, 2021

ProgressDeadlineExceeded not set outside of Deployment rollouts kubernetes/kubernetes#106054

Closed

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Dec 27, 2021

k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 6, 2022

add notes about misunderstandings

cf6306d

atiratree force-pushed the consolidate-status branch from 5a9b174 to cf6306d Compare January 24, 2022 15:41

soltysh reviewed Jan 28, 2022

View reviewed changes

ravisantoshgudimetla changed the title ~~[WIP] Consolidate workload controllers status~~ Consolidate workload controllers status Jan 31, 2022

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 31, 2022

review update

da9573a

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed lgtm "Looks good to me", indicates that a PR is ready to be merged. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Feb 1, 2022

review update

23afbb0

atiratree force-pushed the consolidate-status branch from b3811ae to 23afbb0 Compare February 1, 2022 19:17

ravisantoshgudimetla commented Feb 1, 2022

View reviewed changes

soltysh approved these changes Feb 2, 2022

View reviewed changes

k8s-ci-robot added lgtm "Looks good to me", indicates that a PR is ready to be merged. and removed lgtm "Looks good to me", indicates that a PR is ready to be merged. labels Feb 2, 2022

soltysh approved these changes Feb 2, 2022

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 2, 2022

ehashman reviewed Feb 3, 2022

View reviewed changes

update PRR review

59ea97c

atiratree force-pushed the consolidate-status branch from a06fa4c to 59ea97c Compare February 3, 2022 16:10

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 3, 2022

ehashman reviewed Feb 3, 2022

View reviewed changes

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 3, 2022

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 3, 2022

k8s-ci-robot merged commit 6151861 into kubernetes:master Feb 3, 2022

k8s-ci-robot added this to the v1.24 milestone Feb 3, 2022

atiratree mentioned this pull request Feb 4, 2022

update KEP: consolidate workload controllers status #3211

Merged

atiratree mentioned this pull request Sep 19, 2022

KEP-2804: Consolidate workload controllers status - introduce Operational condition #3524

Closed


		See the [Kubernetes API Conventions](https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#typical-status-properties) for more information on status conditions

		Because of the number of changes that are involved as part of this effort, we are thinking of a phased approach where we introduce these conditions to DaemonSet controller behind a featuregate flag first in one release and then make similar changes to StatefulSet controller. This also needs some code refactoring of existing conditions for Deployment controller.


		###### What happens if we reenable the feature if it was previously rolled back?

		The DaemonSet, StatefulSet controller sstarts respecting the `progressDeadlineSeconds` again

	The DaemonSet, StatefulSet controller sstarts respecting the `progressDeadlineSeconds` again
	The DaemonSet, StatefulSet controller starts respecting the `progressDeadlineSeconds` again.


		###### Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs?

		No.

		@@ -0,0 +1,45 @@
		title: Consolidate Workload controllers life cycle state

		In case feature gate is disabled, the workload controllers will remove the stale conditions.
		In case the feature is missing in the workload controllers the conditions will not be removed and become stale.

Consolidate workload controllers status #2833

Consolidate workload controllers status #2833

Conversation

ravisantoshgudimetla commented Jul 22, 2021

ravisantoshgudimetla commented Jul 22, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ehashman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

k8s-triage-robot commented Dec 27, 2021

atiratree commented Jan 6, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ehashman commented Feb 1, 2022

ravisantoshgudimetla left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Feb 1, 2022

ravisantoshgudimetla commented Feb 1, 2022 • edited Loading

soltysh left a comment

Choose a reason for hiding this comment

soltysh left a comment

Choose a reason for hiding this comment

atiratree commented Feb 3, 2022

ehashman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ehashman left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Feb 3, 2022

ehashman commented Feb 3, 2022

atiratree commented Feb 3, 2022

atiratree commented Feb 4, 2022

ravisantoshgudimetla commented Feb 1, 2022 •

edited

Loading