KEP-4212: Declarative Node Maintenance #4213
Conversation
k8s-ci-robot
added
the
cncf-cla: yes
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: atiratree The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
kind/kep
k8s-ci-robot
added
the
size/XXL
k8s-ci-robot
added
the
do-not-merge/work-in-progress
keps/sig-apps/4212-improve-node-maintenance/node-maintenance.svg
Outdated
Show resolved
Hide resolved
atiratree
force-pushed
the
improve-node-maintenance
branch
7 times, most recently
from
0ed835a
to
81b4589
Compare
atiratree
force-pushed
the
improve-node-maintenance
branch
2 times, most recently
from
6194058
to
00dc7c5
Compare
atiratree
changed the title
atiratree
force-pushed
the
improve-node-maintenance
branch
from
00dc7c5
to
686227c
Compare
Not sure yet, the API itself is SIG Node I guess, but it has a lot of implications for SIG- Apps. Let's delay this decision after I redo the KEP and we have additional rounds of discussions. |
atiratree
force-pushed
the
improve-node-maintenance
branch
from
a15b78d
to
0658ae8
Compare
k8s-ci-robot
added
cncf-cla: yes
atiratree
force-pushed
the
improve-node-maintenance
branch
from
0658ae8
to
00679e9
Compare
|
| One of the problems is that it would be difficult to get real word adoption and thus important | ||
| feedback on this feature. This is mainly due to the requirement that the feature be implemented | ||
| and integrated with multiple components to observe the benefits. And those components are both | ||
| cluster admin and application developer facing. |
There was a problem hiding this comment.
Using the Core APIs to get adoption is not a good argument, if we are assuming there is appetite, official CRDs will be equal desired for the users ... adding experimental features intersecting with the Pod and Node lifecycle than later does not get adoption is a big penalty for the project
There was a problem hiding this comment.
I have changed the section to better reflect the current state of the KEP. Mainly the technical reasons why we prefer the core API. I have kept the adoption part there, but slimmed it down. The adoption was there from the time we had the Evacuation API in the KEP, but since we moved it to a separate KEP, it is not as important anymore.
There was a problem hiding this comment.
The KEP still lacks graduation criteria, and tests, and PRR. We can condition GA graduation on a reasonable adoption, to see if this feature brings a general benefit to the k8s project.
atiratree
force-pushed
the
improve-node-maintenance
branch
from
00679e9
to
063b4bf
Compare
| - Solve the node lifecycle management or automatic shutdown after the node drain is completed. | ||
| Implementation of this is better suited for other cluster components and actors who can use the | ||
| node maintenance as a building block to achieve their desired goals. | ||
| - Synchronize all pod termination mechanisms (see #8 in the [Motivation](#motivation) section), so that they do |
There was a problem hiding this comment.
This require to synchronize the Endpoints/Services too , after a quick skim to the KEP I wonder if this approach will not hit the same problem @smarterclayton is describing here kubernetes/kubernetes#116965 (comment), the Pods should report the status to avoid traffic to be sent to them if they are not ready
There was a problem hiding this comment.
As this is a non goal I think the KEP can comfortably find an easy way to not deliver it.
There was a problem hiding this comment.
- This KEP does not propose to solve the non-goals nor does it suggest how or whether to implement them. It is a simply a list of things that could be done.
- Even the goals section expects additional KEPs on the node and apps side to fully examine the problem. But for the sake of simplicity, I keep the NodeMaintenanceAwareDaemonSet and NodeMaintenanceAwareKubelet features here for now.
Regarding what @smarterclayton wrote, I think we have a lot of room to solve that. By utilizing both the NodeMaintenance and Evacuation API and deciding which parts should be mandatory and which optional.
| A misconfigured .spec.nodeSelector could select all the nodes (or just all master nodes) in the | ||
| cluster. This can cause the cluster to get into a degraded and unrecoverable state. | ||
|
|
||
| An admission plugin ([NodeMaintenance Admission](#nodemaintenance-admission)) is introduced to |
There was a problem hiding this comment.
But this will be unrecoverable, right? if the master nodes are not available it means will you not be able to connect to the apiserver
There was a problem hiding this comment.
The admission restriction wouldn't prevent you marking all the nodes as under maintenance. It'd just warn you that this was a bad idea.
(I'd prefer to have a ValidatingAdmissionPolicy rather than code, but we don't have lifecycle management for this kind of thing. In other words, kubeadm can't deploy recommended ValidatingAdmissionPolicies for us, so we shouldn't assume they are in place.)
- You can mark all the nodes as being covered by maintenance (and this may well be true!)
- If you as a cluster adminstrator then go and pull out the power cords or whatever, you can try to recover by plugging them back in.
- Static Pods don't stop during a node drain, right? So even if you do mark every node as under maintenance, you may still have a control plane. Perhaps, no working cluster network and no DNS, but you still have a control plane.
There was a problem hiding this comment.
Yes, it can still be a valid strategy. Assuming your control plane is deployed correctly - basically that it shuts down last or runs on the side (invisible to kube).
|
|
||
| List of cases where the current solution is not optimal: | ||
|
|
||
| 1. Without extra manual effort, an application running with a single replica has to settle for |
There was a problem hiding this comment.
I am super confused by this. I mean, the answer is "yes". If you have 1 replica, taking that replica down will cause a downtime.
There was a problem hiding this comment.
Some applications have a general ability (e.g. deployments) to scale up first. So they do not have to experience the downtime if configured correctly. This can be generalized to any number of replicas. There are more details in the linked issues.
|
|
||
| ## Proposal | ||
|
|
||
| Most of these issues stem from the lack of a standardized way of detecting a start of the node |
There was a problem hiding this comment.
@jpbetz - this feels like another case for CRX :)
| // +required | ||
| NodeSelector *v1.NodeSelector | ||
|
|
||
| // The order of the stages is Idle -> Cordon -> Drain -> Complete. |
There was a problem hiding this comment.
Be really careful with this language. The way this is specced you have a state machine, which means any futre evolution will be very difficult.
Is it allowable for a cordon'ed node to become idle? This state machine doesn't allow it. I think you'd do better to describe this as a set of mutually exclusive intentions. Modes.
There was a problem hiding this comment.
It is hard to offer this as modes, as it should be possible to manually switch between the states/stages. And also to have a strict transitions between them (please see Supported Stage Transitions) as not all of them are valid. It is up to the admin or a controller when these transitions should occur.
We also want to have a lifecycle for the NodeMaintenance. Once the NM completes we can clean up and recover the node. If an additional NM is required, a new object should be created.
I would be happy to hear ideas on how to improve this.
If we decide for the current solution:
- It may be possible to add new stages into the API.
- The extensibility can be offered through a set of modifiers for each stage, as we do with the
DrainPlan.
| cluster, the admin has to make a potentially harmful decision. | ||
|
|
||
| From an application owner or developer perspective, the only standard tool they have is | ||
| a PodDisruptionBudget. This is sufficient in a basic scenario with a simple multi-replica |
There was a problem hiding this comment.
Do we have a list of all PDB issues somewhere?
There was a problem hiding this comment.
I think the main ones are in the KEP, but I can add the rest of them for the context .
There was a problem hiding this comment.
I have added the list of PDB issues to the Evacuation API KEP: https://github.com/atiratree/kube-enhancements/blob/evacuation-api/keps/sig-apps/4563-evacuation-api/README.md#poddisruptionbudget-standing-issues
| from one node to another. This has been discussed in the issue [kubernetes/kubernetes#66811](https://github.com/kubernetes/kubernetes/issues/66811) | ||
| and in the issue [kubernetes/kubernetes#114877](https://github.com/kubernetes/kubernetes/issues/114877). | ||
| 2. Similar to the first point, it is difficult to use PDBs for applications that can have a variable | ||
| number of pods; for example applications with a configured horizontal pod autoscaler (HPA). These |
There was a problem hiding this comment.
The two possible alternative would be:
- extend HPA to update associated PDB to maintain the required minimum?
- every application should have a reasonable minimum which would allow the application to survive any kind of disruption (planned or unplanned).
There was a problem hiding this comment.
extend HPA to update associated PDB to maintain the required minimum?
- This is not possible when n=1, which is the core of the Allow scaling up to meet PDB constraints kubernetes#93476 issue.
- Although it is possible for n>1, I would be afraid of conflicts when introducing such feature. PDB was not introduced as a controlled/changing object like for example ReplicaSets. It is set by operators/controllers, manually or as part of various devops flows.
- There is no backlink from pod/HPA to a PDB. Even if we introduce such a backlink, there are still problems. What do we do when the PDB does not exist and there is one specified by a HPA. Should it be the responsibility of the HPA to become an operator of a PDB (or become other type of a PDB)? We could run again into the conflicts with existing flows as seen in the previous point.
- There is no mapping from a PDB to an application, there is only a selector. Although not recommended, there could be two applications under one PDB. One controlled by HPA and the other one not. Or any combination of these.
every application should have a reasonable minimum which would allow the application to survive any kind of disruption (planned or unplanned).
This is not always possible/desired, as seen in kubernetes/kubernetes#66811, kubernetes/kubernetes#114877 and kubernetes/kubernetes#93476
| 2. Similar to the first point, it is difficult to use PDBs for applications that can have a variable | ||
| number of pods; for example applications with a configured horizontal pod autoscaler (HPA). These | ||
| applications cannot be disrupted during a low load when they have only pod. However, it is | ||
| possible to disrupt the pods during a high load without experiencing application downtime. If |
There was a problem hiding this comment.
I doubt any admin is interested in node maintenance during high load, unless in a critical situation, in which case I'm not sure any of the considerations from this document might apply, because you're focusing on bringing the cluster back to living.
There was a problem hiding this comment.
High load (of the application) is just another way of saying that the HPA has decided to scale beyond 1 pod. That does not have to correlate to the high load of the cluster.
| This can also happen with other mechanism (e.g., different types of evictions). This has been | ||
| discussed in the issue [kubernetes/kubernetes#124448](https://github.com/kubernetes/kubernetes/issues/124448) | ||
| and in the issue [kubernetes/kubernetes#72129](https://github.com/kubernetes/kubernetes/issues/72129). | ||
| 9. There is not enough metadata about why the node drain was requested. This has been discussed in |
There was a problem hiding this comment.
A lot of the above points nicely collected issues with Graceful Node Shutdown, which is being worked on as part of #2000. Have you considered pointing the authors to this list?
There was a problem hiding this comment.
There has been a cooperation/discussion with sig-node. I have discussed this with @kwilczynski, who is interested in graduating the Graceful Node Shutdown feature.
Many of the problems described are impossible or impractical to be solved by the kubelet alone. This is one of the main goals of this KEP to help resolve this situation.
| not possible to rely solely on this feature as a go-to solution for graceful node and workload | ||
| termination. | ||
|
|
||
| - The Graceful Node Shutdown feature is not application aware and may prematurely disrupt workloads |
There was a problem hiding this comment.
None of the solutions will be ever application aware, that is one of the reasons we have operators per a specific application, which know how to deal with all the intricacies of managing it.
There was a problem hiding this comment.
Yes, I am only stating the fact. Not implying that it should be. The point of this KEP is to freely share the intentions of draining a node and each pod so that each actor (controller, application, user, tools, etc.) can react in a proper way.
|
|
||
| To sum up. Some applications solve the disruption problem by introducing validating admission | ||
| webhooks. This has some drawbacks. The webhooks are not easily discoverable by cluster admins. And | ||
| they can block evictions for other applications if they are misconfigured or misbehave. The |
There was a problem hiding this comment.
Misconfigured webhooks are generally a problem for any kind of cluster operations, and as such I don't think this makes a good argument against it. There's an explicit cautious warning in the official documentation of them https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#experimenting-with-admission-webhooks. Also, "With great power comes great responsibility" 😉
There was a problem hiding this comment.
Some of the solutions today ship with these webhooks, so they are not misconfigured per se. They are part of the cluster operation. As I understand it, this was done to implement safety features that are missing in kubernetes today.
The downside is mostly the observability and breaking the eviction contract (#4213 (comment)). Also, if things go wrong these webhooks will block the node drain, which is a headache for cluster administrators.
| progress is lost. The NodeMaintenance implementation should also utilize existing node's | ||
| `.spec.unschedulable` field, which prevents new pods from being scheduled on such a node. | ||
|
|
||
| We will deprecate the `kubectl drain` as the main mechanism for draining nodes and drive the whole |
There was a problem hiding this comment.
With my sig-cli hat, I will call out this wasn't discussed with that group, so before proceeding with that kind of steps you'll need their approval.
There was a problem hiding this comment.
This is mostly symbolic, as the NodeMaintenance API will essentially replace all of its functionality. The command itself does not have to be removed. A warning that there is a better way, might be enough.
Nevertheless, I plan to present this to sig-cli before committing to it.
There was a problem hiding this comment.
I'd imagined that we'd keep kubectl drain and that by default this subcommand would make a NodeMaintenance if the API is available, otherwise fall back to client-side draining.
|
|
||
| We could implement the NodeMaintenance or Evacuation API out-of-tree first as a CRD. | ||
|
|
||
| The KEP aims to solve graceful termination of any pod in the cluster. This is not possible with a |
There was a problem hiding this comment.
I'm seeing this not as an alternative, but rather as a prerequisite for what this document proposes. You're building a solution on top of existing k8s components, so you first need to address the problems in the underlying pieces. I've mentioned that above when asking whether your reported the problems with Node Graceful Shutdown to sig node.
There was a problem hiding this comment.
This KEP would help resolve the issues around Graceful Node Shutdown: #4213 (comment)
| during the node drain. To solve this, integration with NodeMaintenance is required. See | ||
| [DaemonSet Controller](#daemonset-controller) for more details. | ||
|
|
||
| Also, one of the disadvantages of using a CRD is that it would be more difficult to get real-word |
There was a problem hiding this comment.
One of the reasons for introducing CRDs was to allow authors to iterate faster and provide feedback which can:
- improve the core k8s;
- allow to make an informed decision whether a particular problems is worth integrating in core or not.
So far, the several different projects you've mentioned in this document were able to achieve the functionality you're proposing here outside of core.
There was a problem hiding this comment.
I see, I need to emphasize this better in the goals. The kubelet integration is an important part of this KEP.
The other projects are mostly replicating the kubectl drain logic in a controller with some extra. If this were the only part of the KEP, then I would agree that a CRD would be preferable.
There was a problem hiding this comment.
(I think this strengthens the case for prototyping with a CRD and then switching in-tree when we add the kubelet integration).
There was a problem hiding this comment.
Correct, the whole point of out-of-tree experimentation is to shape up the API and identify what has to be done in the core, that is not possible today, when using an external solution.
| - Taints are used as a mechanism for involuntary disruption; to get pods out of the node for some | ||
| reason (e.g. node is not ready). Modifying the taint mechanism to be less harmful | ||
| (e.g. by adding a PDB support) is not possible due to the original requirements. | ||
| - It is not possible to incrementally scale down according to pod priorities, labels, etc. |
There was a problem hiding this comment.
I don't see an alternative discussed in SIG Apps meeting from 4/29 during which Clayton proposed extending PDBs with the missing functionality like:
- backpreassure
- supporting single-replica apps
- others? (I've asked for a list of PDBs problems above).
There was a problem hiding this comment.
These alternatives are discussed in the Evacuation KEP: https://github.com/atiratree/kube-enhancements/blob/improve-node-maintenance/keps/sig-apps/4212-declarative-node-maintenance/README.md#alternatives
This KEP mentions all the issues, but the implementation part is more high-level in this one.
There was a problem hiding this comment.
I have scheduled a discussion for the next sig-apps call to discuss these options.
atiratree
force-pushed
the
improve-node-maintenance
branch
from
063b4bf
to
55b2526
Compare
| `kubectl cordon` and `kubectl uncordon` commands will be enhanced with a warning that will warn | ||
| the user if a node is made un/schedulable, and it collides with an existing NodeMaintenance object. | ||
| As a consequence the node maintenance controller will reconcile the node back to the old value. | ||
| Because of this we can make these commands noop when under the NodeMaintenance. |
There was a problem hiding this comment.
you mean the NodeMaintainance object override the cordon uncordon commands?
There was a problem hiding this comment.
Yes. I suppose it should also be possible to block scheduling on NodeMaintenance presence only.
| `nodemaintenance` admission plugin will be introduced. | ||
|
|
||
| It will validate all incoming requests for CREATE, UPDATE, and DELETE operations. All nodes | ||
| matching the `.spec.nodeSelector` must pass an authorization check for the DELETE operation. |
There was a problem hiding this comment.
are the nodes selected able to delete the NodeMaintainance object?
There was a problem hiding this comment.
For alpha, I wouldn't have the kubelet update .status of NodeMaintenance only; we can add fields into Node status if we want the kubelet to report something per-node that clients need to see.
After alpha we could revisit to decide what write to NodeMaintenance spec should be done by in-tree kubelet code.
There was a problem hiding this comment.
still don't clear what that means this authorization check, is kind of a nodeauthorizer check?
After alpha we could revisit to decide what write to NodeMaintenance spec should be done by in-tree kubelet code.
what is the rationale of this decision? seems an important thing to decide, at least and unresolved block of test should be explaining this and why is unresolved
There was a problem hiding this comment.
For now we expect the kubelet to only create the NodeMaintenance. This can be re-evaluated in the followup kubelet KEPs.
There was a problem hiding this comment.
still don't clear what that means this authorization check, is kind of a nodeauthorizer check?
@aojea This is for security. So it is not enough for a person to have permissions for NodeMaintenances to drain nodes and disrupt workloads. We were asked by sig-node to include this in the KEP.
After alpha we could revisit to decide what write to NodeMaintenance spec should be done by in-tree kubelet code.
I do not see this as a needed feature at this time. The only field that is reasonable to mutate is stage or maybe a reason. IMO it would be better to explore this in the kubelet KEPs.
| StageStatuses []StageStatus | ||
| // List of a maintenance statuses for all nodes targeted by this maintenance. | ||
| NodeStatuses []NodeMaintenanceNodeStatus |
There was a problem hiding this comment.
@thockin I do not like the pattern of using status as a database
how do we offer consistency , is there only one actor managing this?
If the status is wiped out by mistake are we able to rebuild the current state?
There was a problem hiding this comment.
is this going to support 5000 nodes per example?
There was a problem hiding this comment.
If the status is wiped out by mistake are we able to rebuild the current state?
- Kubernetes no longer expects
.statusto be rebuildable - I think it is reconstructable anyway; start from scratch (as if the kubelet had not previously been running)
There was a problem hiding this comment.
is this going to support 5000 nodes per example?
Good question.
We could add fields into Node status if we want the kubelet to report something per-node (and just report it, not using .status to record cluster administrator intent.
We could also add a SomethingSlice API like we did with Endpoints, but that does not feel like a good fit.
There was a problem hiding this comment.
+1, this is problematic:
I am trying to bend the status somehow, but it is not feasible.
We cannot even limit how many nodes the selectors should select on admission, since new nodes can appear during the NodeMaintenance lifetime (especially Idle).
We could introduce a MultiNodeMaintenance object, whose only point would be to create x number of NodeMaintenance objects, one for each node. And it would synchronize the current stage between itself and the children.
But yeah, storing this in the Node status probably sounds like the best solution.
There was a problem hiding this comment.
This should be solved in the current API design.
| When a `Cordon` or `Drain` stage is detected on the NodeMaintenance object, the controller | ||
| will set (and reconcile) `.spec.Unschedulable` to `true` on all nodes that satisfy | ||
| `.spec.nodeSelector`. It should alert via events if too many occur appear and a race to change | ||
| this field is detected. |
There was a problem hiding this comment.
what if there are other NodeMaintainance objects in other stage matching the same Pods and Nodes?
There was a problem hiding this comment.
Not sure about this but one way to handle cordon / uncordon is to label the NodeMaintenances that result from a cordon. kubectl uncordon could then remove any NodeMaintenance labelled as a cordon and matching the one specific node.
We either reject or warn on setting that label on a NodeMaintenance that selects or could select multiple nodes.
There was a problem hiding this comment.
these dependencies and combinations need to be clear or we can provide a big footgun to users
There was a problem hiding this comment.
Drain of a single node by multiple maintenances is supported, please see https://github.com/atiratree/kube-enhancements/blob/improve-node-maintenance/keps/sig-apps/4212-declarative-node-maintenance/README.md#pod-selection-and-draintargets-example
There was a problem hiding this comment.
If there is at least one maintenance with Cordon or Drain stage, the node remains unschedulable.
atiratree
force-pushed
the
improve-node-maintenance
branch
from
55b2526
to
d537832
Compare
|
As it stands, I support https://github.com/kubernetes/enhancements/blob/d537832d421a1ac9bee27a4a086fe6e49e82bb18/keps/sig-apps/4212-declarative-node-maintenance/README.md#out-of-tree-implementation over the proposal made in this KEP. However, we would / might need to make kubectl aware of the custom API, and so would still need a KEP. I can open a PR with that alternative selected if folks are interested. |
|
@atiratree |
|
@tuibeovince it depends on how your control plane is deployed. Ideally it should have the highest priority and pod type. So that all your workloads drain first and control plane last. You can also pull the plug earlier if you do not need the control plane to shut down gracefully. Do you have a special case in mind that you would like to see solved for a single node cluster? |
atiratree
force-pushed
the
improve-node-maintenance
branch
from
d537832
to
4db6894
Compare
|
@atiratree Thank you! and sorry if I might still be getting lost in the concepts (such as evacuation) being introduced within the scope of DNM and EvacuationAPI in general. In my understanding given a multiple worker node cluster, pretty much like drain, DNM will instigate an evacuation of pods in a node to be maintained, and have them evacuated to an available node. I guess for now what I wish to know about is the intended behavior of when the cluster is a single worker node and that node gets maintained. Like how are the pods handled before, during, and after the maintenance is completed in this case? Or are they simply just shutdown? are they gracefully killing the pods? and such Thank you in advance. |
TODO: