📖 KCP remediation proposal #3676
Conversation
k8s-ci-robot
added
the
cncf-cla: yes
k8s-ci-robot
added
the
size/L
|
|
||
| - KCP remediation is triggered by the MachineHealthCheck controller marking a machine for remediation. see | ||
| [machine-health-checking proposal](https://github.com/kubernetes-sigs/cluster-api/blob/11485f4f817766c444840d8ea7e4e7d1a6b94cc9/docs/proposals/20191030-machine-health-checking.md) | ||
| for additional details. When there are multiple machines that are marked for remediation, the oldest one will be remediate first. |
There was a problem hiding this comment.
Is this true in any case? What if the oldest one shouldn't be remediated because it could impact quorum?
There was a problem hiding this comment.
I agree this is not optimal, but this is by far the simplest solution and so I would advocate this is an acceptable solution for the first iteration on KCP remediations, unless there are evidences that multiple remediations with concurrent sparse etcd failures is a frequent use cases.
However, if during the implementation we find a smarter way to determine the machine to remediate, I will more than happy to amend this proposal ....
| @@ -101,6 +109,9 @@ for the default implementation would not preclude the use of alternative control | |||
| - To manage control plane deployments across failure domains. | |||
| - To support user-initiated remediation: | |||
| E.g. user deletes a Machine. Control Plane Provider reconciles by removing the corresponding etcd member and updating related metadata | |||
| - To support auto remediation triggered by MachineHealthCheck objects: | |||
There was a problem hiding this comment.
Since MachineHealthCheck only looks at Node conditions, may not capture all the things to understand if a control-plane machine needs remediation.
should this be more generic?
There was a problem hiding this comment.
I kind of wish it was a little smarter, but maybe we can split control plane vs worker nodes health checker later in v1alpha4 after #3547
There was a problem hiding this comment.
I agree, but IMO in order to address this we should extend MHC to observe machine conditions, so we could benefit on CAPI conditions for worker nodes as well
|
/lgtm |
k8s-ci-robot
added
the
lgtm
|
@fabriziopandini squash? |
fabriziopandini
force-pushed
the
kcp-remediation-proposal
branch
from
8ab8d17
to
0d91562
Compare
k8s-ci-robot
removed
the
lgtm
|
/test pull-cluster-api-test |
|
/lgtm |
k8s-ci-robot
added
the
lgtm
|
/approve |
|
/milestone v0.3.10 |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: vincepri The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
the
approved
What this PR does / why we need it:
This PR updates the KCP document by introducing support for automatic remediation of unhealthy control-plane machines.
Kudos and credits to @benmoss who kicked off this effort and laid the ground for this PR with https://docs.google.com/document/d/1hJza3X-XbVV_yczB5N6vXbl_97D0bOVQ0OwGovcnth0/edit
Which issue(s) this PR fixes:
Rif #2976