Manual remediation of Machines

[fabriziopandini]

What would you like to be added (User Story)?

As an operator, I would like to be able to recreate single machines in a controlled way

Detailed Description

The traditional way to re-create a single Machine is to delete it manually.

However, directly deleting a machine has some well-known downsides:

• Once the operation is done, it cannot be reverted
• The operation is "low level", it skips several system safeguards and this can be risky especially when you delete control plane nodes (see discussion in Restrict KCP Machine deletion #9919). More speifically
  • With manual deletion, all the checks about max unhealthy, are implicitly skipped
  • With manual deletion, all the checks about the fact the machine can be safely remediated, are implicitly skipped

This proposal is about introducing a new annotation (name TBD) to be applied to a machine that the operator wants to safely delete (and thus to leave "low level" deletion only as the ultimate escape patch).

This annotation will be processed by the MHC controller in healthCheckTargets, and this the machine will be included in the pool of machines to be considered for remediation.

MHC, will then determine if there are the conditions to remediate such a machine, and the remediation with then be taken charge of by the owning controller or by the external remediation tool.

The remediation owner will then perform another round of checks, and finally transform the manual remediation in an actual machine deletion.

Anything else you would like to add?

Once this is in place, we can eventually think of strategies to give higher priority for deletion to machines being remediated/manually remediated, in case more machines are being deleted at the same time.
However this is less important than providing a safe path to deletion (and thus IMO not blocking for a first iteration)

Label(s) to be applied

/kind feature
/area machinehealthcheck
/triage accepted

[chrischdi]

/assign

[Levi080513]

Nice idea!!!
It would be better if we could support this feature in clusterctl, like kubectl drain.

[sbueringer]

I think we could consider additionally implementing a clusterctl command. But in any case this should be implemented in our controllers so other clients which are not clusterctl can also use the feature.