[Question] Does Operator-SDK have some sort of "don't reconcile this" concept? #3418
Labels
kind/feature
Categorizes issue or PR as related to a new feature.
lifecycle/stale
Denotes an issue or PR has remained open with no activity and has become stale.
needs discussion
Milestone
Type of question
Are you asking about community best practices, how to implement a specific feature, or about general context and help around the operator-sdk?
community best practices
Question
Currently I am looking into the concept of a pause reconciliation (further specified in the proposal below), a way for a custom resource to be flagged as do not reconcile me to allow for maintenance and manual intervention. Does this concept exist witthin the operator-sdk framework, or in Kubernetes as a whole? I have looked into this and found little on the topic. Was hoping for advice and to start a wider conversation around this if not already existent.
Additional context
I put together a proposal of sorts internally for my company and wanted to share it, in case additional context is needed:
Pause custom resource reconcile proposal
A user of an operator might want to pause their operator's function for multiple reasons, this could include:
The
spec.pause
field was added to prometheus some time ago: prometheus-operator/prometheus-operator@b8e650bis the behaviour they currently have in code.
I propose ew adopt a model similar to this, though semantically I think retaining any functionality aside from the 'delete' reconciliation action is potentially dangerous on pause.
To expand on this, I think the operator should still do any 'clean-up' operations relating to an actual deletion of a custom resource, but it should not remove a component if that component is no longer specified in the spec of a custom resource.
An example of this in Event Streams is if
spec.pause
is true when you delete aKafka
custom resource, the operator will still delete your PVCs, ifspec.kafka.storage.deleteClaim
is set to true, so that persistent claims are deleted on custom resource deletion.If a custom resource is deleted, since so many resources will be using owner references, it must continue to use deletion logic, even if paused. A pause should not pause deletion logic, the pause is no longer in effect when you delete the custom resource, since the field is actually no longer present (along with the rest of the custom resource).
A user could potentially put the instance in a weird state where bespoke deletion logic would not be able to work, but this is user error and deletion logic should tolerate some elements of failure during deletion.
For the most part deletion logic is to be avoided for operators anyway as owner references are a convenient and simple solution to cleanup.
The advantage of such a mechanism is that when an operator managing an instance would normally have to be uninstalled, instead the operator can continue operation,
but the 'paused' instance will no longer be managed.
Proposal
The exact model I propose is as follows:
spec.pause
is added as a boolean field to all custom resources.If 'paused' a custom resource will exit reconcile before any interactions are done with the application resources, application or related applications, the one exception to this would be the addition of a new status condition that indicates it is
Paused
- or anUnknown
status condition to indicate the operator has no way of verifying the instance state.This means operator validation requires the field is false or not present.
It is not checked on web-hook validation as the
spec.pause
field being true is a valid custom resource state, but on validation within the operator, a custom resource havingspec.pause
true denotes it is paused and that the reconcile should end (successfully) with a pause stateReconciles will still be triggered on custom resource changes and periodic reconciles if configured within the operator, but will drop out of reconcile loop.
Operator will write a
Paused
phase or condition to the custom resource to make clear it is unmanaged.Once manual intervention is finished, user deletes the pause field, or sets the field to false. This will trigger a reconcile and operator will continue normal operations.
Alternative names for spec.pause
spec.pause
may not be the best name, alternatives could be:pauseReconcilliation
skipReconcilliation
unmanaged
ignored
ignoreChanges
The text was updated successfully, but these errors were encountered: