Only KeystoneAPI finalizer from sub-resources when actually needed #401
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We are seeing the cleanup of certain KUTTL tests hang. This happens because
KeystoneAPI
fails to fully delete when multiple service CRs (Nova, Glance, Placement, etc) are being deleted at the same time. This is more likely to occur in operators that have a lot of other operator dependencies in their KUTTL testing (for instance, Octavia).KeystoneService
is used in the enumeration below, but this also applies forKeystoneEndpoint
s.KeystoneService
resource for all the OSP service CRs is being requeued one more time after it has removed its own self-finalizer and is then waiting to be fully deleted.KeystoneService
from the cache and etcd at the same time as that object has been requeued for one more read.KeystoneService
adds a finalizer back ontoKeystoneAPI
(keystone-operator/controllers/keystoneservice_controller.go
Lines 191 to 202 in e5f8627
KeystoneAPI
, you can hit the"object has been modified"
error if other threads for otherKeystoneService
s are also modifying theKeystoneAPI
.KeystoneService
is going to be completely gone at the end of the reconcile (because its self-finalizer was removed in the last reconcile and won't be removed in this one, hence it won't cause a requeue and k8s will completely wipe it from etcd) and yet it has left a finalizer onKeystoneAPI
, forever preventing the deletion of theKeystoneAPI
.To prevent this problem, we are moving the
Update()
to add a finalizer to theKeystoneAPI
in theKeystoneService
andKeystoneEndpoint
reconcile loops to a location that only executes when that finalizer is actually needed. Beforehand we were potentially always adding the finalizer toKeystoneAPI
even if theKeystoneService
orKeystoneEndpoint
had no need of it anymore (i.e. during deletion, after user/service/endpoint had been removed from the OSP Keystone database).