-
Notifications
You must be signed in to change notification settings - Fork 235
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bug 1820472: Finalize namespace when it is stuck in 'Terminating' after migration #633
Conversation
@pliurh: This pull request references Bugzilla bug 1820472, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker. 3 validation(s) were run on this bug
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
@pliurh: This pull request references Bugzilla bug 1820472, which is valid. 3 validation(s) were run on this bug
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
da6035d
to
420e623
Compare
Finalize namespace when it is stuck in 'Terminating' after migration. As controller-runtime client has yet support subresources like, using client-go to invoke the Finalize API of Namespace.
420e623
to
87f9d3d
Compare
/test e2e-ovn-hybrid-step-registry |
if err != nil { | ||
return errors.Wrapf(err, "could not create clientset") | ||
} | ||
if _, err := clientset.CoreV1().Namespaces().Finalize(ns); err != nil { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I may be wrong, but I think you can do instead:
if _, err := client.CoreV1().Namespaces().Finalize(ns); err
And avoid instantiating a new client, removing the lines 64-72
Edit: ah no you can't, disregard this comment.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The client
here is not a k8s client-go clientset, but a controller-runtime client which doesn't support the finalize
call of the namespace object. kubernetes-sigs/controller-runtime#573
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's not going to work. He wants to access the Finalize
method, which is not exposed by k8sclient.Client
, but there might be a smoother way to do this than instantiating a new clientset like this?
/lgtm |
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: juanluisvaladas, pliurh The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
I am not sure convinced this should be done in |
} | ||
if ns.Status.Phase == v1.NamespaceTerminating { | ||
log.Printf("finalize Namespace %s", objDesc) | ||
ns.Spec.Finalizers = []v1.FinalizerName{} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hang on a moment - you can't just go and delete all finalizers. You should only delete the finalizer for which you are responsible.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This logic will only be triggered when user hit a problem with OVN after the migration and node reboot. At that time all the openshift API is not available (as OVN doesn't work), so the finalizer of openshift-sdn
namespace cannot be removed. In such situation, the user cannot rollback, as the openshift-sdn
is hung in the Terminating
, new objects cannot be recreated.
So either we ask user to manually invoke the finalize
API before rolling back, or we add this logic in the code before recreating the openshift-sdn
namespace.
/hold |
Normally, it will work. However, if the OVN doesn't work after the migration. The finalizers in the namespace cannot be removed as expected. And the namespaces will be stuck in |
I understand what you mean, I read the BZ. I am not saying "you should not do this", rather: "you might think about not doing it there and instead in |
Yeah, as written this can't be merged. We cannot just go deleting all namespace finalizers. That will cause all sorts of unpleasantness, like possibly orphaned pods. |
Let me groom the logic a bit. In this patch The reason why we shall not put this logic in |
There will be no orphaned pods, all the openshift-sdn pods will be removed before the reboot. And this finalized shall only be triggered after the reboot by the rollback operation. However, it could be an issue for the openshift API controlled objects under openshift-sdn namespace. To solve that, either
But I don't know whether option 2 is practical or not. |
replace by #641 |
As controller-runtime client has yet support subresources like
spec\finalizers
,using client-go to invoke the Finalize API of Namespace.