-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CAPBk hangs, possibly in slow API server situations? #2073
Comments
im ok closing this but just wanted to go on record w/ a concrete use case in case this plays into the priority of tackling the broader issue. |
This sounds a lot like something @akutz ran into. He found that, I believe, client-go had an infinite timeout when connecting to the API server. Is it possible this is hanging on creating a client to a non-responsive or perhaps very slow API server as you suggest? |
not sure. plausible though bc my infra was a little slow when i saw this. |
related: think i just saw an issue with capv-manager, where restarting the capv-controller-manager- pod immediately caused my cluster creation to resume again. it continually sais ‘succesfully reconciled’ but doesnt actually create a new VM, and then restarting it, creates a new VM. I saw something similar a few weeks ago where restarting cabpk actually solved a stopped up management plane. CAPV Controller Manager spams that resource is not patchedI didnt capture all the logs, but i did manage to capture this being repeated alot in capv-controller-manager:
CABPK logs , nothing suspicious...And i captured all the logs for cabpk during this time period here
Just FYI if any one is curious about this issue. |
@jayunit100 do we have an action items here, or should we close this one out? |
IMO a possible action item is to check all the client-go /controller runtime client have a finite timeout |
Duplicate of #2993 /close |
@vincepri: Closing this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
What steps did you take and what happened:
This is a subissue of #1855, because the solution to it likely will involve a liveness check :)
grace-period=1
.After restarting my
capbk
pod , i found that the cluster came to life instantly, and all VMs came up w/ proper configs.So i figure, this is related to some kind of problem where, if your API server is slow CAPBK hangs . Somehow restarting CAPBK fixes things.
.... I assume CAPBK isnt DOS'ing the API server, but won't rule that out until someone tells me to stop guessing ?
What did you expect to happen:
Capbk should die periodically if its in a hung state or not capable of doing its job.
Anything else you would like to add:
I dont have logs anymore for this capbk instance, but i do have a witness, @fabriziopandini ... :)
Environment:
kubectl version
): 1.16/etc/os-release
): ubuntu 18/kind bug
The text was updated successfully, but these errors were encountered: