vpa-admission-controller: Wire contexts #6891
Labels
area/vertical-pod-autoscaler
kind/bug
Categorizes issue or PR as related to a bug.
triage/accepted
Indicates an issue or PR is ready to be actively worked on.
Which component are you using?:
vertical-pod-autoscaler
What version of the component are you using?:
Component version: 1.1.2
What k8s version are you using (
kubectl version
)?:v1.29
What did you expect to happen?:
Right now, the requests the vpa-admission-controller handles are not contextified. For example, for the handler for Pods,
context.TODO()
is used in few places where the admission-controller is making requests along the way:autoscaler/vertical-pod-autoscaler/pkg/target/fetcher.go
Line 186 in c79bdaf
autoscaler/vertical-pod-autoscaler/pkg/target/controller_fetcher/controller_fetcher.go
Line 251 in 3a3b388
Due to the usages of context.TODO(), when the caller (kube-apiserver) cancels the request (due to client side timeout), the admission-controller's Pod handler is not notified about this and continues to process the requests even when the request is cancelled client side.
We recently faced a VPA related outage (described in #6884) where the vpa-admission-controller was client-side throttled due to the low default kube-api-qps/burst settings.
From the logs we see that it was throttled > 50 minutes:
Hence, the vpa-admission-controller currently would wait the client-side throttling (> 50min) instead of canceling the request.
Meanwhile the kube-apiserver cancelled the request after the configured timeout in the webhook (10s in our case):
What happened instead?:
See above.
How to reproduce it (as minimally and precisely as possible):
Add a big sleep (higher than the kube-apiserver's timeout) in the Pod handler and make sure that the admission request continues to do things after kube-apiserver cancelled the request client-side.
Anything else we need to know?:
N/A
The text was updated successfully, but these errors were encountered: