-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ClusterResourceSet could take up to 16m40s to reconcile again due to locked ClusterCache Tracker #9775
Comments
+1 I think in general it's fine to requeue with a constant interval when we hit this specific lock (e.g. 1m) instead of going into an expontential backoff /triage accepted /cc @fabriziopandini |
(would be nice to get this fixed for v1.6.0) |
The exponential requeue delay gets added because it returns Using |
@chrischdi Can we please follow-up and do the same for all other controllers and CAPV? |
Other places where we return the same:
|
What steps did you take and what happened?
.status.nodeRef
The ClusterResourceSet controller repeatedly hits this line and returns with
ctrl.Result{Requeue: true}
:https://github.com/kubernetes-sigs/cluster-api/blob/main/exp/addons/internal/controllers/clusterresourceset_controller.go#L169
Because the kube-apiserver takes some time to get get reachable: this return gets hit multiple times and because it ends in a Rate Limited Queue it builds up to the maximum requeue time of 1000s (16m40s).
This could also happen in other controllers where we do also return
ctrl.Result{Requeue: true}
when hitting theremote.ErrClusterLocked
error.What did you expect to happen?
CRS to rollout as soon as kube-apiserver is available and by that the test to proceed/succeed.
Cluster API version
v1.6.0-rc.0
for the test.v1.5.3
Kubernetes version
v1.28.0
Anything else you would like to add?
Workarounds are:
Example of a failed test: https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api-provider-vsphere/2517/pull-cluster-api-provider-vsphere-e2e-main/1727951432931348480
Label(s) to be applied
/kind bug
One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels.
The text was updated successfully, but these errors were encountered: