-
Notifications
You must be signed in to change notification settings - Fork 791
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
BOSKOS sometimes fails to be recreated #3600
Comments
|
Direct consequence is the lack of cleaning of GCP resources by boskos. The resources accumulating over time, it was not possible for the tests to pick up resources.
I recreated the ameukam@cloudshell:~ (k8s-infra-oci-proxy-prod)$ kubectl --context gke_k8s-infra-prow-build_us-central1_prow-build get pods -l app=boskos-reaper -n test-pods
NAME READY STATUS RESTARTS AGE
boskos-reaper-7b54fdbb8d-8jzb2 1/1 Running 0 86m Cleanup as be done: I'll try to monitor this over the incoming week-end. |
No incidents until now. Let's keep it under the radar. Long term solution is impactful for the build clusters /milestone v1.25 |
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
No incidents reported until now. The GKE clusters have upgrade to a 1.5 containerd version. I'll considered this as solved. |
@ameukam: Closing this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Description
The Boskos instance running inside the community-owned GKE build clusters sometimes fails to get recreated after rescheduling.
This is seems to be related to a
containerd
bug present in the current version we are currently using.Initial finding: kubernetes/kubernetes#107561
containerd
issue: containerd/containerd#4604Fix merged in
containerd
1.6: containerd/containerd#6478 and backported in 1.5.Possible solutions
Long term
containerd
containing the fix.The fix seems be in
1.22-x.gke-.y
. (still looking for the exact version)Short term
Rapid Mitigation
In case the incident occurs again, delete the boskos pod
Access to the build clusters only accessible to:
k8s.io/groups/sig-testing/groups.yaml
Lines 208 to 224 in b4fed55
Locate the boskos pod:
Delete the pod:
/area prow
/area infra
/priority important-soon
/milestone v1.24
The text was updated successfully, but these errors were encountered: