-
Notifications
You must be signed in to change notification settings - Fork 2.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Time and again Host cluster test pods where the test scripts are run to spin up scalability tests are getting deleted abruptly. #31459
Comments
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands:
Please see the group list for a listing of the SIGs, working groups, and committees available. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
cc @xmudrii |
In the interim, what do you think of running the test pod on the community gke cluster? We can tweak it to assume the correct AWS role to run in the aws scalability account |
I'm open to it but how do we manage the creds if we take that path ? |
This is a known issue unfortunetly and we're working on resolving it. I hope we'll have a fix for this issue as soon as possible. |
This is a problem with the cluster hosting the test pod, on the prior clusters we set suitable (extra high) resource requests / limits to all but guarantee the scalability pods were not interrupted. This problem impacts a lot of CI jobs on the cluster, but it's especially noticeable with lengthy $$$ scale tests. We discussed this at the last K8s Infra meeting, to me it sounds like the cluster's autoscaling behavior needs tuning to not aggressively remove nodes without draining first. |
Given this is happening way more frequently now, Is it possible to run scale tests on it's own cluster ? Or could we leverage Karpenter ? Internally, we leverage karpenter to manage the nodes on Host cluster. |
We're planning to look into this and we have an issue to track this: kubernetes/k8s.io#5168
@BenTheElder It's not really that we need to tune the cluster-autoscaler's behavior, it's more about making sure that it respects ProwJob pods. Let's look into this together after the holidays break. |
@xmudrii just wondering if you guys got a chance to followup ? Given we are being impacted actively, are we thinking in terms of short-term and long-term solutions for this if LOE is too much ? Any rough ETA we are thinking to resolve this issue ? |
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
This shouldn't be an issue any longer |
@xmudrii: Closing this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
What happened:
Following examples test runs :
And some of them show as indefinitely running ( Not sure if it's running or a UI bug here)
What you expected to happen:
These pods should not be deleted and be viewable to look at the logs and artifacts of the testrun. If the pod dies in the middlke of the testrun, tests fail and it's an issue.
Given these are scale tests, as it is they are expensive w.r.t time and cost,
pods got deleted issue
masks all other issues that the test may have.How to reproduce it (as minimally and precisely as possible):
Randomly we encounter this
Please provide links to example occurrences, if any:
Links are provided ablve
Anything else we need to know?:
The text was updated successfully, but these errors were encountered: