-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Occasionally jobs are stuck in "waiting for a runnner to come online" #3649
Comments
Hello! Thank you for filing an issue. The maintainers will triage your issue shortly. In the meantime, please take a look at the troubleshooting guide for bug reports. If this is a feature request, please review our contribution guidelines. |
We are now facing this issue also, same version. |
Bit more info, maybe will help.
You can clearly see that there are 23 runners, failed are 22, but even if I try to restart job and get a hold of pod, don't see any. On top of that in logs, I see (read from bottom to top):
It's super strange, that pod was running just few seconds and than is deleted. |
is there a workaround for this? the issue effectively causes a very poor experience |
experiencing this issue aswell with a new deployment on AKS. controller logs
runner logs
Totally blocked. |
We've seen the same issue. We noticed that there are other issues where this issue was discovered in older versions of actions runner controller, and one of them has a comment saying that the issue is fixed by restarting the listener. This doesn't fix the issue for anyone, but as triage you can deploy a cron job with something like this:
You'll also need to define the ServiceAcccount and its role/rolebinding, but the above should get you started. |
On our side, it somehow magically stopped to be an issue over 2 weeks ago. Since that time, no single job was not picked up (except around an hour, when we messed up our networking in AWS and you couldn't reach the Kubernetes cluster - but it was purely problem on network, which we caused). |
Checks
Controller Version
0.9.3
Deployment Method
Helm
Checks
To Reproduce
Describe the bug
In the example, on the CI piepline:![image](https://private-user-images.githubusercontent.com/139145681/346574429-bafb9ab2-7e76-4ac5-993b-05f0198320a3.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ1MDQxMDYsIm5iZiI6MTcyNDUwMzgwNiwicGF0aCI6Ii8xMzkxNDU2ODEvMzQ2NTc0NDI5LWJhZmI5YWIyLTdlNzYtNGFjNS05OTNiLTA1ZjAxOTgzMjBhMy5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODI0JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgyNFQxMjUwMDZaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1iZTVlZjAyZGJjMTQ3MDc2YWE0ZmRhMzA4YTkwODE3ZDIwZjZiMWUyMTAzMDU5OGFiZTExYjExMDlkZGFiYmExJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.30UvfxSbbrpQtsMjtI0Su_d22YszjvFs0vdbRuTEvZI)
We have multiple cases of jobs waiting a long time for a runner, without any clear explanation.
Describe the expected behavior
Additional Context
There has been attempt to play around the 'minWorkers' configuration. of the runner-set to prevent the waiting issue.
Runner-set are deployed with helm using the values.yaml file:
Controller Logs
Runner Pod Logs
No relevant logs for the runner, as it is the lack of runner that is the issue
The text was updated successfully, but these errors were encountered: