New Runner Pods Not Getting Scheduled

[alexgaganashvili]

Hi,

After some runner pods get scheduled, run jobs, and then get terminated, new pods won't get scheduled and jobs are stuck in the queue. I end up having to reinstall a scaleset (using helm). I'm running version 0.9.0; and runners are registered at an org level as runner groups. I see in a listener's logs that it's getting messages from GitHub:

2024-06-20T19:18:03Z	INFO	listener-app.worker.kubernetesworker	Ephemeral runner set scaled.	{"namespace": "my-scaleset-ns", "name": "my-runner-6pzbd", "replicas": 3}
2024-06-20T19:18:03Z	INFO	listener-app.listener	Getting next message	{"lastMessageID": 11}
2024-06-20T19:18:11Z	INFO	listener-app.listener	Getting next message	{"lastMessageID": 14}
2024-06-20T19:18:53Z	INFO	listener-app.listener	Getting next message	{"lastMessageID": 11}
2024-06-20T19:19:01Z	INFO	listener-app.listener	Getting next message	{"lastMessageID": 14}

2024-06-21T16:22:44Z	INFO	listener-app.worker.kubernetesworker	Ephemeral runner set scaled.	{"namespace": "my-scaleset", "name": "my-runner-rpvp2", "replicas": 10}

I also see "currentRunnerCount": 11, whereas the actual number is 2 until it's 0.

Looking at autoscalingrunnerset, I see:

I have to delete autoscalingrunnerset objects and the listeners - then I can see new runner pods getting scheduled.

I'm not observing this issue in a different cluster where I'm running the same version.

Thanks.

[alexgaganashvili]

cc: @nikola-jokic, I read in some thread that 0.9.2 doesn't have this issue (idk why I'm not running into the same problem on the other cluster). Btw, when's the next release going to be rolled out?

[alexgaganashvili]

kubectl get ephemeralrunner is showing:
my-runner-group-dqfg2-runner-6sg4r https://f.q.d.n/my-org 2374 Failed Failed to create the pod: pods "my-runner-group-dqfg2-runner-6sg4r" is forbidden: exceeded quota: capsule-myinfra-0, requested: requests.cpu=2, used: requests.cpu=60, limited: requests.cpu=60 2m14s

Is it the runner that does the accounting of resource usage in this case? If there are no runner pods running, there should be plenty of resources available.
When I check the corresponding resource quota, I get requests.cpu: 0/60.

So, I do:

kubectl get get EphemeralRunner

and then delete them.

Afterward, new runner pods come up.

The same issue is run into by someone else

[alexgaganashvili]

Ended up creating a cronjob.