Listener randomly turns non-responsive after some time

[audunsolemdal]

Checks

• I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
• I am using charts that are officially provided

Controller Version

0.6.1

Deployment Method

Helm

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions).
• I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

Create a runner group, e.g. `my-group`
Create self hosted runner via ARC, link it to `my-group`
In github actions yaml, make your jobs target the runner group


runs-on:
  group: my-group

Run some jobs
wait patiently
This will work fine for some days, then all of a sudden the listener seems to freeze for no obvious reason

Describe the bug

The listener works fine to start with and jobs are assigned as usual. After some time (days) it seems that the listener freezes. It seems random when this occurs. This results in requested jobs not being assigned.

Once the listener pod is deleted/recreated, jobs are instantly assigned as expected. Without restarting the listener pod it seems that the listener pod stops working without any output indicating that it failed. I do not see any blocked network traffic, so I do not believe that to be the issue.

Describe the expected behavior

Business as usual

Additional Context

The listener does not detect new jobs for some time. Restarting the listener pod solves the issue

YAML for helm chart


securityContext:
  capabilities:
    drop:
    - ALL
  readOnlyRootFilesystem: true
  runAsNonRoot: true
  runAsUser: 1000

labels:
  app: actions-runner-controller

priorityClassName: platform-high

tolerations:
  - key: CriticalAddonsOnly
    operator: Exists



yaml for runnerset:

githubConfigUrl: "https://github.com/myorg"
githubConfigSecret: "pre-defined-secret"

runnerGroup: mygroup

template:
  spec:
    containers:
      - name: runner
        image: mycustomrunnerimage:latest
        command: ["/home/runner/run.sh"]
        resources:
          limits:
            memory: "1Gi"
            cpu: "700m"
          requests:
            cpu: "75m"
            memory: "250Mi"

minRunners: 0

Controller Logs

Logs are completely empty in the relevant period

Runner Pod Logs

Listener logs:
--- several previous successful refreshing_client runs on an hourly basis before this
2023-11-22T18:35:24Z   INFO    refreshing_client    message queue token is expired during GetNextMessage, refreshing...
2023-11-22T18:35:24Z    INFO    refreshing token        {"githubConfigUrl": "https://github.com/myorg"}
2023-11-22T18:35:24Z    INFO    getting access token for GitHub App auth        {"accessTokenURL": "https://api.github.com/app/installations/myappid/access_tokens"}
2023-11-23T08:35:29Z    INFO    getting runner registration token       {"registrationTokenURL": "https://api.github.com/orgs/myorg/actions/runners/registration-token"}
2023-11-23T08:35:30Z    INFO    getting Actions tenant URL and JWT      {"registrationURL": "https://api.github.com/actions/runner-registration"}
-- No further output until pod is restarted

[github-actions]

Hello! Thank you for filing an issue.

The maintainers will triage your issue shortly.

In the meantime, please take a look at the troubleshooting guide for bug reports.

If this is a feature request, please review our contribution guidelines.

[antoninguyot]

Also experiencing this on 0.7.0

[nikola-jokic]

Hey @audunsolemdal @antoninguyot,

Can you please create support tickets ☺️?

[audunsolemdal]

Created ticket here #3102

[nikola-jokic]

Sorry, I thought like this: https://docs.github.com/en/support/contacting-github-support/creating-a-support-ticket

[audunsolemdal]

Ok, I created a regular support ticket now. Notice the header which popped up though, this should be updated if this is the way to go for official support

[nikola-jokic]

Thank you! The message is right, configuring kubernetes cluster and/or ARC is not within the scope of the support. However, investigating this issue may require providing information that shouldn't be visible to the public (such as private repositories, etc) and that is why I asked you to create a support ticket. This issue does not fall into the scope of configuring ARC.

[audunsolemdal]

Ran multiple listeners on version 0.8.1 for 14 days now without issues. Seems to be working fine in tag 0.8.x onwards, thanks