Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Listener pod names conflict when using the same runnerScaleSetName in multiple orgs. #3587

Open
4 tasks done
akhilp6 opened this issue Jun 10, 2024 · 2 comments
Open
4 tasks done
Labels
bug Something isn't working gha-runner-scale-set Related to the gha-runner-scale-set mode

Comments

@akhilp6
Copy link

akhilp6 commented Jun 10, 2024

Checks

Controller Version

0.9.0

Deployment Method

Helm

Checks

  • This isn't a question or user support case (For Q&A and community support, go to Discussions).
  • I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

gha-scale-set-controller installed with helm chart defaults in action-runner-system namespace.
gha-runner-scale-set chart installed with for org1 in runners namespace with values.yaml:

githubConfigUrl: 'https://github.com/org1'
runnerGroup: staging
runnerScaleSetName: staging-centos7

gha-runner-scale-set chart installed with for org2 in runners namespace with values.yaml:

githubConfigUrl: 'https://github.com/org2'
runnerGroup: staging
runnerScaleSetName: staging-centos7

Describe the bug

The controller creates AutoscalingListener/staging-centos7-7c5bdbdc-listener in the local action-runner-system namespace for org1 but this gets stuck in a error/crash loop for the org2 listener as it creates it with the same name. I believe due to a continuous conflict in which org its trying to register the listener.

listener name is computed here. it basically takes in hash of the namespace, since both the scalesets are deployed in runner namespace, it basically gets the same name.
Listener pod logs
https://gist.github.com/akhilp6/9f22d2da2da9a52ed4b02e9aeb8ae0d1

This happens even when we have multiple cluster which different controllers and we try to deploy scalesets with same name, it basically creates on one cluster and then throws an error on the other cluster.
Application returned an error: createSession failed: failed to create session: 409 - had issue communicating with Actions backend: The runner scale set staging-centos7 already has an active session for owner

We probably can workaround with having these scalesets creating in different namespace, but we are having some issue with this approach, one issue is we need to copy the secrets in all the namespace which we create(which we would like to avoid)

Describe the expected behavior

We would like to have "Adding org to the AutoscalingListener object name" or "placing the listener in the runner namespace instead of the local namespace"

Additional Context

org1 values.yaml

githubConfigUrl: 'https://github.com/org1'
runnerGroup: staging
runnerScaleSetName: staging-centos7

org2 values.yaml

githubConfigUrl: 'https://github.com/org2'
runnerGroup: staging
runnerScaleSetName: staging-centos7

Controller Logs

https://gist.github.com/akhilp6/6d759fa1f1bd26b427e8c3c225dc0e73

Runner Pod Logs

N/A
@akhilp6 akhilp6 added bug Something isn't working gha-runner-scale-set Related to the gha-runner-scale-set mode needs triage Requires review from the maintainers labels Jun 10, 2024
Copy link
Contributor

Hello! Thank you for filing an issue.

The maintainers will triage your issue shortly.

In the meantime, please take a look at the troubleshooting guide for bug reports.

If this is a feature request, please review our contribution guidelines.

@nikola-jokic nikola-jokic removed the needs triage Requires review from the maintainers label Jun 11, 2024
@aarondstone
Copy link

Hi team! LinkedIn is asking for an update on where things stand with this issue.

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working gha-runner-scale-set Related to the gha-runner-scale-set mode
Projects
None yet
Development

No branches or pull requests

3 participants