Getting 403 error: "failed to get runner registration token on refresh"

[chen-shapira]

I have installed the gha-runner-scale-set-controller and gha-runner-scale-set on my EKS cluster using Helm.
When the controller tries to get the runner group by name from Github API, I am seeing a 403 error in the controller logs:

2024-04-03T13:39:23Z	INFO	AutoscalingRunnerSet	Runner scale set id annotation does not exist. Creating a new runner scale set.	{"autoscalingrunnerset": {"name":"arc-runner-set","namespace":"arc-runners"}}
2024-04-03T13:39:23Z	INFO	AutoscalingRunnerSet	Creating a new runner scale set	{"autoscalingrunnerset": {"name":"arc-runner-set","namespace":"arc-runners"}}
2024-04-03T13:39:23Z	INFO	actions-clients	retrieve actions client	{"githubConfigURL": "https://github.com/SkyPlug", "namespace": "arc-runners"}
2024-04-03T13:39:23Z	INFO	actions-clients	using cache client	{"githubConfigURL": "https://github.com/SkyPlug", "namespace": "arc-runners"}
2024-04-03T13:39:23Z	INFO	actions-clients	refreshing token	{"githubConfigUrl": "https://github.com/SkyPlug"}
2024-04-03T13:39:23Z	INFO	actions-clients	getting runner registration token	{"registrationTokenURL": "https://api.github.com/orgs/SkyPlug/actions/runners/registration-token"}
2024-04-03T13:39:23Z	ERROR	AutoscalingRunnerSet	Failed to get runner group by name	{"autoscalingrunnerset": {"name":"arc-runner-set","namespace":"arc-runners"}, "runnerGroup": "skyx-runners", "error": "failed to get runner registration token on refresh: unexpected response from Actions service during registration token call: 403 - {\"message\":\"You must be an org admin or have the runners and runner groups fine-grained permission.\",\"documentation_url\":\"https://docs.github.com/rest/actions/self-hosted-runners#create-a-registration-token-for-an-organization\"}"}
github.com/actions/actions-runner-controller/controllers/actions%2egithub%2ecom.(*AutoscalingRunnerSetReconciler).createRunnerScaleSet
	github.com/actions/actions-runner-controller/controllers/actions.github.com/autoscalingrunnerset_controller.go:430
github.com/actions/actions-runner-controller/controllers/actions%2egithub%2ecom.(*AutoscalingRunnerSetReconciler).Reconcile
	github.com/actions/actions-runner-controller/controllers/actions.github.com/autoscalingrunnerset_controller.go:195
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227

I have created a secret with the PAT, after creating a PAT (classic) with org:admin scope permissions.
I have also tested directly with:

curl -L \                                                                                                                                                                                                    
  -H "Accept: application/vnd.github+json" \
  -H "Authorization: Bearer <PAT>" \
  -H "X-GitHub-Api-Version: 2022-11-28" \
  https://api.github.com/orgs/SkyPlug/actions/runners

Getting this response:

  {
  "message": "You must be an org admin or have the runners and runner groups fine-grained permission.",
  "documentation_url": "https://docs.github.com/rest/actions/self-hosted-runners#list-self-hosted-runners-for-an-organization"
}

When I change the user's role in the organization to Owner, the API call is working, but I cannot keep it as is.
The 'admin:org' scope permission was a hard pill to swallow alone.

I would love your help with this issue, as the documents clearly state that the 'admin:org' scope on the PAT (classic) should be enough.

Thanks in advance.

[nixlim]

Have same issue. Although it's a bit of a funny one. Error is identical BUT I run against an enterprise server and, when I originally set this up it worked smoothly without this error. Today I changed the name of the runner group and this error materialized.