Deploy runner with multiple label with gha-runner-scale-set?

[tbirk-bhs]

Hello,

I'm trying to migrate from the legacy ARC to the new gha-runner-scale-set. With the legacy ARC I had a RunnerDeployment and added multiple labels. e.g.

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
...
spec:
  template:
    spec:
      ...
      labels:
        - ubuntu-22.04
        - ubuntu-latest

Is it also possible to add multiple runner labels to a gha-runner-scale-set?

Thanks for helping me!

Timon

[Link-]

Labels are not supported and there are no plans to support them. It's already discussed in our documentation:

Here: https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/deploying-runner-scale-sets-with-actions-runner-controller#about-runner-scale-sets

and here: https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/about-actions-runner-controller#using-arc-runners-in-a-workflow

Runner scale sets cannot be comprised of heterogeneous types of runners (different OSs, specs etc.). As such, each deployment should have a specific runner configuration and a unique name.

If you want different types of runners, you need to configure different runner scale sets.

[thesuperzapper]

I have raised an issue saying that all self-hosted runners should be getting the self-hosted label in addition to whatever the custom label is:

• [CRITICAL] self-hosted label is missing from all scale-set (AutoscalingRunnerSet) runners #3330

Please promote this issue within your teams and to your github account managers!

---

Just so we are clear, for scaling purposes we can just ignore the self-hosted label, so this should not be technically challenging, because EVERY runner should just be assumed to have that label.

This is a critical security issue, as selecting self-hosted in addition to your custom label(s) prevents accidentally running on public runners (if GitHub decides to create a public runner label with the same name as your internal ones). In fact, the official GitHub docs suggest selecting self-hosted runners using runs_on: [self-hosted, MY_LABEL] for this very reason:

All self-hosted runners have the self-hosted label. Using only this label will select any self-hosted runner. To select runners that meet certain criteria, such as operating system or architecture, we recommend providing an array of labels that begins with self-hosted (this must be listed first) and then includes additional labels as needed. When you specify an array of labels, jobs will be queued on runners that have all the labels that you specify.

[danmanners]

I'll second everything @thesuperzapper called out here. Cross referencing this discussion topic from @tmehlinger.

[thesuperzapper]

I have tagged the GitHub leadership in #3330 (comment).

Please show your support by liking that comment, if you agree that the GHA team needs to resolve this.

[dave-schmitz1]

As Runner Scale Sets appear to require unique names, we can't use the same names in both blue and green clusters. And by not being able to assign both of them the same label, we would have to change runs-on for all workflows every time we spin up a new Runner Scale Set. This is of course not viable.

For this specific use case, you can. Check High availability and automatic failover in our docs.

@Link- If I'm following your guidance right and I have a blue / green cluster for dev and blue / green cluster for prod you saying I would make this operational by having a 4 specific runners and runners called runner-prod or runner-dev along with removing runs-on group for all workflows and only use the label to direct workflows?

[naikrovek]

Actions Runner Controller just wasn't thought out well enough to be used in production.

This limitation is silly. If you want people to know how long their job has to wait in the queue before a runner takes it, then show that. I know that info is in the application somewhere, because jobs go into a queue and get worked on mostly in order. What you should not have done is to come up with an entirely new runner system which moves the queue out of GHES entirely and limits users in so many other ways. The tradeoffs made take so much away and offer almost nothing in return.

I am still salty today after reading about this months ago. There's not even a security boundary between ARC runner containers in DinD mode. They all have the host's Docker socket mapped in. What is the point of this mode (and KinD mode, I assume) at all if any running container can easily reach into any other container on the same host?

ARC just isn't production ready. GitHub themselves don't even use it, to my knowledge.

What kinds of other goofy decisions are GitHub going to make in the future?

You wrote GHES and github.com. If you want to show queue depth, then show it. If you can't, then that's a hole you dug yourself, so fill it in and dig the correct hole this time. "lol nah you only get one label and it must be unique and it can't be descriptive. You're welcome."

[nebuk89]

The reason we opted for single labels on ARC (and are doing the same thing on the larger hosted runners as well) is to get us ready to improve the Dev UX about 'why isn't my job starting'.

When we had multiple labels, doing deterministic assignment of jobs to a 'scale set' wasn't possible. The ven diagram of runners that matched labels vs. the labels I was looking for meant that jobs always just waited until 'something was free' and 'where' something would start was less clear. This conflicted with a reasonably high request we had to 'see the queue of jobs and when mine will start'.
I note that some people mitigated this by adding 'lots and lots' of labels to make a run on target very specific – which also isn't great for new users.

To mitigate this, we want to move to a world of more deterministic assignment of jobs to a scale set that takes jobs in a FIFO fashion.

We made the decision that the way to approach this was to move to single named scale sets/queues as a target. That way when a job is assigned it 'goes to its queue and waits' rather than sitting in some meta queue of all jobs for all potential machines.

I do appreciate that this results in a change in workflow files :( this is why we haven't “done away” with multiple labels at a system level yet.

What we don't really want to do, if we can help it, is to create an experience that is good for our hosted machines to review their queue but sucks for self-hosted as no one moves off multi label.

Which lastly led to the decision – what if we used new ARC as a bit of a forcing factor for this to help gently nudge people to the future if people want to come over.

[shapirus]

What an insane decision.

This.

We are now looking (admittedly not very intensively) for an alternative solution. ARC with no labels sucks, it's that plain simple. I'm sure there has to be some self-hosted software that'll allow to launch build runner pods with github used only to fire up webhooks on certain actions on repositories. It'll be a little more hassle to deal with repository secrets etc., though.

Anyone knows of a good alternative to GHA that can be self-hosted and is k8s-friendly?

[thesuperzapper]

I have tagged the GitHub leadership in #3330 (comment).

Please show your support by liking that comment, if you agree that the GHA team needs to resolve this.

[jeffmccune]

Note

Summary request: previously, actions supported highly available runners. Scale sets support HA only for enterprise plan customers. Please consider restoring the ability to configure highly available runners without an enterprise feature gate.

"Highly Available" in this context is a workflow executing on any one of multiple available regional clusters globally.

Cross posting from #3351 as requested.

Thanks for taking the time to answer this. You're correct about my two questions. The only comment I have is that I'm a bit frustrated because with scale sets, groups are required to achieve the same behavior that was previously supported with labels. My frustration is that groups are a paid enterprise feature but labels are not, so this feels like a step backwards.

I'm a paying customer, but some of the github orgs I work with are not, so they cannot use groups and as a result cannot deploy highly available scale sets. Please consider adding back some mechanism to have a workflow execute on any available cluster without requiring a paid feature like was possible previously. Thanks again for your time responding to the question.

[crohr]

What an insane decision.

This.

We are now looking (admittedly not very intensively) for an alternative solution. ARC with no labels sucks, it's that plain simple. I'm sure there has to be some self-hosted software that'll allow to launch build runner pods with github used only to fire up webhooks on certain actions on repositories. It'll be a little more hassle to deal with repository secrets etc., though.

Anyone knows of a good alternative to GHA that can be self-hosted and is k8s-friendly?

@shapirus I'm not sure it has to be k8s friendly, since efficiently launching bare ephemeral VMs is all that's needed, but I believe you might be well served by RunsOn, which is entirely self-hosted in your AWS account.

[shapirus]

Anyone knows of a good alternative to GHA that can be self-hosted and is k8s-friendly?

@shapirus I'm not sure it has to be k8s friendly, since efficiently launching bare ephemeral VMs is all that's needed, but I believe you might be well served by RunsOn, which is entirely self-hosted in your AWS account.

Thanks. This looks interesting.

[nebuk89]

👋 I don't want to dupe out too much and @tmehlinger had put quite a full and detailed single breakdown here #3340 so I have added some updates there following more discussions with @juliandunn and other folks :)

[mbarany]

I am trying to use CodeQL with self-hosted runners. The documentation over here on GitHub's help page says to add a label to the runner, but it seems like ARC does not support adding additional labels.

Would I have to create a scale-set just for CodeQL with a runnerScaleSetName value of code-scanning?
If so, that seems a bit silly.

[luis-lf]

Thats the only way I was able to make it work for dependabot and code-scanning

[stasostrovskyi]

Alternatively, you can have your own workflows for codeql and dependabot and run them on the same runners as regular builds. And having them as shared workflows can simplify rollout across multiple repositories.

[lifeofmoo]

Gah - only 2 weeks ago I finally got over the hurdle that the git cli was finally added to the runner container and now this 🤯

Not being able to even use the self-hosted label is painful

[naikrovek]

Yeah. I still don't understand this decision. If GitHub want to make it so that people can have an idea of when their job will run, which is why they say they made this decision, then show the queue depth of the queue the job is in. It seems very simple to me. Instead they are changing how all runners operate, so that they each get only one label so that queues are simpler and have names... this seems like a lot of work for them when they could simply show a job's position in a queue and let the jobs have labels as we're used to them. I'm going to have to get a job at GitHub in order to truly understand why this decision was made, I think.

[pawelros]

There is not even an option in GitHub org settings to forbid using public runners. With such option, going live with default runner group and this no-label absurd would be easier to swallow.

[ChrisOwen273]

The decision here seems to contradict guidance GitHub give for other parts of their product suite.

See Configuring self-hosted runners for code scanning in your enterprise

Enabling GHAS Code scanning on self hosted runners requires you to set a label called "self-hosted".

Between this thread and that document am I to take it that you cannot use Code scanning on self hosted runners when using ARC?

This seems a bit short sighted. And before anyone suggests it we are moving to EMU and removing access to public runners through that.

[stasostrovskyi]

You can run "advanced" code scanning workflow for codeql and choose runners like you would for any other workflow. Yes, it requires adding one more file to a repo, which is a downside, but it's still doable.

[hicksjacobp]

Enabling GHAS Code scanning on self hosted runners requires you to set a label called "self-hosted".

This doesn't apply to ARC runners.

You can register your runners with the code-scanning label, and it should work just fine, which is what your linked documentation states:

If you are using default setup for code scanning, assign the code-scanning label to your self-hosted runner.

The same goes for running Dependabot on self-hosted runners with the dependabot label (and you've opted-in to running Dependabot on self-hosted runners in your repository's Code security settings).