How to set resource requests/limits?

[davidham]

We are working on a stability issue with our RunnerDeployments. My team inherited a setup where there are several different RunnerDeployments of different sizes, but the resource requests/limits were only being applied to the runner container. I want to set limits on the docker container too, but how do I do it? What runs on the runner container and what runs in the docker container?

[luka5]

Legacy ARC

In order to not leave this question unanswered: If you are still using the legacy ARC approach, the way to go would be to set dockerdContainerResources in the RunnerDeployment object. See also https://github.com/actions/actions-runner-controller/blob/master/docs/automatically-scaling-runners.md#additional-settings

Runner Scale Set

In case one uses the new runner scale set approach, this different.

DIND

If you define the runner and dind container on your own, you can just add the resource limits there. See

actions-runner-controller/charts/gha-runner-scale-set/values.yaml

Lines 115 to 159 in 9fba375

	## initContainers:
	## - name: init-dind-externals
	## image: ghcr.io/actions/actions-runner:latest
	## command: ["cp", "-r", "-v", "/home/runner/externals/.", "/home/runner/tmpDir/"]
	## volumeMounts:
	## - name: dind-externals
	## mountPath: /home/runner/tmpDir
	## containers:
	## - name: runner
	## image: ghcr.io/actions/actions-runner:latest
	## command: ["/home/runner/run.sh"]
	## env:
	## - name: DOCKER_HOST
	## value: unix:///run/docker/docker.sock
	## volumeMounts:
	## - name: work
	## mountPath: /home/runner/_work
	## - name: dind-sock
	## mountPath: /run/docker
	## readOnly: true
	## - name: dind
	## image: docker:dind
	## args:
	## - dockerd
	## - --host=unix:///run/docker/docker.sock
	## - --group=$(DOCKER_GROUP_GID)
	## env:
	## - name: DOCKER_GROUP_GID
	## value: "123"
	## securityContext:
	## privileged: true
	## volumeMounts:
	## - name: work
	## mountPath: /home/runner/_work
	## - name: dind-sock
	## mountPath: /run/docker
	## - name: dind-externals
	## mountPath: /home/runner/externals
	## volumes:
	## - name: work
	## emptyDir: {}
	## - name: dind-sock
	## emptyDir: {}
	## - name: dind-externals
	## emptyDir: {}

But, if you use the containerMode.type=dind it looks like one can not set such an limit.

https://github.com/actions/actions-runner-controller/blob/9fba37540a5dab4bb20ce2d931fa298ad5f132d4/charts/gha-runner-scale-set/templates/autoscalingrunnerset.yaml#L138C21-L138C55

actions-runner-controller/charts/gha-runner-scale-set/templates/_helpers.tpl

Lines 98 to 116 in 9fba375

	{{- define "gha-runner-scale-set.dind-container" -}}
	image: docker:dind
	args:
	- dockerd
	- --host=unix:///run/docker/docker.sock
	- --group=$(DOCKER_GROUP_GID)
	env:
	- name: DOCKER_GROUP_GID
	value: "123"
	securityContext:
	privileged: true
	volumeMounts:
	- name: work
	mountPath: /home/runner/_work
	- name: dind-sock
	mountPath: /run/docker
	- name: dind-externals
	mountPath: /home/runner/externals
	{{- end }}

Kubernetes

In addition, there is also the containerMode.type=kubernetes. This enables Running jobs in a container as well as Docker container action. Both are implemented as runner container hooks: "Running jobs in a container" does create a second pod where the workload is executed. While "Docker container action" does create a job, which itself creates a pod.

From my understanding, resource limits for those can be defined via a ACTIONS_RUNNER_CONTAINER_HOOK_TEMPLATE, but only after actions/runner-container-hooks#134 was merged.

In addition, one could also use another tool such as Kyverno to mutate pods on creation to ensure all of them have proper resource limits, or to ban such pods completely.

@nikola-jokic may I ask you for your opinion on this post? Is it correct like that?

[nikola-jokic]

For k8s mode, you are exactly right! However, for dind, if you need to set requests/limits, or modify dind configuration provided by us when the containerMode.type is set to dind, you should leave the containerMode object commented out, and uncomment the expanded spec that we are using.
This is the documented spec that will be expanded when you use containerMode.type = dind. If you want to customize it, you simply leave containerMode object untouched, i.e. don't set it to dind, uncomment the expanded spec and modify it in place.

[luka5]

Thanks for confirming

[davidham]

Thank you for your replies! In a given workflow run under dind, what does each container do? What runs on the runner container and what runs in the docker container?

[jonwest]

I can take a stab at answering this, though take my reply with a grain of salt.

The runner is responsible for "running" all of the commands that are passed through the actions, and the docker container is running the Docker daemon. The runner container mounts the Docker daemon socket, so even though the docker run command is executed from the runner container, that request is ultimately passed through to the Docker daemon running on the docker container.

TL;DR: Everything runs in the runner container unless it the job specifies a container, which will run within the docker container.

This means if you're applying resource requests and limits, this gets... tricky 😅. If you're planning things out like "I want to allocate 8GB of memory and 4vCPU for each runner", you'll need to consider where the workloads are ultimately going to be run. If the majority of the workload is within the container, then you'd want to allocate the majority of resources to the container. The caveat that we ran into in our implementation of this is that if you're using artifacts that use any compression that you still need to allocate a decent amount of CPU to the runner or compression/decompression becomes slower due to the CPU bottleneck, which slows down the workflow as a whole. Might not apply to your use case, but felt it was worth mentioning.

[shichengripple001]

cant we just do the same that runnerdeployment was implemented? ie, runner and dind in the same container, otherwise it is wasting resource

[IuriGarcia]

Hi @jonwest,

I am encountering an issue with passing resource limits to containers that are run by Docker-in-Docker (DinD). It seems that it is currently impossible to pass flags to the docker create command that creates DinD container jobs. Even if I comment out the containerMode section inside the scaleset values and pass a custom PodSpec with resource.requests/limits, these resource constraints are not being passed to the DinD container. Is there a way to pass resource limits to DinD containers without creating a custom runner image? Did you manage to do that?

[IuriGarcia]

static-analysis-sonarqube:
    runs-on: self-hosted
    container:
      image: sonarsource/sonar-scanner-cli:latest
      options: --memory 5120m

Seems like this will do the trick.

[cboettig]

Is there a better work-around then assuming the action will always 'self-enforce' resource limits in their github workflow (a la @IuriGarcia 's nice example above?) It seems like the resource requests/limits ought to be set in the values.yaml.

[doctorpangloss]

This is really confusing. I just need to see how to add resources to the containers for Kubernetes ARCs. What do I do?

[kanakaraju17]

I agree. Currently, there's no way to automatically set resource requests and limits on the runners. I've opened an issue regarding this, which I didn't fully understand. Please refer to it here.

#3641 (comment)