A Kubernetes operator that provisions ephemeral Pods that run Azure DevOps Pipelines agents, as well as other sidecar containers.
This operator is written in Go, based on controller-runtime. We use kubebuilder for bootstrapping. This solution is completely unrelated to KEDA.
As of 2023, Azure Pipelines has the following methods for self-hosting elastically-scalable agents:
- Azure Virtual Machine Scale Set agents, which are slowly auto-scaling VMs (several minutes of provisioning time), where you get poor resource usage (due to having only one agent per VM)
- KEDA (a general-purpose Kubernetes operator) using its Azure Pipelines scaler, or meta-level solutions based on it, e.g. Azure Pipelines Agent
- Kubernetes operators built specifically for Azure Pipelines agents, such as https://github.com/ogmaresca/azp-agent-autoscaler/ or https://github.com/microsoft/azure-pipelines-orchestrator, which have all been discontinued
While the KEDA-based solution is the most economical one, it has many technical drawbacks:
- With KEDA, running agents whose pods have multiple containers (e.g. because you need the tools contained in the respective images) is cumbersome. Instead of using Azure Pipeline's demands / capabilities feature, you have to create a dedicated agent pool for each set of containers, and maintain a correspondingly large set of KEDA-specific YAML manifests
- It is not easily possible to run agents with containers dynamically-defined in your pipeline YAML file.
Example: job #1 builds and pushes a Docker image (with a tag that depends on an Azure Pipelines variable) that you
want to run with a KEDA-based agent in job #2 that starts after job #1). The only solution is to start a dynamic
container as an ephemeral container (in an already-running agent Pod), which has many other drawbacks (e.g. an
ephemeral container cannot be protected from termination via a
preStop
lifecycle hook, it is invisible in most tools, and its resource usage is not accounted for viarequests
/limits
) - For every agent pool for which you configure KEDA, you need to define at least one agent
Job
/Deployment
withminReplicaCount
larger than 0, as otherwise your jobs would not even start. This disallows you to use the " scale to zero" approach, unless you manually take care of registering a fake/dummy agent for each pool/demand yourself - If you use long-running agent pods (i.e., not providing the
--once
flag to the Azure Pipelines agent container), KEDA may prematurely kill your agent pods, resulting in aborted pipelines and many 'offline' agents in your agent pool. Why? Because KEDA scales your Deployments/Jobs only based on the number of pending jobs. Suppose two jobs are pending, and two Deployments are scheduled by KEDA. One job terminates quickly, the other one takes longer. The pending job count gets reduced from 2 to 1, and KEDA down-scales the Deployments, and Kubernetes may (arbitrarily) try to kill the one that is still runs the active job.- While you could just use ephemeral pods (with the
--once
flag), e.g. as Kubernetes Jobs, as done in https://github.com/clemlesne/azure-pipelines-agent, their disadvantage is that they lack support for cache volumes: Kubernetes has no mechanism to ensure that a cache volume is only concurrently used by one Job (the " Once" in theReadWriteOnce
access mode is highly misleading)
- While you could just use ephemeral pods (with the
For these reasons, this Kubernetes operator provides a better solution solving all of the above problems at once.
azure-pipelines-k8s-agent-scaler
(this project) manages Kubernetes Pods that run the Azure Pipelines (AZP) agent
Docker
image (see here).
The pods are ephemeral, meaning that the agent container is started with the --once
flag, such that it terminates
after having completed one Azure Pipelines job.
Features of azure-pipelines-k8s-agent-scaler
:
- Ability to specify multiple pod configurations, each one for a different set of Azure Pipeline capabilities. For
each pod, you can configure a min/max count for the pods, and define several sidecar containers, e.g. BuildKit, or
any other tools you need in your pipeline
- Because sidecar containers are regular, statically-defined containers (not ephemeral containers), you
can
kubectl exec
into them. This is useful when you build pipelines and run into problems, e.g. the container crashing. Usingkubectl exec
, you can invoke debugging tools (liketop
orps
). You can also temporarily add asleep N
statement in your pipeline YAML for the problematicbash: ...
step, and then interactively run different commands in the container directly, until you figure out the correct command.
- Because sidecar containers are regular, statically-defined containers (not ephemeral containers), you
can
- Automatic termination of agent pods: once the AZP agent container has terminated,
azure-pipelines-k8s-agent-scaler
will automatically stop all other sidecar containers, to transition the pod into a terminated state - Automatic deletion of terminated pods (with the configurable ability to keep the N most recently terminated pods, for debugging purposes)
- Careful termination of superfluous agent pods (which only happens under rare circumstances anyway): only those pods are killed that are currently not running any AZP job
- Configurable definition of cache volumes that are mounted to the defined pods (e.g. to speed up BuildKit via a local
cache).
azure-pipelines-k8s-agent-scaler
provisions new volumes if necessary, and re-mounts existing volumes to new pods, ensuring that a volume is mounted to only one pod - Ability to specify extra containers (including their CPU and memory requests/limits) right in the AZP pipeline YAML
file via demands (example:
ExtraAgentContainers -equals name=containername,image=someImage:someTag,cpu=250m,memory=64Mi||name=otherContainerName,image=someOtherImage:someTag,cpu=500m,memory=128Mi
). Note that the values can also be dynamic, e.g. by populating the demand with AZP variables - Automatic registration of (offline) dummy/fake AZP agents that have the demands that you defined in your
configuration. This is necessary because the AZP platform would otherwise abort jobs that have demands for which there
are no registered agents. This AZP-platform-behavior conflicts with the dynamic registration of agents, as done by
the
azure-pipelines-k8s-agent-scaler
operator, but it can be worked around via the automatic pre-registration of agents. Note: if you defineExtraAgentContainers
you can fully-dynamically register new agents using the companion CLI tool azure-pipelines-agent-registrator - Operator can be installed into your Kubernetes cluster via Kustomize or Helm
Warning
This solution runs Azure DevOps agents as containers in Kubernetes. As documented here, such container-based agents cannot run Docker inside Docker. Consequently, neither the Docker task nor container-based jobs will work! ExtraAgentContainer
is a means to replace these features (using BuildKit to build Docker images).
You need a Kubernetes cluster to run against, e.g. the one by Docker Desktop, k3d, KIND, or a remote cluster.
Note: Your controller automatically uses the current context in your kubeconfig
file (i.e. whatever
cluster kubectl cluster-info
shows).
For local development, it is recommended to use the Kustomize manifests stored in the config/default
folder.
Since this project is based on kubebuilder, it comes with various Make targets (check the Makefile for details).
Examples:
make install
installs only the CRD (make uninstall
removes the CRD again)make docker-build
builds the local Docker image that contains the controller-manager (by default tagged ascontroller:latest
)make deploy
installs the Docker-based version into your cluster (installing the CRD, controller-managerDeployment
and various RBAC-related manifests). You need to manually runmake docker-build
beforehand!make undeploy
reverts the effects ofmake deploy
For a production deployment, it is recommended to use the Helm chart instead. There is also a tutorial available here.
Next, create a dedicated Kubernetes namespace that hosts your AZP agent Pods
. Inside it, create a Secret
that
contains your AZP Personal Access Token:
kubectl create secret generic azure-pat --from-literal=pat=YOUR-PAT-HERE --namespace <your-namespace>
Finally, create your desired CustomResource
(see the sample) and apply it to the
cluster (to <your-namespace>
), or use the demo-agent Helm chart.
Check the logs of the controller Pod/container to identify problems.
If your AZP jobs are pending (and you think that the operator should create Pods, but nothing happens and there is no log output either), you can temporarily enable additional debug-prints:
- Execute into the controller's container, which comes with a minimal shell that only supports creating and deleting files
- Run
touch /home/nonroot/debug
to enable debug printing - Run
rm /home/nonroot/debug
to disable debug printing again
This project aims to follow the Kubernetes Operator pattern.
It uses Controllers, which provide a reconcile function responsible for synchronizing resources until the desired state is reached on the cluster.
- Install the CRDs into the cluster:
make install
- Run your controller (this will run in the foreground, so switch to a new terminal if you want to leave it running):
make run
NOTE: You can also run this in one step by running: make install run
If you are editing the API definitions, generate the manifests such as CRs or CRDs using:
make manifests
NOTE: Run make --help
for more information on all potential make
targets
More information can be found via the Kubebuilder Documentation