Skip to content

Commit

Permalink
Add documentation for setting Kubernetes labels and annotations (#3687)
Browse files Browse the repository at this point in the history 
Co-authored-by: Thomas Anderson <127358482+zc-devs@users.noreply.github.com>
Co-authored-by: qwerty287 <80460567+qwerty287@users.noreply.github.com>
  • Loading branch information
@dominic-p @zc-devs @qwerty287
3 people authored May 13, 2024
1 parent 378ec1a commit fd94a2e
Showing 1 changed file with 49 additions and 19 deletions.
68 changes: 49 additions & 19 deletions docs/docs/30-administration/22-backends/40-kubernetes.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,19 +4,19 @@ toc_max_heading_level: 2

# Kubernetes backend

The kubernetes backend executes steps inside standalone pods. A temporary PVC is created for the lifetime of the pipeline to transfer files between steps.
The Kubernetes backend executes steps inside standalone Pods. A temporary PVC is created for the lifetime of the pipeline to transfer files between steps.

## Images from private registries

In order to pull private container images defined in your pipeline YAML you must provide [registry credentials in Kubernetes secret](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/).
As the secret is Agent-wide, it has to be placed in namespace defined by `WOODPECKER_BACKEND_K8S_NAMESPACE`.
Besides, you need to provide the secret name to Agent via `WOODPECKER_BACKEND_K8S_PULL_SECRET_NAMES`.
In order to pull private container images defined in your pipeline YAML you must provide [registry credentials in Kubernetes Secret](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/).
As the Secret is Agent-wide, it has to be placed in namespace defined by `WOODPECKER_BACKEND_K8S_NAMESPACE`.
Besides, you need to provide the Secret name to Agent via `WOODPECKER_BACKEND_K8S_PULL_SECRET_NAMES`.

## Job specific configuration

### Resources

The kubernetes backend also allows for specifying requests and limits on a per-step basic, most commonly for CPU and memory.
The Kubernetes backend also allows for specifying requests and limits on a per-step basic, most commonly for CPU and memory.
We recommend to add a `resources` definition to all steps to ensure efficient scheduling.

Here is an example definition with an arbitrary `resources` definition below the `backend_options` section:
Expand All @@ -42,13 +42,13 @@ You can use [Limit Ranges](https://kubernetes.io/docs/concepts/policy/limit-rang
### Runtime class
`runtimeClassName` specifies the name of the RuntimeClass which will be used to run this pod. If no `runtimeClassName` is specified, the default RuntimeHandler will be used.
See the [kubernetes documentation](https://kubernetes.io/docs/concepts/containers/runtime-class/) for more information on specifying runtime classes.
`runtimeClassName` specifies the name of the RuntimeClass which will be used to run this Pod. If no `runtimeClassName` is specified, the default RuntimeHandler will be used.
See the [Kubernetes documentation](https://kubernetes.io/docs/concepts/containers/runtime-class/) for more information on specifying runtime classes.

### Service account

`serviceAccountName` specifies the name of the ServiceAccount which the pod will mount. This service account must be created externally.
See the [kubernetes documentation](https://kubernetes.io/docs/concepts/security/service-accounts/) for more information on using service accounts.
`serviceAccountName` specifies the name of the ServiceAccount which the Pod will mount. This service account must be created externally.
See the [Kubernetes documentation](https://kubernetes.io/docs/concepts/security/service-accounts/) for more information on using service accounts.

### Node selector

Expand Down Expand Up @@ -83,8 +83,8 @@ You can use [PodNodeSelector](https://kubernetes.io/docs/reference/access-authn-

### Tolerations

When you use `nodeSelector` and the node pool is configured with Taints, you need to specify the Tolerations. Tolerations allow the scheduler to schedule pods with matching taints.
See the [kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) for more information on using tolerations.
When you use `nodeSelector` and the node pool is configured with Taints, you need to specify the Tolerations. Tolerations allow the scheduler to schedule Pods with matching taints.
See the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) for more information on using tolerations.

Example pipeline configuration:

Expand Down Expand Up @@ -117,7 +117,7 @@ steps:

### Volumes

To mount volumes a persistent volume (PV) and persistent volume claim (PVC) are needed on the cluster which can be referenced in steps via the `volumes` option.
To mount volumes a PersistentVolume (PV) and PersistentVolumeClaim (PVC) are needed on the cluster which can be referenced in steps via the `volumes` option.
Assuming a PVC named `woodpecker-cache` exists, it can be referenced as follows in a step:

```yaml
Expand All @@ -134,7 +134,7 @@ steps:

### Security context

Use the following configuration to set the [Security Context](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/) for the pod/container running a given pipeline step:
Use the following configuration to set the [Security Context](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/) for the Pod/container running a given pipeline step:

```yaml
steps:
Expand All @@ -151,9 +151,9 @@ steps:
[...]
```

Note that the `backend_options.kubernetes.securityContext` object allows you to set both pod and container level security context options in one object.
By default, the properties will be set at the pod level. Properties that are only supported on the container level will be set there instead. So, the
configuration shown above will result in something like the following pod spec:
Note that the `backend_options.kubernetes.securityContext` object allows you to set both Pod and container level security context options in one object.
By default, the properties will be set at the Pod level. Properties that are only supported on the container level will be set there instead. So, the
configuration shown above will result in something like the following Pod spec:

```yaml
kind: Pod
Expand Down Expand Up @@ -195,6 +195,24 @@ backend_options:
AppArmor syntax follows [KEP-24](https://github.com/kubernetes/enhancements/blob/fddcbb9cbf3df39ded03bad71228265ac6e5215f/keps/sig-node/24-apparmor/README.md).
:::

### Annotations and labels

You can specify arbitrary [annotations](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/) and [labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/) to be set on the Pod definition for a given workflow step using the following configuration:

```yaml
backend_options:
kubernetes:
annotations:
workflow-group: alpha
io.kubernetes.cri-o.Devices: /dev/fuse
labels:
environment: ci
app.kubernetes.io/name: builder
```

In order to enable this configuration you need to set the appropriate environment variables to `true` on the woodpecker agent:
[WOODPECKER_BACKEND_K8S_POD_ANNOTATIONS_ALLOW_FROM_STEP](#woodpecker_backend_k8s_pod_annotations_allow_from_step) and/or [WOODPECKER_BACKEND_K8S_POD_LABELS_ALLOW_FROM_STEP](#woodpecker_backend_k8s_pod_labels_allow_from_step).

## Tips and tricks

### CRI-O
Expand All @@ -217,7 +235,7 @@ These env vars can be set in the `env:` sections of the agent.

> Default: `woodpecker`

The namespace to create worker pods in.
The namespace to create worker Pods in.

### `WOODPECKER_BACKEND_K8S_VOLUME_SIZE`

Expand All @@ -241,13 +259,25 @@ Determines if `RWX` should be used for the pipeline volume's [access mode](https

> Default: empty

Additional labels to apply to worker pods. Must be a YAML object, e.g. `{"example.com/test-label":"test-value"}`.
Additional labels to apply to worker Pods. Must be a YAML object, e.g. `{"example.com/test-label":"test-value"}`.

### `WOODPECKER_BACKEND_K8S_POD_LABELS_ALLOW_FROM_STEP`

> Default: `false`

Determines if additional Pod labels can be defined from a step's backend options.

### `WOODPECKER_BACKEND_K8S_POD_ANNOTATIONS`

> Default: empty

Additional annotations to apply to worker pods. Must be a YAML object, e.g. `{"example.com/test-annotation":"test-value"}`.
Additional annotations to apply to worker Pods. Must be a YAML object, e.g. `{"example.com/test-annotation":"test-value"}`.

### `WOODPECKER_BACKEND_K8S_POD_ANNOTATIONS_ALLOW_FROM_STEP`

> Default: `false`

Determines if Pod annotations can be defined from a step's backend options.

### `WOODPECKER_BACKEND_K8S_SECCTX_NONROOT`

Expand Down

0 comments on commit fd94a2e

Please sign in to comment.