Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: Add a guide for configuring the kubernetes executor #873

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
+49 −0
Open

docs: Add a guide for configuring the kubernetes executor #873

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ae397e9
Add a guide for configuring the kubernetes executor
chrishronek Oct 11, 2023
052d9f5
Add create secret config to values.yaml
chrishronek Oct 17, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 47 additions & 0 deletions airflow/plural/docs/kubernetes-executor.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
## Configuring KubernetesExecutor

The [KubernetesExecutor](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/kubernetes.html)
in Apache Airflow is a valuable choice for specific use cases and scenarios. It offers several advantages that make it a
suitable execution engine for running your Airflow tasks.

### edit values.yaml

You'll then want to edit `airflow/helm/airflow/values.yaml` in your installation repo with something like:

```yaml
airflow:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there's also an annoying createPluralSecret: true config variable needed here

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@michaeljguarino - just added it (I think). Can you confirm that I got it right?

airflow:
airflow:
config:
AIRFLOW__KUBERNETES__WORKER_PODS_CREATION_BATCH_SIZE: 5
executor: KubernetesExecutor
kubernetesPodTemplate:
resources:
limits:
cpu: 1
memory: 1Gi
requests:
cpu: 0.5
memory: 512Mi
flower:
enabled: false
redis:
enabled: false
workers:
enabled: false
```

The resources that you configure in the `kubernetesPodTemplate` section will determine the amount of resources that will
be available to your Airflow tasks, so if you get SIGTERM errors from your Airflow tasks, you may need to increase the
default cpu/memory and limits accordingly.

Additionally, we recommend to increase the [worker_pods_creation_batch_size](https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/configurations-ref.html#worker-pods-creation-batch-size)
for your production workloads. This guide has it set to `5`, but you will need to experiment what works best for your
Airflow workloads to decrease any scheduling latency.

Alternatively, you should be able to do this in the configuration section for airflow in your plural console as well.

### redeploy

From there, you should be able to run `plural build --only airflow && plural deploy --commit "use kubernetesexecutor"`
to use the Airflow kubernetesexecutor.