Airflow DAG pod_template

[InitRoot]

I'm having some issues running HELM and my DAGs to utilise a specific pod_template. I've configured the template in the values.yaml and can see the configmap being created fine.

## configs generating the `pod_template.yaml` file for `AIRFLOW__KUBERNETES__POD_TEMPLATE_FILE`
  ## [NOTE] the `dags.gitSync` values will create a git-sync init-container in the pod
  ## [NOTE] the `airflow.extraPipPackages` will NOT be installed
  kubernetesPodTemplate:

    ## the full content of the pod-template file (as a string)
    ## [NOTE] all other `kubernetesPodTemplate.*` are disabled when this is set
    stringOverride: |
      apiVersion: v1
      kind: Pod
      metadata:
        name: pod-template  
      spec:
      ++++++++++

I'm currently using the CeleryKubernetesExecutor as it fits my use case letting smaller non complex jobs run on Celery. I've confirmed that the pod_template.yaml is on my scheduler node:

airflow@airflow-cluster-scheduler-b774c695-svjk2:/opt/airflow$ ls -la
total 132
drwxrwxr-x 1 airflow root   4096 Jan  6 17:21 .
drwxr-xr-x 1 root    root   4096 Sep 20 19:58 ..
-rw------- 1 airflow root 110267 Jan  6 17:21 airflow.cfg
drwxr-xr-x 3    1000 1000   4096 Jan  6 17:20 dags
drwxrwxrwx 8    1000 1000   4096 Jan  6 17:22 logs
drwxr-xr-x 2 root    root   4096 Jan  6 17:21 pod_templates
airflow@airflow-cluster-scheduler-b774c695-svjk2:/opt/airflow$ 

My DAG configuration looks as follows:

from airflow.decorators import dag, task
from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator
from airflow.providers.postgres.hooks.postgres import PostgresHook
from kubernetes.client import models as k8s
from pendulum import datetime
from datetime import timedelta
import json
from airflow.providers.amazon.aws.hooks.s3 import S3Hook
import logging
import re

# Configuration
SCAN_IMAGE = "localhost:32000/disa:latest"
logger = logging.getLogger(__name__)

# Define container resources using k8s format
k8s_resources = k8s.V1ResourceRequirements(
    requests={"cpu": "2500m", "memory": "3Gi"},
    limits={"cpu": "4500m", "memory": "5Gi"}
)


TOOL_COMMANDS = {
    'test' : """
    echo "Hello World"
    """
    }

@dag(
    dag_id='test_pod_template',
    schedule=None,
    start_date=datetime(2024, 10, 22),
    catchup=False,
    tags=['asset-discovery', 'kubernetes'],
)
def test_pod_template():
    
    def create_tool_task(tool: str):
        """Create a KubernetesPodOperator task for a specific tool"""
        return KubernetesPodOperator(
            task_id=f"{tool}",
            image=SCAN_IMAGE,
            image_pull_policy="Never",
            reattach_on_restart="False",
            name=f"{tool}-task",
            is_delete_operator_pod=True,
            in_cluster=True,
            config_file = None,
            container_resources=k8s_resources,
            pod_template_file="/opt/airflow/pod_templates/pod_template.yaml",
            labels={"app": "test-pod-template", "component": tool},
            env_vars={
                "TOOL_NAME": tool,
                "CMD_EXEC": TOOL_COMMANDS[tool]
            },
            retries=1,
            retry_delay=timedelta(seconds=60),
        )

    # Create task instances
    test = create_tool_task('test')

    # Set dependencies
    test

dag = test_pod_template()

Using the below and running my test DAG I see the following:

[2025-01-06, 20:52:29 SAST] {pod_generator.py:559} WARNING - Model file /opt/***/pod_templates/pod_template.yaml does not exist
[2025-01-06, 20:52:29 SAST] {pod.py:1133} INFO - Building pod test-task-3r26oblm with labels: {'dag_id': 'test_pod_template', 'task_id': 'test', 'run_id': 'manual__2025-01-06T185226.0654920000-414426644', 'kubernetes_pod_operator': 'True', 'try_number': '1'}
[2025-01-06, 20:52:29 SAST] {pod.py:555} INFO - Found matching pod test-task-3r26oblm with labels {'airflow_kpo_in_cluster': 'True', 'airflow_version': '2.10.2', 'app': 'test-pod-template', 'component': 'test', 'dag_id': 'test_pod_template', 'kubernetes_pod_operator': 'True', 'run_id': 'manual__2025-01-06T185226.0654920000-414426644', 'task_id': 'test', 'try_number': '1'}
[2025-01-06, 20:52:29 SAST] {pod.py:556} INFO - `try_number` of task_instance: 1
[2025-01-06, 20:52:29 SAST] {pod.py:557} INFO - `try_number` of pod: 1
[2025-01-06, 20:52:29 SAST] {pod_manager.py:378} WARNING - Pod not yet started: test-task-3r26oblm
[2025-01-06, 20:52:34 SAST] {pod.py:982} INFO - Deleting pod: test-task-3r26oblm
[2025-01-06, 20:52:34 SAST] {taskinstance.py:340} ▶ Post task execution logs

I've previously run the same setup without having to copy the template onto my custom image, however, I wasn't using HELM. Am I missing something? Any guidance would be appreciated.

[lucasfonsecads]

do you continue with the error? if yes, could you share your config file? and how do you start your project