SparkKubernetesOperator: delete_on_termination & get_logs not working

[yyu233]

from airflow import DAG
from datetime import timedelta, datetime
from airflow.providers.cncf.kubernetes.operators.spark_kubernetes import SparkKubernetesOperator
from airflow.providers.cncf.kubernetes.sensors.spark_kubernetes import SparkKubernetesSensor
from airflow.models import Variable
from kubernetes.client import models as k8s
from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator

with DAG(
   dag_id='example_k8s_spark_op_py_pi',
   schedule=None,
   start_date=datetime(2024, 11, 19),
   catchup=False,
   tags=['example']
) as dag:
   t1 = SparkKubernetesOperator(
       task_id='spark-op-py-pi',
       application_file="/app/spark-op-pi.yaml",
       namespace="test-af-spark-jobs",
       kubernetes_conn_id="k8s_conn",
       do_xcom_push=True,
       delete_on_termination=False,
       log_events_on_failure=True,
       get_logs=True
   )

airflow version: 2.9.3

I'm not sure if i miss some configuration or there is a bug.

The worker pod is deleted even though i set the delete_on_termination to false.

From web ui, the Logs tab of task displays:

example-k8s-spark-op-py-pi-spark-op-py-pi-s4dbanma
*** Could not read served logs: HTTPConnectionPool(host='example-k8s-spark-op-py-pi-spark-op-py-pi-s4dbanma', port=8793): Max retries exceeded with url: /log/dag_id=example_k8s_spark_op_py_pi/run_id=manual__2024-11-19T10:11:43.433902+00:00/task_id=spark-op-py-pi/attempt=1.log (Caused by NameResolutionError("<urllib3.connection.HTTPConnection object at 0x7f4088fe9c10>: Failed to resolve 'example-k8s-spark-op-py-pi-spark-op-py-pi-s4dbanma' ([Errno -2] Name or service not known)"))