Using kfp v2.0+ not able to add pod labels in kubeflow pipeline workflow

[yifengwang66]

Discussed in #11014

^{Originally posted by emilyyujieli July 17, 2024}
Environment
KFP version: 2.2.0
kubeflow verison: 1.8
KFP SDK version: 2.7.0

Background
We are trying to add istio sidecar in kubeflow pipeline pod then we can use service mesh to do authentication due to project requirement.

Issue
If we use kfp v1 and add parameter kubernetes.add_pod_label(task=task,label_key="sidecar.istio.io/inject",label_value="true") in the code, then use KFP SDK v1 to compile the script. label can be injected in related workflow and run successfully.
However if we add the same parammeter in kfp v2 script and use kfp SDK v2 to compile, l can see the label in compiled file showed as below

platform_spec:
  platforms:
    kubernetes:
      deploymentSpec:
        executors:
          exec-load:
            podMetadata:
              labels:
                sidecar.istio.io/inject: 'true'

However when l try to upload the compiled file and execute it via kubeflow pipeline, the label doesn't be injected in related workflow.

Question
If this parameter kubernetes.add_pod_label still available in KFP v2? How can l add label in kubeflow pipeline workflow yaml if using kfp v2 since not able to find any information on kubeflow official doucument.

Regards & thanks!

[simwijs-fmf]

In kfp-kubernetes (since version 1.3.0) you have Pod Labels and Annotations

Example usage (for Cloud TPU pods)

@dsl.component(base_image="tensorflow/tensorflow:2.18.0-gpu") 
def check_tpu_availability() -> list:
    import tensorflow as tf
    devices = tf.config.list_logical_devices()
    devices = [device for device in devices]
    return devices
 

@dsl.pipeline(name="TF-Device-Check")
def tpu_check_pipeline():
    task = check_tpu_availability()

    kubernetes.add_pod_annotation(
        task,
        annotation_key='tf-version.cloud-tpus.google.com',
        annotation_value='2.18.0',
    )

This however requires kubeflow version 1.9 to support it (SDK version 2.2), otherwise you'll get a nasty Exit Code 1 when you try to launch a pod that upon inspection looks like

failed to unmarshal Kubernetes config, error: unknown field "podMetadata" in kfp_kubernetes.KubernetesExecutorConfig