How to Return Task Value from Python Script (.py) Using DatabricksRunNowOperator

[big-c-note]

I am running a Databricks Workflow from airflow using the DatabricksRunNowOperator. I have set do_xcom_push=True. I have set the task to run a python file (.py file, not a notebook) in a Databricks Repo. That python script contains dbutils.notebook.exit(json.dumps(result)). My expectation would be that the value would appear in xcoms logs in my local Airflow, however I do not see it.

Relevant set up:

apache-airflow-providers-databricks==6.12.0

    task = DatabricksRunNowOperator(
        databricks_conn_id=conn_id,
        job_id=job_id,
        task_id=task_id,
        python_params=[
            table,
            "2024-08-01",
            "2024-08-02",
            100,
        ],
        do_xcom_push=True,
    )

xcom does not show any expected key value output. I realize this may be a Databricks question, but I figured I'd ask the community here in case someone knows.

[big-c-note]

This looks promising, will try tomorrow: https://blog.devgenius.io/send-information-from-databricks-to-airflow-810a7d49ff81