DAG-level failure callback to reduce notification overload

[seifrajhi]

Hi community,

I would like to propose the addition of a DAG-level failure callback feature to Apache Airflow. Currently, we have on_success_callback and on_failure_callback at the task level, which are invoked when a task succeeds or fails, respectively (documentation).

In scenarios where a DAG contains multiple tasks, a failure in several tasks results in multiple notifications. For instance, in my current DAG, I have around 20 tasks, and when they fail, I receive 20 separate notifications. This can be overwhelming and makes it difficult to manage and respond to failures efficiently.

So I think introducing a DAG-level failure callback that is invoked when any task within the DAG fails. This callback would allow us to handle failures at the DAG level, avoiding alerts fatigue by reducing the number of notifications.

[potiuk]

Sure. You can add it if you want. I think in Airflow 3 that is currently developed, it will be easier, because we will be adding a mechanism where callbacks will have to be "distributed", so far problem with DAG level callback was that we did not know in the task whether the whole "dag" succeeded or failed - and callbacks were executed in tasks. So you can definitely contribute that if you want - join the devlist, and our airflow 3 discussions - you can even propose it as a topic for dev3 calls (see devlist of Airflow for all the Airflow 3 devlist messages).