Airflow DAG Run Delay: running State Persists After Tasks Complete

[dineshkumar20]

Apache Airflow version

Other Airflow 2 version (please specify below)

If "Other Airflow 2 version" selected, which one?

2.8.4

What happened?

I’m encountering an issue in Airflow. I’m running 1,000 DAG runs per minute, and while tasks are completing quickly, the DAG run itself remains in a running state for a few minutes even after all tasks have finished. It takes about 5-6 minutes before the DAG run finally switches to success, which causes other DAG runs to get stuck in the queue.

Does anyone know why the DAG run remains in running even though all tasks have completed?

I noticed in the Airflow UI that the dag_run statuses seem to update in batches, with all statuses changing at the same time. Is there a configuration we can update to reduce this wait time and have the status change immediately?

What you think should happen instead?

Dag run should update it state to success immediately after finishing all tasks.

How to reproduce

Create a simple dag with 10 tasks and create 1000 dagruns per minute.

Operating System

Debian GNU/Linux 12

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[potiuk]

This is more of a discussion than issue. Converted it, hopefully someone will be able to help