-
Notifications
You must be signed in to change notification settings - Fork 14.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Setup/Teardown in nested Task Groups creating unwanted dependencies with downstream tasks #36345
Comments
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval. |
CC: @dstandish - I might take a close look (I want to learn more about the setup/teardown implementation) - possibly after holidays but If you happen to be around and know by heart that could be also helpful :) |
Looks like bug, thanks. |
When arrowing `group` >> `task`, the "leaves" of `group` are connected to `task`. When calculating leaves in the group, teardown tasks are ignored, and we recurse upstream to find non-teardowns. What was happening, and what this fixes, is you might recurse to a work task that already has another non-teardown downstream in the group. In that case you should ignore the work task (because it already has a non-teardown descendent). Resolves apache#36345
When arrowing `group` >> `task`, the "leaves" of `group` are connected to `task`. When calculating leaves in the group, teardown tasks are ignored, and we recurse upstream to find non-teardowns. What was happening, and what this fixes, is you might recurse to a work task that already has another non-teardown downstream in the group. In that case you should ignore the work task (because it already has a non-teardown descendent). Resolves #36345
When arrowing `group` >> `task`, the "leaves" of `group` are connected to `task`. When calculating leaves in the group, teardown tasks are ignored, and we recurse upstream to find non-teardowns. What was happening, and what this fixes, is you might recurse to a work task that already has another non-teardown downstream in the group. In that case you should ignore the work task (because it already has a non-teardown descendent). Resolves #36345 (cherry picked from commit 949fc57)
Apache Airflow version
2.8.0
If "Other Airflow 2 version" selected, which one?
No response
What happened?
The code below, in the "How to reproduce" section, produces the following DAG, I think there is an extra connection rendered between
test_task
andouter_end
.What you think should happen instead?
I would expect that
middle_end
is executed aftertest_group
and, thatouter_end
is executed aftertest_group2
. Sinceinner_end
is a teardown task it is not considered a leaf oftest_group
, thus the only leaf oftest_group
istest_task
. So, I would expect only the connection betweentest_task
andmiddle_end
, and not the connection betweentest_task
andouter_end
.How to reproduce
Ddecorate the following function to become a DAG and call it to create the DAG:
Operating System
Ubuntu 22.04.3 LTS
Versions of Apache Airflow Providers
No response
Deployment
Docker-Compose
Deployment details
No response
Anything else?
This seems to be just a rendering issue since I've done some simple tests and the tasks have been executed in order, but I'm not 100% sure.
Are you willing to submit PR?
Code of Conduct
The text was updated successfully, but these errors were encountered: