Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rendered SQL is not available for failed runs. #369

Closed
saikotcharlakota opened this issue Jul 14, 2023 · 5 comments · Fixed by #671
Closed

Rendered SQL is not available for failed runs. #369

saikotcharlakota opened this issue Jul 14, 2023 · 5 comments · Fixed by #671
Milestone
1.3.0

Comments

@saikotcharlakota
Copy link

saikotcharlakota commented Jul 14, 2023 •
edited
Loading

Airflow Rendered template is empty if the task if failed. I wish to see the rendered query/test that was executed resulting in the failure.

IMO, this is because store_compiled_sql is called after the error/exception check local.py file.

exception_handling methods simply raises AirflowException in case of error resulting rendered template as empty.
@jlaneve jlaneve mentioned this issue Jul 24, 2023
Closed
2 tasks
@dosubot DosuBot
Copy link

dosubot bot commented Nov 9, 2023

Hi, @saikotcharlakota! I'm Dosu, and I'm helping the Cosmos team manage their backlog. I wanted to let you know that we are marking this issue as stale.

From what I understand, you reported an issue regarding the rendered SQL not being available for failed runs in Airflow. It seems that the problem lies in the store_compiled_sql function being called after the error/exception check, which results in an empty rendered template.

Since there hasn't been any activity or comments on this issue, we wanted to check with you if it is still relevant to the latest version of the Cosmos repository. If it is, please let the Cosmos team know by commenting on the issue. Otherwise, feel free to close the issue yourself, or it will be automatically closed in 7 days.

Thank you for your contribution, and please don't hesitate to reach out if you have any further questions or concerns.

@dosubot dosubot bot added the stale Issue has not had recent activity or appears to be solved. Stale issues will be automatically closed label Nov 9, 2023
@dosubot DosuBot
Copy link

dosubot bot commented Nov 9, 2023

Hi, @saikotcharlakota! I'm Dosu, and I'm helping the Cosmos team manage their backlog. I wanted to let you know that we are marking this issue as stale.

From what I understand, you reported an issue regarding the rendered SQL not being available for failed runs in Airflow. It seems that the problem lies in the store_compiled_sql function being called after the error/exception check, which results in an empty rendered template.

Since there hasn't been any activity or comments on this issue, we wanted to check with you if it is still relevant to the latest version of the Cosmos repository. If it is, please let the Cosmos team know by commenting on the issue. Otherwise, feel free to close the issue yourself, or it will be automatically closed in 7 days.

Thank you for your contribution, and please don't hesitate to reach out if you have any further questions or concerns.

@saikotcharlakota
Copy link
Author

Hello,
I tested this issue and it still exists.

DBT: 1.5.1
Cosmos: 1.2.0
Airflow: 2.5.1

@dosubot dosubot bot removed the stale Issue has not had recent activity or appears to be solved. Stale issues will be automatically closed label Nov 9, 2023
@agreenburg
Copy link
Contributor

@tatiana I think this would be a very helpful thing to address and I would be happy to provide a submission.

@tatiana tatiana added this to the 1.3.0 milestone Nov 14, 2023
@tatiana
Copy link
Collaborator

tatiana commented Nov 14, 2023

Just read this issue, thanks for looping me in @agreenburg , and thanks for reporting it @saikotcharlakota !

It is a relevant improvement to Cosmos, and we'd appreciate if you'd like to contribute to fix this, @agreenburg !

agreenburg pushed a commit to agreenburg/astronomer-cosmos that referenced this issue Nov 14, 2023
Issue astronomer#369 - store compiled SQL before raising exception on…
74bcd54 
… failure
agreenburg pushed a commit to agreenburg/astronomer-cosmos that referenced this issue Nov 14, 2023
Issue astronomer#369 - fix issue where compiled_sql was getting clobb…
e9caebd 
…ered by addition of full_refresh to template fields
@agreenburg agreenburg mentioned this issue Nov 14, 2023
Merged
2 tasks
agreenburg pushed a commit to agreenburg/astronomer-cosmos that referenced this issue Nov 14, 2023
Issue astronomer#369 - update unit test for template fields
d2d0d47
tatiana pushed a commit that referenced this issue Nov 14, 2023
@agreenburg
Store compiled_sql even when task fails (fixes issue #369) (#671)
ee91ece 
Update `DbtLocalBaseOperator` code to store `compiled_sql` prior to
exception handling so that when a task fails, the `compiled_sql` can still
be reviewed.

In the process found and fixed a related bug where `compiled_sql` was
being dropped on some operations due to the way that the `full_refresh`
field was being added to the `template_fields`.

Closes #369

Fixes bug introduced in
#623 where
compiled_sql was being lost in `DbtSeedLocalOperator` and
`DbtRunLocalOperator`

Co-authored-by: Andrew Greenburg <agreenburg@vergeventures.net>
tatiana pushed a commit that referenced this issue Nov 15, 2023
@agreenburg @tatiana
Store compiled_sql even when task fails (fixes issue #369) (#671)
f28f18d 
Update `DbtLocalBaseOperator` code to store `compiled_sql` prior to
exception handling so that when a task fails, the `compiled_sql` can still
be reviewed.

In the process found and fixed a related bug where `compiled_sql` was
being dropped on some operations due to the way that the `full_refresh`
field was being added to the `template_fields`.

Closes #369

Fixes bug introduced in
#623 where
compiled_sql was being lost in `DbtSeedLocalOperator` and
`DbtRunLocalOperator`

Co-authored-by: Andrew Greenburg <agreenburg@vergeventures.net>
(cherry picked from commit ee91ece)
tatiana pushed a commit that referenced this issue Nov 15, 2023
@agreenburg @tatiana
Store compiled_sql even when task fails (fixes issue #369) (#671)
93396cd 
Update `DbtLocalBaseOperator` code to store `compiled_sql` prior to
exception handling so that when a task fails, the `compiled_sql` can still
be reviewed.

In the process found and fixed a related bug where `compiled_sql` was
being dropped on some operations due to the way that the `full_refresh`
field was being added to the `template_fields`.

Closes #369

Fixes bug introduced in
#623 where
compiled_sql was being lost in `DbtSeedLocalOperator` and
`DbtRunLocalOperator`

Co-authored-by: Andrew Greenburg <agreenburg@vergeventures.net>
(cherry picked from commit ee91ece)
arojasb3 pushed a commit to arojasb3/astronomer-cosmos that referenced this issue Jul 14, 2024
@agreenburg @arojasb3
Store compiled_sql even when task fails (fixes issue astronomer#369) (a…
077728b 
…stronomer#671)

Update `DbtLocalBaseOperator` code to store `compiled_sql` prior to
exception handling so that when a task fails, the `compiled_sql` can still
be reviewed.

In the process found and fixed a related bug where `compiled_sql` was
being dropped on some operations due to the way that the `full_refresh`
field was being added to the `template_fields`.

Closes astronomer#369

Fixes bug introduced in
astronomer#623 where
compiled_sql was being lost in `DbtSeedLocalOperator` and
`DbtRunLocalOperator`

Co-authored-by: Andrew Greenburg <agreenburg@vergeventures.net>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
1.3.0
3 participants
@tatiana @agreenburg @saikotcharlakota