Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Deterministic kedro-airflow DAGs #380

Closed
sbrugman opened this issue Oct 9, 2023 · 1 comment · Fixed by #525
Closed

Deterministic kedro-airflow DAGs #380

sbrugman opened this issue Oct 9, 2023 · 1 comment · Fixed by #525
Labels
Community Issue/PR opened by the open-source community enhancement New feature or request
Milestone
Fix and improve k...

Comments

@sbrugman
Copy link
Contributor

sbrugman commented Oct 9, 2023 •
edited
Loading

Description

The DAGs generated by kedro-airflow are not deterministic, as they are only topologically sorted.

Context

As a user that version controls the DAGs, I would like to have a diff only when the DAG changes (the ordering of the edges does not).

Diff in Edges

Avoiding situations where the diff consists only of edges ordered differently.

Possible Implementation

Sort the edges deterministically in kedro-airflow

Possible Alternatives

(Optional) Describe any alternative solutions or features you've considered.
@ankatiyar ankatiyar added the Community Issue/PR opened by the open-source community label Oct 9, 2023
@astrojuanlu astrojuanlu added the enhancement New feature or request label Oct 9, 2023
@sbrugman sbrugman mentioned this issue Oct 9, 2023
Closed
@noklam
Copy link
Contributor

noklam commented Jan 24, 2024

I can confirm this behavior with kedro-airflow 0.8.0 and kedro 0.19.2.

@noklam noklam mentioned this issue Jan 24, 2024
Merged
4 tasks
@astrojuanlu astrojuanlu moved this to In Review in Kedro Framework Feb 19, 2024
@github-project-automation github-project-automation bot moved this from In Review to Done in Kedro Framework Feb 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Community Issue/PR opened by the open-source community enhancement New feature or request
Projects
Kedro Framework
Archived in project
Milestone
Fix and improve kedro-airflow
Development

Successfully merging a pull request may close this issue.

feat(airflow): make sure kedro airflow create is deterministic kedro-org/kedro-plugins
5 participants
@astrojuanlu @sbrugman @noklam @merelcht @ankatiyar