Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature request: Group kedro nodes in same DAG task #138

Open
sbrugman opened this issue May 10, 2022 · 0 comments
Open

feature request: Group kedro nodes in same DAG task #138

sbrugman opened this issue May 10, 2022 · 0 comments

Comments

@sbrugman
Copy link
Contributor

The documentation reads:

Every kedro node is transformed into Airflow DAG task.

As software engineers, we may want to have multiple kedro nodes correspond to a single DAG task when logical separation of code in multiple nodes is preferred (e.g. for readability or reuse). This would also allow datasets of type "MemoryDataset", that would need grouping by default.

For this kedro-airflow-k8s could detect groups of nodes similar to spark grouping.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant