How to add task_group argument? [kedro-airflow] [task_group] [group_id]

[kevin-koga-mckinsey]

Description

I need to add a task_group to the kedro pipelines in order to create a better hierarchy of execution.

Context

I have a airflow pipeline that needs to have a task_group (with group_id, etc) due to order of execution progress. It is like a cross dependency between all input are ready to the modeling process starts.

Possible Implementation

Bind nodes tags to the task_group.group_id by ordering the tags list in alphabetical order then concatenating them in order to create a task_group.group_id! OR even getting the name of the referenced pipeline as a task_group.group_id.

Possible Alternatives

Adapting the generated project_name_dag.py and manually adding the groups.

Attention! This is not a complaint, this is just an idea ! : )

Happy to hear you back folks, cheers : )

[astrojuanlu]

Thanks @kevin-koga-mckinsey for making this suggestion! I think this resonates with your research @datajoely , in particular enabling users to manually group nodes.

@kevin-koga-mckinsey just FYI notice there's an ongoing PR by @sbrugman that adds some automatic grouping #241 but it's still automatic.