Explore monorepo approaches?

[astrojuanlu]

With the deprecation of micropackaging #3854, it would be good to explore monorepo approaches. What "monorepo" actually means depends on the use case.

[astrojuanlu]

One request that comes up from time to time is "1 Kedro project, different groups of dependencies". At the moment, 1 Kedro project = 1 Python package, therefore having completely disjoint dependencies is not possible. There are several possible solutions:

• 1 Python package
  • one could have a very minimal set of common dependencies (say, just kedro>=0.19.6) and then groups of dependencies per use case using PEP 621 [project.optional-dependencies].
  • There's also the draft PEP 735 about dependency groups (to be submitted in 2 weeks?)
• 1 "root" Python package, other connected packages
  • There's uv workspaces (akin to Cargo monorepos) https://docs.astral.sh/uv/concepts/workspaces/

[astrojuanlu]

Requested clarification on "non-workspaces" astral-sh/uv#5605 (comment)

[astrojuanlu]

Beyond the "different groups of dependencies" use case, today a user asked about having different packages under src:

my_project
├── src
|   ├── use_case_1
|       ├── __init__.py
|       ├── __main__.py
|       ├── pipelines
|       |   ├── use_case_1_data_engineering
|       |   ├── use_case_1_modeling
|   ├── use_case_2
|       ├── __init__.py
|       ├── __main__.py
|       ├── pipelines
|       |   ├── use_case_2_data_ingestion
|       |   ├── use_case_2_data_science

I tried this with both setuptools and PDM build backends and it worked like a charm (hatchling on the other hand chocked). Sample pyproject.toml:

[project]
name = "multi"
version = "0.1.0"
readme = "README.md"
requires-python = ">=3.11"
dependencies = []

[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

However, I'm sure Kedro has some hardcoded logic that looks into a specific path inside src/. That logic also enables kedro pipeline create and other commands that manipulate the source code. So supporting this fully in Kedro might require some thought.

[astrojuanlu]

To note, PEP 735 (Dependency Groups) is now accepted. https://peps.python.org/pep-0735/

[astrojuanlu]

uv and tox already shipped support for it

...but pip hasn't, and might never do it pypa/pip#12963 (comment)

[astrojuanlu]

Useful ecosystem overview of how other languages do this https://gist.github.com/konstin/6d04f111563641beb10facb617fe0eb3

[astrojuanlu]

Yet another user concerned about the 1 Kedro pipeline = 1 set of requirements https://kedro.hall.community/evaluating-kedro-for-data-engineering-processes-qFYCwWk5VKQh