Package kedro.extras.datasets into its own kedro-datasets package

[idanov]

Problem

Currently Kedro as a framework and kedro.extras.datasets are one package and this has caused a number of issues:

• Kedro Framework was held back adding support for Python 3.9/3.10 by breaking changes in the datasets dependencies
• Adding support for newer versions of the datasets was held back by the need for more conservative versioning of the framework
• The difference of breaking change velocity between the Framework (more stable) and the datasets (less stable) meant a painful accounting of what was new in 0.18 and what was merely a change in a dataset

Staged approach

Ripping kedro.extras.datasets from the framework is itself a breaking change, but we can phase this out in two stages:

1. Make a separate package and alias everything in kedro.extras.datasets to that new package, including installing kedro[xxx]
2. When we ship 0.19, we can officially remove kedro.extras.datasets and now both kedro and kedro-datasets will be installed separately

Tasks for first stage (0.18.X)

• Create a new package kedro-datasets as part of kedro-plugins #1492
  • Copy all datasets to the new kedro-datasets package including the relevant extras_require=
    

    kedro/setup.py

    Lines 107 to 148 in 1b15589

    	extras_require = {
    	"api": _collect_requirements(api_require),
    	"biosequence": _collect_requirements(biosequence_require),
    	"dask": _collect_requirements(dask_require),
    	"docs": [
    	"docutils==0.16",
    	"sphinx~=3.4.3",
    	"sphinx_rtd_theme==0.4.1",
    	"nbsphinx==0.8.1",
    	"nbstripout~=0.4",
    	"recommonmark==0.7.1",
    	"sphinx-autodoc-typehints==1.11.1",
    	"sphinx_copybutton==0.3.1",
    	"ipykernel>=5.3, <7.0",
    	],
    	"geopandas": _collect_requirements(geopandas_require),
    	"matplotlib": _collect_requirements(matplotlib_require),
    	"holoviews": _collect_requirements(holoviews_require),
    	"networkx": _collect_requirements(networkx_require),
    	"notebook_templates": ["nbconvert>=5.3.1, <6.0", "nbformat~=4.4"],
    	"pandas": _collect_requirements(pandas_require),
    	"pillow": _collect_requirements(pillow_require),
    	"plotly": _collect_requirements(plotly_require),
    	"profilers": ["memory_profiler>=0.50.0, <1.0"],
    	"redis": _collect_requirements(redis_require),
    	"spark": _collect_requirements(spark_require),
    	"tensorflow": _collect_requirements(tensorflow_required),
    	"yaml": _collect_requirements(yaml_require),
    	**api_require,
    	**biosequence_require,
    	**dask_require,
    	**geopandas_require,
    	**matplotlib_require,
    	**holoviews_require,
    	**networkx_require,
    	**pandas_require,
    	**pillow_require,
    	**plotly_require,
    	**spark_require,
    	**tensorflow_required,
    	**yaml_require,
    	}

  • Move all dataset tests to kedro-datasets
• Setup CI/CD with automatic releases to PyPI for kedro-datasets and release the first version [Parent] #1493
  - [x] Namespace kedro-datasets  #1693
• [User Research] How do users feel about moving kedro.extras.datasets to kedro-datasets #1850
• Make Kedro instantiate datasets from kedro-datasets with higher priority than kedro.extras.datasets #1494
• Release version 1.0.0 for kedro-datasets and test it's fully working #1711
  -[ ] Add deprecation warnings to all the kedro.extras.datasets to explain they're moving #1501 This no longer needs to be done after it was decided to namespace kedro-datasets
• Update all starters to install kedro-datasets[xxxxx] instead of kedro[xxxx] #1495
• Update contribution doc and PR template to make it clear to make dataset changes in kedro-datasets #1497
• How to make the documentations works for kedro-datasets? #1651
• Update documentation from pip install kedro[...] to the new pip install kedro kedro-datasets[...] #2101

Tasks for second stage (preparing release 0.19)

• Make Kedro not instantiate datasets from kedro.extras.datasets #2125
• Remove all datasets related extras_require= from kedro #2127
• Remove kedro.extras.datasets from Kedro #2126
• Add kedro-datasets in kedro[all] #2128
• Rename all mentions of DataSet to Dataset #2129
• Check if improvements are needed for test_requirements.txt setup #1498
• Clean up kedro's CI config and Makefile #1502

[antonymilne]

This is great! 🚀 Just to wanted another motivation for doing this, and also some more tasks that will be needed. Datasets and their dependencies are the number 1 cause of random CI failures which have slowed us down quite a lot. I presume that your intention was that all the dataset tests should move to kedro-datasets also? If so then I'd add to "tasks for first stage":

• Move all dataset tests to kedro-datasets
• Try and figure out a nicer way of installing all the dependencies listed in the setup.py file rather than needing to maintain a separate test_requirements.txt file there
• Clean up kedro's test_requirements.txt to remove requirements that aren't needed any more
• Clean up kedro's CI config and Makefile. e.g. no need to conda install pytables any more or have the separate make test-no-spark. There's quite a few dataset-specific workarounds that we could remove

Also I think we should have:

• Add deprecation warnings to all the kedro.extras.datasets (or the AbstractDataset classes? Not sure whether they're also moving) to explain they're moving

One small question: were you thinking of kedro-datasets living in kedro-plugins or a whole new repo? It's not a plugin in the strict sense of the word since it doesn't use entrypoints, but maybe it's easier to maintain if it's in the monorepo.

[merelcht]

One small question: were you thinking of kedro-datasets living in kedro-plugins or a whole new repo? It's not a plugin in the strict sense of the word since it doesn't use entrypoints, but maybe it's easier to maintain if it's in the monorepo.

I like this suggestion! True kedro-datasets isn't technically a plugin, but if we add it to kedro-plugins we can make use of the already existing CI setup and add in automatic push to pypi for all plugins in one go.

Some more things we should do as part of never work in kedro.extras.datasets is make sure this is clear to open source contributors by adding it to the contribution guide, and at the top of every dataset class and perhaps also in the PR template.

[noklam]

Follow up on some unresolved issues discussed in kedro-org/kedro-plugins#38

1. How will docs work for kedro-datasets? We will need to generate documentation (at least the dataset part) from kedro-datasets repo.
2. Deepyaman has an idea with namespace package (i.e. we keep the namespace kedro.extra.datasets in the kedro-datasets repo.

[idanov]

Here's just a wild idea, I've played a bit with Python's meta_path and came up with an interesting PoC about aliasing imports globally as shown here:

import sys
import importlib
from importlib.abc import MetaPathFinder

class Custom(MetaPathFinder):
    def find_spec(fullname, path, target=None):
        if not 'kedro.datasets' in fullname:
            return None
       return importlib.util.find_spec('kedro_datasets')

sys.meta_path.append(Custom)

import kedro.datasets as kd

The code here allows you to import anything from kedro_datasets as if it were kedro.datasets. I did not explore it enough for potential problems, but maybe we can investigate whether we can somehow create the alias kedro.datasets work out of the box for any Kedro project? WDYT?

The modification of the meta_path can happen in configure_project so it works in all entrypoints. I am not sure how it will work with language servers or IDE support though...

[deepyaman]

@idanov That's very interesting, and I didn't know about this functionality. That being said, if we're going to investigate this, I think we should prioritize spending some time to make it work using namespace packages, because that would ideally be the "standard" solution to the problem (that doesn't involve path hacking). I just saw some of the discussion around why namespace packages may not work in #1758, and even though I don't fully understand all of the reasons having not spent time on it (except for some of the challenges with pip install -e), this could be a good alternative.

[merelcht]

Closing this because all sub-tasks are done and Kedro 0.19.0 and kedro-datasets 2.0.0 will be released soon.