Replace and deprecate `DataSet` use in class names #2500

deepyaman · 2023-04-10T14:35:34Z

NOTE: Kedro datasets are moving from kedro.extras.datasets to a separate kedro-datasets package in
kedro-plugins repository. Any changes to the dataset implementations
should be done by opening a pull request in that repository.

Description

Deprecate core "DataSet" names and warn about the impending rename to "Dataset".

Once 0.19.0 is released, the deprecated aliases (and even the utils, if so desired) can be removed.

Development notes

Checklist

Read the contributing guidelines
Opened this PR as a 'Draft Pull Request' if it is work-in-progress
Updated the documentation to reflect the code changes
Added a description of this change in the RELEASE.md file
Added tests to cover my changes

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

merelcht · 2023-04-11T08:59:33Z

@deepyaman I'm not sure it actually makes sense to the rename on the Kedro framework side. It's a breaking change and we're removing all datasets from here when releasing 0.19.0 anyway.

deepyaman · 2023-04-11T13:21:29Z

@deepyaman I'm not sure it actually makes sense to the rename on the Kedro framework side. It's a breaking change and we're removing all datasets from here when releasing 0.19.0 anyway.

@merelcht Won't the datasets in kedro.io.core stay in core framework, even in 0.19.0? Looking at #2129, are you suggesting that we would keep DataSet in the core repo and Dataset in plugins? Or that we change DataSet -> Dataset in core repo without deprecation in 0.19.0, since it's a breaking release anyway?

merelcht · 2023-04-11T13:35:00Z

Ignore me, I should've had my morning coffee before commenting 😂

We should rename all of them, but it will be a breaking change so maybe it's better to wait until we start work on 0.19.0 to make sure develop and main don't diverge too much until then.

merelcht · 2023-04-11T13:36:04Z

If we do a deprecation warning what will be users see and how often? It might be annoying if there's nothing they can do about it..

deepyaman · 2023-04-11T14:12:49Z

If we do a deprecation warning what will be users see and how often? It might be annoying if there's nothing they can do about it..

If this functionality were to be released, they would get a warning if they used MemoryDataSet (or pandas.CSVDataSet in an updated kedro-datasets); but they could change their code to MemoryDataset (or pandas.CSVDataset) and avoid the warning. So the idea would be, yes, they can do something about it, and they have a grace period until it's removed that their projects wouldn't be broken on a newer version.

kedro/utils.py

Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com>

…into rename-data-set

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

stichbury · 2023-06-13T11:48:13Z

You are truly the patron saint of consistency @deepyaman and I'm happy to approve this change because it seems wholesale and straightforward. However, I'm not able to speak for @merelcht and @idanov in terms of whether it's the right point to merge this change or the implications thereof. But if they're happy with the intent, I'm happy with the execution.

merelcht · 2023-06-13T14:23:58Z

kedro/utils.py

+                if alias is not None:
+                    warn(
+                        f"{mcs.__name__} has been renamed to {alias.__name__}, and the "
+                        f"alias will be removed Kedro 0.19.0",


Suggested change

f"alias will be removed Kedro 0.19.0",

f"alias will be removed in Kedro 0.19.0",

Ah, thanks! I fixed it in a separate place but forgot to add it back in here. :)

merelcht

Thanks for persisting with this @deepyaman ! I'm happy for this to get merged ⭐

stichbury

If it's OK with @merelcht it's a 👍 from me too.

astrojuanlu · 2023-06-15T15:45:53Z

RTD failing with:

sphinx.errors.SphinxWarning: /home/docs/checkouts/readthedocs.org/user_builds/kedro/checkouts/2500/kedro/extras/datasets/api/api_dataset.py:docstring of kedro.io.core.AbstractDataSet.exists:1:py:exc reference target not found: DatasetError

* Replace and deprecate `DataSet` use in class names * Replace another format string with an f-string * Perform deprecations for cached, lambda, and partitioned datasets * Deprecated `MemoryDataSet` in favor of `MemoryDataset` Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Fix keyword argument to specify metaclass on `CachedDataSet` * Fix reference to `PartitionedDataset` * Keep `AbstractDataSet` subscriptable * Update __init__.py files, __all__ definitions, etc * Warn of impending Kedro 0.19 (not abstract future) Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> * Update `VideoDataSet` to `VideoDataset` (and refs) * Add missing `kedro.utils.DeprecatedClassMeta` imps * Change deprecated references to `AbstractDataSet` Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Warn of impending Kedro 0.19 (not abstract future) * Rename `pandas.CSVDataSet` to `pandas.CSVDataset` * Fix some pylint errors and blacken code * Update `dask.ParquetDataSet` * Undo changes for `VideoDataSet`, inherit from new base * Undo changes to `APIDataSet`, inherit from new base * Fix some imports and missed references * Undo changes to `BioSequenceDataSet`, inherit from new base * Undo changes to Dask and Pandas datasets, inherit from new bases * Remove the `AbstractDataset` and `AbstractVersionedDataset` alias, update `kedro.io.core` * Undo changes in `kedro/extras/datasets` * Update branch * Change `DataSetError` to `DatasetError` * Remove deprecated aliases for Abstract*DataSet * Change `DataSetError` to `DatasetError` in tests/ * Change DataSet*Error to Dataset*Error in tests/ * Fix references to DataSet in a lot of tests * Change `CSVDataset` back to `CSBVDataSet` * Rename core datasets used across `tests` directory * Fix "Saving 'None' to a 'DataSet' is not allowed." messages * Fix `test_http_filesystem_no_versioning` everywhere * Fix removal of "data" * Deprecate `_SharedMemoryDataSet` in favor of `_SharedMemoryDataset` * Fix tests/pipeline/test_pipeline_from_missing.py * Fix list datasets test * Change patched IncrementalDataSet to IncrementalDataset * Fix default checkpoint dataset * Fix data catalog tests * Fix error message * Use `MemoryDataset`, not `MemoryDataSet`, by default * Use `MemoryDataset`, not `MemoryDataSet`, for missing datasets in data catalog * Rename DefaultDataSet key to DefaultDataset * Change `LambdaDataSet` to `LambdaDataset` in `test_node_run.py` * Update error message--but should I? * Update error message--but should I? * Update error message in kedro/io/core.py--but should I? * Update RELEASE.md * Fix remaining tests * Fix lint issues * Align capitalization * Add `DeprecatedClassMeta` tests from StackOverflow * Blacken kedro/utils.py * Ignore "No value for argument 'subclass' in unbound method call" * Rename 'foo' to 'value' to satisfy linter * Disable pylint messages on deprecated class definitions * Blacken kedro/utils.py * Wrap `kedro.io.core` to fix error deprecation * Simplify deprecation of error names to try to fix docs * Undo attempt to make docs pass Revert "Simplify deprecation of error names to try to fix docs" This reverts commit e9294be. Revert "Wrap `kedro.io.core` to fix error deprecation" This reverts commit db9b7ac. * Replace `DataSetError` with `DatasetError` in test * Add missing "in" to a `DeprecationWarning` message * Add "Dataset" versions of errors to `kedro.io` doc * Add updated "Dataset" names to `kedro.io.rst` and sort the entries * Add `_SharedMemoryDataset` to type targets in conf --------- Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* Replace and deprecate `DataSet` use in class names * Replace another format string with an f-string * Perform deprecations for cached, lambda, and partitioned datasets * Deprecated `MemoryDataSet` in favor of `MemoryDataset` Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Fix keyword argument to specify metaclass on `CachedDataSet` * Fix reference to `PartitionedDataset` * Keep `AbstractDataSet` subscriptable * Update __init__.py files, __all__ definitions, etc * Warn of impending Kedro 0.19 (not abstract future) Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> * Update `VideoDataSet` to `VideoDataset` (and refs) * Add missing `kedro.utils.DeprecatedClassMeta` imps * Change deprecated references to `AbstractDataSet` Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Warn of impending Kedro 0.19 (not abstract future) * Rename `pandas.CSVDataSet` to `pandas.CSVDataset` * Fix some pylint errors and blacken code * Update `dask.ParquetDataSet` * Undo changes for `VideoDataSet`, inherit from new base * Undo changes to `APIDataSet`, inherit from new base * Fix some imports and missed references * Undo changes to `BioSequenceDataSet`, inherit from new base * Undo changes to Dask and Pandas datasets, inherit from new bases * Remove the `AbstractDataset` and `AbstractVersionedDataset` alias, update `kedro.io.core` * Undo changes in `kedro/extras/datasets` * Update branch * Change `DataSetError` to `DatasetError` * Remove deprecated aliases for Abstract*DataSet * Change `DataSetError` to `DatasetError` in tests/ * Change DataSet*Error to Dataset*Error in tests/ * Fix references to DataSet in a lot of tests * Change `CSVDataset` back to `CSBVDataSet` * Rename core datasets used across `tests` directory * Fix "Saving 'None' to a 'DataSet' is not allowed." messages * Fix `test_http_filesystem_no_versioning` everywhere * Fix removal of "data" * Deprecate `_SharedMemoryDataSet` in favor of `_SharedMemoryDataset` * Fix tests/pipeline/test_pipeline_from_missing.py * Fix list datasets test * Change patched IncrementalDataSet to IncrementalDataset * Fix default checkpoint dataset * Fix data catalog tests * Fix error message * Use `MemoryDataset`, not `MemoryDataSet`, by default * Use `MemoryDataset`, not `MemoryDataSet`, for missing datasets in data catalog * Rename DefaultDataSet key to DefaultDataset * Change `LambdaDataSet` to `LambdaDataset` in `test_node_run.py` * Update error message--but should I? * Update error message--but should I? * Update error message in kedro/io/core.py--but should I? * Update RELEASE.md * Fix remaining tests * Fix lint issues * Align capitalization * Add `DeprecatedClassMeta` tests from StackOverflow * Blacken kedro/utils.py * Ignore "No value for argument 'subclass' in unbound method call" * Rename 'foo' to 'value' to satisfy linter * Disable pylint messages on deprecated class definitions * Blacken kedro/utils.py * Wrap `kedro.io.core` to fix error deprecation * Simplify deprecation of error names to try to fix docs * Undo attempt to make docs pass Revert "Simplify deprecation of error names to try to fix docs" This reverts commit e9294be. Revert "Wrap `kedro.io.core` to fix error deprecation" This reverts commit db9b7ac. * Replace `DataSetError` with `DatasetError` in test * Add missing "in" to a `DeprecationWarning` message * Add "Dataset" versions of errors to `kedro.io` doc * Add updated "Dataset" names to `kedro.io.rst` and sort the entries * Add `_SharedMemoryDataset` to type targets in conf --------- Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>

* Replace and deprecate `DataSet` use in class names (#2500) * Replace and deprecate `DataSet` use in class names * Replace another format string with an f-string * Perform deprecations for cached, lambda, and partitioned datasets * Deprecated `MemoryDataSet` in favor of `MemoryDataset` Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Fix keyword argument to specify metaclass on `CachedDataSet` * Fix reference to `PartitionedDataset` * Keep `AbstractDataSet` subscriptable * Update __init__.py files, __all__ definitions, etc * Warn of impending Kedro 0.19 (not abstract future) Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> * Update `VideoDataSet` to `VideoDataset` (and refs) * Add missing `kedro.utils.DeprecatedClassMeta` imps * Change deprecated references to `AbstractDataSet` Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Warn of impending Kedro 0.19 (not abstract future) * Rename `pandas.CSVDataSet` to `pandas.CSVDataset` * Fix some pylint errors and blacken code * Update `dask.ParquetDataSet` * Undo changes for `VideoDataSet`, inherit from new base * Undo changes to `APIDataSet`, inherit from new base * Fix some imports and missed references * Undo changes to `BioSequenceDataSet`, inherit from new base * Undo changes to Dask and Pandas datasets, inherit from new bases * Remove the `AbstractDataset` and `AbstractVersionedDataset` alias, update `kedro.io.core` * Undo changes in `kedro/extras/datasets` * Update branch * Change `DataSetError` to `DatasetError` * Remove deprecated aliases for Abstract*DataSet * Change `DataSetError` to `DatasetError` in tests/ * Change DataSet*Error to Dataset*Error in tests/ * Fix references to DataSet in a lot of tests * Change `CSVDataset` back to `CSBVDataSet` * Rename core datasets used across `tests` directory * Fix "Saving 'None' to a 'DataSet' is not allowed." messages * Fix `test_http_filesystem_no_versioning` everywhere * Fix removal of "data" * Deprecate `_SharedMemoryDataSet` in favor of `_SharedMemoryDataset` * Fix tests/pipeline/test_pipeline_from_missing.py * Fix list datasets test * Change patched IncrementalDataSet to IncrementalDataset * Fix default checkpoint dataset * Fix data catalog tests * Fix error message * Use `MemoryDataset`, not `MemoryDataSet`, by default * Use `MemoryDataset`, not `MemoryDataSet`, for missing datasets in data catalog * Rename DefaultDataSet key to DefaultDataset * Change `LambdaDataSet` to `LambdaDataset` in `test_node_run.py` * Update error message--but should I? * Update error message--but should I? * Update error message in kedro/io/core.py--but should I? * Update RELEASE.md * Fix remaining tests * Fix lint issues * Align capitalization * Add `DeprecatedClassMeta` tests from StackOverflow * Blacken kedro/utils.py * Ignore "No value for argument 'subclass' in unbound method call" * Rename 'foo' to 'value' to satisfy linter * Disable pylint messages on deprecated class definitions * Blacken kedro/utils.py * Wrap `kedro.io.core` to fix error deprecation * Simplify deprecation of error names to try to fix docs * Undo attempt to make docs pass Revert "Simplify deprecation of error names to try to fix docs" This reverts commit e9294be. Revert "Wrap `kedro.io.core` to fix error deprecation" This reverts commit db9b7ac. * Replace `DataSetError` with `DatasetError` in test * Add missing "in" to a `DeprecationWarning` message * Add "Dataset" versions of errors to `kedro.io` doc * Add updated "Dataset" names to `kedro.io.rst` and sort the entries * Add `_SharedMemoryDataset` to type targets in conf --------- Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Update `DataSetError` to `DatasetError` in test_api_dataset.py --------- Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Co-authored-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

deepyaman added 7 commits April 10, 2023 10:30

Replace and deprecate DataSet use in class names

bcf66a5

Replace another format string with an f-string

87172c0

Perform deprecations for cached, lambda, and partitioned datasets

af5f4a1

Deprecated MemoryDataSet in favor of MemoryDataset

53eaf4c

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

Fix keyword argument to specify metaclass on CachedDataSet

6377e12

Fix reference to PartitionedDataset

82b99f1

Keep AbstractDataSet subscriptable

a8a0da6

merelcht reviewed Apr 12, 2023

View reviewed changes

kedro/utils.py Outdated Show resolved Hide resolved

merelcht reviewed Apr 12, 2023

View reviewed changes

kedro/utils.py Outdated Show resolved Hide resolved

deepyaman and others added 16 commits April 27, 2023 14:24

Update __init__.py files, __all__ definitions, etc

0f1b3be

Merge branch 'main' into rename-data-set

ccdf78e

Warn of impending Kedro 0.19 (not abstract future)

e738c32

Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com>

Update VideoDataSet to VideoDataset (and refs)

7f02f86

Merge branch 'rename-data-set' of https://github.com/kedro-org/kedro …

64b1a76

…into rename-data-set

Add missing kedro.utils.DeprecatedClassMeta imps

cae65ca

Change deprecated references to AbstractDataSet

904b5d6

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

Warn of impending Kedro 0.19 (not abstract future)

f5654a7

Rename pandas.CSVDataSet to pandas.CSVDataset

139c4b0

Fix some pylint errors and blacken code

686db58

Update dask.ParquetDataSet

f3359fc

Undo changes for VideoDataSet, inherit from new base

2987744

Undo changes to APIDataSet, inherit from new base

5d92194

Fix some imports and missed references

4fde3d7

Undo changes to BioSequenceDataSet, inherit from new base

c6c2177

Undo changes to Dask and Pandas datasets, inherit from new bases

1ef2ee2

deepyaman added 3 commits June 7, 2023 15:21

Merge branch 'main' into rename-data-set

5ef91bb

Merge branch 'main' into rename-data-set

5ad59dd

Merge branch 'main' into rename-data-set

2d41aec

deepyaman force-pushed the rename-data-set branch from c0ebba2 to 2d41aec Compare June 12, 2023 17:17

deepyaman mentioned this pull request Jun 12, 2023

Deprecate dataset errors with module __getattr__ #2673

Merged

5 tasks

merelcht removed the request for review from idanov June 13, 2023 14:19

merelcht reviewed Jun 13, 2023

View reviewed changes

merelcht approved these changes Jun 13, 2023

View reviewed changes

stichbury approved these changes Jun 13, 2023

View reviewed changes

Merge branch 'main' into rename-data-set

582bf9a

deepyaman added 4 commits June 15, 2023 11:55

Merge branch 'main' into rename-data-set

71b7a6a

Replace DataSetError with DatasetError in test

3af6a03

Add missing "in" to a DeprecationWarning message

355c41c

Add "Dataset" versions of errors to kedro.io doc

5b17331

deepyaman requested review from yetudada and astrojuanlu as code owners June 15, 2023 16:23

deepyaman added 2 commits June 15, 2023 12:53

Add updated "Dataset" names to kedro.io.rst and sort the entries

2120812

Add _SharedMemoryDataset to type targets in conf

32b2bb5

deepyaman merged commit 1fb1ff7 into main Jun 16, 2023

deepyaman deleted the rename-data-set branch June 16, 2023 04:27

deepyaman mentioned this pull request Jun 20, 2023

test(datasets): update the "no versioning" message kedro-org/kedro-plugins#245

Merged

4 tasks

PtrBld mentioned this pull request Jun 28, 2023

feat(datasets): Migrated PartitionedDataSet and IncrementalDataSet from main repository to kedro-datasets kedro-org/kedro-plugins#253

Merged

4 tasks

noklam mentioned this pull request Jun 28, 2023

Plan for Renaming DataSet to Dataset #2740

Closed

6 tasks

astrojuanlu mentioned this pull request Jun 28, 2023

Library should use either "data set" or "dataset" consistently #533

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace and deprecate `DataSet` use in class names #2500

Replace and deprecate `DataSet` use in class names #2500

deepyaman commented Apr 10, 2023 •

edited

Loading

merelcht commented Apr 11, 2023

deepyaman commented Apr 11, 2023 •

edited

Loading

merelcht commented Apr 11, 2023

merelcht commented Apr 11, 2023

deepyaman commented Apr 11, 2023

stichbury commented Jun 13, 2023

merelcht Jun 13, 2023

deepyaman Jun 14, 2023

merelcht left a comment

stichbury left a comment

astrojuanlu commented Jun 15, 2023

	f"alias will be removed Kedro 0.19.0",
	f"alias will be removed in Kedro 0.19.0",

Replace and deprecate DataSet use in class names #2500

Replace and deprecate DataSet use in class names #2500

Conversation

deepyaman commented Apr 10, 2023 • edited Loading

Description

Development notes

Checklist

merelcht commented Apr 11, 2023

deepyaman commented Apr 11, 2023 • edited Loading

merelcht commented Apr 11, 2023

merelcht commented Apr 11, 2023

deepyaman commented Apr 11, 2023

stichbury commented Jun 13, 2023

merelcht Jun 13, 2023

Choose a reason for hiding this comment

deepyaman Jun 14, 2023

Choose a reason for hiding this comment

merelcht left a comment

Choose a reason for hiding this comment

stichbury left a comment

Choose a reason for hiding this comment

astrojuanlu commented Jun 15, 2023

Replace and deprecate `DataSet` use in class names #2500

Replace and deprecate `DataSet` use in class names #2500

deepyaman commented Apr 10, 2023 •

edited

Loading

deepyaman commented Apr 11, 2023 •

edited

Loading