feat(datasets): Migrated `PartitionedDataSet` and `IncrementalDataSet` from main repository to kedro-datasets #253

PtrBld · 2023-06-27T13:53:39Z

Description

As discussed the PartitionedDataSet and IncrementalDataSet should be moved to this repository. Issue #249.

Development notes

DataSets are added here kedro-datasets/kedro_datasets/partitions
Unit Tests are added here kedro-datasets/tests/partitions/

Checklist

Opened this PR as a 'Draft Pull Request' if it is work-in-progress
Updated the documentation to reflect the code changes
Added a description of this change in the relevant RELEASE.md file
Added tests to cover my changes

Signed-off-by: Peter Bludau <ptrbld.dev@gmail.com>

PtrBld · 2023-06-28T05:57:59Z

Hi, can I also work on #250 in this pull request or should I wait until this one gets accepted and then work on it in a new PR?

astrojuanlu · 2023-06-28T05:59:54Z

Hi @PtrBld , thanks for this! Let's have a separate PR for gh-250 , this one is already big

PtrBld · 2023-06-28T10:23:17Z

I just now recognized that kedro-org/kedro#2500 renamed all DataSets to datasets in the main reopository and therefore the tests fail with my code. Most datasets in kedro-datasets are not yet updated. For the two datasets here, should I already apply the new naming convention?

astrojuanlu · 2023-06-28T11:15:32Z

This renaming got us a bit all over the place and we're now discussing the way forward in kedro-org/kedro#2740. Thanks for your patience 🙏🏽

astrojuanlu · 2023-08-26T11:45:14Z

We're good to continue this @PtrBld, do you still have time to pursue it?

…edro-org#257) * Fix indentation Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Modify myst-parser Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Add newline Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Fix inconsistent extra["docs"] dependencies https://github.com/kedro-org/kedro/blob/2f3811aeab25c25fba8476f1152651753e85ee7e/setup.py#L88-L107 When we revamp our sphinx doc last time, the dependencies isn't sync, we should fix this so it is consistent. * Remove unnecessary type hints from doc string Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> --------- Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> Co-authored-by: Nok Lam Chan 陳諾林 <nok.lam.chan@quantumblack.com>

* Prepare kedro-dataset 1.4.2 Signed-off-by: Nok <nok.lam.chan@quantumblack.com> * Move release note to 1.4.2 Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> --------- Signed-off-by: Nok <nok.lam.chan@quantumblack.com> Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> Co-authored-by: Jannic Holzer <jannic.holzer@quantumblack.com> Co-authored-by: Jannic <37243923+jmholzer@users.noreply.github.com>

* Add format for ImageDataSet save When using the ImageDataSet with S3 storage you need to manually specify the format in the save_args dict despite the suffix in the file. This is since the save method on the PIL images are called with an open file. The PIL documentation states the following about the format argument: "If a file object was used instead of a filename, this parameter should always be used." Signed-off-by: Daniel Falk <daniel.falk.1@fixedit.ai> * Add unit test + lint Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update Release notes Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> --------- Signed-off-by: Daniel Falk <daniel.falk.1@fixedit.ai> Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Co-authored-by: Ankita Katiyar <110245118+ankatiyar@users.noreply.github.com> Co-authored-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

instructions on custom Airflow Operators Signed-off-by: Simon Brugman <sfbbrugman@gmail.com>

chore: bump semver Closes: kedro-org#269 Signed-off-by: Simon Brugman <sfbbrugman@gmail.com>

…o-org#259) * Create .pre-commit-config.yaml * Update .pre-commit-config.yaml * remove pre-commit-config from each plugin * fix relative paths * add exclude regex * Update .pre-commit-config.yaml * rework pre-commit * test * Revert "test" This reverts commit edf731e. * Update Makefile * Update .pre-commit-config.yaml * Update .pre-commit-config.yaml * Update .pre-commit-config.yaml * Update .pre-commit-config.yaml * TEST * Update .pre-commit-config.yaml * fix isort * undo test * lint * add secret scan and bandit again * pass_filenames: true * Update check-plugin.yml * Revert "pass_filenames: true" This reverts commit 1e85806. * disable import errors * lint * Update .pre-commit-config.yaml * Update .pre-commit-config.yaml * isort * try running hooks seperatley * revert telemetry isort * fix ga yaml * Update check-plugin.yml * add hook stage * changes based on review * add features and tests for docker, airflow and telemetry * Update Makefile * indent * lint * lint for airflow and docker * Update check-plugin.yml * undo plugin lint

…oml (kedro-org#263) * Chore: Move test requirements to optional dependencies in pyproject.toml Signed-off-by: Simon Brugman <sfbbrugman@gmail.com> * Chore: Refactor setup.py for static dependencies Signed-off-by: Simon Brugman <sfbbrugman@gmail.com> * Chore: Replace setup.py with pyproject.toml Signed-off-by: Simon Brugman <sfbbrugman@gmail.com> * chore: remove `requirements-fixer` from pre-commit Signed-off-by: Simon Brugman <sfbbrugman@gmail.com> --------- Signed-off-by: Simon Brugman <sfbbrugman@gmail.com>

* feat: added delta table dataset Signed-off-by: Afaque Ahmad <linkedafaque@gmail.com> * test: lint Signed-off-by: k9 <kylechun9@gmail.com> * chore: adjusted docstring line length Signed-off-by: Afaque Ahmad <linkedafaque@gmail.com> * chore: fix requirements order Signed-off-by: k9 <kylechun9@gmail.com> * chore: add .pylintrc to ignore line too long for url Signed-off-by: k9 <kylechun9@gmail.com> * chore: remove invalid noqa comment Signed-off-by: k9 <kylechun9@gmail.com> * fix: do not import TableNotFoundError from private module to avoid pylint complains Signed-off-by: k9 <kylechun9@gmail.com> * fix: fixed linting issues Signed-off-by: Afaque Ahmad <afaque.ahmad@quantumblack.com> * Move pylintrc to pyproject.toml Signed-off-by: Nok <nok.lam.chan@quantumblack.com> * Fix pylint config Signed-off-by: Nok <nok.lam.chan@quantumblack.com> * test: use mocker fixture to replace unittest.mock Signed-off-by: k9 <kylechun9@gmail.com> * chore: lint for line too long Signed-off-by: k9 <kylechun9@gmail.com> * test: increase coverage for pandas delta table dataset Signed-off-by: Kyle Chung <kylechun9@gmail.com> * chore: lint Signed-off-by: Kyle Chung <kylechun9@gmail.com> --------- Signed-off-by: Afaque Ahmad <linkedafaque@gmail.com> Signed-off-by: k9 <kylechun9@gmail.com> Signed-off-by: Afaque Ahmad <afaque.ahmad@quantumblack.com> Signed-off-by: Nok <nok.lam.chan@quantumblack.com> Signed-off-by: Kyle Chung <kylechun9@gmail.com> Co-authored-by: k9 <kylechun9@gmail.com> Co-authored-by: Nok <nok.lam.chan@quantumblack.com>

* Unpin pytest-xdist to avoid deprecation error Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Remove deprecation warnings from kedro-datasets Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Adapt future deprecation of Abstract*DataSet Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Add tests for backwards and forwards compatibility Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Typo Co-authored-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Make _io tests restore old values Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Lint Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Ignore faulty pylint check See pylint-dev/pylint#8865 Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Disable a few more faulty pylint checks Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Remove shim from tensorflow datasets This needs to be addressed later on, see TODO. Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> --------- Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Co-authored-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

add nightly build

bump dask version to the latest Signed-off-by: Jian <jian.wang@cpnet.io> Co-authored-by: Nok Lam Chan <nok.lam.chan@quantumblack.com>

Minor docs tweaks Signed-off-by: Simon Brugman <sfbbrugman@gmail.com> Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>

* perf(datasets): lazily load datasets in init files (api) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * perf(datasets): lazily load datasets in init files (pandas) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * fix(datasets): fix no name in module in api/pandas Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * perf(datasets): lazily load datasets in init files (biosequence) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * perf(datasets): lazily load datasets in init files (dask) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * perf(datasets): lazily load datasets in init files (databricks) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * perf(datasets): lazily load datasets in init files (email) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * perf(datasets): lazily load datasets in init files (geopandas) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * perf(datasets): lazily load datasets in init files (holoviews) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * perf(datasets): lazily load datasets in init files (json) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * fix(datasets): resolve "too few public attributes" Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * perf(datasets): lazily load datasets in init files (matplotlib) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * perf(datasets): lazily load datasets in init files (networkx) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * perf(datasets): lazily load datasets in init files (pickle) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * perf(datasets): lazily load datasets in init files (pillow) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * perf(datasets): lazily load datasets in init files (plotly) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * perf(datasets): lazily load datasets in init files (polars) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * perf(datasets): lazily load datasets in init files (redis) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * perf(datasets): lazily load datasets in init files (snowflake) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * perf(datasets): lazily load datasets in init files (spark) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * perf(datasets): lazily load datasets in init files (svmlight) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * perf(datasets): lazily load datasets in init files (tensorflow) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * perf(datasets): lazily load datasets in init files (text) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * perf(datasets): lazily load datasets in init files (tracking) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * perf(datasets): lazily load datasets in init files (video) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * perf(datasets): lazily load datasets in init files (yaml) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Update RELEASE.md --------- Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com>

merelcht · 2023-10-04T15:26:45Z

Hi @PtrBld , I've jumped in to resolve merge conflicts and linting issues. I hope that's okay. There's still some tests failing. I will try and fix them if I have some time!

Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com>

… names in implementations + tests Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com>

…ugins into add-partitioned-datasets

Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com>

AhdraMeraliQB

Just one suggestion:

kedro-datasets/tests/partitions/test_incremental_dataset.py

Co-authored-by: Ahdra Merali <90615669+AhdraMeraliQB@users.noreply.github.com>

astrojuanlu · 2023-10-11T14:35:06Z

kedro-datasets/RELEASE.md

@@ -4,6 +4,9 @@

 ## Bug fixes and other changes
 ## Upcoming deprecations for Kedro-Datasets 2.0.0
+## Community contributions
+Many thanks to the following Kedroids for contributing PRs to this release:
+* [PtrBld](https://github.com/PtrBld)


…` from main repository to kedro-datasets (kedro-org#253) Signed-off-by: Peter Bludau <ptrbld.dev@gmail.com> Co-authored-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: Riley Brady <riley_brady@mckinsey.com>

* initialize template and early additions Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * add placeholder for remote file system load Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * switch to versioned dataset Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * add initial remote -> local get for S3 Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * further generalize remote retrieval Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * add in credentials Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * make temppath optional for remote datasets Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * add initial idea for multifile glob Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * style: Introduce `ruff` for linting in all plugins. (#354) Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * add suggested style changes Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * add temppath to attributes Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * more temppath fixes Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * more temppath updates Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * add better tempfile deletion and work on saving files Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * make __del__ flexible Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * formatting Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * feat(datasets): create custom `DeprecationWarning` (#356) * feat(datasets): create custom `DeprecationWarning` Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * feat(datasets): use the custom deprecation warning Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * chore(datasets): show Kedro's deprecation warnings Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * fix(datasets): remove unused imports in test files Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> --------- Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * docs(datasets): add note about DataSet deprecation (#357) Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * test(datasets): skip `tensorflow` tests on Windows (#363) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * ci: Pin `tables` version (#370) * Pin tables version Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Also fix kedro-airflow Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Revert trying to fix airflow Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> --------- Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * build(datasets): Release `1.7.1` (#378) Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * docs: Update CONTRIBUTING.md and add one for `kedro-datasets` (#379) Update CONTRIBUTING.md + add one for kedro-datasets Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * ci(datasets): Run tensorflow tests separately from other dataset tests (#377) Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * feat: Kedro-Airflow convert all pipelines option (#335) * feat: kedro airflow convert --all option Signed-off-by: Simon Brugman <sfbbrugman@gmail.com> * docs: release docs Signed-off-by: Simon Brugman <sfbbrugman@gmail.com> --------- Signed-off-by: Simon Brugman <sfbbrugman@gmail.com> Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * docs(datasets): blacken code in rst literal blocks (#362) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * docs: cloudpickle is an interesting extension of the pickle functionality (#361) Signed-off-by: H. Felix Wittmann <hfwittmann@gmail.com> Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * fix(datasets): Fix secret scan entropy error (#383) Fix secret scan entropy error Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * style: Rename mentions of `DataSet` to `Dataset` in `kedro-airflow` and `kedro-telemetry` (#384) Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * feat(datasets): Migrated `PartitionedDataSet` and `IncrementalDataSet` from main repository to kedro-datasets (#253) Signed-off-by: Peter Bludau <ptrbld.dev@gmail.com> Co-authored-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * fix: backwards compatibility for `kedro-airflow` (#381) Signed-off-by: Simon Brugman <sfbbrugman@gmail.com> Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * fix(datasets): Don't warn for SparkDataset on Databricks when using s3 (#341) Signed-off-by: Alistair McKelvie <alistair.mckelvie@gmail.com> Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * update docs API and release notes Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * add netcdf requirements to setup Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * lint Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * add initial tests Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * update dataset exists for multifile Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * Add full test suite for NetCDFDataSet Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * Add docstring examples Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * change xarray version req Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * update dask req Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * rename DataSet -> Dataset Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * Update xarray reqs for earlier python versions Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * fix setup Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * update test coverage Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * exclude init from test coverage Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * Sub in pathlib for os.remove Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * add metadata to dataset Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * add doctest for the new datasets Signed-off-by: Nok <nok.lam.chan@quantumblack.com> * add patch for supporting http/https Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * Small fixes post-merge Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Lint Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Fix import Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Un-ignore NetCDF doctest Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Add fixture Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Mark problematic test as xfail Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Skip problematic test instead of making it fail Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Skip problematic tests and fix failing tests Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Remove comment Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> --------- Signed-off-by: Riley Brady <riley_brady@mckinsey.com> Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: Simon Brugman <sfbbrugman@gmail.com> Signed-off-by: H. Felix Wittmann <hfwittmann@gmail.com> Signed-off-by: Peter Bludau <ptrbld.dev@gmail.com> Signed-off-by: Alistair McKelvie <alistair.mckelvie@gmail.com> Signed-off-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Signed-off-by: Nok Lam Chan <nok.lam.chan@quantumblack.com> Signed-off-by: Nok <nok.lam.chan@quantumblack.com> Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Signed-off-by: Ankita Katiyar <110245118+ankatiyar@users.noreply.github.com> Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Co-authored-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Co-authored-by: Ankita Katiyar <110245118+ankatiyar@users.noreply.github.com> Co-authored-by: Simon Brugman <sbrugman@users.noreply.github.com> Co-authored-by: Felix Wittmann <hfwittmann@users.noreply.github.com> Co-authored-by: PtrBld <7523956+PtrBld@users.noreply.github.com> Co-authored-by: Merel Theisen <merel.theisen@quantumblack.com> Co-authored-by: Alistair McKelvie <alistair.mckelvie@gmail.com> Co-authored-by: Nok Lam Chan <nok.lam.chan@quantumblack.com> Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Co-authored-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

…` from main repository to kedro-datasets (kedro-org#253) Signed-off-by: Peter Bludau <ptrbld.dev@gmail.com> Co-authored-by: Merel Theisen <merel.theisen@quantumblack.com>

* initialize template and early additions Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * add placeholder for remote file system load Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * switch to versioned dataset Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * add initial remote -> local get for S3 Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * further generalize remote retrieval Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * add in credentials Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * make temppath optional for remote datasets Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * add initial idea for multifile glob Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * style: Introduce `ruff` for linting in all plugins. (kedro-org#354) Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * add suggested style changes Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * add temppath to attributes Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * more temppath fixes Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * more temppath updates Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * add better tempfile deletion and work on saving files Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * make __del__ flexible Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * formatting Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * feat(datasets): create custom `DeprecationWarning` (kedro-org#356) * feat(datasets): create custom `DeprecationWarning` Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * feat(datasets): use the custom deprecation warning Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * chore(datasets): show Kedro's deprecation warnings Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * fix(datasets): remove unused imports in test files Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> --------- Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * docs(datasets): add note about DataSet deprecation (kedro-org#357) Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * test(datasets): skip `tensorflow` tests on Windows (kedro-org#363) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * ci: Pin `tables` version (kedro-org#370) * Pin tables version Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Also fix kedro-airflow Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Revert trying to fix airflow Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> --------- Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * build(datasets): Release `1.7.1` (kedro-org#378) Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * docs: Update CONTRIBUTING.md and add one for `kedro-datasets` (kedro-org#379) Update CONTRIBUTING.md + add one for kedro-datasets Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * ci(datasets): Run tensorflow tests separately from other dataset tests (kedro-org#377) Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * feat: Kedro-Airflow convert all pipelines option (kedro-org#335) * feat: kedro airflow convert --all option Signed-off-by: Simon Brugman <sfbbrugman@gmail.com> * docs: release docs Signed-off-by: Simon Brugman <sfbbrugman@gmail.com> --------- Signed-off-by: Simon Brugman <sfbbrugman@gmail.com> Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * docs(datasets): blacken code in rst literal blocks (kedro-org#362) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * docs: cloudpickle is an interesting extension of the pickle functionality (kedro-org#361) Signed-off-by: H. Felix Wittmann <hfwittmann@gmail.com> Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * fix(datasets): Fix secret scan entropy error (kedro-org#383) Fix secret scan entropy error Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * style: Rename mentions of `DataSet` to `Dataset` in `kedro-airflow` and `kedro-telemetry` (kedro-org#384) Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * feat(datasets): Migrated `PartitionedDataSet` and `IncrementalDataSet` from main repository to kedro-datasets (kedro-org#253) Signed-off-by: Peter Bludau <ptrbld.dev@gmail.com> Co-authored-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * fix: backwards compatibility for `kedro-airflow` (kedro-org#381) Signed-off-by: Simon Brugman <sfbbrugman@gmail.com> Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * fix(datasets): Don't warn for SparkDataset on Databricks when using s3 (kedro-org#341) Signed-off-by: Alistair McKelvie <alistair.mckelvie@gmail.com> Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * update docs API and release notes Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * add netcdf requirements to setup Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * lint Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * add initial tests Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * update dataset exists for multifile Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * Add full test suite for NetCDFDataSet Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * Add docstring examples Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * change xarray version req Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * update dask req Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * rename DataSet -> Dataset Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * Update xarray reqs for earlier python versions Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * fix setup Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * update test coverage Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * exclude init from test coverage Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * Sub in pathlib for os.remove Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * add metadata to dataset Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * add doctest for the new datasets Signed-off-by: Nok <nok.lam.chan@quantumblack.com> * add patch for supporting http/https Signed-off-by: Riley Brady <riley_brady@mckinsey.com> * Small fixes post-merge Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Lint Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Fix import Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Un-ignore NetCDF doctest Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Add fixture Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Mark problematic test as xfail Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Skip problematic test instead of making it fail Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Skip problematic tests and fix failing tests Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Remove comment Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> --------- Signed-off-by: Riley Brady <riley_brady@mckinsey.com> Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: Simon Brugman <sfbbrugman@gmail.com> Signed-off-by: H. Felix Wittmann <hfwittmann@gmail.com> Signed-off-by: Peter Bludau <ptrbld.dev@gmail.com> Signed-off-by: Alistair McKelvie <alistair.mckelvie@gmail.com> Signed-off-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Signed-off-by: Nok Lam Chan <nok.lam.chan@quantumblack.com> Signed-off-by: Nok <nok.lam.chan@quantumblack.com> Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Signed-off-by: Ankita Katiyar <110245118+ankatiyar@users.noreply.github.com> Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Co-authored-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Co-authored-by: Ankita Katiyar <110245118+ankatiyar@users.noreply.github.com> Co-authored-by: Simon Brugman <sbrugman@users.noreply.github.com> Co-authored-by: Felix Wittmann <hfwittmann@users.noreply.github.com> Co-authored-by: PtrBld <7523956+PtrBld@users.noreply.github.com> Co-authored-by: Merel Theisen <merel.theisen@quantumblack.com> Co-authored-by: Alistair McKelvie <alistair.mckelvie@gmail.com> Co-authored-by: Nok Lam Chan <nok.lam.chan@quantumblack.com> Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Co-authored-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com>

* refactor(datasets): deprecate "DataSet" type names (#328) * refactor(datasets): deprecate "DataSet" type names (api) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (biosequence) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (dask) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (databricks) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (email) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (geopandas) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (holoviews) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (json) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (matplotlib) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (networkx) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (pandas) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (pandas.csv_dataset) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (pandas.deltatable_dataset) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (pandas.excel_dataset) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (pandas.feather_dataset) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (pandas.gbq_dataset) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (pandas.generic_dataset) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (pandas.hdf_dataset) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (pandas.json_dataset) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (pandas.parquet_dataset) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (pandas.sql_dataset) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (pandas.xml_dataset) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (pickle) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (pillow) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (plotly) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (polars) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (redis) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (snowflake) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (spark) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (svmlight) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (tensorflow) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (text) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (tracking) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (video) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (yaml) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * chore(datasets): ignore TensorFlow coverage issues Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> --------- Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * added basic code for geotiff Signed-off-by: tgoelles <thomas.goelles@gmail.com> * renamed to xarray Signed-off-by: tgoelles <thomas.goelles@gmail.com> * renamed to xarray Signed-off-by: tgoelles <thomas.goelles@gmail.com> * added load and self args Signed-off-by: tgoelles <thomas.goelles@gmail.com> * only local files Signed-off-by: tgoelles <thomas.goelles@gmail.com> * added empty test Signed-off-by: tgoelles <thomas.goelles@gmail.com> * added test data Signed-off-by: tgoelles <thomas.goelles@gmail.com> * added rioxarray requirements Signed-off-by: tgoelles <thomas.goelles@gmail.com> * reformat with black Signed-off-by: tgoelles <thomas.goelles@gmail.com> * rioxarray 0.14 Signed-off-by: tgoelles <thomas.goelles@gmail.com> * rioxarray 0.15 Signed-off-by: tgoelles <thomas.goelles@gmail.com> * rioxarray 0.12 Signed-off-by: tgoelles <thomas.goelles@gmail.com> * rioxarray 0.9 Signed-off-by: tgoelles <thomas.goelles@gmail.com> * fixed dataset typo Signed-off-by: tgoelles <thomas.goelles@gmail.com> * fixed docstring for sphinx Signed-off-by: tgoelles <thomas.goelles@gmail.com> * run black Signed-off-by: tgoelles <thomas.goelles@gmail.com> * sort imports Signed-off-by: tgoelles <thomas.goelles@gmail.com> * class docstring Signed-off-by: tgoelles <thomas.goelles@gmail.com> * black Signed-off-by: tgoelles <thomas.goelles@gmail.com> * fixed pylint Signed-off-by: tgoelles <thomas.goelles@gmail.com> * added release notes Signed-off-by: tgoelles <thomas.goelles@gmail.com> * added yaml example Signed-off-by: tgoelles <thomas.goelles@gmail.com> * improve testing WIP Signed-off-by: tgoelles <thomas.goelles@gmail.com> * basic test success Signed-off-by: tgoelles <thomas.goelles@gmail.com> * test reloaded Signed-off-by: tgoelles <thomas.goelles@gmail.com> * test exists Signed-off-by: tgoelles <thomas.goelles@gmail.com> * added version Signed-off-by: tgoelles <thomas.goelles@gmail.com> * basic test suite Signed-off-by: tgoelles <thomas.goelles@gmail.com> * run black Signed-off-by: tgoelles <thomas.goelles@gmail.com> * added example and test it Signed-off-by: tgoelles <thomas.goelles@gmail.com> * deleted duplications Signed-off-by: tgoelles <thomas.goelles@gmail.com> * fixed position of example Signed-off-by: tgoelles <thomas.goelles@gmail.com> * black Signed-off-by: tgoelles <thomas.goelles@gmail.com> * style: Introduce `ruff` for linting in all plugins. (#354) Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> * feat(datasets): create custom `DeprecationWarning` (#356) * feat(datasets): create custom `DeprecationWarning` Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * feat(datasets): use the custom deprecation warning Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * chore(datasets): show Kedro's deprecation warnings Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * fix(datasets): remove unused imports in test files Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> --------- Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * docs(datasets): add note about DataSet deprecation (#357) Signed-off-by: tgoelles <thomas.goelles@gmail.com> * test(datasets): skip `tensorflow` tests on Windows (#363) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * ci: Pin `tables` version (#370) * Pin tables version Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Also fix kedro-airflow Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Revert trying to fix airflow Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> --------- Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * build(datasets): Release `1.7.1` (#378) Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * docs: Update CONTRIBUTING.md and add one for `kedro-datasets` (#379) Update CONTRIBUTING.md + add one for kedro-datasets Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * ci(datasets): Run tensorflow tests separately from other dataset tests (#377) Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * feat: Kedro-Airflow convert all pipelines option (#335) * feat: kedro airflow convert --all option Signed-off-by: Simon Brugman <sfbbrugman@gmail.com> * docs: release docs Signed-off-by: Simon Brugman <sfbbrugman@gmail.com> --------- Signed-off-by: Simon Brugman <sfbbrugman@gmail.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * docs(datasets): blacken code in rst literal blocks (#362) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * docs: cloudpickle is an interesting extension of the pickle functionality (#361) Signed-off-by: H. Felix Wittmann <hfwittmann@gmail.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * fix(datasets): Fix secret scan entropy error (#383) Fix secret scan entropy error Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * style: Rename mentions of `DataSet` to `Dataset` in `kedro-airflow` and `kedro-telemetry` (#384) Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * feat(datasets): Migrated `PartitionedDataSet` and `IncrementalDataSet` from main repository to kedro-datasets (#253) Signed-off-by: Peter Bludau <ptrbld.dev@gmail.com> Co-authored-by: Merel Theisen <merel.theisen@quantumblack.com> * fix: backwards compatibility for `kedro-airflow` (#381) Signed-off-by: Simon Brugman <sfbbrugman@gmail.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * added metadata Signed-off-by: tgoelles <thomas.goelles@gmail.com> * after linting Signed-off-by: tgoelles <thomas.goelles@gmail.com> * ignore ruff PLR0913 Signed-off-by: tgoelles <thomas.goelles@gmail.com> * fix(datasets): Don't warn for SparkDataset on Databricks when using s3 (#341) Signed-off-by: Alistair McKelvie <alistair.mckelvie@gmail.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * chore: Hot fix for RTD due to bad pip version (#396) fix RTD Signed-off-by: Nok <nok.lam.chan@quantumblack.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * chore: Pin pip version temporarily (#398) * Pin pip version temporarily Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Hive support failures Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Also pin pip on lint Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Temporary ignore databricks spark tests Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> --------- Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * perf(datasets): don't create connection until need (#281) * perf(datasets): delay `Engine` creation until need Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * chore: don't check coverage in TYPE_CHECKING block Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * perf(datasets): don't connect in `__init__` method Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * test(datasets): fix tests to touch `create_engine` Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * perf(datasets): don't connect in `__init__` method Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * style(datasets): exec Ruff on sql_dataset.py :dog: Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Undo changes to `engines` values type (for Sphinx) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Patch Sphinx build by removing `Engine` references * perf(datasets): don't connect in `__init__` method Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * chore(datasets): don't require coverage for import * chore(datasets): del unused `TYPE_CHECKING` import * docs(datasets): document lazy connection in README * perf(datasets): remove create in `SQLQueryDataset` Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): do not return the created conn Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> --------- Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * chore: Drop Python 3.7 support for kedro-plugins (#392) * Remove references to Python 3.7 Signed-off-by: lrcouto <laurarccouto@gmail.com> * Revert kedro-dataset changes Signed-off-by: lrcouto <laurarccouto@gmail.com> * Revert kedro-dataset changes Signed-off-by: lrcouto <laurarccouto@gmail.com> * Add information to release docs Signed-off-by: lrcouto <laurarccouto@gmail.com> --------- Signed-off-by: lrcouto <laurarccouto@gmail.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * feat(datasets): support Polars lazy evaluation (#350) * feat(datasets) add PolarsDataset to support Polars's Lazy API Signed-off-by: Matthias Roels <matthias.roels21@gmail.com> * Fix(datasets): rename PolarsDataSet to PolarsDataSet Add PolarsDataSet as an alias for PolarsDataset with deprecation warning. Signed-off-by: Matthias Roels <matthias.roels21@gmail.com> * Fix(datasets): apply ruff linting rules Signed-off-by: Matthias Roels <matthias.roels21@gmail.com> * Fix(datasets): Correct pattern matching when Raising exceptions Corrected PolarsDataSet to PolarsDataset in the pattern to match in test_load_missing_file Signed-off-by: Matthias Roels <matthias.roels21@gmail.com> * fix(datasets): clean up PolarsDataset related code Remove reference to PolarsDataSet as this is not required for new dataset implementations. Signed-off-by: Matthias Roels <matthias.roels21@gmail.com> * feat(datasets): Rename Polars Datasets to better describe their intent Signed-off-by: Matthias Roels <matthias.roels21@gmail.com> * feat(datasets): clean up LazyPolarsDataset Signed-off-by: Matthias Roels <matthias.roels21@gmail.com> * fix(datasets): increase test coverage for PolarsDataset classes Signed-off-by: Matthias Roels <matthias.roels21@gmail.com> * docs(datasets): add renamed Polars datasets to docs Signed-off-by: Matthias Roels <matthias.roels21@gmail.com> * docs(datasets): Add new polars datasets to release notes Signed-off-by: Matthias Roels <mroels2@its.jnj.com> * fix(datasets): load_args not properly passed to LazyPolarsDataset.load Signed-off-by: Matthias Roels <matthias.roels21@gmail.com> * docs(datasets): fix spelling error in release notes Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Signed-off-by: Matthias Roels <matthias.roels21@gmail.com> --------- Signed-off-by: Matthias Roels <matthias.roels21@gmail.com> Signed-off-by: Matthias Roels <mroels2@its.jnj.com> Signed-off-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Co-authored-by: Matthias Roels <mroels2@its.jnj.com> Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * build(datasets): Release `1.8.0` (#406) Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * build(airflow): Release 0.7.0 (#407) * bump version Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update release notes Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> --------- Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * build(telemetry): Release 0.3.0 (#408) Bump version Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: Ankita Katiyar <110245118+ankatiyar@users.noreply.github.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * build(docker): Release 0.4.0 (#409) Bump version Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: Ankita Katiyar <110245118+ankatiyar@users.noreply.github.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * style(airflow): blacken README.md of Kedro-Airflow (#418) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * fix(datasets): Fix missing jQuery (#414) Fix missing jQuery Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * fix(datasets): Fix Lazy Polars dataset to use the new-style base class (#413) * Fix Lazy Polars dataset to use the new-style base class Fix gh-412 Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Update release notes Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Revert "Update release notes" This reverts commit 92ceea6d8fa412abf3d8abd28a2f0a22353867ed. --------- Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Signed-off-by: Sajid Alam <90610031+SajidAlamQB@users.noreply.github.com> Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Co-authored-by: Sajid Alam <90610031+SajidAlamQB@users.noreply.github.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * chore(datasets): lazily load `partitions` classes (#411) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * docs(datasets): fix code blocks and `data_set` use (#417) * chore(datasets): lazily load `partitions` classes Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * test(datasets): run doctests to check examples run Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * test(datasets): keep running tests amidst failures Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * docs(datasets): format ManagedTableDataset example Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * chore(datasets): ignore breaking mods for doctests Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * style(airflow): black code in Kedro-Airflow README Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * docs(datasets): fix example syntax, and autoformat Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * docs(datasets): remove `kedro.extras.datasets` ref Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * docs(datasets): remove `>>> ` prefix for YAML code Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * docs(datasets): remove `kedro.extras.datasets` ref Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * docs(datasets): replace `data_set`s with `dataset`s Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * chore(datasets): undo changes for running doctests Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * revert(datasets): undo lazily load `partitions` classes Refs: 3fdc5a8efa034fa9a18b7683a942415915b42fb5 Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * revert(airflow): undo black code in Kedro-Airflow README Refs: dc3476ea36bac98e2adcc0b52a11b0f90001e31d Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> --------- Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * fix: TF model load failure when model is saved as a TensorFlow Saved Model format (#410) * fixes TF model load failure when model is saved as a TensorFlow Saved Model format when a model is saved in the TensorFlow SavedModel format ("tf" default option in tf.save_model when using TF 2.x) via the catalog.xml file, the subsequent loading of that model for further use in a subsequent node fails. The issue is linked to the fact that the model files don't get copied into the temporary folder, presumably because the _fs.get function "thinks" that the provided path is a file and not a folder. Adding an terminating "/" to the path fixes the issue. Signed-off-by: Edouard59 <68538605+Edouard59@users.noreply.github.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * chore: Drop support for Python 3.7 on kedro-datasets (#419) * Drop support for Python 3.7 on kedro-datasets Signed-off-by: lrcouto <laurarccouto@gmail.com> * Remove redundant 3.8 markers Signed-off-by: lrcouto <laurarccouto@gmail.com> --------- Signed-off-by: lrcouto <laurarccouto@gmail.com> Signed-off-by: L. R. Couto <57910428+lrcouto@users.noreply.github.com> Signed-off-by: Sajid Alam <90610031+SajidAlamQB@users.noreply.github.com> Co-authored-by: Sajid Alam <90610031+SajidAlamQB@users.noreply.github.com> * test(datasets): run doctests to check examples run (#416) * chore(datasets): lazily load `partitions` classes Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * test(datasets): run doctests to check examples run Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * test(datasets): keep running tests amidst failures Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * docs(datasets): format ManagedTableDataset example Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * chore(datasets): ignore breaking mods for doctests Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * style(airflow): black code in Kedro-Airflow README Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * docs(datasets): fix example syntax, and autoformat Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * docs(datasets): remove `kedro.extras.datasets` ref Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * docs(datasets): remove `>>> ` prefix for YAML code Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * docs(datasets): remove `kedro.extras.datasets` ref Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * docs(datasets): replace `data_set`s with `dataset`s Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): run doctests separately Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * separate dataset-doctests Signed-off-by: Nok <nok.lam.chan@quantumblack.com> * chore(datasets): ignore non-passing tests to make CI pass Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * chore(datasets): fix comment location Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * chore(datasets): fix .py.py Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * chore(datasets): don't measure coverage on doctest run Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * build(datasets): fix windows and snowflake stuff in Makefile Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> --------- Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: Nok <nok.lam.chan@quantumblack.com> Co-authored-by: Nok <nok.lam.chan@quantumblack.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * feat(datasets): Add support for `databricks-connect>=13.0` (#352) Signed-off-by: Miguel Rodriguez Gutierrez <miguel7r@hotmail.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * fix(telemetry): remove double execution by moving to after catalog created hook (#422) * remove double execution by moving to after catalog created hook Signed-off-by: Florian Roessler <roessler.fd@gmail.com> * update release notes Signed-off-by: Florian Roessler <roessler.fd@gmail.com> * fix tests Signed-off-by: Florian Roessler <roessler.fd@gmail.com> * remove unsued fixture Signed-off-by: Florian Roessler <roessler.fd@gmail.com> --------- Signed-off-by: Florian Roessler <roessler.fd@gmail.com> Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * docs: Add python version support policy to plugin `README.md`s (#425) * Add python version support policy to plugin readmes Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> * Temporarily pin connexion Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> --------- Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * docs(airflow): Use new docs link (#393) Use new docs link Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * style: Add shared CSS and meganav to datasets docs (#400) * Add shared CSS and meganav Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Add end of file Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Add new heap data source Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * adjust heap parameter Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Remove nav_version next to Kedro logo in top left; add Kedro logo * Revise project name and author name Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Use full kedro icon and type for logo * Add close btn to mobile nav Signed-off-by: vladimir-mck <106236933+vladimir-mck@users.noreply.github.com> * Add css for mobile nav logo image Signed-off-by: vladimir-mck <106236933+vladimir-mck@users.noreply.github.com> * Update close button for mobile nav Signed-off-by: vladimir-mck <106236933+vladimir-mck@users.noreply.github.com> * Add open button to mobile nav Signed-off-by: vladimir-mck <106236933+vladimir-mck@users.noreply.github.com> * Delete kedro-datasets/docs/source/kedro-horizontal-color-on-light.svg Signed-off-by: vladimir-mck <106236933+vladimir-mck@users.noreply.github.com> * Update conf.py Signed-off-by: vladimir-mck <106236933+vladimir-mck@users.noreply.github.com> * Update layout.html Add links to subprojects Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Remove svg from docs -- not needed?? Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * linter error fix Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> --------- Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> Signed-off-by: vladimir-mck <106236933+vladimir-mck@users.noreply.github.com> Co-authored-by: Tynan DeBold <thdebold@gmail.com> Co-authored-by: vladimir-mck <106236933+vladimir-mck@users.noreply.github.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * feat(datasets): Add Hugging Face datasets (#344) * Add HuggingFace datasets Co-authored-by: Danny Farah <danny_farah@mckinsey.com> Co-authored-by: Kevin Koga <Kevin_Koga@mckinsey.com> Co-authored-by: Mate Scharnitzky <Mate_Scharnitzky@mckinsey.com> Co-authored-by: Tomer Shor <Tomer_Shor@mckinsey.com> Co-authored-by: Pierre-Yves Mousset <Pierre-Yves_Mousset@mckinsey.com> Co-authored-by: Bela Chupal <Bela_chuphal@mckinsey.com> Co-authored-by: Khangjrakpam Arjun <Khangjrakpam_Arjun@mckinsey.com> Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Apply suggestions from code review Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Co-authored-by: Joel <35801847+datajoely@users.noreply.github.com> Co-authored-by: Nok Lam Chan <nok.lam.chan@quantumblack.com> * Typo Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Fix docstring Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Add docstring for HFTransformerPipelineDataset Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Use intersphinx for cross references in Hugging Face docstrings Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Add docstring for HFDataset Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Add missing test dependencies Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Add tests for huggingface datasets Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Fix HFDataset.save Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Add test for HFDataset.list_datasets Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Use new name Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Consolidate imports Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> --------- Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Co-authored-by: Danny Farah <danny_farah@mckinsey.com> Co-authored-by: Kevin Koga <Kevin_Koga@mckinsey.com> Co-authored-by: Mate Scharnitzky <Mate_Scharnitzky@mckinsey.com> Co-authored-by: Tomer Shor <Tomer_Shor@mckinsey.com> Co-authored-by: Pierre-Yves Mousset <Pierre-Yves_Mousset@mckinsey.com> Co-authored-by: Bela Chupal <Bela_chuphal@mckinsey.com> Co-authored-by: Khangjrakpam Arjun <Khangjrakpam_Arjun@mckinsey.com> Co-authored-by: Joel <35801847+datajoely@users.noreply.github.com> Co-authored-by: Nok Lam Chan <nok.lam.chan@quantumblack.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * test(datasets): fix `dask.ParquetDataset` doctests (#439) * test(datasets): fix `dask.ParquetDataset` doctests Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * test(datasets): use `tmp_path` fixture in doctests Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * test(datasets): simplify by not passing the schema Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * test(datasets): ignore conftest for doctests cover Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Create MANIFEST.in Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> --------- Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * refactor: Remove `DataSet` aliases and mentions (#440) Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> * chore(datasets): replace "Pyspark" with "PySpark" (#423) Consistently write "PySpark" rather than "Pyspark" Also, fix list formatting Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * test(datasets): make `api.APIDataset` doctests run (#448) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * chore(datasets): Fix `pandas.GenericDataset` doctest (#445) Fix pandas.GenericDataset doctest Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * feat(datasets): make datasets arguments keywords only (#358) * feat(datasets): make `APIDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `BioSequenceDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `ParquetDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `EmailMessageDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `GeoJSONDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `HoloviewsWriter.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `JSONDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `MatplotlibWriter.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `GMLDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `GraphMLDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make NetworkX `JSONDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `PickleDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `ImageDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make plotly `JSONDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `PlotlyDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make polars `CSVDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make polars `GenericDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make redis `PickleDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `SnowparkTableDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `SVMLightDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `TensorFlowModelDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `TextDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `YAMLDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `ManagedTableDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `VideoDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `CSVDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `DeltaTableDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `ExcelDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `FeatherDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `GBQTableDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `GenericDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make pandas `JSONDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make pandas `ParquerDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `SQLTableDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `XMLDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `HDFDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `DeltaTableDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `SparkDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `SparkHiveDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `SparkJDBCDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `SparkStreamingDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `IncrementalDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `LazyPolarsDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * docs(datasets): update doctests for HoloviewsWriter Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * Update release notes Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> --------- Signed-off-by: Felix Scherz <felixwscherz@gmail.com> Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Co-authored-by: Felix Scherz <felixwscherz@gmail.com> Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Co-authored-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * chore: Drop support for python 3.8 on kedro-datasets (#442) * Drop support for python 3.8 on kedro-datasets --------- Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com> Signed-off-by: Dmitry Sorokin <40151847+DimedS@users.noreply.github.com> Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * test(datasets): add outputs to matplotlib doctests (#449) * test(datasets): add outputs to matplotlib doctests Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Update Makefile Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Reformat code example, line length is short enough * Update kedro-datasets/kedro_datasets/matplotlib/matplotlib_writer.py Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> --------- Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * chore(datasets): Fix more doctest issues (#451) Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Co-authored-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * test(datasets): fix failing doctests in Windows CI (#457) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * chore(datasets): fix accidental reference to NumPy (#450) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * chore(datasets): don't pollute dev env in doctests (#452) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * feat: Add tools to heap event (#430) * Add add-on data to heap event Signed-off-by: lrcouto <laurarccouto@gmail.com> * Move addons logic to _get_project_property Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Add condition for pyproject.toml Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Fix tests Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Fix tests Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * add tools to mock Signed-off-by: lrcouto <laurarccouto@gmail.com> * lint Signed-off-by: lrcouto <laurarccouto@gmail.com> * Update tools test Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Add after_context_created tools test Signed-off-by: lrcouto <laurarccouto@gmail.com> * Update rename to tools Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update kedro-telemetry/tests/test_plugin.py Co-authored-by: Sajid Alam <90610031+SajidAlamQB@users.noreply.github.com> Signed-off-by: Ankita Katiyar <110245118+ankatiyar@users.noreply.github.com> --------- Signed-off-by: lrcouto <laurarccouto@gmail.com> Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: Ankita Katiyar <110245118+ankatiyar@users.noreply.github.com> Co-authored-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Co-authored-by: Ankita Katiyar <110245118+ankatiyar@users.noreply.github.com> Co-authored-by: Sajid Alam <90610031+SajidAlamQB@users.noreply.github.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * ci(datasets): install deps in single `pip install` (#454) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * build(datasets): Bump s3fs (#463) * Use mocking for AWS responses Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> * Add change to release notes Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> * Update release notes Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> * Use pytest xfail instead of commenting out test Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> --------- Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * test(datasets): make SQL dataset examples runnable (#455) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * fix(datasets): correct pandas-gbq as py311 dependency (#460) * update pandas-gbq dependency declaration Signed-off-by: Onur Kuru <kuru.onur1@gmail.com> * fix fmt Signed-off-by: Onur Kuru <kuru.onur1@gmail.com> --------- Signed-off-by: Onur Kuru <kuru.onur1@gmail.com> Co-authored-by: Ahdra Merali <90615669+AhdraMeraliQB@users.noreply.github.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * docs(datasets): Document `IncrementalDataset` (#468) Document IncrementalDataset Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * chore: Update datasets to be arguments keyword only (#466) Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * chore: Clean up code for old dataset syntax compatibility (#465) Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * chore: Update scikit-learn version (#469) Update scikit-learn version Signed-off-by: Nok Lam Chan <nok.lam.chan@quantumblack.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * feat(datasets): support versioning data partitions (#447) * feat(datasets): support versioning data partitions Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Remove unused import Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * chore(datasets): use keyword arguments when needed Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Apply suggestions from code review Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Update kedro-datasets/kedro_datasets/partitions/partitioned_dataset.py Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> --------- Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * docs(datasets): Improve documentation index (#428) Rework documentation index Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * docs(datasets): update wrong docstring about `con` (#461) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * build(datasets): Release `2.0.0` (#472) Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * ci(telemetry): Pin `PyYAML` (#474) Pin PyYaml Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * build(telemetry): Release 0.3.1 (#475) Signed-off-by: tgoelles <thomas.goelles@gmail.com> * docs(datasets): Fix broken links in README (#477) Fix broken links in README Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * chore(datasets): replace more "data_set" instances (#476) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * chore(datasets): Fix doctests (#488) Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * chore(datasets): Fix delta + incremental dataset docstrings (#489) Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * chore(airflow): Post 0.19 cleanup (#478) * bump version Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Unbump version and clean test Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update e2e tests Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update e2e tests Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update e2e tests Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update e2e tests Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Split big test into smaller tests Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update conftest Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update conftest Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Fix coverage Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Try unpin airflow Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * remove datacatalog step Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Change node Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * update tasks test step Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Revert to older airflow and constraint pendulum Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update template Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update message in e2e step Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Final cleanup Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update kedro-airflow/pyproject.toml Signed-off-by: Nok Lam Chan <nok.lam.chan@quantumblack.com> * Pin apache-airflow again Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> --------- Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: Nok Lam Chan <nok.lam.chan@quantumblack.com> Co-authored-by: Nok Lam Chan <nok.lam.chan@quantumblack.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * build(airflow): Release 0.8.0 (#491) Bump version Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * fix: telemetry metadata (#495) --------- Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * fix: Update tests on kedro-docker for 0.5.0 release. (#496) * bump version to 0.5.0 Signed-off-by: lrcouto <laurarccouto@gmail.com> * bump version to 0.5.0 Signed-off-by: lrcouto <laurarccouto@gmail.com> * update e2e tests to use new starters Signed-off-by: lrcouto <laurarccouto@gmail.com> * Lint Signed-off-by: lrcouto <laurarccouto@gmail.com> * update e2e tests to use new starters Signed-off-by: lrcouto <laurarccouto@gmail.com> * fix test path for e2e tests Signed-off-by: lrcouto <laurarccouto@gmail.com> * fix requirements path on dockerfiles Signed-off-by: lrcouto <laurarccouto@gmail.com> * update tests to fit with current log format Signed-off-by: lrcouto <laurarccouto@gmail.com> * update tests to fit with current log format Signed-off-by: lrcouto <laurarccouto@gmail.com> * update tests to fit with current log format Signed-off-by: lrcouto <laurarccouto@gmail.com> * Remove redundant test Signed-off-by: lrcouto <laurarccouto@gmail.com> * Alter test for custom GID and UID Signed-off-by: lrcouto <laurarccouto@gmail.com> * Update release notes Signed-off-by: lrcouto <laurarccouto@gmail.com> * Revert version bump to put in in separate PR Signed-off-by: lrcouto <laurarccouto@gmail.com> --------- Signed-off-by: lrcouto <laurarccouto@gmail.com> Signed-off-by: L. R. Couto <57910428+lrcouto@users.noreply.github.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * build: Release kedro-docker 0.5.0 (#497) * bump version to 0.5.0 Signed-off-by: lrcouto <laurarccouto@gmail.com> * bump version to 0.5.0 Signed-off-by: lrcouto <laurarccouto@gmail.com> * update e2e tests to use new starters Signed-off-by: lrcouto <laurarccouto@gmail.com> * Lint Signed-off-by: lrcouto <laurarccouto@gmail.com> * update e2e tests to use new starters Signed-off-by: lrcouto <laurarccouto@gmail.com> * fix test path for e2e tests Signed-off-by: lrcouto <laurarccouto@gmail.com> * fix requirements path on dockerfiles Signed-off-by: lrcouto <laurarccouto@gmail.com> * update tests to fit with current log format Signed-off-by: lrcouto <laurarccouto@gmail.com> * update tests to fit with current log format Signed-off-by: lrcouto <laurarccouto@gmail.com> * update tests to fit with current log format Signed-off-by: lrcouto <laurarccouto@gmail.com> * Remove redundant test Signed-off-by: lrcouto <laurarccouto@gmail.com> * Alter test for custom GID and UID Signed-off-by: lrcouto <laurarccouto@gmail.com> * Update release notes Signed-off-by: lrcouto <laurarccouto@gmail.com> * Revert version bump to put in in separate PR Signed-off-by: lrcouto <laurarccouto@gmail.com> * Bump kedro-docker to 0.5.0 Signed-off-by: lrcouto <laurarccouto@gmail.com> * Add release notes Signed-off-by: lrcouto <laurarccouto@gmail.com> * Update kedro-docker/RELEASE.md Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Signed-off-by: L. R. Couto <57910428+lrcouto@users.noreply.github.com> --------- Signed-off-by: lrcouto <laurarccouto@gmail.com> Signed-off-by: L. R. Couto <57910428+lrcouto@users.noreply.github.com> Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * chore(datasets): Update partitioned dataset docstring (#502) Update partitioned dataset docstring Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * Fix GeotiffDataset import + casing Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> * Fix lint Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * fix(datasets): Relax pandas.HDFDataSet dependencies which are broken on Windows (#426) * Relax pandas.HDFDataSet dependencies which are broken on Window (#402) Signed-off-by: Yolan Honoré-Rougé <yolan.honore.rouge@gmail.com> * Update RELEASE.md Signed-off-by: Yolan Honoré-Rougé <yolan.honore.rouge@gmail.com> * Apply suggestions from code review Signed-off-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> * Update kedro-datasets/setup.py Signed-off-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> --------- Signed-off-by: Yolan Honoré-Rougé <yolan.honore.rouge@gmail.com> Signed-off-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * fix: airflow metadata (#498) * Add example pipeline entry to metadata declaration Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Fix entry Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Make entries consistent Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Add tools to config Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * fix: telemetry metadata (#495) --------- Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com> Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Revert "Add tools to config" This reverts commit 14732d772a3c2f4787063071a68fdf1512c93488. Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Quick fix Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Lint Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Remove outdated config key Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Use kedro new instead of cookiecutter Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> --------- Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com> Co-authored-by: Dmitry Sorokin <40151847+DimedS@users.noreply.github.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * chore(airflow): Bump `apache-airflow` version (#511) * Bump apache airflow Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Change starter Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update e2e test steps Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update e2e test steps Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> --------- Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * ci(datasets): Unpin dask (#522) * Unpin dask Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update doctest Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update doctest Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update kedro-datasets/setup.py Co-authored-by: Nok Lam Chan <nok.lam.chan@quantumblack.com> Signed-off-by: Ankita Katiyar <110245118+ankatiyar@users.noreply.github.com> --------- Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: Ankita Katiyar <110245118+ankatiyar@users.noreply.github.com> Co-authored-by: Nok Lam Chan <nok.lam.chan@quantumblack.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * feat(datasets): Add `MatlabDataset` to `kedro-datasets` (#515) * Refork and commit kedro matlab datasets Signed-off-by: samuelleeshemen <samuel_lee_sj@aiap.sg> * Fix lint, add to docs Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Try fixing docstring Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Try fixing save Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Try fix docstest Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Fix unit tests Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update release notes: Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Not hardcode load mode Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> --------- Signed-off-by: samuelleeshemen <samuel_lee_sj@aiap.sg> Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Co-authored-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * ci(airflow): Pin `Flask-Session` version (#521) * Restrict pendulum version Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update airflow init step Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Remove pendulum pin Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update create connections step Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Pin flask session Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Add comment Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> --------- Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * feat: `kedro-airflow` group in memory nodes (#241) * feat: option to group in-memory nodes Signed-off-by: Simon Brugman <sfbbrugman@gmail.com> * fix: MemoryDataset Signed-off-by: Simon Brugman <sfbbrugman@gmail.com> * Update kedro-airflow/README.md Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Signed-off-by: Simon Brugman <sbrugman@users.noreply.github.com> * Update kedro-airflow/README.md Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Signed-off-by: Simon Brugman <sbrugman@users.noreply.github.com> * Update kedro-airflow/README.md Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Signed-off-by: Simon Brugman <sbrugman@users.noreply.github.com> * Update kedro-airflow/RELEASE.md Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Signed-off-by: Simon Brugman <sbrugman@users.noreply.github.com> * Update kedro-airflow/kedro_airflow/grouping.py Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Signed-off-by: Simon Brugman <sbrugman@users.noreply.github.com> * Update kedro-airflow/kedro_airflow/plugin.py Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Signed-off-by: Simon Brugman <sbrugman@users.noreply.github.com> * Update kedro-airflow/tests/test_node_grouping.py Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Signed-off-by: Simon Brugman <sbrugman@users.noreply.github.com> * Update kedro-airflow/tests/test_node_grouping.py Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Signed-off-by: Simon Brugman <sbrugman@users.noreply.github.com> * Update kedro-airflow/kedro_airflow/grouping.py Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Signed-off-by: Simon Brugman <sbrugman@users.noreply.github.com> * Update kedro-airflow/kedro_airflow/grouping.py Co-authored-by: Ankita Katiyar <110245118+ankatiyar@users.noreply.github.com> Signed-off-by: Simon Brugman <sbrugman@users.noreply.github.com> * fix: tests Signed-off-by: Simon Brugman <sfbbrugman@gmail.com> * Bump minimum kedro version Signed-off-by: Simon Brugman <sfbbrugman@gmail.com> * fixes Signed-off-by: Simon Brugman <sfbbrugman@gmail.com> --------- Signed-off-by: Simon Brugman <sfbbrugman@gmail.com> Signed-off-by: Simon Brugman <sbrugman@users.noreply.github.com> Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Co-authored-by: Ankita Katiyar <110245118+ankatiyar@users.noreply.github.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * ci(datasets): Update pyproject.toml to pin Kedro 0.19 for kedro-datasets (#526) Update pyproject.toml Signed-off-by: Nok Lam Chan <nok.lam.chan@quantumblack.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * feat(airflow): include environment name in DAG filename (#492) * feat: include environment name in DAG file Signed-off-by: Simon Brugman <sfbbrugman@gmail.com> * doc: add update to release notes Signed-off-by: Simon Brugman <sfbbrugman@gmail.com> --------- Signed-off-by: Simon Brugman <sfbbrugman@gmail.com> Co-authored-by: Ankita Katiyar <110245118+ankatiyar@users.noreply.github.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * feat(datasets): Enable search-as-you type on Kedro-datasets docs (#532) * done Signed-off-by: rashidakanchwala <rashida_kanchwala@mckinsey.com> * fix lint Signed-off-by: rashidakanchwala <rashida_kanchwala@mckinsey.com> --------- Signed-off-by: rashidakanchwala <rashida_kanchwala@mckinsey.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * fix(datasets): Debug and fix `kedro-datasets` nightly build failures (#541) * pin deltalake * Update kedro-datasets/setup.py Signed-off-by: Juan Luis Cano Rodríguez <hello@juanlu.space> * Update setup.py Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * sort order and compare * Update setup.py * lint * pin deltalake * add comment to pin --------- Signed-off-by: Juan Luis Cano Rodríguez <hello@juanlu.space> Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * feat(datasets): Dataset Preview Refactor (#504) * test * done * change from _preview to preview * fix lint and tests * added docstrings * rtd fix * rtd fix * fix rtd Signed-off-by: rashidakanchwala <rashida_kanchwala@mckinsey.com> * fix rtd Signed-off-by: rashidakanchwala <rashida_kanchwala@mckinsey.com> * fix rtd - pls" Signed-off-by: rashidakanchwala <rashida_kanchwala@mckinsey.com> * add nitpick ignore Signed-off-by: rashidakanchwala <rashida_kanchwala@mckinsey.com> * test again Signed-off-by: rashidakanchwala <rashida_kanchwala@mckinsey.com> * move tracking datasets to constant Signed-off-by: rashidakanchwala <rashida_kanchwala@mckinsey.com> * remove comma Signed-off-by: rashidakanchwala <rashida_kanchwala@mckinsey.com> * remove Newtype from json_dataset" Signed-off-by: rashidakanchwala <rashida_kanchwala@mckinsey.com> * pls work Signed-off-by: rashidakanchwala <rashida_kanchwala@mckinsey.com> * confirm rtd works locally Signed-off-by: rashidakanchwala <rashida_kanchwala@mckinsey.com> * juanlu's fix Signed-off-by: rashidakanchwala <rashida_kanchwala@mckinsey.com> * fix tests Signed-off-by: rashidakanchwala <rashida_kanchwala@mckinsey.com> * remove unnecessary stuff from conf.py Signed-off-by: rashidakanchwala <rashida_kanchwala@mckinsey.com> * fixes based on review Signed-off-by: rashidakanchwala <rashida_kanchwala@mckinsey.com> * changes based on review Signed-off-by: rashidakanchwala <rashida_kanchwala@mckinsey.com> * fix tests Signed-off-by: rashidakanchwala <rashida_kanchwala@mckinsey.com> * add suffix Preview Signed-off-by: rashidakanchwala <rashida_kanchwala@mckinsey.com> * change img return type to bytes Signed-off-by: rashidakanchwala <rashida_kanchwala@mckinsey.com> * fix tests Signed-off-by: rashidakanchwala <rashida_kanchwala@mckinsey.com> * update release note * fix lint --------- Signed-off-by: rashidakanchwala <rashida_kanchwala@mckinsey.com> Co-authored-by: ravi-kumar-pilla <ravi_kumar_pilla@mckinsey.com> Co-authored-by: Sajid Alam <90610031+SajidAlamQB@users.noreply.github.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * fix(datasets): Drop pyarrow constraint when using snowpark (#538) * Free pyarrow req Signed-off-by: Felipe Monroy <felipe.m02@gmail.com> * Free pyarrow req Signed-off-by: Felipe Monroy <felipe.m02@gmail.com> --------- Signed-off-by: Felipe Monroy <felipe.m02@gmail.com> Co-authored-by: Nok Lam Chan <nok.lam.chan@quantumblack.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * docs: Update kedro-telemetry docs on which data is collected (#546) * Update data being collected --------- Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com> Signed-off-by: Dmitry Sorokin <40151847+DimedS@users.noreply.github.com> Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * ci(docker): Trying to fix e2e tests (#548) * Pin psutil Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Add no capture to test Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update pip version Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * U…

* refactor(datasets): deprecate "DataSet" type names (#328) * refactor(datasets): deprecate "DataSet" type names (api) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (biosequence) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (dask) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (databricks) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (email) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (geopandas) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (holoviews) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (json) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (matplotlib) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (networkx) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (pandas) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (pandas.csv_dataset) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (pandas.deltatable_dataset) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (pandas.excel_dataset) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (pandas.feather_dataset) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (pandas.gbq_dataset) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (pandas.generic_dataset) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (pandas.hdf_dataset) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (pandas.json_dataset) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (pandas.parquet_dataset) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (pandas.sql_dataset) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (pandas.xml_dataset) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (pickle) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (pillow) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (plotly) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (polars) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (redis) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (snowflake) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (spark) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (svmlight) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (tensorflow) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (text) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (tracking) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (video) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): deprecate "DataSet" type names (yaml) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * chore(datasets): ignore TensorFlow coverage issues Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> --------- Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * added basic code for geotiff Signed-off-by: tgoelles <thomas.goelles@gmail.com> * renamed to xarray Signed-off-by: tgoelles <thomas.goelles@gmail.com> * renamed to xarray Signed-off-by: tgoelles <thomas.goelles@gmail.com> * added load and self args Signed-off-by: tgoelles <thomas.goelles@gmail.com> * only local files Signed-off-by: tgoelles <thomas.goelles@gmail.com> * added empty test Signed-off-by: tgoelles <thomas.goelles@gmail.com> * added test data Signed-off-by: tgoelles <thomas.goelles@gmail.com> * added rioxarray requirements Signed-off-by: tgoelles <thomas.goelles@gmail.com> * reformat with black Signed-off-by: tgoelles <thomas.goelles@gmail.com> * rioxarray 0.14 Signed-off-by: tgoelles <thomas.goelles@gmail.com> * rioxarray 0.15 Signed-off-by: tgoelles <thomas.goelles@gmail.com> * rioxarray 0.12 Signed-off-by: tgoelles <thomas.goelles@gmail.com> * rioxarray 0.9 Signed-off-by: tgoelles <thomas.goelles@gmail.com> * fixed dataset typo Signed-off-by: tgoelles <thomas.goelles@gmail.com> * fixed docstring for sphinx Signed-off-by: tgoelles <thomas.goelles@gmail.com> * run black Signed-off-by: tgoelles <thomas.goelles@gmail.com> * sort imports Signed-off-by: tgoelles <thomas.goelles@gmail.com> * class docstring Signed-off-by: tgoelles <thomas.goelles@gmail.com> * black Signed-off-by: tgoelles <thomas.goelles@gmail.com> * fixed pylint Signed-off-by: tgoelles <thomas.goelles@gmail.com> * added release notes Signed-off-by: tgoelles <thomas.goelles@gmail.com> * added yaml example Signed-off-by: tgoelles <thomas.goelles@gmail.com> * improve testing WIP Signed-off-by: tgoelles <thomas.goelles@gmail.com> * basic test success Signed-off-by: tgoelles <thomas.goelles@gmail.com> * test reloaded Signed-off-by: tgoelles <thomas.goelles@gmail.com> * test exists Signed-off-by: tgoelles <thomas.goelles@gmail.com> * added version Signed-off-by: tgoelles <thomas.goelles@gmail.com> * basic test suite Signed-off-by: tgoelles <thomas.goelles@gmail.com> * run black Signed-off-by: tgoelles <thomas.goelles@gmail.com> * added example and test it Signed-off-by: tgoelles <thomas.goelles@gmail.com> * deleted duplications Signed-off-by: tgoelles <thomas.goelles@gmail.com> * fixed position of example Signed-off-by: tgoelles <thomas.goelles@gmail.com> * black Signed-off-by: tgoelles <thomas.goelles@gmail.com> * style: Introduce `ruff` for linting in all plugins. (#354) Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> * feat(datasets): create custom `DeprecationWarning` (#356) * feat(datasets): create custom `DeprecationWarning` Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * feat(datasets): use the custom deprecation warning Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * chore(datasets): show Kedro's deprecation warnings Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * fix(datasets): remove unused imports in test files Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> --------- Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * docs(datasets): add note about DataSet deprecation (#357) Signed-off-by: tgoelles <thomas.goelles@gmail.com> * test(datasets): skip `tensorflow` tests on Windows (#363) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * ci: Pin `tables` version (#370) * Pin tables version Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Also fix kedro-airflow Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Revert trying to fix airflow Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> --------- Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * build(datasets): Release `1.7.1` (#378) Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * docs: Update CONTRIBUTING.md and add one for `kedro-datasets` (#379) Update CONTRIBUTING.md + add one for kedro-datasets Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * ci(datasets): Run tensorflow tests separately from other dataset tests (#377) Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * feat: Kedro-Airflow convert all pipelines option (#335) * feat: kedro airflow convert --all option Signed-off-by: Simon Brugman <sfbbrugman@gmail.com> * docs: release docs Signed-off-by: Simon Brugman <sfbbrugman@gmail.com> --------- Signed-off-by: Simon Brugman <sfbbrugman@gmail.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * docs(datasets): blacken code in rst literal blocks (#362) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * docs: cloudpickle is an interesting extension of the pickle functionality (#361) Signed-off-by: H. Felix Wittmann <hfwittmann@gmail.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * fix(datasets): Fix secret scan entropy error (#383) Fix secret scan entropy error Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * style: Rename mentions of `DataSet` to `Dataset` in `kedro-airflow` and `kedro-telemetry` (#384) Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * feat(datasets): Migrated `PartitionedDataSet` and `IncrementalDataSet` from main repository to kedro-datasets (#253) Signed-off-by: Peter Bludau <ptrbld.dev@gmail.com> Co-authored-by: Merel Theisen <merel.theisen@quantumblack.com> * fix: backwards compatibility for `kedro-airflow` (#381) Signed-off-by: Simon Brugman <sfbbrugman@gmail.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * added metadata Signed-off-by: tgoelles <thomas.goelles@gmail.com> * after linting Signed-off-by: tgoelles <thomas.goelles@gmail.com> * ignore ruff PLR0913 Signed-off-by: tgoelles <thomas.goelles@gmail.com> * fix(datasets): Don't warn for SparkDataset on Databricks when using s3 (#341) Signed-off-by: Alistair McKelvie <alistair.mckelvie@gmail.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * chore: Hot fix for RTD due to bad pip version (#396) fix RTD Signed-off-by: Nok <nok.lam.chan@quantumblack.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * chore: Pin pip version temporarily (#398) * Pin pip version temporarily Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Hive support failures Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Also pin pip on lint Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Temporary ignore databricks spark tests Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> --------- Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * perf(datasets): don't create connection until need (#281) * perf(datasets): delay `Engine` creation until need Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * chore: don't check coverage in TYPE_CHECKING block Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * perf(datasets): don't connect in `__init__` method Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * test(datasets): fix tests to touch `create_engine` Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * perf(datasets): don't connect in `__init__` method Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * style(datasets): exec Ruff on sql_dataset.py :dog: Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Undo changes to `engines` values type (for Sphinx) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Patch Sphinx build by removing `Engine` references * perf(datasets): don't connect in `__init__` method Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * chore(datasets): don't require coverage for import * chore(datasets): del unused `TYPE_CHECKING` import * docs(datasets): document lazy connection in README * perf(datasets): remove create in `SQLQueryDataset` Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): do not return the created conn Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> --------- Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * chore: Drop Python 3.7 support for kedro-plugins (#392) * Remove references to Python 3.7 Signed-off-by: lrcouto <laurarccouto@gmail.com> * Revert kedro-dataset changes Signed-off-by: lrcouto <laurarccouto@gmail.com> * Revert kedro-dataset changes Signed-off-by: lrcouto <laurarccouto@gmail.com> * Add information to release docs Signed-off-by: lrcouto <laurarccouto@gmail.com> --------- Signed-off-by: lrcouto <laurarccouto@gmail.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * feat(datasets): support Polars lazy evaluation (#350) * feat(datasets) add PolarsDataset to support Polars's Lazy API Signed-off-by: Matthias Roels <matthias.roels21@gmail.com> * Fix(datasets): rename PolarsDataSet to PolarsDataSet Add PolarsDataSet as an alias for PolarsDataset with deprecation warning. Signed-off-by: Matthias Roels <matthias.roels21@gmail.com> * Fix(datasets): apply ruff linting rules Signed-off-by: Matthias Roels <matthias.roels21@gmail.com> * Fix(datasets): Correct pattern matching when Raising exceptions Corrected PolarsDataSet to PolarsDataset in the pattern to match in test_load_missing_file Signed-off-by: Matthias Roels <matthias.roels21@gmail.com> * fix(datasets): clean up PolarsDataset related code Remove reference to PolarsDataSet as this is not required for new dataset implementations. Signed-off-by: Matthias Roels <matthias.roels21@gmail.com> * feat(datasets): Rename Polars Datasets to better describe their intent Signed-off-by: Matthias Roels <matthias.roels21@gmail.com> * feat(datasets): clean up LazyPolarsDataset Signed-off-by: Matthias Roels <matthias.roels21@gmail.com> * fix(datasets): increase test coverage for PolarsDataset classes Signed-off-by: Matthias Roels <matthias.roels21@gmail.com> * docs(datasets): add renamed Polars datasets to docs Signed-off-by: Matthias Roels <matthias.roels21@gmail.com> * docs(datasets): Add new polars datasets to release notes Signed-off-by: Matthias Roels <mroels2@its.jnj.com> * fix(datasets): load_args not properly passed to LazyPolarsDataset.load Signed-off-by: Matthias Roels <matthias.roels21@gmail.com> * docs(datasets): fix spelling error in release notes Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Signed-off-by: Matthias Roels <matthias.roels21@gmail.com> --------- Signed-off-by: Matthias Roels <matthias.roels21@gmail.com> Signed-off-by: Matthias Roels <mroels2@its.jnj.com> Signed-off-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Co-authored-by: Matthias Roels <mroels2@its.jnj.com> Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * build(datasets): Release `1.8.0` (#406) Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * build(airflow): Release 0.7.0 (#407) * bump version Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update release notes Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> --------- Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * build(telemetry): Release 0.3.0 (#408) Bump version Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: Ankita Katiyar <110245118+ankatiyar@users.noreply.github.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * build(docker): Release 0.4.0 (#409) Bump version Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: Ankita Katiyar <110245118+ankatiyar@users.noreply.github.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * style(airflow): blacken README.md of Kedro-Airflow (#418) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * fix(datasets): Fix missing jQuery (#414) Fix missing jQuery Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * fix(datasets): Fix Lazy Polars dataset to use the new-style base class (#413) * Fix Lazy Polars dataset to use the new-style base class Fix gh-412 Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Update release notes Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Revert "Update release notes" This reverts commit 92ceea6d8fa412abf3d8abd28a2f0a22353867ed. --------- Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Signed-off-by: Sajid Alam <90610031+SajidAlamQB@users.noreply.github.com> Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Co-authored-by: Sajid Alam <90610031+SajidAlamQB@users.noreply.github.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * chore(datasets): lazily load `partitions` classes (#411) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * docs(datasets): fix code blocks and `data_set` use (#417) * chore(datasets): lazily load `partitions` classes Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * test(datasets): run doctests to check examples run Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * test(datasets): keep running tests amidst failures Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * docs(datasets): format ManagedTableDataset example Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * chore(datasets): ignore breaking mods for doctests Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * style(airflow): black code in Kedro-Airflow README Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * docs(datasets): fix example syntax, and autoformat Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * docs(datasets): remove `kedro.extras.datasets` ref Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * docs(datasets): remove `>>> ` prefix for YAML code Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * docs(datasets): remove `kedro.extras.datasets` ref Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * docs(datasets): replace `data_set`s with `dataset`s Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * chore(datasets): undo changes for running doctests Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * revert(datasets): undo lazily load `partitions` classes Refs: 3fdc5a8efa034fa9a18b7683a942415915b42fb5 Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * revert(airflow): undo black code in Kedro-Airflow README Refs: dc3476ea36bac98e2adcc0b52a11b0f90001e31d Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> --------- Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * fix: TF model load failure when model is saved as a TensorFlow Saved Model format (#410) * fixes TF model load failure when model is saved as a TensorFlow Saved Model format when a model is saved in the TensorFlow SavedModel format ("tf" default option in tf.save_model when using TF 2.x) via the catalog.xml file, the subsequent loading of that model for further use in a subsequent node fails. The issue is linked to the fact that the model files don't get copied into the temporary folder, presumably because the _fs.get function "thinks" that the provided path is a file and not a folder. Adding an terminating "/" to the path fixes the issue. Signed-off-by: Edouard59 <68538605+Edouard59@users.noreply.github.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * chore: Drop support for Python 3.7 on kedro-datasets (#419) * Drop support for Python 3.7 on kedro-datasets Signed-off-by: lrcouto <laurarccouto@gmail.com> * Remove redundant 3.8 markers Signed-off-by: lrcouto <laurarccouto@gmail.com> --------- Signed-off-by: lrcouto <laurarccouto@gmail.com> Signed-off-by: L. R. Couto <57910428+lrcouto@users.noreply.github.com> Signed-off-by: Sajid Alam <90610031+SajidAlamQB@users.noreply.github.com> Co-authored-by: Sajid Alam <90610031+SajidAlamQB@users.noreply.github.com> * test(datasets): run doctests to check examples run (#416) * chore(datasets): lazily load `partitions` classes Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * test(datasets): run doctests to check examples run Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * test(datasets): keep running tests amidst failures Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * docs(datasets): format ManagedTableDataset example Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * chore(datasets): ignore breaking mods for doctests Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * style(airflow): black code in Kedro-Airflow README Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * docs(datasets): fix example syntax, and autoformat Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * docs(datasets): remove `kedro.extras.datasets` ref Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * docs(datasets): remove `>>> ` prefix for YAML code Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * docs(datasets): remove `kedro.extras.datasets` ref Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * docs(datasets): replace `data_set`s with `dataset`s Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): run doctests separately Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * separate dataset-doctests Signed-off-by: Nok <nok.lam.chan@quantumblack.com> * chore(datasets): ignore non-passing tests to make CI pass Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * chore(datasets): fix comment location Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * chore(datasets): fix .py.py Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * chore(datasets): don't measure coverage on doctest run Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * build(datasets): fix windows and snowflake stuff in Makefile Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> --------- Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: Nok <nok.lam.chan@quantumblack.com> Co-authored-by: Nok <nok.lam.chan@quantumblack.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * feat(datasets): Add support for `databricks-connect>=13.0` (#352) Signed-off-by: Miguel Rodriguez Gutierrez <miguel7r@hotmail.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * fix(telemetry): remove double execution by moving to after catalog created hook (#422) * remove double execution by moving to after catalog created hook Signed-off-by: Florian Roessler <roessler.fd@gmail.com> * update release notes Signed-off-by: Florian Roessler <roessler.fd@gmail.com> * fix tests Signed-off-by: Florian Roessler <roessler.fd@gmail.com> * remove unsued fixture Signed-off-by: Florian Roessler <roessler.fd@gmail.com> --------- Signed-off-by: Florian Roessler <roessler.fd@gmail.com> Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * docs: Add python version support policy to plugin `README.md`s (#425) * Add python version support policy to plugin readmes Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> * Temporarily pin connexion Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> --------- Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * docs(airflow): Use new docs link (#393) Use new docs link Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * style: Add shared CSS and meganav to datasets docs (#400) * Add shared CSS and meganav Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Add end of file Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Add new heap data source Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * adjust heap parameter Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Remove nav_version next to Kedro logo in top left; add Kedro logo * Revise project name and author name Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Use full kedro icon and type for logo * Add close btn to mobile nav Signed-off-by: vladimir-mck <106236933+vladimir-mck@users.noreply.github.com> * Add css for mobile nav logo image Signed-off-by: vladimir-mck <106236933+vladimir-mck@users.noreply.github.com> * Update close button for mobile nav Signed-off-by: vladimir-mck <106236933+vladimir-mck@users.noreply.github.com> * Add open button to mobile nav Signed-off-by: vladimir-mck <106236933+vladimir-mck@users.noreply.github.com> * Delete kedro-datasets/docs/source/kedro-horizontal-color-on-light.svg Signed-off-by: vladimir-mck <106236933+vladimir-mck@users.noreply.github.com> * Update conf.py Signed-off-by: vladimir-mck <106236933+vladimir-mck@users.noreply.github.com> * Update layout.html Add links to subprojects Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Remove svg from docs -- not needed?? Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * linter error fix Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> --------- Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> Signed-off-by: vladimir-mck <106236933+vladimir-mck@users.noreply.github.com> Co-authored-by: Tynan DeBold <thdebold@gmail.com> Co-authored-by: vladimir-mck <106236933+vladimir-mck@users.noreply.github.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * feat(datasets): Add Hugging Face datasets (#344) * Add HuggingFace datasets Co-authored-by: Danny Farah <danny_farah@mckinsey.com> Co-authored-by: Kevin Koga <Kevin_Koga@mckinsey.com> Co-authored-by: Mate Scharnitzky <Mate_Scharnitzky@mckinsey.com> Co-authored-by: Tomer Shor <Tomer_Shor@mckinsey.com> Co-authored-by: Pierre-Yves Mousset <Pierre-Yves_Mousset@mckinsey.com> Co-authored-by: Bela Chupal <Bela_chuphal@mckinsey.com> Co-authored-by: Khangjrakpam Arjun <Khangjrakpam_Arjun@mckinsey.com> Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Apply suggestions from code review Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Co-authored-by: Joel <35801847+datajoely@users.noreply.github.com> Co-authored-by: Nok Lam Chan <nok.lam.chan@quantumblack.com> * Typo Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Fix docstring Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Add docstring for HFTransformerPipelineDataset Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Use intersphinx for cross references in Hugging Face docstrings Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Add docstring for HFDataset Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Add missing test dependencies Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Add tests for huggingface datasets Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Fix HFDataset.save Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Add test for HFDataset.list_datasets Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Use new name Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Consolidate imports Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> --------- Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Co-authored-by: Danny Farah <danny_farah@mckinsey.com> Co-authored-by: Kevin Koga <Kevin_Koga@mckinsey.com> Co-authored-by: Mate Scharnitzky <Mate_Scharnitzky@mckinsey.com> Co-authored-by: Tomer Shor <Tomer_Shor@mckinsey.com> Co-authored-by: Pierre-Yves Mousset <Pierre-Yves_Mousset@mckinsey.com> Co-authored-by: Bela Chupal <Bela_chuphal@mckinsey.com> Co-authored-by: Khangjrakpam Arjun <Khangjrakpam_Arjun@mckinsey.com> Co-authored-by: Joel <35801847+datajoely@users.noreply.github.com> Co-authored-by: Nok Lam Chan <nok.lam.chan@quantumblack.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * test(datasets): fix `dask.ParquetDataset` doctests (#439) * test(datasets): fix `dask.ParquetDataset` doctests Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * test(datasets): use `tmp_path` fixture in doctests Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * test(datasets): simplify by not passing the schema Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * test(datasets): ignore conftest for doctests cover Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Create MANIFEST.in Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> --------- Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * refactor: Remove `DataSet` aliases and mentions (#440) Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> * chore(datasets): replace "Pyspark" with "PySpark" (#423) Consistently write "PySpark" rather than "Pyspark" Also, fix list formatting Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * test(datasets): make `api.APIDataset` doctests run (#448) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * chore(datasets): Fix `pandas.GenericDataset` doctest (#445) Fix pandas.GenericDataset doctest Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * feat(datasets): make datasets arguments keywords only (#358) * feat(datasets): make `APIDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `BioSequenceDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `ParquetDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `EmailMessageDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `GeoJSONDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `HoloviewsWriter.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `JSONDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `MatplotlibWriter.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `GMLDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `GraphMLDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make NetworkX `JSONDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `PickleDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `ImageDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make plotly `JSONDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `PlotlyDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make polars `CSVDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make polars `GenericDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make redis `PickleDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `SnowparkTableDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `SVMLightDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `TensorFlowModelDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `TextDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `YAMLDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `ManagedTableDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `VideoDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `CSVDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `DeltaTableDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `ExcelDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `FeatherDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `GBQTableDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `GenericDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make pandas `JSONDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make pandas `ParquerDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `SQLTableDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `XMLDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `HDFDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `DeltaTableDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `SparkDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `SparkHiveDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `SparkJDBCDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `SparkStreamingDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `IncrementalDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * feat(datasets): make `LazyPolarsDataset.__init__` keyword only Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * docs(datasets): update doctests for HoloviewsWriter Signed-off-by: Felix Scherz <felixwscherz@gmail.com> * Update release notes Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> --------- Signed-off-by: Felix Scherz <felixwscherz@gmail.com> Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Co-authored-by: Felix Scherz <felixwscherz@gmail.com> Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Co-authored-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * chore: Drop support for python 3.8 on kedro-datasets (#442) * Drop support for python 3.8 on kedro-datasets --------- Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com> Signed-off-by: Dmitry Sorokin <40151847+DimedS@users.noreply.github.com> Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * test(datasets): add outputs to matplotlib doctests (#449) * test(datasets): add outputs to matplotlib doctests Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Update Makefile Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Reformat code example, line length is short enough * Update kedro-datasets/kedro_datasets/matplotlib/matplotlib_writer.py Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> --------- Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * chore(datasets): Fix more doctest issues (#451) Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Co-authored-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * test(datasets): fix failing doctests in Windows CI (#457) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * chore(datasets): fix accidental reference to NumPy (#450) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * chore(datasets): don't pollute dev env in doctests (#452) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * feat: Add tools to heap event (#430) * Add add-on data to heap event Signed-off-by: lrcouto <laurarccouto@gmail.com> * Move addons logic to _get_project_property Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Add condition for pyproject.toml Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Fix tests Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Fix tests Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * add tools to mock Signed-off-by: lrcouto <laurarccouto@gmail.com> * lint Signed-off-by: lrcouto <laurarccouto@gmail.com> * Update tools test Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Add after_context_created tools test Signed-off-by: lrcouto <laurarccouto@gmail.com> * Update rename to tools Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update kedro-telemetry/tests/test_plugin.py Co-authored-by: Sajid Alam <90610031+SajidAlamQB@users.noreply.github.com> Signed-off-by: Ankita Katiyar <110245118+ankatiyar@users.noreply.github.com> --------- Signed-off-by: lrcouto <laurarccouto@gmail.com> Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: Ankita Katiyar <110245118+ankatiyar@users.noreply.github.com> Co-authored-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Co-authored-by: Ankita Katiyar <110245118+ankatiyar@users.noreply.github.com> Co-authored-by: Sajid Alam <90610031+SajidAlamQB@users.noreply.github.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * ci(datasets): install deps in single `pip install` (#454) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * build(datasets): Bump s3fs (#463) * Use mocking for AWS responses Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> * Add change to release notes Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> * Update release notes Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> * Use pytest xfail instead of commenting out test Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> --------- Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * test(datasets): make SQL dataset examples runnable (#455) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * fix(datasets): correct pandas-gbq as py311 dependency (#460) * update pandas-gbq dependency declaration Signed-off-by: Onur Kuru <kuru.onur1@gmail.com> * fix fmt Signed-off-by: Onur Kuru <kuru.onur1@gmail.com> --------- Signed-off-by: Onur Kuru <kuru.onur1@gmail.com> Co-authored-by: Ahdra Merali <90615669+AhdraMeraliQB@users.noreply.github.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * docs(datasets): Document `IncrementalDataset` (#468) Document IncrementalDataset Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * chore: Update datasets to be arguments keyword only (#466) Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * chore: Clean up code for old dataset syntax compatibility (#465) Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * chore: Update scikit-learn version (#469) Update scikit-learn version Signed-off-by: Nok Lam Chan <nok.lam.chan@quantumblack.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * feat(datasets): support versioning data partitions (#447) * feat(datasets): support versioning data partitions Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Remove unused import Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * chore(datasets): use keyword arguments when needed Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Apply suggestions from code review Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Update kedro-datasets/kedro_datasets/partitions/partitioned_dataset.py Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> --------- Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * docs(datasets): Improve documentation index (#428) Rework documentation index Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * docs(datasets): update wrong docstring about `con` (#461) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * build(datasets): Release `2.0.0` (#472) Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * ci(telemetry): Pin `PyYAML` (#474) Pin PyYaml Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * build(telemetry): Release 0.3.1 (#475) Signed-off-by: tgoelles <thomas.goelles@gmail.com> * docs(datasets): Fix broken links in README (#477) Fix broken links in README Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * chore(datasets): replace more "data_set" instances (#476) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * chore(datasets): Fix doctests (#488) Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * chore(datasets): Fix delta + incremental dataset docstrings (#489) Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * chore(airflow): Post 0.19 cleanup (#478) * bump version Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Unbump version and clean test Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update e2e tests Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update e2e tests Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update e2e tests Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update e2e tests Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Split big test into smaller tests Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update conftest Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update conftest Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Fix coverage Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Try unpin airflow Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * remove datacatalog step Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Change node Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * update tasks test step Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Revert to older airflow and constraint pendulum Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update template Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update message in e2e step Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Final cleanup Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update kedro-airflow/pyproject.toml Signed-off-by: Nok Lam Chan <nok.lam.chan@quantumblack.com> * Pin apache-airflow again Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> --------- Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: Nok Lam Chan <nok.lam.chan@quantumblack.com> Co-authored-by: Nok Lam Chan <nok.lam.chan@quantumblack.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * build(airflow): Release 0.8.0 (#491) Bump version Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * fix: telemetry metadata (#495) --------- Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * fix: Update tests on kedro-docker for 0.5.0 release. (#496) * bump version to 0.5.0 Signed-off-by: lrcouto <laurarccouto@gmail.com> * bump version to 0.5.0 Signed-off-by: lrcouto <laurarccouto@gmail.com> * update e2e tests to use new starters Signed-off-by: lrcouto <laurarccouto@gmail.com> * Lint Signed-off-by: lrcouto <laurarccouto@gmail.com> * update e2e tests to use new starters Signed-off-by: lrcouto <laurarccouto@gmail.com> * fix test path for e2e tests Signed-off-by: lrcouto <laurarccouto@gmail.com> * fix requirements path on dockerfiles Signed-off-by: lrcouto <laurarccouto@gmail.com> * update tests to fit with current log format Signed-off-by: lrcouto <laurarccouto@gmail.com> * update tests to fit with current log format Signed-off-by: lrcouto <laurarccouto@gmail.com> * update tests to fit with current log format Signed-off-by: lrcouto <laurarccouto@gmail.com> * Remove redundant test Signed-off-by: lrcouto <laurarccouto@gmail.com> * Alter test for custom GID and UID Signed-off-by: lrcouto <laurarccouto@gmail.com> * Update release notes Signed-off-by: lrcouto <laurarccouto@gmail.com> * Revert version bump to put in in separate PR Signed-off-by: lrcouto <laurarccouto@gmail.com> --------- Signed-off-by: lrcouto <laurarccouto@gmail.com> Signed-off-by: L. R. Couto <57910428+lrcouto@users.noreply.github.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * build: Release kedro-docker 0.5.0 (#497) * bump version to 0.5.0 Signed-off-by: lrcouto <laurarccouto@gmail.com> * bump version to 0.5.0 Signed-off-by: lrcouto <laurarccouto@gmail.com> * update e2e tests to use new starters Signed-off-by: lrcouto <laurarccouto@gmail.com> * Lint Signed-off-by: lrcouto <laurarccouto@gmail.com> * update e2e tests to use new starters Signed-off-by: lrcouto <laurarccouto@gmail.com> * fix test path for e2e tests Signed-off-by: lrcouto <laurarccouto@gmail.com> * fix requirements path on dockerfiles Signed-off-by: lrcouto <laurarccouto@gmail.com> * update tests to fit with current log format Signed-off-by: lrcouto <laurarccouto@gmail.com> * update tests to fit with current log format Signed-off-by: lrcouto <laurarccouto@gmail.com> * update tests to fit with current log format Signed-off-by: lrcouto <laurarccouto@gmail.com> * Remove redundant test Signed-off-by: lrcouto <laurarccouto@gmail.com> * Alter test for custom GID and UID Signed-off-by: lrcouto <laurarccouto@gmail.com> * Update release notes Signed-off-by: lrcouto <laurarccouto@gmail.com> * Revert version bump to put in in separate PR Signed-off-by: lrcouto <laurarccouto@gmail.com> * Bump kedro-docker to 0.5.0 Signed-off-by: lrcouto <laurarccouto@gmail.com> * Add release notes Signed-off-by: lrcouto <laurarccouto@gmail.com> * Update kedro-docker/RELEASE.md Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Signed-off-by: L. R. Couto <57910428+lrcouto@users.noreply.github.com> --------- Signed-off-by: lrcouto <laurarccouto@gmail.com> Signed-off-by: L. R. Couto <57910428+lrcouto@users.noreply.github.com> Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * chore(datasets): Update partitioned dataset docstring (#502) Update partitioned dataset docstring Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * Fix GeotiffDataset import + casing Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> * Fix lint Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * fix(datasets): Relax pandas.HDFDataSet dependencies which are broken on Windows (#426) * Relax pandas.HDFDataSet dependencies which are broken on Window (#402) Signed-off-by: Yolan Honoré-Rougé <yolan.honore.rouge@gmail.com> * Update RELEASE.md Signed-off-by: Yolan Honoré-Rougé <yolan.honore.rouge@gmail.com> * Apply suggestions from code review Signed-off-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> * Update kedro-datasets/setup.py Signed-off-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> --------- Signed-off-by: Yolan Honoré-Rougé <yolan.honore.rouge@gmail.com> Signed-off-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * fix: airflow metadata (#498) * Add example pipeline entry to metadata declaration Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Fix entry Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Make entries consistent Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Add tools to config Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * fix: telemetry metadata (#495) --------- Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com> Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Revert "Add tools to config" This reverts commit 14732d772a3c2f4787063071a68fdf1512c93488. Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Quick fix Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Lint Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Remove outdated config key Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Use kedro new instead of cookiecutter Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> --------- Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com> Co-authored-by: Dmitry Sorokin <40151847+DimedS@users.noreply.github.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * chore(airflow): Bump `apache-airflow` version (#511) * Bump apache airflow Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Change starter Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update e2e test steps Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update e2e test steps Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> --------- Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * ci(datasets): Unpin dask (#522) * Unpin dask Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update doctest Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update doctest Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update kedro-datasets/setup.py Co-authored-by: Nok Lam Chan <nok.lam.chan@quantumblack.com> Signed-off-by: Ankita Katiyar <110245118+ankatiyar@users.noreply.github.com> --------- Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: Ankita Katiyar <110245118+ankatiyar@users.noreply.github.com> Co-authored-by: Nok Lam Chan <nok.lam.chan@quantumblack.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * feat(datasets): Add `MatlabDataset` to `kedro-datasets` (#515) * Refork and commit kedro matlab datasets Signed-off-by: samuelleeshemen <samuel_lee_sj@aiap.sg> * Fix lint, add to docs Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Try fixing docstring Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Try fixing save Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Try fix docstest Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Fix unit tests Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update release notes: Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Not hardcode load mode Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> --------- Signed-off-by: samuelleeshemen <samuel_lee_sj@aiap.sg> Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Co-authored-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * ci(airflow): Pin `Flask-Session` version (#521) * Restrict pendulum version Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update airflow init step Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Remove pendulum pin Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update create connections step Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Pin flask session Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Add comment Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> --------- Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * feat: `kedro-airflow` group in memory nodes (#241) * feat: option to group in-memory nodes Signed-off-by: Simon Brugman <sfbbrugman@gmail.com> * fix: MemoryDataset Signed-off-by: Simon Brugman <sfbbrugman@gmail.com> * Update kedro-airflow/README.md Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Signed-off-by: Simon Brugman <sbrugman@users.noreply.github.com> * Update kedro-airflow/README.md Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Signed-off-by: Simon Brugman <sbrugman@users.noreply.github.com> * Update kedro-airflow/README.md Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Signed-off-by: Simon Brugman <sbrugman@users.noreply.github.com> * Update kedro-airflow/RELEASE.md Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Signed-off-by: Simon Brugman <sbrugman@users.noreply.github.com> * Update kedro-airflow/kedro_airflow/grouping.py Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Signed-off-by: Simon Brugman <sbrugman@users.noreply.github.com> * Update kedro-airflow/kedro_airflow/plugin.py Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Signed-off-by: Simon Brugman <sbrugman@users.noreply.github.com> * Update kedro-airflow/tests/test_node_grouping.py Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Signed-off-by: Simon Brugman <sbrugman@users.noreply.github.com> * Update kedro-airflow/tests/test_node_grouping.py Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Signed-off-by: Simon Brugman <sbrugman@users.noreply.github.com> * Update kedro-airflow/kedro_airflow/grouping.py Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Signed-off-by: Simon Brugman <sbrugman@users.noreply.github.com> * Update kedro-airflow/kedro_airflow/grouping.py Co-authored-by: Ankita Katiyar <110245118+ankatiyar@users.noreply.github.com> Signed-off-by: Simon Brugman <sbrugman@users.noreply.github.com> * fix: tests Signed-off-by: Simon Brugman <sfbbrugman@gmail.com> * Bump minimum kedro version Signed-off-by: Simon Brugman <sfbbrugman@gmail.com> * fixes Signed-off-by: Simon Brugman <sfbbrugman@gmail.com> --------- Signed-off-by: Simon Brugman <sfbbrugman@gmail.com> Signed-off-by: Simon Brugman <sbrugman@users.noreply.github.com> Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Co-authored-by: Ankita Katiyar <110245118+ankatiyar@users.noreply.github.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * ci(datasets): Update pyproject.toml to pin Kedro 0.19 for kedro-datasets (#526) Update pyproject.toml Signed-off-by: Nok Lam Chan <nok.lam.chan@quantumblack.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * feat(airflow): include environment name in DAG filename (#492) * feat: include environment name in DAG file Signed-off-by: Simon Brugman <sfbbrugman@gmail.com> * doc: add update to release notes Signed-off-by: Simon Brugman <sfbbrugman@gmail.com> --------- Signed-off-by: Simon Brugman <sfbbrugman@gmail.com> Co-authored-by: Ankita Katiyar <110245118+ankatiyar@users.noreply.github.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * feat(datasets): Enable search-as-you type on Kedro-datasets docs (#532) * done Signed-off-by: rashidakanchwala <rashida_kanchwala@mckinsey.com> * fix lint Signed-off-by: rashidakanchwala <rashida_kanchwala@mckinsey.com> --------- Signed-off-by: rashidakanchwala <rashida_kanchwala@mckinsey.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * fix(datasets): Debug and fix `kedro-datasets` nightly build failures (#541) * pin deltalake * Update kedro-datasets/setup.py Signed-off-by: Juan Luis Cano Rodríguez <hello@juanlu.space> * Update setup.py Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * sort order and compare * Update setup.py * lint * pin deltalake * add comment to pin --------- Signed-off-by: Juan Luis Cano Rodríguez <hello@juanlu.space> Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * feat(datasets): Dataset Preview Refactor (#504) * test * done * change from _preview to preview * fix lint and tests * added docstrings * rtd fix * rtd fix * fix rtd Signed-off-by: rashidakanchwala <rashida_kanchwala@mckinsey.com> * fix rtd Signed-off-by: rashidakanchwala <rashida_kanchwala@mckinsey.com> * fix rtd - pls" Signed-off-by: rashidakanchwala <rashida_kanchwala@mckinsey.com> * add nitpick ignore Signed-off-by: rashidakanchwala <rashida_kanchwala@mckinsey.com> * test again Signed-off-by: rashidakanchwala <rashida_kanchwala@mckinsey.com> * move tracking datasets to constant Signed-off-by: rashidakanchwala <rashida_kanchwala@mckinsey.com> * remove comma Signed-off-by: rashidakanchwala <rashida_kanchwala@mckinsey.com> * remove Newtype from json_dataset" Signed-off-by: rashidakanchwala <rashida_kanchwala@mckinsey.com> * pls work Signed-off-by: rashidakanchwala <rashida_kanchwala@mckinsey.com> * confirm rtd works locally Signed-off-by: rashidakanchwala <rashida_kanchwala@mckinsey.com> * juanlu's fix Signed-off-by: rashidakanchwala <rashida_kanchwala@mckinsey.com> * fix tests Signed-off-by: rashidakanchwala <rashida_kanchwala@mckinsey.com> * remove unnecessary stuff from conf.py Signed-off-by: rashidakanchwala <rashida_kanchwala@mckinsey.com> * fixes based on review Signed-off-by: rashidakanchwala <rashida_kanchwala@mckinsey.com> * changes based on review Signed-off-by: rashidakanchwala <rashida_kanchwala@mckinsey.com> * fix tests Signed-off-by: rashidakanchwala <rashida_kanchwala@mckinsey.com> * add suffix Preview Signed-off-by: rashidakanchwala <rashida_kanchwala@mckinsey.com> * change img return type to bytes Signed-off-by: rashidakanchwala <rashida_kanchwala@mckinsey.com> * fix tests Signed-off-by: rashidakanchwala <rashida_kanchwala@mckinsey.com> * update release note * fix lint --------- Signed-off-by: rashidakanchwala <rashida_kanchwala@mckinsey.com> Co-authored-by: ravi-kumar-pilla <ravi_kumar_pilla@mckinsey.com> Co-authored-by: Sajid Alam <90610031+SajidAlamQB@users.noreply.github.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * fix(datasets): Drop pyarrow constraint when using snowpark (#538) * Free pyarrow req Signed-off-by: Felipe Monroy <felipe.m02@gmail.com> * Free pyarrow req Signed-off-by: Felipe Monroy <felipe.m02@gmail.com> --------- Signed-off-by: Felipe Monroy <felipe.m02@gmail.com> Co-authored-by: Nok Lam Chan <nok.lam.chan@quantumblack.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * docs: Update kedro-telemetry docs on which data is collected (#546) * Update data being collected --------- Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com> Signed-off-by: Dmitry Sorokin <40151847+DimedS@users.noreply.github.com> Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * ci(docker): Trying to fix e2e tests (#548) * Pin psutil Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Add no capture to test Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update pip version Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update call Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update pip Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * pip ruamel Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * change pip v Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * change pip v Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * show stdout Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * use no cache dir Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * revert extra changes Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * pin pip Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * gitpod Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * pip inside dockerfile Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * pip pip inside dockerfile Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> --------- Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * chore: bump actions versions (#539) * Unpin pip and bump actions versions Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * remove version Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Revert unpinning of pip Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> --------- Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * docs(telemetry): Direct readers to Kedro documentation for further information on telemetry (#555) * Direct readers to Kedro documentation for further information on telemetry Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Wording improvements Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> Signed-off-by: Juan Luis Cano Rodríguez <hello@juanlu.space> * Amend README section Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> --------- Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Signed-off-by: Juan Luis Cano Rodríguez <hello@juanlu.space> Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> Signed-off-by: tgoelles <thomas.goelles@gmail.com> * fix: kedro-telemetry masking (#552) * Fix masking Signed-off-by: Dmitr…

Peter Bludau added 5 commits June 26, 2023 18:29

Added DataSets and tests

48d13bd

Signed-off-by: Peter Bludau <ptrbld.dev@gmail.com>

Linted files with project settings

e29bd93

Signed-off-by: Peter Bludau <ptrbld.dev@gmail.com>

Add missing typing

e545924

Signed-off-by: Peter Bludau <ptrbld.dev@gmail.com>

Update documentation

c8872cd

Signed-off-by: Peter Bludau <ptrbld.dev@gmail.com>

Added release notes

c4862b1

Signed-off-by: Peter Bludau <ptrbld.dev@gmail.com>

PtrBld changed the title ~~feat(datasets): Migrated PartitionedDataSet and IncrementalDataSet to from main repository to kedro-datasets~~ feat(datasets): Migrated PartitionedDataSet and IncrementalDataSet from main repository to kedro-datasets Jun 27, 2023

Peter Bludau and others added 4 commits June 27, 2023 15:54

Fix typo

57bfe3d

Signed-off-by: Peter Bludau <ptrbld.dev@gmail.com>

Merge branch 'main' into add-partitioned-datasets

1d01351

Make typing compatible with older versions of python

f73a588

Signed-off-by: Peter Bludau <ptrbld.dev@gmail.com>

Linting tests

76f2df1

Signed-off-by: Peter Bludau <ptrbld.dev@gmail.com>

astrojuanlu mentioned this pull request Jun 28, 2023

Plan for Renaming DataSet to Dataset kedro-org/kedro#2740

Closed

6 tasks

ankatiyar added the Community Issue/PR opened by the open-source community label Jul 14, 2023

jmholzer and others added 13 commits August 27, 2023 14:18

docs: Instructions on custom Airflow Operators (kedro-org#267)

3f27b95

instructions on custom Airflow Operators Signed-off-by: Simon Brugman <sfbbrugman@gmail.com>

chore(airflow): relax semver requirement bound (kedro-org#270)

4462e88

chore: bump semver Closes: kedro-org#269 Signed-off-by: Simon Brugman <sfbbrugman@gmail.com>

chore: Add nightly build (kedro-org#283)

d62e720

add nightly build

feat(datasets): bump dask version to the latest (kedro-org#282)

1b3e938

bump dask version to the latest Signed-off-by: Jian <jian.wang@cpnet.io> Co-authored-by: Nok Lam Chan <nok.lam.chan@quantumblack.com>

docs: Minor docs tweaks (kedro-org#266)

3b53085

Minor docs tweaks Signed-off-by: Simon Brugman <sfbbrugman@gmail.com> Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>

merelcht added 2 commits October 4, 2023 16:01

Add change to release notes + lint

706deb5

Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com>

Fix links

eb5aca4

Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com>

merelcht and others added 3 commits October 4, 2023 17:09

Fix tests

664b00b

Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com>

Merge branch 'main' into add-partitioned-datasets

69c17dc

Merge branch 'main' into add-partitioned-datasets

b0f0b29

merelcht mentioned this pull request Oct 5, 2023

ci(datasets): Run tensorflow tests separately from other dataset tests #377

Merged

4 tasks

merelcht and others added 3 commits October 6, 2023 14:16

Merge branch 'main' into add-partitioned-datasets

9b9f3ef

Mention rename to PartionedDataset and IncrementalDataset and use new…

d6ef0e5

… names in implementations + tests Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com>

Merge branch 'add-partitioned-datasets' of github.com:PtrBld/kedro-pl…

7e27344

…ugins into add-partitioned-datasets

merelcht requested review from astrojuanlu, deepyaman and SajidAlamQB October 6, 2023 13:58

Acknowledge contributor in release notes

96ff24c

Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com>

merelcht linked an issue Oct 9, 2023 that may be closed by this pull request

Migrate PartitionedDataSet and IncrementalDataSet to kedro-datasets #249

Closed

merelcht mentioned this pull request Oct 10, 2023

Rename all DataSet mentions to Dataset (excl. docs) kedro-org/kedro#3147

Merged

7 tasks

AhdraMeraliQB approved these changes Oct 10, 2023

View reviewed changes

kedro-datasets/tests/partitions/test_incremental_dataset.py Outdated Show resolved Hide resolved

merelcht and others added 2 commits October 10, 2023 17:19

Apply suggestions from code review

0be7a98

Co-authored-by: Ahdra Merali <90615669+AhdraMeraliQB@users.noreply.github.com>

Merge branch 'main' into add-partitioned-datasets

a728ae2

astrojuanlu reviewed Oct 11, 2023

View reviewed changes

Merge branch 'main' into add-partitioned-datasets

d8f214e

merelcht approved these changes Oct 11, 2023

View reviewed changes

merelcht merged commit 6d1fca4 into kedro-org:main Oct 11, 2023
16 checks passed

SajidAlamQB mentioned this pull request Oct 17, 2023

Remove PartitionedDataset and IncrementalDataset from kedro.io kedro-org/kedro#3187

Merged

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(datasets): Migrated `PartitionedDataSet` and `IncrementalDataSet` from main repository to kedro-datasets #253

feat(datasets): Migrated `PartitionedDataSet` and `IncrementalDataSet` from main repository to kedro-datasets #253

PtrBld commented Jun 27, 2023

PtrBld commented Jun 28, 2023

astrojuanlu commented Jun 28, 2023

PtrBld commented Jun 28, 2023

astrojuanlu commented Jun 28, 2023

astrojuanlu commented Aug 26, 2023

merelcht commented Oct 4, 2023

AhdraMeraliQB left a comment

astrojuanlu Oct 11, 2023

feat(datasets): Migrated PartitionedDataSet and IncrementalDataSet from main repository to kedro-datasets #253

feat(datasets): Migrated PartitionedDataSet and IncrementalDataSet from main repository to kedro-datasets #253

Conversation

PtrBld commented Jun 27, 2023

Description

Development notes

Checklist

PtrBld commented Jun 28, 2023

astrojuanlu commented Jun 28, 2023

PtrBld commented Jun 28, 2023

astrojuanlu commented Jun 28, 2023

astrojuanlu commented Aug 26, 2023

merelcht commented Oct 4, 2023

AhdraMeraliQB left a comment

Choose a reason for hiding this comment

astrojuanlu Oct 11, 2023

Choose a reason for hiding this comment

feat(datasets): Migrated `PartitionedDataSet` and `IncrementalDataSet` from main repository to kedro-datasets #253

feat(datasets): Migrated `PartitionedDataSet` and `IncrementalDataSet` from main repository to kedro-datasets #253