-
Notifications
You must be signed in to change notification settings - Fork 89
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: add matlab dataset into kedro-datasets #435
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks a lot for your pull request @samuel-lee-sj ! We'll need a couple of iterations for this to be ready, but this is a great start 💪🏽
You can also do make lint plugin=kedro-datasets
(from the kedro-plugins
directory) to automatically lint the code
When I try I get an error saying 'No hook with id 'ruff-' in stage 'manual'. Below is the full error message:
|
@samuel-lee-sj it's because of a typo: you wrote Lines 16 to 17 in f59e930
|
Hi @samuel-lee-sj , would you like to continue this PR or do you need some help finishing it? |
Hello @merelcht. I was under the impression that @ankatiyar was going to takeover. Let me know what you think! |
@samuel-lee-sj Could you fix the sign-off on the commits by following the instructions here and I can help with the rest! 😄 |
Hello @ankatiyar. I think I did it. Let me know if there are any more problems. |
@samuel-lee-sj, seems like the DCO test is still not passing. Would you mind trying rebasing again? |
…` from main repository to kedro-datasets (#253) Signed-off-by: Peter Bludau <ptrbld.dev@gmail.com> Co-authored-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: samuelleeshemen <samuel_lee_sj@aiap.sg>
Signed-off-by: Simon Brugman <sfbbrugman@gmail.com> Signed-off-by: samuelleeshemen <samuel_lee_sj@aiap.sg>
#341) Signed-off-by: Alistair McKelvie <alistair.mckelvie@gmail.com> Signed-off-by: samuelleeshemen <samuel_lee_sj@aiap.sg>
fix RTD Signed-off-by: Nok <nok.lam.chan@quantumblack.com> Signed-off-by: samuelleeshemen <samuel_lee_sj@aiap.sg>
* Pin pip version temporarily Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Hive support failures Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Also pin pip on lint Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Temporary ignore databricks spark tests Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> --------- Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: samuelleeshemen <samuel_lee_sj@aiap.sg>
* perf(datasets): delay `Engine` creation until need Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * chore: don't check coverage in TYPE_CHECKING block Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * perf(datasets): don't connect in `__init__` method Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * test(datasets): fix tests to touch `create_engine` Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * perf(datasets): don't connect in `__init__` method Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * style(datasets): exec Ruff on sql_dataset.py 🐶 Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Undo changes to `engines` values type (for Sphinx) Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Patch Sphinx build by removing `Engine` references * perf(datasets): don't connect in `__init__` method Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * chore(datasets): don't require coverage for import * chore(datasets): del unused `TYPE_CHECKING` import * docs(datasets): document lazy connection in README * perf(datasets): remove create in `SQLQueryDataset` Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * refactor(datasets): do not return the created conn Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> --------- Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: samuelleeshemen <samuel_lee_sj@aiap.sg>
* Remove references to Python 3.7 Signed-off-by: lrcouto <laurarccouto@gmail.com> * Revert kedro-dataset changes Signed-off-by: lrcouto <laurarccouto@gmail.com> * Revert kedro-dataset changes Signed-off-by: lrcouto <laurarccouto@gmail.com> * Add information to release docs Signed-off-by: lrcouto <laurarccouto@gmail.com> --------- Signed-off-by: lrcouto <laurarccouto@gmail.com> Signed-off-by: samuelleeshemen <samuel_lee_sj@aiap.sg>
* feat(datasets) add PolarsDataset to support Polars's Lazy API Signed-off-by: Matthias Roels <matthias.roels21@gmail.com> * Fix(datasets): rename PolarsDataSet to PolarsDataSet Add PolarsDataSet as an alias for PolarsDataset with deprecation warning. Signed-off-by: Matthias Roels <matthias.roels21@gmail.com> * Fix(datasets): apply ruff linting rules Signed-off-by: Matthias Roels <matthias.roels21@gmail.com> * Fix(datasets): Correct pattern matching when Raising exceptions Corrected PolarsDataSet to PolarsDataset in the pattern to match in test_load_missing_file Signed-off-by: Matthias Roels <matthias.roels21@gmail.com> * fix(datasets): clean up PolarsDataset related code Remove reference to PolarsDataSet as this is not required for new dataset implementations. Signed-off-by: Matthias Roels <matthias.roels21@gmail.com> * feat(datasets): Rename Polars Datasets to better describe their intent Signed-off-by: Matthias Roels <matthias.roels21@gmail.com> * feat(datasets): clean up LazyPolarsDataset Signed-off-by: Matthias Roels <matthias.roels21@gmail.com> * fix(datasets): increase test coverage for PolarsDataset classes Signed-off-by: Matthias Roels <matthias.roels21@gmail.com> * docs(datasets): add renamed Polars datasets to docs Signed-off-by: Matthias Roels <matthias.roels21@gmail.com> * docs(datasets): Add new polars datasets to release notes Signed-off-by: Matthias Roels <mroels2@its.jnj.com> * fix(datasets): load_args not properly passed to LazyPolarsDataset.load Signed-off-by: Matthias Roels <matthias.roels21@gmail.com> * docs(datasets): fix spelling error in release notes Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Signed-off-by: Matthias Roels <matthias.roels21@gmail.com> --------- Signed-off-by: Matthias Roels <matthias.roels21@gmail.com> Signed-off-by: Matthias Roels <mroels2@its.jnj.com> Signed-off-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Co-authored-by: Matthias Roels <mroels2@its.jnj.com> Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Signed-off-by: samuelleeshemen <samuel_lee_sj@aiap.sg>
* bump version Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update release notes Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> --------- Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: samuelleeshemen <samuel_lee_sj@aiap.sg>
Bump version Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: Ankita Katiyar <110245118+ankatiyar@users.noreply.github.com> Signed-off-by: samuelleeshemen <samuel_lee_sj@aiap.sg>
Bump version Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: Ankita Katiyar <110245118+ankatiyar@users.noreply.github.com> Signed-off-by: samuelleeshemen <samuel_lee_sj@aiap.sg>
Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: samuelleeshemen <samuel_lee_sj@aiap.sg>
Fix missing jQuery Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Signed-off-by: samuelleeshemen <samuel_lee_sj@aiap.sg>
#413) * Fix Lazy Polars dataset to use the new-style base class Fix gh-412 Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Update release notes Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Revert "Update release notes" This reverts commit 92ceea6. --------- Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Signed-off-by: Sajid Alam <90610031+SajidAlamQB@users.noreply.github.com> Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Co-authored-by: Sajid Alam <90610031+SajidAlamQB@users.noreply.github.com> Signed-off-by: samuelleeshemen <samuel_lee_sj@aiap.sg>
Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: samuelleeshemen <samuel_lee_sj@aiap.sg>
* chore(datasets): lazily load `partitions` classes Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * test(datasets): run doctests to check examples run Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * test(datasets): keep running tests amidst failures Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * docs(datasets): format ManagedTableDataset example Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * chore(datasets): ignore breaking mods for doctests Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * style(airflow): black code in Kedro-Airflow README Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * docs(datasets): fix example syntax, and autoformat Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * docs(datasets): remove `kedro.extras.datasets` ref Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * docs(datasets): remove `>>> ` prefix for YAML code Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * docs(datasets): remove `kedro.extras.datasets` ref Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * docs(datasets): replace `data_set`s with `dataset`s Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * chore(datasets): undo changes for running doctests Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * revert(datasets): undo lazily load `partitions` classes Refs: 3fdc5a8 Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * revert(airflow): undo black code in Kedro-Airflow README Refs: dc3476e Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> --------- Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: samuelleeshemen <samuel_lee_sj@aiap.sg>
Signed-off-by: samuelleeshemen <samuel_lee_sj@aiap.sg>
* Add python version support policy to plugin readmes Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> * Temporarily pin connexion Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> --------- Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com>
Signed-off-by: samuelleeshemen <samuel_lee_sj@aiap.sg>
Signed-off-by: samuelleeshemen <samuel_lee_sj@aiap.sg>
Signed-off-by: samuelleeshemen <samuel_lee_sj@aiap.sg>
Signed-off-by: samuelleeshemen <samuel_lee_sj@aiap.sg>
* Add shared CSS and meganav Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Add end of file Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Add new heap data source Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * adjust heap parameter Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Remove nav_version next to Kedro logo in top left; add Kedro logo * Revise project name and author name Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Use full kedro icon and type for logo * Add close btn to mobile nav Signed-off-by: vladimir-mck <106236933+vladimir-mck@users.noreply.github.com> * Add css for mobile nav logo image Signed-off-by: vladimir-mck <106236933+vladimir-mck@users.noreply.github.com> * Update close button for mobile nav Signed-off-by: vladimir-mck <106236933+vladimir-mck@users.noreply.github.com> * Add open button to mobile nav Signed-off-by: vladimir-mck <106236933+vladimir-mck@users.noreply.github.com> * Delete kedro-datasets/docs/source/kedro-horizontal-color-on-light.svg Signed-off-by: vladimir-mck <106236933+vladimir-mck@users.noreply.github.com> * Update conf.py Signed-off-by: vladimir-mck <106236933+vladimir-mck@users.noreply.github.com> * Update layout.html Add links to subprojects Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Remove svg from docs -- not needed?? Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * linter error fix Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> --------- Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> Signed-off-by: vladimir-mck <106236933+vladimir-mck@users.noreply.github.com> Co-authored-by: Tynan DeBold <thdebold@gmail.com> Co-authored-by: vladimir-mck <106236933+vladimir-mck@users.noreply.github.com> Signed-off-by: samuelleeshemen <samuel_lee_sj@aiap.sg>
* Add HuggingFace datasets Co-authored-by: Danny Farah <danny_farah@mckinsey.com> Co-authored-by: Kevin Koga <Kevin_Koga@mckinsey.com> Co-authored-by: Mate Scharnitzky <Mate_Scharnitzky@mckinsey.com> Co-authored-by: Tomer Shor <Tomer_Shor@mckinsey.com> Co-authored-by: Pierre-Yves Mousset <Pierre-Yves_Mousset@mckinsey.com> Co-authored-by: Bela Chupal <Bela_chuphal@mckinsey.com> Co-authored-by: Khangjrakpam Arjun <Khangjrakpam_Arjun@mckinsey.com> Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Apply suggestions from code review Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Co-authored-by: Joel <35801847+datajoely@users.noreply.github.com> Co-authored-by: Nok Lam Chan <nok.lam.chan@quantumblack.com> * Typo Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Fix docstring Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Add docstring for HFTransformerPipelineDataset Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Use intersphinx for cross references in Hugging Face docstrings Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Add docstring for HFDataset Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Add missing test dependencies Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Add tests for huggingface datasets Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Fix HFDataset.save Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Add test for HFDataset.list_datasets Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Use new name Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Consolidate imports Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> --------- Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Co-authored-by: Danny Farah <danny_farah@mckinsey.com> Co-authored-by: Kevin Koga <Kevin_Koga@mckinsey.com> Co-authored-by: Mate Scharnitzky <Mate_Scharnitzky@mckinsey.com> Co-authored-by: Tomer Shor <Tomer_Shor@mckinsey.com> Co-authored-by: Pierre-Yves Mousset <Pierre-Yves_Mousset@mckinsey.com> Co-authored-by: Bela Chupal <Bela_chuphal@mckinsey.com> Co-authored-by: Khangjrakpam Arjun <Khangjrakpam_Arjun@mckinsey.com> Co-authored-by: Joel <35801847+datajoely@users.noreply.github.com> Co-authored-by: Nok Lam Chan <nok.lam.chan@quantumblack.com>
Signed-off-by: samuelleeshemen <samuel_lee_sj@aiap.sg>
Signed-off-by: samuelleeshemen <samuel_lee_sj@aiap.sg>
Signed-off-by: samuelleeshemen <samuel_lee_sj@aiap.sg>
Signed-off-by: samuelleeshemen <samuel_lee_sj@aiap.sg>
* test(datasets): fix `dask.ParquetDataset` doctests Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * test(datasets): use `tmp_path` fixture in doctests Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * test(datasets): simplify by not passing the schema Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * test(datasets): ignore conftest for doctests cover Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Create MANIFEST.in Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> --------- Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>
Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com>
Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: samuelleeshemen <samuel_lee_sj@aiap.sg>
* test(datasets): add outputs to matplotlib doctests Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Update Makefile Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Reformat code example, line length is short enough * Update kedro-datasets/kedro_datasets/matplotlib/matplotlib_writer.py Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> --------- Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: samuelleeshemen <samuel_lee_sj@aiap.sg>
Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: samuelleeshemen <samuel_lee_sj@aiap.sg>
* Add add-on data to heap event Signed-off-by: lrcouto <laurarccouto@gmail.com> * Move addons logic to _get_project_property Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Add condition for pyproject.toml Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Fix tests Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Fix tests Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * add tools to mock Signed-off-by: lrcouto <laurarccouto@gmail.com> * lint Signed-off-by: lrcouto <laurarccouto@gmail.com> * Update tools test Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Add after_context_created tools test Signed-off-by: lrcouto <laurarccouto@gmail.com> * Update rename to tools Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update kedro-telemetry/tests/test_plugin.py Co-authored-by: Sajid Alam <90610031+SajidAlamQB@users.noreply.github.com> Signed-off-by: Ankita Katiyar <110245118+ankatiyar@users.noreply.github.com> --------- Signed-off-by: lrcouto <laurarccouto@gmail.com> Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: Ankita Katiyar <110245118+ankatiyar@users.noreply.github.com> Co-authored-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Co-authored-by: Ankita Katiyar <110245118+ankatiyar@users.noreply.github.com> Co-authored-by: Sajid Alam <90610031+SajidAlamQB@users.noreply.github.com> Signed-off-by: samuelleeshemen <samuel_lee_sj@aiap.sg>
* update pandas-gbq dependency declaration Signed-off-by: Onur Kuru <kuru.onur1@gmail.com> * fix fmt Signed-off-by: Onur Kuru <kuru.onur1@gmail.com> --------- Signed-off-by: Onur Kuru <kuru.onur1@gmail.com> Co-authored-by: Ahdra Merali <90615669+AhdraMeraliQB@users.noreply.github.com> Signed-off-by: samuelleeshemen <samuel_lee_sj@aiap.sg>
Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: samuelleeshemen <samuel_lee_sj@aiap.sg>
Update scikit-learn version Signed-off-by: Nok Lam Chan <nok.lam.chan@quantumblack.com> Signed-off-by: samuelleeshemen <samuel_lee_sj@aiap.sg>
Signed-off-by: samuelleeshemen <samuel_lee_sj@aiap.sg>
Fix broken links in README Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Signed-off-by: samuelleeshemen <samuel_lee_sj@aiap.sg>
Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Signed-off-by: samuelleeshemen <samuel_lee_sj@aiap.sg>
Signed-off-by: samuel-lee-sj <143670400+samuel-lee-sj@users.noreply.github.com>
Hello @ankatiyar. I think I messed up the rebase. Could you give me some help? The DCO is saying that: I don't think I edited this commit. If I did I am sorry. |
Description
This is implementation of a .mat dataset from MatLab for kedro
Development notes
Added two files, matlab_dataset.py, test_matlab_dataset.py. Tested on pytest.
Checklist
RELEASE.md
file