Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

docs: cloudpickle is an interesting extension of the pickle functionality #361

Merged
merged 8 commits into from
Oct 10, 2023
Merged

docs: cloudpickle is an interesting extension of the pickle functionality #361

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
9ceabad
cloudpickle is an interesting extension of the pickle functionality.
hfwittmann Sep 30, 2023
5113108
fix: changed version
hfwittmann Sep 30, 2023
f34ec08
fix: removed Anchors in links in docs (otherwise sphinx crashes)
hfwittmann Sep 30, 2023
05e5342
fix: removed some more Anchors in links in docs (otherwise sphinx cra…
hfwittmann Sep 30, 2023
62fb981
removed trailing whitespace
hfwittmann Sep 30, 2023
2b72928
removed whitespace
hfwittmann Oct 9, 2023
79fe1d2
Merge branch 'main' into cloudpickle
stichbury Oct 10, 2023
41dc267
Update release notes
merelcht Oct 10, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions kedro-datasets/kedro_datasets/pickle/pickle_dataset.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,8 @@ def __init__( # noqa: PLR0913
dill.load: https://dill.readthedocs.io/en/latest/index.html#dill.load
compress_pickle.load:
https://lucianopaz.github.io/compress_pickle/html/api/compress_pickle.html#compress_pickle.compress_pickle.load
cloudpickle.load:
https://github.com/cloudpipe/cloudpickle/blob/0f330b6afe55313fc1efc090a7d350f5ad5c9317/tests/cloudpickle_test.py
All defaults are preserved.
save_args: Pickle options for saving pickle files.
You can pass in arguments that the backend dump function specified accepts, e.g:
Expand All @@ -118,6 +120,8 @@ def __init__( # noqa: PLR0913
dill.dump: https://dill.readthedocs.io/en/latest/index.html#dill.dump
compress_pickle.dump:
https://lucianopaz.github.io/compress_pickle/html/api/compress_pickle.html#compress_pickle.compress_pickle.dump
cloudpickle.dump:
https://github.com/cloudpipe/cloudpickle/blob/0f330b6afe55313fc1efc090a7d350f5ad5c9317/tests/cloudpickle_test.py
All defaults are preserved.
version: If specified, should be an instance of
``kedro.io.core.Version``. If its ``load`` attribute is
Expand Down
5 changes: 5 additions & 0 deletions kedro-datasets/kedro_datasets/redis/redis_dataset.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,7 @@ def __init__( # noqa: PLR0913
* `pickle`
* `dill`
* `compress_pickle`
* `cloudpickle`

Example backends that are incompatible:
* `torch`
Expand All @@ -94,13 +95,17 @@ def __init__( # noqa: PLR0913
dill.loads: https://dill.readthedocs.io/en/latest/index.html#dill.loads
compress_pickle.loads:
https://lucianopaz.github.io/compress_pickle/html/api/compress_pickle.html#compress_pickle.compress_pickle.loads
cloudpickle.loads:
https://github.com/cloudpipe/cloudpickle/blob/0f330b6afe55313fc1efc090a7d350f5ad5c9317/tests/cloudpickle_test.py
All defaults are preserved.
save_args: Pickle options for saving pickle files.
You can pass in arguments that the backend dump function specified accepts, e.g:
pickle.dumps: https://docs.python.org/3/library/pickle.html#pickle.dump
dill.dumps: https://dill.readthedocs.io/en/latest/index.html#dill.dumps
compress_pickle.dumps:
https://lucianopaz.github.io/compress_pickle/html/api/compress_pickle.html#compress_pickle.compress_pickle.dumps
cloudpickle.dumps:
https://github.com/cloudpipe/cloudpickle/blob/0f330b6afe55313fc1efc090a7d350f5ad5c9317/tests/cloudpickle_test.py
All defaults are preserved.
credentials: Credentials required to get access to the redis server.
E.g. `{"password": None}`.
Expand Down
1 change: 1 addition & 0 deletions kedro-datasets/setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -153,6 +153,7 @@ def _collect_requirements(requires):
"biopython~=1.73",
"blacken-docs==1.9.2",
"black~=22.0",
"cloudpickle<=2.0.0",
"compress-pickle[lz4]~=2.1.0",
"coverage[toml]",
"dask[complete]~=2021.10", # pinned by Snyk to avoid a vulnerability
Expand Down
1 change: 1 addition & 0 deletions kedro-datasets/tests/pickle/test_pickle_dataset.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,7 @@ class TestPickleDataset:
("pickle", None, None),
("joblib", None, None),
("dill", None, None),
("cloudpickle", None, None),
("compress_pickle", {"compression": "lz4"}, {"compression": "lz4"}),
],
indirect=True,
Expand Down
1 change: 1 addition & 0 deletions kedro-datasets/tests/redis/test_redis_dataset.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,7 @@ class TestPickleDataset:
[
("a", "pickle", None, None),
(1, "dill", None, None),
(2, "cloudpickle", None, None),
("key", "compress_pickle", {"compression": "lz4"}, {"compression": "lz4"}),
],
indirect=True,
Expand Down
Loading