Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make torch.Tensor and spacy models cacheable #5191

Merged
merged 5 commits into from
Nov 2, 2022
Merged

Conversation

mariosasko
Copy link
Collaborator

Override Pickler.save to implement deterministic reduction (lazily registered; inspired by https://github.com/uqfoundation/dill/blob/master/dill/_dill.py#L343) functions for torch.Tensor and spaCy models.

Fix #5170, fix #3178

@HuggingFaceDocBuilderDev
Copy link

HuggingFaceDocBuilderDev commented Nov 2, 2022

The documentation is not available anymore as the PR was closed or merged.

@mariosasko mariosasko marked this pull request as ready for review November 2, 2022 14:36
Copy link
Member

@lhoestq lhoestq left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Love it thanks ! Added some minor comments, but overall LGTM :)

src/datasets/utils/py_utils.py Outdated Show resolved Hide resolved
src/datasets/utils/py_utils.py Outdated Show resolved Hide resolved
@mariosasko mariosasko merged commit 0d9c12a into main Nov 2, 2022
@mariosasko mariosasko deleted the improve-caching branch November 2, 2022 17:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants