Fix wrong unpickling with dask 2024.11 #1993
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Co-authored-by: Trevor James Smith <10819524+Zeitsperre@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Pull Request Checklist:
dask>=2024.11#1992number) and pull request (:pull:number) has been addedWhat kind of change does this PR introduce?
The issue introduced with dask 2024.11 concerned the passing of arguments through the dask graph. Most SDBA classes are children of
Parametrizable, itself a child ofdict. When stored in the graph, objects like this are pickled. and then repickled when the function is actually executed.For some reason, the repickling stopped working and objects where reinstated as dictionaries instead of their subclasses. This particularly touched the
Grouper, one of the only such objects actually been passed through dask-backed computations.Inheriting from
dictwas once not recommended but I think this is not the case anymore. Anyway, I found that inheriting fromcollections.UserDictactually solved the issue (after adapting the internals). Not sure if I should raise the issue back to dask...Does this PR introduce a breaking change?
I changed how all classes of
xclim.sdbaare made, but in theory the change is at a low, private level.Other information: