-
Notifications
You must be signed in to change notification settings - Fork 51
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Pickle w/o Statics #1458
Comments
@ax3l I tried using dask delayed with openpmd-api for parallelization over iterations. I couldn't get it to work, I think, because dask wasn't able to pickle the series object. Would it make sense to add |
@ax3l @franzpoeschel This also doesn't seem to work when working with one series at a time but more than one in a kernel instance. I don't understand why, but I'm trying to use Are you aware of any workaround? |
That's precisely what Axel means above by
Unpickling e.g. a single // Create a new openPMD Series and keep it alive.
// This is a big hack for now, but it works for our use
// case, which is spinning up remote serial read series
// for DASK.
static auto series = openPMD::Series(filename, Access::READ_ONLY); ... which leads exactly to the behavior that you see. I do have an idea though how we could fix this short-term, lemme see |
For the first implementation of multi-process (multi-node) Dask, we pickle objects like the Record and RecordComponents + their series.
The series is unpickled into a static function member, to avoid:
This works as a hack until you need to work with two series at a time.
https://github.com/openPMD/openPMD-api/blob/0.15.1/include/openPMD/binding/python/Pickle.hpp#L73-L77
The text was updated successfully, but these errors were encountered: