Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug: zarr arrays cannot be pickled #64

Closed
ieivanov opened this issue Feb 6, 2021 · 2 comments
Closed

Bug: zarr arrays cannot be pickled #64

ieivanov opened this issue Feb 6, 2021 · 2 comments
Labels
enhancement New feature or request

Comments

@ieivanov
Copy link

ieivanov commented Feb 6, 2021

zarr arrays created with tifffile cannot be pickled. This prevents using zarr arrays in multiprocessing applications.

Running:

zarr_store = tifffile.imread('path_to_data', aszarr=True)
zarrArray = zarr.open(zarr_store, mode='r')
pickle.dumps(zarrArray)

throws the error:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-4-343d376ee953> in <module>
----> 1 pickle.dumps(zarrArray)

AttributeError: Can't pickle local object 'TIFF.CHUNKMODE.<locals>.CHUNKMODE'

I believe this is caused by this part of the code:

tifffile/tifffile/tifffile.py

Lines 12454 to 12461 in d2fe871

def CHUNKMODE():
class CHUNKMODE(enum.IntEnum):
NONE = 0
PLANE = 1
PAGE = 2
FILE = 3
return CHUNKMODE
and can be solved using functools.wraps as described here: http://gael-varoquaux.info/programming/decoration-in-python-done-right-decorating-and-pickling.html

@cgohlke cgohlke added the enhancement New feature or request label Feb 6, 2021
@cgohlke
Copy link
Owner

cgohlke commented Feb 6, 2021

This is a known limitation (see discussion at AllenCellModeling/aicsimageio#178 (comment)) but I don't think it's a bug since zarr does not strictly require stores to be picklable (https://github.com/zarr-developers/zarr-python/blob/master/docs/tutorial.rst#pickle-support). I think aicsimageio works around this limitation by opening a new ZarrTiffStore in each process.
Fundamentally, tifffile.ZarrTiffStore is not picklable because it uses a TiffFile instance, which uses system handles (file handles and threads). To make tifffile.ZarrTiffStore picklable would need a different approach outlined in AllenCellModeling/aicsimageio#178 (comment). However I expect that to be significantly slower than the current implementation, both in initialization and chunk access.

@cgohlke cgohlke closed this as completed Feb 6, 2021
@joshmoore
Copy link

@ieivanov, the zarr.convenience.copy_store helper method may do what you want. It won't produce pickled data, but it should convert the data from TIFF to Zarr on disk for later re-use.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants