Issue reading szip compressed datasets with anaconda install vs pip install

[dwr-psandhu]

Issue:
conda install h5py and reading a h5 file with szip compression causes error above. No issues reading tables in same file with no compression.

h5py/h5py#1293

[tacaswell]

Please copy over the example and include the details about your system asked for in the issue template.

[dwr-psandhu]

The issue now confirmed on Windows 10 and MacOS Mojave using Python 3.7.x and h5py 2.9.0

Example below

Use this sample file https://support.hdfgroup.org/ftp/HDF5/examples/files/exbyapi/h5ex_d_szip.h5

Python 3.7.3 (default, Mar 27 2019, 16:54:48) 
[Clang 4.0.1 (tags/RELEASE_401/final)] :: Anaconda, Inc. on darwin


Type "help", "copyright", "credits" or "license" for more information.
>>> import h5py
>>> h5f=h5py.File('h5ex_d_szip.h5')
>>> h5f.get('/DS1').compression
'szip'
>>> h5f.get('/DS1')[:]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "//anaconda3/lib/python3.7/site-packages/h5py/_hl/dataset.py", line 573, in __getitem__
    self.id.read(mspace, fspace, arr, mtype, dxpl=self._dxpl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5d.pyx", line 181, in h5py.h5d.DatasetID.read
  File "h5py/_proxy.pyx", line 130, in h5py._proxy.dset_rw
  File "h5py/_proxy.pyx", line 84, in h5py._proxy.H5PY_H5Dread
OSError: Can't read data (can't open directory: //anaconda3/lib/hdf5/plugin)
>>>
 

[dwr-psandhu]

Issue:

Environment (conda list):

(base) nickys-mbp:data nsandhu$ conda list
# packages in environment at //anaconda3:
#
# Name                    Version                   Build  Channel
_ipyw_jlab_nb_ext_conf    0.1.0                    py37_0  
alabaster                 0.7.12                   py37_0  
anaconda                  2019.07                  py37_0  
anaconda-client           1.7.2                    py37_0  
anaconda-navigator        1.9.7                    py37_0  
anaconda-project          0.8.3                      py_0  
appnope                   0.1.0                    py37_0  
appscript                 1.1.0            py37h1de35cc_0  
asn1crypto                0.24.0                   py37_0  
astroid                   2.2.5                    py37_0  
astropy                   3.2.1            py37h1de35cc_0  
atomicwrites              1.3.0                    py37_1  
attrs                     19.1.0                   py37_1  
babel                     2.7.0                      py_0  
backcall                  0.1.0                    py37_0  
backports                 1.0                        py_2  
backports.functools_lru_cache 1.5                        py_2  
backports.os              0.1.1                    py37_0  
backports.shutil_get_terminal_size 1.0.0                    py37_2  
backports.tempfile        1.0                        py_1  
backports.weakref         1.0.post1                  py_1  
beautifulsoup4            4.7.1                    py37_1  
bitarray                  0.9.3            py37h1de35cc_0  
bkcharts                  0.2                      py37_0  
blas                      1.0                         mkl  
bleach                    3.1.0                    py37_0  
blosc                     1.16.3               hd9629dc_0  
bokeh                     1.2.0                    py37_0  
boto                      2.49.0                   py37_0  
bottleneck                1.2.1            py37h1d22016_1  
bzip2                     1.0.8                h1de35cc_0  
ca-certificates           2019.5.15                     0  
certifi                   2019.6.16                py37_0  
cffi                      1.12.3           py37hb5b8e2f_0  
chardet                   3.0.4                    py37_1  
click                     7.0                      py37_0  
cloudpickle               1.2.1                      py_0  
clyent                    1.2.2                    py37_1  
colorama                  0.4.1                    py37_0  
conda                     4.7.11                   py37_0  
conda-build               3.18.8                   py37_0  
conda-env                 2.6.0                         1  
conda-package-handling    1.3.11                   py37_0  
conda-verify              3.4.2                      py_1  
contextlib2               0.5.5                    py37_0  
cryptography              2.7              py37ha12b0ac_0  
curl                      7.65.2               ha441bb4_0  
cycler                    0.10.0                   py37_0  
cython                    0.29.12          py37h0a44026_0  
cytoolz                   0.10.0           py37h1de35cc_0  
dask                      2.1.0                      py_0  
dask-core                 2.1.0                      py_0  
dbus                      1.13.6               h90a0687_0  
decorator                 4.4.0                    py37_1  
defusedxml                0.6.0                      py_0  
distributed               2.1.0                      py_0  
docutils                  0.14                     py37_0  
entrypoints               0.3                      py37_0  
et_xmlfile                1.0.1                    py37_0  
expat                     2.2.6                h0a44026_0  
fastcache                 1.1.0            py37h1de35cc_0  
filelock                  3.0.12                     py_0  
flask                     1.1.1                      py_0  
freetype                  2.9.1                hb4e5f40_0  
future                    0.17.1                   py37_0  
get_terminal_size         1.0.0                h7520d66_0  
gettext                   0.19.8.1             h15daf44_3  
gevent                    1.4.0            py37h1de35cc_0  
glib                      2.56.2               hd9629dc_0  
glob2                     0.7                        py_0  
gmp                       6.1.2                hb37e062_1  
gmpy2                     2.0.8            py37h6ef4df4_2  
greenlet                  0.4.15           py37h1de35cc_0  
h5py                      2.9.0            py37h3134771_0  
hdf5                      1.10.4               hfa1e0ec_0  
heapdict                  1.0.0                    py37_2  
html5lib                  1.0.1                    py37_0  
icu                       58.2                 h4b95b61_1  
idna                      2.8                      py37_0  
imageio                   2.5.0                    py37_0  
imagesize                 1.1.0                    py37_0  
importlib_metadata        0.17                     py37_1  
intel-openmp              2019.4                      233  
ipykernel                 5.1.1            py37h39e3cac_0  
ipython                   7.6.1            py37h39e3cac_0  
ipython_genutils          0.2.0                    py37_0  
ipywidgets                7.5.0                      py_0  
isort                     4.3.21                   py37_0  
itsdangerous              1.1.0                    py37_0  
jbig                      2.1                  h4d881f8_0  
jdcal                     1.4.1                      py_0  
jedi                      0.13.3                   py37_0  
jinja2                    2.10.1                   py37_0  
joblib                    0.13.2                   py37_0  
jpeg                      9b                   he5867d9_2  
json5                     0.8.4                      py_0  
jsonschema                3.0.1                    py37_0  
jupyter                   1.0.0                    py37_7  
jupyter_client            5.3.1                      py_0  
jupyter_console           6.0.0                    py37_0  
jupyter_core              4.5.0                      py_0  
jupyterlab                1.0.2            py37hf63ae98_0  
jupyterlab_server         1.0.0                      py_0  
keyring                   18.0.0                   py37_0  
kiwisolver                1.1.0            py37h0a44026_0  
krb5                      1.16.1               hddcf347_7  
lazy-object-proxy         1.4.1            py37h1de35cc_0  
libarchive                3.3.3                h786848e_5  
libcurl                   7.65.2               h051b688_0  
libcxx                    4.0.1                hcfea43d_1  
libcxxabi                 4.0.1                hcfea43d_1  
libedit                   3.1.20181209         hb402a30_0  
libffi                    3.2.1                h475c297_4  
libgfortran               3.0.1                h93005f0_2  
libiconv                  1.15                 hdd342a3_7  
liblief                   0.9.0                h2a1bed3_2  
libpng                    1.6.37               ha441bb4_0  
libsodium                 1.0.16               h3efe00b_0  
libssh2                   1.8.2                ha12b0ac_0  
libtiff                   4.0.10               hcb84e12_2  
libxml2                   2.9.9                hf6e021a_1  
libxslt                   1.1.33               h33a18ac_0  
llvm-openmp               4.0.1                hcfea43d_1  
llvmlite                  0.29.0           py37h98b8051_0  
locket                    0.2.0                    py37_1  
lxml                      4.3.4            py37hef8c89e_0  
lz4-c                     1.8.1.2              h1de35cc_0  
lzo                       2.10                 h362108e_2  
markupsafe                1.1.1            py37h1de35cc_0  
matplotlib                3.1.0            py37h54f8f79_0  
mccabe                    0.6.1                    py37_1  
mistune                   0.8.4            py37h1de35cc_0  
mkl                       2019.4                      233  
mkl-service               2.0.2            py37h1de35cc_0  
mkl_fft                   1.0.12           py37h5e564d8_0  
mkl_random                1.0.2            py37h27c97d8_0  
mock                      3.0.5                    py37_0  
more-itertools            7.0.0                    py37_0  
mpc                       1.1.0                h6ef4df4_1  
mpfr                      4.0.1                h3018a27_3  
mpmath                    1.1.0                    py37_0  
msgpack-python            0.6.1            py37h04f5b5a_1  
multipledispatch          0.6.0                    py37_0  
navigator-updater         0.2.1                    py37_0  
nbconvert                 5.5.0                      py_0  
nbformat                  4.4.0                    py37_0  
ncurses                   6.1                  h0a44026_1  
networkx                  2.3                        py_0  
nltk                      3.4.4                    py37_0  
nose                      1.3.7                    py37_2  
notebook                  6.0.0                    py37_0  
numba                     0.44.1           py37h6440ff4_0  
numexpr                   2.6.9            py37h7413580_0  
numpy                     1.16.4           py37hacdab7b_0  
numpy-base                1.16.4           py37h6575580_0  
numpydoc                  0.9.1                      py_0  
olefile                   0.46                     py37_0  
openpyxl                  2.6.2                      py_0  
openssl                   1.1.1c               h1de35cc_1  
packaging                 19.0                     py37_0  
pandas                    0.24.2           py37h0a44026_0  
pandoc                    2.2.3.2                       0  
pandocfilters             1.4.2                    py37_1  
parso                     0.5.0                      py_0  
partd                     1.0.0                      py_0  
path.py                   12.0.1                     py_0  
pathlib2                  2.3.4                    py37_0  
patsy                     0.5.1                    py37_0  
pcre                      8.43                 h0a44026_0  
pep8                      1.7.1                    py37_0  
pexpect                   4.7.0                    py37_0  
pickleshare               0.7.5                    py37_0  
pillow                    6.1.0            py37hb68e598_0  
pip                       19.1.1                   py37_0  
pkginfo                   1.5.0.1                  py37_0  
pluggy                    0.12.0                     py_0  
ply                       3.11                     py37_0  
prometheus_client         0.7.1                      py_0  
prompt_toolkit            2.0.9                    py37_0  
psutil                    5.6.3            py37h1de35cc_0  
ptyprocess                0.6.0                    py37_0  
py                        1.8.0                    py37_0  
py-lief                   0.9.0            py37h1413db1_2  
pycodestyle               2.5.0                    py37_0  
pycosat                   0.6.3            py37h1de35cc_0  
pycparser                 2.19                     py37_0  
pycrypto                  2.6.1            py37h1de35cc_9  
pycurl                    7.43.0.3         py37ha12b0ac_0  
pyflakes                  2.1.1                    py37_0  
pygments                  2.4.2                      py_0  
pylint                    2.3.1                    py37_0  
pyodbc                    4.0.26           py37h0a44026_0  
pyopenssl                 19.0.0                   py37_0  
pyparsing                 2.4.0                      py_0  
pyqt                      5.9.2            py37h655552a_2  
pyrsistent                0.14.11          py37h1de35cc_0  
pysocks                   1.7.0                    py37_0  
pytables                  3.5.2            py37h5bccee9_1  
pytest                    5.0.1                    py37_0  
pytest-arraydiff          0.3              py37h39e3cac_0  
pytest-astropy            0.5.0                    py37_0  
pytest-doctestplus        0.3.0                    py37_0  
pytest-openfiles          0.3.2                    py37_0  
pytest-remotedata         0.3.1                    py37_0  
python                    3.7.3                h359304d_0  
python-dateutil           2.8.0                    py37_0  
python-libarchive-c       2.8                     py37_11  
python.app                2                        py37_9  
pytz                      2019.1                     py_0  
pywavelets                1.0.3            py37h1d22016_1  
pyyaml                    5.1.1            py37h1de35cc_0  
pyzmq                     18.0.0           py37h0a44026_0  
qt                        5.9.7                h468cd18_1  
qtawesome                 0.5.7                    py37_1  
qtconsole                 4.5.1                      py_0  
qtpy                      1.8.0                      py_0  
readline                  7.0                  h1de35cc_5  
requests                  2.22.0                   py37_0  
rope                      0.14.0                     py_0  
ruamel_yaml               0.15.46          py37h1de35cc_0  
scikit-image              0.15.0           py37h0a44026_0  
scikit-learn              0.21.2           py37h27c97d8_0  
scipy                     1.3.0            py37h1410ff5_0  
seaborn                   0.9.0                    py37_0  
send2trash                1.5.0                    py37_0  
setuptools                41.0.1                   py37_0  
simplegeneric             0.8.1                    py37_2  
singledispatch            3.4.0.3                  py37_0  
sip                       4.19.8           py37h0a44026_0  
six                       1.12.0                   py37_0  
snappy                    1.1.7                he62c110_3  
snowballstemmer           1.9.0                      py_0  
sortedcollections         1.1.2                    py37_0  
sortedcontainers          2.1.0                    py37_0  
soupsieve                 1.8                      py37_0  
sphinx                    2.1.2                      py_0  
sphinxcontrib             1.0                      py37_1  
sphinxcontrib-applehelp   1.0.1                      py_0  
sphinxcontrib-devhelp     1.0.1                      py_0  
sphinxcontrib-htmlhelp    1.0.2                      py_0  
sphinxcontrib-jsmath      1.0.1                      py_0  
sphinxcontrib-qthelp      1.0.2                      py_0  
sphinxcontrib-serializinghtml 1.1.3                      py_0  
sphinxcontrib-websupport  1.1.2                      py_0  
spyder                    3.3.6                    py37_0  
spyder-kernels            0.5.1                    py37_0  
sqlalchemy                1.3.5            py37h1de35cc_0  
sqlite                    3.29.0               ha441bb4_0  
statsmodels               0.10.0           py37h1d22016_0  
sympy                     1.4                      py37_0  
tblib                     1.4.0                      py_0  
terminado                 0.8.2                    py37_0  
testpath                  0.4.2                    py37_0  
tk                        8.6.8                ha441bb4_0  
toolz                     0.10.0                     py_0  
tornado                   6.0.3            py37h1de35cc_0  
tqdm                      4.32.1                     py_0  
traitlets                 4.3.2                    py37_0  
unicodecsv                0.14.1                   py37_0  
unixodbc                  2.3.7                h1de35cc_0  
urllib3                   1.24.2                   py37_0  
wcwidth                   0.1.7                    py37_0  
webencodings              0.5.1                    py37_1  
werkzeug                  0.15.4                     py_0  
wheel                     0.33.4                   py37_0  
widgetsnbextension        3.5.0                    py37_0  
wrapt                     1.11.2           py37h1de35cc_0  
wurlitzer                 1.0.2                    py37_0  
xlrd                      1.2.0                    py37_0  
xlsxwriter                1.1.8                      py_0  
xlwings                   0.15.8                   py37_0  
xlwt                      1.3.0                    py37_0  
xz                        5.2.4                h1de35cc_4  
yaml                      0.1.7                hc338f04_2  
zeromq                    4.3.1                h0a44026_3  
zict                      1.0.0                      py_0  
zipp                      0.5.1                      py_0  
zlib                      1.2.11               h1de35cc_3  
zstd                      1.3.7                h5bba6e5_0  

Details about conda and system ( conda info ):

(base) nickys-mbp:data nsandhu$ conda info

     active environment : base
    active env location : //anaconda3
            shell level : 2
       user config file : /Users/nsandhu/.condarc
 populated config files : 
          conda version : 4.7.11
    conda-build version : 3.18.8
         python version : 3.7.3.final.0
       virtual packages : 
       base environment : //anaconda3  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/osx-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/osx-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : //anaconda3/pkgs
                          /Users/nsandhu/.conda/pkgs
       envs directories : //anaconda3/envs
                          /Users/nsandhu/.conda/envs
               platform : osx-64
             user-agent : conda/4.7.11 requests/2.22.0 CPython/3.7.3 Darwin/18.7.0 OSX/10.14.6
                UID:GID : 501:20
             netrc file : None
           offline mode : False

[dwr-psandhu]

Install via pip on macos was failing but I was able to pip install the same version 2.9.0
It works after the wheel installation (this was reproduced on windows as well)

(dsm2) nickys-mbp:data nsandhu$ pip install h5py==2.9.0
Collecting h5py==2.9.0
  Downloading https://files.pythonhosted.org/packages/80/c5/eec74d7324628f1b640c6e706981c4ed51afcaa1656ece26cb08d862598e/h5py-2.9.0-cp37-cp37m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (6.2MB)
     |████████████████████████████████| 6.2MB 1.7MB/s 
Requirement already satisfied: numpy>=1.7 in /anaconda3/envs/dsm2/lib/python3.7/site-packages (from h5py==2.9.0) (1.16.4)
Requirement already satisfied: six in /anaconda3/envs/dsm2/lib/python3.7/site-packages (from h5py==2.9.0) (1.12.0)
Installing collected packages: h5py
Successfully installed h5py-2.9.0
(dsm2) nickys-mbp:data nsandhu$ python
Python 3.7.4 (default, Aug 13 2019, 15:17:50) 
[Clang 4.0.1 (tags/RELEASE_401/final)] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import h5py
>>> h5f=h5py.File('h5ex_d_szip.h5')
>>> h5f.get('/DS1').compression
'szip'
>>> h5f.get('/DS1')[:]
array([[   0,   -1,   -2, ...,  -61,  -62,  -63],
       [   0,    0,    0, ...,    0,    0,    0],
       [   0,    1,    2, ...,   61,   62,   63],
       ...,
       [   0,   28,   56, ..., 1708, 1736, 1764],
       [   0,   29,   58, ..., 1769, 1798, 1827],
       [   0,   30,   60, ..., 1830, 1860, 1890]], dtype=int32)

[tacaswell]

This looks like you are installing the packages from the main anaconda channel, not conda-forge, this should go to https://github.com/ContinuumIO/anaconda-issues/issues/

Do you have HDF5_PLUGIN_PATH set?

[dwr-psandhu]

Ok. So i created new environment with conda-forge install for h5py. Still getting the same error

(test_h5py) C:\Users\psandhu\Downloads>python
Python 3.7.4 (default, Aug  9 2019, 18:34:13) [MSC v.1915 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import h5py
>>> h5f=h5py.File('h5ex_d_szip.h5')
>>> h5f.get('/DS1').compression
'szip'
>>> h5f.get('/DS1')[:]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "h5py\_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py\_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "d:\Programs\Anaconda2\envs\test_h5py\lib\site-packages\h5py\_hl\dataset.py", line 573, in __getitem__
    self.id.read(mspace, fspace, arr, mtype, dxpl=self._dxpl)
  File "h5py\_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py\_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py\h5d.pyx", line 181, in h5py.h5d.DatasetID.read
  File "h5py\_proxy.pyx", line 130, in h5py._proxy.dset_rw
  File "h5py\_proxy.pyx", line 84, in h5py._proxy.H5PY_H5Dread
OSError: Can't read data (can't open directory)
>>> exit()

Conda list

(test_h5py) C:\Users\psandhu\Downloads>conda list
# packages in environment at d:\Programs\Anaconda2\envs\test_h5py:
#
# Name                    Version                   Build  Channel
ca-certificates           2019.6.16            hecc5488_0    conda-forge
certifi                   2019.6.16                py37_1    conda-forge
h5py                      2.9.0           nompi_py37h422b98e_1104    conda-forge
hdf5                      1.10.5          nompi_ha405e13_1103    conda-forge
intel-openmp              2019.4                      245
libblas                   3.8.0                    12_mkl    conda-forge
libcblas                  3.8.0                    12_mkl    conda-forge
liblapack                 3.8.0                    12_mkl    conda-forge
mkl                       2019.4                      245
numpy                     1.17.1           py37hc71023c_0    conda-forge
openssl                   1.1.1c               hfa6e2cd_0    conda-forge
pip                       19.2.2                   py37_0
pyreadline                2.1                   py37_1000    conda-forge
python                    3.7.4                h5263a28_0
setuptools                41.0.1                   py37_0
six                       1.12.0                py37_1000    conda-forge
sqlite                    3.29.0               he774522_0
vc                        14.1                 h0510ff6_4
vs2015_runtime            14.16.27012          hf0eaf9b_0
wheel                     0.33.4                   py37_0
wincertstore              0.2                      py37_0
zlib                      1.2.11            h2fa13f4_1005    conda-forge

Conda Info

(test_h5py) C:\Users\psandhu\Downloads>conda info

     active environment : test_h5py
    active env location : d:\Programs\Anaconda2\envs\test_h5py
            shell level : 2
       user config file : C:\Users\psandhu\.condarc
 populated config files : C:\Users\psandhu\.condarc
          conda version : 4.7.11
    conda-build version : 3.17.8
         python version : 2.7.16.final.0
       virtual packages : __cuda=8.0
       base environment : d:\Programs\Anaconda2  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : d:\Programs\Anaconda2\pkgs
                          C:\Users\psandhu\.conda\pkgs
                          C:\Users\psandhu\AppData\Local\conda\conda\pkgs
       envs directories : d:\Programs\Anaconda2\envs
                          C:\Users\psandhu\.conda\envs
                          C:\Users\psandhu\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/4.7.11 requests/2.21.0 CPython/2.7.16 Windows/10 Windows/10.0.16299
          administrator : False
             netrc file : None
           offline mode : False

[dwr-psandhu]

No, I don't have HDF5_PLUGIN_PATH set.

[RUCO13]

Install via pip on macos was failing but I was able to pip install the same version 2.9.0
It works after the wheel installation (this was reproduced on windows as well)

(dsm2) nickys-mbp:data nsandhu$ pip install h5py==2.9.0
Collecting h5py==2.9.0
  Downloading https://files.pythonhosted.org/packages/80/c5/eec74d7324628f1b640c6e706981c4ed51afcaa1656ece26cb08d862598e/h5py-2.9.0-cp37-cp37m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (6.2MB)
     |████████████████████████████████| 6.2MB 1.7MB/s 
Requirement already satisfied: numpy>=1.7 in /anaconda3/envs/dsm2/lib/python3.7/site-packages (from h5py==2.9.0) (1.16.4)
Requirement already satisfied: six in /anaconda3/envs/dsm2/lib/python3.7/site-packages (from h5py==2.9.0) (1.12.0)
Installing collected packages: h5py
Successfully installed h5py-2.9.0
(dsm2) nickys-mbp:data nsandhu$ python
Python 3.7.4 (default, Aug 13 2019, 15:17:50) 
[Clang 4.0.1 (tags/RELEASE_401/final)] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import h5py
>>> h5f=h5py.File('h5ex_d_szip.h5')
>>> h5f.get('/DS1').compression
'szip'
>>> h5f.get('/DS1')[:]
array([[   0,   -1,   -2, ...,  -61,  -62,  -63],
       [   0,    0,    0, ...,    0,    0,    0],
       [   0,    1,    2, ...,   61,   62,   63],
       ...,
       [   0,   28,   56, ..., 1708, 1736, 1764],
       [   0,   29,   58, ..., 1769, 1798, 1827],
       [   0,   30,   60, ..., 1830, 1860, 1890]], dtype=int32)

Hello, I had the same error with anaconda3 on Linux (Ubuntu 20.04), I tried installed h5py from conda-forge channel and the error was same. Finally, I did a h5py environment with h5py==2.9 then I could read the dataset with the szip compression. Thanks!

[ocefpaf]

BTW, I'm pretty sure we don't build hdf5 against szip due to license issues. This probably won't work and it is not a bug, just unsupported.

[mkitti]

I believe this can be closed due to adoption of libaec.

[jakirkham]

libaec support was added to hdf5 ( conda-forge/hdf5-feedstock#179 ) for UNIX. Windows is unsupported currently.

Were there any changes that needed to happen on the h5py side accordingly?

[mkitti]

Hmm. I see that MSYS2 has enabled libaec support on Windows, so is this a Visual Studio based issue?
https://packages.msys2.org/base/mingw-w64-hdf5

[mkitti]

Maybe it was even me who patched it there:
https://github.com/msys2/MINGW-packages/blob/master/mingw-w64-hdf5/hdf5-fix-find-szip.patch

[mkitti]

Upstream fix via cmake fetchcontent:
HDFGroup/hdf5#2492

Perhaps we should wait for a patch release from upstream?

[jakirkham]

For clarity there is already a libaec package on Windows. Just hdf5 Windows builds have trouble detecting it ( conda-forge/hdf5-feedstock#179 (comment) ).

If someone has a Windows machine and can try things out/suggest a fix, that would help us close out that issue. Applying an upstream patch to fix the issue would also be reasonable if we have a known fix.

[mkitti]

I do not think there actually anything to do on this feedstock. It works fine on non-Windows systems. The issue is with the hdf5-feedstock, so I created a new issue there:

conda-forge/hdf5-feedstock#197