Potential test failures with libnetcdf 4.5.0

[spencerahill]

A heads up: @spencerkclark unearthed problems with libnetcdf 4.5.0 that will cause xarray test failures. See Unidata/netcdf4-python#742 and link therein for more info.

[shoyer]

@spencerahill how did you install libnetcdf 4.5.0? I'm still getting 4.4.1 with conda, on both the default and conda-forge channel.

[spencerkclark]

@shoyer I initially ran into that too. I think depending on the libraries you have installed (or seek to install) conda may not let you upgrade to libnetcdf 4.5.0. To get libnetcdf 4.5.0 I created a clean environment with the following specification file:

name: test_xarray_libnetcdf45
channels:
  - conda-forge
dependencies:
  - python=3.6
  - netcdf4
  - libnetcdf=4.5.0
  - numpy
  - mock
  - pandas
  - pytest
  - pip
  - dask
  - scipy

Then from within that environment I did an editable install of the xarray master branch and ran the test suite.

[ocefpaf]

@spencerkclark conda-forge has netcdf4 built with both 4.5.0 and 4.4.1.1, so conda can get one or the other depending on the scenario. The only way to ensure you are building an env with libnetcdf 4.5.0 is to add it explicitly in that list.

[spencerkclark]

@ocefpaf thanks, empirically the original did the trick, but I have edited it for correctness.

[shoyer]

I ran our test suite with libnetcdf 4.5.0 and it turned up two failures, for GenericNetCDFDataTest.test_cross_engine_read_write_netcdf3 and GenericNetCDFDataTestAutocloseTrue.test_cross_engine_read_write_netcdf3:
https://travis-ci.org/shoyer/xarray/jobs/303718182

I'll have to do a little digging to turn up exactly which cross read/write cases failed. (It would probably be best to edit the test cases to use parameterized tests of some sort rather than the explicit "for" loop)

[spencerkclark]

Thanks @shoyer, I think it's the files written by the 'scipy' engine and read by the 'netcdf4' engine. Here is a minimal example:

In [1]: import xarray as xr

In [2]: da = xr.DataArray([1, 2, 3], dims=['x'])

In [3]: da.to_netcdf('test_netcdf4.nc', format='NETCDF3_CLASSIC', engine='netcdf4')

In [4]: ds = xr.open_dataset('test_netcdf4.nc', engine='netcdf4')

In [5]: ds = xr.open_dataset('test_netcdf4.nc', engine='scipy')

In [6]: da.to_netcdf('test_scipy.nc', format='NETCDF3_CLASSIC', engine='scipy')

In [7]: ds = xr.open_dataset('test_scipy.nc', engine='scipy')

In [8]: ds = xr.open_dataset('test_scipy.nc', engine='netcdf4')
---------------------------------------------------------------------------
IOError                                   Traceback (most recent call last)
<ipython-input-8-13ddc3810e0b> in <module>()
----> 1 ds = xr.open_dataset('test_scipy.nc', engine='netcdf4')

//anaconda/envs/research/lib/python2.7/site-packages/xarray/backends/api.pyc in open_dataset(filename_or_obj, group, decode_cf, mask_and_scale, decode_times, autoclose, concat_characters, decode_coords, engine, chunks, lock, cache, drop_variables)
    280         if engine == 'netcdf4':
    281             store = backends.NetCDF4DataStore(filename_or_obj, group=group,
--> 282                                               autoclose=autoclose)
    283         elif engine == 'scipy':
    284             store = backends.ScipyDataStore(filename_or_obj,

//anaconda/envs/research/lib/python2.7/site-packages/xarray/backends/netCDF4_.pyc in __init__(self, filename, mode, format, group, writer, clobber, diskless, persist, autoclose)
    208                                    diskless=diskless, persist=persist,
    209                                    format=format)
--> 210         self.ds = opener()
    211         self._autoclose = autoclose
    212         self._isopen = True

//anaconda/envs/research/lib/python2.7/site-packages/xarray/backends/netCDF4_.pyc in _open_netcdf4_group(filename, mode, group, **kwargs)
    183     import netCDF4 as nc4
    184
--> 185     ds = nc4.Dataset(filename, mode=mode, **kwargs)
    186
    187     with close_on_error(ds):

netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Dataset.__init__()

netCDF4/_netCDF4.pyx in netCDF4._netCDF4._ensure_nc_success()

IOError: [Errno -36] NetCDF: Invalid argument: '/Users/skc/test_scipy.nc'

Here's a link to the issue upstream: Unidata/netcdf-c#657.

[stale]

In order to maintain a list of currently relevant issues, we mark issues as stale after a period of inactivity

If this issue remains relevant, please comment here or remove the stale label; otherwise it will be marked as closed automatically