should check to prevent szip error: "potential pixels per block greater than total number of elements in the chunk"

[edwardhartnett]

When using szip compression, it's possible to specify incorrect parameters, which result in HDF5 error when the write is attempted (i.e. not at the time the settings are set, unfortunately).

For example, this (fortran) code causes the failure, if the variable identified by varid(1) has a size of less than 32 elements:

C     Turn on szip compression for var 1.
      retval = nf_def_var_szip(ncid, varid(1), 32, 32)
      if (retval .ne. 0) stop 3

In this case, we get the following output from HDF5:

ERROR: file nc4hdf.c, line 1054.
NetCDF: HDF error
HDF5-DIAG: Error detected in HDF5 (1.10.6) thread 0:
  #000: H5D.c line 151 in H5Dcreate2(): unable to create dataset
    major: Dataset
    minor: Unable to initialize object
  #001: H5Dint.c line 337 in H5D__create_named(): unable to create and link to dataset
    major: Dataset
    minor: Unable to initialize object
  #002: H5L.c line 1592 in H5L_link_object(): unable to create new link to object
    major: Links
    minor: Unable to initialize object
  #003: H5L.c line 1833 in H5L__create_real(): can't insert link
    major: Links
    minor: Unable to insert object
  #004: H5Gtraverse.c line 851 in H5G_traverse(): internal path traversal failed
    major: Symbol table
    minor: Object not found
  #005: H5Gtraverse.c line 627 in H5G__traverse_real(): traversal operator failed
    major: Symbol table
    minor: Callback failed
  #006: H5L.c line 1639 in H5L__link_cb(): unable to create object
    major: Links
    minor: Unable to initialize object
  #007: H5Oint.c line 2460 in H5O_obj_create(): unable to open object
    major: Object header
    minor: Can't open object
  #008: H5Doh.c line 300 in H5O__dset_create(): unable to create dataset
    major: Dataset
    minor: Unable to initialize object
  #009: H5Dint.c line 1233 in H5D__create(): unable to set local filter parameters
    major: Dataset
    minor: Unable to initialize object
  #010: H5Z.c line 925 in H5Z_set_local(): local filter parameters not set
    major: Data filters
    minor: Error from filter 'set local' callback
  #011: H5Z.c line 854 in H5Z_prepare_prelude_callback_dcpl(): unable to apply filter
    major: Data filters
    minor: Error from filter 'can apply' callback
  #012: H5Z.c line 769 in H5Z_prelude_callback(): error during user callback
    major: Data filters
    minor: Error from filter 'set local' callback
  #013: H5Zszip.c line 213 in H5Z_set_local_szip(): pixels per block greater than total number of elements in the chunk
    major: Invalid arguments to routine
    minor: Bad value

This would also occur if the chunksize was set to < 32 for that dim in that var. (That;'s what is happening behind the scenes, the dim length is only 22, so that ends up being the chunksize on that dim, and that's < 32).

One challenge here is that we often don't know the chunksizes when szip compression is turned on.

However, some things that can easily be done:

• Check against the size of the var dims, which is known, and would have caught this case.
• Document this clearly in C and Fortran docs for szip compression.
• We can detect this error at the time that metadata are written (i.e. enddef) rather than waiting for an attempted write to the variable and then failing.

I am working on szip in netcdf-fortran now; once I get that sorted I will swing back around and fix this issue.