Got "dimensionality of chunks doesn't match the dataspace" when compressing 4D TwoPhotonSeries data

[yzhaoinuw]

Hello, I wanted to compress my data stored as TwoPhotonSeries following the example for DataPipe under the "chunking" Section. Specifically, I want to experiment with setting the chunk size when compressing because it seems to result in faster speed and smaller size according to the example.

I have TwoPhotonSeries data (call it myData) in the shape of (1, width, height, depth). For specificity, let width = height = 512, and depth = 130. I tried using the following code to compress it

fData_compressed = types.untyped.DataPipe( ...
    'data', myData, ...
    'chunkSize', [1, 512], ...
    'axis', 2);

Inspired by the example, I chose the chunk size such that it spans each row of my 4D matrix. I set axis to 2 because the first dimension of my matrix is just 1. However, I got "dimensionality of chunks doesn't match the dataspace" when exporting the nwb. Any ideas on how I should compress if I were to set the chunk size?

The complete traceback is

Error using hdf5lib2
The HDF5 library encountered an error and produced the following stack trace information:

    H5D__chunk_construct    dimensionality of chunks doesn't match the dataspace

Error in H5D.create (line 63)
id = H5ML.hdf5lib2('H5Dcreate', varargin{:} );

Error in types.untyped.datapipe.BlueprintPipe/write (line 219)
            did = H5D.create(fid, fullpath, tid, sid, lcpl, dcpl, dapl);

Error in types.untyped.DataPipe/export (line 278)
            obj.internal = obj.internal.write(fid, fullpath);

Error in indexing (line 297)
                B = builtin('subsref', obj, S);

Error in types.core.TimeSeries/export (line 364)
            refs = obj.data.export(fid, [fullpath '/data'], refs);

Error in types.core.ImageSeries/export (line 154)
        refs = export@types.core.TimeSeries(obj, fid, fullpath, refs);

Error in types.core.TwoPhotonSeries/export (line 110)
        refs = export@types.core.ImageSeries(obj, fid, fullpath, refs);

Error in types.untyped.Set/export (line 181)
                    refs = v.export(fid, propfp, refs);

Error in types.core.NWBFile/export (line 843)
        refs = obj.acquisition.export(fid, [fullpath '/acquisition'], refs);

Error in NwbFile/export (line 62)
                refs = export@types.core.NWBFile(obj, output_file_id, '/', {});

Error in nwbExport (line 36)
    nwb(i).export(filename);

Error in sketch (line 89)
nwbExport(nwb, nwbFilePath);

[bendichter]

Hi @yzhaoinuw ! Great to see you are exploring the advanced writing tools!

NWB does not allow you to save a TwoPhotonSeries with data shape (1, width, height, depth). It should be height x width x time or depth x height x width x time.

[yzhaoinuw]

Hi @bendichter, thank you for the reply. In my case, I got z-stack data from a two-photon microscope that have up to three channels. They are not time series data, but images at different depths. I asked in a separate discussion post #92 on whether the TwoPhotonSeries or RGBImage is the more appropriate object for my data. Following the suggestion given by @CodyCBakerPhD, I proceeded with creating a separate TwoPhotonSeries for the data in each channel made them into the shape of (1, width, height, depth). So the first dimension, 1, is a dummy dimension to work around the data shape requirement of TwoPhotonSeries, which says that the first dimension must be time or frame. That's why I had my data in this shape.

Given my situation, is there any way I can further compress the data by setting the chunk size, or is there any alternative that you would recommend?

[yzhaoinuw]

Setting the chunkSize to [1, width, height, 1] works. Using a concrete example, suppose the dimension of my TwoPhotonSeries data is [1, 512, 512, 131]. Then, the chunkSize can be [1, 512, 512, 1]. Credits to ChatGPT, who further explains that

This means that the chunks span the entire second and third dimensions of your data, and a single element along the first and fourth dimensions. You can adjust the chunkSize values depending on your specific needs and the desired compression performance.

It worked and passed the NWBInspector. The resulting compressed size is smaller than what it would be without setting the chunkSize