Reading HDF5 - control creation of anonymous dimensions

[kmuehlbauer]

I've read the Interoperability with HDF5 section in the docs.

Assuming a HDF5 file is written in accordance with the netCDF-4 rules (i.e. no strange types, no looping groups), and assuming that every dataset has a dimension scale attached to each dimension, the netCDF-4 API can be used to read and edit the file, quite easily.
[...]
If dimension scales are not used, then netCDF-4 can still edit the file, and will invent anonymous dimensions for each variable shape.

This works quite well when reading hdf5 weather radar data, but has one quirk.

If the variable has different size for first and second dimension (eg. (100,200)) two anonymous dimensions are invented. If the 2D dataset has same size for first and second dimension (eg. (100,100)) then only one anonymous dimension is invented.

In my use case this contradicts the original meaning of the 2D dataset dimensions to be different. Maybe I do not see it immediately, but I also can't think of any use case where the dimensions of a 2D dataset would be the same dimension (sound pretty bad, but I do not know how to explain better).

On my way to find some working solution I've already asked on the netCDF4-python github issue tracker and on the h5py mailing-list but it seems a solution is only possible by enhancing/changing behaviour of netcdf-c.

From my perspective netcdf-c would need to invent two anonymous dimensions from a 2D dataset, even if the dimensions are of the same size. Another option would be to define the wanted behaviour at reading time. Are there any other options?

Software Version Information

output of `nc-config --all (nectCDF 4.6.2):

This netCDF 4.6.2 has been built with the following features:

--cc -> x86_64-conda_cos6-linux-gnu-cc
--cflags -> -I/home/kai/miniconda/envs/wradlib_150/include
--libs -> -L/home/kai/miniconda/envs/wradlib_150/lib -lnetcdf -lmfhdf -ldf -lhdf5_hl -lhdf5 -lrt -lpthread -lz -ldl -lm -lcurl

--has-c++ -> no
--cxx ->

--has-c++4 -> no
--cxx4 ->

--has-fortran -> no
--has-dap -> yes
--has-dap2 -> yes
--has-dap4 -> yes
--has-nc2 -> yes
--has-nc4 -> yes
--has-hdf5 -> yes
--has-hdf4 -> yes
--has-logging -> no
--has-pnetcdf -> no
--has-szlib -> no
--has-cdf5 -> yes
--has-parallel4 -> no
--has-parallel -> no

--prefix -> /home/kai/miniconda/envs/wradlib_150
--includedir -> /home/kai/miniconda/envs/wradlib_150/include
--libdir -> /home/kai/miniconda/envs/wradlib_150/lib
--version -> netCDF 4.6.2

output of `h5cc --showconfig (hdf 1.10.5):

SUMMARY OF THE HDF5 CONFIGURATION =================================

General Information:

               HDF5 Version: 1.10.5                                                                                                                                 
              Configured on: Wed Aug 21 19:09:16 UTC 2019                                                                                                           
              Configured by: conda@752c21c7e9bc                                                                                                                     
                Host system: x86_64-conda_cos6-linux-gnu                                                                                                            
          Uname information: Linux 752c21c7e9bc 4.15.0-1052-azure #57-Ubuntu SMP Tue Jul 23 19:07:16 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux                        
                   Byte sex: little-endian                                                                                                                          
         Installation point: /home/kai/miniconda/envs/wradlib_150                                                                                                   

Compiling Options:

                 Build Mode: production
          Debugging Symbols: no
                    Asserts: no
                  Profiling: no
         Optimization Level: high

Linking Options:

                  Libraries: static, shared

Statically Linked Executables:
LDFLAGS: -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/home/kai/miniconda/envs/wradlib_150/lib -Wl,-rpath-link,/home/kai/miniconda/envs/wradlib_150/lib -L/home/kai/miniconda/envs/wradlib_150/lib
H5_LDFLAGS:
AM_LDFLAGS: -L/home/kai/miniconda/envs/wradlib_150/lib
Extra libraries: -lrt -lpthread -lz -ldl -lm
Archiver: /home/conda/feedstock_root/build_artifacts/hdf5_split_1566414109997/_build_env/bin/x86_64-conda_cos6-linux-gnu-ar
AR_FLAGS: cr
Ranlib: /home/conda/feedstock_root/build_artifacts/hdf5_split_1566414109997/_build_env/bin/x86_64-conda_cos6-linux-gnu-ranlib

Languages:

                          C: yes
                 C Compiler: /home/conda/feedstock_root/build_artifacts/hdf5_split_1566414109997/_build_env/bin/x86_64-conda_cos6-linux-gnu-cc
                   CPPFLAGS: -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -I/home/kai/miniconda/envs/wradlib_150/include
                H5_CPPFLAGS: -D_GNU_SOURCE -D_POSIX_C_SOURCE=200809L   -DNDEBUG -UH5_DEBUG_API
                AM_CPPFLAGS:  -I/home/kai/miniconda/envs/wradlib_150/include
                    C Flags: -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/home/kai/miniconda/envs/wradlib_150/include -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/hdf5_split_1566414109997/work=/usr/local/src/conda/hdf5_split-1.10.5 -fdebug-prefix-map=/home/kai/miniconda/envs/wradlib_150=/usr/local/src/conda-prefix
                 H5 C Flags:  -std=c99  -pedantic -Wall -Wextra -Wbad-function-cast -Wc++-compat -Wcast-align -Wcast-qual -Wconversion -Wdeclaration-after-statement -Wdisabled-optimization -Wfloat-equal -Wformat=2 -Winit-self -Winvalid-pch -Wmissing-declarations -Wmissing-include-dirs -Wmissing-prototypes -Wnested-externs -Wold-style-definition -Wpacked -Wpointer-arith -Wredundant-decls -Wshadow -Wstrict-prototypes -Wswitch-default -Wswitch-enum -Wundef -Wunused-macros -Wunsafe-loop-optimizations -Wwrite-strings -finline-functions -s -Wno-inline -Wno-aggregate-return -Wno-missing-format-attribute -Wno-missing-noreturn -O
                 AM C Flags: 
           Shared C Library: yes
           Static C Library: yes


                    Fortran: yes
           Fortran Compiler: /home/conda/feedstock_root/build_artifacts/hdf5_split_1566414109997/_build_env/bin/x86_64-conda_cos6-linux-gnu-gfortran
              Fortran Flags: 
           H5 Fortran Flags:  -pedantic -Wall -Wextra -Wunderflow -Wimplicit-interface -Wsurprising -Wno-c-binding-type  -s -O2
           AM Fortran Flags: 
     Shared Fortran Library: yes
     Static Fortran Library: yes

                        C++: yes
               C++ Compiler: /home/conda/feedstock_root/build_artifacts/hdf5_split_1566414109997/_build_env/bin/x86_64-conda_cos6-linux-gnu-c++
                  C++ Flags: -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/home/kai/miniconda/envs/wradlib_150/include -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/hdf5_split_1566414109997/work=/usr/local/src/conda/hdf5_split-1.10.5 -fdebug-prefix-map=/home/kai/miniconda/envs/wradlib_150=/usr/local/src/conda-prefix
               H5 C++ Flags:   -pedantic -Wall -W -Wundef -Wshadow -Wpointer-arith -Wcast-qual -Wcast-align -Wwrite-strings -Wconversion -Wredundant-decls -Winline -Wsign-promo -Woverloaded-virtual -Wold-style-cast -Weffc++ -Wreorder -Wnon-virtual-dtor -Wctor-dtor-privacy -Wabi -finline-functions -s -O
               AM C++ Flags: 
         Shared C++ Library: yes
         Static C++ Library: yes

                       Java: no

Features:

               Parallel HDF5: no

Parallel Filtered Dataset Writes: no
Large Parallel I/O: no
High-level library: yes
Threadsafety: yes
Default API mapping: v110
With deprecated public symbols: yes
I/O filters (external): deflate(zlib)
MPE: no
Direct VFD: no
dmalloc: no
Packages w/ extra debug output: none
API tracing: no
Using memory checker: yes
Memory allocation sanity checks: no
Function stack tracing: no
Strict file format checks: no
Optimization instrumentation: no

Python Version

Python 3.7.3 | packaged by conda-forge | (default, Jul  1 2019, 21:52:21) 
[GCC 7.3.0] :: Anaconda, Inc. on linux

netCDF4 version

'1.5.1.2'

h5py version

'2.9.0'

[edhartnett]

What would make this easier would be a test program that builds your HDF5 file (with HDF5 code) and then tries to open the file with netcdf.

[kmuehlbauer]

@edhartnett Thanks for looking into this.

Attached below is sample code taken from HDF Group. It creates a sample file with a single 2d dataset. Using ncdump -h on this file:

netcdf dset {
dimensions:
        phony_dim_0 = 100 ;
variables:
        int dset(phony_dim_0, phony_dim_0) ;
}

Changing the second dimension in the source code to 200:

netcdf dset {
dimensions:
        phony_dim_0 = 100 ;
        phony_dim_1 = 200 ;
variables:
        int dset(phony_dim_0, phony_dim_1) ;
}

/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
 * Copyright by The HDF Group.                                               *
 * Copyright by the Board of Trustees of the University of Illinois.         *
 * All rights reserved.                                                      *
 *                                                                           *
 * This file is part of HDF5.  The full HDF5 copyright notice, including     *
 * terms governing use, modification, and redistribution, is contained in    *
 * the COPYING file, which can be found at the root of the source code       *
 * distribution tree, or in https://support.hdfgroup.org/ftp/HDF5/releases.  *
 * If you do not have access to either file, you may request a copy from     *
 * help@hdfgroup.org.                                                        *
 * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */

/*
 *  This example illustrates how to create a dataset that is a 4 x 6 
 *  array.  It is used in the HDF5 Tutorial.
 */

#include "hdf5.h"
#define FILE "dset.h5"

int main() {

   hid_t       file_id, dataset_id, dataspace_id;  /* identifiers */
   hsize_t     dims[2];
   herr_t      status;

   /* Create a new file using default properties. */
   file_id = H5Fcreate(FILE, H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);

   /* Create the data space for the dataset. */
   dims[0] = 100; 
   dims[1] = 100; 
   dataspace_id = H5Screate_simple(2, dims, NULL);

   /* Create the dataset. */
   dataset_id = H5Dcreate2(file_id, "/dset", H5T_STD_I32BE, dataspace_id, 
                          H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);

   /* End access to the dataset and release resources used by it. */
   status = H5Dclose(dataset_id);

   /* Terminate access to the data space. */ 
   status = H5Sclose(dataspace_id);

   /* Close the file. */
   status = H5Fclose(file_id);
}

Update: intermix of output updated

[edharnett]

Hi You have the wrong person or email. Thanks Ed Harnett

…

On Mon, 2 Sep 2019 at 13:28, Kai Mühlbauer ***@***.***> wrote: @edharnett <https://github.com/edharnett> Thanks for looking into this. Attached below is sample code taken from HDF Group <https://support.hdfgroup.org/HDF5/examples/intro.html>. It creates a sample file with a single 2d dataset. Using ncdump -h on this file: netcdf dset { dimensions: phony_dim_0 = 100 ; phony_dim_1 = 200 ; variables: int dset(phony_dim_0, phony_dim_1) ; } Changing the second dimension in the source code to 200: netcdf dset { dimensions: phony_dim_0 = 100 ; variables: int dset(phony_dim_0, phony_dim_0) ; } /* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * Copyright by The HDF Group. * * Copyright by the Board of Trustees of the University of Illinois. * * All rights reserved. * * * * This file is part of HDF5. The full HDF5 copyright notice, including * * terms governing use, modification, and redistribution, is contained in * * the COPYING file, which can be found at the root of the source code * * distribution tree, or in https://support.hdfgroup.org/ftp/HDF5/releases. * * If you do not have access to either file, you may request a copy from * * ***@***.*** * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */ /* * This example illustrates how to create a dataset that is a 4 x 6 * array. It is used in the HDF5 Tutorial. */ #include "hdf5.h" #define FILE "dset.h5" int main() { hid_t file_id, dataset_id, dataspace_id; /* identifiers */ hsize_t dims[2]; herr_t status; /* Create a new file using default properties. */ file_id = H5Fcreate(FILE, H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT); /* Create the data space for the dataset. */ dims[0] = 100; dims[1] = 100; dataspace_id = H5Screate_simple(2, dims, NULL); /* Create the dataset. */ dataset_id = H5Dcreate2(file_id, "/dset", H5T_STD_I32BE, dataspace_id, H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT); /* End access to the dataset and release resources used by it. */ status = H5Dclose(dataset_id); /* Terminate access to the data space. */ status = H5Sclose(dataspace_id); /* Close the file. */ status = H5Fclose(file_id); } — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1484?email_source=notifications&email_token=ABKKQ7ULZBUVPNQLR4XOJEDQHUBGRA5CNFSM4IS2TBKKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5VVZDA#issuecomment-527129740>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABKKQ7WI3PY46C4AXWLDHNTQHUBGRANCNFSM4IS2TBKA> .

[kmuehlbauer]

Ping @edhartnett, because of using wrong github username

[kmuehlbauer]

@edhartnett Is there any news on this issue? Can I be of any further help?

[edharnett]

Wrong email again!

…

On Thu, 5 Sep 2019 at 17:15, Kai Mühlbauer ***@***.***> wrote: @edhartnett <https://github.com/edhartnett> Is there any news on this issue? Can I be of any further help? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1484?email_source=notifications&email_token=ABKKQ7US4LDH2IQEH2ZJ2MLQIEWD5A5CNFSM4IS2TBKKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD57W2VQ#issuecomment-528444758>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABKKQ7T4VCDGODPIG4Q3YD3QIEWD5ANCNFSM4IS2TBKA> .

[kmuehlbauer]

@edharnett I double checked to not use your github name this time but @edhartnett instead. Not sure why you got notified again. If this persists you might need to mute the thread via the link in your email. Sorry for any inconvenience.

[DennisHeimbigner]

My 2cents. The problem is that without a name for the dimension,
it is impossible for netcdf to assign semantics to anonymous dimensions.
Given a variable (100,100), it just has no way to decide if those are
semantically the same or just by accident the same.
IMO, there are two defensible positions:

1. all anonymous dimensions of the same size will refer to the same
   dimension. This is the current situation.
2. every anonymous dimension is different and there should be
   a dimension created for every individual anonymous dimension
   even if two of them have the same size.
   I think you are saying that case 2 is what you would prefer, correct?

The only problem with case 2 is coming up with a naming scheme for
each anonymous dimension. E.g. something like dim_100_1 vs dim_100_2.
I am not sure what the consequences would be -- vis-a-vis back
compatibility -- of moving to case 2.
=Dennis Heimbigner
Unidata

[edhartnett]

There is no further news and unfortunately it's unlikely that I will get time to look at this before I come back from a long vacation in November. ;-)

@DennisHeimbigner I believe 2 is not a good idea, because mostly HDF5 files are actually trying to use netcdf-like dimensions. In other words, if there are 10 variables and they all share a dimension of length 100, then probably the file writer intended that all 10 variables are sharing a netCDF-like dimension.

In any case, we cannot change this behavior. This code has been out there and stable for a long time (10+ years), and I believe it is widely used. So we can't break anyone's existing code.

Perhaps we should handle the case differently in which one variable has two dimensions of the same size?

But then what should we do? What should we do to other variables that also have that size? (So if we have a var[100][100], we decide that there are two dimensions, dim1 = 100 and dim2 = 100. Now we encounter another variable var2[100] - which dimension do we use?)

Of course, using approach 1 also solves this problem, every dimension is unique and if 10 variables all share it in HDF5 then there will be 10 dimensions in the netCDF file.

Perhaps we should add a mode flag to control this behavior? So the user can decide if shared dimensions are assumed (the default), or distinct dimensions will be created (if this mode flag is used), when opening a HDF5 file that was not created with netCDF. But if we want to add a mode flag we must finally confront the issue of expanding the number of mode flags (see #864).

[DennisHeimbigner]

Another possibility (not necessarily a good one) is to
allow an nccopy option to specify the names for selected anonymous
dimensions. Then the user could fix things up by copying the hdf5 file
to a different, true netcdf4 file, with the dimension names set as desired.

[edhartnett]

I think it would be better to fix this in the C library. In fact, I think we should just assume, as @kmuehlbauer points out, that a variable will never have the same dimension twice in its list of dimensions. So if we encounter:
var[100][100]
we should assume 2 dimensions, dim1 and dim2.

What happens if we encounter another?
var2[100]

Well we should just assign it dim1 and move on with our lives.

I can implement this, but probably not til after vacation. ;-)

[kmuehlbauer]

@edhartnett @DennisHeimbigner Thank you for your extensive insights.

I now see, that it is not that easy to conquer. Your example with var1 and var2 is quite convincing and this is possibly the main reason for having named dimensions (or dimension scales).

In my datasets this is only appearing in some corner cases. But this means several years of 5 minute spaced datasets are involved. My current workaround is to touch the file with h5py and resize one colliding dimension by one element. This works but copying the file from read-only filesystem to local filesystem is necessary.

Nevertheless some solution/handling which does not involve special treatment would be favourable.

I'll think a bit more about approaching this the next days.

[gsjaardema]

If you do start implementing this, let me know... I have another somewhat related proposal, but it would result in a change to the dimensions and would make it so that older libraries would misread the dimensions in the new files. Would still be possible to read old files with new library, but not vice-versa...

In other words, if the implementation planned for this would result in a new mode, I have something else to bundle with it to avoid yet another mode flag soon after.

Basically -- proposal can help with parallel scalability of netcdf-4 files on large processor counts

[edhartnett]

Well, @gsjaardema, we are certainly always prepared to listen. Your proposal relates to reading non-netcdf-4 HDF5 files as well?

[gsjaardema]

@edharnett No, NetCDF-4 files only. I need to do some more testing and timing; should have something at the start of our next FY -- October

[edhartnett]

OK ping me in November when I get back...

[DennisHeimbigner]

You probably should give us an overview of what you are proposing
so we can analyze if from the point of view of back compatibility,etc.

[gsjaardema]

@DennisHeimbigner That is the plan, but I am not yet ready. Hopefully early October.

[kmuehlbauer]

Ping @edhartnett approaching mid-november, so just to get this back on the plate. If I can be of any help resolving this issue, please let me know.

[edhartnett]

Thanks @kmuehlbauer for bringing this up again. I'm included to go with the solution I outlined above:

if we encounter:
var[100][100]
we should assume 2 dimensions, dim1 and dim2.

What happens if we encounter another?
var2[100]

Well we should just assign it dim1 and move on with our lives.

@gsjaardema now would be the time to bring up your proposed alternate solution...

[edhartnett]

Well @kmuehlbauer such a joy to know my documentation has at least one reader! As a programmer, I honestly don't expect people to read the documentation. ;-)

I have included an update to the tutorial on this topic.

Thanks for your help on this! Please test it out and let me know if it solves your issues...

[gsjaardema]

I have an experimental version in which I use a "hidden attribute" to specify the size of the dim instead of the length of the HDF5 dataset. This did work, but breaks any applications that are using an older version of the library since they now see every dimension as having a value of '1'.

The original reason that I investigated this (parallel scalability) did not show the performance benefit that I was hoping for (basically, since the data in the "fake" dataset is never written, the size of the dataset does not matter) so I don't know whether it is worth the trouble of figuring out how best to productionize it. Maybe if there is another format-breaking change in the future, this could be included.

It does have the benefit that it makes it easier to use HDF5 tools on NetCDF-4 files. For example, if I do an h5dump on a NetCDF-4 file, it does show me all of the non-written data in the fake dim datasets. For dims with values of 2 billion or more, this makes h5dump not useful...

netcdf-dim-size.patch.txt

The attached patch should show the basics of what I was doing. I tried to filter out the other changes I was investigating at the time (and will propose soon), but if something doesn't look quite right, it may be a vestige of the other changes. The changes did work, but didn't substantially change parallel performance and do break old libraries reading new files.

[edhartnett]

OK, but a recent change to netCDF (4.7.1?) added hidden attributes for all the dimension info. Have you looked at them? So it may be that the info you need is already there.

Adding a new hidden attribute is a problem. It breaks compatibility! So I piggybacked on some existing hidden attributes.

[kmuehlbauer]

Well @kmuehlbauer such a joy to know my documentation has at least one reader! As a programmer, I honestly don't expect people to read the documentation. ;-)

Yes, that's true @edhartnett . But RTFM (and the code, if one is able to do so) is still the best approach to gain as much information as needed. I did not want to annoy people with things RTFM would have resolved. So, thanks for having neat documentation out there!

I can try to build netcdf-c and netCDF4 locally to test this. But there is a good chance that this is released before I find the time.

Again, thank you very much for taking the time and considering that feature request.