Using BrainGrid with HDF5

What is HDF5?

HDF5 is an open source Hierarchical Data Format product consisting of a data format specification and a supporting library implementation. It's a general-purpose, machine-independent standard for storing large amount of data in files, typically used in research applications to distribute and access very large datasets without using a database.

Why HDF5?

Similar to XML documents, HDF files are self-describing and allow users to specify complex data relationships and dependencies. In contrast to XML documents, HDF files can contain binary data and allow direct access to parts of the file without first parsing the entire contents.

Think of HDF as a file system within a file. It lets you organize the data hierarchically and manage large amount of data very efficiently. HDF5 is also used by a wide range of engineering and scientific fields that want a standard way to store data so that it can be shared.

For more information about the HDF5 file format, read the HDF5 documentation available at the HDF Web site.

Use HDF5 with BrainGrid

If you have the HDF5 package installed and ready to use HDF5 as your output file format, follow these steps:

1. Go into the Makefile and set CUSEHDF5 variable under Conditional Flags from "no" to "yes"

   CUSEHDF5 = yes
   

2. Check if the following paths in Makefile under Source Directories point to where HDF5 is installed on your system. If not, modify them accordingly to represent the new paths.

   ifeq ($(CUSEHDF5), yes)
        H5INCDIR = /usr/include/hdf5/serial                # include dir
        H5LIBDIR = /usr/lib/x86_64-linux-gnu/hdf5/serial/  # library dir
   else
        H5INCDIR = .
   endif
   

3. Lastly, go into the Config files and change the output file extension from .xml to .h5. The output file name is specified inside the SimInfoParams section.

   <OutputParams stateOutputFileName="test-small-out.h5"/>
   

At this point you should be able to compile and run BrainGrid. However, we have encountered a few issues while using HDF5 with BrainGrid. If you have problem compiling BrainGrid code with HDF5 option, take a look at the possible issues and solutions listed below:

Possible Issues and fixes

• undefined reference to linker error

  if the error message looks like this:

  undefined reference to `H5check_version'
  undefined reference to `H5Fcreate'
  undefined reference to `H5Fclose'
  

  First, make sure your HDF5 library files are indeed under your H5LIBDIR. You should be able to find files like libhdf5_cpp.a, libhdf5_hl.a, libhdf5_hl_cpp.a inside your library directory.

  Then change LD linker variable under Build Tools from g++ to h5c++.

  LD = h5c++
  

  The compiler helper script h5c++ takes care of specifying where the HDF5 header files and libraries are on the command line which should fix the problem. For more HDF5 tools, see HDF5 Reference Manual