Tested version of hdf5?

[kurtsansom]

I am trying to run a relatively simple spherical polar test case and I am hitting a segfault at the very beginning when athena writes the first output file. It appears to be happening in the hdf5 library but I am not 100% sure.

What verisons of hdf5 have been tested with athena++

I am getting a double free or corruption (out) It's possible its an error on my end. but it was working a few weeks ago for me without an issue.

[kurtsansom]

Does thread safe version of hdf5 need to be used either with MPI or openMPI?

Things were working before when I was using OpenMP, then I tried switching to MPI and that's when it stopped working, but then trying to revert back to using openMP or without parallelism it continued to not work.
(obviously I am using make clean, python configure.py and make workflow. it really has me confused.

[kurtsansom]

I think I may have figured it out. make clean does not appear to remove /src/defs.hpp
EDIT: nope that doesn't work.

[felker]

Only configure.py rewrites that file

[felker]

You need to build a parallel HDF5 library: https://support.hdfgroup.org/HDF5/PHDF5/

[kurtsansom]

Based on your feedback I believe I can confirm there is some kind of segfault when using the hdf5 libraries. My current hypotheses include hdf5 build configuration issue, and hdf5 library version issue as it appears that Athena is tested against older hdf5 libraries. I have some more tests to try but I believe that is the issue. The regression tests appear to work fine and they don't use hdf5, running an example separately without hdf5 also appears to work.

[kurtsansom]

I get the similar segfaults with hdf5@1.10.6, hdf5@1.10.7, and hdf5@1.10.8

$ gdb ../../athena/bin/athena
...
Reading symbols from ../../athena/bin/athena...
(gdb) r -i athinput_junxiang.cyl_test 
Starting program: /media/store/krs/build/athena/athena/bin/athena -i athinput_junxiang.cyl_test
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
munmap_chunk(): invalid pointer

Program received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
51	../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007ffff6abe921 in __GI_abort () at abort.c:79
#2  0x00007ffff6b07967 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7ffff6c34b0d "%s\n")
    at ../sysdeps/posix/libc_fatal.c:181
#3  0x00007ffff6b0e9da in malloc_printerr (str=str@entry=0x7ffff6c36720 "munmap_chunk(): invalid pointer")
    at malloc.c:5342
#4  0x00007ffff6b15fbc in munmap_chunk (p=0x7ffff1c20000) at malloc.c:2846
#5  __GI___libc_free (mem=0x7ffff1c20010) at malloc.c:3127
#6  0x000055555569937a in ATHDF5Output::WriteOutputFile (this=0x55555658f430, pm=0x55555595b000, 
    pin=0x55555595ada0, flag=false) at src/outputs/athena_hdf5.cpp:784
#7  0x00005555556a27c2 in Outputs::MakeOutputs (this=0x55555658d010, pm=0x55555595b000, pin=0x55555595ada0, 
    wtflag=false) at src/outputs/outputs.cpp:774
#8  0x0000555555565907 in main (argc=3, argv=0x7fffffffccd8) at src/main.cpp:397

[kurtsansom]

if I modify the code here,

athena/src/outputs/athena_hdf5.cpp

Lines 783 to 784 in b8244b0

	for (int n = 0; n < num_datasets; ++n)
	delete[] data_buffers[n];

to

  for (int n = num_datasets; n > 0 ; --n) {
    delete[] data_buffers[n];
  }

the segfault goes away. but this is incorrect. basically deleting both data_buffers[1], data_buffers[0], and data_buffers appear to cause the issue.

[felker]

Athena++ has been tested with nearly every major version of HDF5 released in the last 5 years. Which version are you using? How did you build the library?

What is your input file and configure/compile command?

[kurtsansom]

I am using spack and gcc@10.3.0 on ubuntu Ubuntu 18.04.6 LTS

(I have also used gcc@7.5.0 for athena but linking to hdf5 built with gcc@10.3.0)

$ spack env create athena_hdf5_dbg
$ spack env activate athena_hdf5_dbg
$ spack add  hdf5@1.10.6+cxx+fortran+hl+mpi+tools^openmpi+cxx openmpi+cxx
$ spack install

I had to modify the configure script to configure with g++-10

$ cd athena
$ python configure.py --cxx=g++-10 --prob=blast --coord=cylindrical -debug -b -hdf5 --hdf5_path=/home/krs/tools/spack/var/spack/environments/athena_hdf5_dbg/.spack-env/view
$ make clean
$ make

$ cd ../
$ mkdir test
$ cd test
$ cp ../athena/inputs/mhd/athinput.blast_cyl .

Modify file_type to file_type = hdf5

$../athena/bin/athena -i athinput.blast_cyl

I also ran it with valgrind and the culprit appears to be

athena/src/outputs/athena_hdf5.cpp

Lines 386 to 387 in b8244b0

	data_buffers[n_dataset][(ndv*num_blocks_local+nba)*nx3*nx2*nx1+index]
	= pod->data(v,k,j,i);

Caveat: Some print statements were added

valgrind -q --tool=memcheck --leak-check=full --main-stacksize=32392192 ../../athena/bin/athena -i athinput.blast_cyl
num_datasets: 2
5  1 64 64 64
5 1310720
3 786432
n_dataset: 0
n_dataset: 0
n_dataset: 0
n_dataset: 0
n_dataset: 0
==25735== Invalid write of size 4
==25735==    at 0x24B47D: ATHDF5Output::WriteOutputFile(Mesh*, ParameterInput*, bool) (athena_hdf5.cpp:400)
==25735==    by 0x256A7F: Outputs::MakeOutputs(Mesh*, ParameterInput*, bool) (outputs.cpp:774)
==25735==    by 0x119956: main (main.cpp:397)
==25735==  Address 0x13d75040 is 0 bytes after a block of size 5,242,880 alloc'd
==25735==    at 0x4C3289F: operator new[](unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==25735==    by 0x24A098: ATHDF5Output::WriteOutputFile(Mesh*, ParameterInput*, bool) (athena_hdf5.cpp:273)
==25735==    by 0x256A7F: Outputs::MakeOutputs(Mesh*, ParameterInput*, bool) (outputs.cpp:774)
==25735==    by 0x119956: main (main.cpp:397)
==25735== 
n_dataset: 0
==25735== 
==25735== Process terminating with default action of signal 11 (SIGSEGV)
==25735==  Access not within mapped region at address 0x14176000
==25735==    at 0x24B47D: ATHDF5Output::WriteOutputFile(Mesh*, ParameterInput*, bool) (athena_hdf5.cpp:400)
==25735==    by 0x256A7F: Outputs::MakeOutputs(Mesh*, ParameterInput*, bool) (outputs.cpp:774)
==25735==    by 0x119956: main (main.cpp:397)
==25735==  If you believe this happened as a result of a stack
==25735==  overflow in your program's main thread (unlikely but
==25735==  possible), you can try to increase the size of the
==25735==  main thread stack using the --main-stacksize= flag.
==25735==  The main thread stack size used in this run was 32395264.
Segmentation fault (core dumped)

[kurtsansom]

Thank you for helping resolve this issue.