Diablo 3

This version is based on the original version of J. R. Taylor (cf. DIABLO2.0 ReadMe). It includes updated FFTW and pHDF5 libraries. It also requires the toml-f library. The appropriate paths need to be set in the Makefile (for/Makefile0) and added to the LD_LIBRARY_PATH.

Test case

The test case is a 10 minute simulation on a fairly small domain. The initial conditions are pretty much nonsense.

The number of grid points in x,y,z are set in grid_def.all. The number of grid points in y must match grid.h5.

The number of processes used in the y and z directions are passed as command line arguments to for/setup_run. The number of processes in y must divide NY - 1 = 193 - 1 = 192 = 6 x 32 and the number of processes in z must divide NZ = 32.

In input.toml the VERBOSITY is set to its highest setting. This includes printing out a message every time mpi_alltoall is called in fft_xz_to_fourier.

Running the test case:

1. Copy test_case directory to $SCRATCH
2. Modify the path in diablo.slurm that points to for/setup_run
3. Submit slurm job

Important files

• for/Makefile0 is the makefile. We usually recompile the code every time we run it as the grid sizes and processor distribution need to be known at compile time.
• for/setup_run creates the grid files included in the fortran code and then compiles the code. This also sets a number of flags that set options in the makefile. The --toml flag is necessary whereas the --debug flag is optional. We don't use the --shared flag. This was an attempt to improve the performance of the code using mpi shared memory directives but it turns out to be slower. Turning this option on would actually bypass the mpi_alltoall call that keeps failing but I tried this and this code also fails.
• for/domain.f90 this is where mpi_initis run and the mpi_subcarts are created.
• for/fft.f90 this file isolates all the calls to fftw. Contains the subroutine fft_xz_to_fourier in which mpi_alltoall keeps failing. Note that fft_xz_to_physical also has a call to mpi_alltoall which doesn't seem to fail.