Problem with running GRMHD examples with the use of MPI

[AlexSokoliuk]

Hi. Recently I installed Athena++ in the clean Ubuntu 22.04 LTS system. I encounter a bunch of problems while running examples with hdf5 (without tag -hdf5 everything works fine). I installed paralell hdf5 with the help of conda install. More specifically, when I'm running configuration
./configure.py -g -b --prob gr_torus --coord=kerr-schild --flux hlle --nghost 4 -hdf5 --hdf5_path="${CONDA_PREFIX}" -mpi --mpiccmd=mpic++
And trying to analyse the results with
~/athena/bin/athena -i ~/athena/inputs/mhd_gr/athinput.fm_torus time/nlim=0
I get an error

### FATAL ERROR in function [OutputType::LoadOutputData]
Output variable 'uov' not implemented

What could it be? If I'm running aforementioned configuration without mpi, I get more complicated error with make -j:

/usr/bin/ld: obj/bvals.o: in function `BoundaryValues::ApplyPhysicalBoundaries(double, double, std::vector<BoundaryVariable*, std::allocator<BoundaryVariable*> >)':
bvals.cpp:(.text+0x3146): undefined reference to `std::__throw_bad_array_new_length()'
/usr/bin/ld: obj/bvals_refine.o: in function `BoundaryValues::ApplyPhysicalBoundariesOnCoarseLevel(NeighborBlock const&, double, double, int, int, int, int, int, int, std::vector<BoundaryVariable*, std::allocator<BoundaryVariable*> >)':
bvals_refine.cpp:(.text+0x1186): undefined reference to `std::__throw_bad_array_new_length()'
/usr/bin/ld: obj/bvals_refine.o: in function `BoundaryValues::ProlongateBoundaries(double, double, std::vector<BoundaryVariable*, std::allocator<BoundaryVariable*> >)':
bvals_refine.cpp:(.text+0x1e1b): undefined reference to `std::__throw_bad_array_new_length()'
/usr/bin/ld: obj/meshblock_tree.o: in function `std::_Hashtable<LogicalLocation, std::pair<LogicalLocation const, int>, std::allocator<std::pair<LogicalLocation const, int> >, std::__detail::_Select1st, std::equal_to<LogicalLocation>, LogicalLocationHash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_rehash(unsigned long, unsigned long const&)':
meshblock_tree.cpp:(.text._ZNSt10_HashtableI15LogicalLocationSt4pairIKS0_iESaIS3_ENSt8__detail10_Select1stESt8equal_toIS0_E19LogicalLocationHashNS5_18_Mod_range_hashingENS5_20_Default_ranged_hashENS5_20_Prime_rehash_policyENS5_17_Hashtable_traitsILb1ELb0ELb1EEEE9_M_rehashEmRKm[_ZNSt10_HashtableI15LogicalLocationSt4pairIKS0_iESaIS3_ENSt8__detail10_Select1stESt8equal_toIS0_E19LogicalLocationHashNS5_18_Mod_range_hashingENS5_20_Default_ranged_hashENS5_20_Prime_rehash_policyENS5_17_Hashtable_traitsILb1ELb0ELb1EEEE9_M_rehashEmRKm]+0x107): undefined reference to `std::__throw_bad_array_new_length()'
/usr/bin/ld: obj/mesh.o: in function `Mesh::CorrectMidpointInitialCondition()':
mesh.cpp:(.text+0xbf0d): undefined reference to `std::__throw_bad_array_new_length()'
/usr/bin/ld: obj/mesh.o:mesh.cpp:(.text+0xca2e): more undefined references to `std::__throw_bad_array_new_length()' follow
collect2: error: ld returned 1 exit status
make: *** [Makefile:96: bin/athena] Error 1

I'll be grateful for any response.

[tomidakn]

This is nothing relevant to HDF5 nor GR but is the same issue as #436. For now, please turn off the uov output in the input file.

[kahoooo]

Just wondering, is the second issue related?

[c-white]

The second issue would need more information, but I suspect it's unrelated. How was the environment reconfigured? Was the HDF5 library switch to a serial version? Was make clean called before recompiling?

[tomidakn]

I do not think so...

I am not sure yet, but the second issue looks like related to versions of the standard library.
JuliaLang/julia#40703
This is Fedora, but Ubuntu 22.04 uses gcc11 as well.

As @c-white said, it is important to (re)build the code in a clean environment.

[AlexSokoliuk]

Unfortunately, make clean does not help. However, I planned to use Athena with MPI, so it's not a big loss. I wanted to ask one more question. When I'm trying to obtain a plot for MPI configuration mentioned earlier (this time I disable <output3>, that respond for uov variable), I get a python error

Traceback (most recent call last):
  File "/home/oleksii/athena/vis/python/plot_spherical.py", line 395, in <module>
    main(**vars(args))
  File "/home/oleksii/athena/vis/python/plot_spherical.py", line 180, in main
    vals_left = 0.5 * (data[kwargs['quantity']][(nx3/2)-1, :, :]
IndexError: only integers, slices (`:`), ellipsis (`...`), numpy.newaxis (`None`) and integer or boolean arrays are valid indices

Does it arise because of the newest version of python installed? I saw a similar issue before (check #169). Do I need to downgrade my python version in order to produce plots?

[c-white]

Try replacing all / with // and see if that works for plotting.

[AlexSokoliuk]

It doesn't help as well:

(base) oleksii@oleksii-VivoBook-ASUSLaptop-X512JP-X512JP:~/work$ ~//athena//vis//python//plot_spherical.py fm_torus.prim.00000.athdf rho show     --vmin=1.0e-6 --vmax=1.0e0 --logc -s Bcc --stream_density 2.0
Traceback (most recent call last):
  File "/home/oleksii//athena//vis//python//plot_spherical.py", line 395, in <module>
    main(**vars(args))
  File "/home/oleksii//athena//vis//python//plot_spherical.py", line 180, in main
    vals_left = 0.5 * (data[kwargs['quantity']][(nx3/2)-1, :, :]
IndexError: only integers, slices (`:`), ellipsis (`...`), numpy.newaxis (`None`) and integer or boolean arrays are valid indices

[kahoooo]

@c-white may be referring to the / in the (nx3/2)-1
This is due to / default to be integer division in Python 2. But in Python 3, / always return floating-point numbers, and hence, the IndexError

[AlexSokoliuk]

It worked! Thank you all for helping such a silly person as me :)

[felker]

@c-white do we need to update the division operators in all Python plotting scripts?

[c-white]

@felker I actually thought we had already done it, but I guess there are some places that fell through the cracks.