Installation on ALCF Theta

[smpark7]

I'm currently trying to install a MOOSE App on Theta. I don't have any specific requirements concerning PETSc and libMesh versions. Can anyone provide me with the appropriate environment for installing MOOSE on Theta?

I tried the following:

module load cray-hdf5-parallel
module load cray-fftw
module load cray-petsc
module load cray-tpsl
module load cray-python
export CC=cc
export CXX=CC
export FC=ftn
export EPYTHON=python3.8
export PETSCDIR=$PETSC_DIR
export CRAYPE_LINK_TYPE=dynamic
export CRAY_ADD_RPATH=yes

which I (blindly) adapted from my experience with another supercomputer. This resulted in a seemingly successful installation, but when I run an input file (that works on other machines), the simulation run gets killed by the OOM killer despite using only ~32GB of the available 192GB memory per node.

[GiudGiud]

Hello

Does the test suite pass?
How did you measure the 32 GB ?
How did you request the 192 GB ?

Guillaume

[fdkong]

Regarding OOM, could you let me know what is the total DoFs of the simulation? What are the solver options that you are using? It could be a true OOM. How many nodes are you using?

[smpark7]

The simulation is a MultiApp run. The main app is a Transient simulation with 160k DOFs and 200k aux DOFs, NEWTON solve, asm with lu (1 overlap), while the sub app is a InversePowerMethod simulation with 560k DOFs and 240 aux DOFs, NEWTON solve, asm with lu (1 overlap).

I ran it on 8 compute nodes twice. When run on 64 ranks per node (512 total ranks), the latest logged memory use (according to MemoryUsage) before OOM is ~32 GB total. When run on 32 ranks per node (256 total ranks), the latest logged memory use before OOM is ~10 GB total. Both runs encounter the OOM killer during the fifth time step, in between the sub app and main app runs.

It's weird because I successfully ran this input file in early 2021 on Theta under a different project allocation. I did a clean install of Moltres using the same environment, but now I encounter this OOM killer issue.

[fdkong]

This size of problem should be fine on eight compute nodes. IIRC, there is 192 GB per node. You could compile the code to a "devel" mode to trigger more libmesh checks. It could be some error during the simulation, and this kind of error we usually do not check in the opt mode because of the performance. If this type of error happens in the simulation, we will receive a segfault or other error messages.

Even though it is unlikely to be a real out of memory, it does not hurt to try "asm with ilu". LU is a lot more expensive than ILU in memory

[smpark7]

Thanks for the suggestions. I'll try out devel mode and see what I get. ILU comes with a slight hit on performance for my problem, but I agree it is an option.

[smpark7]

Somehow the OOM error doesn't occur if I use aprun -n 256 -N 64 -d 1 -j 1, which is baffling to me because -d and -j should both default to 1 if they're not specified.