Building APEX on NERSC Perlmutter

The APEX CMake configuration will download and build all requested dependencies, if desired. The easiest way to install APEX for Perlmutter is with spack, doing:

spack install apex +cuda +mpi +lmsensors +binutils +papi

If you're not using spack, the next easiest way is to do the following:

module load papi

cwd=`pwd`
builddir=${cwd}/build
instdir=${cwd}/install

rm -rf ${builddir} ${instdir}

set -x
cmake -B ${builddir} \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_INSTALL_PREFIX=${instdir} \
-DCMAKE_CXX_COMPILER=`which CC` \
-DCMAKE_C_COMPILER=`which cc` \
-DAPEX_WITH_CUDA=TRUE \
-DAPEX_WITH_BFD=TRUE \
-DAPEX_WITH_LM_SENSORS=TRUE \
-DAPEX_WITH_PLUGINS=TRUE \
-DAPEX_WITH_MPI=TRUE \
-DPAPI_ROOT=${CRAY_PAPI_PREFIX} \

cmake --build build -j8
cmake --install build

Running with APEX

After building, you can run with APEX by inserting apex_exec between your srun command and your application executable name, for example:

srun --label --unbuffered -N ${SLURM_JOB_NUM_NODES} -n ${n_mpi_ranks} -c ${OMP_NUM_THREADS} --cpu-bind=threads --ntasks-per-node=${n_mpi_ranks_per_node} --gpus-per-task=1 --gpu-bind=single:1 \
../apex/install/bin/apex_exec --apex:mpi --apex:monitor-gpu --apex:scatter \
${MY_EXE}

For a list of apex_exec options, run apex_exec --help.