Skip to content

Commit

Permalink
Docs: Reorder HPC Profiles + Batch Scripts (#2757)
Browse files Browse the repository at this point in the history 
* Docs: Reorder Summit Files

* Docs: Reorder Spock Files

* Docs: Reorder Cori Files

* Docs: Reorder Perlmutter Files

* Docs: Reorder Juwels Files

* Docs: Reorder Lassen Files

* Docs: Reorder Quartz Files

* Docs: Reorder Ookami Files

* Docs: Also Move Summit Profile Script

* Listing Captions: Location in Source
  • Loading branch information
@ax3l
ax3l authored Jan 20, 2022
1 parent 76ae829 commit aa531aa
Show file tree
Hide file tree
Showing 30 changed files with 422 additions and 403 deletions.
103 changes: 20 additions & 83 deletions Docs/source/install/hpc/cori.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -32,29 +32,9 @@ KNL

We use the following modules and environments on the system (``$HOME/knl_warpx.profile``).

.. code-block:: bash
module swap craype-haswell craype-mic-knl
module swap PrgEnv-intel PrgEnv-gnu
module load cmake/3.21.3
module switch cray-libsci cray-libsci/20.09.1
module load cray-hdf5-parallel/1.10.5.2
module load cray-fftw/3.3.8.4
module load cray-python/3.7.3.2
export PKG_CONFIG_PATH=$FFTW_DIR/pkgconfig:$PKG_CONFIG_PATH
export CMAKE_PREFIX_PATH=$HOME/sw/c-blosc-1.12.1-knl-install:$CMAKE_PREFIX_PATH
export CMAKE_PREFIX_PATH=$HOME/sw/adios2-2.7.1-knl-install:$CMAKE_PREFIX_PATH
export CMAKE_PREFIX_PATH=$HOME/sw/blaspp-master-knl-install:$CMAKE_PREFIX_PATH
export CMAKE_PREFIX_PATH=$HOME/sw/lapackpp-master-knl-install:$CMAKE_PREFIX_PATH
if [ -d "$HOME/sw/venvs/knl_warpx" ]
then
source $HOME/sw/venvs/knl_warpx/bin/activate
fi
export CXXFLAGS="-march=knl"
export CFLAGS="-march=knl"
.. literalinclude:: ../../../../Tools/machines/cori-nersc/knl_warpx.profile.example
:language: bash
:caption: You can copy this file from ``Tools/machines/cori-nersc/knl_warpx.profile.example``.

And install ADIOS2, BLAS++ and LAPACK++:

Expand Down Expand Up @@ -105,25 +85,9 @@ Haswell

We use the following modules and environments on the system (``$HOME/haswell_warpx.profile``).

.. code-block:: bash
module swap PrgEnv-intel PrgEnv-gnu
module load cmake/3.21.3
module switch cray-libsci cray-libsci/20.09.1
module load cray-hdf5-parallel/1.10.5.2
module load cray-fftw/3.3.8.4
module load cray-python/3.7.3.2
export PKG_CONFIG_PATH=$FFTW_DIR/pkgconfig:$PKG_CONFIG_PATH
export CMAKE_PREFIX_PATH=$HOME/sw/c-blosc-1.12.1-haswell-install:$CMAKE_PREFIX_PATH
export CMAKE_PREFIX_PATH=$HOME/sw/adios2-2.7.1-haswell-install:$CMAKE_PREFIX_PATH
export CMAKE_PREFIX_PATH=$HOME/sw/blaspp-master-haswell-install:$CMAKE_PREFIX_PATH
export CMAKE_PREFIX_PATH=$HOME/sw/lapackpp-master-haswell-install:$CMAKE_PREFIX_PATH
if [ -d "$HOME/sw/venvs/haswell_warpx" ]
then
source $HOME/sw/venvs/haswell_warpx/bin/activate
fi
.. literalinclude:: ../../../../Tools/machines/cori-nersc/haswell_warpx.profile.example
:language: bash
:caption: You can copy this file from ``Tools/machines/cori-nersc/haswell_warpx.profile.example``.

And install ADIOS2, BLAS++ and LAPACK++:

Expand Down Expand Up @@ -174,41 +138,11 @@ GPU (V100)

Cori provides a partition with `18 nodes that include V100 (16 GB) GPUs <https://docs-dev.nersc.gov/cgpu/>`__.
We use the following modules and environments on the system (``$HOME/gpu_warpx.profile``).
You can copy this file from ``Tools/machines/cori-nersc/gpu_warpx.profile.example``:

.. code-block:: bash
export proj="m1759"
module purge
module load modules
module load cgpu
module load esslurm
module load gcc/8.3.0 cuda/11.4.0 cmake/3.21.3
module load openmpi
export CMAKE_PREFIX_PATH=$HOME/sw/c-blosc-1.12.1-gpu-install:$CMAKE_PREFIX_PATH
export CMAKE_PREFIX_PATH=$HOME/sw/adios2-2.7.1-gpu-install:$CMAKE_PREFIX_PATH
if [ -d "$HOME/sw/venvs/gpu_warpx" ]
then
source $HOME/sw/venvs/gpu_warpx/bin/activate
fi
# compiler environment hints
export CC=$(which gcc)
export CXX=$(which g++)
export FC=$(which gfortran)
export CUDACXX=$(which nvcc)
export CUDAHOSTCXX=$(which g++)
# optimize CUDA compilation for V100
export AMREX_CUDA_ARCH=7.0
# allocate a GPU, e.g. to compile on
# 10 logical cores (5 physical), 1 GPU
function getNode() {
salloc -C gpu -N 1 -t 30 -c 10 --gres=gpu:1 -A $proj
}
.. literalinclude:: ../../../../Tools/machines/cori-nersc/gpu_warpx.profile.example
:language: bash
:caption: You can copy this file from ``Tools/machines/cori-nersc/gpu_warpx.profile.example``.

And install ADIOS2:

Expand Down Expand Up @@ -306,14 +240,15 @@ Do not forget to first ``source $HOME/knl_warpx.profile`` if you have not done s

For PICMI Python runs, the ``<path/to/executable>`` has to read ``python3`` and the ``<input file>`` is the path to your PICMI input script.

.. literalinclude:: ../../../../Tools/BatchScripts/batch_cori.sh
.. literalinclude:: ../../../../Tools/machines/cori-nersc/cori_knl.sbatch
:language: bash
:caption: You can copy this file from ``Tools/machines/cori-nersc/cori_knl.sbatch``.

To run a simulation, copy the lines above to a file ``batch_cori.sh`` and run
To run a simulation, copy the lines above to a file ``cori_knl.sbatch`` and run

.. code-block:: bash
sbatch batch_cori.sh
sbatch cori_knl.sbatch
to submit the job.

Expand All @@ -338,15 +273,16 @@ The batch script below can be used to run a WarpX simulation on 1 `Haswell node

Do not forget to first ``source $HOME/haswell_warpx.profile`` if you have not done so already for this terminal session.

.. literalinclude:: ../../../../Tools/BatchScripts/batch_cori_haswell.sh
.. literalinclude:: ../../../../Tools/machines/cori-nersc/cori_haswell.sbatch
:language: bash
:caption: You can copy this file from ``Tools/machines/cori-nersc/cori_haswell.sbatch``.

To run a simulation, copy the lines above to a file ``batch_cori_haswell.sh`` and
To run a simulation, copy the lines above to a file ``cori_haswell.sbatch`` and
run

.. code-block:: bash
sbatch batch_cori_haswell.sh
sbatch cori_haswell.sbatch
to submit the job.

Expand All @@ -367,8 +303,9 @@ For single-node runs, try to run one grid per GPU.

A multi-node batch script template can be found below:

.. literalinclude:: ../../../../Tools/BatchScripts/batch_cori_gpu.sh
.. literalinclude:: ../../../../Tools/machines/cori-nersc/cori_gpu.sbatch
:language: bash
:caption: You can copy this file from ``Tools/machines/cori-nersc/cori_gpu.sbatch``.


.. _post-processing-cori:
Expand Down
3 changes: 2 additions & 1 deletion Docs/source/install/hpc/crusher.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ We use the following modules and environments on the system (``$HOME/crusher_war

.. literalinclude:: ../../../../Tools/machines/crusher-olcf/crusher_warpx.profile.example
:language: bash

:caption: You can copy this file from ``Tools/machines/crusher-olcf/crusher_warpx.profile.example``.

We recommend to store the above lines in a file, such as ``$HOME/crusher_warpx.profile``, and load it into your shell after a login:

Expand Down Expand Up @@ -75,6 +75,7 @@ Or in non-interactive runs:

.. literalinclude:: ../../../../Tools/machines/crusher-olcf/submit.sh
:language: bash
:caption: You can copy this file from ``Tools/machines/crusher-olcf/submit.sh``.


.. _post-processing-crusher:
Expand Down
34 changes: 7 additions & 27 deletions Docs/source/install/hpc/juwels.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -31,38 +31,17 @@ Use the following commands to download the WarpX source code and switch to the c
We use the following modules and environments on the system.

.. code-block:: bash
# please set your project account
export proj=<yourProject>
# required dependencies
module load ccache
module load CMake
module load GCC
module load CUDA/11.3
module load OpenMPI
module load FFTW
module load HDF5
module load Python
# JUWELS' job scheduler may not map ranks to GPUs,
# so we give a hint to AMReX about the node layout.
# This is usually done in Make.<supercomputing center> files in AMReX
# but there is no such file for JSC yet.
export GPUS_PER_SOCKET=2
export GPUS_PER_NODE=4
# optimize CUDA compilation for V100 (7.0) or for A100 (8.0)
export AMREX_CUDA_ARCH=8.0
.. literalinclude:: ../../../../Tools/machines/juwels-jsc/juwels_warpx.profile.example
:language: bash
:caption: You can copy this file from ``Tools/machines/juwels-jsc/juwels_warpx.profile.example``.

Note that for now WarpX must rely on OpenMPI instead of the recommended MPI implementation on this platform MVAPICH2.

We recommend to store the above lines in a file, such as ``$HOME/warpx.profile``, and load it into your shell after a login:
We recommend to store the above lines in a file, such as ``$HOME/juwels_warpx.profile``, and load it into your shell after a login:

.. code-block:: bash
source $HOME/warpx.profile
source $HOME/juwels_warpx.profile
Then, ``cd`` into the directory ``$HOME/src/warpx`` and use the following commands to compile:

Expand Down Expand Up @@ -98,8 +77,9 @@ The `Juwels GPUs <https://apps.fz-juelich.de/jsc/hps/juwels/configuration.html>`

An example submission script reads

.. literalinclude:: ../../../../Tools/BatchScripts/batch_juwels.sh
.. literalinclude:: ../../../../Tools/machines/juwels-jsc/juwels.sbatch
:language: bash
:caption: You can copy this file from ``Tools/machines/juwels-jsc/juwels.sbatch``.

Queue: batch (2 x Intel Xeon Platinum 8168 CPUs, 24 Cores + 24 Hyperthreads/CPU)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Expand Down
54 changes: 8 additions & 46 deletions Docs/source/install/hpc/lassen.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,50 +25,11 @@ Use the following commands to download the WarpX source code and switch to the c
git clone https://github.com/ECP-WarpX/WarpX.git $HOME/src/warpx
We use the following modules and environments on the system (``$HOME/warpx.profile``).

.. code-block:: bash
# please set your project account
export proj=<yourProject>
# required dependencies
module load cmake/3.20.2
module load gcc/8.3.1
module load cuda/11.2.0
# optional: for PSATD support
module load fftw/3.3.8
# optional: for QED lookup table generation support
module load boost/1.70.0
# optional: for openPMD support
# TODO ADIOS2
module load hdf5-parallel/1.10.4
# optional: for PSATD in RZ geometry support
# TODO: blaspp lapackpp
# optional: for Python bindings
module load python/3.8.2
# optional: an alias to request an interactive node for two hours
alias getNode="bsub -G $proj -W 2:00 -nnodes 1 -Is /bin/bash"
# fix system defaults: do not escape $ with a \ on tab completion
shopt -s direxpand
# optimize CUDA compilation for V100
export AMREX_CUDA_ARCH=7.0
# compiler environment hints
export CC=$(which gcc)
export CXX=$(which g++)
export FC=$(which gfortran)
export CUDACXX=$(which nvcc)
export CUDAHOSTCXX=$(which g++)
We use the following modules and environments on the system (``$HOME/lassen_warpx.profile``).

.. literalinclude:: ../../../../Tools/machines/lassen-llnl/lassen_warpx.profile.example
:language: bash
:caption: You can copy this file from ``Tools/machines/lassen-llnl/lassen_warpx.profile.example``.

We recommend to store the above lines in a file, such as ``$HOME/lassen_warpx.profile``, and load it into your shell after a login:

Expand Down Expand Up @@ -104,14 +65,15 @@ The batch script below can be used to run a WarpX simulation on 2 nodes on the s
Replace descriptions between chevrons ``<>`` by relevant values, for instance ``<input file>`` could be ``plasma_mirror_inputs``.
Note that the only option so far is to run with one MPI rank per GPU.

.. literalinclude:: ../../../../Tools/BatchScripts/batch_lassen.sh
.. literalinclude:: ../../../../Tools/machines/lassen-llnl/lassen.bsub
:language: bash
:caption: You can copy this file from ``Tools/machines/lassen-llnl/lassen.bsub``.

To run a simulation, copy the lines above to a file ``batch_lassen.sh`` and run
To run a simulation, copy the lines above to a file ``lassen.bsub`` and run

.. code-block:: bash
bsub batch_lassen.sh
bsub lassen.bsub
to submit the job.

Expand Down
35 changes: 5 additions & 30 deletions Docs/source/install/hpc/ookami.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -31,36 +31,9 @@ Use the following commands to download the WarpX source code and switch to the c
We use the following modules and environments on the system (``$HOME/warpx_gcc10.profile``).

.. code-block:: bash
# please set your project account (not relevant yet)
#export proj=<yourProject>
# required dependencies
module load cmake/3.19.0
module load gcc/10.3.0
module load openmpi/gcc10/4.1.0
# optional: faster builds (not available yet)
#module load ccache
#module load ninja
# optional: for PSATD support (not available yet)
#module load fftw
# optional: for QED lookup table generation support (not available yet)
#module load boost
# optional: for openPMD support
#module load adios2 # not available yet
#module load hdf5 # only serial
# compiler environment hints
export CC=$(which gcc)
export CXX=$(which g++)
export FC=$(which gfortran)
export CXXFLAGS="-mcpu=a64fx"
.. literalinclude:: ../../../../Tools/machines/ookami-sbu/ookami_warpx.profile.example
:language: bash
:caption: You can copy this file from ``Tools/machines/ookami-sbu/ookami_warpx.profile.example``.

We recommend to store the above lines in a file, such as ``$HOME/warpx_gcc10.profile``, and load it into your shell after a login:

Expand Down Expand Up @@ -126,3 +99,5 @@ We compiled with the Fujitsu Compiler (Clang) with the following build string:
-DAMReX_MPI_THREAD_MULTIPLE=FALSE \
-DWarpX_COMPUTE=OMP
cmake --build build -j 10
Note that the best performance for A64FX is currently achieved with the GCC or ARM compilers.
Loading

0 comments on commit aa531aa

Please sign in to comment.