-
Notifications
You must be signed in to change notification settings - Fork 8
ucvm2mesh_mpi Example
UCVMC can be used to generate large meshes. The UCVM distribution contains a serial meshing program called ucvm2mesh. This entry describes a parallel version of the meshing code, used to speed up the meshing process, called ucvm2mesh-mpi.
Notice: ucvm2mesh-mpi is meant to be run as a MPI job, not as a standalone executable.
ucvm2mesh-mpi generates a mesh in either IJK-12, IJK-20, IJK-32, or SORD format. Unlike its serial version, ucvm2mesh this command can use multiple cores to generate the mesh. It does so by reading in the specifications in a given configuration file.
First, we'll describe special considerations about running on a computing cluster. Then, we'll show an example of running on a specific cluster, one on campus at USC.
The configuration files required by ucvm2mesh-mpi are more extensive than other ucvm commands. A working example of a ucvm2mesh_mpi program is provided below.
First, ucvmc must be installed on a computer cluster. Most current HPC clusters are based on Linux operating systems, and in most cases, they provide the software tools needed to compile and build the ucvmc distribution, including the gnu compilers, a python interpreter, and the automake tools
To successfully install and use ucvm2mesh-mpi on a cluster, consider special conditions.
Clusters typically have a variety of disks and filesystems, and the user must figure out how to use them appropriate when building a ucvm mesh. Often a cluster will set a quota on a specific user directory. A cluster will provide a large temporary disk for use by running jobs. Sometimes a cluster will provide a group directory, where members of a group can install codes or store data used by the group.
An important consideration for each cluster file system is whether the filesystem is visible from the compute nodes. In some cases, the computers and disks that the user sees when the log into the cluster is not visible when a job is running on the cluster. In some case, the cluster nodes are different types than the login nodes. UCVMC should be installed on a filesystem that is visible from the compute nodes.
The user should identify locations for the UCVMC/source directory (25GB) and the UCVMC/installation directory (20GB). After installation, the source directory can be deleted.
Large meshes require large disk storage. ucvm2mesh-mpi can be used to generate multi-terabyte meshes. When planning to use ucvm2mesh-mpi, the user should identify the output file system with room to store the mesh being created.
In some cases, the user can write the output file to the cluster /scratch. However, files written to /scratch can be removed by the cluster software, so typically the user should move the output files off of scratch as part of their cluster job.
Typically, ucvm2mesh-mpi configurations are organized around processors. The mesh extraction job is divided onto multiple processor, which speeds up the mesh building process.
Cluster hardware is often discussed using the term node, in which case, a cluster node will often have multiple processors. Typically, you cannot schedule a job on anything less than 1 node. If your node has 32 processors, and you schedule 1 processor per node, the other 31 processors will be idle. However, the RAM memory on a node is often shared by all the processors, so if you try to run 32 processors on a node, each of your jobs will have less RAM to work with. If your job runs out of memory, you may need to run with fewer processors per node.
Worse case, you can run ucvm2mesh-mpi on one processor per node. If your nodes have more then one processor, you can run ucvm2mesh-mpi on multiple processors per node. If you have a 32 processor node, you might be able to run ucvm2mesh-mpi on 32 processors per node.
Some cluster require that all executables are statically linked. The default ucvm2mesh-mpi is dynamic linking. Dynamically linked executables are typically smaller, but they achieve lower performance than statically linked programs. Statically linked ucvm2mesh-mpi are advanced topic covered in the advanced user guide on the SCEC wiki.
To figure our how to divide your ucvm2mesh-mpi job onto multiple processors, first calculate the total mesh points. We will use the following example, which defines a 1760800 mesh point mesh.
# Number of cells along each dim nx=384 ny=248 nz=25
To divide the ucvm queries among several processors, we need to specify the length, width, and depth of our simulation volume. The length of the volume is defined as nx * spacing, the width is ny * spacing, and the depth is nz * spacing.
For the MPI version of ucvm2mesh-mpi, we need to specify the number of processors to dedicate to each axis. The total number of processors required is px * py * pz. For this example, we would need 20 cores dedicated to this task. Using our example above, we can divide the nx mesh points by 2, the number of ny mesh points by 2, and the number of nz mesh points by 5. So the number of processors in each direction can be used:
Then we divide the mesh points onto a number of processors, in each direction. An important constraint is that the number of mesh points must evenly by the number of processors. The guideline for the processors are that px * py * pz = num processors in the job. So for each x, y, and z, nx / px, ny / py, nz / pz, must be whole numbers. So if nx is 1000, px cannot be 3 but px can be 5. If you have a prime number of mesh points in a direction, you must use 1, or the prime number of processors.
# Partitioning of grid among processors px=2 py=2 pz=5
For ucvm2mesh-mpi, we need to specify the number of processors to dedicate to each axis. The total number of processors required is px * py * pz. For this example, we would need 20 cores dedicated to this task.
The user must also specify the ucvm model, or models, to be queried. Also, the user should specify the min Vp and min Vs values in the mesh. These are described as floor values for Vs and Vp. If the user specifies a min vp or min vs, if ucvm retrieves a lower vp or vs from the CVM while it is building ht mesh, it will replace the lower vp or vs with the minimum value defined here. Setting these values to 0, and ucmv will not modify the material properties it receives from CVMs in use.
model=cvmsi,bbp1d # Vs/Vp minimum vp_min=0 vs_min=0
We must also specify the location for our mesh file and the grid file. A mesh file is the list of lon,lat,depths. The grid file is the mesh file populated with material properties.
# Mesh and grid files, format meshfile=/lustre/scratch/user/mesh_cvmh_ijk12_2000m.media gridfile=/lustre/scratch/user/mesh_cvmh_ijk12_2000m.grid
ucvm2mesh-mpi needs to be submitted as a job to the cluster scheduler. In some cases, this can be done on the command line, like this. mpirun -np 20 ucvm2mesh_mpi -f ./ucvm2mesh_example.conf
However, in most cases, the user will create a job description scrip the defines the program, the required input parameters, and other information.
In this example, we assume that ucvmc has been installed on USC HPC disk system visible to the compute nodes. Then, to run on ucvm2mesh-mpi on USC HPC cluster, the user defines two main configuration files.
First, the user defines a specification for the mesh they wish to create. This file defines the CVM or CVMS to use, the mesh size, the number of processors to use, the output file names and directories. A working example file for a 1.7M mesh point mesh is given here:
This file contains important information about the ucvm software, specifically the location of the ucvm.conf file in the ucvmc/installation directory. The ucvm2mesh-mpi binary, and the ucvm.conf file must be visible from the compute nodes.
# List of CVMs to query ucvmlist=cvmh,bbp1d # UCVM conf file ucvmconf=/auto/scec-00/maechlin/t1/conf/ucvm.conf # Gridding cell centered or vertex gridtype=CENTER # Spacing of cells spacing=2000.0 # Projection proj=+proj=utm +datum=WGS84 +zone=11 rot=-40.0 x0=-122.3 y0=34.7835 z0=0.0 # Number of cells along each dim nx=384 ny=248 nz=25 # Partitioning of grid among processors px=2 py=2 pz=5 # Vs/Vp minimum vp_min=0 vs_min=0 # Mesh and grid files, format meshfile=/auto/scec-00/maechlin/ucvm_mpi/mesh_cvmhbbp1d_ijk12_2000m.media gridfile=/auto/scec-00/maechlin/ucvm_mpi/mesh_cvmhbbp1d_ijk12_2000m.grid meshtype=IJK-12 # Location of scratch dir scratch=/scratch
As you can see, this defines specific parameters about the mesh to be created.
USC HPC uses a pbs scheduler, so we create pbs script that will submit a job that runs ucvm2mesh-mpi, and that will pass this configuration file on the command line.
Our ucvm2mesh.pbs file looks like this:
-bash-4.2$ more ucvm2mesh.pbs #!/bin/bash # # This is the pbs script ucvm2mesh # #PBS -l nodes=4:ppn=5 #PBS -l walltime=00:10:00 #PBS -m ae #PBS -M maechlin@usc.edu ##PBS -q main #PBS -o $PBS_JOBID.out #PBS -e $PBS_JOBID.err cd $PBS_O_WORKDIR np=$(cat $PBS_NODEFILE | wc -l) cat $PBS_NODEFILE mpirun -np $np -machinefile $PBS_NODEFILE /auto/scec-00/maechlin/t1/bin/ucvm2mesh-mpi -f /auto/scec-00/maechlin/ucvm_mpi/ucvm2mesh_20cores.conf exit
This job runs on 4 nodes, with 5 processors per node. It outputs the mesh files to a disk visible from the compute nodes. the output files created when this ran were up to 28Mb is size:
-rw-r--r-- 1 maechlin scec 2.2M Feb 2 21:51 mesh_cvmhbbp1d_ijk12_2000m.grid -rw-r--r-- 1 maechlin scec 28M Feb 2 21:52 mesh_cvmhbbp1d_ijk12_2000m.media
In this section we define specific configuration files we used to run on OLCF Titan. These are similar to the files used to run on HPC, with some differences.
# List of CVMs to query ucvmlist=cvmh # UCVM conf file ucvmconf=../conf/kraken/ucvm.conf # Gridding cell centered or vertex gridtype=CENTER # Spacing of cells spacing=2000.0 # Projection proj=+proj=utm +datum=WGS84 +zone=11 rot=-40.0 x0=-122.3 y0=34.7835 z0=0.0 # Number of cells along each dim nx=384 ny=248 nz=25 # Partitioning of grid among processors px=2 py=2 pz=5 # Vs/Vp minimum vp_min=0 vs_min=0 # Mesh and grid files, format meshfile=/lustre/scratch/user/mesh_cvmh_ijk12_2000m.media gridfile=/lustre/scratch/user/mesh_cvmh_ijk12_2000m.grid meshtype=IJK-12 # Location of scratch dir scratch=/lustre/scratch/user/scratch
Older, but more detailed documentation on ucvm2mesh is posted on a SCEC website at: UCVM Documentation
This wiki is licensed by University of Southern California (USC) to the public under a Creative Commons Attribution 4.0 license.