Feature 1926 gridstat openmp (#1977)

Co-authored-by: johnhg <johnhg@ucar.edu>
Co-authored-by: John Halley Gotway <johnhg@ucar.edu>
Co-authored-by: Julie Prestopnik <jpresto@seneca.rap.ucar.edu>

met/configure.ac

@@ -8,6 +8,13 @@ AC_CONFIG_HEADERS([config.h])
 
 AM_INIT_AUTOMAKE([1.9 foreign])
 
+# OpenMP
+
+AC_OPENMP()
+
+CPPFLAGS="${CPPFLAGS} ${OPENMP_CFLAGS}"
+LDFLAGS="${LDFLAGS} ${OPENMP_CFLAGS}"
+
 #
 # Look for the NetCDF library
 #

met/docs/Users_Guide/config_options.rst

@@ -202,13 +202,22 @@ to use one configuration file rather than maintianing many very similar ones.
 An error in the syntax of a configuration file will result in an error from the
 MET tool stating the location of the parsing error.
 
+Runtime Environment Variables
+-----------------------------
+
+MET_BASE
+^^^^^^^^
+
 The MET_BASE variable is defined in the code at compilation time as the path
 to the MET shared data. These are things like the default configuration files,
 common polygons and color scales. MET_BASE may be used in the MET configuration
 files when specifying paths and the appropriate path will be substituted in.
 If MET_BASE is defined as an environment variable, its value will be used
 instead of the one defined at compilation time.
 
+MET_OBS_ERROR_TABLE
+^^^^^^^^^^^^^^^^^^^
+
 The MET_OBS_ERROR_TABLE environment variable can be set to specify the location
 of an ASCII file defining observation error information. The default table can
 be found in the installed *share/met/table_files/obs_error_table.txt*. This
@@ -236,6 +245,9 @@ values and randomly perturbed ensemble member values. Values less than MIN are
 reset to the mimimum value and values greater than MAX are reset to the maximum
 value. A value of NA indicates that the variable is unbounded.
 
+MET_GRIB_TABLES
+^^^^^^^^^^^^^^^
+
 The MET_GRIB_TABLES environment variable can be set to specify the location of
 custom GRIB tables. It can either be set to a specific file name or to a
 directory containing custom GRIB tables files. These file names must begin with
@@ -289,9 +301,96 @@ References:
 | `NCEP WMO GRIB2 Documentation <http://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc>`_
 |
 
+OMP_NUM_THREADS
+^^^^^^^^^^^^^^^
+
+**Introduction**
+
+There are a number of different ways of parallelizing code. OpenMP offers
+parallelism within a single shared-memory workstation or supercomputer node.
+The programmer writes OpenMP directives into the code to parallelize
+particular code regions.
+
+When a parallelized code region is reached, which we shall hereafter call a
+parallel region, a number of threads are spawned and work is shared among them.
+Running on different cores, this reduces the execution time. At the end of the
+parallel region, the code returns to single-thread execution.
+
+A limited number of code regions are parallelized in MET. As a consequence,
+there are limits to the overall speed gains acheivable. Only the parallel
+regions of code will get faster with more threads, leaving the remaining
+serial portions to dominate the runtime.
+
+Not all top-level executables use parallelized code. If OpenMP is available,
+a log message will appear inviting the user to increase the number of threads
+for faster runtimes.
+
+**Setting the number of threads**
+
+The number of threads is controlled by the environment variable
+*OMP_NUM_THREADS* . For example, on a quad core machine, the user might choose
+to run on 4 threads:
+
+.. code :: bash
+
+  export OMP_NUM_THREADS=4
+
+Alternatively, the variable may be specified as a prefix to the executable
+itself. For example:
+
+.. code :: bash
+
+  OMP_NUM_THREADS=4 <exec>
+
+The case where this variable remains unset is handled inside the code, which
+defaults to a single thread.
+
+There are choices when deciding how many threads to use. To perform a single run
+as fast as possible, it would likely be appropriate to use as many threads as
+there are (physical) cores available on the specific system. However, it is not
+a cast-iron guarantee that more threads will always translate into more speed.
+In theory, there is a chance that running across multiple non-uniform memory
+access (NUMA) regions may carry negative performance impacts. This has not been
+observed in practice, however.
+
+A lower thread count is appropriate when time-to-solution is not so critical,
+because cores remain idle when the code is not inside a parallel region. Fewer
+threads typically means better resource utilization.
+
+**Which code is parallelized?**
+
+Regions of parallelized code are:
+
+  * :code:`fractional_coverage (data_plane_util.cc)`
+
+Only the following top-level executables can presently benefit from OpenMP
+parallelization:
+
+  * :code:`grid_stat`
+  * :code:`ensemble_stat`
+  * :code:`grid_ens_prod`
+
+**Thread Binding**    
+
+It is normally beneficial to bind threads to particular cores, sometimes called
+*affinitization*. There are a few reasons for this, but at the very least it
+guarantees that threads remain evenly distributed across the available cores.
+Otherwise, the operating system may migrate threads between cores during a run.
+
+OpenMP provides some environment variables to handle this: :code:`OMP_PLACES`
+and  :code:`OMP_PROC_BIND`.  We anticipate that the effect of setting only
+:code:`OMP_PROC_BIND=true` would be neutral-to-positive.
+
+However, there are sometimes compiler-specific environment variables. Instead,
+thread affinitization is sometimes handled by MPI launchers, since OpenMP is
+often used in MPI codes to reduce intra-node communications.
+
+Where code is running in a production context, it is worth being familiar with
+the binding / affinitization method on the particular system and building it
+into any relevant scripting.
 
 Settings common to multiple tools
-_________________________________
+---------------------------------
 
 .. _exit_on_warning:
 
@@ -2190,10 +2289,10 @@ are empty. Note: grib_code 11 is equivalent to obs_var "TMP".
   }
 
 Settings specific to individual tools
-_____________________________________
+-------------------------------------
 
 EnsembleStatConfig_default
-~~~~~~~~~~~~~~~~~~~~~~~~~~
+^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 .. _ens:
 
@@ -2505,7 +2604,7 @@ used for random assignment of ranks when they are tied.
   }
 
 MODEAnalysisConfig_default
-~~~~~~~~~~~~~~~~~~~~~~~~~~
+^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 MODE line options are used to create filters that determine which MODE output
 lines are read in and processed. The MODE line options are numerous. They
@@ -2843,7 +2942,7 @@ MET User's Guide for a description of these attributes.
 
 
 MODEConfig_default
-~~~~~~~~~~~~~~~~~~
+^^^^^^^^^^^^^^^^^^
 
 .. _quilt:
 
@@ -3158,7 +3257,7 @@ much more flexible "regrid" option may be used instead.
   shift_right = 0;
 
 PB2NCConfig_default
-~~~~~~~~~~~~~~~~~~~
+^^^^^^^^^^^^^^^^^^^
 
 The PB2NC tool filters out observations from PREPBUFR or BUFR files using the
 following criteria:
@@ -3484,7 +3583,7 @@ stack (most quality controlled) or the bottom of the event stack (most raw).
   event_stack_flag = TOP;
 
 SeriesAnalysisConfig_default
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 .. _block_size:
 
@@ -3539,7 +3638,7 @@ grid is large.
   }
 
 STATAnalysisConfig_default
-~~~~~~~~~~~~~~~~~~~~~~~~~~
+^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 .. _jobs:
 
@@ -4008,7 +4107,7 @@ confidence intervals computed for the aggregated statistics.
   vif_flag = FALSE;
 
 WaveletStatConfig_default
-~~~~~~~~~~~~~~~~~~~~~~~~~
+^^^^^^^^^^^^^^^^^^^^^^^^^
 
 .. _grid_decomp_flag:
 
@@ -4099,7 +4198,7 @@ similar to the "fcst_raw_plot" described in the "Settings common to multiple
 tools" section.
 
 WWMCARegridConfig_default
-~~~~~~~~~~~~~~~~~~~~~~~~~
+^^^^^^^^^^^^^^^^^^^^^^^^^
 
 .. _to_grid:
 

met/docs/Users_Guide/installation.rst

@@ -290,6 +290,14 @@ Enable compilation of the MODE-Graphics tool. Requires $MET_CAIRO and $MET_FREET
 
 Disable use of BLOCK4 in the compilation. Use this if you have trouble using PrepBUFR files.
 
+.. code-block:: none
+
+  --disable-openmp
+
+Disable compilation of OpenMP directives within the code which allows some code
+regions to benefit from thread-parallel execution. Runtime environment variable
+:code:`OMP_NUM_THREADS` controls the number of threads.
+
 Run the configure script with the **-help** argument to see the full list of configuration options.
 
 Make Targets

met/src/basic/vx_util/Makefile.am

@@ -69,6 +69,7 @@ libvx_util_a_SOURCES = ascii_table.cc ascii_table.h \
                GridOffset.h GridOffset.cc \
                observation.h observation.cc \
                stat_column_defs.h \
+               handle_openmp.h handle_openmp.cc \
                RectangularTemplate.h RectangularTemplate.cc $(OPT_PYTHON_SOURCES)
 
 libvx_util_a_CPPFLAGS = ${MET_CPPFLAGS}