Update develop-ref after #2371, #2372, and #2373 (#2376)

Co-authored-by: Howard Soh <hsoh@seneca.rap.ucar.edu> Co-authored-by: Dave Albo <dave@seneca.rap.ucar.edu> Co-authored-by: John Halley Gotway <johnhg@ucar.edu> Co-authored-by: Seth Linden <linden@seneca.rap.ucar.edu> Co-authored-by: johnhg <johnhg@ucar.edu> Co-authored-by: Lisa Goodrich <lisag@seneca.rap.ucar.edu> Co-authored-by: jprestop <jpresto@ucar.edu> Co-authored-by: MET Tools Test Account <met_test@seneca.rap.ucar.edu> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> Co-authored-by: Julie Prestopnik <jpresto@seneca.rap.ucar.edu> Co-authored-by: Jonathan Vigh <jvigh@ucar.edu> Co-authored-by: hsoh-u <hsoh@ucar.edu> Co-authored-by: bikegeek <3753118+bikegeek@users.noreply.github.com> Co-authored-by: davidalbo <dave@ucar.edu> Co-authored-by: Seth Linden <linden@ucar.edu> Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu>
dtcenter · Dec 12, 2022 · c0f28f7 · c0f28f7
1 parent e19445f
commit c0f28f7
Show file tree

Hide file tree

Showing 28 changed files with 791 additions and 461 deletions.
diff --git a/.github/workflows/documentation.yml b/.github/workflows/documentation.yml
@@ -21,7 +21,7 @@ jobs:
       - uses: actions/checkout@v3
       - uses: actions/setup-python@v4
         with:
-          python-version: '3.6'
+          python-version: '3.8'
       - name: Install dependencies
         run: |
           python -m pip install --upgrade python-dateutil requests sphinx \

diff --git a/docs/Users_Guide/appendixC.rst b/docs/Users_Guide/appendixC.rst
@@ -382,6 +382,29 @@ For cost / loss ratio above the base rate, the ECLV is defined as:
 
 .. math:: \text{ECLV } = \frac{(cl \ast (h + f)) + m - b}{b \ast (cl - 1)}.
 
+Stable Equitable Error in Probability Space (SEEPS)
+---------------------------------------------------
+
+Included in SEEPS output :numref:`table_PS_format_info_SEEPS` and SEEPS_MPR output :numref:`table_PS_format_info_SEEPS_MPR`
+
+The SEEPS scoring matrix (equation 15 from :ref:`Rodwell et al, 2010 <Rodwell-2010>`) is:
+
+.. math:: \{S^{S}_{vf}\} = \frac{1}{2}
+          \begin{Bmatrix}
+             0 & \frac{1}{1-p_1} & \frac{1}{p_3} + \frac{1}{1-p_1}\\
+             \frac{1}{p_1} & 0 & \frac{1}{p_3}\\
+             \frac{1}{p_1} + \frac{1}{1-p_3} & \frac{1}{1-p_3} & 0
+          \end{Bmatrix}
+
+In addition, Rodwell et al (2011) note that SEEPS can be written as the mean of two 2-category scores that individually assess the dry/light and light/heavy thresholds (:ref:`Rodwell et al., 2011 <Rodwell-2011>`). Each of these scores is like 1 – HK, but written as:
+
+.. math:: \frac{n_{01}}{\text{Expected n}_{.1}} + \frac{n_{10}}{\text{Expected n}_{.0}}
+
+
+where the word expected refers to the mean value deduced from the climatology, rather than the sample mean.
+
+SEEPS scores are expected to lie between 0 and 1, with a perfect forecast having a value of 0. Individual values can be much larger than 1. Results can be presented as a skill score by using the value of 1 – SEEPS.
+
 MET verification measures for continuous variables
 ==================================================
 

diff --git a/docs/Users_Guide/config_options.rst b/docs/Users_Guide/config_options.rst
@@ -240,6 +240,7 @@ Referencing that environment variable inside a MET configuration file:
 In addition to supporting user-specified environment variables within configuration
 files, the environment variables listed below have special meaning if set at runtime.
 
+.. _met_airnow_stations:
 
 MET_AIRNOW_STATIONS
 ^^^^^^^^^^^^^^^^^^^
@@ -249,7 +250,17 @@ will override the default file. If set, it should be the full path to the file.
 The default table can be found in the installed
 *share/met/table_files/airnow_monitoring_site_locations_v2.dat*. This file contains
 ascii column data that allows lookups of latitude, longitude, and elevation for all
-airnow stations based on stationId and/or AqSid.
+AirNow stations based on stationId and/or AqSid.
+
+Additional information and updated site locations can be found at the
+`EPA AirNow website <https://www.airnow.gov>`_. While some monitoring stations are
+permanent, others are temporary, and theirs locations can change. When running the
+ascii2nc tool with the `-format airnowhourly` option, users should
+`download <https://test.airnowtech.org/>`_ the `Monitoring_Site_Locations_V2.dat` data file
+data file corresponding to the date being processed and set the MET_AIRNOW_STATIONS
+envrionment variable to define its location.
+
+.. _met_ndbc_stations:
 
 MET_NDBC_STATIONS
 ^^^^^^^^^^^^^^^^^

diff --git a/docs/Users_Guide/grid-stat.rst b/docs/Users_Guide/grid-stat.rst
@@ -71,6 +71,15 @@ There are several ways to present the results of the neighborhood approaches, su
 
 The user must specify several parameters in the grid_stat configuration file to utilize the neighborhood approach, such as the interpolation method, size of the smoothing window, and required fraction of valid data points within the smoothing window. For FSS-specific results, the user must specify the size of the neighborhood window, the required fraction of valid data points within the window, and the fractional coverage threshold from which the contingency tables are defined. These parameters are described further in the practical information section below.
 
+.. _grid-stat_seeps:
+
+SEEPS scores
+------------
+
+The Stable Equitable Error in Probability Space (SEEPS) was devised for monitoring global deterministic forecasts of precipitation against the WMO gauge network (:ref:`Rodwell et al., 2010 <Rodwell-2010>`; :ref:`Haiden et al., 2012 <Haiden-2012>`) and is a multi-category score which uses a climatology to account for local variations in behavior. Please see Point-Stat documentation :numref:`PS_seeps` for more details.
+
+The capability to calculate the SEEPS has also been added to Grid-Stat. This follows the method described in :ref:`North et al, 2022 <North-2022>`, which uses the TRMM 3B42 v7 gridded satellite product for the climatological values and interpolates the forecast and observed products onto this grid for evaluation. A 24-hour TRMM climatology (valid at 00 UTC) constructed from data over the time period 1998-2015 is supplied with the release. Expansion of the capability to other fields will occur as well vetted examples and funding allow.
+
 Fourier Decomposition
 ---------------------
 

diff --git a/docs/Users_Guide/point-stat.rst b/docs/Users_Guide/point-stat.rst
@@ -136,6 +136,17 @@ The HiRA framework provides a unique method for evaluating models in the neighbo
 
 Often, the neighborhood size is chosen so that multiple models to be compared have approximately the same horizontal resolution. Then, standard metrics for probabilistic forecasts, such as Brier Score, can be used to compare those forecasts. HiRA was developed using surface observation stations so the neighborhood lies completely within the horizontal plane. With any type of upper air observation, the vertical neighborhood must also be defined.
 
+.. _PS_seeps:
+
+SEEPS scores
+------------
+
+The Stable Equitable Error in Probability Space (SEEPS) was devised for monitoring global deterministic forecasts of precipitation against the WMO gauge network (:ref:`Rodwell et al., 2010 <Rodwell-2010>`; :ref:`Haiden et al., 2012 <Haiden-2012>`) and is a multi-category score which uses a climatology to account for local variations in behavior. Since the score uses probability space to define categories using the climatology, it can be aggregated over heterogeneous climate regions. Even though it was developed for use with precipitation forecasts, in principle it could be applied to any forecast parameter for which a sufficiently long time period of observations exists to create a suitable climatology. The computation of SEEPS for precipitation is only supported for now.
+
+For use with precipitation, three categories are used, named ‘dry’, ‘light’ and ‘heavy’. The ‘dry’ category is defined (using the WMO observing guidelines) with any accumulation (rounded to the nearest 0.1 millimeter) that is less than or equal to 0.2 mm.  The remaining precipitation is divided into ‘light’ and ‘heavy’ categories whose thresholds are with respect to a climatology and thus location specific. The light precipitation is defined to occur twice as often as heavy precipitation.
+
+When calculating a single SEEPS value over observing stations for a particular region, the scores should have a density weighting applied which accounts for uneven station distribution in the region of interest (see Section 9.1 in :ref:`Rodwell et al., 2010 <Rodwell-2010>`). This density weighting has not yet been implemented in MET. Global precipitation climatologies calculated from the WMO SYNOP records from 1980-2009 are supplied with the release. At the moment, a 24-hour climatology is available (valid at 00 UTC or 12 UTC), but in future a 6-hour climatology will become available.
+
 .. _PS_Statistical-measures:
 
 Statistical measures
@@ -487,7 +498,7 @@ Note that writing out matched pair data (MPR lines) for a large number of cases
 
 If all line types corresponding to a particular verification method are set to NONE, the computation of those statistics will be skipped in the code and thus make the Point-Stat tool run more efficiently. For example, if FHO, CTC, and CTS are all set to NONE, the Point-Stat tool will skip the categorical verification step.
 
-The default SEEPS climo file exists at MET_BASE/share/met/climo/seeps/PPT24_seepsweights.nc. It can be overridden by using the environment variable, MET_SEEPS_POINT_CLIMO_NAME.
+The default SEEPS climo file exists at MET_BASE/climo/seeps/PPT24_seepsweights.nc. It can be overridden by using the environment variable, MET_SEEPS_POINT_CLIMO_NAME.
 
 .. _point_stat-output:
 

diff --git a/docs/Users_Guide/reformat_point.rst b/docs/Users_Guide/reformat_point.rst
@@ -451,9 +451,9 @@ While initial versions of the ASCII2NC tool only supported a simple 11 column AS
 
 • Western Wind and Solar Integration Study (WWSIS) format. WWSIS data are available by request from National Renewable Energy Laboratory (NREL) in Boulder, CO. 
 
-• `AirNow DailyData_v2, AirNow HourlyData, and AirNow HourlyAQObs formats <https://www.epa.gov/outdoor-air-quality-data>`_
+• `AirNow DailyData_v2, AirNow HourlyData, and AirNow HourlyAQObs formats <https://www.epa.gov/outdoor-air-quality-data>`_. See the :ref:`MET_AIRNOW_STATIONS` environment variable.
 
-• `National Data Buoy (NDBC) Standard Meteorlogical Data format <https://www.ndbc.noaa.gov/measdes.shtml>`_
+• `National Data Buoy (NDBC) Standard Meteorlogical Data format <https://www.ndbc.noaa.gov/measdes.shtml>`_. See the :ref:`MET_NDBC_STATIONS` environment variable.
 
 • `AErosol RObotic NEtwork (AERONET) versions 2 and 3 format <http://aeronet.gsfc.nasa.gov/>`_
 
@@ -1191,10 +1191,10 @@ The script can be found at:
 
 .. code-block:: none
 
-   MET_BASE/shared/met/utility/print_pointnc2ascii.py
+   MET_BASE/utility/print_pointnc2ascii.py
 
 For how to use the script, issue the command:
 
 .. code-block:: none
 
-   python3 MET_BASE/shared/met/utility/print_pointnc2ascii.py -h
+   python3 MET_BASE/utility/print_pointnc2ascii.py -h
diff --git a/docs/Users_Guide/refs.rst b/docs/Users_Guide/refs.rst
@@ -7,14 +7,14 @@ References
 .. _Aberson-1998:
 
 | Aberson, S.D., 1998: Five-day Tropical cyclone track forecasts in the North
-|   Atlantic Basin. *Weather & Forecasting*,  13, 1005-1015.
+|   Atlantic Basin. *Weather and Forecasting*,  13, 1005-1015.
 |
 
 .. _Ahijevych-2009:
 
-| Ahijevych, D., E. Gilleland, B.G. Brown, and E.E. Ebert, 2009. Application of
+| Ahijevych, D., E. Gilleland, B.G. Brown, and E.E. Ebert, 2009: Application of
 |   spatial verification methods to idealized and NWP-gridded precipitation forecasts.
-|   Weather Forecast., 24 (6), 1485 - 1497, doi: 10.1175/2009WAF2222298.1.
+|   *Weather and Forecasting*, 24 (6), 1485 - 1497, doi: 10.1175/2009WAF2222298.1.
 |
 
 .. _Barker-1991:
@@ -127,31 +127,30 @@ References
 
 .. _Gilleland-2017:
 
-| Gilleland, E., 2017. A new characterization in the spatial verification
+| Gilleland, E., 2017: A new characterization in the spatial verification
 |   framework for false alarms, misses, and overall patterns.
-|   Weather Forecast., 32 (1), 187 - 198, doi: 10.1175/WAF-D-16-0134.1.
+|   *Weather and Forecasting*, 32 (1), 187 - 198, doi: 10.1175/WAF-D-16-0134.1.
 |
 
-
 .. _Gilleland_PartI-2020:
 
-| Gilleland, E., 2020. Bootstrap methods for statistical inference.
+| Gilleland, E., 2020: Bootstrap methods for statistical inference.
 |   Part I: Comparative forecast verification for continuous variables.
-|   Journal of Atmospheric and Oceanic Technology, 37 (11), 2117 - 2134,
+|   *Journal of Atmospheric and Oceanic Technology*, 37 (11), 2117 - 2134,
 |   doi: 10.1175/JTECH-D-20-0069.1.
 |
 
 .. _Gilleland_PartII-2020:
 
-| Gilleland, E., 2020. Bootstrap methods for statistical inference.
-|   Part II: Extreme-value analysis. Journal of Atmospheric and Oceanic
-|   Technology, 37 (11), 2135 - 2144, doi: 10.1175/JTECH-D-20-0070.1.
+| Gilleland, E., 2020: Bootstrap methods for statistical inference.
+|   Part II: Extreme-value analysis. *Journal of Atmospheric and Oceanic*
+|   *Technology*, 37 (11), 2135 - 2144, doi: 10.1175/JTECH-D-20-0070.1.
 |
 
 .. _Gilleland-2021:
 
-| Gilleland, E., 2021. Novel measures for summarizing high-resolution forecast
-|   performance. Advances in Statistical Climatology, Meteorology and Oceanography,
+| Gilleland, E., 2021: Novel measures for summarizing high-resolution forecast
+|   performance. *Advances in Statistical Climatology, Meteorology and Oceanography*,
 |   7 (1), 13 - 34, doi: 10.5194/ascmo-7-13-2021.
 |
 
@@ -164,6 +163,14 @@ References
 |   http://www.stat.washington.edu/www/research/reports/
 | 
 
+.. _Haiden-2012:
+
+| Haiden, T., M.J. Rodwell, D.S. Richardson, A. Okagaki, T. Robinson, T. Hewson, 2012:
+|   Intercomparison of Global Model Precipitation Forecast Skill in 2010/11
+|   Using the SEEPS Score. *Monthly Weather Review*, 140, 2720-2733.
+|   https://doi.org/10.1175/MWR-D-11-00301.1
+|
+
 .. _Hamill-2001:
 
 | Hamill, T. M., 2001: Interpretation of rank histograms for verifying ensemble
@@ -186,7 +193,7 @@ References
 
 | Knaff, J.A., M. DeMaria, C.R. Sampson, and J.M. Gross, 2003: Statistical,
 |   Five-Day Tropical Cyclone Intensity Forecasts Derived from Climatology
-|   and Persistence. *Weather & Forecasting,* Vol. 18 Issue 2, p. 80-92.
+|   and Persistence. *Weather and Forecasting,* Vol. 18 Issue 2, p. 80-92.
 | 
 
 .. _Mason-2004:
@@ -228,20 +235,44 @@ References
 |   verification. *Monthly Weather Review*, 115, 1330-1338.
 | 
 
+
+.. _North-2022:
+
+| North, R.C.,  M.P. Mittermaier, S.F. Milton, 2022. *Using SEEPS with a*
+|   TRMM-derived Climatology to Assess Global NWP Precipitation Forecast Skill.
+|   *Monthly Weather Review*, 150, 135-155.
+|   https://doi.org/10.1175/MWR-D-20-0347.1
+|
+
 .. _Ou-2016:
 
 | Ou, M. H., Charles, M., & Collins, D. C. 2016: Sensitivity of calibrated week-2
 |   probabilistic forecast skill to reforecast sampling of the NCEP global
-|   ensemble forecast system. *Weather and Forecasting,* 31(4), 1093-1107.
+|   ensemble forecast system. *Weather and Forecasting*, 31(4), 1093-1107.
 |
 
 .. _Roberts-2008:
 
 | Roberts, N.M., and H.W. Lean, 2008: Scale-selective verification of rainfall
 |   accumulations from high-resolution forecasts of convective events.
-|   *Monthly Weather Review,* 136, 78-97.
+|   *Monthly Weather Review*, 136, 78-97.
 | 
 
+.. _Rodwell-2010:
+
+| Rodwell, M.J., D.S. Richardson, T.D. Hewson and T. Haiden, 2010: A new equitable
+|   score suitable for verifying precipitation in numerical weather prediction.
+|   *Quarterly Journal of the Royal Meteorological Society*, 136: 1344-1463.
+|   https://doi.org/10.1002/qj.656
+|
+
+.. _Rodwell-2011:
+
+| Rodwell, M.J., T. Haiden, D.S. Richardson, 2011: Developments in Precipitation
+|   Verification. *ECMWF Newsletter* Number 128.
+|   https://www.ecmwf.int/node/14595
+|
+
 .. _Saetra-2004:
 
 | Saetra O., H. Hersbach, J-R Bidlot, D. Richardson, 2004: Effects of
@@ -279,22 +310,22 @@ References
 .. _Todter-2012:
 
 | Tödter, J. and B. Ahrens, 2012: Generalization of the Ignorance Score:
-|   Continuous ranked version and its decomposition. *Mon. Wea. Rev.*,
+|   Continuous ranked version and its decomposition. *Monthly Weather Review*,
 |   140 (6), 2005 - 2017, doi: 10.1175/MWR-D-11-00266.1.
 |
 
 .. _Weniger-2016:
 
 | Weniger, M., F. Kapp, and P. Friederichs, 2016: Spatial Verification Using
 |   Wavelet Transforms: A Review. *Quarterly Journal of the Royal*
-|   *Meteorological Society,* 143, 120-136.
+|   *Meteorological Society*, 143, 120-136.
 | 
 
 .. _Wilks-2010:
 
 | Wilks, D.S. 2010: Sampling distributions of the Brier score and Brier skill
 |   score under serial dependence. *Quarterly Journal of the Royal*
-|   *Meteorological Society,*, 136, 2109-2118. doi:10.1002/qj.709
+|   *Meteorological Society*, 136, 2109-2118. doi:10.1002/qj.709
 | 
 
 .. _Wilks-2011: