Skip to content

Commit

Permalink
Per #1673, making changes requested by Eric.
Browse files Browse the repository at this point in the history
  • Loading branch information
JohnHalleyGotway committed Aug 2, 2021
1 parent 92168b8 commit e64b74e
Show file tree
Hide file tree
Showing 2 changed files with 16 additions and 5 deletions.
7 changes: 6 additions & 1 deletion met/docs/Users_Guide/appendixC.rst
Original file line number Diff line number Diff line change
Expand Up @@ -1178,10 +1178,13 @@ where MED *(A,B)* is as in the Mean-error distance, *N* is the total number of g

The range for ZHU is 0 to infinity, with a score of 0 indicating a perfect forecast.

.. _App_C-gbeta:

:math:`G` and :math:`G_\beta`
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Called "G" and "GBETA" in the DMAP output :numref:`table_GS_format_info_DMAP`
See :numref:`grid-stat_gbeta` for a description of these statistics.

Let :math:`y = {y_1}{y_2}` where :math:`y_1 = n_A + n_B - 2n_{AB}`, and :math:`y_2 = MED(A,B) \cdot n_B + MED(B,A) \cdot n_A`, with the mean-error distance (:math:`MED`) as described above, and where :math:`n_{A}`, :math:`n_{B}`, and :math:`n_{AB}` are the number of events within event areas *A*, *B*, and the intersection of *A* and *B*, respectively.

Expand All @@ -1193,7 +1196,9 @@ and the :math:`G_\beta` performance measure is given by

.. math:: G_\beta(A,B) = max\{1-\frac{y}{\beta}, 0\}

where :math:`\beta > 0` is a user-chosen parameter with a default value of :math:`n^2 / 2.0` with :math:`n` equal to the number of points in the domain.
where :math:`\beta > 0` is a user-chosen parameter with a default value of :math:`n^2 / 2.0` with :math:`n` equal to the number of points in the domain. The square-root of :math:`G` will give units of grid points, where :math:`y^{1/3}` gives units of grid points squared.

The range for :math:`G_\beta` is 0 to 1, with a score of 1 indicating a perfect forecast.

Calculating Percentiles
_______________________
Expand Down
14 changes: 10 additions & 4 deletions met/docs/Users_Guide/grid-stat.rst
Original file line number Diff line number Diff line change
Expand Up @@ -122,18 +122,24 @@ While :numref:`grid-stat_fig1` and :numref:`grid-stat_fig2` are helpful in illus

The statistics derived from these distance maps are described in :numref:`Appendix C, Section %s <App_C-distance_maps>`. For each combination of input field and categorical threshold requested in the configuration file, Grid-Stat applies that threshold to define events in the forecast and observation fields and computes distance maps for those binary fields. Statistics for all requested masking regions are derived from those distance maps. Note that the distance maps are computed only once over the full verification domain, not separately for each masking region. Events occurring outside of a masking region can affect the distance map values inside that masking region and, therefore, can also affect the distance maps statistics for that region.

Selecting :math:`\beta` for :math:`G_\beta`
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. _grid-stat_gbeta:

The :math:`G_\beta` statistic provides a summary measure of forecast quality for each user-defined threshold chosen. Its value ranges from 0 to 1, with 1 being a perfect score. It is sensitive to the choice of :math:`\beta`, which depends on the (i) specific domain, (ii) variable, and (iii) user’s needs. Smaller values make :math:`G_\beta` more stringent and larger values make it more lenient. :numref:`grid-stat_fig6` shows an example of applying :math:`G_\beta` over a range of thresholds to a precipitation verification set where the binary fields are created by applying a threshold of :math:`2.1 mmh^{-1}`. Color choice and human bias can make it difficult to determine the quality of the forecast for a human observer looking at the raw images in the top row of the figure (:ref:`Ahijevych et al., 2009 <Ahijevych-2009>`). The bottom left panel of the figure displays the differences in their binary fields, which highlights that the forecast captured the overall shapes of the observed rain areas but suffers from a spatial displacement error (perhaps really a timing error).
:math:`\beta` and :math:`G_\beta`
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

See :numref:`App_C-gbeta` for the equations for :math:`G` and :math:`G_\beta`.

:math:`G_\beta` provides a summary measure of forecast quality for each user-defined threshold chosen. It falls into a range from zero to one where one is a perfect forecast and zero is considered to be a very poor forecast as determined by the user through the value of :math:`\beta`. Values of :math:`G_\beta` closer to one represent better forecasts and worse forecasts as it decreases toward zero. Although a particular value cannot be universally compared against any forecast, when applied with the same choice of :math:`\beta` for the same variable and on the same domain, it is highly effective at ranking such forecasts.

:math:`G_\beta` is sensitive to the choice of :math:`\beta`, which depends on the (i) specific domain, (ii) variable, and (iii) user’s needs. Smaller values make :math:`G_\beta` more stringent and larger values make it more lenient. :numref:`grid-stat_fig6` shows an example of applying :math:`G_\beta` over a range of :math:`\beta` values to a precipitation verification set where the binary fields are created by applying a threshold of :math:`2.1 mmh^{-1}`. Color choice and human bias can make it difficult to determine the quality of the forecast for a human observer looking at the raw images in the top row of the figure (:ref:`Ahijevych et al., 2009 <Ahijevych-2009>`). The bottom left panel of the figure displays the differences in their binary fields, which highlights that the forecast captured the overall shapes of the observed rain areas but suffers from a spatial displacement error (perhaps really a timing error).

Whether or not the forecast from :numref:`grid-stat_fig6` is “good” or not depends on the specific user. Is it sufficient that the forecast came as close as it did to the observation field? If the answer is yes for the user, then a higher choice of :math:`\beta`, such as :math:`N/2`, with :math:`N` equal to the number of points in the domain, will correctly inform this user that it is a “good” forecast as it will lead to a :math:`G_\beta` value near one. If the user requires the forecast to be much better aligned spatially with the observation field, then a lower choice, perhaps :math:`\beta = N`, will correctly inform that the forecast suffers from spatial displacement errors that are too large for this user to be pleased. If the goal is to rank a series of ensemble forecasts, for example, then a choice of :math:`\beta` that falls in the steep part of the curve shown in the lower right panel of the figure should be preferred, say somewhere between :math:`\beta = N` and :math:`\beta = N^2/2`. Such a choice will ensure that each member is differentiated by the measure.

.. _grid-stat_fig6:

.. figure:: figure/grid-stat_fig6.png

Top left is an example of an accumulated precipitation (mm/h) forecast with the corresponding observed field on the top right. Bottom left shows the difference in binary fields, where the binary fields are created by setting all values in the original fields that fall above :math:`2.1 mmh^{-1}` to one and the rest to zero. Bottom right shows the results for :math:`G_\beta` over a range of choices for :math:`\beta`.
Top left is an example of an accumulated precipitation (mm/h) forecast with the corresponding observed field on the top right. Bottom left shows the difference in binary fields, where the binary fields are created by setting all values in the original fields that fall above :math:`2.1 mmh^{-1}` to one and the rest to zero. Bottom right shows the results for :math:`G_\beta` calculated on the binary fields using the threshold of :math:`2.1 mmh^{-1}` over a range of choices for :math:`\beta`.

In some cases, a user may be interested in a much higher threshold than :math:`2.1 mmh^{-1}` of the above example. :ref:`Gilleland, 2021 (Fig. 4) <Gilleland-2021>`, for example, shows this same forecast using a threshold of :math:`40 mmh^{-1}`. Only a small area in Mississippi has such extreme rain predicted at this valid time; yet none was observed. Small spatial areas of extreme rain in the observed field, however, did occur in a location far away from Mississippi that was not predicted. Generally, for this type of verification, the Hausdorff metric is a good choice of measure. However, a small choice of :math:`\beta` will provide similar results as the Hausdorff distance (:ref:`Gilleland, 2021 <Gilleland-2021>`). The user should think about the average size of storm areas and multiply this value by the displacement distance they are comfortable with in order to get a good initial choice for :math:`\beta`, and may have to increase or decrease its value by trial-and-error using one or two example cases from their verification set.

Expand Down

0 comments on commit e64b74e

Please sign in to comment.