From 893d98783d33dbce2e8cda7815133957ec0e46e0 Mon Sep 17 00:00:00 2001 From: George McCabe <23407799+georgemccabe@users.noreply.github.com> Date: Fri, 18 Nov 2022 13:35:38 -0700 Subject: [PATCH 01/12] per #1835, add documentation to describe how to use PCPCombine wrapper, including how to use [FCST/OBS]_PCP_COMBINE_CONSTANT_INIT to find files with the same initialization time --- docs/Users_Guide/wrappers.rst | 55 +++++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+) diff --git a/docs/Users_Guide/wrappers.rst b/docs/Users_Guide/wrappers.rst index 5be9555a3e..db6270afb3 100644 --- a/docs/Users_Guide/wrappers.rst +++ b/docs/Users_Guide/wrappers.rst @@ -5123,6 +5123,61 @@ The PCPCombine wrapper is a Python script that encapsulates the MET PCPCombine tool. It provides the infrastructure to combine or extract from files to build desired accumulations. +PCPCombine wrapper can be configured to process forecast and/or observation +data. Setting :term:`FCST_PCP_COMBINE_RUN` = True will process forecast data +and setting :term:`OBS_PCP_COMBINE_RUN` = True will process observation data. + +PCPCombine wrapper can be configured to build a command for sum, add, subtract, +and derive methods using :term:`FCST_PCP_COMBINE_METHOD` and/or +:term:`OBS_PCP_COMBINE_METHOD`. Each method executes logic to gather the +desired input files to build the command based on specific examples. + +Accumulations +^^^^^^^^^^^^^ + +The desired accumulation to build is defined using +:term:`FCST_PCP_COMBINE_OUTPUT_ACCUM` or :term:`OBS_PCP_COMBINE_OUTPUT_ACCUM`. +The default units are hours unless otherwise specified. +The output field name can be set explicitly using +:term:`FCST_PCP_COMBINE_OUTPUT_NAME` or :term:`OBS_PCP_COMBINE_OUTPUT_NAME`. + +For ADD and DERIVE methods, the input accumulation(s) can be specified using +:term:`FCST_PCP_COMBINE_INPUT_ACCUMS` or :term:`OBS_PCP_COMBINE_INPUT_ACCUMS`. +The default units are hours unless otherwise specified. +This can be a list of accumulation amounts in order of preference. +If the remaining accumulation needed to build the desired accumulation is +less than the first accumulation, then the next value in the list will be used. +The name and level of the field to read for each input accumulation can be +specified with +:term:`FCST_PCP_COMBINE_INPUT_NAMES`/:term:`FCST_PCP_COMBINE_INPUT_LEVELS` or +:term:`OBS_PCP_COMBINE_INPUT_NAMES`/:term:`OBS_PCP_COMBINE_INPUT_LEVELS`. +These lists must be the same length as +:term:`FCST_PCP_COMBINE_INPUT_ACCUMS` or :term:`OBS_PCP_COMBINE_INPUT_ACCUMS`. + +Constant Initialization Time +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +For ADD and DERIVE methods, :term:`FCST_PCP_COMBINE_CONSTANT_INIT` or +:term:`OBS_PCP_COMBINE_CONSTANT_INIT` can be set to **True** to gather input +files that all contain the same initialization time. + + +User-Defined Commands +^^^^^^^^^^^^^^^^^^^^^ + +There are many ways to utilize PCPCombine that may not align with the logic +used to gather files. If this is the case, then the method can be set to +**USER_DEFINED** and the explicit command arguments can be specified using +:term:`FCST_PCP_COMBINE_COMMAND` or :term:`OBS_PCP_COMBINE_COMMAND`. +Other METplus configuration variables and filename template tags can be +referenced in the explicit command. Note that the path to the pcp_combine +executable and the output path should not be included in the command value. +The output path is controlled by +:term:`FCST_PCP_COMBINE_INPUT_TEMPLATE`/:term:`FCST_PCP_COMBINE_INPUT_DIR` or +:term:`OBS_PCP_COMBINE_INPUT_TEMPLATE`/:term:`OBS_PCP_COMBINE_INPUT_DIR` and +will automatically be added to the end of the command. + + METplus Configuration --------------------- From b20f63cca5f8f1598d61c51425879663b734483a Mon Sep 17 00:00:00 2001 From: George McCabe <23407799+georgemccabe@users.noreply.github.com> Date: Fri, 18 Nov 2022 14:12:53 -0700 Subject: [PATCH 02/12] per #1783, add documentation to describe how to specify multiple files to read --- docs/Users_Guide/systemconfiguration.rst | 96 ++++++++++++++++++++---- 1 file changed, 82 insertions(+), 14 deletions(-) diff --git a/docs/Users_Guide/systemconfiguration.rst b/docs/Users_Guide/systemconfiguration.rst index 17a164b051..c80edeee5e 100644 --- a/docs/Users_Guide/systemconfiguration.rst +++ b/docs/Users_Guide/systemconfiguration.rst @@ -1640,10 +1640,79 @@ and will substitute the current day into the template, giving a filename of a file time of 20190131_23Z, so the filename will be 20190131.ext that is generated by the template. -Using Windows to find Valid Files -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -The [FCST/OBS]_FILE_WINDOW_[BEGIN/END] configuration variables can be used +Multiple Input Files +^^^^^^^^^^^^^^^^^^^^ + +If a tool supports reading multiple files for a given input, then there are +a variety of ways to configure the METplus wrappers to read them. + +* :ref:`multiple-files-wildcard ` +* :ref:`multiple-files-list ` +* :ref:`multiple-files-window ` + +.. _multiple-files-wildcard: + +Using Wildcards to find Multiple Files +"""""""""""""""""""""""""""""""""""""" + +Wildcard characters can be used in filename template variables. +The \* character is used to match 1 or more characters and +the \? character is used to match a single character. + +For example, if a directory /my/files contains the following files: + +* filename_AAA.nc +* filename_ABA.nc +* filename_BBB.nc +* filename.nc + +The following template will match +filename_AAA.nc, filename_ABA.nc and filename_BBB.nc, but not filename.nc:: + + INPUT_TEMPLATE = /my/files/filename_*.nc + +The following template will match filename_AAA.nc and filename_ABA.nc:: + + INPUT_TEMPLATE = /my/files/filename_A?A.nc + + +.. _multiple-files-list: + +Using a List of Templates to find Multiple Files +"""""""""""""""""""""""""""""""""""""""""""""""" + +A comma-separated list of templates can be specified in a \_TEMPLATE variable. +Each value in the list will be added to the corresponding \_DIR variable. + +For example, if a directory /my/files contains the following files: + +* filename_AAA.nc +* filename_ABA.nc +* filename_BBB.nc +* filename.nc + +The following configuration will look for files +/my/files/filename_AAA.nc and /my/files/filename_BBB.nc:: + + INPUT_DIR = /my/files + INPUT_TEMPLATE = filename_AAA.nc, filename_BBB.nc + +Lists of templates can be used with :ref:`multiple-files-wildcard `. +The following configuration will find all 4 files in /my/files:: + + INPUT_DIR = /my/files + INPUT_TEMPLATE = filename.nc, filename_*.nc + + +.. _multiple-files-window: + +Using File Windows to find Multiple Files +""""""""""""""""""""""""""""""""""""""""" + +The :term:`FCST_FILE_WINDOW_BEGIN`, :term:`FCST_FILE_WINDOW_END`, +:term:`OBS_FILE_WINDOW_BEGIN`, and :term:`OBS_FILE_WINDOW_END` +configuration variables can be used if the time information in the input data does not exactly line up with the run time but the user still wants to process the data. The default value of the file window begin and end variables are both 0 seconds. If both values are @@ -1665,12 +1734,11 @@ configuration:: For a run time of 20190201_00Z, and a set of files in the input directory that looks like this: -| /my/grid_stat/input/obs/20190131/pre.20190131_22.ext -| /my/grid_stat/input/obs/20190131/pre.20190131_23.ext -| /my/grid_stat/input/obs/20190201/othertype.20190201_00.ext -| /my/grid_stat/input/obs/20190201/pre.20190201_01.ext -| /my/grid_stat/input/obs/20190201/pre.20190201_02.ext -| +* /my/grid_stat/input/obs/20190131/pre.20190131_22.ext +* /my/grid_stat/input/obs/20190131/pre.20190131_23.ext +* /my/grid_stat/input/obs/20190201/othertype.20190201_00.ext +* /my/grid_stat/input/obs/20190201/pre.20190201_01.ext +* /my/grid_stat/input/obs/20190201/pre.20190201_02.ext The following behavior can be expected for each file: @@ -1690,12 +1758,11 @@ The following behavior can be expected for each file: 5. The fifth file matches the template but it is a further distance away from the closest file (7200 seconds versus 3600 seconds) so it is ignored. -Therefore, METplus Wrappers will use -/my/grid_stat/input/obs/20190131/pre.20190131_23.ext as the input to -grid_stat in this example. +Therefore, /my/grid_stat/input/obs/20190131/pre.20190131_23.ext will be used +as the input to grid_stat in this example. -Wrapper Specific Windows -^^^^^^^^^^^^^^^^^^^^^^^^ + +**Wrapper Specific Windows** A user may need to specify a different window on a wrapper-by-wrapper basis. If this is the case, the user can override the file window values for each @@ -1715,6 +1782,7 @@ EnsembleStat will use -0/+3600 for observation data. :term:`OBS_ENSEMBLE_STAT_FILE_WINDOW_BEGIN` was not set, so the EnsembleStat wrapper will use :term:`OBS_FILE_WINDOW_BEGIN`. + .. _Runtime_Freq: Runtime Frequency From cffab460b07ed65ff8ec33346e157a44ab066631 Mon Sep 17 00:00:00 2001 From: George McCabe <23407799+georgemccabe@users.noreply.github.com> Date: Fri, 18 Nov 2022 14:36:10 -0700 Subject: [PATCH 03/12] per #1783, add a section to describe the begin_end_incr syntax and link to this section where it is referenced elsewhere --- docs/Users_Guide/systemconfiguration.rst | 49 +++++++++++++++++++----- 1 file changed, 40 insertions(+), 9 deletions(-) diff --git a/docs/Users_Guide/systemconfiguration.rst b/docs/Users_Guide/systemconfiguration.rst index c80edeee5e..2c38c7dfc4 100644 --- a/docs/Users_Guide/systemconfiguration.rst +++ b/docs/Users_Guide/systemconfiguration.rst @@ -760,10 +760,8 @@ then three times will be processed: 2. Initialized on 2019-02-01 at 00Z / valid on 2019-02-01 at 06Z 3. Initialized on 2019-02-01 at 00Z / valid on 2019-02-01 at 09Z -The user can also define :term:`LEAD_SEQ` using a special notation for many -forecast leads. The notation is **begin_end_incr(b,e,i)** where b = the -first lead value, e = the last lead value (inclusive), and -i = the increment between leads. For example:: +The user can also define :term:`LEAD_SEQ` using :ref:`begin_end_incr` for many +forecast leads. For example:: [config] LEAD_SEQ = begin_end_incr(0,12,3) @@ -782,9 +780,8 @@ Any number of these groups can be defined by setting configuration variables LEAD_SEQ_1, LEAD_SEQ_2, ..., :term:`LEAD_SEQ_\`. The value can be defined with a comma-separated list of integers (currently only hours are supported here) -or using the special begin_end_incr(b,e,i) notation described just -above. Each :term:`LEAD_SEQ_\` must have a corresponding -variable :term:`LEAD_SEQ__LABEL`. For example:: +or using :ref:`begin_end_incr`. Each :term:`LEAD_SEQ_\` must have a +corresponding variable :term:`LEAD_SEQ__LABEL`. For example:: [config] @@ -891,8 +888,7 @@ Example 3:: This will skip every 30th and 31st day **and** every 3rd month. -**begin_end_incr(b,e,i)** syntax can be used to define a range of times to -skip. +:ref:`begin_end_incr` syntax can be used to define a range of times to skip. b = begin value, e = end value, @@ -1982,6 +1978,41 @@ Files Processed:: I2020101912_F006_V18 +.. _config-utilities: + +Config Utilities +---------------- + +.. _begin_end_incr: + +Begin End Increment (begin_end_incr) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +In configuration variables that can accept a list of values, the +Begin End Increment syntax can be used to easily create a sequence of numbers +without having to type out the entire list explicitly. +This functionality is similar to the Python range() function except that it is +inclusive, meaning that the end value is also included in the list. + +The notation is **begin_end_incr(b,e,i)** where b = the first lead value, +e = the last lead value (inclusive), and i = the increment between values. + +begin_end_incr(0,6,2) will expand to a list of numbers from 0 to 6 by 2, or +0, 2, 4, 6. + +An optional 4th argument can be provided to specify the zero padding. +begin_end_incr(8,10,1,2) will expand to 08, 09, 10. + +If this syntax is found within a configuration variable, it will expand a +string into a list with each number included. For example:: + + INPUT_TEMPLATE = ens01.nc, ens02.nc, ens03.nc, ens04.nc, ens05.nc, ens06.nc, ens07.nc, ens08.nc + +can be simplified as:: + + INPUT_TEMPLATE = ensbegin_end_incr(1,8,1,2).nc + + .. _metplus-control-met: How METplus controls MET configuration variables From ffc9479175ef4776b5974096756edf7dc5428ad4 Mon Sep 17 00:00:00 2001 From: George McCabe <23407799+georgemccabe@users.noreply.github.com> Date: Fri, 18 Nov 2022 14:40:45 -0700 Subject: [PATCH 04/12] fixed incorrect rst reference syntax --- docs/Users_Guide/systemconfiguration.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/Users_Guide/systemconfiguration.rst b/docs/Users_Guide/systemconfiguration.rst index 2c38c7dfc4..4bd2b6408c 100644 --- a/docs/Users_Guide/systemconfiguration.rst +++ b/docs/Users_Guide/systemconfiguration.rst @@ -1643,9 +1643,9 @@ Multiple Input Files If a tool supports reading multiple files for a given input, then there are a variety of ways to configure the METplus wrappers to read them. -* :ref:`multiple-files-wildcard ` -* :ref:`multiple-files-list ` -* :ref:`multiple-files-window ` +* :ref:`Wildcards ` +* :ref:`List of Templates ` +* :ref:`File Windows ` .. _multiple-files-wildcard: @@ -1694,7 +1694,7 @@ The following configuration will look for files INPUT_DIR = /my/files INPUT_TEMPLATE = filename_AAA.nc, filename_BBB.nc -Lists of templates can be used with :ref:`multiple-files-wildcard `. +Lists of templates can be used with :ref:`wildcards `. The following configuration will find all 4 files in /my/files:: INPUT_DIR = /my/files From 1db3459b83234c75225f09232cf6475c8f09ac45 Mon Sep 17 00:00:00 2001 From: George McCabe <23407799+georgemccabe@users.noreply.github.com> Date: Mon, 21 Nov 2022 11:16:16 -0700 Subject: [PATCH 05/12] added a note that begin_end_incr syntax can be used including a link to the section about it --- docs/Users_Guide/systemconfiguration.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/Users_Guide/systemconfiguration.rst b/docs/Users_Guide/systemconfiguration.rst index 4bd2b6408c..17922d3be1 100644 --- a/docs/Users_Guide/systemconfiguration.rst +++ b/docs/Users_Guide/systemconfiguration.rst @@ -1700,6 +1700,7 @@ The following configuration will find all 4 files in /my/files:: INPUT_DIR = /my/files INPUT_TEMPLATE = filename.nc, filename_*.nc +The :ref:`begin_end_incr` syntax can be used to generate lists of file paths. .. _multiple-files-window: From 03ee1454ca98c2ba2b042cd398d4cd8d551c7a9d Mon Sep 17 00:00:00 2001 From: George McCabe <23407799+georgemccabe@users.noreply.github.com> Date: Mon, 21 Nov 2022 11:39:26 -0700 Subject: [PATCH 06/12] update URL for files in dtcenter/MET repository to use new location of files (moved from met/scripts to scripts) --- .../met_tool_wrapper/ASCII2NC/ASCII2NC_python_embedding.py | 2 +- .../EnsembleStat/EnsembleStat_python_embedding.py | 2 +- .../met_tool_wrapper/GridStat/GridStat_python_embedding.py | 2 +- docs/use_cases/met_tool_wrapper/MODE/MODE_python_embedding.py | 2 +- docs/use_cases/met_tool_wrapper/MTD/MTD_python_embedding.py | 2 +- .../PlotDataPlane/PlotDataPlane_python_embedding.py | 2 +- docs/use_cases/met_tool_wrapper/PyEmbedIngest/PyEmbedIngest.py | 2 +- .../PyEmbedIngest/PyEmbedIngest_multi_field_one_file.py | 2 +- .../RegridDataPlane/RegridDataPlane_python_embedding.py | 2 +- .../SeriesAnalysis/SeriesAnalysis_python_embedding.py | 2 +- .../StatAnalysis/StatAnalysis_python_embedding.py | 2 +- 11 files changed, 11 insertions(+), 11 deletions(-) diff --git a/docs/use_cases/met_tool_wrapper/ASCII2NC/ASCII2NC_python_embedding.py b/docs/use_cases/met_tool_wrapper/ASCII2NC/ASCII2NC_python_embedding.py index 3e92e73c06..3f3ad5b5a2 100644 --- a/docs/use_cases/met_tool_wrapper/ASCII2NC/ASCII2NC_python_embedding.py +++ b/docs/use_cases/met_tool_wrapper/ASCII2NC/ASCII2NC_python_embedding.py @@ -63,7 +63,7 @@ # This use case calls a Python script to read the input data. # The Python script is stored in the MET repository: /path/to/MET/installation/share/met/python/read_ascii_point.py # -# `read_ascii_point.py `_ +# `read_ascii_point.py `_ ############################################################################## # Running METplus diff --git a/docs/use_cases/met_tool_wrapper/EnsembleStat/EnsembleStat_python_embedding.py b/docs/use_cases/met_tool_wrapper/EnsembleStat/EnsembleStat_python_embedding.py index e4e71665c4..37007eeccb 100644 --- a/docs/use_cases/met_tool_wrapper/EnsembleStat/EnsembleStat_python_embedding.py +++ b/docs/use_cases/met_tool_wrapper/EnsembleStat/EnsembleStat_python_embedding.py @@ -78,7 +78,7 @@ # This use case calls a Python script to read the input data. # The Python script is stored in the MET repository: /path/to/MET/installation/share/met/python/read_ascii_numpy.py # -# `read_ascii_numpy.py `_ +# `read_ascii_numpy.py `_ ############################################################################## # Running METplus diff --git a/docs/use_cases/met_tool_wrapper/GridStat/GridStat_python_embedding.py b/docs/use_cases/met_tool_wrapper/GridStat/GridStat_python_embedding.py index 40bade8833..aa3ceb9189 100644 --- a/docs/use_cases/met_tool_wrapper/GridStat/GridStat_python_embedding.py +++ b/docs/use_cases/met_tool_wrapper/GridStat/GridStat_python_embedding.py @@ -74,7 +74,7 @@ # This use case calls a Python script to read the input data. # The Python script is stored in the MET repository: /path/to/MET/installation/share/met/python/read_ascii_numpy.py # -# `read_ascii_numpy.py `_ +# `read_ascii_numpy.py `_ ############################################################################## # Running METplus diff --git a/docs/use_cases/met_tool_wrapper/MODE/MODE_python_embedding.py b/docs/use_cases/met_tool_wrapper/MODE/MODE_python_embedding.py index 3138e02293..53eb66d46d 100644 --- a/docs/use_cases/met_tool_wrapper/MODE/MODE_python_embedding.py +++ b/docs/use_cases/met_tool_wrapper/MODE/MODE_python_embedding.py @@ -73,7 +73,7 @@ # This use case calls a Python script to read the input data. # The Python script is stored in the MET repository: /path/to/MET/installation/share/met/python/read_ascii_numpy.py # -# `read_ascii_numpy.py `_ +# `read_ascii_numpy.py `_ ############################################################################## # Running METplus diff --git a/docs/use_cases/met_tool_wrapper/MTD/MTD_python_embedding.py b/docs/use_cases/met_tool_wrapper/MTD/MTD_python_embedding.py index 7b54ffc6a1..6651fb1555 100644 --- a/docs/use_cases/met_tool_wrapper/MTD/MTD_python_embedding.py +++ b/docs/use_cases/met_tool_wrapper/MTD/MTD_python_embedding.py @@ -73,7 +73,7 @@ # This use case calls a Python script to read the input data. # The Python script is stored in the MET repository: /path/to/MET/installation/share/met/python/read_ascii_numpy.py # -# `read_ascii_numpy.py `_ +# `read_ascii_numpy.py `_ ############################################################################## # Running METplus diff --git a/docs/use_cases/met_tool_wrapper/PlotDataPlane/PlotDataPlane_python_embedding.py b/docs/use_cases/met_tool_wrapper/PlotDataPlane/PlotDataPlane_python_embedding.py index 2a8821b1ba..00aebc197e 100644 --- a/docs/use_cases/met_tool_wrapper/PlotDataPlane/PlotDataPlane_python_embedding.py +++ b/docs/use_cases/met_tool_wrapper/PlotDataPlane/PlotDataPlane_python_embedding.py @@ -66,7 +66,7 @@ # This use case calls a Python script to read the input data. # The Python script is stored in the MET repository: /path/to/MET/installation/share/met/python/read_ascii_numpy.py # -# `read_ascii_numpy.py `_ +# `read_ascii_numpy.py `_ ############################################################################## # Running METplus diff --git a/docs/use_cases/met_tool_wrapper/PyEmbedIngest/PyEmbedIngest.py b/docs/use_cases/met_tool_wrapper/PyEmbedIngest/PyEmbedIngest.py index 030a1da461..08f1d5215c 100644 --- a/docs/use_cases/met_tool_wrapper/PyEmbedIngest/PyEmbedIngest.py +++ b/docs/use_cases/met_tool_wrapper/PyEmbedIngest/PyEmbedIngest.py @@ -59,7 +59,7 @@ # This use case calls a Python script to read the input data. # The Python script is stored in the MET repository: /path/to/MET/installation/share/met/python/read_ascii_numpy.py # -# `read_ascii_numpy.py `_ +# `read_ascii_numpy.py `_ ############################################################################## # Running METplus diff --git a/docs/use_cases/met_tool_wrapper/PyEmbedIngest/PyEmbedIngest_multi_field_one_file.py b/docs/use_cases/met_tool_wrapper/PyEmbedIngest/PyEmbedIngest_multi_field_one_file.py index 793f6eb3e0..b50d5c270d 100644 --- a/docs/use_cases/met_tool_wrapper/PyEmbedIngest/PyEmbedIngest_multi_field_one_file.py +++ b/docs/use_cases/met_tool_wrapper/PyEmbedIngest/PyEmbedIngest_multi_field_one_file.py @@ -62,7 +62,7 @@ # This use case calls a Python script to read the input data. # The Python script is stored in the MET repository: /path/to/MET/installation/share/met/python/read_ascii_numpy.py # -# `read_ascii_numpy.py `_ +# `read_ascii_numpy.py `_ ############################################################################## # Running METplus diff --git a/docs/use_cases/met_tool_wrapper/RegridDataPlane/RegridDataPlane_python_embedding.py b/docs/use_cases/met_tool_wrapper/RegridDataPlane/RegridDataPlane_python_embedding.py index 1d12d7a036..5277b5f340 100644 --- a/docs/use_cases/met_tool_wrapper/RegridDataPlane/RegridDataPlane_python_embedding.py +++ b/docs/use_cases/met_tool_wrapper/RegridDataPlane/RegridDataPlane_python_embedding.py @@ -64,7 +64,7 @@ # This use case calls a Python script to read the input data. # The Python script is stored in the MET repository: /path/to/MET/installation/share/met/python/read_ascii_numpy.py # -# `read_ascii_numpy.py `_ +# `read_ascii_numpy.py `_ ############################################################################## # Running METplus diff --git a/docs/use_cases/met_tool_wrapper/SeriesAnalysis/SeriesAnalysis_python_embedding.py b/docs/use_cases/met_tool_wrapper/SeriesAnalysis/SeriesAnalysis_python_embedding.py index f2f9ebdc13..8ea1b0b298 100644 --- a/docs/use_cases/met_tool_wrapper/SeriesAnalysis/SeriesAnalysis_python_embedding.py +++ b/docs/use_cases/met_tool_wrapper/SeriesAnalysis/SeriesAnalysis_python_embedding.py @@ -74,7 +74,7 @@ # This use case calls a Python script to read the input data. # The Python script is stored in the MET repository: /path/to/MET/installation/share/met/python/read_ascii_numpy.py # -# `read_ascii_numpy.py `_ +# `read_ascii_numpy.py `_ ############################################################################## # Running METplus diff --git a/docs/use_cases/met_tool_wrapper/StatAnalysis/StatAnalysis_python_embedding.py b/docs/use_cases/met_tool_wrapper/StatAnalysis/StatAnalysis_python_embedding.py index 0ad8ffc118..4ea9b70748 100644 --- a/docs/use_cases/met_tool_wrapper/StatAnalysis/StatAnalysis_python_embedding.py +++ b/docs/use_cases/met_tool_wrapper/StatAnalysis/StatAnalysis_python_embedding.py @@ -84,7 +84,7 @@ # This use case calls a Python script to read matched pair lines from an input source. # The Python script is stored in the MET repository: /path/to/MET/installation/share/met/python/read_ascii_mpr.py # -# `read_ascii_mpr.py `_ +# `read_ascii_mpr.py `_ ############################################################################## # Running METplus From 83d2d81b7284c389702810f1d6bb7f719ede5bf8 Mon Sep 17 00:00:00 2001 From: George McCabe <23407799+georgemccabe@users.noreply.github.com> Date: Mon, 21 Nov 2022 11:40:27 -0700 Subject: [PATCH 07/12] update URL for files in dtcenter/MET repository to use new location of files (moved from met/data to data). Note: These URLs use HEAD which is the current default branch. These links will not work until the MET v11.0.0 release is created and the default branch is moved to main_v11.0 --- docs/Users_Guide/wrappers.rst | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/docs/Users_Guide/wrappers.rst b/docs/Users_Guide/wrappers.rst index db6270afb3..e8f5a5b5b1 100644 --- a/docs/Users_Guide/wrappers.rst +++ b/docs/Users_Guide/wrappers.rst @@ -70,7 +70,7 @@ Environment variables are used to control entries in this configuration file. The default value for each environment variable is obtained from (except where noted below): -`MET_INSTALL_DIR/share/met/config/Ascii2NcConfig_default `_ +`MET_INSTALL_DIR/share/met/config/Ascii2NcConfig_default `_ Below the file contents are descriptions of each environment variable referenced in this file and the corresponding METplus configuration item used @@ -333,7 +333,7 @@ Environment variables are used to control entries in this configuration file. The default value for each environment variable is obtained from (except where noted below): -`MET_INSTALL_DIR/share/met/config/EnsembleStatConfig_default `_ +`MET_INSTALL_DIR/share/met/config/EnsembleStatConfig_default `_ Below the file contents are descriptions of each environment variable referenced in this file and the corresponding METplus configuration item used @@ -1124,7 +1124,7 @@ Environment variables are used to control entries in this configuration file. The default value for each environment variable is obtained from (except where noted below): -`MET_INSTALL_DIR/share/met/config/GenEnsProdConfig_default `_ +`MET_INSTALL_DIR/share/met/config/GenEnsProdConfig_default `_ Below the file contents are descriptions of each environment variable referenced in this file and the corresponding METplus configuration item used @@ -2686,7 +2686,7 @@ Environment variables are used to control entries in this configuration file. The default value for each environment variable is obtained from (except where noted below): -`MET_INSTALL_DIR/share/met/config/GridDiagConfig_default `_ +`MET_INSTALL_DIR/share/met/config/GridDiagConfig_default `_ Below the file contents are descriptions of each environment variable referenced in this file and the corresponding METplus configuration item used @@ -2964,7 +2964,7 @@ Environment variables are used to control entries in this configuration file. The default value for each environment variable is obtained from (except where noted below): -`MET_INSTALL_DIR/share/met/config/GridStatConfig_default `_ +`MET_INSTALL_DIR/share/met/config/GridStatConfig_default `_ Below the file contents are descriptions of each environment variable referenced in this file and the corresponding METplus configuration item used @@ -3501,7 +3501,7 @@ Environment variables are used to control entries in this configuration file. The default value for each environment variable is obtained from (except where noted below): -`MET_INSTALL_DIR/share/met/config/IODA2NCConfig_default `_ +`MET_INSTALL_DIR/share/met/config/IODA2NCConfig_default `_ Below the file contents are descriptions of each environment variable referenced in this file and the corresponding METplus configuration item used @@ -4044,7 +4044,7 @@ Environment variables are used to control entries in this configuration file. The default value for each environment variable is obtained from (except where noted below): -`MET_INSTALL_DIR/share/met/config/MODEConfig_default `_ +`MET_INSTALL_DIR/share/met/config/MODEConfig_default `_ Below the file contents are descriptions of each environment variable referenced in this file and the corresponding METplus configuration item used @@ -4669,7 +4669,7 @@ Environment variables are used to control entries in this configuration file. The default value for each environment variable is obtained from (except where noted below): -`MET_INSTALL_DIR/share/met/config/MTDConfig_default `_ +`MET_INSTALL_DIR/share/met/config/MTDConfig_default `_ Below the file contents are descriptions of each environment variable referenced in this file and the corresponding METplus configuration item used @@ -4941,7 +4941,7 @@ Environment variables are used to control entries in this configuration file. The default value for each environment variable is obtained from (except where noted below): -`MET_INSTALL_DIR/share/met/config/PB2NCConfig_default `_ +`MET_INSTALL_DIR/share/met/config/PB2NCConfig_default `_ Below the file contents are descriptions of each environment variable referenced in this file and the corresponding METplus configuration item used @@ -5367,7 +5367,7 @@ Environment variables are used to control entries in this configuration file. The default value for each environment variable is obtained from (except where noted below): -`MET_INSTALL_DIR/share/met/config/PlotPointObsConfig_default `_ +`MET_INSTALL_DIR/share/met/config/PlotPointObsConfig_default `_ Below the file contents are descriptions of each environment variable referenced in this file and the corresponding METplus configuration item used @@ -5861,7 +5861,7 @@ Environment variables are used to control entries in this configuration file. The default value for each environment variable is obtained from (except where noted below): -`MET_INSTALL_DIR/share/met/config/PointStatConfig_default `_ +`MET_INSTALL_DIR/share/met/config/PointStatConfig_default `_ Below the file contents are descriptions of each environment variable referenced in this file and the corresponding METplus configuration item used @@ -6496,7 +6496,7 @@ Environment variables are used to control entries in this configuration file. The default value for each environment variable is obtained from (except where noted below): -`MET_INSTALL_DIR/share/met/config/SeriesAnalysisConfig_default `_ +`MET_INSTALL_DIR/share/met/config/SeriesAnalysisConfig_default `_ Below the file contents are descriptions of each environment variable referenced in this file and the corresponding METplus configuration item used @@ -7161,7 +7161,7 @@ Environment variables are used to control entries in this configuration file. The default value for each environment variable is obtained from (except where noted below): -`MET_INSTALL_DIR/share/met/config/STATAnalysisConfig_default `_ +`MET_INSTALL_DIR/share/met/config/STATAnalysisConfig_default `_ Below the file contents are descriptions of each environment variable referenced in this file and the corresponding METplus configuration item used @@ -7661,7 +7661,7 @@ Environment variables are used to control entries in this configuration file. The default value for each environment variable is obtained from (except where noted below): -`MET_INSTALL_DIR/share/met/config/TCGenConfig_default `_ +`MET_INSTALL_DIR/share/met/config/TCGenConfig_default `_ Below the file contents are descriptions of each environment variable referenced in this file and the corresponding METplus configuration item used @@ -8359,7 +8359,7 @@ Environment variables are used to control entries in this configuration file. The default value for each environment variable is obtained from (except where noted below): -`MET_INSTALL_DIR/share/met/config/TCPairsConfig_default `_ +`MET_INSTALL_DIR/share/met/config/TCPairsConfig_default `_ Below the file contents are descriptions of each environment variable referenced in this file and the corresponding METplus configuration item used @@ -8700,7 +8700,7 @@ Environment variables are used to control entries in this configuration file. The default value for each environment variable is obtained from (except where noted below): -`MET_INSTALL_DIR/share/met/config/TCRMWConfig_default `_ +`MET_INSTALL_DIR/share/met/config/TCRMWConfig_default `_ Below the file contents are descriptions of each environment variable referenced in this file and the corresponding METplus configuration item used @@ -9030,7 +9030,7 @@ Environment variables are used to control entries in this configuration file. The default value for each environment variable is obtained from (except where noted below): -`MET_INSTALL_DIR/share/met/config/TCStatConfig_default `_ +`MET_INSTALL_DIR/share/met/config/TCStatConfig_default `_ Below the file contents are descriptions of each environment variable referenced in this file and the corresponding METplus configuration item used From fda461850d921eed5abbd27de87141ba155cecba Mon Sep 17 00:00:00 2001 From: George McCabe <23407799+georgemccabe@users.noreply.github.com> Date: Mon, 21 Nov 2022 12:26:38 -0700 Subject: [PATCH 08/12] updated MET version in installation chapter --- docs/Users_Guide/installation.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Users_Guide/installation.rst b/docs/Users_Guide/installation.rst index 18a248e4f9..5317aa69b9 100644 --- a/docs/Users_Guide/installation.rst +++ b/docs/Users_Guide/installation.rst @@ -44,7 +44,7 @@ The following software is required to run METplus Wrappers: - Python 3.8.6 or above -- MET version 10.0.0 or above - +- MET version 11.0.0 or above - For information on installing MET please see the `Software Installation/Getting Started `_ section of the MET User's Guide. From 7de0d9f1bd9a29331104380853dc4a135661b29d Mon Sep 17 00:00:00 2001 From: George McCabe <23407799+georgemccabe@users.noreply.github.com> Date: Mon, 21 Nov 2022 13:08:21 -0700 Subject: [PATCH 09/12] per #1171, add documentaiton on how to run METplus using Docker --- docs/Users_Guide/getting_started.rst | 73 ++++++++++++++++++++++++++++ 1 file changed, 73 insertions(+) diff --git a/docs/Users_Guide/getting_started.rst b/docs/Users_Guide/getting_started.rst index 7941a97f68..942d192bb5 100644 --- a/docs/Users_Guide/getting_started.rst +++ b/docs/Users_Guide/getting_started.rst @@ -377,3 +377,76 @@ the path to the METplus log file that was generated. * Review the :ref:`metplus_final.conf` file to see all of the settings that were used in the use case. + +.. _metplus-docker: + +METplus in Docker +================= + +METplus is available on DockerHub. The METplus Docker image includes all of +the MET executables from the corresponding METplus Coordinated Release and +the METplus wrappers are pre-configured to use them. + +To pull the latest official release, run:: + + docker pull dtcenter/metplus:latest + +Tags for previous releases and development releases are also available. +Refer to the list of +:ref:`available tags `_ +on DockerHub. + +.. _docker-sample-input: + +Sample Input Data +----------------- + +Sample input data for all of the use cases provided with the METplus wrappers +are also available on DockerHub. These data are found in the +dtcenter/metplus-data DockerHub repository and are named with the X.Y version +of the corresponding METplus release and the name of the use case category +separated by a dash, e.g. 4.1-data_assimilation or 4.0-met_tool_wrapper. +A list of +:ref:`available tags for input data `_ +can also be found on DockerHub. + +To make these data available in a METplus Docker container, first create a +Docker data volume from the desired tag and give it a name with the *--name* +argument:: + + docker create --name met_tool_wrapper dtcenter/metplus-data:4.1-met_tool_wrapper + +Then mount the data volume to the container using the *--volumes-from* argument +to the *docker run* command:: + + docker run --rm -it --volumes-from met_tool_wrapper dtcenter/metplus:4.1.4 bash + +The input data will be available inside the container under +/data/input/METplus_Data. + +Running METplus in Docker +------------------------- + +The run_metplus.py script is in the user's path inside the container. +The use case configuration files can be found under +/metplus/METplus/parm/use_cases. +The values for :ref:`sys_conf_met_install_dir`, :ref:`sys_conf_input_base`, and +:ref:`sys_conf_output_base` are set to appropriate default values. +A use case can be run by simply passing the use case configuration file to the +run script:: + + run_metplus.py /metplus/METplus/parm/use_cases/met_tool_wrapper/Example/Example.conf + +Output from the use case will be written to /data/output by default. +The value for :ref:`sys_conf_output_base` can be changed to write elsewhere:: + + run_metplus.py /metplus/METplus/parm/use_cases/met_tool_wrapper/Example/Example.conf config.OUTPUT_BASE=/data/output/my_data_dir + +If :ref:`docker-sample-input` is mounted to the container, then use cases from +the corresponding category can be run:: + + run_metplus.py /metplus/METplus/parm/use_cases/met_tool_wrapper/GridStat/GridStat.conf + +Please note that use cases that have additional Python package dependencies +may not run successfully unless those packages are installed inside the +container or obtained elsewhere. From 4ecae5487f77d678b4ae3ccefc8e9e97ab46f5e6 Mon Sep 17 00:00:00 2001 From: George McCabe <23407799+georgemccabe@users.noreply.github.com> Date: Mon, 21 Nov 2022 13:15:35 -0700 Subject: [PATCH 10/12] fixed urls --- docs/Users_Guide/getting_started.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/Users_Guide/getting_started.rst b/docs/Users_Guide/getting_started.rst index 942d192bb5..a9aabaa928 100644 --- a/docs/Users_Guide/getting_started.rst +++ b/docs/Users_Guide/getting_started.rst @@ -393,7 +393,7 @@ To pull the latest official release, run:: Tags for previous releases and development releases are also available. Refer to the list of -:ref:`available tags `_ +`available tags`_ on DockerHub. .. _docker-sample-input: @@ -407,7 +407,7 @@ dtcenter/metplus-data DockerHub repository and are named with the X.Y version of the corresponding METplus release and the name of the use case category separated by a dash, e.g. 4.1-data_assimilation or 4.0-met_tool_wrapper. A list of -:ref:`available tags for input data `_ +`available tags for input data `_ can also be found on DockerHub. To make these data available in a METplus Docker container, first create a From e4e5404c468b639d0012236f6c64c29bd3d4986a Mon Sep 17 00:00:00 2001 From: George McCabe <23407799+georgemccabe@users.noreply.github.com> Date: Mon, 21 Nov 2022 13:20:39 -0700 Subject: [PATCH 11/12] another fix --- docs/Users_Guide/getting_started.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Users_Guide/getting_started.rst b/docs/Users_Guide/getting_started.rst index a9aabaa928..6bf82c9c57 100644 --- a/docs/Users_Guide/getting_started.rst +++ b/docs/Users_Guide/getting_started.rst @@ -393,7 +393,7 @@ To pull the latest official release, run:: Tags for previous releases and development releases are also available. Refer to the list of -`available tags`_ +`available tags `_ on DockerHub. .. _docker-sample-input: From 98839c8dd59ea84d8fa06a2d8c472125da6a6536 Mon Sep 17 00:00:00 2001 From: George McCabe <23407799+georgemccabe@users.noreply.github.com> Date: Thu, 1 Dec 2022 12:01:58 -0700 Subject: [PATCH 12/12] Apply suggestions from code review Co-authored-by: Dan Adriaansen --- docs/Users_Guide/getting_started.rst | 2 +- docs/Users_Guide/systemconfiguration.rst | 2 +- docs/Users_Guide/wrappers.rst | 6 +++--- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/Users_Guide/getting_started.rst b/docs/Users_Guide/getting_started.rst index 6bf82c9c57..5d7287bd25 100644 --- a/docs/Users_Guide/getting_started.rst +++ b/docs/Users_Guide/getting_started.rst @@ -404,7 +404,7 @@ Sample Input Data Sample input data for all of the use cases provided with the METplus wrappers are also available on DockerHub. These data are found in the dtcenter/metplus-data DockerHub repository and are named with the X.Y version -of the corresponding METplus release and the name of the use case category +of the corresponding METplus Coordinated Release and the name of the use case category separated by a dash, e.g. 4.1-data_assimilation or 4.0-met_tool_wrapper. A list of `available tags for input data `_ diff --git a/docs/Users_Guide/systemconfiguration.rst b/docs/Users_Guide/systemconfiguration.rst index 17922d3be1..507f67b7b4 100644 --- a/docs/Users_Guide/systemconfiguration.rst +++ b/docs/Users_Guide/systemconfiguration.rst @@ -1990,7 +1990,7 @@ Begin End Increment (begin_end_incr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ In configuration variables that can accept a list of values, the -Begin End Increment syntax can be used to easily create a sequence of numbers +Begin End Increment utility can be used to easily create a sequence of numbers without having to type out the entire list explicitly. This functionality is similar to the Python range() function except that it is inclusive, meaning that the end value is also included in the list. diff --git a/docs/Users_Guide/wrappers.rst b/docs/Users_Guide/wrappers.rst index e8f5a5b5b1..2dca4b54e3 100644 --- a/docs/Users_Guide/wrappers.rst +++ b/docs/Users_Guide/wrappers.rst @@ -5127,7 +5127,7 @@ PCPCombine wrapper can be configured to process forecast and/or observation data. Setting :term:`FCST_PCP_COMBINE_RUN` = True will process forecast data and setting :term:`OBS_PCP_COMBINE_RUN` = True will process observation data. -PCPCombine wrapper can be configured to build a command for sum, add, subtract, +PCPCombine wrapper can be configured to build a command for the sum, add, subtract, and derive methods using :term:`FCST_PCP_COMBINE_METHOD` and/or :term:`OBS_PCP_COMBINE_METHOD`. Each method executes logic to gather the desired input files to build the command based on specific examples. @@ -5141,7 +5141,7 @@ The default units are hours unless otherwise specified. The output field name can be set explicitly using :term:`FCST_PCP_COMBINE_OUTPUT_NAME` or :term:`OBS_PCP_COMBINE_OUTPUT_NAME`. -For ADD and DERIVE methods, the input accumulation(s) can be specified using +For the ADD and DERIVE methods, the input accumulation(s) can be specified using :term:`FCST_PCP_COMBINE_INPUT_ACCUMS` or :term:`OBS_PCP_COMBINE_INPUT_ACCUMS`. The default units are hours unless otherwise specified. This can be a list of accumulation amounts in order of preference. @@ -5157,7 +5157,7 @@ These lists must be the same length as Constant Initialization Time ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -For ADD and DERIVE methods, :term:`FCST_PCP_COMBINE_CONSTANT_INIT` or +For the ADD and DERIVE methods, :term:`FCST_PCP_COMBINE_CONSTANT_INIT` or :term:`OBS_PCP_COMBINE_CONSTANT_INIT` can be set to **True** to gather input files that all contain the same initialization time.