Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature #1711, #1783, #1814, and #1835 Documentation - Multiple files, PCPCombine constant init, Docker instructions, and MET links #1967

Merged
merged 13 commits into from
Dec 5, 2022
Merged
Show file tree
Hide file tree
Changes from 11 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
73 changes: 73 additions & 0 deletions docs/Users_Guide/getting_started.rst
Original file line number Diff line number Diff line change
Expand Up @@ -377,3 +377,76 @@ the path to the METplus log file that was generated.
* Review the :ref:`metplus_final.conf<metplus_final_conf>` file to see all
of the settings that were used in the use case.


.. _metplus-docker:

METplus in Docker
=================

METplus is available on DockerHub. The METplus Docker image includes all of
the MET executables from the corresponding METplus Coordinated Release and
the METplus wrappers are pre-configured to use them.

To pull the latest official release, run::

docker pull dtcenter/metplus:latest

Tags for previous releases and development releases are also available.
Refer to the list of
`available tags <https://hub.docker.com/repository/docker/dtcenter/metplus/tags>`_
on DockerHub.

.. _docker-sample-input:

Sample Input Data
-----------------

Sample input data for all of the use cases provided with the METplus wrappers
are also available on DockerHub. These data are found in the
dtcenter/metplus-data DockerHub repository and are named with the X.Y version
of the corresponding METplus release and the name of the use case category
georgemccabe marked this conversation as resolved.
Show resolved Hide resolved
separated by a dash, e.g. 4.1-data_assimilation or 4.0-met_tool_wrapper.
A list of
`available tags for input data <https://hub.docker.com/repository/docker/dtcenter/metplus-data/tags>`_
can also be found on DockerHub.

To make these data available in a METplus Docker container, first create a
Docker data volume from the desired tag and give it a name with the *--name*
argument::

docker create --name met_tool_wrapper dtcenter/metplus-data:4.1-met_tool_wrapper

Then mount the data volume to the container using the *--volumes-from* argument
to the *docker run* command::

docker run --rm -it --volumes-from met_tool_wrapper dtcenter/metplus:4.1.4 bash

The input data will be available inside the container under
/data/input/METplus_Data.

Running METplus in Docker
-------------------------

The run_metplus.py script is in the user's path inside the container.
The use case configuration files can be found under
/metplus/METplus/parm/use_cases.
The values for :ref:`sys_conf_met_install_dir`, :ref:`sys_conf_input_base`, and
:ref:`sys_conf_output_base` are set to appropriate default values.
A use case can be run by simply passing the use case configuration file to the
run script::

run_metplus.py /metplus/METplus/parm/use_cases/met_tool_wrapper/Example/Example.conf

Output from the use case will be written to /data/output by default.
The value for :ref:`sys_conf_output_base` can be changed to write elsewhere::

run_metplus.py /metplus/METplus/parm/use_cases/met_tool_wrapper/Example/Example.conf config.OUTPUT_BASE=/data/output/my_data_dir

If :ref:`docker-sample-input` is mounted to the container, then use cases from
the corresponding category can be run::

run_metplus.py /metplus/METplus/parm/use_cases/met_tool_wrapper/GridStat/GridStat.conf

Please note that use cases that have additional Python package dependencies
may not run successfully unless those packages are installed inside the
container or obtained elsewhere.
2 changes: 1 addition & 1 deletion docs/Users_Guide/installation.rst
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ The following software is required to run METplus Wrappers:

- Python 3.8.6 or above

- MET version 10.0.0 or above -
- MET version 11.0.0 or above -
For information on installing MET please see the `Software Installation/Getting Started <https://met.readthedocs.io/en/latest/Users_Guide/installation.html>`_
section of the MET User's Guide.

Expand Down
146 changes: 123 additions & 23 deletions docs/Users_Guide/systemconfiguration.rst
Original file line number Diff line number Diff line change
Expand Up @@ -760,10 +760,8 @@ then three times will be processed:
2. Initialized on 2019-02-01 at 00Z / valid on 2019-02-01 at 06Z
3. Initialized on 2019-02-01 at 00Z / valid on 2019-02-01 at 09Z

The user can also define :term:`LEAD_SEQ` using a special notation for many
forecast leads. The notation is **begin_end_incr(b,e,i)** where b = the
first lead value, e = the last lead value (inclusive), and
i = the increment between leads. For example::
The user can also define :term:`LEAD_SEQ` using :ref:`begin_end_incr` for many
forecast leads. For example::

[config]
LEAD_SEQ = begin_end_incr(0,12,3)
Expand All @@ -782,9 +780,8 @@ Any number of these groups can be defined by setting
configuration variables LEAD_SEQ_1, LEAD_SEQ_2, ..., :term:`LEAD_SEQ_\<n\>`.
The value can be defined with a
comma-separated list of integers (currently only hours are supported here)
or using the special begin_end_incr(b,e,i) notation described just
above. Each :term:`LEAD_SEQ_\<n\>` must have a corresponding
variable :term:`LEAD_SEQ_<n>_LABEL`. For example::
or using :ref:`begin_end_incr`. Each :term:`LEAD_SEQ_\<n\>` must have a
corresponding variable :term:`LEAD_SEQ_<n>_LABEL`. For example::


[config]
Expand Down Expand Up @@ -891,8 +888,7 @@ Example 3::
This will skip every 30th and 31st day **and** every 3rd month.


**begin_end_incr(b,e,i)** syntax can be used to define a range of times to
skip.
:ref:`begin_end_incr` syntax can be used to define a range of times to skip.

b = begin value, e = end value,

Expand Down Expand Up @@ -1640,10 +1636,80 @@ and will substitute the current day into the template, giving a filename of
a file time of 20190131_23Z, so the filename will be 20190131.ext that is
generated by the template.

Using Windows to find Valid Files
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The [FCST/OBS]_FILE_WINDOW_[BEGIN/END] configuration variables can be used
Multiple Input Files
^^^^^^^^^^^^^^^^^^^^

If a tool supports reading multiple files for a given input, then there are
a variety of ways to configure the METplus wrappers to read them.

* :ref:`Wildcards <multiple-files-wildcard>`
* :ref:`List of Templates <multiple-files-list>`
* :ref:`File Windows <multiple-files-window>`

.. _multiple-files-wildcard:

Using Wildcards to find Multiple Files
""""""""""""""""""""""""""""""""""""""

Wildcard characters can be used in filename template variables.
The \* character is used to match 1 or more characters and
the \? character is used to match a single character.

For example, if a directory /my/files contains the following files:

* filename_AAA.nc
* filename_ABA.nc
* filename_BBB.nc
* filename.nc

The following template will match
filename_AAA.nc, filename_ABA.nc and filename_BBB.nc, but not filename.nc::

INPUT_TEMPLATE = /my/files/filename_*.nc

The following template will match filename_AAA.nc and filename_ABA.nc::

INPUT_TEMPLATE = /my/files/filename_A?A.nc


.. _multiple-files-list:

Using a List of Templates to find Multiple Files
""""""""""""""""""""""""""""""""""""""""""""""""

A comma-separated list of templates can be specified in a \_TEMPLATE variable.
Each value in the list will be added to the corresponding \_DIR variable.

For example, if a directory /my/files contains the following files:

* filename_AAA.nc
* filename_ABA.nc
* filename_BBB.nc
* filename.nc

The following configuration will look for files
/my/files/filename_AAA.nc and /my/files/filename_BBB.nc::

INPUT_DIR = /my/files
INPUT_TEMPLATE = filename_AAA.nc, filename_BBB.nc

Lists of templates can be used with :ref:`wildcards <multiple-files-wildcard>`.
The following configuration will find all 4 files in /my/files::

INPUT_DIR = /my/files
INPUT_TEMPLATE = filename.nc, filename_*.nc

The :ref:`begin_end_incr` syntax can be used to generate lists of file paths.
DanielAdriaansen marked this conversation as resolved.
Show resolved Hide resolved

.. _multiple-files-window:

Using File Windows to find Multiple Files
"""""""""""""""""""""""""""""""""""""""""

The :term:`FCST_FILE_WINDOW_BEGIN`, :term:`FCST_FILE_WINDOW_END`,
:term:`OBS_FILE_WINDOW_BEGIN`, and :term:`OBS_FILE_WINDOW_END`
configuration variables can be used
if the time information in the input data does not exactly line up with the
run time but the user still wants to process the data. The default value of the
file window begin and end variables are both 0 seconds. If both values are
Expand All @@ -1665,12 +1731,11 @@ configuration::
For a run time of 20190201_00Z, and a set of files in the input directory
that looks like this:

| /my/grid_stat/input/obs/20190131/pre.20190131_22.ext
| /my/grid_stat/input/obs/20190131/pre.20190131_23.ext
| /my/grid_stat/input/obs/20190201/othertype.20190201_00.ext
| /my/grid_stat/input/obs/20190201/pre.20190201_01.ext
| /my/grid_stat/input/obs/20190201/pre.20190201_02.ext
|
* /my/grid_stat/input/obs/20190131/pre.20190131_22.ext
* /my/grid_stat/input/obs/20190131/pre.20190131_23.ext
* /my/grid_stat/input/obs/20190201/othertype.20190201_00.ext
* /my/grid_stat/input/obs/20190201/pre.20190201_01.ext
* /my/grid_stat/input/obs/20190201/pre.20190201_02.ext

The following behavior can be expected for each file:

Expand All @@ -1690,12 +1755,11 @@ The following behavior can be expected for each file:
5. The fifth file matches the template but it is a further distance away
from the closest file (7200 seconds versus 3600 seconds) so it is ignored.

Therefore, METplus Wrappers will use
/my/grid_stat/input/obs/20190131/pre.20190131_23.ext as the input to
grid_stat in this example.
Therefore, /my/grid_stat/input/obs/20190131/pre.20190131_23.ext will be used
as the input to grid_stat in this example.

Wrapper Specific Windows
^^^^^^^^^^^^^^^^^^^^^^^^

**Wrapper Specific Windows**

A user may need to specify a different window on a wrapper-by-wrapper basis.
If this is the case, the user can override the file window values for each
Expand All @@ -1715,6 +1779,7 @@ EnsembleStat will use -0/+3600 for observation data.
:term:`OBS_ENSEMBLE_STAT_FILE_WINDOW_BEGIN` was not set, so the
EnsembleStat wrapper will use :term:`OBS_FILE_WINDOW_BEGIN`.


.. _Runtime_Freq:

Runtime Frequency
Expand Down Expand Up @@ -1914,6 +1979,41 @@ Files Processed::
I2020101912_F006_V18


.. _config-utilities:

Config Utilities
----------------

.. _begin_end_incr:

Begin End Increment (begin_end_incr)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

In configuration variables that can accept a list of values, the
Begin End Increment syntax can be used to easily create a sequence of numbers
georgemccabe marked this conversation as resolved.
Show resolved Hide resolved
without having to type out the entire list explicitly.
This functionality is similar to the Python range() function except that it is
inclusive, meaning that the end value is also included in the list.

The notation is **begin_end_incr(b,e,i)** where b = the first lead value,
e = the last lead value (inclusive), and i = the increment between values.

begin_end_incr(0,6,2) will expand to a list of numbers from 0 to 6 by 2, or
0, 2, 4, 6.

An optional 4th argument can be provided to specify the zero padding.
begin_end_incr(8,10,1,2) will expand to 08, 09, 10.

If this syntax is found within a configuration variable, it will expand a
string into a list with each number included. For example::

INPUT_TEMPLATE = ens01.nc, ens02.nc, ens03.nc, ens04.nc, ens05.nc, ens06.nc, ens07.nc, ens08.nc

can be simplified as::

INPUT_TEMPLATE = ensbegin_end_incr(1,8,1,2).nc


.. _metplus-control-met:

How METplus controls MET configuration variables
Expand Down
Loading