Skip to content

Commit

Permalink
per #1783, add documentation to describe how to specify multiple file…
Browse files Browse the repository at this point in the history
…s to read
  • Loading branch information
georgemccabe committed Nov 18, 2022
1 parent 893d987 commit b20f63c
Showing 1 changed file with 82 additions and 14 deletions.
96 changes: 82 additions & 14 deletions docs/Users_Guide/systemconfiguration.rst
Original file line number Diff line number Diff line change
Expand Up @@ -1640,10 +1640,79 @@ and will substitute the current day into the template, giving a filename of
a file time of 20190131_23Z, so the filename will be 20190131.ext that is
generated by the template.

Using Windows to find Valid Files
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The [FCST/OBS]_FILE_WINDOW_[BEGIN/END] configuration variables can be used
Multiple Input Files
^^^^^^^^^^^^^^^^^^^^

If a tool supports reading multiple files for a given input, then there are
a variety of ways to configure the METplus wrappers to read them.

* :ref:`multiple-files-wildcard <Wildcards>`
* :ref:`multiple-files-list <List of Templates>`
* :ref:`multiple-files-window <File Windows>`

.. _multiple-files-wildcard:

Using Wildcards to find Multiple Files
""""""""""""""""""""""""""""""""""""""

Wildcard characters can be used in filename template variables.
The \* character is used to match 1 or more characters and
the \? character is used to match a single character.

For example, if a directory /my/files contains the following files:

* filename_AAA.nc
* filename_ABA.nc
* filename_BBB.nc
* filename.nc

The following template will match
filename_AAA.nc, filename_ABA.nc and filename_BBB.nc, but not filename.nc::

INPUT_TEMPLATE = /my/files/filename_*.nc

The following template will match filename_AAA.nc and filename_ABA.nc::

INPUT_TEMPLATE = /my/files/filename_A?A.nc


.. _multiple-files-list:

Using a List of Templates to find Multiple Files
""""""""""""""""""""""""""""""""""""""""""""""""

A comma-separated list of templates can be specified in a \_TEMPLATE variable.
Each value in the list will be added to the corresponding \_DIR variable.

For example, if a directory /my/files contains the following files:

* filename_AAA.nc
* filename_ABA.nc
* filename_BBB.nc
* filename.nc

The following configuration will look for files
/my/files/filename_AAA.nc and /my/files/filename_BBB.nc::

INPUT_DIR = /my/files
INPUT_TEMPLATE = filename_AAA.nc, filename_BBB.nc

Lists of templates can be used with :ref:`multiple-files-wildcard <wildcards>`.
The following configuration will find all 4 files in /my/files::

INPUT_DIR = /my/files
INPUT_TEMPLATE = filename.nc, filename_*.nc


.. _multiple-files-window:

Using File Windows to find Multiple Files
"""""""""""""""""""""""""""""""""""""""""

The :term:`FCST_FILE_WINDOW_BEGIN`, :term:`FCST_FILE_WINDOW_END`,
:term:`OBS_FILE_WINDOW_BEGIN`, and :term:`OBS_FILE_WINDOW_END`
configuration variables can be used
if the time information in the input data does not exactly line up with the
run time but the user still wants to process the data. The default value of the
file window begin and end variables are both 0 seconds. If both values are
Expand All @@ -1665,12 +1734,11 @@ configuration::
For a run time of 20190201_00Z, and a set of files in the input directory
that looks like this:

| /my/grid_stat/input/obs/20190131/pre.20190131_22.ext
| /my/grid_stat/input/obs/20190131/pre.20190131_23.ext
| /my/grid_stat/input/obs/20190201/othertype.20190201_00.ext
| /my/grid_stat/input/obs/20190201/pre.20190201_01.ext
| /my/grid_stat/input/obs/20190201/pre.20190201_02.ext
|
* /my/grid_stat/input/obs/20190131/pre.20190131_22.ext
* /my/grid_stat/input/obs/20190131/pre.20190131_23.ext
* /my/grid_stat/input/obs/20190201/othertype.20190201_00.ext
* /my/grid_stat/input/obs/20190201/pre.20190201_01.ext
* /my/grid_stat/input/obs/20190201/pre.20190201_02.ext

The following behavior can be expected for each file:

Expand All @@ -1690,12 +1758,11 @@ The following behavior can be expected for each file:
5. The fifth file matches the template but it is a further distance away
from the closest file (7200 seconds versus 3600 seconds) so it is ignored.

Therefore, METplus Wrappers will use
/my/grid_stat/input/obs/20190131/pre.20190131_23.ext as the input to
grid_stat in this example.
Therefore, /my/grid_stat/input/obs/20190131/pre.20190131_23.ext will be used
as the input to grid_stat in this example.

Wrapper Specific Windows
^^^^^^^^^^^^^^^^^^^^^^^^

**Wrapper Specific Windows**

A user may need to specify a different window on a wrapper-by-wrapper basis.
If this is the case, the user can override the file window values for each
Expand All @@ -1715,6 +1782,7 @@ EnsembleStat will use -0/+3600 for observation data.
:term:`OBS_ENSEMBLE_STAT_FILE_WINDOW_BEGIN` was not set, so the
EnsembleStat wrapper will use :term:`OBS_FILE_WINDOW_BEGIN`.


.. _Runtime_Freq:

Runtime Frequency
Expand Down

0 comments on commit b20f63c

Please sign in to comment.