Feature 1289 explicit file list

docs/Users_Guide/glossary.rst

@@ -2695,7 +2695,7 @@ METplus Configuration Glossary
      | *Used by:*  EnsembleStat, GridStat, MODE, MTD, PB2NC, PointStat
 
    OBS_FILE_WINDOW_END
-     Used to control the upper bound of the window around the valid time to determine if a file should be used for processing. See :ref:`Directory_and_Filename_Template_Info` subsection called 'Using Windows to Find Valid Files.' Units are seconds.This value will be used for all wrappers that look for an observation file unless it is overridden by a wrapper specific configuration variable. For example, if :term:`OBS_GRID_STAT_WINDOW_END` is set, the GridStat wrapper will use that value. If :term:`PB2NC_WINDOW_END` is not set, then the PB2NC wrapper will use :term:`OBS_WINDOW_END`. A corresponding variable exists for forecast data called :term:`FCST_FILE_WINDOW_END`.
+     Used to control the upper bound of the window around the valid time to determine if a file should be used for processing. See :ref:`Directory_and_Filename_Template_Info` subsection called 'Using Windows to Find Valid Files.' Units are seconds.This value will be used for all wrappers that look for an observation file unless it is overridden by a wrapper specific configuration variable. For example, if :term:`OBS_GRID_STAT_FILE_WINDOW_END` is set, the GridStat wrapper will use that value. If :term:`PB2NC_FILE_WINDOW_END` is not set, then the PB2NC wrapper will use :term:`OBS_FILE_WINDOW_END`. A corresponding variable exists for forecast data called :term:`FCST_FILE_WINDOW_END`.
 
      | *Used by:*  EnsembleStat, GridStat, MODE, MTD, PB2NC, PointStat
 
@@ -3173,12 +3173,12 @@ METplus Configuration Glossary
      .. warning:: **DEPRECATED:** Please use :term:`OBS_WINDOW_BEGIN`.
 
    OBS_WINDOW_BEGIN
-     Passed to the MET config file to determine the range of data within a file that should be used for processing.Units are seconds. This value will be used for all wrappers that look for an observation file unless it is overridden by a wrapper specific configuration variable. For example, if :term:`OBS_POINT_STAT_WINDOW_BEGIN` is set, the PointStat wrapper will use that value. If :term:`PB2NC_WINDOW_BEGIN` is not set, then the PB2NC wrapper will use :term:`OBS_WINDOW_BEGIN`. A corresponding variable exists for forecast data called :term:`FCST_WINDOW_BEGIN`.
+     Passed to the MET config file to determine the range of data within a file that should be used for processing.Units are seconds. This value will be used for all wrappers that look for an observation file unless it is overridden by a wrapper specific configuration variable. For example, if :term:`OBS_POINT_STAT_WINDOW_BEGIN` is set, the PointStat wrapper will use that value. If :term:`PB2NC_OBS_WINDOW_BEGIN` is not set, then the PB2NC wrapper will use :term:`OBS_WINDOW_BEGIN`. A corresponding variable exists for forecast data called :term:`FCST_WINDOW_BEGIN`.
 
      | *Used by:*  PB2NC, PointStat
 
    OBS_WINDOW_END
-     Passed to the MET config file to determine the range of data within a file that should be used for processing.Units are seconds. This value will be used for all wrappers that look for an observation file unless it is overridden by a wrapper specific configuration variable. For example, if :term:`OBS_POINT_STAT_WINDOW_END` is set, the PointStat wrapper will use that value. If :term:`PB2NC_WINDOW_END` is not set, then the PB2NC wrapper will use :term:`OBS_WINDOW_END`. A corresponding variable exists for forecast data called :term:`FCST_WINDOW_END`.
+     Passed to the MET config file to determine the range of data within a file that should be used for processing.Units are seconds. This value will be used for all wrappers that look for an observation file unless it is overridden by a wrapper specific configuration variable. For example, if :term:`OBS_POINT_STAT_WINDOW_END` is set, the PointStat wrapper will use that value. If :term:`PB2NC_OBS_WINDOW_END` is not set, then the PB2NC wrapper will use :term:`OBS_WINDOW_END`. A corresponding variable exists for forecast data called :term:`FCST_WINDOW_END`.
 
      | *Used by:*  PB2NC, PointStat
 
@@ -3317,12 +3317,12 @@ METplus Configuration Glossary
 
      | *Used by:*  PB2NC
 
-   PB2NC_WINDOW_BEGIN
+   PB2NC_OBS_WINDOW_BEGIN
      Passed to the pb2nc MET config file to determine the range of data within a file that should be used for processing.Units are seconds. If the variable is not set, pb2nc will use :term:`OBS_WINDOW_BEGIN`.
 
      | *Used by:*  PB2NC
 
-   PB2NC_WINDOW_END
+   PB2NC_OBS_WINDOW_END
      Passed to the pb2nc MET config file to determine the range of data within a file that should be used for processing. Units are seconds. If the variable is not set, pb2nc will use :term:`OBS_WINDOW_END`.
 
      | *Used by:*  PB2NC
@@ -3649,6 +3649,20 @@ METplus Configuration Glossary
 
      | *Used by:*  SeriesAnalysis
 
+   BOTH_SERIES_ANALYSIS_INPUT_DIR
+     Specify the directory to read forecast and observation input from the
+     same file in SeriesAnalysis.
+     See also :term:`BOTH_SERIES_ANALYSIS_INPUT_TEMPLATE`
+
+     | *Used by:*  SeriesAnalysis
+
+   BOTH_SERIES_ANALYSIS_INPUT_TEMPLATE
+     Template to find forecast and observation input from the
+     same file in SeriesAnalysis.
+     See also :term:`BOTH_SERIES_ANALYSIS_INPUT_DIR`
+
+     | *Used by:*  SeriesAnalysis
+
    SERIES_ANALYSIS_OUTPUT_DIR
      Specify the directory where files will be written from the MET series analysis tool.
 
@@ -8960,3 +8974,74 @@ METplus Configuration Glossary
      See also :term:`SERIES_ANALYSIS_CLIMO_STDEV_USE_FCST`.
 
      | *Used by:* SeriesAnalysis
+
+   FCST_SERIES_ANALYSIS_INPUT_FILE_LIST
+     Specifies an explicit path to a file list file to pass into
+     series_analysis with the -fcst argument. If set,
+     :term:`OBS_SERIES_ANALYSIS_INPUT_FILE_LIST` must also be set and
+     :term:`FCST_SERIES_ANALYSIS_INPUT_TEMPLATE` and
+     :term:`FCST_SERIES_ANALYSIS_INPUT_DIR` are ignored.
+     See also :term:`BOTH_SERIES_ANALYSIS_INPUT_FILE_LIST`.
+
+     | *Used by:* SeriesAnalysis
+
+   OBS_SERIES_ANALYSIS_INPUT_FILE_LIST
+     Specifies an explicit path to a file list file to pass into
+     series_analysis with the -fcst argument. If set,
+     :term:`FCST_SERIES_ANALYSIS_INPUT_FILE_LIST` must also be set and
+     :term:`OBS_SERIES_ANALYSIS_INPUT_TEMPLATE` and
+     :term:`OBS_SERIES_ANALYSIS_INPUT_DIR` are ignored.
+     See also :term:`BOTH_SERIES_ANALYSIS_INPUT_FILE_LIST`.
+
+     | *Used by:* SeriesAnalysis
+
+   BOTH_SERIES_ANALYSIS_INPUT_FILE_LIST
+     Specifies an explicit path to a file list file to pass into
+     series_analysis with the -both argument. If set,
+     :term:`BOTH_SERIES_ANALYSIS_INPUT_TEMPLATE` and
+     :term:`BOTH_SERIES_ANALYSIS_INPUT_DIR` are ignored.
+     See also :term:`FCST_SERIES_ANALYSIS_INPUT_FILE_LIST` and
+     :term:`OBS_SERIES_ANALYSIS_INPUT_FILE_LIST`.
+
+     | *Used by:* SeriesAnalysis
+
+   FCST_MTD_INPUT_FILE_LIST
+     Specifies an explicit path to a file list file to pass into
+     mtd with the -fcst or -single argument. If set,
+     :term:`FCST_MTD_INPUT_TEMPLATE` and
+     :term:`FCST_MTD_INPUT_DIR` are ignored.
+     See also :term:`OBS_MTD_INPUT_FILE_LIST`.
+
+     | *Used by:* MTD
+
+   OBS_MTD_INPUT_FILE_LIST
+     Specifies an explicit path to a file list file to pass into
+     mtd with the -obs or -single argument. If set,
+     :term:`OBS_MTD_INPUT_TEMPLATE` and
+     :term:`OBS_MTD_INPUT_DIR` are ignored.
+     See also :term:`FCST_MTD_INPUT_FILE_LIST`.
+
+     | *Used by:* MTD
+
+   TC_RMW_INPUT_FILE_LIST
+     Specifies an explicit path to a file list file to pass into tc_rmw.
+     If set, :term:`TC_RMW_INPUT_TEMPLATE` and :term:`TC_RMW_INPUT_DIR`
+     are ignored.
+
+     | *Used by:* TCRMW
+
+   FCST_ENSEMBLE_STAT_INPUT_FILE_LIST
+     Specifies an explicit path to a file list file to pass ensembles into
+     ensemble_stat. If set,
+     :term:`FCST_ENSEMBLE_STAT_INPUT_TEMPLATE` and
+     :term:`FCST_ENSEMBLE_STAT_INPUT_DIR` are ignored.
+
+     | *Used by:* EnsembleStat
+
+   GEN_ENS_PROD_INPUT_FILE_LIST
+     Specifies an explicit path to a file list file to pass ensembles into
+     gen_ens_prod. If set,
+     :term:`GEN_ENS_PROD_INPUT_TEMPLATE` and
+     :term:`GEN_ENS_PROD_INPUT_DIR` are ignored.
+
+     | *Used by:* GenEnsProd

docs/Users_Guide/wrappers.rst

@@ -182,6 +182,7 @@ METplus Configuration
 | :term:`OBS_ENSEMBLE_STAT_POINT_INPUT_TEMPLATE`
 | :term:`OBS_ENSEMBLE_STAT_GRID_INPUT_TEMPLATE`
 | :term:`FCST_ENSEMBLE_STAT_INPUT_TEMPLATE`
+| :term:`FCST_ENSEMBLE_STAT_INPUT_FILE_LIST`
 | :term:`ENSEMBLE_STAT_OUTPUT_TEMPLATE`
 | :term:`ENSEMBLE_STAT_CTRL_INPUT_DIR`
 | :term:`ENSEMBLE_STAT_CTRL_INPUT_TEMPLATE`
@@ -1031,6 +1032,7 @@ METplus Configuration
 
 | :term:`GEN_ENS_PROD_INPUT_DIR`
 | :term:`GEN_ENS_PROD_INPUT_TEMPLATE`
+| :term:`GEN_ENS_PROD_INPUT_FILE_LIST`
 | :term:`GEN_ENS_PROD_CTRL_INPUT_DIR`
 | :term:`GEN_ENS_PROD_CTRL_INPUT_TEMPLATE`
 | :term:`GEN_ENS_PROD_OUTPUT_DIR`
@@ -4598,6 +4600,8 @@ METplus Configuration
 | :term:`MTD_OUTPUT_DIR`
 | :term:`FCST_MTD_INPUT_TEMPLATE`
 | :term:`OBS_MTD_INPUT_TEMPLATE`
+| :term:`FCST_MTD_INPUT_FILE_LIST`
+| :term:`OBS_MTD_INPUT_FILE_LIST`
 | :term:`MTD_OUTPUT_TEMPLATE`
 | :term:`MTD_CONFIG_FILE`
 | :term:`MTD_MIN_VOLUME`
@@ -4876,8 +4880,8 @@ METplus Configuration
 | :term:`PB2NC_TIME_SUMMARY_END`
 | :term:`PB2NC_TIME_SUMMARY_VAR_NAMES`
 | :term:`PB2NC_TIME_SUMMARY_TYPES`
-| :term:`PB2NC_WINDOW_BEGIN`
-| :term:`PB2NC_WINDOW_END`
+| :term:`PB2NC_OBS_WINDOW_BEGIN`
+| :term:`PB2NC_OBS_WINDOW_END`
 | :term:`PB2NC_VALID_BEGIN`
 | :term:`PB2NC_VALID_END`
 | :term:`PB2NC_CUSTOM_LOOP_LIST`
@@ -4958,9 +4962,9 @@ see :ref:`How METplus controls MET config file settings<metplus-control-met>`.
 
    * - METplus Config(s)
      - MET Config File
-   * - :term:`PB2NC_WINDOW_BEGIN`
+   * - :term:`PB2NC_OBS_WINDOW_BEGIN`
      - obs_window.beg
-   * - :term:`PB2NC_WINDOW_END`
+   * - :term:`PB2NC_OBS_WINDOW_END`
      - obs_window.end
 
 **${METPLUS_MASK_DICT}**
@@ -5936,10 +5940,15 @@ METplus Configuration
 | :term:`SERIES_ANALYSIS_MET_CONFIG_OVERRIDES`
 | :term:`FCST_SERIES_ANALYSIS_INPUT_DIR`
 | :term:`OBS_SERIES_ANALYSIS_INPUT_DIR`
+| :term:`BOTH_SERIES_ANALYSIS_INPUT_DIR`
 | :term:`SERIES_ANALYSIS_TC_STAT_INPUT_DIR`
 | :term:`SERIES_ANALYSIS_OUTPUT_DIR`
 | :term:`FCST_SERIES_ANALYSIS_INPUT_TEMPLATE`
 | :term:`OBS_SERIES_ANALYSIS_INPUT_TEMPLATE`
+| :term:`BOTH_SERIES_ANALYSIS_INPUT_TEMPLATE`
+| :term:`FCST_SERIES_ANALYSIS_INPUT_FILE_LIST`
+| :term:`OBS_SERIES_ANALYSIS_INPUT_FILE_LIST`
+| :term:`BOTH_SERIES_ANALYSIS_INPUT_FILE_LIST`
 | :term:`SERIES_ANALYSIS_TC_STAT_INPUT_TEMPLATE`
 | :term:`SERIES_ANALYSIS_OUTPUT_TEMPLATE`
 | :term:`SERIES_ANALYSIS_CLIMO_MEAN_FILE_NAME`
@@ -7894,6 +7903,7 @@ METplus Configuration
 | :term:`TC_RMW_OUTPUT_DIR`
 | :term:`TC_RMW_DECK_TEMPLATE`
 | :term:`TC_RMW_INPUT_TEMPLATE`
+| :term:`TC_RMW_INPUT_FILE_LIST`
 | :term:`TC_RMW_OUTPUT_TEMPLATE`
 | :term:`LOG_TC_RMW_VERBOSITY`
 | :term:`TC_RMW_CONFIG_FILE`

internal_tests/pytests/series_analysis/test_series_analysis.py

@@ -692,9 +692,9 @@ def test_get_all_files_and_subset(metplus_config, time_info, expect_fcst_subset,
          ]),
     ]
 )
-def test_create_ascii_storm_files_list(metplus_config, config_overrides,
-                                       time_info, storm_id, lead_group,
-                                       expect_fcst_subset, expect_obs_subset):
+def test_get_fcst_and_obs_path(metplus_config, config_overrides,
+                               time_info, storm_id, lead_group,
+                               expect_fcst_subset, expect_obs_subset):
     all_config_overrides = {
         'LOOP_BY': 'INIT',
         'SERIES_ANALYSIS_RUNTIME_FREQ': 'RUN_ONCE',
@@ -757,9 +757,9 @@ def test_create_ascii_storm_files_list(metplus_config, config_overrides,
     if os.path.exists(obs_file_path):
         os.remove(obs_file_path)
 
-    fcst_path, obs_path = wrapper._create_ascii_storm_files_list(time_info,
-                                                                 storm_id,
-                                                                 lead_group)
+    fcst_path, obs_path = wrapper._get_fcst_and_obs_path(time_info,
+                                                         storm_id,
+                                                         lead_group)
     assert(fcst_path == fcst_file_path and obs_path == obs_file_path)
 
     with open(fcst_file_path, 'r') as file_handle:

metplus/wrappers/command_builder.py

@@ -831,11 +831,6 @@ def find_input_files_ensemble(self, time_info):
             @param time_info dictionary containing timing information
             @returns True on success
         """
-        # get list of ensemble files to process
-        input_files = self.find_model(time_info, return_list=True)
-        if not input_files:
-            self.log_error("Could not find any input files")
-            return False
 
         # get control file if requested
         if self.c_dict.get('CTRL_INPUT_TEMPLATE'):
@@ -847,6 +842,28 @@ def find_input_files_ensemble(self, time_info):
 
             self.args.append(f'-ctrl {ctrl_file}')
 
+        # if explicit file list file is specified, use it and
+        # bypass logic to error check model files
+        if self.c_dict.get('FCST_INPUT_FILE_LIST'):
+            file_list_path = do_string_sub(self.c_dict['FCST_INPUT_FILE_LIST'],
+                                           **time_info)
+            self.logger.debug(f"Explicit file list file: {file_list_path}")
+            if not os.path.exists(file_list_path):
+                self.log_error("Could not find file list file")
+                return False
+
+            self.infiles.append(file_list_path)
+            return True
+
+        # get list of ensemble files to process
+        input_files = self.find_model(time_info, return_list=True)
+        if not input_files:
+            self.log_error("Could not find any input files")
+            return False
+
+        # if control file is requested, remove it from input list
+        if self.c_dict.get('CTRL_INPUT_TEMPLATE'):
+
             # check if control file is found in ensemble list
             if ctrl_file in input_files:
                 # warn and remove control file if found

metplus/wrappers/ensemble_stat_wrapper.py

@@ -172,14 +172,14 @@ def create_c_dict(self):
           self.config.getdir('OBS_ENSEMBLE_STAT_POINT_INPUT_DIR', '')
 
         c_dict['OBS_POINT_INPUT_TEMPLATE'] = \
-          self.config.getraw('filename_templates',
+          self.config.getraw('config',
                              'OBS_ENSEMBLE_STAT_POINT_INPUT_TEMPLATE')
 
         c_dict['OBS_GRID_INPUT_DIR'] = \
           self.config.getdir('OBS_ENSEMBLE_STAT_GRID_INPUT_DIR', '')
 
         c_dict['OBS_GRID_INPUT_TEMPLATE'] = \
-          self.config.getraw('filename_templates',
+          self.config.getraw('config',
                              'OBS_ENSEMBLE_STAT_GRID_INPUT_TEMPLATE')
 
         # The ensemble forecast files input directory and filename templates
@@ -189,8 +189,13 @@ def create_c_dict(self):
         c_dict['FCST_INPUT_TEMPLATE'] = (
             self.config.getraw('config', 'FCST_ENSEMBLE_STAT_INPUT_TEMPLATE')
         )
-        if not c_dict['FCST_INPUT_TEMPLATE']:
-            self.log_error("Must set FCST_ENSEMBLE_STAT_INPUT_TEMPLATE")
+        c_dict['FCST_INPUT_FILE_LIST'] = (
+            self.config.getraw('config', 'FCST_ENSEMBLE_STAT_INPUT_FILE_LIST')
+        )
+        if (not c_dict['FCST_INPUT_TEMPLATE'] and
+                not c_dict['FCST_INPUT_FILE_LIST']):
+            self.log_error("Must set FCST_ENSEMBLE_STAT_INPUT_TEMPLATE or "
+                           "FCST_ENSEMBLE_STAT_INPUT_FILE_LIST")
 
         c_dict['OUTPUT_DIR'] = self.config.getdir('ENSEMBLE_STAT_OUTPUT_DIR',
                                                   '')
@@ -475,84 +480,6 @@ def get_all_field_info(self, var_list, data_type):
 
         return ','.join(field_list)
 
-
-    def find_model_members(self, time_info):
-        """! Finds the model member files to compare
-              Args:
-                @param time_info dictionary containing timing information
-                @rtype string
-                @return Returns a list of the paths to the ensemble model files
-        """
-        model_dir = self.c_dict['FCST_INPUT_DIR']
-        # used for filling in missing files to ensure ens_thresh check is accurate
-        fake_dir = '/ensemble/member/is/missing'
-
-        # model_template is a list of 1 or more.
-        ens_members_path = []
-
-        # get all files that exist
-        for ens_member_template in self.c_dict['FCST_INPUT_TEMPLATE']:
-            member_file = do_string_sub(ens_member_template,
-                                        **time_info)
-            expected_path = os.path.join(model_dir, member_file)
-
-            # if wildcard expression, get all files that match
-            if '?' in expected_path or '*' in expected_path:
-                wildcard_files = sorted(glob.glob(expected_path))
-                self.logger.debug('Ensemble members file pattern: {}'
-                                  .format(expected_path))
-                self.logger.debug('{} members match file pattern'
-                                  .format(str(len(wildcard_files))))
-
-                # add files to list of ensemble members
-                for wildcard_file in wildcard_files:
-                    ens_members_path.append(wildcard_file)
-            else:
-                # otherwise check if file exists
-                expected_path = util.preprocess_file(expected_path,
-                                                     self.c_dict['FCST_INPUT_DATATYPE'],
-                                                     self.config)
-
-                # if the file exists, add it to the list
-                if expected_path:
-                    ens_members_path.append(expected_path)
-                else:
-                    # add relative path to fake dir and add to list
-                    ens_members_path.append(os.path.join(fake_dir, member_file))
-                    self.logger.warning('Expected ensemble file {} not found'
-                                        .format(member_file))
-
-        # if more files found than expected, error and exit
-        if len(ens_members_path) > self.c_dict['N_MEMBERS']:
-            msg = 'Found more files than expected! ' +\
-                  'Found {} expected {}. '.format(len(ens_members_path),
-                                                  self.c_dict['N_MEMBERS']) +\
-                  'Adjust wildcard expression in [filename_templates] '+\
-                  'FCST_ENSEMBLE_STAT_INPUT_TEMPLATE or adjust [config] '+\
-                  'ENSEMBLE_STAT_N_MEMBERS. Files found: {}'.format(ens_members_path)
-            self.log_error(msg)
-            self.logger.error("Could not file files in {} for init {} f{} "
-                              .format(model_dir, time_info['init_fmt'],
-                                      str(time_info['lead_hours'])))
-            return False
-        # if fewer files found than expected, warn and add fake files
-        elif len(ens_members_path) < self.c_dict['N_MEMBERS']:
-            msg = 'Found fewer files than expected. '+\
-              'Found {} expected {}.'.format(len(ens_members_path),
-                                             self.c_dict['N_MEMBERS'])
-            self.logger.warning(msg)
-            # add fake files to list to get correct number of files for ens_thresh
-            diff = self.c_dict['N_MEMBERS'] - len(ens_members_path)
-            self.logger.warning('Adding {} fake files to '.format(str(diff))+\
-                                'ensure ens_thresh check is accurate')
-            for _ in range(0, diff, 1):
-                ens_members_path.append(fake_dir)
-
-        # write file that contains list of ensemble files
-        list_filename = time_info['init_fmt'] + '_' + \
-          str(time_info['lead_hours']) + '_ensemble.txt'
-        return self.write_list_file(list_filename, ens_members_path)
-
     def set_environment_variables(self, time_info):
         self.add_env_var("MET_OBS_ERROR_TABLE",
                          self.c_dict.get('MET_OBS_ERR_TABLE', ''))