Fix IODA2NC to handle the same input file being provided multiple times

[hsoh-u]

Replace italics below with details for this issue.

Describe the Problem

ioda2nc fails if the same input file is given with -iodafile option

Expected Behavior

It should work without errors.

Environment

Describe your runtime environment:
1. Machine: (Linux Workstation)
2. OS: (RedHat Linux)
3. Software version number(s): 10.1.0 and 11.0 beta

To Reproduce

Describe the steps to reproduce the behavior:
1. Go to seneca
2. run the following command

/usr/local/met/bin/ioda2nc -v 2 /d1/projects/METplus/METplus_Data/development/feature_1203_ioda2nc/met_test/new/ioda
/ioda.NC001007.2020031012.nc ioda.NC001007.2020031012.summary.nc
-iodafile /d1/projects/METplus/METplus_Data/development/feature_1203_ioda2nc/met_test/new/ioda/ioda.NC001007.2020031012.nc

3. See error

terminate called after throwing an instance of 'netCDF::exceptions::NcEdge'
what(): NetCDF: Start+count exceeds dimension bound
file: ncVar.cpp line:958
Aborted

4. run the following command

export METPLUS_ELEVATION_RANGE_DICT="elevation_range = {beg = -1000;end = 100000;}"
export METPLUS_LEVEL_RANGE_DICT=""
export METPLUS_MASK_DICT=""
export METPLUS_MESSAGE_TYPE=""
export METPLUS_MESSAGE_TYPE_GROUP_MAP=""
export METPLUS_MESSAGE_TYPE_MAP=""
export METPLUS_METADATA_MAP=""
export METPLUS_MET_CONFIG_OVERRIDES=""
export METPLUS_MISSING_THRESH=""
export METPLUS_OBS_NAME_MAP="obs_name_map = [{ key = "wind_direction"; val = "WDIR"; }, { key = "wind_speed"; val = "WIND"; }];"
export METPLUS_OBS_VAR=""
export METPLUS_OBS_WINDOW_DICT="obs_window = {beg = -5400;end = 5400;}"
export METPLUS_QUALITY_MARK_THRESH="quality_mark_thresh = 0;"
export METPLUS_STATION_ID=""
export METPLUS_TIME_SUMMARY_DICT="time_summary = {flag = TRUE;raw_data = TRUE;beg = "000000";end = "235959";step = 300;width = 600;grib_code = [];obs_var = ["WIND"];type = ["min", "max", "range", "mean", "stdev", "median", "p80"];vld_freq = 0;vld_thresh = 0.0;}"
export MET_TMP_DIR="/d1/personal/mccabe/out2/tmp"

/usr/local/met/bin/ioda2nc -v 2 /d1/projects/METplus/METplus_Data/development/feature_1203_ioda2nc/met_test/new/ioda
/ioda.NC001007.2020031012.nc ioda.NC001007.2020031012.summary.nc
-config /d1/personal/mccabe/METplus/parm/met_config/IODA2NCConfig_wrapped
-iodafile /d1/projects/METplus/METplus_Data/development/feature_1203_ioda2nc/met_test/new/ioda/ioda.NC001007.2020031012.nc

5. See error

ERROR :
ERROR : get_obs_data_float() -> WDIR@ObsValue does not exist!
ERROR :

Post relevant sample data following these instructions:
https://dtcenter.org/community-code/model-evaluation-tools-met/met-help-desk#ftp

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

2702691

Define the Metadata

Assignee

• Select engineer(s) or no engineer required
• Select scientist(s) or no scientist required: no scientist

Labels

• Select component(s)
• Select priority
• Select requestor(s)

Projects and Milestone

• Select Organization level Project for support of the current coordinated release
• Select Repository level Project for development toward the next official release or add alert: NEED PROJECT ASSIGNMENT label
• Select Milestone as the next bugfix version

Define Related Issue(s)

Consider the impact to the other METplus components.

• METplus, MET, METdatadb, METviewer, METexpress, METcalcpy, METplotpy

Bugfix Checklist

See the METplus Workflow for details.

• Complete the issue definition above, including the Time Estimate and Funding Source.
• Fork this repository or create a branch of main_<Version>.
  Branch name: bugfix_<Issue Number>_main_<Version>_<Description>
• Fix the bug and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Push local changes to GitHub.
• Submit a pull request to merge into main_<Version>.
  Pull request: bugfix <Issue Number> main_<Version> <Description>
• Define the pull request metadata, as permissions allow.
  Select: Reviewer(s) and Linked issues
  Select: Organization level software support Project for the current coordinated release
  Select: Milestone as the next bugfix version
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Complete the steps above to fix the bug on the develop branch.
  Branch name: bugfix_<Issue Number>_develop_<Description>
  Pull request: bugfix <Issue Number> develop <Description>
  Select: Reviewer(s) and Linked issues
  Select: Repository level development cycle Project for the next official release
  Select: Milestone as the next official version
• Close this issue.

[JohnHalleyGotway]

@hsoh-u it seems to me that this issue would be very easy to fix. Unless I'm misunderstanding, the description makes it sound like telling ioda2nc to process the same input file multiple times triggers the error. If that's the case, let's just require that the input files be unique.

So just compare the length of ioda_files to ioda_files.uniq(). If they differ, then there's at least one duplicate present and the tool could error out. Starting on line 265 of ioda2nc.cc:

   // Store the input file names
   ioda_files.add(cline[0]);
   if(cline.n() > 1) ncfile = cline[1];

   // Check for unique input file names
   if(ioda_files.n() != ioda_files.uniq().n()) {
      mlog << Error << "\n" << method_name
           << "all input files specified using \"-iodafile\" must be unique!\n\n";
      exit(1);
   }

   // Create the default config file name 
   ConcatString default_config_file = replace_path(DEF_CONFIG_NAME);

Or is it more complicated than what I'm assuming?