GRID_STAT_SKIP_IF_OUTPUT_EXISTS not writing for all requested forecast hours

[malloryprow]

I’m running grid_stat for a single valid time over a bunch of forecast hours. I’m running with GRID_STAT_SKIP_IF_OUTPUT_EXISTS = True; however, it only produces output for the first forecast hour. It looks like GridStat is determining if output exists by using wildcards. I was hoping to use this configuration setting to not run if individual forecast hours already existed.

03/20 15:41:06.946 metplus (command_builder.py:1003) DEBUG: Skip writing output /lfs/h2/emc/stmp/mallory.row/evs_global_det_atmos_test/dev/tmp/jevs_gfs_atmos_grid2grid_stats_00.58711663.dbqs01/grid2grid_stats/METplus_output/atmos.20230319/gfs/grid2grid/grid_stat_pres_levs_GeoHeightAnom*20230319_000000V* because it already exists. Remove file or change GRID_STAT_SKIP_IF_OUTPUT_EXISTS to False to process

Relevant config file options (in my config I use environment variables but I replaced them here with their values):
OUTPUT_BASE = /lfs/h2/emc/stmp/mallory.row/evs_global_det_atmos_test/dev/tmp/jevs_gfs_atmos_grid2grid_stats_00.58711663.dbqs01/grid2grid_stats/METplus_output
GRID_STAT_OUTPUT_DIR = {OUTPUT_BASE}
GRID_STAT_OUTPUT_TEMPLATE = atmos.{valid?fmt=%Y%m%d}/gfs/grid2grid
GRID_STAT_OUTPUT_PREFIX = pres_levs_GeoHeightAnom
GRID_STAT_OUTPUT_FLAG_FHO = NONE
GRID_STAT_OUTPUT_FLAG_CTC = NONE
GRID_STAT_OUTPUT_FLAG_CTS = NONE
GRID_STAT_OUTPUT_FLAG_MCTC = NONE
GRID_STAT_OUTPUT_FLAG_MCTS = NONE
GRID_STAT_OUTPUT_FLAG_MPR = NONE
GRID_STAT_OUTPUT_FLAG_CNT = NONE
GRID_STAT_OUTPUT_FLAG_SL1L2 = NONE
GRID_STAT_OUTPUT_FLAG_SAL1L2 = NONE
GRID_STAT_OUTPUT_FLAG_VL1L2 = NONE
GRID_STAT_OUTPUT_FLAG_VAL1L2 = NONE
GRID_STAT_OUTPUT_FLAG_VCNT = NONE
GRID_STAT_OUTPUT_FLAG_PCT = NONE
GRID_STAT_OUTPUT_FLAG_PSTD = NONE
GRID_STAT_OUTPUT_FLAG_PJC = NONE
GRID_STAT_OUTPUT_FLAG_PRC = NONE
GRID_STAT_OUTPUT_FLAG_ECLV = NONE
GRID_STAT_OUTPUT_FLAG_NBRCTC = NONE
GRID_STAT_OUTPUT_FLAG_NBRCTS = NONE
GRID_STAT_OUTPUT_FLAG_NBRCNT = NONE
GRID_STAT_OUTPUT_FLAG_GRAD = NONE
GRID_STAT_OUTPUT_FLAG_DMAP = NONE
GRID_STAT_NC_PAIRS_FLAG_LATLON = TRUE
GRID_STAT_NC_PAIRS_FLAG_RAW = TRUE
GRID_STAT_NC_PAIRS_FLAG_DIFF = FALSE
GRID_STAT_NC_PAIRS_FLAG_CLIMO = FALSE
GRID_STAT_NC_PAIRS_FLAG_CLIMO_CDP = FALSE
GRID_STAT_NC_PAIRS_FLAG_WEIGHT = FALSE
GRID_STAT_NC_PAIRS_FLAG_NBRHD = FALSE
GRID_STAT_NC_PAIRS_FLAG_FOURIER = FALSE
GRID_STAT_NC_PAIRS_FLAG_GRADIENT = FALSE
GRID_STAT_NC_PAIRS_FLAG_DISTANCE_MAP = FALSE
GRID_STAT_NC_PAIRS_FLAG_APPLY_MASK = FALSE

I attached the log file
reformat_data_GridStat_fcstgfs_obsgfs_anl_climoERA5_NetCDF_grid2grid_stats_pres_levs_GeoHeightAnom_for2023031900_runon20230320154105.log

[georgemccabe]

Hi @malloryprow,

Thank you for providing this information. The logic to check if the output files already exist currently use {self.app_name}_{prefix}*{valid_format}V* as the search string, which will find any files with the prefix for a given valid time. It sounds like the check should also include the forecast lead to prevent incorrect skipping in this case. I will create a bugfix issue.

Is this a fix that should be applied to METplus 5.0.2?

In the meantime, you should be able to get around this by adding the forecast lead to the GRID_STAT_OUTPUT_PREFIX`:

GRID_STAT_OUTPUT_PREFIX = pres_levs_GeoHeightAnom_f{lead?fmt=%H}

[georgemccabe]

Here is the bugfix issue: #2096

[malloryprow]

@georgemccabe Thanks!

[malloryprow]

@georgemccabe The workaround does work. I thought it was interesting though that METplus logged an error.

03/21 12:27:08.513 metplus (config_util.py:166) ERROR: No commands were run. Skip writing all_commands file
03/21 12:27:08.514 metplus INFO: Overwrite final conf here: /lfs/h2/emc/stmp/mallory.row/evs_global_det_atmos_test/dev/tmp/jevs_gfs_atmos_grid2grid_stats_00.58786843.dbqs01/grid2grid_stats/METplus_output/confs/reformat_data_GridStat_fcstgfs_obsgfs_anl_climoERA5_NetCDF_grid2grid_stats_pres_levs_GeoHeightAnom_for2023032000_runon20230321122708.conf
03/21 12:27:08.516 metplus (run_util.py:188) DEBUG: METplus took 0:00:00.516172 to run.
03/21 12:27:08.516 metplus (run_util.py:203) ERROR: METplus has finished running as user mallory.row(32288) but had 1 error.

We were hoping to use this method to help with restart capabilities in EVS, but in operations we can't have error messages.

[georgemccabe]

Yes, the run fails if no commands are run because it was assumed that something was set up incorrectly when that happens.

To get around this for now, you could add a UserScript instance to the process list that simply runs "ls" on the output directory.

Any suggestions for how this should be improved? A few initial thoughts:

1. Change the error to a warning if no commands were run.
2. Add a config option to prevent an error if no commands were run.

[georgemccabe]

We discussed this in the METplus Wrappers engineering meeting today. @j-opatz suggested that we should change the message that is output if no commands were run to INFO level. I like this idea. If anything goes wrong in the building or running of commands, an error will still be reported if this error message was removed. If needed, we could add a config option to force the run to error out if no commands are run that users could set if they want to be informed of this case.

Does anyone else at EMC or elsewhere have any strong feelings about how this should behave? @LoganDawson-NOAA? @PerryShafran-NOAA? @robdarvell? Please tag or reach out to anyone else who may want to weigh in.

[LoganDawson-NOAA]

@georgemccabe I agree with @j-opatz's suggestion of running this message at the INFO level instead of the ERROR level. I also think it would be good to give the users a config option to print an error, if desired. I think that option would be desirable in instances like someone developing a new use case and needing to know that no commands were built or run.

[robdarvell]

How we operate within our system in the UK at present is we have the two main options, which are either repeat rows into our results file (which then can be housekept to store the most recent entry (you can't select which to keep)) or fail the job saying you are trying to over-write output.
There are occasions where you would want to be informed that you are trying to over-write something which already exists, and I think for most of us that would preferably be with a failed job than trying to scroll through output seeing whether a message appears in the output. Most people (especially in our trialling community) just want to set the job off and forget about it until it finishes (without really checking the output), so knowing something is (possibly) wrong at an early stage (ie through a failure) is their preferred approach.
In conclusion, it might be that this variable perhaps comes with 3 options (as has been mentioned above) rather than perhaps a logical true/false where a user could specify OUTPUT_EXISTS={TrueError/TrueInfo/False}, then the TrueError would fail the job and TrueInfo would just write the info message and continue.

[georgemccabe]

@robdarvell, an easy enhancement to achieve that behavior could be to add a <tool>_FAIL_IF_OUTPUT_EXISTS boolean in addition to <tool>_SKIP_IF_OUTPUT_EXISTS that will cause an error if the desired output file already exists. I created issue #2110 to track this enhancement.

Also, we have an (old) open issue #196 to add an option to stop a run immediately if an error occurs. It is currently labeled as low priority but could be elevated if desired. Please let us know if so.