Time Shifting not working in Stat Analysis

[CPKalb]

Hi:

I am having a struggle trying to get time shifting to work in Stat Analysis. Previously, this was working from a version of METplus on the computer atmosphere. However, checking out the latest version of develop and main, it fails. The error I get is as follows:
05/06 03:45:28.219Z metplus.StatAnalysis.sanal_lead (run_util.py:166) ERROR: Fatal error occurred
Traceback (most recent call last):
File "/home/aspire/kalb/METplus/metplus/util/run_util.py", line 143, in run_metplus
new_commands = process.run_all_times()
File "/home/aspire/kalb/METplus/metplus/wrappers/runtime_freq_wrapper.py", line 115, in run_all_times
self.run_all_times_custom(custom_string)
File "/home/aspire/kalb/METplus/metplus/wrappers/runtime_freq_wrapper.py", line 126, in run_all_times_custom
self.run_once(custom)
File "/home/aspire/kalb/METplus/metplus/wrappers/runtime_freq_wrapper.py", line 153, in run_once
return self.run_at_time_once(time_input)
File "/home/aspire/kalb/METplus/metplus/wrappers/stat_analysis_wrapper.py", line 238, in run_at_time_once
self._run_stat_analysis(time_input)
File "/home/aspire/kalb/METplus/metplus/wrappers/stat_analysis_wrapper.py", line 246, in _run_stat_analysis
runtime_settings_dict_list = self._get_all_runtime_settings(time_input)
File "/home/aspire/kalb/METplus/metplus/wrappers/stat_analysis_wrapper.py", line 275, in _get_all_runtime_settings
model_info = self._get_model_obtype_and_lookindir(runtime_settings)
File "/home/aspire/kalb/METplus/metplus/wrappers/stat_analysis_wrapper.py", line 1333, in _get_model_obtype_and_lookindir
self._get_lookin_dir(model_info['dir'], runtime_settings_dict)
File "/home/aspire/kalb/METplus/metplus/wrappers/stat_analysis_wrapper.py", line 961, in _get_lookin_dir
**stringsub_dict)
File "/home/aspire/kalb/METplus/metplus/util/string_template_substitution.py", line 359, in do_string_sub
skip_missing_tags)
File "/home/aspire/kalb/METplus/metplus/util/string_template_substitution.py", line 403, in find_and_replace_tags_in_template
shift_seconds = get_seconds_from_template(split_string, SHIFT_STRING, kwargs)
File "/home/aspire/kalb/METplus/metplus/util/string_template_substitution.py", line 232, in get_seconds_from_template
valid_time=valid_time))
File "/home/aspire/kalb/METplus/metplus/util/time_util.py", line 106, in get_seconds_from_string
return ti_get_seconds_from_relativedelta(rd_obj, valid_time)
File "/home/aspire/kalb/METplus/metplus/util/time_util.py", line 163, in ti_get_seconds_from_relativedelta
return int((valid_time - (valid_time - lead)).total_seconds())
TypeError: unsupported operand type(s) for -: 'str' and 'relativedelta'
05/06 03:45:28.220Z metplus.StatAnalysis.sanal_lead INFO: Check the log file for more information: /data1/work/aspire/stats/logs/metplus.log.20230506034528
05/06 03:45:28.221Z metplus INFO: Overwrite final conf here: /data1/work/aspire/stats/metplus_final_CTREC.conf
05/06 03:45:28.236Z metplus INFO: METplus took 0:00:00.236548 to run.
05/06 03:45:28.236Z metplus (run_util.py:216) ERROR: METplus has finished running as user aspire(10424) but had 1 error.
05/06 03:45:28.236Z metplus INFO: Check the log file for more information: /data1/work/aspire/stats/logs/metplus.log.20230506034528

The offending line in the METplus configuration:MODEL1_STAT_ANALYSIS_LOOKIN_DIR = {OUTPUT_BASE}/CTREC/grid_stat/{init_end?fmt=%Y%m%d?shift=-86400},{OUTPUT_BASE}/CTREC/grid_stat/{init_beg?fmt=%Y%m%d}

I tried pulling the version of METplus I have installed on atmosphere, and the error does not happen. Should time shifting work in the Stat Analysis wrapper?

Configuration files:
GridStat_fcstCTREC_obsRadar_refc.conf.txt

Christina

[georgemccabe]

It looks like the time value is stored as a string instead of a datetime object. I am actually surprised that it used to work in a previous version, because from what I remembered the StatAnalysis wrapper was previously storing all of the additional time template values as strings. Which version of METplus is installed on atmosphere? This will require some code changes to properly store any date value as a datetime object instead of a string.

[CPKalb]

Hi George:
The version on atmosphere is from a branch. How do I determine what version this is, or get the needed information?

Tina

[georgemccabe]

The version number is output at the beginning of the logs. It is also stored in the metplus/VERSION file. It is likely a version number with the -dev suffix to signify that it is from the development cycle working towards the version listed.

[CPKalb]

This is what is says on atmosphere:

Running METplus v5.0.0-beta5-dev

[georgemccabe]

Thanks, @CPKalb. I confirmed that there is a bug in the logic. I created issue #2168 to fix this. I made a fix off of the develop branch in bugfix_2168_develop_time_shift. Could you test that this fixes the issue for your use case? Also, do we need a bugfix for METplus 5.0 or is adding it for the upcoming 5.1.0 release sufficient?

[CPKalb]

Thanks! I will test the branch.

When are we expected to release METplus 5.1.0?

Tina

[georgemccabe]

At the end of June

[CPKalb]

It's probably fine if we just add it to 5.1.0 then. I tested the branch and it is working as expected.

Tina

[georgemccabe]

Great, thank you for confirming!