Flexible restart write times (restart_fh) for med, ocn, ice, wav

[NickSzapiro-NOAA]

Commit Queue Requirements:

• Fill out all sections of this template.
• All sub component pull requests have been reviewed by their code managers.
• Run the full Intel+GNU RT suite (compared to current baselines) on either Hera/Derecho/Hercules
• Commit 'test_changes.list' from previous step

---

Description:

This PR enables writing forecast hour defined restarts ("restart_fh") to have option of forecast hour restarts in addition to existing functionality like RESTART_N. Restart writes are triggered when input restart_fh forecast hours are evenly divisible by a component's timestep (internally compared in units of integer seconds) and skipped otherwise. The implementation builds on current restart_fh in MOM6 for CMEPS, MOM6, CICE, and WW3 via a new shr_is_restart_fh_mod shared module (located at UFS level in CDEPS-interface). With this, restart writes can be triggered at any forecast time via UFS configuration for these components.

Currently, space-delimited floating point forecast hours are input in model_configure attributes, like restart_fh: 0.25 2.5 6 17 24. Such an attribute is well tested (e.g., output_fh for FV3 implemented in NOAA-EMC/fv3atm#345), but it may be more natural to move attribute to ufs.configure for all components once issue with reading list of floating points is resolved (https://github.com/orgs/esmf-org/discussions/291).

Testing with cpld_control_gfsv17 confirms that restarts are bit-for-bit when (1) sharing common times with RESTART_N and (2) instead of RESTART_N. This option is exercised in the cpld_control_p8 RT for med, ocn, ice, wav with restart files added to the baselines for comparison. Baselines are otherwise unchanged.

Commit Message:

* UFSWM - Flexible restart write times (restart_fh) for med, ocn, ice, wav
  * CICE - Flexible restart write times (restart_fh)
  * CMEPS - Flexible restart write times (restart_fh)
  * MOM6 - Flexible restart write times (restart_fh)
  * WW3 - Flexible restart write times (restart_fh)

Priority:

• Normal

Git Tracking

UFSWM:

• Closes # Enable flexible writing of restart file output time in UFS coupled configuration  #2348 . Note that FV3 is separate with its own encoded restart_interval implementation

Sub component Pull Requests:

• CICE: Flexible restart write times (restart_fh) NOAA-EMC/CICE#85
• CMEPS: Flexible restart write times (restart_fh) NOAA-EMC/CMEPS#125
• MOM6: Flexible restart write times (restart_fh) NOAA-EMC/MOM6#139
• WW3: Flexible restart write times (restart_fh) NOAA-EMC/WW3#1320

UFSWM Blocking Dependencies:

• Update WW3 for PIO/netCDF restarts #2445
• WW3 cannot build pre-and-post processing jobs  #2502

---

Changes

Regression Test Changes (Please commit test_changes.list):

• Adds additional restart files to baseline

Input data Changes:

• None

Library Changes/Upgrades:

• No Updates

---

Testing Log:

• RDHPCS
  • Hera
  • Orion
  • Hercules
  • Jet
  • Gaea
  • Derecho
• WCOSS2
  • Dogwood/Cactus
  • Acorn
• CI
• opnReqTest (complete task if unnecessary)

[DeniseWorthen]

@NickSzapiro-NOAA Could you start making the required changes in WW3 also, starting with my PR branch for the netCDF restarts? That branch is NOAA-EMC/WW3#1303. That would allow all components to have this feature available at the same time, which I think would be a priority.

There are script level changes required in UWM also to enable PIO in WW3. See my UWM PR #2445.

[NickSzapiro-NOAA]

Sounds great, @DeniseWorthen!

So the "restart_fh" module is now shared at:
https://github.com/NickSzapiro-NOAA/ufs-weather-model/blob/restart_fh_marine/CDEPS-interface/ufs/cdeps_share/shr_is_restart_fh_mod.F90

For WW3, it looks like it makes sense to keep restart_fh near alarm_restart in model/src/wav_comp_nuopc.F90.
If we init in ModelSetRunClock and check in ModelAdvance, this is the same as in MOM6 and CICE.

May I check that there is no concern about conflicts between the restart strides with the netcdf output option?

[DeniseWorthen]

@NickSzapiro-NOAA Thanks. My gut instinct is that there will be no similar stride conflict. When use_restartnc is set, the code totally bypasses the current code which determines whether a restart will be written. See for example the use of the logical flag (and it's .not. variant) in w3wavemd.F90 in my feature branch.

[NickSzapiro-NOAA]

I know I've mucked this up before, but I don't think a new bl_date is needed.
Only these new files are added to the cpld_control_p8 baseline:

RESTART/20210322.090000.MOM.res.nc
RESTART/iced.2021-03-22-32400.nc 
RESTART/ufs.cpld.cpl.r.2021-03-22-32400.nc 
ufs.cpld.ww3.r.2021-03-22-32400.nc

[FernandoAndrade-NOAA]

Ok, we should be all set with the testing, I'll let sub components know to continue with merging updates.

[jiandewang]

Ok, we should be all set with the testing, I'll let sub components know to continue with merging updates.

time for me to do the MOM6 merging, right ?

[FernandoAndrade-NOAA]

Ok, we should be all set with the testing, I'll let sub components know to continue with merging updates.

time for me to do the MOM6 merging, right ?

Correct, please continue with merging on the sub component side.

[jiandewang]

MOM6 merged, hash # 8cc8b70f

[NickSzapiro-NOAA]

Thank you, @jiandewang . CICE merged, hash # ff7fd76

[JessicaMeixner-NOAA]

WW3 hash is 29063ec7b631d3a967172e8c45c5af13a18e0a82