flexible restart file update breaks IAU in DA

[jswhit]

PR #691 adds the iau_offset to the restart_interval specified in model_configure to determine when restarts are dumped. Currently the DA cycle uses iau_offset=6 and restart_interval=6, which means that restarts will be dumped 12 hours into the forecast instead of 6. In fact, there appears to be no way to get a restart dumped 6 hours into the forecast since that would require setting restart_interval=0, which the model interprets as a restart at the end of the forecast.

Suggested fix: remove the logic that adds iau_offset to the restart time (around line 366 here)

@RussTreadon-NOAA

[junwang-noaa]

Jeff, In the PR691, we are trying to make restart_interval and output_fh consistent, as I was told it's confusing to have restart time different from output time. So in the IAU case, since it is a restart run from previous cycle, but we treat it as a forecast from current cycle, which means, the restart from fh=0 (or the output_fh at fh=0) now means to write out restart file (or history file) forecast hour 0 in current cycle (or the previous cycle + iauoffset=6), similarly restart_time at 6hr or output_fh=6 means to write restart file or history file at forecast time fh=6 in current cycle (it's also previous cycle +12h). So if possible, please update the restart_interval to 0, 3,6,9. Also now restart interval (6, -1) means it's restart frequency, while (6) means just to write out restart at fh=6 in the current cycle, this is also consistent with output_fh. This change was specified in the description of PR#691, item 3).

[junwang-noaa]

@jswhit I talked to Rahul. workflow will be updated with this model change, basically the restart interval for DA will be unified to "3,6" with or without IAU. Please let me know if you still have issues.

[jswhit]

@junwang-noaa @aerorahul I don't think that will work - I think you need 0,3 not 3,6 (since iau_offset=6 is added)? Testing now..

[jswhit]

Don't think the logic for restart frequency works with nonzero iau_offset. If I specify "3,-1" with iauoffset=6 and nhours_fcst=15 (for a forecast that whose initial time is 3 hours before the current start time) I get frestart=9 and a restart is not written until 9 hours into the forecast (fhour 12) instead of 3 hours in (fhour=6).

[aerorahul]

@jswhit
Should the iau_offset be 3, since it is an offset from the "analysis time" (middle of the 6 hour window)?
I am not sure if that is what iau_offset is supposed to mean.

Per the conversation with @junwang-noaa this AM, the restart_interval is now relative to the "analysis time", and thus restart_interval=3 would produce the restarts at 3 hours past the analysis time, which is the beginning of the next assimilation window.

The restart_interval in this case refers to the explicit time (relative to the "analysis time") at which the restarts will be written out.

[jswhit]

@aerorahul the forecast start time is the previous nominal analysis time, and the initial time is at the beginning of the assim window (so 3 hours into the forecast). The nominal analysis time is 6 hours into the forecast, so iau_offset=6 so that a fh=0 history file is dumped at that time.

[jswhit]

setting restart_interval=3 does not produce a restart at the nominal analysis time, since iau_offset is added you get a restart at the next analysis time (the middle of the next assimilation window). What we want is a restart at the end of the current assimilation window (which would appear to require restart_interval=-3)

[jswhit]

Setting restart_interval="0,3" does not work either - you end up with a restart at the end of the forecast only (the end of the next assimilation window). With the current logic there does not appear to be any way to dump a restart 6 hours into the forecast (at fhour=9) with iau_offset=6.

[junwang-noaa]

Maybe I misunderstood the requirement of IAU. So for gdas run without IAU, do we need to write out any restart files, if yes, at what time? With IAU run, we need to write out fh=3 and 6 relative to current cycle, is that correct?

…

On Thu, Sep 2, 2021 at 1:41 PM Jeff Whitaker ***@***.***> wrote: Setting restart_interval="0,3" does not work either - you end up with a restart at the end of the forecast only (the end of the next assimilation window). With the current logic there does not appear to be any way to dump a restart 6 hours into the forecast with iau_offset=6. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#785 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AI7D6TMFQI4OR4CJ6PCDSYLT76ZMNANCNFSM5DH4AVRQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[jswhit]

Yes, we need restart files regardless of whether we are using IAU (at 6 hours into the forecast for both IAU and no-IAU). For the no-IAU case this should produce a restart at the nominal analysis time for the next analysis window, for the IAU case it should produce a restart at the beginning of the next analysis window (nominal analysis time - 3h).

I think the fundamental problem is that FMS writes the restarts, and it does not know anything about the iau_offset. The iau_offset logic is only applied to the history files, so that F00 is at the nominal analysis time. I have created a patch that removes iau_offset from the frestart logic and things seem to be working again. I can create a PR if need be.

[junwang-noaa]

Actually that causes lots of confusion on setting the forecast time on the history files and restart files with IAU. Because the restart time is from the current start time (fh=-3) and we use the bucket time=6 for accumulation fields for many applications, people can't use restart interval frequency for restart writing time. Those are the reasons we hope to keep forecast time for history files and restart files consistent. I see your fix will change the restart writing time back to previous one, but that will continue the forecast time inconsistency between history/restart files.

[junwang-noaa]

Yes, FMS does not need to know about the IAU, frestart controls the restart writing time. The iau_offset is only used to specify how the configuration variable "restart_interval" is defined. The issue is with IAU offset (6) and bucket hour 6, the restart interval breaks reproducibility with previous implementation, please let me know if this is not true. I fixed the issue of restart interval (3 6), (0)[write out at the end of forecast], (0 -1) [write at forecast time 0 relative to current cycle) with IAU.

[jswhit]

@junwang-noaa I don't think the change to the restart_interval logic achieves what you want to achieve. For the history files, the iau_offset is applied to the time since the original start time of the forecast, so iau_offset=6 causes F09 to become F03 For the restart times, you are applying iau_offset to the time since the initialization of the current forecast, so with iau_offset=6 and restart_interval=3, the first restart is written 9 hours into the forecast (instead of 6).

[junwang-noaa]

@jswhit To clearify, with iau_offset=6 and restart_interval=3, the restart is written after 6 hour integration. Please see:
/scratch1/NCEPDEV/stmp2/Jun.Wang/FV3_RT/rt_152668/control_c384gdastest1/RESTART

The time stamps for restart files are saved in coupler$RESTART_WRITTEN_TIME.res, with model initialization time (which is previous cycle time) and current model time when restart is written. It does not know if it is an IAU run or not. With the code changes I proposed, we can still use restart frequency feature for long forecast IAU runs. E.g. set restart_interval: 24 -1 (second element -1 means frequency) to write out restart files every 24 hours relative to current cycle time. This feature was broken with previous coding for IAU.

[jswhit]

@junwang-noaa Here's what I am seeing that is problematic for the GDAS with IAU. I have a coupler.res with this:

    2        (Calendar: no_calendar=0, thirty_day_months=1, julian=2, gregorian=3, noleap=4)
  2015  12  05  18     0     0        Model start time:   year, month, day, hour, minute, second
  2015  12  05  21     0     0        Current model time: year, month, day, hour, minute, second

and model_configure with this

start_year:              2015
start_month:             12
start_day:               05
start_hour:              18
start_minute:            0
start_second:            0
nhours_fcst:             15
restart_interval:        3 -1
iau_offset :                6

The nominal analysis time is 2015120600 and I need to dump a restart at 2015120600 and 2015120603 (3 and 6 hours into the integration, where history file will be labelled f000 and f006). However, I end up with

frestart= 9 12 15 0 0

and the model dumps a restart at 2015120606.

ls -l RESTART/*fv_core*tile1*

-rw-r--r-- 1 Jeffrey.S.Whitaker esrl 684110455 Sep  3 15:32 RESTART/20151206.060000.fv_core.res.tile1.nc
-rw-r--r-- 1 Jeffrey.S.Whitaker esrl 684343647 Sep  3 15:42 RESTART/fv_core.res.tile1.nc

This is the problem I'm having and the only solution I have found is to revert your changes to how frestart is calculated.

Seems like frestart is hours relative to the current time at the start of the forecast, not hours relative to the initial forecast start time.

[junwang-noaa]

@jswhit Sorry, I fotgot to mention, as you noticed there are some issues with IAU restart in current develop branch. I made an fv3 PR380, and the runs I made are with the fix. In this PR frestart is the forecast time relative to the model start time, meanwhile:

1. restart_interval: 0 will write out restart at the end of forecast, as before
2. restart_interval: 0 -1 will write out restart at forecast hour 0 relative to current cycle (3 hour into integration) only,
3. restart_interval: 3 -1 will write out restart at forecast hour 3, 6, 9 relative to current cycle.
   e.g. with the test run:

start_year:              2021
start_month:             03
start_day:               21
start_hour:              18
start_minute:            0
start_second:            0
nhours_fcst:             15
fhrot:                   3

dt_atmos:                225
calendar:                'julian'
restart_interval:        3 -1

frestart=           9          12          15           0           0 

20210322.030000.coupler.res  20210322.060000.coupler.res  coupler.res

4) restart_interval: 0 3 will write out the restart at forecast hour 0 and 3 relative to current cycle time. 
5) When forecast length is set to 48, restart_interval: 24 -1 will write out restart at fh=24 and 48 relative to current cycle.

[jswhit]

Thank you @junwang-noaa - I can confirm that your PR works as advertised. This should solve all the problems I was having with restarts and iau_offeset.

[junwang-noaa]

Thanks for the conformation. We plan to commit this PR next week, will let you know when it is done.

[junwang-noaa]

@jswhit @KateFriedman-NOAA The restart_interval fix for IAU is committed in PR#782

[KateFriedman-NOAA]

@junwang-noaa Awesome, thanks for the update!