FHZERO bug

[pjpegion]

Description

Changing the output frequency affect the resulting forecast.

To Reproduce:

What compilers/machines are you seeing this with?
Intel compiler in hera

Give explicit steps to reproduce the behavior.

1. Checkout latest ufs-weather-model
2. copy the control_p7 test to control_p7_fhzero
3. add export FHZERO=3 to control_p7_fhzero
4. run both control_p7 and control_p7_fhzero tests keeping the output

Compare the restart files, and any instantaneous field, such as ugrd, and you will see differences

Additional context

I tested this with "control", which is GFS_V16 physics, and that physics suite doesn't contain this bug.

[pjpegion]

I did some backing off of p7, and when I turn off the GWD the problem goes away.

[junwang-noaa]

Thanks Phil for identifying the feature that causes the problem. I did some test runs with control_p7 and confirmed that the results are changed when the value of fhzero is changed. @yangfanglin @mdtoy FYI.

[yangfanglin]

Is using FHZERO=6 working fine with uGWD ?

[junwang-noaa]

The instant fields should not be changed if a different FHZERO is used. Since they do, I guess there might be some issue with FHZERO related computation with uGWD

[mdtoy]

Back in May, 2021 I had noticed this problem and documented it in NCAR/ccpp-physics Issue #649 <NCAR/ccpp-physics#649> I eventually narrowed it down to the non-stationary GWD of the UGWP_v1 suite. Setting the namelist variable “do_ugwp_v1” to “false” eliminates the problem, but then we don’t have non-stationary GWD. I couldn’t see a bug on the CCPP side, and it was suggested that it was an FV3ATM issue.

…

On Nov 10, 2021, at 9:14 AM, Jun Wang ***@***.*** ***@***.***>> wrote: The instant fields should not be changed if a different FHZERO is used. Since they do, I guess there might be some issue with FHZERO related computation with uGWD — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#908 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFMODLFVOLIX5446AGAAIMTULKK57ANCNFSM5HYEWNZQ>. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[pjpegion]

@mdtoy I found the bug. The precipitation going into ugwp is from the Diagnostic container, and should be coming from the surface container. It is just a few lines switches in the meta files:

diff --git a/physics/ugwpv1_gsldrag.meta b/physics/ugwpv1_gsldrag.meta
index 64d6b0d6..86b46477 100644
--- a/physics/ugwpv1_gsldrag.meta
+++ b/physics/ugwpv1_gsldrag.meta
@@ -595,8 +595,8 @@
   kind = kind_phys
   intent = in
 [rain]
-  standard_name = lwe_thickness_of_precipitation_amount_on_dynamics_timestep
-  long_name = total rain at this time step
+  standard_name = nonnegative_lwe_thickness_of_precipitation_amount_on_dynamics_timestep
+  long_name = total precipitation amount in each time step
   units = m
   dimensions = (horizontal_loop_extent)
   type = real
diff --git a/physics/unified_ugwp.meta b/physics/unified_ugwp.meta
index 54725668..14179862 100644
--- a/physics/unified_ugwp.meta
+++ b/physics/unified_ugwp.meta
@@ -1014,8 +1014,8 @@
   kind = kind_phys
   intent = in
 [rain]
-  standard_name = lwe_thickness_of_precipitation_amount_on_dynamics_timestep
-  long_name = total rain at this time step
+  standard_name = nonnegative_lwe_thickness_of_precipitation_amount_on_dynamics_timestep
+  long_name = total precipitation amount in each time step
   units = m
   dimensions = (horizontal_loop_extent)
   type = real

[junwang-noaa]

Excellent! Thanks for fixing the bug, Phil!

[climbfuji]

Yes, thanks very much Phil, great detective work.

[mdtoy]

@pjpegion Thank you again for finding the bug. I implemented your fix to the meta data files and verified that it indeed fixes the forecast reproducibility issue associated with fhzero. That was a tough bug, and I learned something from what you found.
@climbfuji This looks like a problem that could pop up again from time to time, and that could already exist in other physics schemes. The "fhzero reproducibility" test should probably be added to the suite of Regression Tests.
@climbfuji Do you want me to issue a PR for this bug? Or is it already being handled?

[climbfuji]

@pjpegion Thank you again for finding the bug. I implemented your fix to the meta data files and verified that it indeed fixes the forecast reproducibility issue associated with fhzero. That was a tough bug, and I learned something from what you found.
@climbfuji This looks like a problem that could pop up again from time to time, and that could already exist in other physics schemes. The "fhzero reproducibility" test should probably be added to the suite of Regression Tests.

That's a good point, maybe as part of the OpnReqTest - @MinsukJi-NOAA @junwang-noaa what do you think?

@climbfuji Do you want me to issue a PR for this bug? Or is it already being handled?

Yes, please create a PR and we will include it as soon as possible. I could, for example, pull your new PR into NCAR/ccpp-physics#775, if it doesn't change the regression test results. Please state if it does or not in your PR. Thank you @mdtoy and @pjpegion .