OM_05 crashes when using some mask_tables

[nikizadehgfdl]

The global area mean diagnostics produce NaNs and cause the model to crash:

FATAL from PE   754: NaN in input field of reproducing_sum(_2d).

Image              PC                Routine            Line        Source
fms_MOM6_SIS2_com  000000000151E5D1  mpp_mod_mp_mpp_er          50  mpp_util_mpi.inc
fms_MOM6_SIS2_com  00000000006FCFF5  mom_coms_mp_repro         157  MOM_coms.F90
fms_MOM6_SIS2_com  00000000007A51E1  mom_spatial_means          39  MOM_spatial_means.F90
fms_MOM6_SIS2_com  0000000000C95F50  mom_diagnostics_m        1200  MOM_diagnostics.F90
fms_MOM6_SIS2_com  00000000007CD441  mom_mp_step_mom_          774  MOM.F90
fms_MOM6_SIS2_com  0000000000729F84  ocean_model_mod_m         581  ocean_model_MOM.F90

Where line 39 of referred to above is:

  global_area_mean = reproducing_sum( tmpForSumming ) * G%IareaT_global

This happens for all diagnostics that use the global_area_mean or global_area_integral subroutine:

ssh_ga, tosga, sosga, zos, zossq, volo

After I comment all these out of the diag_table the model crashes with a non-diagnostic instance of calling reproducing_sum:

FATAL from PE   754: NaN in input field of reproducing_sum(_2d).

fms_MOM6_SIS2_com  000000000151E5F1  mpp_mod_mp_mpp_er          50  mpp_util_mpi.inc
fms_MOM6_SIS2_com  00000000006FCFF5  mom_coms_mp_repro         157  MOM_coms.F90
fms_MOM6_SIS2_com  0000000000858727  mom_surface_forci         578  MOM_surface_forcing.F90
fms_MOM6_SIS2_com  00000000007295D7  ocean_model_mod_m         547  ocean_model_MOM.F90
fms_MOM6_SIS2_com  000000000040B972  MAIN__                   1020  coupler_main.F90

where line 578 of MOM_surface_forcing.F90 is:

      fluxes%netFWGlobalAdj = reproducing_sum(net_FW(:,:), isr, ier, jsr, jer) / CS%area_surf

[nikizadehgfdl]

The model runs fine with the same layout and no mask_table.

Not all mask_tables cause crash, only some of them.

One thing that is common to all the doomed quantities is a factor of G%areaT * G%mask2dT.
Could a bad value lurk into either of these arrays when using a mask_table that has a common boundary with the actual land mask?

[nikizadehgfdl]

Running in debug mode does not trap any such NaNs ! How does reproducing_sum do it?

The model crashes on day 5. But if I run for 4 days and restart the model it does not crash till day 7!!!

[nikizadehgfdl]

I think I might have found the root cause. The generation of mask_table takes the number of halos and assumes it to be 1 if not provided. Is 1 the correct value for 1/2 degree model?

The mask generation tool can also take a --sea_level argument and assumes 0 if not passed. Shouldn't this match the MOM6 parameter MINIMUM_DEPTH = 9.5 ?

[adcroft]

What does "number of halos" mean? Do you mean the size of the halo?

And yes, the pre-calculation of the mask by fre tools can be inconsistent with the mask that the ocean dynamically generates. MOM6 does not read a mask. There is another parameter which controls the masking to help make these things more consistent:

  call get_param(PF, mdl, "MINIMUM_DEPTH", min_depth, &
                 "If MASKING_DEPTH is unspecified, then anything shallower than\n"//&
                 "MINIMUM_DEPTH is assumed to be land and all fluxes are masked out.\n"//&
                 "If MASKING_DEPTH is specified, then all depths shallower than\n"//&
                 "MINIMUM_DEPTH but deeper than MASKING_DEPTH are rounded to MINIMUM_DEPTH.", &
                 units="m", default=0.0)
  call get_param(PF, mdl, "MASKING_DEPTH", mask_depth, &
                 "The depth below which to mask points as land points, for which all\n"//&
                 "fluxes are zeroed out. MASKING_DEPTH is ignored if negative.", &
                 units="m", default=-9999.0)

[Zhi-Liang]

Hi Niki, Did you run the experiment for 1 day to see if the experiment with mask table can reproduce the one without mask table? Greetings, Zhi

…

On Tue, Mar 13, 2018 at 7:25 PM, Niki Zadeh ***@***.***> wrote: I think I might have found the root cause. The generation of mask_table takes the number of halos and assumes it to be 1 if not provided. Is 1 the correct value for 1/2 degree model? The mask generation tool can also take a --sea_level argument and assumes 0 if not passed. Shouldn't this match the MOM6 parameter MINIMUM_DEPTH = 9.5 ? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <NOAA-GFDL#734 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFkEkAkXCVA9aZpYTy0-eCyQtt6yp2pDks5teFV8gaJpZM4SpeWG> .

[nikizadehgfdl]

I did some experiments with the --halo argument of the check_mask tool and found that if I provide it with --halo 3 (at the least) then the produced mask_table allows the model to run for the layout that was crashing. What this does is to push the masked pe boundaries away from the land boundaries which we guess is the source of the problem.

Is there a requirement that there should be pes spanning the ocean-land boundary?

[nikizadehgfdl]

Zhi, yes, when the model runs with a mask_table it reproduces the answers without a mask_table (except for some spurious values on the land).

[nikizadehgfdl]

I put a print statement in global_area_mean, it's the variable itself (ssh in this case) that prints as "NaN", the G%areaT is finite and G%mask2dT is 0 or 1 (so NaN could happen on land or sea).

[nikizadehgfdl]

MEKE%Ku first gets a NaN that propagates on. The model runs fine when I turn MEKE off. This could be related to issue #262 !

[Hallberg-NOAA]

This issue was addressed by PR NOAA-GFDL#739, and is being closed.

[nikizadehgfdl]

Just a note for future reference: Creating mask_tables with --halo 3 (check_mask --halo 3 ...) saved another SPEAR based coupled model from invalid answers with mask_tables. Without --halo 3 some land processors which are too close to land-ocean boundary get masked and cause the model not to reproduce (or even crash) the non-mask-table answers.