new test case OM4_SIS_DT_THERM_7200 does not reproduce across a restart for longer runs

[nikizadehgfdl]

1x2day = 2x1day but 1x10day != 2x5days

/// /lustre/f1/Niki.Zadeh/ulm_mom6_2015.04.20_0/OM4_SIS_DT_THERM_7200/ncrc2.intel-prod/1x0m2d_1152pe/restart/19000103.tar
\\\ /lustre/f1/Niki.Zadeh/ulm_mom6_2015.04.20_0/OM4_SIS_DT_THERM_7200/ncrc2.intel-prod/1x0m2d_2304pe/restart/19000103.tar
WARNING : NONESSENTIAL FILES DIFFER : input.nml
WARNING : NONESSENTIAL FILE NOT FOUND : OM4_SIS_DT_THERM_7200_1x0m2d_1152pe
/// /lustre/f1/Niki.Zadeh/ulm_mom6_2015.04.20_0/OM4_SIS_DT_THERM_7200/ncrc2.intel-prod/1x0m2d_1152pe/restart/19000103.tar
\\\ /lustre/f1/Niki.Zadeh/ulm_mom6_2015.04.20_0/OM4_SIS_DT_THERM_7200/ncrc2.intel-prod/2x0m1d_2304pe/restart/19000103.tar
WARNING : NONESSENTIAL FILES DIFFER : input.nml
WARNING : NONESSENTIAL FILE NOT FOUND : OM4_SIS_DT_THERM_7200_1x0m2d_1152pe
/// /lustre/f1/Niki.Zadeh/ulm_mom6_2015.04.20_0/OM4_SIS_DT_THERM_7200/ncrc2.intel-prod/1x0m10d_2997pe/restart/19000111.tar
\\\ /lustre/f1/Niki.Zadeh/ulm_mom6_2015.04.20_0/OM4_SIS_DT_THERM_7200/ncrc2.intel-prod/2x0m5d_2997pe/restart/19000111.tar
WARNING : NONESSENTIAL FILES DIFFER : input.nml
DIFFER : MOM.res_1.nc
DIFFER : MOM.res_3.nc
DIFFER : MOM.res.nc

Something must be breaking for runs longer than 2 days
OR
The non-reproducibility has something to do with the layout and mask_table.

I am going to try a 10day run with the layouts that reproduced.

[nikizadehgfdl]

This non-repro has something to do with the mask_table. The model reproduces across restarts when not using a mask_table !!

/// /lustre/f1/Niki.Zadeh/ulm_mom6_2015.04.24_0/OM4_SIS_DT_THERM_7200/ncrc2.intel-prod/1x0m10d_2304pe/restart/19000111.tar
\\\ /lustre/f1/Niki.Zadeh/ulm_mom6_2015.04.24_0/OM4_SIS_DT_THERM_7200/ncrc2.intel-prod/2x0m5d_2304pe/restart/19000111.tar
  CROSSOVER   PASSED: OM4_SIS_DT_THERM_7200

[nikizadehgfdl]

The non-repro is only in 3 restart variables h, h2 and age. They differ on a few isolated points on the map on the land !!

[nikizadehgfdl]

Given the interest in running with "spanning" option this needs to be resolved. It resembles the restart mismatch we have between pe-masked and non-pe-masked runs.

[nikizadehgfdl]

Odd enough this issue does not happen when I replace SIS1 with SIS2!
The experiment OM4_SIS2_DT_THERM_7200 passes restart tests even with mask_table.

[adcroft]

!

Dr Alistair Adcroft (Alistair.Adcroft@noaa.gov)
Princeton University Tel: (609) 987-5073
NOAA/GFDL, 201 Forrestal Road, Princeton, NJ 08540

On Wed, Aug 5, 2015 at 5:08 PM, Niki Zadeh notifications@github.com wrote:

Odd enough this issue does not happen when I replace SIS1 with SIS2!
The experiment OM4_SIS2_DT_THERM_7200 passes restart tests even with
mask_table.

—
Reply to this email directly or view it on GitHub
NOAA-GFDL#146 (comment).

[Hallberg-NOAA]

Perhaps this is yet another argument in favor of the widespread replacement of SIS with SIS2 in all new configurations... 😄

[Hallberg-NOAA]

This issue may have been addressed in the years since it was first raised, but because SIS1 has been declared obsolete and is not in use for any active MOM6 development, and this issue never arose with SIS2, we have no way to verify whether it was, in fact corrected. In any case, it is no longer relevant, and is therefore appropriate to declare this issue closed. Please reopen this as a new issue if similar problems are ever encountered with MOM6/SIS2 configurations.