test io layout for restart

[junwang-noaa]

It is found that writing restart files is very slow in C768L127 runs. So far we are using io_layout = 1,1 in namelist input.nml. Tests have been done to use different layout to reduce the write time. FV3 GFS restart test will be updated to use non-1 io_layout.

[junwang-noaa]

Some test results show that using different io layout reduces the restart file writing time, therefore reduce the total model run time.

Control: Io layout:
&fv_core_nml
layout = 16,12
io_layout = 1,1
npx = 769
npy = 769
ntiles = 6
npz = 127

Results: /gpfs/dell1/ptmp/Jun.Wang/v16rt4_gdasfcst_06.run6
PASS: fcstRUN phase 1, na = 142 time is 5.97872209548950
PASS: fcstRUN phase 2, na = 142 time is 0.175957918167114
PASS: fcstRUN phase 1, na = 143 time is 5.89624500274658
PASS: fcstRUN phase 2, na = 143 time is 291.485043048859
PASS: fcstRUN phase 1, na = 144 time is 16.1452939510345
PASS: fcstRUN phase 2, na = 144 time is 1.70220208168030

Io layout:
&fv_core_nml
layout = 16,12
io_layout = 4,4
npx = 769
npy = 769
ntiles = 6
npz = 127

/gpfs/dell1/ptmp/Jun.Wang/v16rt4_gdasfcst_06.run3:
PASS: fcstRUN phase 1, na = 142 time is 5.97238397598267
PASS: fcstRUN phase 2, na = 142 time is 0.172916889190674
PASS: fcstRUN phase 1, na = 143 time is 5.88842296600342
PASS: fcstRUN phase 2, na = 143 time is 20.5420670509338
PASS: fcstRUN phase 1, na = 144 time is 14.6955599784851
PASS: fcstRUN phase 2, na = 144 time is 1.63663101196289
PASS: fcstRUN phase 1, na = 145 time is 6.87134408950806

io layout:
&fv_core_nml
layout = 16,12
io_layout = 8,6
npx = 769
npy = 769
ntiles = 6
npz = 127

/gpfs/dell1/ptmp/Jun.Wang/v16rt4_gdasfcst_06.run4

PASS: fcstRUN phase 1, na = 142 time is 5.99817705154419
PASS: fcstRUN phase 2, na = 142 time is 0.155758142471313
PASS: fcstRUN phase 1, na = 143 time is 5.92022585868835
PASS: fcstRUN phase 2, na = 143 time is 10.2844760417938
PASS: fcstRUN phase 1, na = 144 time is 15.4237310886383
PASS: fcstRUN phase 2, na = 144 time is 1.56829094886780
PASS: fcstRUN phase 1, na = 145 time is 6.94577097892761

We need to verify that using the subdomain restart files can generate identical results for restart runs

[junwang-noaa]

Dusan tested the io_layout change and confirmed that the restart run from the restart file generated by io_layout produce identical results as cold start straight run.

Results from Dusan:
diff --git a/tests/tests/fv3_control b/tests/tests/fv3_control
index c66e74c..2fed89c 100644
+export I_IO_LAYOUT=1
+export J_IO_LAYOUT=1

to:
export_fv3
+export I_IO_LAYOUT=3
+export J_IO_LAYOUT=2

script change, copy new restart files to INPUT directory for restart:

• rm -f INPUT/fv_core.res.*
• rm -f INPUT/fv_srf_wnd.res.*
• rm -f INPUT/fv_tracer.res.*
• rm -f INPUT/phy_data.*c
• rm -f INPUT/sfc_data.*
• for RFILE in ../fv3_control${RT_SUFFIX}/RESTART/20161003.120000.*; do

•   RFILE_OLD=$(basename $RFILE)
  

•   RFILE_NEW="${RFILE_OLD//20161003.120000./}"
  

•   cp $RFILE INPUT/${RFILE_NEW}
  

• done

If you look at log_wcoss_dell_p3/rt_00* in /gpfs/dell2/emc/modeling/noscrub/Dusan.Jovic/ufs/ctrl_rest/ufs-weather-model/tests you'll see that all history outputs are identical. Restart files are obviously different because of different naming for multi tile outputs.

Looks like restarting with different io_layout gives b4b identical history outputs, which is good.