Stoch updates

[pjpegion]

Description

This PR references issue #371

Addition of stochastic physics restarts in netCDF format that will be written at the restart interval, and at end of run.
Addition of stochastic cloud perturbation and Micropyiscs perturbations for SPPT.
and
clean up of stochastic physics code.

Is a change of answers expected from this PR?
No changes are expected with regression tests except the stochastic physics related regression tests. Stochastic physics regression test answer does change due to using a local version of ffpack instead of the fftpack in splib.

Issue(s) addressed

• fixes ocean stochastic physics #371

Testing

testing of new science additions were tested with 1-degree coupled model ensemble forecasts out to 35-day
What compilers / HPCs was it tested with? All tests on hera with intel and GNU compuler.
Are the changes covered by regression tests? (If not, why? Do new tests need to be added?)
The cloud perturbations and microphhysics perturbations are not included in any RTs.

Have the ufs-weather-model regression test been run? Currently a subset of regression tests were run on hera with intel and GNU compliers

Will the code updates change regression test baseline? If yes, why?
Yes, stochastic physics test will be changed due to using a local version of fftpack
I am waiting on code review before doing the full regression test suite.

• hera.intel
• hera.gnu
• orion.intel
• cheyenne.intel
• cheyenne.gnu
• gaea.intel
• jet.intel
• wcoss_cray
• wcoss_dell_p3

new baseline date tag is 20210318.

Dependencies

If testing this branch requires non-default branches in other repositories, list them.
Those branches should have matching names (ideally)

• waiting on Cld perts NOAA-EMC/fv3atm#233
• waiting on Stochastic MP and cloud perturbations NCAR/ccpp-physics#553
• waiting on Ocean stochastic physics and netCDF restarts NOAA-PSL/stochastic_physics#35

[DusanJovic-NOAA]

I do not see cheyenne log files.

[pjpegion]

I do not see cheyenne log files.

I just pushed them.

[pjpegion]

@BrianCurtis-NOAA can you provide more infomation on what you are doing? I ran the hera.gnu test this morning, and I see that just up ran to too, and now I see that your hera.intel test failed, but mine worked.

[climbfuji]

@BrianCurtis-NOAA can you provide more infomation on what you are doing? I ran the hera.gnu test this morning, and I see that just up ran to too, and now I see that your hera.intel test failed, but mine worked.

Duplication of efforts it seems ... hera just had intel compiler (license?) issues.

[DusanJovic-NOAA]

@BrianCurtis-NOAA can you provide more infomation on what you are doing? I ran the hera.gnu test this morning, and I see that just up ran to too, and now I see that your hera.intel test failed, but mine worked.

It's auto-rt jobs that I submitted. It's not Brian.

[pjpegion]

@DusanJovic-NOAA thanks for the clarification

[BrianCurtis-NOAA]

Machine: jet
Compiler: intel
Repo location: /lfs4/HFIP/h-nems/emc.nemspara/autort/pr/555022768/20210319183026/ufs-weather-model
Please manually delete: /lfs4/HFIP/hfv3gfs/emc.nemspara/RT_RUNDIRS/emc.nemspara/FV3_RT/rt_287340
Test fv3_HAFS_v0_hwrf_thompson_debug 065 failed in run_test failed
Test fv3_gfs_v16_ras_debug 068 failed in run_test failed
Test fv3_gfsv16_ugwpv1_debug 067 failed in run_test failed
Please make changes and add the following label back:
jet-intel-RT

[pjpegion]

Those three failed tests on jets have the same system error:

sbatch: error: Batch job submission failed: Socket timed out on send/recv operation

I compared the output of fv3_HAFS_v0_hwrf_thompson_debug and it has zero difference from baseline at FHR24.

[DusanJovic-NOAA]

Those three failed tests on jets have the same system error:

sbatch: error: Batch job submission failed: Socket timed out on send/recv operation

I compared the output of fv3_HAFS_v0_hwrf_thompson_debug and it has zero difference from baseline at FHR24.

I reran the test manually. Log file is here:
/lfs4/HFIP/hfv3gfs/Dusan.Jovic/ufs/pr372/ufs-weather-model/tests/RegressionTests_jet.intel.log