Update ufs-weather for latest CICE emc/develop; Update to esmf-bs47; add export of openwater_frac_in_atm from fv3ATM when coupled #475

DeniseWorthen · 2021-03-14T14:47:54Z

Description

This PR will:

update CICE in ufs-weather for the latest emc/develop at NOAA-EMC/CICE.
update fv3ATM to use esmf-bs47, add the export of open-water fraction from fv3ATM and update module files across all platforms to use bs47 (PR Update to esmf-bs47 across all platforms; add export of openwater_frac_in_atm from FV3 when coupled #493)

Issue(s) addressed

CICE Issue #25
UFS Issue Update ESMF to bs47 across all platforms #494
partial fix to issue UFS the coupled model requires FV3 and CMEPS updates for frac grid normalization #366. FV3atm will export the required field but no use will be made of it until changes are made in the CMEPS mediator.

Testing

This PR will change all ufs-coupled and ufs-datm baselines due to a non-b4b bugfix in Icepack. A test baseline was created on hera.intel and all cpld and datm regression tests passed against that test baseline.
This PR also changes baselines due to the change to esmf-bs47 because some forecast files will contain different metadata or different ordering of fields. See fv3ATM Issue #265 for more details on the changes expected.

Dependencies

NEMS PR#94
CICE PR #24
CMEPS PR #39
FV3-ATM PR #266

use updcmeps branch of CMEPS-interface/CMEPS add file to CMakeLists shorten name of nems.configure file for coupled model clean up white space

add rahul's fix for optionally loading fv3_debug if exists and debug=y

add file to cice_files.cmake update CICE with use statement added to CICEInitMod

skip-ci

add regression test log showing that debug test passes baseline cpld_control fails due to refactor

* Add $INPUTDATA_ROOT * Update utest * rt.sh: Remove LOCKDIR before trying to stop ecflow in cleanup()

DeniseWorthen · 2021-04-02T14:26:11Z

@junwang-noaa all the submodules have approvals at this point except CICE--can you please review/approve?

DeniseWorthen · 2021-04-02T15:53:17Z

I put the jet baseline in the wrong location for the auto-rt to find it (/lfs4/HFIP/hfv3gfs/RT/NEMSfv3gfs vs /lfs4/HFIP/hfv3gfs/emc.nemspara/RT/NEMSfv3gfs) so we'll be restarting the jet auto-rt.

BrianCurtis-NOAA · 2021-04-02T18:09:17Z

Machine: jet
Compiler: intel
Repo location: /lfs4/HFIP/h-nems/emc.nemspara/autort/pr/592556664/20210402160009/ufs-weather-model
Please manually delete: /lfs4/HFIP/hfv3gfs/emc.nemspara/RT_RUNDIRS/emc.nemspara/FV3_RT/rt_18641
Test fv3_stretched_nest 019 failed in run_test failed
Test fv3_HAFS_v0_hwrf_thompson 046 failed in check_result failed
Test fv3_HAFS_v0_hwrf_thompson 046 failed in run_test failed
Please make changes and add the following label back:
jet-intel-RT

DeniseWorthen · 2021-04-02T18:31:30Z

The fv3_stretched_nest failed due to time wall clock but it actually seemed to have hung. It is a 48hr forecast and it looks like it only got a few timesteps.

The HAFS test failed on the 4x daily tile2 because the baseline file has no time level information written:

time = UNLIMITED ; // (0 currently)

The other 4x daily files for the other tiles have 4 timesteps correctly written and they compare identically with the verify run.

I will add that I'm having some trouble getting response from jet when I ls or cd. So perhaps there is a system issue going on here too.

DeniseWorthen · 2021-04-03T10:21:25Z

The baselines on jet are not completing (compiling) because the disk is full:

CMakeFiles/ccppphys.dir/physics/GFS_phys_time_vary.fv3.F90.o: catastrophic error: Disk is full, no space to write object file

This is in my own attempt to run the last two baselines as well as the auto-rt I started late yesterday.

BrianCurtis-NOAA · 2021-04-05T02:40:22Z

Machine: jet
Compiler: intel
Repo location: /lfs4/HFIP/h-nems/emc.nemspara/autort/pr/592556664/20210402230015/ufs-weather-model
Please manually delete: /lfs4/HFIP/hfv3gfs/emc.nemspara/RT_RUNDIRS/emc.nemspara/FV3_RT/rt_150647
Test compile 014 failed failed
Test compile 013 failed failed
Test fv3_cpt 032 failed in run_test failed
Test fv3_wrtGauss_nemsio_c192 010 failed in run_test failed
Test fv3_gfs_v16_RRTMGP_2thrd 041 failed in run_test failed
Test fv3_esg_HAFS_v0_hwrf_thompson 047 failed in run_test failed
Please make changes and add the following label back:
jet-intel-RT

* update CICE, CMEPS and NEMS submodules

DeniseWorthen · 2021-04-05T14:30:23Z

Ready for review & merge.

climbfuji · 2021-04-05T14:35:27Z

tests/rt.conf

@@ -40,7 +40,7 @@ RUN     | fv3_stretched_nest
 COMPILE | APP=ATM SUITES=FV3_GFS_v15_thompson_mynn 32BIT=Y                                                                        |                                         | fv3 |
 RUN     | fv3_regional_control                                                                                                    |                                         | fv3 |
 RUN     | fv3_regional_restart                                                                                                    |                                         | fv3 | fv3_regional_control
-RUN     | fv3_regional_quilt                                                                                                      |                                         | fv3 |
+#RUN     | fv3_regional_quilt                                                                                                      |                                         | fv3 |


Do we have a timeline for fixing this?

I think Jun has tracked down the issues here. She's opened an associated issue in EMC_post.

climbfuji · 2021-04-05T14:35:54Z

tests/tests/fv3_gfdlmprad

@@ -56,6 +56,7 @@ export LIST_FILES="atmos_4xdaily.tile1.nc \
 export_fv3

 export TASKS=192
+export WLCLK=60


These longer runtimes need to be investigated.

A full 60 minutes is not required but this is what I tested out of frustration. The runs ended up taking about 20-25 minutes in my case.

MinsukJi-NOAA · 2021-04-05T15:03:40Z

@DeniseWorthen could you include these changes? It fixes the esmf bs47 docker container issue.

diff --git a/tests/ci/Dockerfile b/tests/ci/Dockerfile
index c685eaf..8d01299 100644
--- a/tests/ci/Dockerfile
+++ b/tests/ci/Dockerfile
@@ -1,4 +1,4 @@
-From noaaemc/ubuntu-hpc:v1.1
+From noaaemc/ubuntu-hpc:v1.2
 
 ENV HOME=/home/builder
 COPY --chown=builder:builder . $HOME/ufs-weather-model

DeniseWorthen · 2021-04-05T15:52:14Z

@MinsukJi-NOAA Should I now try run-ci?

MinsukJi-NOAA · 2021-04-05T15:54:11Z

Yes, that would be great.

DeniseWorthen · 2021-04-05T16:52:30Z

@MinsukJi-NOAA Two CI tests failed. Should I hold off merging until you can take a look?

MinsukJi-NOAA · 2021-04-05T17:02:45Z

I will take a look. I suggest you go ahead with merging for this PR.

@SMoorthi-emc

…(combined) (ufs-community#475) * add lsm_cold_start variable in GFS_typedefs.F90/meta and update ccpp/physics for testing * includes @SMoorthi-emc 's bugfix for gwdps.f.

…e-offset (#475) * change PDY to yyyymmdd for time-offset * change prod in nco we2e script * modify ex-script rather than machine file * fix typo * Set PDYext cycext Co-authored-by: chan-hoo <chan-hoo.jeon@clogin01.cactus.wcoss2.ncep.noaa.gov>

DeniseWorthen and others added 30 commits October 28, 2020 12:47

update cmeps

ca288f4

use updcmeps branch of CMEPS-interface/CMEPS add file to CMakeLists shorten name of nems.configure file for coupled model clean up white space

fix for debug mode compile

a21c7e9

add rahul's fix for optionally loading fv3_debug if exists and debug=y

update cmeps

0b655ed

Merge branch 'develop' into feature/updcmeps

1699be7

point to updcice branch

399d21d

updates for ice_dyn_vp

06bad08

add file to cice_files.cmake update CICE with use statement added to CICEInitMod

update to last CMEPS master prior to perf updates

5d51334

update CMEPS submodule

21543a2

Merge branch 'develop' into feature/updcmeps

2017af3

skip-ci

Merge branch 'develop' into feature/updcice

b558028

Merge branch 'feature/updcmeps' into feature/updcice

394c7ad

Merge branch 'develop' into feature/updcice

5d19c0f

update cice branch of ufs-weather

2d12b93

add regression test log showing that debug test passes baseline cpld_control fails due to refactor

update gitmodules and CICE

0e2cc16

set n_aero to zero

e4af1ec

Merge branch 'develop' into feature/updcice

a356d4c

Merge branch 'develop' into feature/updcice

0564cdf

Merge branch 'develop' into feature/updcice

381142b

Merge branch 'develop' into feature/updcice

42fe791

Merge branch 'develop' into feature/updcice

c73dcd9

Merge branch 'develop' into feature/updcice

d320bd5

Merge branch 'develop' into feature/updcice

81b7165

Merge branch 'develop' into feature/updcice skip-ci

e93cfae

add cheyenne.intel for ufs-cpld and ufs-datm skip-ci

ab1ca9f

rt.sh separate input (#2)

541c714

* Add $INPUTDATA_ROOT * Update utest * rt.sh: Remove LOCKDIR before trying to stop ecflow in cleanup()

add hera gnu logs; update baseline date; skip-ci

8950d96

add orion.intel logs; skip-ci

b3d2cf2

add hera.intel log; skip-ci

79205a6

add wcoss-cray log; skip-ci

d8d1d1d

add dell-p3 log; skip-ci

6c760ac

MinsukJi-NOAA mentioned this pull request Apr 2, 2021

How to maintain hpc-stack up to date with ufs-weather-model NOAA-EMC/hpc-stack#205

Closed

BrianCurtis-NOAA added jet-intel-RT and removed jet-intel-RT labels Apr 2, 2021

DeniseWorthen added the jet-intel-RT label Apr 2, 2021

BrianCurtis-NOAA removed the jet-intel-RT label Apr 2, 2021

add jet.intel log

50c1999

DeniseWorthen added 2 commits April 5, 2021 09:02

update submodules

343da16

* update CICE, CMEPS and NEMS submodules

update FV3 submodule

446da5b

DusanJovic-NOAA approved these changes Apr 5, 2021

View reviewed changes

climbfuji approved these changes Apr 5, 2021

View reviewed changes

BrianCurtis-NOAA approved these changes Apr 5, 2021

View reviewed changes

update Dockerfile

5bdae36

run-ci

d067cc4

DeniseWorthen merged commit b8ed338 into ufs-community:develop Apr 5, 2021

This was referenced Apr 5, 2021

CMEPS needs to use imported ocean fraction from FV3 to properly normalize fluxes NOAA-EMC/CMEPS#29

Closed

compare_ncfile error on cray #500

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update ufs-weather for latest CICE emc/develop; Update to esmf-bs47; add export of openwater_frac_in_atm from fv3ATM when coupled #475

Update ufs-weather for latest CICE emc/develop; Update to esmf-bs47; add export of openwater_frac_in_atm from fv3ATM when coupled #475

DeniseWorthen commented Mar 14, 2021 •

edited

Loading

DeniseWorthen commented Apr 2, 2021

DeniseWorthen commented Apr 2, 2021

BrianCurtis-NOAA commented Apr 2, 2021

DeniseWorthen commented Apr 2, 2021 •

edited

Loading

DeniseWorthen commented Apr 3, 2021

BrianCurtis-NOAA commented Apr 5, 2021

DeniseWorthen commented Apr 5, 2021

climbfuji Apr 5, 2021

DeniseWorthen Apr 5, 2021

climbfuji Apr 5, 2021

DeniseWorthen Apr 5, 2021

MinsukJi-NOAA commented Apr 5, 2021

DeniseWorthen commented Apr 5, 2021

MinsukJi-NOAA commented Apr 5, 2021

DeniseWorthen commented Apr 5, 2021

MinsukJi-NOAA commented Apr 5, 2021

Update ufs-weather for latest CICE emc/develop; Update to esmf-bs47; add export of openwater_frac_in_atm from fv3ATM when coupled #475

Update ufs-weather for latest CICE emc/develop; Update to esmf-bs47; add export of openwater_frac_in_atm from fv3ATM when coupled #475

Conversation

DeniseWorthen commented Mar 14, 2021 • edited Loading

Description

Issue(s) addressed

Testing

Dependencies

DeniseWorthen commented Apr 2, 2021

DeniseWorthen commented Apr 2, 2021

BrianCurtis-NOAA commented Apr 2, 2021

DeniseWorthen commented Apr 2, 2021 • edited Loading

DeniseWorthen commented Apr 3, 2021

BrianCurtis-NOAA commented Apr 5, 2021

DeniseWorthen commented Apr 5, 2021

climbfuji Apr 5, 2021

Choose a reason for hiding this comment

DeniseWorthen Apr 5, 2021

Choose a reason for hiding this comment

climbfuji Apr 5, 2021

Choose a reason for hiding this comment

DeniseWorthen Apr 5, 2021

Choose a reason for hiding this comment

MinsukJi-NOAA commented Apr 5, 2021

DeniseWorthen commented Apr 5, 2021

MinsukJi-NOAA commented Apr 5, 2021

DeniseWorthen commented Apr 5, 2021

MinsukJi-NOAA commented Apr 5, 2021

DeniseWorthen commented Mar 14, 2021 •

edited

Loading

DeniseWorthen commented Apr 2, 2021 •

edited

Loading