Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update gsl/develop from develop 2021/05/21 #79

Merged

Conversation

climbfuji
Copy link

@climbfuji climbfuji commented May 21, 2021

Description

Update of gsl/develop from develop as of 2021/05/21. Changes include:

  • Several build system updates
  • Updates to coupled model components MOM6, CICE6, CDEPS
  • Updates to FV3 and stochastic physics as described in the associated PRs below
  • NEMS driver cleanup

There are no real science changes in these PRs. Answer changes are due to infrastructure updates and the use of assumed-size array definitions, which cause the compilers - in particular the Intel compiler with PROD flags - to optimize the code differently.

Associated PRs

NOAA-GSL/ccpp-framework#14
NOAA-GSL/ccpp-physics#89
NOAA-GSL/fv3atm#89
NOAA-GSL/stochastic_physics#5
#79

Testing

Regression testing against existing baselines on Hera with Intel and GNU: all tests pass, except the newly added tendency tests in gsl/develop (because there is no baseline to compare against). But these tests run to completion with both compilers:

fv3_gfs_v16_dtend_debug 086 failed in check_result
fv3_gfs_v16_dtend_debug 086 failed in run_test
fv3_gsd_dtend_debug 092 failed in check_result
fv3_gsd_dtend_debug 092 failed in run_test

rt_hera_gnu_verify_against_existing.log
rt_hera_intel_verify_against_existing_rerun.log
rt_hera_intel_verify_against_existing_fail_test.log
rt_hera_intel_verify_against_existing.log
rt_hera_gnu_verify_against_existing_fail_test.log

Regression testing using rt_ccpp_dev.conf on Hera with Intel and GNU: first create new baselines, then verify against it:

All tests pass with GNU.

rt_ccpp_dev_hera_gnu_create.log
rt_ccpp_dev_hera_gnu_verify.log

All tests pass with Intel.

rt_ccpp_dev_hera_intel_create.log
rt_ccpp_dev_hera_intel_verify.log

DeniseWorthen and others added 27 commits April 5, 2021 13:09
…add export of openwater_frac_in_atm from fv3ATM when coupled (ufs-community#475)


* update CICE in ufs-weather for the latest emc/develop
* fix ice_in_template; sw_redist has been moved to shortwave_nml and removed two unused settings
* update fv3ATM to use esmf-bs47 and add the export of open-water fraction from fv3ATM 
* update module files to use bs47 (PR ufs-community#493)
* fix wcoss-cray for alt-compare functionality
* update CMEPS and NEMS for added export field
* update Dockerfile
…l on in the coupled model (Replace PR ufs-community#453) (ufs-community#483)

* point to fv3 branch
* Add one more test cpld_control_nsst to test added ccpp suite (FV3_GFS_v15p2_couplednsst)
* Modify rt.conf to add FV3_GFS_v15p2_couplednsst
* Modify rt.conf to add a new test, cpld_bmarkfrac_v16_nsst,  remove test cpld_control_nsst
* Modify tests/tests/cpld_bmarkfrac_v16_nsst: 1. cpld_bmarkfrac_v16 to cpld_bmarkfrac_v16_nsst. 2. export NSTF_NAME=2,1,0,0,0 to export nstf_name=2,1,0,0,0.
* Modify input.benchmark_v16.nml.IN & cpld_bmarkfrac_v16_nsst for a consistent definition of nstf_name namelist
* Modify cpld_bmarkfrac_v16_nsst by moving the NSTF_NAME to the namelist field updates section
* RegressionTests_orion.intel.log of the rt run and BL_DATE=20210406 in rt.sh
* RT JOBS PASSED: hera.intel. Log file uploaded.
* run-ci, commit 7 RegressionTest log files
* Push RegressionTests_wcoss_dell_p3.log

Co-authored-by: Brian Curtis <brian.curtis@noaa.gov>
…modules (ufs-community#492)


* update ufs for CMEPS master with PIO changes
* update pio to 2.5.2 across platforms
* replace variable SUITE_NAME with CCPP_SUITE
* Merge remote-tracking branch 'DusanJovic/module_common' into feature/updcmeps
* switch to h-nems area on jet

Co-authored-by: Bin Li <Bin.Li@gaea13.ncrc.gov>
Co-authored-by: Dusan Jovic <dusan.jovic@noaa.gov>
…fs-community#488)

* switch MOM6 to feature branch
*allow topo-edits to change the land mask when run with CICE6 in ufs
* Comment out line that instructs cmake to not cache the configuration
* tests/fv3_conf/fv3_qsub.IN_cheyenne: prepend model output with rank, similar to other platforms
This reverts commit cea6fc6.
* Revert "update FV3"
This reverts commit 7b826d4.
* Change default queue on Cheyenne to economy
* using MOM6 branch of update-to-GFDL-20210402 from JIandewang
* update to MOM6 PR branch
* add parameter for allowing land mask changes
* switch two variables to character strings for nems.configure
* Regression test log for hera.gnu and hera.intel; run-ci
* Regression test logs for cheyenne.gnu and cheyenne.intel,
* RT JOBS PASSED: gaea.intel. Log file uploaded.
* RT JOBS PASSED: orion.intel. Log file uploaded.
* RT JOBS PASSED: jet.intel. Log file uploaded.
* Regression test logs for wcoss_dell_p3 and wcoss_cray
* Revert change to .gitmodules and update submodule pointer fors for mom6 and fv3atm

Co-authored-by: denise.worthen <Denise.Worthen@noaa.gov>
Co-authored-by: jiandewang <jiande.wang@noaa.gov>
Co-authored-by: Brian Curtis <brian.curtis@noaa.gov>
… zorl interstitial, ocn -> wat, merra2 threading (ufs-community#520)

* changed .gitmodules to point to merra2  fv3atm
* Update .gitmodules and submodule pointer for fv3atm for code review and testing
* update submodule pointer for fv3atm
* Update regression test baseline date in tests/rt.sh; run-ci
* RT JOBS PASSED: hera.gnu. Log file uploaded.
* RT JOBS PASSED: gaea.intel. Log file uploaded.
* RT JOBS PASSED: hera.intel. Log file uploaded.
* RT JOBS PASSED: orion.intel. Log file uploaded.
* Regression test logs for wcoss_dell_p3 and wcoss_cray
* Regression test logs for cheyenne.gnu
* Regression test logs for cheyenne.intel
* Regression test log for jet.intel
* Revert change to .gitmodules and update submodule pointer for fv3atm
Co-authored-by: anning.cheng <anning.cheng@noaa.gov>
Co-authored-by: Brian Curtis <brian.curtis@noaa.gov>
…y#524)

* point to Jun fv3atm personal fork
* add CDEPS submodule
* add building instructions
* fix issues that came from building
* builds CDEPS targets
* missed w3nco dep
* cleanup use of FoX and remove redundant library links in components
* Add 3 files generated by genf90.pl to the CDEPS-interface/ufs directory. (#1)
* remove dependency on genf90.pl for CDEPS/share
* remove FoX (#2)
remove FoX dependency building and use ESMF Config instead.
Co-authored-by: Jun Wang <junwang-noaa@users.noreply.github.com>
Co-authored-by: BinLi-NOAA <bin.li@noaa.gov>
Co-authored-by: Brian Curtis <brian.curtis@noaa.gov>
…bs47 debug on WCOSS_Cray. (ufs-community#532)

* .gitmodules for fv3atm, FV3 pointer update
* fv3 pointer update
* RT JOBS PASSED: cheyenne.gnu. Log file uploaded.
* RT JOBS PASSED: hera.gnu. Log file uploaded.
* RT JOBS PASSED: hera.intel. Log file uploaded.
* RT JOBS PASSED: gaea.intel. Log file uploaded.
* RT JOBS PASSED: jet.intel. Log file uploaded.
* RT JOBS PASSED: cheyenne.intel. Log file uploaded.
* WCOSS Dell P3 Log
* Orion Regression Test
* run-ci
* WCOSS CRAY Log. ESMF-Debug issue
* Updated FV3 pointer to NOAA-EMC/fv3atm, reverted .gitmodules changes
* update regional and fv3-wave tests mpi tasks
* fix lam post uninitialized value
* fix hafs test tasks
* fix the spval in openmp
* add lam inline post test back
* update to upp 10.0.6 and regional.nml
* update baseline date
* RT JOBS PASSED: gaea.intel. Log file uploaded.
* RT JOBS PASSED: jet.intel. Log file uploaded.
* RT JOBS PASSED: orion.intel. Log file uploaded.
* RT JOBS PASSED: cheyenne.intel. Log file uploaded.
* RT JOBS PASSED: cheyenne.gnu. Log file uploaded.
* RT JOBS PASSED: hera.gnu. Log file uploaded.
* RT JOBS PASSED: hera.intel. Log file uploaded.
* update dell RT log file; run-ci
* update cray log file
Co-authored-by: Brian Curtis <brian.curtis@noaa.gov>
Co-authored-by: Jun Wang <junwang-noaa@users.noreply.github.com>
* Add the RegressionTests_orion.intel.log.

* RT JOBS PASSED: cheyenne.gnu. Log file uploaded.

* RT JOBS PASSED: gaea.intel. Log file uploaded.

* RT JOBS PASSED: cheyenne.intel. Log file uploaded.

* WCOSS Dell P3 Log

* RT JOBS PASSED: hera.gnu. Log file uploaded.

* WCOSS Cray Logs. run-ci

* hera.intel log

Co-authored-by: Brian Curtis <brian.curtis@noaa.gov>
…fs-community#531)

Use OPENMP option to specify dependency on OpenMP instead of using OpenMP_Fortran_FOUND.
Remove redundant compiler flags in cmake/Intel.cmake and GNU.cmake
Update CDEPS.
Fix 'make install' in stochastic_physics.
Updates to the documentation of the ufs-weather-model, developed as part of the v2.0.0 release, and a few bug fixes that were also part of that release.

ufs-weather-model: doc/ subdirectory, add the RRFS_v1alpha to rt.conf
NEMS: doc directory only
fv3atm: add RRFS_v1alpha SDF
ccpp-physics: documentation, fix assumed-size array dimensions (gnu10 fix), rename gmtb-scm to scm (Single Column Model)
atmos_cubed_sphere: doc directory only
…ity#522)


* implement unity normalization in CMEPS for coupled and datm applications
Various CCPP PRs as described in NOAA-EMC/fv3atm#291
Improvement and update for CI

Co-authored-by: Dom Heinzeller <dom.heinzeller@icloud.com>
Co-authored-by: Brian Curtis <brian.curtis@noaa.gov>
* Updates to RRTMGP in FV3/CCPP. New RT for regional configuration using RRTMGP
* Updated submodule to my forked repo.
* Changed default namelist setting for RRTMGP.
* Turned on feature to use GP LW flux-adjustment.
* Removed mistake in previous commit
* Updates to RRTMGP RTs.
* Updated RT nml.
* Updated GP regional RTs
* Reverted deprecated code in FV3/ccpp-physics.
* Removed GP c192L127 RT
* Commented out RRTMGP c192l127 RT in rt.conf.
* GP_2thread RT working.
* Updated FV3/physics
* Revert "GP_2thread RT working."
Co-authored-by: Dom Heinzeller <climbfuji@ymail.com>
…n CCPP (ufs-community#527)

- Update the submodule pointers for ccpp-physics and fv3atm and the regression test baseline date tag
- Small update of `tests/ci/repo_check.sh` from @MinsukJi-NOAA
- Move coupled tests in `rt.conf` to the top, because these take the longest to wait in the queue and to run; **BUT**: needed to move ATMW tests to end of `rt.conf`, since WW3 is not able to support parallel builds. By moving it to the end, we are as "lucky" as before in a sense that only ten builds can happen at the same time, and the two builds that involve WW3 are sufficiently far apart to not overlap - see issue ufs-community#550
- Change location of temporary/personal baseline directories on Cheyenne from $WORK to $SCRATCH to avoid disk space limitations
- Increase walltime for compile jobs on wcoss_dell_p3 from 50min to 1hr

Co-authored-by: Dustin Swales <dustin.swales@noaa.gov>
…s-community#542), add CPLD&DATM tests on Jet (ufs-community#545) (ufs-community#533)


* update ufs-weather for removing nems mediator
* set mediator as cmeps in nems.configure*
* copy yaml field dictionary when runnning ATM,ATMW
* update NEMS and CMakeLists, nems.configure
* add a gcc module so that gfortran can be found when building
the WW3 switch preprocessing routines
* Add datm and coupled tests to Jet.

Co-authored-by: Jessica.Meixner <Jessica.Meixner@noaa.gov>
Co-authored-by: Bin.Li <bin.li@noaa.gov>
…s test (ufs-community#538)

* Update NEMS component for CDEPS data atmosphere model.
* Update CMEPS to revise land-sea masks used for cfsr and gefs data sources in CDEPS.
* Update CMakeLists.txt to build the mom6_cice6_cdeps application.
* Use a new ESMF configuration file to provide data stream input parameters.
* Use new input forcing files that are CF-1.0 compliant in time axis.
* Use new input mesh files that are generated from existing grid files.
* Add 8 cdeps regression tests.
* Fix bugs in the datm_bulk_gefs test.
…ufs-community#544)

* updating WW3 to add gaea, jet and now use '$platform.intel' to
simplify some of the CMake and add future flexibility for gnu

* add WW3 tests on more platforms

* current updates for balancing wave tests

* update WW3

* updates for wave template input file

* add ww3_multi.inp as a templated parm file instead of being fixed

* add a gcc module so that gfortran can be found when building
the WW3 switch preprocessing routines

* update WW3 to its latest develop branch

* add ATMW tests to all machines except jet leaves C768 as hera/orion only

* update for better load balancing of wave tests

* fix for multiple wave grids with template

* update to latest WW3

* update ATMW to use existing format for output and include gaea for S2SW

* revert unintended changes

* add CPLWAV=F to datm section of defaults

* point to new WW3 input

* turn on ATMW tests on jet

* switch cpld v16 tests to gaussian output

* add v16 tests on gaea

* remove the C768 test on orion

* update cpld v16 tests for fhmax=9 hours

* add updated restart test

* reduced node usage for fv3_wrtGauss_nemsio_c768 to 40 nodes
the job now took 927s to complete and with only 2 threads
did not run into memory issues that had been the prior reason
for using so few pets per node.  This also added threading so
that the full node would be used

* update cpld v16 tests 

* add fhzero=3 variable to get 3/3/6 restart test to reproduce
the 6 hour forecast file
* cpld v16 35d tests are left at original fdiag=6,fhzero=fdiag
setting to match workflow

* update comment in rt.conf

* updating fix per code review

* removing wcoss-cray from wave tests to not add more
tests there, however they do now run

* regression test log for wcoss
--- Note the first set of regression tests, the compile for cpld
 timed out so I re-ran just the coupled test.  The first log is
on top and the second log with just the coupled is on the bottom

* Jet RT Log, minus datm over quota tests

Co-authored-by: denise.worthen <Denise.Worthen@noaa.gov>
Co-authored-by: Brian Curtis <brian.curtis@noaa.gov>
Update compiler flags in cmake/Intel.cmake and cmake/GNU.cmke so that CCPP can use the flags set by UFS instead of maintaining it's own copy of all flags
Use ESMF version 8.1.1
Update MOM6 submodule
…C routines (re-opened) (ufs-community#567)

* Update FV3
* RT JOBS PASSED: hera.intel. Log file uploaded.
* RT JOBS PASSED: hera.gnu. Log file uploaded.
* RT JOBS PASSED: cheyenne.intel. Log file uploaded.
* RT JOBS PASSED: gaea.intel. Log file uploaded.
* RT JOBS PASSED: cheyenne.gnu. Log file uploaded.
* RT jobs passed: wcoss_dell_p3, wcoss_cray
* Add orion RT log; run-ci
* Add Jet log; run-ci
* update cpld run scripts
* add level indication ("L") for the FV3_input_frac directories
and remove conditional if statement
* switch to using mom mesh file for mom6 cap instead of grid
* make nst_anl false in input.benchmark_v16.nml.IN
* update FV3 containing Kate's dycore fix for z_bot
* update input-data date
* update FV3 with new suite files
* Remove FMS git submodule
* Update module files and top-level CMakeLists.txt to use fms module (library) built by hpc-stack.
* Update ci for fms library ufs-community#575 (#13)

Co-authored-by: Minsuk Ji <57227195+MinsukJi-NOAA@users.noreply.github.com>
…pp-physics (ufs-community#572)

* Change CI triggering method ufs-community#558: CI runs when run-ci label is created
* Remove FMS remnant from compile.sh
* Move load cmake call to after hpc-stack on wcoss_dell_p3
* CDEPS updates

Co-authored-by: MinsukJi-NOAA <minsuk.ji@noaa.gov>
Co-authored-by: Bin Li <bin.li@noaa.gov>
Co-authored-by: Brian Curtis <brian.curtis@noaa.gov>
Co-authored-by: denise.worthen <Denise.Worthen@noaa.gov>
@DomHeinzeller
Copy link

@tanyasmirnova @joeolson42 @SamuelTrahanNOAA this is reaydy to merge, can I get a review/approval, please? Thank you ...

@DomHeinzeller
Copy link

Thanks, @tanyasmirnova !

@DomHeinzeller DomHeinzeller merged commit eae7d9c into NOAA-GSL:gsl/develop May 21, 2021
Copy link
Collaborator

@SamuelTrahanNOAA SamuelTrahanNOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only some very minor issues, and they're all from the upstream repository.

@@ -11,12 +11,12 @@ Input files
=============

There are three types of files needed to execute a run: static datasets (*fix* files containing climatological
information), files that depend on grid resolution and initial conditions, and model configuration files (such as namelists).
information), files that depend on grid resolution, initial and boundary conditions, and model configuration files (such as namelists).
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing a comma before final conjunction.

"resolution, initial, and boundary conditions"

https://www.thesaurus.com/e/grammar/s/serial-comma/#do-you-call-it-the-serial-comma-or-the-oxford-comma

@@ -79,19 +79,19 @@ The static input files are listed and described in :numref:`Table %s <FixFiles>`
---------------------------------------------
Grid description and initial condition files
---------------------------------------------
The input files containing grid information and the initial conditions are listed and described in :numref:`Table %s <GridICFiles>`.
The input files containing grid information and the initial conditions for global configurations are listed and described in :numref:`Table %s <GridICFiles>`. The input files for a limited area model (LAM) configuration including grid information and initial and lateral boundary conditions are listed and described in :numref:`Table %s <RegionalGridICFiles>`. Note that the regional grid is referred to as Tile 7 here, and are generated by several pre-processing utilities.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When defining an acronym, you must capitalize the first letter of each word, as in a title. Also, "limited-area" should be hyphenated because it is being used as an adjective.

Limited-Area Model (LAM) configuration

Also, replace "and...and" with a comma-separated series. The sentence ran away from itself; it's better to rewrite it in a more simple manner like:

The input files for a Limited-Area Model (LAM) configuration includes initial and boundary conditions, and grid information (see :numref: Table...)

Also, this "are" should be "is"

Note that the regional grid is referred to as Tile 7 here, and is generated by several pre-processing utilities

Additional Information about the Write Component
==============================================================

The UFS Weather Model is built using the Earth System Modeling Framework (ESMF). As part of this framework, the output history files written by the model use an ESMF component, referred to as the *write component*. This model component is configured with settings in the model_configure file, as described in :numref:`Section %s <model_configureFile>`. By using the ESMF capabilities, the write component can generate output files in several different formats and several different map projections. For example, a Gaussian global grid in NEMSIO format, or a native grid in NetCDF format. The write component also runs on independent MPI tasks, and so the computational tasks can continue while the write component completes its work asynchronously. The configuration of write component tasks, balanced with compute tasks, is part of the tuning for each specific application of the model (HPC, write frequency, i/o speed, model domain, etc). For the global grid, if the write component is not selected (quilting=.false.), the FV3 code will write tiled output in the native projection in NetCDF format. The regional grid requires the use of the write component.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this line, "i/o" should be "I/O" since it is an acronym. Also, the parentheses after the abbreviation "etc" should be after the period.

specific application of the model (HPC, write frequency, I/O speed, model domain, etc.)

#%Module

proc ModulesHelp {} {
puts stderr "\tcit - loads modules required for building and running UFS Model on Cheyenne/GNU"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be, "loads modules that are required for..."

#%Module

proc ModulesHelp {} {
puts stderr "\tcit - loads modules required for building and running UFS Model on Cheyenne/GNU"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be, "loads modules that are required for..."

export ATM_NY_GLB=$JATM
export ATMRES='1536x768'
export DATM_SRC="GEFS"
export FILENAME_BASE='gefs.'
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These quotes are extraneous.

export FILENAME_BASE='cfsr.'
export IATM=1760
export JATM=880
export DAYS="0.25"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These quotes are extraneous.


export_datm_cdeps
export DATM_SRC="CFSR"
export FILENAME_BASE='cfsr.'
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These quotes are extraneous.


export_datm_cdeps
export WLCLK=40
export DATM_SRC="CFSR"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These quotes are extraneous.

export ATM_NY_GLB=$JATM
export ATMRES='1536x768'
export DATM_SRC="GEFS"
export FILENAME_BASE='gefs.'
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These quotes are extraneous.

@climbfuji
Copy link
Author

Only some very minor issues, and they're all from the upstream repository.

Thanks, Sam, for your review - since these changes are coming from the upstream repository, it would be great if you could (time permitting) open a PR to fix them in those repositories (not in the NOAA-GSL fork)?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.