Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update UFS_UTILS version to new ufs_utils_1_8_0 tag (develop) #974

Closed
KateFriedman-NOAA opened this issue Aug 15, 2022 · 10 comments · Fixed by #1001
Closed

Update UFS_UTILS version to new ufs_utils_1_8_0 tag (develop) #974

KateFriedman-NOAA opened this issue Aug 15, 2022 · 10 comments · Fixed by #1001
Assignees
Labels
maintenance Regular updates and maintenance work

Comments

@KateFriedman-NOAA
Copy link
Member

Description

A new UFS_UTILS tag (ufs_utils_1_8_0) has been released. It includes the following changes compared to the current hash (a2b0817):

https://github.com/ufs-community/UFS_UTILS/compare/a2b0817..735e2ba

The changes include cleanup to remove old/decommissioned platforms, add WCOSS2 support, and updates documented in these UFS_UTILS PRs:

Requirements

This new tag is required to add WCOSS2 support into the develop branch and go in before WCOSS2 build-related PRs can be sent to the develop branch for Issue #419.

Acceptance Criteria (Definition of Done)

The new tag builds and works on Hera and Orion.

Dependencies

None

@KateFriedman-NOAA KateFriedman-NOAA added the maintenance Regular updates and maintenance work label Aug 15, 2022
@KateFriedman-NOAA KateFriedman-NOAA self-assigned this Aug 15, 2022
@KateFriedman-NOAA
Copy link
Member Author

KateFriedman-NOAA commented Aug 22, 2022

Now running atmos-only and coupled controls and tests on Hera for this tag update.

@KateFriedman-NOAA
Copy link
Member Author

Testing on Hera moves forward very slowly due to long wait times in queues because of low fairshare value for fv3-cpu account.

@KateFriedman-NOAA
Copy link
Member Author

The coupled control and test match each other (not showing atmos/input.nml diffs, which are from clone path diffs and are expected):

-bash-4.2$ ./diff_ROTDIR.sh /scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot 2013010100 devctrlcpl uuctrlcpl > diff_cpl_uu_test.out 2>&1
-bash-4.2$ pwd
/scratch1/NCEPDEV/global/Kate.Friedman/git/feature_dev-wcoss2/test
-bash-4.2$ cat diff_cpl_uu_test.out 
=== atmos/storms.gfso.atcf_gen.2013010100 ===
=== atmos/trak.gfso.atcfunix.2013010100 ===
=== ice/ice_in ===
=== ocean/MOM_input ===
=== atmos/gfs.t00z.master.grb2f000 ===
All fields are identical!
=== atmos/gfs.t00z.master.grb2f006 ===
All fields are identical!
=== atmos/gfs.t00z.master.grb2f012 ===
All fields are identical!
=== atmos/gfs.t00z.master.grb2f018 ===
All fields are identical!
=== atmos/gfs.t00z.master.grb2f024 ===
All fields are identical!
=== atmos/gfs.t00z.master.grb2f030 ===
All fields are identical!
=== atmos/gfs.t00z.master.grb2f036 ===
All fields are identical!
=== atmos/gfs.t00z.master.grb2f042 ===
All fields are identical!
=== atmos/gfs.t00z.master.grb2f048 ===
All fields are identical!
=== atmos/gfs.t00z.master.grb2if000 ===
All fields are identical!
=== atmos/gfs.t00z.master.grb2if006 ===
All fields are identical!
=== atmos/gfs.t00z.master.grb2if012 ===
All fields are identical!
=== atmos/gfs.t00z.master.grb2if018 ===
All fields are identical!
=== atmos/gfs.t00z.master.grb2if024 ===
All fields are identical!
=== atmos/gfs.t00z.master.grb2if030 ===
All fields are identical!
=== atmos/gfs.t00z.master.grb2if036 ===
All fields are identical!
=== atmos/gfs.t00z.master.grb2if042 ===
All fields are identical!
=== atmos/gfs.t00z.master.grb2if048 ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2.0p25.f000 ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2.0p25.f000.idx ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2.0p25.f006 ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2.0p25.f006.idx ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2.0p25.f012 ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2.0p25.f012.idx ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2.0p25.f018 ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2.0p25.f018.idx ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2.0p25.f024 ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2.0p25.f024.idx ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2.0p25.f030 ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2.0p25.f030.idx ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2.0p25.f036 ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2.0p25.f036.idx ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2.0p25.f042 ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2.0p25.f042.idx ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2.0p25.f048 ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2.0p25.f048.idx ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2.0p50.f000 ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2.0p50.f000.idx ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2.0p50.f006 ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2.0p50.f006.idx ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2.0p50.f030 ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2.0p50.f030.idx ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2.0p50.f036 ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2.0p50.f036.idx ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2.1p00.f000 ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2.1p00.f000.idx ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2.1p00.f006 ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2.1p00.f006.idx ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2.1p00.f030 ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2.1p00.f030.idx ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2.1p00.f036 ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2.1p00.f036.idx ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2b.0p25.f000 ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2b.0p25.f000.idx ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2b.0p25.f006 ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2b.0p25.f006.idx ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2b.0p25.f012 ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2b.0p25.f012.idx ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2b.0p25.f018 ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2b.0p25.f018.idx ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2b.0p25.f024 ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2b.0p25.f024.idx ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2b.0p25.f030 ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2b.0p25.f030.idx ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2b.0p25.f036 ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2b.0p25.f036.idx ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2b.0p25.f042 ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2b.0p25.f042.idx ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2b.0p25.f048 ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2b.0p25.f048.idx ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2b.0p50.f000 ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2b.0p50.f000.idx ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2b.0p50.f006 ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2b.0p50.f006.idx ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2b.0p50.f030 ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2b.0p50.f030.idx ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2b.0p50.f036 ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2b.0p50.f036.idx ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2b.1p00.f000 ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2b.1p00.f000.idx ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2b.1p00.f006 ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2b.1p00.f006.idx ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2b.1p00.f030 ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2b.1p00.f030.idx ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2b.1p00.f036 ===
All fields are identical!
=== atmos/gfs.t00z.pgrb2b.1p00.f036.idx ===
All fields are identical!
=== atmos/gfs.t00z.flux.1p00.f000 ===
All fields are identical!
=== atmos/gfs.t00z.flux.1p00.f000.idx ===
All fields are identical!
=== atmos/gfs.t00z.flux.1p00.f006 ===
All fields are identical!
=== atmos/gfs.t00z.flux.1p00.f006.idx ===
All fields are identical!
=== atmos/gfs.t00z.flux.1p00.f012 ===
All fields are identical!
=== atmos/gfs.t00z.flux.1p00.f012.idx ===
All fields are identical!
=== atmos/gfs.t00z.flux.1p00.f018 ===
All fields are identical!
=== atmos/gfs.t00z.flux.1p00.f018.idx ===
All fields are identical!
=== atmos/gfs.t00z.flux.1p00.f024 ===
All fields are identical!
=== atmos/gfs.t00z.flux.1p00.f024.idx ===
All fields are identical!
=== atmos/gfs.t00z.flux.1p00.f030 ===
All fields are identical!
=== atmos/gfs.t00z.flux.1p00.f030.idx ===
All fields are identical!
=== atmos/gfs.t00z.flux.1p00.f036 ===
All fields are identical!
=== atmos/gfs.t00z.flux.1p00.f036.idx ===
All fields are identical!
=== atmos/gfs.t00z.flux.1p00.f042 ===
All fields are identical!
=== atmos/gfs.t00z.flux.1p00.f042.idx ===
All fields are identical!
=== atmos/gfs.t00z.flux.1p00.f048 ===
All fields are identical!
=== atmos/gfs.t00z.flux.1p00.f048.idx ===
All fields are identical!
=== wave/gridded/gfswave.t00z.global.0p50.f000.grib2 ===
All fields are identical!
=== wave/gridded/gfswave.t00z.global.0p50.f003.grib2 ===
All fields are identical!
=== wave/gridded/gfswave.t00z.global.0p50.f006.grib2 ===
All fields are identical!
=== wave/gridded/gfswave.t00z.global.0p50.f009.grib2 ===
All fields are identical!
=== wave/gridded/gfswave.t00z.global.0p50.f012.grib2 ===
All fields are identical!
=== wave/gridded/gfswave.t00z.global.0p50.f015.grib2 ===
All fields are identical!
=== wave/gridded/gfswave.t00z.global.0p50.f018.grib2 ===
All fields are identical!
=== wave/gridded/gfswave.t00z.global.0p50.f021.grib2 ===
All fields are identical!
=== wave/gridded/gfswave.t00z.global.0p50.f024.grib2 ===
All fields are identical!
=== wave/gridded/gfswave.t00z.global.0p50.f027.grib2 ===
All fields are identical!
=== wave/gridded/gfswave.t00z.global.0p50.f030.grib2 ===
All fields are identical!
=== wave/gridded/gfswave.t00z.global.0p50.f033.grib2 ===
All fields are identical!
=== wave/gridded/gfswave.t00z.global.0p50.f036.grib2 ===
All fields are identical!
=== wave/gridded/gfswave.t00z.global.0p50.f039.grib2 ===
All fields are identical!
=== wave/gridded/gfswave.t00z.global.0p50.f042.grib2 ===
All fields are identical!
=== wave/gridded/gfswave.t00z.global.0p50.f045.grib2 ===
All fields are identical!
=== wave/gridded/gfswave.t00z.global.0p50.f048.grib2 ===
All fields are identical!
=== ocean/ocn_ice2013010106.01.2013010100_0p25x0p25.grb2 ===
All fields are identical!
=== ocean/ocn_ice2013010106.01.2013010100_0p5x0p5.grb2 ===
All fields are identical!
=== ocean/ocn_ice2013010112.01.2013010100_0p25x0p25.grb2 ===
All fields are identical!
=== ocean/ocn_ice2013010112.01.2013010100_0p5x0p5.grb2 ===
All fields are identical!
=== ocean/ocn_ice2013010118.01.2013010100_0p25x0p25.grb2 ===
All fields are identical!
=== ocean/ocn_ice2013010118.01.2013010100_0p5x0p5.grb2 ===
All fields are identical!
=== ocean/ocn_ice2013010200.01.2013010100_0p25x0p25.grb2 ===
All fields are identical!
=== ocean/ocn_ice2013010200.01.2013010100_0p5x0p5.grb2 ===
All fields are identical!
=== ocean/ocn_ice2013010206.01.2013010100_0p25x0p25.grb2 ===
All fields are identical!
=== ocean/ocn_ice2013010206.01.2013010100_0p5x0p5.grb2 ===
All fields are identical!
=== ocean/ocn_ice2013010212.01.2013010100_0p25x0p25.grb2 ===
All fields are identical!
=== ocean/ocn_ice2013010212.01.2013010100_0p5x0p5.grb2 ===
All fields are identical!
=== ocean/ocn_ice2013010218.01.2013010100_0p25x0p25.grb2 ===
All fields are identical!
=== ocean/ocn_ice2013010218.01.2013010100_0p5x0p5.grb2 ===
All fields are identical!
=== ocean/ocn_ice2013010300.01.2013010100_0p25x0p25.grb2 ===
All fields are identical!
=== ocean/ocn_ice2013010300.01.2013010100_0p5x0p5.grb2 ===
All fields are identical!
=== atmos/gfs.t00z.atmf000.nc ===
0 differences found
=== atmos/gfs.t00z.atmf006.nc ===
0 differences found
=== atmos/gfs.t00z.atmf012.nc ===
0 differences found
=== atmos/gfs.t00z.atmf018.nc ===
0 differences found
=== atmos/gfs.t00z.atmf024.nc ===
0 differences found
=== atmos/gfs.t00z.atmf030.nc ===
0 differences found
=== atmos/gfs.t00z.atmf036.nc ===
0 differences found
=== atmos/gfs.t00z.atmf042.nc ===
0 differences found
=== atmos/gfs.t00z.atmf048.nc ===
0 differences found
=== atmos/gfs.t00z.sfcf000.nc ===

Examining atmos-only 1.5 cycle cycled test now, which is where diffs would likely appear anyway since UFS_UTILS codes are invoked in cycled mode more-so than a single cycle coupled test.

@KateFriedman-NOAA
Copy link
Member Author

FYI, ignore the dumb experiment names I used for this (including "ctrl" in all of them incorrectly facepalm). Had a brain fart when setting experiment names. Experiments are:

  • devctrlatmos - atmos-only control
  • uuctrlatmos - atmos-only test with new UFS_UTILS tag
  • devctrlcpl - coupled control
  • uuctrlcpl - coupled test with new UFS_UTILS tag

@KateFriedman-NOAA
Copy link
Member Author

Examining the cycled control and test for new UFS_UTILS tag

Spot check of first half cycle test (C192C96L127) shows they match:

-bash-4.2$ pwd
/scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot/uuctrlatmos
-bash-4.2$ nccmp -dgB ../devctrlatmos/gdas.20220101/18/atmos/gdas.t18z.sfcf009.nc gdas.20220101/18/atmos/gdas.t18z.sfcf009.nc                        
-bash-4.2$ nccmp -dgB ../devctrlatmos/gdas.20220101/18/atmos/gdas.t18z.atmf009.nc gdas.20220101/18/atmos/gdas.t18z.atmf009.nc                                              
-bash-4.2$ cmp ../devctrlatmos/gdas.20220101/18/atmos/gdas.t18z.master.grb2f009 gdas.20220101/18/atmos/gdas.t18z.master.grb2f009
-bash-4.2$ cmp ../devctrlatmos/gdas.20220101/18/atmos/gdas.t18z.pgrb2.0p25.f009 gdas.20220101/18/atmos/gdas.t18z.pgrb2.0p25.f009
-bash-4.2$ cmp ../devctrlatmos/gdas.20220101/18/atmos/gdas.t18z.pgrb2b.0p25.f009 gdas.20220101/18/atmos/gdas.t18z.pgrb2b.0p25.f009
-bash-4.2$ cmp ../devctrlatmos/gdas.20220101/18/atmos/gdas.t18z.sfluxgrbf009.grib2 gdas.20220101/18/atmos/gdas.t18z.sfluxgrbf009.grib2
-bash-4.2$ nccmp -dgB ../devctrlatmos/enkfgdas.20220101/18/atmos/mem001/gdas.t18z.sfcf009.nc enkfgdas.20220101/18/atmos/mem001/gdas.t18z.sfcf009.nc             
-bash-4.2$ nccmp -dgB ../devctrlatmos/enkfgdas.20220101/18/atmos/mem001/gdas.t18z.atmf009.nc enkfgdas.20220101/18/atmos/mem001/gdas.t18z.atmf009.nc                        
-bash-4.2$ nccmp -dgB ../devctrlatmos/enkfgdas.20220101/18/atmos/mem023/gdas.t18z.sfcf009.nc enkfgdas.20220101/18/atmos/mem023/gdas.t18z.sfcf009.nc                        
-bash-4.2$ nccmp -dgB ../devctrlatmos/enkfgdas.20220101/18/atmos/mem023/gdas.t18z.atmf009.nc enkfgdas.20220101/18/atmos/mem023/gdas.t18z.atmf009.nc
-bash-4.2$ nccmp -dgB ../devctrlatmos/enkfgdas.20220101/18/atmos/gdas.t18z.atmf006.ensmean.nc enkfgdas.20220101/18/atmos/gdas.t18z.atmf006.ensmean.nc
-bash-4.2$ nccmp -dgB ../devctrlatmos/enkfgdas.20220101/18/atmos/gdas.t18z.sfcf004.ensmean.nc enkfgdas.20220101/18/atmos/gdas.t18z.sfcf004.ensmean.nc
-bash-4.2$ nccmp -dgB ../devctrlatmos/enkfgdas.20220101/18/atmos/gdas.t18z.sfcf009.ensmean.nc enkfgdas.20220101/18/atmos/gdas.t18z.sfcf009.ensmean.nc
-bash-4.2$ nccmp -dgB ../devctrlatmos/enkfgdas.20220101/18/atmos/gdas.t18z.atmf009.ensspread.nc enkfgdas.20220101/18/atmos/gdas.t18z.atmf009.ensspread.nc
-bash-4.2$

Examining next (first full) cycle next...

@KateFriedman-NOAA
Copy link
Member Author

KateFriedman-NOAA commented Aug 25, 2022

Examining the cycled control and test for new UFS_UTILS tag

The following match:

-bash-4.2$ pwd
/scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot/uuctrlatmos
-bash-4.2$ cmp ../devctrlatmos/gdas.20220102/00/atmos/gdas.t00z.prepbufr gdas.20220102/00/atmos/gdas.t00z.prepbufr
-bash-4.2$ cmp ../devctrlatmos/gdas.20220102/00/atmos/gdas.t00z.nsstbufr gdas.20220102/00/atmos/gdas.t00z.nsstbufr
-bash-4.2$ nccmp -dgB ../devctrlatmos/gdas.20220102/00/atmos/gdas.t00z.atmgm3.nc gdas.20220102/00/atmos/gdas.t00z.atmgm3.nc
-bash-4.2$ cmp ../devctrlatmos/gdas.20220102/00/atmos/gdas.t00z.abias_int gdas.20220102/00/atmos/gdas.t00z.abias_int
-bash-4.2$ nccmp -dgB ../devctrlatmos/gdas.20220102/00/atmos/gdas.t00z.atmi003.nc gdas.20220102/00/atmos/gdas.t00z.atmi003.nc
-bash-4.2$ nccmp -dgB ../devctrlatmos/gdas.20220102/00/atmos/gdas.t00z.atminc.nc gdas.20220102/00/atmos/gdas.t00z.atminc.nc
-bash-4.2$ nccmp -dgB ../devctrlatmos/gdas.20220102/00/atmos/gdas.t00z.dtfanl.nc gdas.20220102/00/atmos/gdas.t00z.dtfanl.nc
-bash-4.2$ nccmp -dgB ../devctrlatmos/gdas.20220102/00/atmos/gdas.t00z.atmi009.nc gdas.20220102/00/atmos/gdas.t00z.atmi009.nc
-bash-4.2$ cmp ../devctrlatmos/gdas.20220102/00/atmos/gdas.t00z.abias_pc gdas.20220102/00/atmos/gdas.t00z.abias_pc
-bash-4.2$ cmp ../devctrlatmos/gdas.20220102/00/atmos/gdas.t00z.abias_air gdas.20220102/00/atmos/gdas.t00z.abias_air                                                       
-bash-4.2$ cmp ../devctrlatmos/gdas.20220102/00/atmos/gdas.t00z.abias gdas.20220102/00/atmos/gdas.t00z.abias 
-bash-4.2$ nccmp -dgB ../devctrlatmos/gdas.20220102/00/atmos/gdas.t00z.atmanl.nc gdas.20220102/00/atmos/gdas.t00z.atmanl.nc
-bash-4.2$ nccmp -dgB ../devctrlatmos/gdas.20220102/00/atmos/gdas.t00z.atma003.ensres.nc gdas.20220102/00/atmos/gdas.t00z.atma003.ensres.nc
-bash-4.2$ nccmp -dgB ../devctrlatmos/gdas.20220102/00/atmos/gdas.t00z.atmanl.ensres.nc gdas.20220102/00/atmos/gdas.t00z.atmanl.ensres.nc
-bash-4.2$ nccmp -dgB ../devctrlatmos/gdas.20220102/00/atmos/gdas.t00z.atma009.ensres.nc gdas.20220102/00/atmos/gdas.t00z.atma009.ensres.nc

Diffs start with the sfcanl file:

-bash-4.2$ nccmp -dgB ../devctrlatmos/gdas.20220102/00/atmos/gdas.t00z.sfcanl.nc gdas.20220102/00/atmos/gdas.t00z.sfcanl.nc
DIFFER : VARIABLE : icetk : POSITION : [0,15,639] : VALUES : 3 <> 3.00061

...and then the forecast understandably differs:

-bash-4.2$ nccmp -dgB ../devctrlatmos/gdas.20220102/00/atmos/gdas.t00z.sfcf000.nc gdas.20220102/00/atmos/gdas.t00z.sfcf000.nc
DIFFER : VARIABLE : acond : POSITION : [0,0,0] : VALUES : 0.0152784 <> 0.0152784
-bash-4.2$ nccmp -dgB ../devctrlatmos/gdas.20220102/00/atmos/gdas.t00z.sfcf009.nc gdas.20220102/00/atmos/gdas.t00z.sfcf009.nc                                              
DIFFER : VARIABLE : acond : POSITION : [0,0,0] : VALUES : 0.0120907 <> 0.012088

The sfcanl files differ in size, with the test one using the newer UFS_UTILS being slightly larger:

-bash-4.2$ ll ../devctrlatmos/gdas.20220102/00/atmos/gdas.t00z.sfcanl.nc gdas.20220102/00/atmos/gdas.t00z.sfcanl.nc
-rw-r--r-- 1 Kate.Friedman stmp 23595158 Aug 25 08:35 ../devctrlatmos/gdas.20220102/00/atmos/gdas.t00z.sfcanl.nc
-rw-r--r-- 1 Kate.Friedman stmp 23595203 Aug 25 08:58 gdas.20220102/00/atmos/gdas.t00z.sfcanl.nc

Investigating...

@KateFriedman-NOAA
Copy link
Member Author

Further compared files in the atmos-only run. Both of the tile3 sfcanl_data files differ between the runs while the other tiles match:

-bash-4.2$ pwd
/scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot/uuctrlatmos
-bash-4.2$ nccmp -dgB ../devctrlatmos/gdas.20220102/00/atmos/RESTART/20220101.210000.sfcanl_data.tile1.nc gdas.20220102/00/atmos/RESTART/20220101.210000.sfcanl_data.tile1.nc
-bash-4.2$ nccmp -dgB ../devctrlatmos/gdas.20220102/00/atmos/RESTART/20220101.210000.sfcanl_data.tile2.nc gdas.20220102/00/atmos/RESTART/20220101.210000.sfcanl_data.tile2.nc
-bash-4.2$ nccmp -dgB ../devctrlatmos/gdas.20220102/00/atmos/RESTART/20220101.210000.sfcanl_data.tile3.nc gdas.20220102/00/atmos/RESTART/20220101.210000.sfcanl_data.tile3.nc
DIFFER : VARIABLE : hice : POSITION : [0,99,75] : VALUES : 3 <> 0.998376
-bash-4.2$ nccmp -dgB ../devctrlatmos/gdas.20220102/00/atmos/RESTART/20220101.210000.sfcanl_data.tile4.nc gdas.20220102/00/atmos/RESTART/20220101.210000.sfcanl_data.tile4.nc
-bash-4.2$ nccmp -dgB ../devctrlatmos/gdas.20220102/00/atmos/RESTART/20220101.210000.sfcanl_data.tile5.nc gdas.20220102/00/atmos/RESTART/20220101.210000.sfcanl_data.tile5.nc
-bash-4.2$ nccmp -dgB ../devctrlatmos/gdas.20220102/00/atmos/RESTART/20220101.210000.sfcanl_data.tile6.nc gdas.20220102/00/atmos/RESTART/20220101.210000.sfcanl_data.tile6.nc
-bash-4.2$ nccmp -dgB ../devctrlatmos/gdas.20220102/00/atmos/RESTART/20220102.000000.sfcanl_data.tile1.nc gdas.20220102/00/atmos/RESTART/20220102.000000.sfcanl_data.tile1.nc
-bash-4.2$ nccmp -dgB ../devctrlatmos/gdas.20220102/00/atmos/RESTART/20220102.000000.sfcanl_data.tile2.nc gdas.20220102/00/atmos/RESTART/20220102.000000.sfcanl_data.tile2.nc
-bash-4.2$ nccmp -dgB ../devctrlatmos/gdas.20220102/00/atmos/RESTART/20220102.000000.sfcanl_data.tile3.nc gdas.20220102/00/atmos/RESTART/20220102.000000.sfcanl_data.tile3.nc
DIFFER : VARIABLE : hice : POSITION : [0,99,75] : VALUES : 3 <> 0.999024
-bash-4.2$ nccmp -dgB ../devctrlatmos/gdas.20220102/00/atmos/RESTART/20220102.000000.sfcanl_data.tile4.nc gdas.20220102/00/atmos/RESTART/20220102.000000.sfcanl_data.tile4.nc
-bash-4.2$ nccmp -dgB ../devctrlatmos/gdas.20220102/00/atmos/RESTART/20220102.000000.sfcanl_data.tile5.nc gdas.20220102/00/atmos/RESTART/20220102.000000.sfcanl_data.tile5.nc
-bash-4.2$ nccmp -dgB ../devctrlatmos/gdas.20220102/00/atmos/RESTART/20220102.000000.sfcanl_data.tile6.nc gdas.20220102/00/atmos/RESTART/20220102.000000.sfcanl_data.tile6.nc

These files are produced in the sfcanal job, which uses global_cycle from UFS_UTILS. The sfcanl.nc file is then produced from those sfcanl_data.tile files in the analcalc job (which runs after the sfcanl job has run).

The following changed between the UFS_UTILS hash used in develop currently (a2b0817) and the new tag hash (735e2ba).
https://github.com/ufs-community/UFS_UTILS/compare/a2b0817..735e2ba

Those diffs show the following change to the ush/global_cycle.sh script:

- FNAISC=${FNAISC:-${FIXam}/CFSR.SEAICE.1982.2012.monthly.clim.grb}
+ FNAISC=${FNAISC:-${FIXam}/IMS-NIC.blended.ice.monthly.clim.grb}

I confirmed that different FNAISC monthly.clim.grb fix files were used between the control and test:

-bash-4.2$ grep FNAISC= ../devctrlatmos/logs/2022010200/gdassfcanl.log 
+ global_cycle.sh[283]: FNAISC=/scratch1/NCEPDEV/global/Kate.Friedman/git/develop/fix/fix_am/CFSR.SEAICE.1982.2012.monthly.clim.grb
+ global_cycle.sh[283]: FNAISC=/scratch1/NCEPDEV/global/Kate.Friedman/git/develop/fix/fix_am/CFSR.SEAICE.1982.2012.monthly.clim.grb
-bash-4.2$ grep FNAISC= logs/2022010200/gdassfcanl.log                                                                                                                    
+ global_cycle.sh[283]: FNAISC=/scratch1/NCEPDEV/global/Kate.Friedman/git/feature_dev-wcoss2/fix/fix_am/IMS-NIC.blended.ice.monthly.clim.grb
+ global_cycle.sh[283]: FNAISC=/scratch1/NCEPDEV/global/Kate.Friedman/git/feature_dev-wcoss2/fix/fix_am/IMS-NIC.blended.ice.monthly.clim.grb

I don't see any other diffs between those UFS_UTILS hashes that would impact the sfcanl file.

@GeorgeGayno-NOAA Is the difference I see between my control and test sfcanl files (see prior issue comment) to be expected given the change to the FNAISC fix file? It's odd I only see differences in one of the six tile files. Should I have seen differences in all six tile files? Thanks!

@GeorgeGayno-NOAA
Copy link
Contributor

Further compared files in the atmos-only run. Both of the tile3 sfcanl_data files differ between the runs while the other tiles match:

-bash-4.2$ pwd
/scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot/uuctrlatmos
-bash-4.2$ nccmp -dgB ../devctrlatmos/gdas.20220102/00/atmos/RESTART/20220101.210000.sfcanl_data.tile1.nc gdas.20220102/00/atmos/RESTART/20220101.210000.sfcanl_data.tile1.nc
-bash-4.2$ nccmp -dgB ../devctrlatmos/gdas.20220102/00/atmos/RESTART/20220101.210000.sfcanl_data.tile2.nc gdas.20220102/00/atmos/RESTART/20220101.210000.sfcanl_data.tile2.nc
-bash-4.2$ nccmp -dgB ../devctrlatmos/gdas.20220102/00/atmos/RESTART/20220101.210000.sfcanl_data.tile3.nc gdas.20220102/00/atmos/RESTART/20220101.210000.sfcanl_data.tile3.nc
DIFFER : VARIABLE : hice : POSITION : [0,99,75] : VALUES : 3 <> 0.998376
-bash-4.2$ nccmp -dgB ../devctrlatmos/gdas.20220102/00/atmos/RESTART/20220101.210000.sfcanl_data.tile4.nc gdas.20220102/00/atmos/RESTART/20220101.210000.sfcanl_data.tile4.nc
-bash-4.2$ nccmp -dgB ../devctrlatmos/gdas.20220102/00/atmos/RESTART/20220101.210000.sfcanl_data.tile5.nc gdas.20220102/00/atmos/RESTART/20220101.210000.sfcanl_data.tile5.nc
-bash-4.2$ nccmp -dgB ../devctrlatmos/gdas.20220102/00/atmos/RESTART/20220101.210000.sfcanl_data.tile6.nc gdas.20220102/00/atmos/RESTART/20220101.210000.sfcanl_data.tile6.nc
-bash-4.2$ nccmp -dgB ../devctrlatmos/gdas.20220102/00/atmos/RESTART/20220102.000000.sfcanl_data.tile1.nc gdas.20220102/00/atmos/RESTART/20220102.000000.sfcanl_data.tile1.nc
-bash-4.2$ nccmp -dgB ../devctrlatmos/gdas.20220102/00/atmos/RESTART/20220102.000000.sfcanl_data.tile2.nc gdas.20220102/00/atmos/RESTART/20220102.000000.sfcanl_data.tile2.nc
-bash-4.2$ nccmp -dgB ../devctrlatmos/gdas.20220102/00/atmos/RESTART/20220102.000000.sfcanl_data.tile3.nc gdas.20220102/00/atmos/RESTART/20220102.000000.sfcanl_data.tile3.nc
DIFFER : VARIABLE : hice : POSITION : [0,99,75] : VALUES : 3 <> 0.999024
-bash-4.2$ nccmp -dgB ../devctrlatmos/gdas.20220102/00/atmos/RESTART/20220102.000000.sfcanl_data.tile4.nc gdas.20220102/00/atmos/RESTART/20220102.000000.sfcanl_data.tile4.nc
-bash-4.2$ nccmp -dgB ../devctrlatmos/gdas.20220102/00/atmos/RESTART/20220102.000000.sfcanl_data.tile5.nc gdas.20220102/00/atmos/RESTART/20220102.000000.sfcanl_data.tile5.nc
-bash-4.2$ nccmp -dgB ../devctrlatmos/gdas.20220102/00/atmos/RESTART/20220102.000000.sfcanl_data.tile6.nc gdas.20220102/00/atmos/RESTART/20220102.000000.sfcanl_data.tile6.nc

These files are produced in the sfcanal job, which uses global_cycle from UFS_UTILS. The sfcanl.nc file is then produced from those sfcanl_data.tile files in the analcalc job (which runs after the sfcanl job has run).

The following changed between the UFS_UTILS hash used in develop currently (a2b0817) and the new tag hash (735e2ba). https://github.com/ufs-community/UFS_UTILS/compare/a2b0817..735e2ba

Those diffs show the following change to the ush/global_cycle.sh script:

- FNAISC=${FNAISC:-${FIXam}/CFSR.SEAICE.1982.2012.monthly.clim.grb}
+ FNAISC=${FNAISC:-${FIXam}/IMS-NIC.blended.ice.monthly.clim.grb}

I confirmed that different FNAISC monthly.clim.grb fix files were used between the control and test:

-bash-4.2$ grep FNAISC= ../devctrlatmos/logs/2022010200/gdassfcanl.log 
+ global_cycle.sh[283]: FNAISC=/scratch1/NCEPDEV/global/Kate.Friedman/git/develop/fix/fix_am/CFSR.SEAICE.1982.2012.monthly.clim.grb
+ global_cycle.sh[283]: FNAISC=/scratch1/NCEPDEV/global/Kate.Friedman/git/develop/fix/fix_am/CFSR.SEAICE.1982.2012.monthly.clim.grb
-bash-4.2$ grep FNAISC= logs/2022010200/gdassfcanl.log                                                                                                                    
+ global_cycle.sh[283]: FNAISC=/scratch1/NCEPDEV/global/Kate.Friedman/git/feature_dev-wcoss2/fix/fix_am/IMS-NIC.blended.ice.monthly.clim.grb
+ global_cycle.sh[283]: FNAISC=/scratch1/NCEPDEV/global/Kate.Friedman/git/feature_dev-wcoss2/fix/fix_am/IMS-NIC.blended.ice.monthly.clim.grb

I don't see any other diffs between those UFS_UTILS hashes that would impact the sfcanl file.

@GeorgeGayno-NOAA Is the difference I see between my control and test sfcanl files (see prior issue comment) to be expected given the change to the FNAISC fix file? It's odd I only see differences in one of the six tile files. Should I have seen differences in all six tile files? Thanks!

Tiles 3 and 6 cover the poles, so they are normally the only tiles with ice. When I tested the IMS-NIC ice in my C768 regression test, tile 6 had one ice point different (in the hice record). All other tiles were bit identical. So if you are getting changes to a few points in tile 3 and/or 6, I would say it the change you are seeing is expected.

@KateFriedman-NOAA
Copy link
Member Author

Tiles 3 and 6 cover the poles, so they are normally the only tiles with ice. When I tested the IMS-NIC ice in my C768 regression test, tile 6 had one ice point different (in the hice record). All other tiles were bit identical. So if you are getting changes to a few points in tile 3 and/or 6, I would say it the change you are seeing is expected.

Ok cool, that jives with the difference I got in tile 3 to the hice variable:

-bash-4.2$ nccmp -dgB ../devctrlatmos/gdas.20220102/00/atmos/RESTART/20220101.210000.sfcanl_data.tile3.nc gdas.20220102/00/atmos/RESTART/20220101.210000.sfcanl_data.tile3.nc
DIFFER : VARIABLE : hice : POSITION : [0,99,75] : VALUES : 3 <> 0.998376

The resulting sfcanl.nc file also shows a difference in the icetk (sea ice thickness variable):

-bash-4.2$ nccmp -dgB ../devctrlatmos/gdas.20220102/00/atmos/gdas.t00z.sfcanl.nc gdas.20220102/00/atmos/gdas.t00z.sfcanl.nc
DIFFER : VARIABLE : icetk : POSITION : [0,15,639] : VALUES : 3 <> 3.00061

Thanks for confirming this difference is expected in the ice data by using the different IMS-NIC file!

@KateFriedman-NOAA
Copy link
Member Author

If I copy in the sfcanl_data tile files from the control run into the test run and rerun gdasanalcalc I get an identical sfcanl.nc file between the runs. I then reran the gdasfcst job in the test and obtained matching output with the control run.

The enkf suite between the control and test runs also matched and the output in the test run was not impacted by the global_cycle ice climo file change.

In the end, the only difference between the control and test related to the newer UFS_UTILS tag is a small change in the ice fields related to using the IMS-NIC.blended.ice.monthly.clim.grb now instead of CFSR.SEAICE.1982.2012.monthly.clim.grb.

The change in the UFS_UTILS to use this newer ice climo file was done @ ufs-community/UFS_UTILS@f2e1203 from PR ufs-community/UFS_UTILS#664 and from issue ufs-community/UFS_UTILS#661 in response to global-workflow issue #886.

Replace current default ice climo "CFSR.SEAICE.1982.2012.monthly.clim.grb"
with new IMS-based climo "IMS-NIC.blended.ice.monthly.clim.grb".

Updating to the newer UFS_UTILS ufs_utils_1_8_0 will complete the move to the newer ice climo file as described in issue #886.

KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Aug 26, 2022
- Update develop to the newly released ufs_utils_1_8_0 UFS_UTILS tag.
- New tag contains:
  - cleanup of decommissioned platforms
  - adding support for WCOSS2
  - regression test updates
  - code updates to chgres_cube and emcsfc_snow2mdl
  - some small script updates and cleanup
  - change to the FNAISC file from CFSR.SEAICE.1982.2012.monthly.clim.grb
    to IMS-NIC.blended.ice.monthly.clim.grb

Refs NOAA-EMC#974
KateFriedman-NOAA added a commit that referenced this issue Aug 26, 2022
Update UFS_UTILS tag to ufs_utils_1_8_0

New tag contains:
- cleanup of decommissioned platforms
- adding support for WCOSS2
- regression test updates
- code updates to chgres_cube and emcsfc_snow2mdl
- some small script updates and cleanup
- change to the FNAISC file from CFSR.SEAICE.1982.2012.monthly.clim.grb to IMS-NIC.blended.ice.monthly.clim.grb

Refs #974
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
maintenance Regular updates and maintenance work
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants