Add tracer interpolation method for missing levels when reading grib2 input #556

JiliDong-NOAA · 2021-07-29T20:26:00Z

A new method ("intrp") is added for tracers to interpolate to the missing isobaric levels for grib2 input. This pressure to pressure interpolation is mostly to handle the combined GFSv16 grib2 files where specific humidity has missing levels all through the depth and assigning a small constant value (e.g. 1E-7) in the whole column is not always desirable particularly in mid to lower troposphere.

The interpolation is a simple linear one in log(P) coordinate.

To use the option, just replace spfh method from "set_to_fill" to "intrp" in varmap. As this is a new option, it is not expected to change/break any current existing applications if "intrp" is not used.

Fixes #555

JiliDong-NOAA · 2021-07-29T20:29:23Z

This is related to issue #555

JiliDong-NOAA · 2021-07-30T03:02:55Z

Not sure how to solve this test error:

/home/runner/work/UFS_UTILS/UFS_UTILS/ufs_utils/sorc/chgres_cube.fd/input_data.F90:6717: error: Member dint2p(PPIN, XXIN, NPIN, PPOUT, XXOUT, NPOUT, LINLOG, XMSG, IER) (function) of module input_data is not documented. (warning treated as error, aborting now)

dint2p is a function added for pres to pres interpolation. The compiling and testing are good on Hera though.

GeorgeGayno-NOAA · 2021-07-30T13:25:32Z

Not sure how to solve this test error:

/home/runner/work/UFS_UTILS/UFS_UTILS/ufs_utils/sorc/chgres_cube.fd/input_data.F90:6717: error: Member dint2p(PPIN, XXIN, NPIN, PPOUT, XXOUT, NPOUT, LINLOG, XMSG, IER) (function) of module input_data is not documented. (warning treated as error, aborting now)

dint2p is a function added for pres to pres interpolation. The compiling and testing are good on Hera though.

@JiliDong-NOAA That is a doxygen error. All new routines must be documented with doxygen.

GeorgeGayno-NOAA · 2021-07-30T13:32:09Z

@JiliDong-NOAA We should add a unit test for this new routine. I can help with that.

JiliDong-NOAA · 2021-07-30T15:54:23Z

Not sure how to solve this test error:
/home/runner/work/UFS_UTILS/UFS_UTILS/ufs_utils/sorc/chgres_cube.fd/input_data.F90:6717: error: Member dint2p(PPIN, XXIN, NPIN, PPOUT, XXOUT, NPOUT, LINLOG, XMSG, IER) (function) of module input_data is not documented. (warning treated as error, aborting now)
dint2p is a function added for pres to pres interpolation. The compiling and testing are good on Hera though.

@JiliDong-NOAA That is a doxygen error. All new routines must be documented with doxygen.

Thanks @GeorgeGayno-NOAA . Doxygen documentation is added to dint2p and all checks have passed.

JiliDong-NOAA · 2021-07-30T15:55:57Z

@JiliDong-NOAA We should add a unit test for this new routine. I can help with that.

@GeorgeGayno-NOAA Please let me know how I should go forward with the unit test as I have no idea how to do that.

GeorgeGayno-NOAA · 2021-07-30T15:59:14Z

@JiliDong-NOAA We should add a unit test for this new routine. I can help with that.

@GeorgeGayno-NOAA Please let me know how I should go forward with the unit test as I have no idea how to do that.

I can help with a unit test when I return from lunch. I will create a simple test and check it into your branch. Make sure I can access your fork.

sorc/chgres_cube.fd/input_data.F90

Fixes ufs-community#555.

GeorgeGayno-NOAA · 2021-07-30T20:16:07Z

I added a unit test at c3bf3fb. You will need to add some comments to the source code. Also, you will want to improve and expand on what I did. For example, the routine has different interpolation options - linear, logp, extrapolation. You want to test each option.

JiliDong-NOAA · 2021-07-30T20:39:29Z

I added a unit test at c3bf3fb. You will need to add some comments to the source code. Also, you will want to improve and expand on what I did. For example, the routine has different interpolation options - linear, logp, extrapolation. You want to test each option.

Thanks @GeorgeGayno-NOAA I will work on it. How would I run the unit test? Using a single job card for "srun ftst_dint2p"?

sorc/chgres_cube.fd/input_data.F90

GeorgeGayno-NOAA · 2021-07-30T20:53:28Z

I added a unit test at c3bf3fb. You will need to add some comments to the source code. Also, you will want to improve and expand on what I did. For example, the routine has different interpolation options - linear, logp, extrapolation. You want to test each option.

Thanks @GeorgeGayno-NOAA I will work on it. How would I run the unit test? Using a single job card for "srun ftst_dint2p"?

You can run it on Hera, Jet or Orion. First, to build_all.sh, add 'make test' after the 'make install'. Then, under ./cmake, edit the mpiexec file for your machine for an account and queue you are eligible to use. That will automatically run all the tests. Log files from the tests will be under ./build/Testing/Temporary.

GeorgeGayno-NOAA · 2021-08-02T14:45:09Z

@JiliDong-NOAA Since you are adding a new option, you will need to update the related documentation:

…olation

JiliDong-NOAA · 2021-08-02T15:46:42Z

I added a unit test at c3bf3fb. You will need to add some comments to the source code. Also, you will want to improve and expand on what I did. For example, the routine has different interpolation options - linear, logp, extrapolation. You want to test each option.

Thanks @GeorgeGayno-NOAA I will work on it. How would I run the unit test? Using a single job card for "srun ftst_dint2p"?

You can run it on Hera, Jet or Orion. First, to build_all.sh, add 'make test' after the 'make install'. Then, under ./cmake, edit the mpiexec file for your machine for an account and queue you are eligible to use. That will automatically run all the tests. Log files from the tests will be under ./build/Testing/Temporary.

dint2p unit test successful after a bug is fixed when using lnP extrapolation (the routine won't use extrapolation option when being called but it is safe to get the big fixed).

The unit test consider the four scenarios:

linear interpolation; no extrapolation
linear interpolation; linear extrapolation
lnP interpolation; no extrapolation
lnP interpolation; lnP extrapolation

Start 6: chgres_cube-ftst_dint2p
6/18 Test CHGRES_CUBE grib2 atmospheric component support #6: chgres_cube-ftst_dint2p ..................... Passed 0.03 sec

There are several unit tests failed which seem not related to my change:

The following tests FAILED:
9 - chgres_cube-ftst_read_sfc_gfs_nemsio (Failed)
10 - chgres_cube-ftst_read_sfc_nemsio (Failed)
11 - chgres_cube-ftst_read_sfc_netcdf (Failed)
12 - chgres_cube-ftst_read_nst_netcdf (Failed)
13 - chgres_cube-ftst_read_nst_nemsio (Failed)
14 - chgres_cube-ftst_read_atm_gaussian_netcdf (Failed)
18 - sfc_climo_gen-ftst_interp (Failed)

GeorgeGayno-NOAA · 2021-08-02T16:10:49Z

@JiliDong-NOAA Don't worry about the tests that failed on Hera. You did not break anything.

JiliDong-NOAA · 2021-08-02T16:26:09Z

@JiliDong-NOAA Since you are adding a new option, you will need to update the related documentation:

https://github.com/JiliDong-NOAA/UFS_UTILS/blob/add_intrp/parm/varmap_tables/README

https://github.com/JiliDong-NOAA/UFS_UTILS/blob/add_intrp/sorc/chgres_cube.fd/docs/user_guide.md

https://github.com/JiliDong-NOAA/UFS_UTILS/blob/add_intrp/docs/source/chgres_cube.rst

related documentation updated

JiliDong-NOAA · 2021-08-11T15:56:21Z

I made one minor change to move zeroing out tracer to the end of filling in default value. Otherwise it may zero out pre-assigned missing value of -999 before we use it to determine whether to fill in. @LarissaReames-NOAA could you review it? Other than that, it looks good to me.

LarissaReames-NOAA · 2021-08-11T16:02:41Z

I made one minor change to move zeroing out tracer to the end of filling in default value. Otherwise it may zero out pre-assigned missing value of -999 before we use it to determine whether to fill in. @LarissaReames-NOAA could you review it? Other than that, it looks good to me.

Ah, yep, great catch. That looks good to me.

JiliDong-NOAA · 2021-08-11T16:08:39Z

I made one minor change to move zeroing out tracer to the end of filling in default value. Otherwise it may zero out pre-assigned missing value of -999 before we use it to determine whether to fill in. @LarissaReames-NOAA could you review it? Other than that, it looks good to me.

Ah, yep, great catch. That looks good to me.

Thanks @LarissaReames-NOAA. I am doing consistency tests on Hera but unable to get consistency.log and summary.log. The individual log files (consistency.log[01-16]) seem ok. Anything I need to change to get those two summary log files?

LarissaReames-NOAA · 2021-08-11T17:43:14Z

I made one minor change to move zeroing out tracer to the end of filling in default value. Otherwise it may zero out pre-assigned missing value of -999 before we use it to determine whether to fill in. @LarissaReames-NOAA could you review it? Other than that, it looks good to me.

Ah, yep, great catch. That looks good to me.

Thanks @LarissaReames-NOAA. I am doing consistency tests on Hera but unable to get consistency.log and summary.log. The individual log files (consistency.log[01-16]) seem ok. Anything I need to change to get those two summary log files?

That's odd. The summary file should be produced so long as all the individual tests complete successfully. I usually only fail to get a summary file when a test fails before it can test for file consistency, so the chgres_cube code fails during execution. I just ran the tests on the most recent code and I get the summary file just fine. You can see the output here: /scratch1/BMC/gsd-fv3/Larissa.Reames/UFS_UTILS_jili/reg_tests/chgres_cube

JiliDong-NOAA · 2021-08-11T18:28:35Z

I made one minor change to move zeroing out tracer to the end of filling in default value. Otherwise it may zero out pre-assigned missing value of -999 before we use it to determine whether to fill in. @LarissaReames-NOAA could you review it? Other than that, it looks good to me.

Ah, yep, great catch. That looks good to me.

Thanks @LarissaReames-NOAA. I am doing consistency tests on Hera but unable to get consistency.log and summary.log. The individual log files (consistency.log[01-16]) seem ok. Anything I need to change to get those two summary log files?

That's odd. The summary file should be produced so long as all the individual tests complete successfully. I usually only fail to get a summary file when a test fails before it can test for file consistency, so the chgres_cube code fails during execution. I just ran the tests on the most recent code and I get the summary file just fine. You can see the output here: /scratch1/BMC/gsd-fv3/Larissa.Reames/UFS_UTILS_jili/reg_tests/chgres_cube

@LarissaReames-NOAA Now I see where the problem is. I didn't change HOMEreg to /scratch1/BMC/gsd-fv3/Larissa.Reames/chgres_cube/reg_tests. After changing that, I can get the summary.log. Noticed several reg test failures 08, 09 and 14 though. I guess the reg tests will be updated later.

LarissaReames-NOAA · 2021-08-11T20:53:11Z

All unit and consistency tests pass on Hera and Jet (other than the expected failures for previous tests using GFS grib2 data). @GeorgeGayno-NOAA can you help run tests on Orion and WCOSS?

GeorgeGayno-NOAA · 2021-08-12T12:39:30Z

All unit and consistency tests pass on Hera and Jet (other than the expected failures for previous tests using GFS grib2 data). @GeorgeGayno-NOAA can you help run tests on Orion and WCOSS?

The tests that failed - will the baseline data need to be updated?

LarissaReames-NOAA · 2021-08-12T15:25:01Z

All unit and consistency tests pass on Hera and Jet (other than the expected failures for previous tests using GFS grib2 data). @GeorgeGayno-NOAA can you help run tests on Orion and WCOSS?

The tests that failed - will the baseline data need to be updated?

Yes, there should be three failures -- all the old tests that used GFS grib2 data. They'll need to be updated, and the change is due to the update of all variable mapping tables to interpolate scattered missing levels.

GeorgeGayno-NOAA · 2021-08-12T15:58:27Z

I ran the tests on WCOSS-Dell. I got four failures, not three:

consistency.log01:<<< 25-KM CONUS GFS GRIB2 TEST FAILED. >>>
consistency.log06:<<< 13-KM NA GFS NCEI GRIB2 TEST FAILED. >>>
consistency.log14:<<< C192 GFS GRIB2 TEST FAILED. >>>
consistency.log16:<<< 25-km CONUS GFS GRIB2 TEST FAILED. <<<

LarissaReames-NOAA · 2021-08-12T16:00:51Z

I ran the tests on WCOSS-Dell. I got four failures, not three:

consistency.log01:<<< 25-KM CONUS GFS GRIB2 TEST FAILED. >>>
consistency.log06:<<< 13-KM NA GFS NCEI GRIB2 TEST FAILED. >>>
consistency.log14:<<< C192 GFS GRIB2 TEST FAILED. >>>
consistency.log16:<<< 25-km CONUS GFS GRIB2 TEST FAILED. <<<

Ah yes, well that last one will fail if you haven't already created the correct baseline file for the system since it's new in this PR. So the first three will need new baselines, and the last one (test 16) will just need to have whatever file you get from the test itself inserted as the standard baseline result.

GeorgeGayno-NOAA · 2021-08-12T16:09:45Z

I ran the tests on WCOSS-Dell. I got four failures, not three:
consistency.log01:<<< 25-KM CONUS GFS GRIB2 TEST FAILED. >>>
consistency.log06:<<< 13-KM NA GFS NCEI GRIB2 TEST FAILED. >>>
consistency.log14:<<< C192 GFS GRIB2 TEST FAILED. >>>
consistency.log16:<<< 25-km CONUS GFS GRIB2 TEST FAILED. <<<
Ah yes, well that last one will fail if you haven't already created the correct baseline file for the system since it's new in this PR. So the first three will need new baselines, and the last one (test 16) will just need to have whatever file you get from the test itself inserted as the standard baseline result.

Oh, I forgot you added a new test. Is this the input data for it? /scratch1/BMC/gsd-fv3/Larissa.Reames/chgres_cube/reg_tests/input_data/gfs.pbgrib2

LarissaReames-NOAA · 2021-08-12T16:14:43Z

Ah yes, well that last one will fail if you haven't already created the correct baseline file for the system since it's new in this PR. So the first three will need new baselines, and the last one (test 16) will just need to have whatever file you get from the test itself inserted as the standard baseline result.

Oh, I forgot you added a new test. Is this the input data for it? /scratch1/BMC/gsd-?>fv3/Larissa.Reames/chgres_cube/reg_tests/input_data/gfs.pbgrib2

Yep, that's the one.

GeorgeGayno-NOAA · 2021-08-13T14:57:36Z

@JiliDong-NOAA I found some bugs in some of the regression test scripts. I will check in the fixes to your branch.

Fixes ufs-community#555.

Fixes ufs-community#555

GeorgeGayno-NOAA · 2021-08-13T18:58:20Z

The consistency tests were run on all Tier 1 machines - Jet, Orion, Hera, WCOSS-Cray and WCOSS-Dell. The baseline data was updated for the three tests that were affected by changes under this PR. And input and baseline data was added for the new consistency test.

GeorgeGayno-NOAA

I have no further comments.

GeorgeGayno-NOAA · 2021-08-13T18:59:20Z

Are we ready to merge?

JeffBeck-NOAA · 2021-08-13T19:01:45Z

Are we ready to merge?

I believe so. Thanks for your help with this PR.

JiliDong-NOAA added 2 commits July 29, 2021 20:02

add interpolation method to fill in missing levels for grib2 input

7f44154

add comment

4e0e6b1

JiliDong-NOAA added 2 commits July 30, 2021 01:42

change double precision to real*8 in dint2p for argument consistency

9e9512f

change dlog in dint2p to log

4b2f9f0

JiliDong-NOAA added 2 commits July 30, 2021 15:23

add Doxygen documentation for dint2p; change intrp from linear to lnp

99220ae

more Doxygen documentation for dint2p

5385997

GeorgeGayno-NOAA reviewed Jul 30, 2021

View reviewed changes

sorc/chgres_cube.fd/input_data.F90 Outdated Show resolved Hide resolved

GeorgeGayno-NOAA reviewed Jul 30, 2021

View reviewed changes

sorc/chgres_cube.fd/input_data.F90 Outdated Show resolved Hide resolved

GeorgeGayno-NOAA reviewed Jul 30, 2021

View reviewed changes

sorc/chgres_cube.fd/input_data.F90 Show resolved Hide resolved

JiliDong-NOAA and others added 3 commits July 30, 2021 18:39

modify Doxygen documentation format for dint2p

86c4919

modify Doxygen documentation date for dint2p

32fdf16

Add unit test for new chgres_cube routine "dint2p".

c3bf3fb

Fixes ufs-community#555.

GeorgeGayno-NOAA reviewed Jul 30, 2021

View reviewed changes

sorc/chgres_cube.fd/input_data.F90 Outdated Show resolved Hide resolved

update unit test for dint2p; bug fix for dint2p when using lnP extrap…

5ad839f

…olation

JiliDong-NOAA added 2 commits August 2, 2021 16:15

add extra point for dint2p unit test

905f6dc

documentation added for new chgres_cube option

2218973

JiliDong-NOAA and others added 2 commits August 11, 2021 11:14

Merge branch 'NOAA-EMC:develop' into add_intrp

657c753

move zeroing out tracers

8ee35eb

LarissaReames-NOAA approved these changes Aug 11, 2021

View reviewed changes

Merge branch 'NOAA-EMC:develop' into add_intrp

1da01e2

GeorgeGayno-NOAA and others added 4 commits August 13, 2021 10:03

Fix bug in ./reg_tests/chgres_cube/driver.orion.sh

2c34a18

Fixes ufs-community#555.

Fix bug in ./reg_tests/chgres_cube/driver.jet.sh

299c53b

Fixes ufs-community#555

Merge branch 'NOAA-EMC:develop' into add_intrp

b3e0e66

Update consistency test script comments.

81915df

Fixes ufs-community#555

GeorgeGayno-NOAA approved these changes Aug 13, 2021

View reviewed changes

JeffBeck-NOAA approved these changes Aug 13, 2021

View reviewed changes

GeorgeGayno-NOAA merged commit 469fd5b into ufs-community:develop Aug 13, 2021

GeorgeGayno-NOAA mentioned this pull request Apr 20, 2022

Update documentation to support SRW APP v2.0 #643

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add tracer interpolation method for missing levels when reading grib2 input #556

Add tracer interpolation method for missing levels when reading grib2 input #556

JiliDong-NOAA commented Jul 29, 2021 •

edited by GeorgeGayno-NOAA

Loading

JiliDong-NOAA commented Jul 29, 2021

JiliDong-NOAA commented Jul 30, 2021

GeorgeGayno-NOAA commented Jul 30, 2021

GeorgeGayno-NOAA commented Jul 30, 2021

JiliDong-NOAA commented Jul 30, 2021

JiliDong-NOAA commented Jul 30, 2021

GeorgeGayno-NOAA commented Jul 30, 2021

GeorgeGayno-NOAA commented Jul 30, 2021

JiliDong-NOAA commented Jul 30, 2021

GeorgeGayno-NOAA commented Jul 30, 2021

GeorgeGayno-NOAA commented Aug 2, 2021

JiliDong-NOAA commented Aug 2, 2021

GeorgeGayno-NOAA commented Aug 2, 2021

JiliDong-NOAA commented Aug 2, 2021

JiliDong-NOAA commented Aug 11, 2021

LarissaReames-NOAA commented Aug 11, 2021

JiliDong-NOAA commented Aug 11, 2021

LarissaReames-NOAA commented Aug 11, 2021

JiliDong-NOAA commented Aug 11, 2021

LarissaReames-NOAA commented Aug 11, 2021

GeorgeGayno-NOAA commented Aug 12, 2021

LarissaReames-NOAA commented Aug 12, 2021

GeorgeGayno-NOAA commented Aug 12, 2021

LarissaReames-NOAA commented Aug 12, 2021

GeorgeGayno-NOAA commented Aug 12, 2021

LarissaReames-NOAA commented Aug 12, 2021 •

edited

Loading

GeorgeGayno-NOAA commented Aug 13, 2021

GeorgeGayno-NOAA commented Aug 13, 2021

GeorgeGayno-NOAA left a comment •

edited

Loading

GeorgeGayno-NOAA commented Aug 13, 2021

JeffBeck-NOAA commented Aug 13, 2021

Add tracer interpolation method for missing levels when reading grib2 input #556

Add tracer interpolation method for missing levels when reading grib2 input #556

Conversation

JiliDong-NOAA commented Jul 29, 2021 • edited by GeorgeGayno-NOAA Loading

JiliDong-NOAA commented Jul 29, 2021

JiliDong-NOAA commented Jul 30, 2021

GeorgeGayno-NOAA commented Jul 30, 2021

GeorgeGayno-NOAA commented Jul 30, 2021

JiliDong-NOAA commented Jul 30, 2021

JiliDong-NOAA commented Jul 30, 2021

GeorgeGayno-NOAA commented Jul 30, 2021

GeorgeGayno-NOAA commented Jul 30, 2021

JiliDong-NOAA commented Jul 30, 2021

GeorgeGayno-NOAA commented Jul 30, 2021

GeorgeGayno-NOAA commented Aug 2, 2021

JiliDong-NOAA commented Aug 2, 2021

GeorgeGayno-NOAA commented Aug 2, 2021

JiliDong-NOAA commented Aug 2, 2021

JiliDong-NOAA commented Aug 11, 2021

LarissaReames-NOAA commented Aug 11, 2021

JiliDong-NOAA commented Aug 11, 2021

LarissaReames-NOAA commented Aug 11, 2021

JiliDong-NOAA commented Aug 11, 2021

LarissaReames-NOAA commented Aug 11, 2021

GeorgeGayno-NOAA commented Aug 12, 2021

LarissaReames-NOAA commented Aug 12, 2021

GeorgeGayno-NOAA commented Aug 12, 2021

LarissaReames-NOAA commented Aug 12, 2021

GeorgeGayno-NOAA commented Aug 12, 2021

LarissaReames-NOAA commented Aug 12, 2021 • edited Loading

GeorgeGayno-NOAA commented Aug 13, 2021

GeorgeGayno-NOAA commented Aug 13, 2021

GeorgeGayno-NOAA left a comment • edited Loading

Choose a reason for hiding this comment

GeorgeGayno-NOAA commented Aug 13, 2021

JeffBeck-NOAA commented Aug 13, 2021

JiliDong-NOAA commented Jul 29, 2021 •

edited by GeorgeGayno-NOAA

Loading

LarissaReames-NOAA commented Aug 12, 2021 •

edited

Loading

GeorgeGayno-NOAA left a comment •

edited

Loading