Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature #1626 TCDiag updates to wrapper and basic use case #2248

Merged
merged 61 commits into from
Jul 18, 2023
Merged

Feature #1626 TCDiag updates to wrapper and basic use case #2248

merged 61 commits into from
Jul 18, 2023

Commits on Jun 14, 2023

  1. update diff util and test that changes properly diff failling diffs, … 

    …ci-run-diff
    @georgemccabe
    georgemccabe committed Jun 14, 2023
    Configuration menu
    Copy the full SHA
    2c0332a View commit details
    Browse the repository at this point in the history

  2. turn off use case

    @georgemccabe
    georgemccabe committed Jun 14, 2023
    Configuration menu
    Copy the full SHA
    905b409 View commit details
    Browse the repository at this point in the history

  3. catch any exception that occurs in NetCDF file diff, ci-skip-unit-tests

    @georgemccabe
    georgemccabe committed Jun 14, 2023
    Configuration menu
    Copy the full SHA
    c2b6e3a View commit details
    Browse the repository at this point in the history

  4. catch any exception from file diff instead of just in NetCDF diff, ci… 

    …-skip-unit-tests
    @georgemccabe
    georgemccabe committed Jun 14, 2023
    Configuration menu
    Copy the full SHA
    5fdee80 View commit details
    Browse the repository at this point in the history

  5. forgot to trigger diff, ci-run-diff, ci-skip-unit-tests

    @georgemccabe
    georgemccabe committed Jun 14, 2023
    Configuration menu
    Copy the full SHA
    21af4b2 View commit details
    Browse the repository at this point in the history

  6. use pandas.isnull instead of numpy.isnan to prevent exceptions from o… 

    …ccurring, ci-run-diff, ci-skip-unit-tests
    @georgemccabe
    georgemccabe committed Jun 14, 2023
    Configuration menu
    Copy the full SHA
    2172134 View commit details
    Browse the repository at this point in the history

  7. capture failure in diff script, ci-run-diff, ci-skip-unit-tests

    @georgemccabe
    georgemccabe committed Jun 14, 2023
    Configuration menu
    Copy the full SHA
    eefaa0c View commit details
    Browse the repository at this point in the history

  8. ignore .idea directory that was auto-generated

    @georgemccabe
    georgemccabe committed Jun 14, 2023
    Configuration menu
    Copy the full SHA
    0201daa View commit details
    Browse the repository at this point in the history

  9. fix incorrect syntax, ci-run-diff, ci-skip-unit-tests

    @georgemccabe
    georgemccabe committed Jun 14, 2023
    Configuration menu
    Copy the full SHA
    194088d View commit details
    Browse the repository at this point in the history

  10. handle failing comparison when NetCDF values are stored as a string, … 

    …ci-run-diff, ci-skip-unit-tests
    @georgemccabe
    georgemccabe committed Jun 14, 2023
    Configuration menu
    Copy the full SHA
    8017d99 View commit details
    Browse the repository at this point in the history

  11. Preserve backwards compatibility by forcing TC_PAIRS_RUN_ONCE=False i… 

    …f LOOP_ORDER is still used and warn that it should be removed. Add debug log message to alert user that only the first init/valid time is being processed and how to change that if desired
    @georgemccabe
    georgemccabe committed Jun 14, 2023
    Configuration menu
    Copy the full SHA
    d7981f2 View commit details
    Browse the repository at this point in the history

  12. Merge branch 'bugfix_gha_diff' of github.com:dtcenter/METplus into bu… 

    …gfix_gha_diff
    @georgemccabe
    georgemccabe committed Jun 14, 2023
    Configuration menu
    Copy the full SHA
    edbbe39 View commit details
    Browse the repository at this point in the history

  13. add pandas to diff conda environment used for diff tests in GHA to pr… 

    …operly check for null/NaN values
    @georgemccabe
    georgemccabe committed Jun 14, 2023
    Configuration menu
    Copy the full SHA
    c8a80b2 View commit details
    Browse the repository at this point in the history

  14. turn off use cases after tests succeeded

    @georgemccabe
    georgemccabe committed Jun 14, 2023
    Configuration menu
    Copy the full SHA
    353fd15 View commit details
    Browse the repository at this point in the history

Commits on Jun 15, 2023

  1. update apt-get to fix poppler-utils install error, add specific versi… 

    …on for pandas
    @georgemccabe
    georgemccabe committed Jun 15, 2023
    Configuration menu
    Copy the full SHA
    52f5ceb View commit details
    Browse the repository at this point in the history

  2. run use case to test that diff.v5.1 conda env was created properly, c… 

    …i-run-diff
    @georgemccabe
    georgemccabe committed Jun 15, 2023
    Configuration menu
    Copy the full SHA
    1421c18 View commit details
    Browse the repository at this point in the history

  3. check if values are completely equal inside _is_equal_rounded functio… 

    …n, call that function in NC value diff to prevent differences that are smaller than 5 decimal places
    @georgemccabe
    georgemccabe committed Jun 15, 2023
    Configuration menu
    Copy the full SHA
    b29a7ac View commit details
    Browse the repository at this point in the history

  4. turn off all use case groups before re-opening PR

    @georgemccabe
    georgemccabe committed Jun 15, 2023
    Configuration menu
    Copy the full SHA
    2b9bea5 View commit details
    Browse the repository at this point in the history

  5. if both netcdf values are masked, consider them equal

    @georgemccabe
    georgemccabe committed Jun 15, 2023
    Configuration menu
    Copy the full SHA
    0cbcff2 View commit details
    Browse the repository at this point in the history

Commits on Jun 16, 2023

  1. updates to the wrapper to properly set the -data arguments

    @georgemccabe
    georgemccabe committed Jun 16, 2023
    Configuration menu
    Copy the full SHA
    70e5b5e View commit details
    Browse the repository at this point in the history

Commits on Jun 28, 2023

  1. Merge branch 'develop' into feature_1626_tc_diag_v2

    @georgemccabe
    georgemccabe committed Jun 28, 2023
    Configuration menu
    Copy the full SHA
    d8c4298 View commit details
    Browse the repository at this point in the history

Commits on Jul 5, 2023

  1. Added some additional description to the 'Description' section for TC… 

    …-DIAG in the METplus wrappers page, as the initial description seems overly terse.
    @jvigh
    jvigh committed Jul 5, 2023
    Configuration menu
    Copy the full SHA
    7dee5a9 View commit details
    Browse the repository at this point in the history

Commits on Jul 6, 2023

  1. As per #2201 (for #1626), reviewed and enhanced the TC-Diag basic use… 

    … case example documentation.
    @jvigh
    jvigh committed Jul 6, 2023
    Configuration menu
    Copy the full SHA
    2e39caa View commit details
    Browse the repository at this point in the history

  2. Corrected link to MET config file.

    @jvigh
    jvigh committed Jul 6, 2023
    Configuration menu
    Copy the full SHA
    a839509 View commit details
    Browse the repository at this point in the history

  3. add TropicalCycloneUseCase quick search keyword and add it the keywor… 

    …d to each relevant use case
    @georgemccabe
    georgemccabe committed Jul 6, 2023
    Configuration menu
    Copy the full SHA
    e2756b7 View commit details
    Browse the repository at this point in the history

Commits on Jul 7, 2023

  1. As per #1626, modified the use case to use data form a more recent st… 

    …orm (Bret 2023).
    @jvigh
    jvigh committed Jul 7, 2023
    Configuration menu
    Copy the full SHA
    30927a9 View commit details
    Browse the repository at this point in the history

  2. As per #1626, found one more fv3 ref to remove.

    @jvigh
    jvigh committed Jul 7, 2023
    Configuration menu
    Copy the full SHA
    30f84a5 View commit details
    Browse the repository at this point in the history

  3. As per #1626, changing GFSO to AVNO.

    @jvigh
    jvigh committed Jul 7, 2023
    Configuration menu
    Copy the full SHA
    1a18c36 View commit details
    Browse the repository at this point in the history

  4. As per #1626, hmmm, the use case didn't work with AVNO. So now trying… 

    … using a TECH of GFSO.
    @jvigh
    jvigh committed Jul 7, 2023
    Configuration menu
    Copy the full SHA
    991d6ad View commit details
    Browse the repository at this point in the history

Commits on Jul 10, 2023

  1. As per #2616, changed GFSO to AVNO.

    @jvigh
    jvigh committed Jul 10, 2023
    Configuration menu
    Copy the full SHA
    fc5e1f3 View commit details
    Browse the repository at this point in the history

  2. As per #1626, running with LOG_LEVEL=DEBUG as per @georgemccabe's sug… 

    …gestion.
    @jvigh
    jvigh committed Jul 10, 2023
    Configuration menu
    Copy the full SHA
    e3436b6 View commit details
    Browse the repository at this point in the history

  3. As per #1626, changing input file template.

    @jvigh
    jvigh committed Jul 10, 2023
    Configuration menu
    Copy the full SHA
    829b965 View commit details
    Browse the repository at this point in the history

  4. As per @georgemccabe's suggestion, upping the logging verbosity for #… 

    …1626.
    @jvigh
    jvigh committed Jul 10, 2023
    Configuration menu
    Copy the full SHA
    0179b00 View commit details
    Browse the repository at this point in the history

  5. As per @georgemccabe's suggestion, also setting SCRUB_STAGING_DIR = F… 

    …alse for #1626.
    @jvigh
    jvigh committed Jul 10, 2023
    Configuration menu
    Copy the full SHA
    a0c16fb View commit details
    Browse the repository at this point in the history

  6. As per @georgemccabe's request for #1626, changing MODEL to GFSO. 

    TC_DIAG_INPUT1_TECH_ID_LIST is still AVNO.
    @jvigh
    jvigh committed Jul 10, 2023
    Configuration menu
    Copy the full SHA
    e50088b View commit details
    Browse the repository at this point in the history

  7. As per #1626, setting 

    TC_DIAG_INIT_INCLUDE = 2023062012

As it threw an error:
```
ERROR  :
ERROR  : process_tracks() -> set the "init_inc" config option to select one of the 31 track initialization times between 2023061700 and 2023062412.
ERROR  :
```

@georgemccabe, it seems kind of redudant to have to set the init time twice (once through this setting and again through the BEGIN and END looper). Is this the expected behavior?
    @jvigh
    jvigh committed Jul 10, 2023
    Configuration menu
    Copy the full SHA
    37c86e6 View commit details
    Browse the repository at this point in the history

  8. #1626: 

    @georgemccabe, I got it to run further, but it barfed when it couldn't find PRMSL/L0.

As per: https://luckgrib.com/blog/2018/08/28/gfs-sea-level-pressure.html
the GFS now has two sea level pressures. PRMSL is heavily smoothed, while MSLET is much closer to reality. So changing PRMSL to MSLET in this use case.
    @jvigh
    jvigh committed Jul 10, 2023
    Configuration menu
    Copy the full SHA
    48e8955 View commit details
    Browse the repository at this point in the history

Commits on Jul 11, 2023

  1. @georgemccabe It seems that the "old" FV3-GFS data used "L0", but now… 

    … "Z0" might be the correct level. I'm about to try it to see if this works.
    @jvigh
    jvigh committed Jul 11, 2023
    Configuration menu
    Copy the full SHA
    64fc15c View commit details
    Browse the repository at this point in the history

  2. #1626 @georgemccabe. Using MSLET/Z0 didn't work. Now trying PRMSL/Z0.

    @jvigh
    jvigh committed Jul 11, 2023
    Configuration menu
    Copy the full SHA
    e3b05f7 View commit details
    Browse the repository at this point in the history

  3. #1626 and @georgemccabe: Trying specificying no level.

    @jvigh
    jvigh committed Jul 11, 2023
    Configuration menu
    Copy the full SHA
    1895c3e View commit details
    Browse the repository at this point in the history

  4. use Z0 for PRMSL level and use current init time for init_inc value

    @georgemccabe
    georgemccabe committed Jul 11, 2023
    Configuration menu
    Copy the full SHA
    2079e5b View commit details
    Browse the repository at this point in the history

  5. #1626: Using time templates instead of hard-coding the times.

    @jvigh
    jvigh committed Jul 11, 2023
    Configuration menu
    Copy the full SHA
    6f54c37 View commit details
    Browse the repository at this point in the history

  6. Merge branch 'feature_1626_tc_diag_v2' of github.com:dtcenter/METplus… 

    … into feature_1626_tc_diag_v2

# Conflicts:
#	parm/use_cases/met_tool_wrapper/TCDiag/TCDiag.conf
    @jvigh
    jvigh committed Jul 11, 2023
    Configuration menu
    Copy the full SHA
    7d678a4 View commit details
    Browse the repository at this point in the history

  7. #1626: Trying LEAD_SEQ to limit the extent of the run.

    @jvigh
    jvigh committed Jul 11, 2023
    Configuration menu
    Copy the full SHA
    d3f9ca1 View commit details
    Browse the repository at this point in the history

  8. fixed typo in variable name

    @georgemccabe
    georgemccabe committed Jul 11, 2023
    Configuration menu
    Copy the full SHA
    6493671 View commit details
    Browse the repository at this point in the history

  9. #1626: Now checking to see if it can run using MSLET instead of PRMSL.

    @jvigh
    jvigh committed Jul 11, 2023
    Configuration menu
    Copy the full SHA
    19d6bd2 View commit details
    Browse the repository at this point in the history

  10. #1626: Cleaning up for finalization.

    @jvigh
    jvigh committed Jul 11, 2023
    Configuration menu
    Copy the full SHA
    cc58315 View commit details
    Browse the repository at this point in the history

  11. #1626: Removed data staging.

    @jvigh
    jvigh committed Jul 11, 2023
    Configuration menu
    Copy the full SHA
    225a83e View commit details
    Browse the repository at this point in the history

  12. add example of complex subset of grib2 data

    @georgemccabe
    georgemccabe committed Jul 11, 2023
    Configuration menu
    Copy the full SHA
    4de1c62 View commit details
    Browse the repository at this point in the history

  13. Merge branch 'feature_1626_tc_diag_v2' of github.com:dtcenter/METplus… 

    … into feature_1626_tc_diag_v2
    @georgemccabe
    georgemccabe committed Jul 11, 2023
    Configuration menu
    Copy the full SHA
    a1c6d82 View commit details
    Browse the repository at this point in the history

  14. #1626 @georgemccabe: 

    Verifying that the use case works on subsetted data.
Corrected a typo in the Contributor's Guide for the complex subsetting example.
    @jvigh
    jvigh committed Jul 11, 2023
    Configuration menu
    Copy the full SHA
    99cbe4d View commit details
    Browse the repository at this point in the history

Commits on Jul 12, 2023

  1. add TCDiag basic use case to automated tests and set it to run to test

    @georgemccabe
    georgemccabe committed Jul 12, 2023
    Configuration menu
    Copy the full SHA
    3d207c7 View commit details
    Browse the repository at this point in the history

  2. turn off use case group after confirming it works in automated tests

    @georgemccabe
    georgemccabe committed Jul 12, 2023
    Configuration menu
    Copy the full SHA
    781c085 View commit details
    Browse the repository at this point in the history

Commits on Jul 13, 2023

  1. updated use case to include pressure levels that are found in the new… 

    …ly revised input dataset
    @georgemccabe
    georgemccabe committed Jul 13, 2023
    Configuration menu
    Copy the full SHA
    ff909ed View commit details
    Browse the repository at this point in the history

  2. turn on new use case to test, ci-skip-unit-tests

    @georgemccabe
    georgemccabe committed Jul 13, 2023
    Configuration menu
    Copy the full SHA
    54f0aee View commit details
    Browse the repository at this point in the history

  3. turn off use case for PR

    @georgemccabe
    georgemccabe committed Jul 13, 2023
    Configuration menu
    Copy the full SHA
    cb7678b View commit details
    Browse the repository at this point in the history

Commits on Jul 14, 2023

  1. As per #1616, updated the documentation for the basic TCDiag use case… 

    … to reflect the init time and lead times for this case. Also updated the expected output.
    @jvigh
    jvigh committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    c843beb View commit details
    Browse the repository at this point in the history

  2. As per #1626, fixing formatting of the introductory doc block which g… 

    …ot split off from the main part.
    @jvigh
    jvigh committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    c21d6af View commit details
    Browse the repository at this point in the history

  3. As per #1626, another formatting fix.

    @jvigh
    jvigh committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    408bf1e View commit details
    Browse the repository at this point in the history

  4. @georgemccabe, as per #1626, changing the version of MET in the TCDia… 

    …gConfig_wrapped to 11.1.0. Is this correct?
    @jvigh
    jvigh committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    08aefa8 View commit details
    Browse the repository at this point in the history

  5. #1626. I think my previous attempt at a bulleted list for the novel f… 

    …eatures of the use case broke the documentation build. Trying this mod.
    @jvigh
    jvigh committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    31043a1 View commit details
    Browse the repository at this point in the history