Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle HRDPS change from west domain to continental domain #156

Merged
merged 71 commits into from
Mar 15, 2023

Commits on Feb 23, 2023

  1. Chg weather config to 2.5km HRDPS continental product

    ECCC stopped producing 2.5km HRDPS west product on 22-Feb-2023. It was part of
    the polar stereographic grid product suite that they replaced with the rotated
    lat-lon grid product, but that product is only available for the continental
    domain.
    douglatornell committed Feb 23, 2023
    Configuration menu
    Copy the full SHA
    440f897 View commit details
    Browse the repository at this point in the history

Commits on Feb 24, 2023

  1. Add --run-date option to download_weather worker

    Replaces --yesterday option. ECCC hpfx server holds ~60 days of archived
    model products. So, we now have the option of downloading for dates further into
    the past than just the previous day.
    douglatornell committed Feb 24, 2023
    Configuration menu
    Copy the full SHA
    5a4bde3 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    a7427a1 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    6e10eef View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    1782ea4 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    cb1381e View commit details
    Browse the repository at this point in the history

Commits on Feb 26, 2023

  1. Configuration menu
    Copy the full SHA
    00c2f51 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    4922459 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    a1d97c0 View commit details
    Browse the repository at this point in the history
  4. Refactor _define_forecast_segments_nowcast()

    Includes part of work required for issue #23
    douglatornell committed Feb 26, 2023
    Configuration menu
    Copy the full SHA
    4292e18 View commit details
    Browse the repository at this point in the history
  5. Refactor _define_forecast_segments_forecast2()

    Includes part of work required for issue #23
    douglatornell committed Feb 26, 2023
    Configuration menu
    Copy the full SHA
    5a36354 View commit details
    Browse the repository at this point in the history

Commits on Mar 2, 2023

  1. Add pywgrib2_xr package as dependency

    Use SalishSeaCast branch in SalishSeaCast/pywgrib2_xr fork to work under
    Python 3.10 and xarray>=2022.11. Also requires installation of libwgrib2
    from SalishSeaCast branch in SalishSeaCast/libwgrib2 fork.
    douglatornell committed Mar 2, 2023
    Configuration menu
    Copy the full SHA
    5621470 View commit details
    Browse the repository at this point in the history
  2. Refactor _rotate_grib_wind()

    Includes part of work required for issues #23 and #43
    douglatornell committed Mar 2, 2023
    Configuration menu
    Copy the full SHA
    fd18ca7 View commit details
    Browse the repository at this point in the history

Commits on Mar 3, 2023

  1. Improve GHA pytest-with-coverage workflow comments

    Mostly to trigger another workflow run with tweaked setup.py in the pywgrib2_xr
    fork.
    douglatornell committed Mar 3, 2023
    Configuration menu
    Copy the full SHA
    dfc9520 View commit details
    Browse the repository at this point in the history
  2. Install vendored libwgrib2 in GHA pytest-with-coverage

    Mostly to trigger another workflow run with tweaked setup.py in the pywgrib2_xr
    fork.
    douglatornell committed Mar 3, 2023
    Configuration menu
    Copy the full SHA
    6212d6f View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    d54891d View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    9b8fab3 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    63b4eff View commit details
    Browse the repository at this point in the history
  6. Drop GRIB2 file existence & 0 length checks

    Artefact of early days of SalishSeaNowcast development before automation
    included assurance that
    douglatornell committed Mar 3, 2023
    Configuration menu
    Copy the full SHA
    0b126c1 View commit details
    Browse the repository at this point in the history
  7. Factor out _wgrib2_append() function

    Handles finicky details of pywgrib2_xr.wgrib():
    * all pywgrib2_xr.wgrib() args must be strings
    * files must be freed after use to close them
    douglatornell committed Mar 3, 2023
    Configuration menu
    Copy the full SHA
    2763c47 View commit details
    Browse the repository at this point in the history
  8. Refactor _collect_grib_scalars()

    Includes part of work required for issues #23 and #43
    douglatornell committed Mar 3, 2023
    Configuration menu
    Copy the full SHA
    8dd5d2f View commit details
    Browse the repository at this point in the history
  9. Refactor _concat_hourly_gribs()

    Includes part of work required for issue #23.
    douglatornell committed Mar 3, 2023
    Configuration menu
    Copy the full SHA
    5d6a8c0 View commit details
    Browse the repository at this point in the history
  10. Drop TCDC_Sfc variable from weather collection

    Total cloud in percent that was used for parametrization of radiation missing
    from 2007-2014 GEMLAM. No longer needed.
    douglatornell committed Mar 3, 2023
    Configuration menu
    Copy the full SHA
    6ff7944 View commit details
    Browse the repository at this point in the history

Commits on Mar 4, 2023

  1. Refactor _crop_to_watersheds()

    Includes factoring out _wgrib2_crop() function to handles finicky details of
    pywgrib2_xr.wgrib():
    * all pywgrib2_xr.wgrib() args must be strings
    * files must be freed after use to close them
    
    Also includes part of work required for issue #23.
    douglatornell committed Mar 4, 2023
    Configuration menu
    Copy the full SHA
    e16fad3 View commit details
    Browse the repository at this point in the history
  2. Temporarily remove launch of upload_forcing worker

    Need this so that weather collection, etc. can run in automation before
    grib_to_netcdf rework is ready for use.
    douglatornell committed Mar 4, 2023
    Configuration menu
    Copy the full SHA
    ecb2e84 View commit details
    Browse the repository at this point in the history

Commits on Mar 6, 2023

  1. Configuration menu
    Copy the full SHA
    fcfff41 View commit details
    Browse the repository at this point in the history

Commits on Mar 8, 2023

  1. Revert addition of pywgrib2_xr pkg as dependency

    Using forks of pywgrib2_xr and libwgrib2 was yucky, but ultimately, the
    inability of wgrib2 to export the rotated lat-lon grid to netCDF was the
    death knell.
    douglatornell committed Mar 8, 2023
    Configuration menu
    Copy the full SHA
    a0f70a0 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    aa88e58 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    72c428d View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    e782d24 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    043d3b7 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    42248d2 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    db6b23c View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    417412e View commit details
    Browse the repository at this point in the history

Commits on Mar 9, 2023

  1. Change 2.5km weather variable names to triplets

    We need the MSC file variable name, the GRIB standard variable name, and the
    variable name to use in the NEMO forcing file for processing of the HRDPS
    continental GRIB files.
    douglatornell committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    14da1b9 View commit details
    Browse the repository at this point in the history
  2. Chg forcing filename template from ops_ to hrdps_

    So that we can clearly distinguish from HRDPS continental grid in contrast
    to those from HRDPS west for GEMLAM experimental grids.
    douglatornell committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    5ca1efd View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    8395af2 View commit details
    Browse the repository at this point in the history

Commits on Mar 10, 2023

  1. Set logging level for cfgrib library to WARNING

    DEBUG is too noisy.
    douglatornell committed Mar 10, 2023
    Configuration menu
    Copy the full SHA
    e90f26d View commit details
    Browse the repository at this point in the history
  2. Add _apportion_accumulation_vars()

    Add list of 2.5km forecast accumulation variables to weather config.
    
    Also fix bugs in grib files date and hours selection.
    douglatornell committed Mar 10, 2023
    Configuration menu
    Copy the full SHA
    fb92d18 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    30b58f1 View commit details
    Browse the repository at this point in the history
  4. Add testing mock for _apportion_accumulation_vars()

    Missed in fb92d1.
    douglatornell committed Mar 10, 2023
    Configuration menu
    Copy the full SHA
    112e132 View commit details
    Browse the repository at this point in the history
  5. Add cfgrib to loggers config test

    Missed in e90f26.
    douglatornell committed Mar 10, 2023
    Configuration menu
    Copy the full SHA
    3898e16 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    e5635f3 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    c00117d View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    a80de78 View commit details
    Browse the repository at this point in the history
  9. Add _calc_grid_angle()

    douglatornell committed Mar 10, 2023
    Configuration menu
    Copy the full SHA
    c1ad362 View commit details
    Browse the repository at this point in the history

Commits on Mar 11, 2023

  1. Configuration menu
    Copy the full SHA
    3aaf5ac View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    db26d36 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    6559288 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    944f8ba View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    5175936 View commit details
    Browse the repository at this point in the history

Commits on Mar 12, 2023

  1. Drop unneeded coordinate variables

    "time", "longitude", and "latitude" variables from GRIB files are replaced by
    time_counter, "nav_lon", and "nav_lat" in NEMO datasets, so they are unneeded.
    douglatornell committed Mar 12, 2023
    Configuration menu
    Copy the full SHA
    ffc1fc6 View commit details
    Browse the repository at this point in the history
  2. Add _improve_metadata()

    douglatornell committed Mar 12, 2023
    Configuration menu
    Copy the full SHA
    711d49f View commit details
    Browse the repository at this point in the history
  3. Move setting dataset history attr to _write_netcdf()

    Necessary so that we can use xarray.combine_by_coords() to build 3-section
    run-date dataset.
    douglatornell committed Mar 12, 2023
    Configuration menu
    Copy the full SHA
    5ab4f7e View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    2efbc28 View commit details
    Browse the repository at this point in the history

Commits on Mar 13, 2023

  1. Configuration menu
    Copy the full SHA
    b42988f View commit details
    Browse the repository at this point in the history
  2. More timezone fun

    douglatornell committed Mar 13, 2023
    Configuration menu
    Copy the full SHA
    f71fdb9 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    fd5f233 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    deebae2 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    26b02b5 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    b095d7b View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    e08666b View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    f16a9ca View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    5b1b0bc View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    27cfb2a View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    faf3024 View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    5445eba View commit details
    Browse the repository at this point in the history
  13. Configuration menu
    Copy the full SHA
    18f7adc View commit details
    Browse the repository at this point in the history

Commits on Mar 14, 2023

  1. Configuration menu
    Copy the full SHA
    3b19e9d View commit details
    Browse the repository at this point in the history
  2. Use persistent dask cluster on salish for processing

    Reduces grib_to_netcdf memory footprint and processing time a little compared to
    using dask's default threads scheduler.
    douglatornell committed Mar 14, 2023
    Configuration menu
    Copy the full SHA
    f43d102 View commit details
    Browse the repository at this point in the history
  3. Chg _write_netcdf() file created log msg to info level

    Also improve test coverage of _write_netcdf().
    douglatornell committed Mar 14, 2023
    Configuration menu
    Copy the full SHA
    b547803 View commit details
    Browse the repository at this point in the history