Feature #2769 tc_diag_driver_v0.11.0 (#2812)

* Per #2769, update to tc_diag_driver version 0.11.0. * Per #2769, add tmp_nc_diag_flag config option to retain temporary files. * Per #2769, change default regrdding method from NEAREST to BILIN. * Per #2769, add checks to make sure input files exist. * Fix the handling of diagnostic name and units strings. Both are limited to 7 characters but the units are enclosed in parantheses, increasing the field width to 9. If the strings are too long, truncate them and print warnings like this: WARNING: OutFileInfo::write_cira_diag_vals() -> long diagnostic name "850TANGXXX" truncated to "850TANG"! WARNING: OutFileInfo::write_cira_diag_vals() -> long diagnostic units string "(10^7C/MXXX)" truncated to "(10^7C/M)"! Pad the first 2 columns out to widths of 7 and 9 and set the inter-column spacing between columns 2 and 3 to 0. This could technically introduce parsing problems when the units are 7 characters long and the values have 5 digits, but this is the logic needed to exactly replicate the existing output. * Just whitespace * Per #2769, remove the tmp_nc_diag_flag TC-Diag config file option in favor of the MET_KEEP_TEMP_FILE environment variable option that Howard is adding on his feature_2772_python_embedding_json branch. I did add documentation about that option here even though the functionality is coming from Howard's branch. * Per #2769, remove tmp_nc_diag_flag that has been removed. * Per #2769, remove mention of tmp_nc_diag_flag option that has been removed. * Per #2769, upgrade to METbaseimage version 3.2 to provide SciPy in the Python environment which is needed by TC-Diag. * Per #2769, add SciPy to the list of required Python packages. * Per #2769, tweak AppendixF details. * Per #2769, update release date for MET-12.0.0-beta3 to be 20240207 since we didn't get it out on the 6th.
dtcenter · Feb 7, 2024 · afb4576 · afb4576
1 parent e6706e6
commit afb4576
Show file tree

Hide file tree

Showing 22 changed files with 569 additions and 288 deletions.
diff --git a/.github/jobs/set_job_controls.sh b/.github/jobs/set_job_controls.sh
@@ -6,7 +6,7 @@ run_unit_tests=false
 run_diff=false
 run_update_truth=false
 met_base_repo=met-base
-met_base_tag=v3.1
+met_base_tag=v3.2
 input_data_version=develop
 truth_data_version=develop
 

diff --git a/.github/workflows/build_docker_and_trigger_metplus.yml b/.github/workflows/build_docker_and_trigger_metplus.yml
@@ -29,7 +29,7 @@ jobs:
         env:
           SOURCE_BRANCH: ${{ steps.get_branch_name.outputs.branch_name }}-lite
           MET_BASE_REPO: met-base
-          MET_BASE_TAG: v3.1
+          MET_BASE_TAG: v3.2
 
       - name: Push Docker Image
         run: .github/jobs/push_docker_image.sh

diff --git a/data/config/TCDiagConfig_default b/data/config/TCDiagConfig_default
@@ -85,8 +85,8 @@ domain_info = [
 // May be set separately in each data "field" array entry
 //
 regrid = {
-   method     = NEAREST;
-   width      = 1;
+   method     = BILIN;
+   width      = 2;
    vld_thresh = 0.5;
    shape      = SQUARE;
 }

diff --git a/docs/Users_Guide/appendixF.rst b/docs/Users_Guide/appendixF.rst
@@ -22,15 +22,19 @@ In order to use Python embedding, a local Python installation must be available
 
 3. **NumPy** Python package
 
-4. **netCDF4** Python package
+4. **Pandas** Python package
 
-5. **Pandas** Python package
+5. **Xarray** Python package
 
-6. **Xarray** Python package
+6. **YAML** Python package
 
-7. **YAML** Python package
+7. **SciPy** Python package
 
-Users should be aware that in some cases, the C-language Python header files and libraries may be deleted at the end of the Python installation process, and they may need to confirm their availability prior to compiling MET. Once the user has confirmed the above requirements are satisfied, they can compile the MET software for Python embedding by passing the **\-\-enable-python** option to the **configure** script on the command line. This will link the MET C++ code directly to the Python libraries. The **NumPy** and **netCDF4** Python packages are required by the Python scripts included with the MET software that facilitate the passing of data in memory and the reading and writing of temporary files when Python embedding is used. The **YAML** Python package is required by the tropical cyclone diagnostics Python scripts called by the TC-Diag tool.
+8. **netCDF4** Python package
+
+Users should be aware that in some cases, the C-language Python header files and libraries may be deleted at the end of the Python installation process, and they may need to confirm their availability prior to compiling MET. Once the user has confirmed the above requirements are satisfied, they can compile the MET software for Python embedding by passing the **\-\-enable-python** option to the **configure** script on the command line. This will link the MET C++ code directly to the Python libraries.
+
+The **NumPy**, **Xarray**, and **Pandas** Python packages are required by the Python scripts included with the MET software that facilitate the passing of data in memory. The *SciPy** and **YAML** Python packages are required by the tropical cyclone diagnostics Python scripts called by the TC-Diag tool. The **netCDF4** package is used for reading and writing temporary files for Python embedding, but only when the **MET_PYTHON_TMP_FORMAT** environment variable is set to `netcdf` at runtime. 
 
 In addition to using **\-\-enable-python** with **configure** as mentioned above, the following environment variables must also be set prior to executing **configure**: **MET_PYTHON_BIN_EXE**, **MET_PYTHON_CC**, and **MET_PYTHON_LD**. These may either be set as environment variables or as command line options to **configure**. These environment variables are used when building MET to enable the compiler to find the requisite Python executable, header files, and libraries in the user's local filesystem. Fortunately, Python provides a way to set these variables properly. This frees the user from the necessity of having any expert knowledge of the compiling and linking process. Along with the **Python** executable in the users local Python installation, there should be another executable called **python3-config**, whose output can be used to set these environment variables as follows:
 

diff --git a/docs/Users_Guide/release-notes.rst b/docs/Users_Guide/release-notes.rst
@@ -9,11 +9,12 @@ When applicable, release notes are followed by the GitHub issue number which des
 enhancement, or new feature (`MET GitHub issues <https://github.com/dtcenter/MET/issues>`_).
 Important issues are listed **in bold** for emphasis.
 
-MET Version 12.0.0-beta3 Release Notes (20240206)
+MET Version 12.0.0-beta3 Release Notes (20240207)
 -------------------------------------------------
 
   .. dropdown:: Repository, build, and test
 
+     * Enhance METbaseimage to install SciPy Python package needed by the MET TC-Diag tool (`METbaseimage#20 <https://github.com/dtcenter/METbaseimage/issues/20>`_).
      * Remove the SonarQube token from the properties file (`#2757 <https://github.com/dtcenter/MET/issues/2757>`_).
      * Repository cleanup of stale code and configuration consistency (`#2776 <https://github.com/dtcenter/MET/issues/2776>`_).
      * Add new example installation configuration files for Intel compiler users (`#2785 <https://github.com/dtcenter/MET/issues/2785>`_).

diff --git a/docs/Users_Guide/tc-diag.rst b/docs/Users_Guide/tc-diag.rst
@@ -63,6 +63,8 @@ Optional Arguments for tc_diag
 
 6. The **-v level** option indicates the desired level of verbosity. The contents of "level" will override the default setting of 2. Setting the verbosity to 0 will make the tool run with no log messages, while increasing the verbosity above 1 will increase the amount of logging.
 
+.. note:: Setting the **MET_KEEP_TEMP_FILE** (:numref:`met_keep_temp_file`) environment variable retains the temporary NetCDF cylindrical coordinate files for development, testing, and debugging purposes.
+
 tc_diag Configuration File
 --------------------------
 
@@ -155,13 +157,13 @@ Configuring regridding options
 .. code-block:: none
 
   regrid = {
-     method     = NEAREST;
-     width      = 1;
+     method     = BILIN;
+     width      = 2;
      vld_thresh = 0.5;
      shape      = SQUARE;
   }
 
-The **regrid** dictionary is common to multiple MET tools and is described in :numref:`config_options`. It specifies how the input data should be regridded to cylindrical coordinates prior to compute diagnostics. It can be specified separately in each **data.field** array entry, described below. The default setting uses nearest neighbor interpolation for all fields.
+The **regrid** dictionary is common to multiple MET tools and is described in :numref:`config_options`. It specifies how the input data should be regridded to cylindrical coordinates prior to compute diagnostics. It can be specified separately in each **data.field** array entry, described below. The default setting uses bilinear interpolation for all fields.
 
 Configuring Fields, Levels, and Domains
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

diff --git a/docs/conf.py b/docs/conf.py
@@ -24,7 +24,7 @@
 verinfo = version
 release = f'{version}'
 release_year = '2024'
-release_date = f'{release_year}-02-06'
+release_date = f'{release_year}-02-07'
 copyright = f'{release_year}, {author}'
 
 # -- General configuration ---------------------------------------------------

diff --git a/internal/scripts/docker/Dockerfile b/internal/scripts/docker/Dockerfile
@@ -1,5 +1,5 @@
 ARG MET_BASE_REPO=met-base
-ARG MET_BASE_TAG=v3.1
+ARG MET_BASE_TAG=v3.2
 
 FROM dtcenter/${MET_BASE_REPO}:${MET_BASE_TAG}
 MAINTAINER John Halley Gotway <johnhg@ucar.edu>

diff --git a/internal/scripts/docker/Dockerfile.copy b/internal/scripts/docker/Dockerfile.copy
@@ -1,5 +1,5 @@
 ARG MET_BASE_REPO=met-base-unit-test
-ARG MET_BASE_TAG=v3.1
+ARG MET_BASE_TAG=v3.2
 
 FROM dtcenter/${MET_BASE_REPO}:${MET_BASE_TAG}
 MAINTAINER John Halley Gotway <johnhg@ucar.edu>

diff --git a/internal/test_unit/config/TCDiagConfig_ian b/internal/test_unit/config/TCDiagConfig_ian
@@ -77,8 +77,8 @@ domain_info = [
 // May be set separately in each data "field" array entry
 //
 regrid = {
-   method     = NEAREST;
-   width      = 1;
+   method     = BILIN;
+   width      = 2;
    vld_thresh = 0.5;
    shape      = SQUARE;
 }

diff --git a/scripts/python/tc_diag/compute_tc_diag.py b/scripts/python/tc_diag/compute_tc_diag.py
@@ -85,6 +85,19 @@ def main():
     land_filename   = os.path.expandvars(args.land_file)
     data_filename   = os.path.expandvars(args.data_file)
 
+    # Validate inputs
+    if not os.path.isfile(config_filename):
+        print("Error: Input Config File (" + config_filename + ") does not exist!")
+        sys.exit(1)
+
+    if not os.path.isfile(land_filename):
+        print("Error: Input Land File (" + land_filename + ")does not exist!")
+        sys.exit(1)
+
+    if not os.path.isfile(data_filename):
+        print("Error: Input Data File (" + data_filename + ") does not exist!")
+        sys.exit(1)
+
     # Print verbose arguments
     if args.verbose:
         print("Python Script:\t" + os.path.expandvars(sys.argv[0]))

diff --git a/scripts/python/tc_diag/config/post_resample.yml b/scripts/python/tc_diag/config/post_resample.yml
@@ -100,23 +100,25 @@ pressure_independent_computation_specs:
     callable: *div_vort_func 
     output_vars: [850DVRG, 850VORT] 
     kwargs: {u_name: *u_vert_name, v_name: *v_vert_name, level_hPa: 850, radius_km: *div_vort_max_km}
+    units: [/S, /S]
   div_vort_200: 
     callable: *div_vort_func
     output_vars: [200DVRG, 200VORT] 
     kwargs: {u_name: *u_vert_name, v_name: *v_vert_name, level_hPa: 200, radius_km: *div_vort_max_km}
+    units: [/S, /S]
 
 
   shear: 
     batch_order: 1
     callable: tc_diag_driver.met_diag_vars.shear 
     output_vars: [SHR_MAG, SHR_HDG] 
-    kwargs: {u_name: u, v_name: v, bottom_hPa: 850, top_hPa: 200}
+    kwargs: {u_name: U, v_name: V, bottom_hPa: 850, top_hPa: 200}
     unit_converters: [tc_diag_driver.met_post_process.mps_to_kt, pass]
     units: [KT, DEG]
   TGRD:
     batch_order: 1
     callable: tc_diag_driver.met_diag_vars.temperature_gradient
-    kwargs: {u_name: u, v_name: v, bottom_hPa: 850, top_hPa: 700}
+    kwargs: {u_name: U, v_name: V, bottom_hPa: 850, top_hPa: 700}
     units: "10^7C/M"
 
 sounding_computation_specs:
@@ -161,4 +163,4 @@ comment_format: |
   *   850VORT, 200DVRG averaged from {div_vort_min}-{div_vort_max} km around storm center [x10^7 /s, x10^7 /s]   *
   *   850TANG averaged from {rad_tan_min}-{rad_tan_max} km around storm center [x10 m/s]   *
   *   T, R, Z, P averaged from {therm_min}-{therm_max} km around storm center [x10 C, %, dm, mb]   *
-  *   TPW averaged from {tpw_min}-{tpw_max} km around storm center [mm]   *
+  *   TPW averaged from {tpw_min}-{tpw_max} km around storm center [mm]   *
diff --git a/scripts/python/tc_diag/config/post_resample_nest.yml b/scripts/python/tc_diag/config/post_resample_nest.yml
@@ -100,23 +100,25 @@ pressure_independent_computation_specs:
     callable: *div_vort_func 
     output_vars: [850DVRG, 850VORT] 
     kwargs: {u_name: *u_vert_name, v_name: *v_vert_name, level_hPa: 850, radius_km: *div_vort_max_km}
+    units: [/S, /S]
   div_vort_200: 
     callable: *div_vort_func
     output_vars: [200DVRG, 200VORT] 
     kwargs: {u_name: *u_vert_name, v_name: *v_vert_name, level_hPa: 200, radius_km: *div_vort_max_km}
+    units: [/S, /S]
 
 
   shear: 
     batch_order: 1
     callable: tc_diag_driver.met_diag_vars.shear 
     output_vars: [SHR_MAG, SHR_HDG] 
-    kwargs: {u_name: u, v_name: v, bottom_hPa: 850, top_hPa: 200}
+    kwargs: {u_name: U, v_name: V, bottom_hPa: 850, top_hPa: 200}
     unit_converters: [tc_diag_driver.met_post_process.mps_to_kt, pass]
     units: [KT, DEG]
   TGRD:
     batch_order: 1
     callable: tc_diag_driver.met_diag_vars.temperature_gradient
-    kwargs: {u_name: u, v_name: v, bottom_hPa: 850, top_hPa: 700}
+    kwargs: {u_name: U, v_name: V, bottom_hPa: 850, top_hPa: 700}
     units: "10^7C/M"
 
 sounding_computation_specs:
@@ -161,4 +163,4 @@ comment_format: |
   *   850VORT, 200DVRG averaged from {div_vort_min}-{div_vort_max} km around storm center [x10^7 /s, x10^7 /s]   *
   *   850TANG averaged from {rad_tan_min}-{rad_tan_max} km around storm center [x10 m/s]   *
   *   T, R, Z, P averaged from {therm_min}-{therm_max} km around storm center [x10 C, %, dm, mb]   *
-  *   TPW averaged from {tpw_min}-{tpw_max} km around storm center [mm]   *
+  *   TPW averaged from {tpw_min}-{tpw_max} km around storm center [mm]   *
diff --git a/scripts/python/tc_diag/tc_diag_driver/__init__.py b/scripts/python/tc_diag/tc_diag_driver/__init__.py
@@ -1 +1 @@
-__version__ = "0.9.0"
+__version__ = "0.11.0"
diff --git a/scripts/python/tc_diag/tc_diag_driver/computation_engine.py b/scripts/python/tc_diag/tc_diag_driver/computation_engine.py
@@ -251,6 +251,21 @@ def get_result_names(computations: List[DiagComputation]) -> List[str]:
     return names
 
 
+def get_all_result_units(
+    pressure_indedpendent: List[DiagComputation], sounding: List[DiagComputation]
+) -> Tuple[List[str], List[str]]:
+    pi_var_units = get_result_units(pressure_indedpendent)
+    snd_var_units = get_result_units(sounding)
+    return pi_var_units, snd_var_units
+
+
+def get_result_units(computations: List[DiagComputation]) -> List[str]:
+    units = []
+    for c in computations:
+        units.extend(c.units)
+    return units
+
+
 def get_computation_batches(
     pressure_indedpendent: List[DiagComputation], sounding: List[DiagComputation]
 ) -> List[ComputationBatch]: