Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bugfix: Investigate unexpected number of derived HPBL observations in PB2NC #2687

Closed
9 of 23 tasks
jprestop opened this issue Sep 20, 2023 · 8 comments · Fixed by #2719
Closed
9 of 23 tasks

Bugfix: Investigate unexpected number of derived HPBL observations in PB2NC #2687

jprestop opened this issue Sep 20, 2023 · 8 comments · Fixed by #2719
Assignees
Labels
alert: NEED ACCOUNT KEY Need to assign an account key to this issue component: build process Build process issue priority: high High Priority requestor: METplus Team METplus Development Team type: bug Fix something that is not working
Milestone

Comments

@jprestop
Copy link
Collaborator

jprestop commented Sep 20, 2023

Describe the Problem

When committing ONLY a filename change directly in the develop branch, the GitHub Actions run produced a failure at the "Check for Differences" step. Asking about it on GitHub, @JohnHalleyGotway indicated that these derived PBL obs also recently caused diffs when we switched METbaseimage to Debian 12:
https://github.com/dtcenter/MET/actions/runs/6101796818/job/16560081693?pr=2675
The PBL derivation doesn't seem to be as repeatable as it should be. We need to look into the details of calpbl.f to debug and tighten up the logic.
@georgemccabe noted that this issue is the likely cause of the use case diffs as described in Enhancement: Improve PBL use case to generate reproducible results #2246.

Expected Behavior

The 9 missing derived HPBL obs should not be missing.

python MET-develop/scripts/python/utility/print_pointnc2ascii.py nam.20210311.t00z.prepbufr.tm00.pbl_TRUTH.nc > nam.20210311.t00z.prepbufr.tm00.pbl_TRUTH.txt

python MET-develop/scripts/python/utility/print_pointnc2ascii.py nam.20210311.t00z.prepbufr.tm00.pbl_OUTPUT.nc > nam.20210311.t00z.prepbufr.tm00.pbl_OUTPUT.txt

diff nam.20210311.t00z.prepbufr.tm00.pbl_TRUTH.txt nam.20210311.t00z.prepbufr.tm00.pbl_OUTPUT.txt

Here are the differences:

99d98
< ADPUPA 70398   20210311_000000 55.05 -131.5900   35 HPBL   NA 0 9  103.87126160
102d100
< ADPUPA 70326   20210311_000000 58.68 -156.6700    8 HPBL   NA 0 9  130.87126160
108d105
< ADPUPA 70273   20210311_000000 61.16 -149.9900   52 HPBL   NA 0 9   86.87126160
117d113
< ADPUPA 71957   20210311_000000 68.32 -133.5300  103 HPBL   NA 0 9   35.87126160
132d127
< ADPUPA 76903   20210311_000000 14.88  -92.3000  119 HPBL   NA 0 9   19.87126160
145d139
< ADPUPA 72240   20210311_000000 30.13  -93.2200    5 HPBL   NA 0 9  133.87126160
377d370
< ADPUPA 70261   20210311_000000 64.82 -147.8800  134 HPBL   NA 0 9    4.87126160
383d375
< ADPUPA 71917   20210311_000000 80.00  -85.9300   10 HPBL   NA 0 9  128.87126160
386d377
< ADPUPA 71082   20210311_000000 82.50  -62.3500   76 HPBL   NA 0 9   62.87126160

Environment

Describe your runtime environment:
GitHub Actions

To Reproduce

Describe the steps to reproduce the behavior:
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error
Post relevant sample data following these instructions:
https://dtcenter.org/community-code/model-evaluation-tools-met/met-help-desk#ftp

Relevant Deadlines

Beta2

Funding Source

2792543 - NOAA Base

Define the Metadata

Assignee

  • Select engineer(s) or no engineer required
  • Select scientist(s) or no scientist required

Labels

  • Review default alert labels
  • Select component(s)
  • Select priority
  • Select requestor(s)

Milestone and Projects

  • Select Milestone as the next bugfix version
  • Select Coordinated METplus-X.Y Support project for support of the current coordinated release
  • Select MET-X.Y.Z Development project for development toward the next official release

Define Related Issue(s)

Consider the impact to the other METplus components.

Bugfix Checklist

See the METplus Workflow for details.

  • Complete the issue definition above, including the Time Estimate and Funding Source.
  • Fork this repository or create a branch of main_<Version>.
    Branch name: bugfix_<Issue Number>_main_<Version>_<Description>
  • Fix the bug and test your changes.
  • Add/update log messages for easier debugging.
  • Add/update unit tests.
  • Add/update documentation.
  • Push local changes to GitHub.
  • Submit a pull request to merge into main_<Version>.
    Pull request: bugfix <Issue Number> main_<Version> <Description>
  • Define the pull request metadata, as permissions allow.
    Select: Reviewer(s) and Development issue
    Select: Milestone as the next bugfix version
    Select: Coordinated METplus-X.Y Support project for support of the current coordinated release
  • Iterate until the reviewer(s) accept and merge your changes.
  • Delete your fork or branch.
  • Complete the steps above to fix the bug on the develop branch.
    Branch name: bugfix_<Issue Number>_develop_<Description>
    Pull request: bugfix <Issue Number> develop <Description>
    Select: Reviewer(s) and Development issue
    Select: Milestone as the next official version
    Select: MET-X.Y.Z Development project for development toward the next official release
  • Close this issue.
@jprestop jprestop added type: bug Fix something that is not working component: build process Build process issue alert: NEED ACCOUNT KEY Need to assign an account key to this issue requestor: METplus Team METplus Development Team priority: high High Priority labels Sep 20, 2023
@jprestop jprestop added this to the MET 12.0.0 milestone Sep 20, 2023
@hsoh-u
Copy link
Collaborator

hsoh-u commented Sep 28, 2023

Where can I get the "nam.20210311.t00z.prepbufr.tm00.pbl_TRUTH.nc"?

@jprestop
Copy link
Collaborator Author

Great question @hsoh-u. I did a quick check on seneca and see nothing:

seneca:jpresto:/d1/projects/MET> locate nam.20210311.t00z.prepbufr.tm00.pbl_TRUTH.nc
seneca:jpresto:/d1/projects/MET>

@JohnHalleyGotway Where can @hsoh-u access this file?

@georgemccabe
Copy link
Collaborator

It should be in the diff artifact from the GHA run that flagged differences.

@hsoh-u
Copy link
Collaborator

hsoh-u commented Sep 29, 2023

The matching log for the first missing one ADPUPA 70398 20210311_000000 55.05 -131.5900 35 HPBL NA 0 9 103.87126160. The PBL value is different and filtered out.

  • 2 records for the TQZ (at 101700mb and 1000mb)
  • 36 records for UV (two records were discarded during merging: pressure = 930mb and 860mb)
  • Common pressure: 1 (pressure 1000mb)
  • 35 merged records (35 = 2 + 34 - 1)
DEBUG 8: compute_pbl() input to calpbl_ (buffer): index, P, Q, T, Z, U, V
DEBUG 8: compute_pbl()   0       -136673597379257383059938344960.00000 0.00000 -106153509484448810401246740480.00000 0.00000 13.90000 3.70000
DEBUG 8: compute_pbl()   1       1010.00006 0.00002 216.38368 30480.00000 6.10000 -2.80000
DEBUG 8: compute_pbl()   2       1110.00000 0.00008 217.65215 29870.00000 14.30000 1.30000
DEBUG 8: compute_pbl()   3       1220.00000 0.00013 218.92169 29261.00000 0.20000 2.60000
DEBUG 8: compute_pbl()   4       1280.00000 0.00016 219.56673 28956.00000 6.20000 0.00000
DEBUG 8: compute_pbl()   5       1350.00000 0.00019 220.28210 28639.00000 7.70000 0.70000
DEBUG 8: compute_pbl()   6       1629.99988 0.00029 222.81439 27432.00000 3.30000 3.90000
DEBUG 8: compute_pbl()   7       1710.00000 0.00032 223.45815 27127.00000 0.90000 5.00000
DEBUG 8: compute_pbl()   8       1979.99988 0.00040 225.42786 26213.00000 3.70000 8.00000
DEBUG 8: compute_pbl()   9       2080.00000 0.00043 226.08984 25908.00000 1.90000 6.90000
DEBUG 8: compute_pbl()   10      2190.00000 0.00046 226.78223 25603.00000 -0.90000 5.00000
DEBUG 8: compute_pbl()   11      2410.00000 0.00051 228.06837 24994.00000 2.90000 4.20000
DEBUG 8: compute_pbl()   12      2530.00000 0.00054 228.72124 24689.00000 0.00000 0.00000
DEBUG 8: compute_pbl()   13      2660.00000 0.00057 229.39445 24384.00000 0.60000 -1.40000
DEBUG 8: compute_pbl()   14      2790.00000 0.00060 230.03555 24079.00000 5.10000 0.40000
DEBUG 8: compute_pbl()   15      2920.00000 0.00062 230.64745 23774.00000 3.30000 7.00000
DEBUG 8: compute_pbl()   16      3070.00000 0.00065 231.32050 23470.00000 -2.60000 5.60000
DEBUG 8: compute_pbl()   17      3220.00000 0.00068 231.96143 23165.00000 -4.40000 -2.50000
DEBUG 8: compute_pbl()   18      3380.00000 0.00070 232.61298 22860.00000 -0.90000 -2.40000
DEBUG 8: compute_pbl()   19      3909.99976 0.00079 234.57004 21946.00000 3.60000 -3.60000
DEBUG 8: compute_pbl()   20      4310.00000 0.00084 235.87868 21336.00000 7.10000 -4.10000
DEBUG 8: compute_pbl()   21      4520.00000 0.00087 236.51787 21031.00000 5.50000 -4.60000
DEBUG 8: compute_pbl()   22      4980.00000 0.00092 237.82002 20422.00000 4.80000 1.70000
DEBUG 8: compute_pbl()   23      5230.00000 0.00095 238.47812 20117.00000 3.90000 -3.30000
DEBUG 8: compute_pbl()   24      5490.00000 0.00098 239.12997 19812.00000 6.20000 -0.50000
DEBUG 8: compute_pbl()   25      5760.00000 0.00100 239.77501 19507.00000 5.00000 0.90000
DEBUG 8: compute_pbl()   26      6650.00000 0.00108 241.70543 18593.00000 1.30000 -4.90000
DEBUG 8: compute_pbl()   27      6969.99951 0.00111 242.33688 18288.00000 2.90000 -4.20000
DEBUG 8: compute_pbl()   28      7319.99951 0.00114 242.99518 17983.00000 0.00000 -5.70000
DEBUG 8: compute_pbl()   29      7669.99951 0.00116 243.62271 17678.00000 5.70000 -9.80000
DEBUG 8: compute_pbl()   30      8050.00000 0.00119 244.27238 17374.00000 6.70000 -4.70000
DEBUG 8: compute_pbl()   31      8440.00000 0.00122 244.90804 17069.00000 6.20000 -3.60000
DEBUG 8: compute_pbl()   32      8850.00000 0.00125 245.54536 16764.00000 3.90000 -8.40000
DEBUG 8: compute_pbl()   33      98370.00000 0.00260 277.90271 305.00000 4.00000 0.70000
DEBUG 8: compute_pbl()   34      101700.00000 0.00262 278.35001 35.00000 -9999.00000 -9999.00000
DEBUG 4: Not saved because the computed PBL (3.078372466e-41) is less than the station elevation (35). 20210311_000000 ADPUPA 70398

The pressure of the first PBL input record is not correct (should 1000). It's corrected, but still filtered out

DEBUG 7: compute_pbl() pbl_level= 35 from TQ (2) and UV (36)
DEBUG 8:
DEBUG 8: compute_pbl() input to calpbl_ (buffer): index, P, Q, T, Z, U, V
DEBUG 8: compute_pbl()   0       1000.00000 0.00002 216.25000 31496.00000 13.90000 3.70000
DEBUG 8: compute_pbl()   1       1010.00006 0.00002 216.38368 30480.00000 6.10000 -2.80000
DEBUG 8: compute_pbl()   2       1110.00000 0.00008 217.65215 29870.00000 14.30000 1.30000
...
DEBUG 7:  hpbl: 31496, obs_arr[4]: 31461   lat: 55.04999924, lon: -131.5899963, elv: 35 20210311_000000 ADPUPA 70398
DEBUG 7:
DEBUG 4: Not saved the computed PBL (31461 from 31496) because of the MAX PBL 10000  (20210311_000000 ADPUPA 70398)

@hsoh-u
Copy link
Collaborator

hsoh-u commented Sep 29, 2023

9 PBL values were filtered out with the nightly build:

/d1/projects/MET/MET_regression/develop/NB20230929/MET-develop/bin/pb2nc /d1/personal/hsoh/data/MET-1715/nam.20210311.t00z.prepbufr.tm00 nam.20210311.t00z.prepbufr.tm00.pbl.nc /d1/personal/hsoh/data/MET-1715/PB2NCConfig_case_6_10 -v 4 | grep " Not saved because the computed PBL"  | egrep  "70398|70326|70273|71957|76903|72240|70261|71917|71082"
DEBUG 4: Not saved because the computed PBL (3.07627e-41) is less than the station elevation (35). 20210311_000000 ADPUPA 70398
DEBUG 4: Not saved because the computed PBL (3.07627e-41) is less than the station elevation (8). 20210311_000000 ADPUPA 70326
DEBUG 4: Not saved because the computed PBL (3.07627e-41) is less than the station elevation (52). 20210311_000000 ADPUPA 70273
DEBUG 4: Not saved because the computed PBL (3.07627e-41) is less than the station elevation (103). 20210311_000000 ADPUPA 71957
DEBUG 4: Not saved because the computed PBL (3.07627e-41) is less than the station elevation (119). 20210311_000000 ADPUPA 76903
DEBUG 4: Not saved because the computed PBL (3.07627e-41) is less than the station elevation (5). 20210311_000000 ADPUPA 72240
DEBUG 4: Not saved because the computed PBL (3.07627e-41) is less than the station elevation (134). 20210311_000000 ADPUPA 70261
DEBUG 4: Not saved because the computed PBL (3.07627e-41) is less than the station elevation (10). 20210311_000000 ADPUPA 71917
DEBUG 4: Not saved because the computed PBL (3.07627e-41) is less than the station elevation (76). 20210311_000000 ADPUPA 71082

@JohnHalleyGotway
Copy link
Collaborator

JohnHalleyGotway commented Oct 10, 2023

@hsoh-u and @JohnHalleyGotway met on Oct 10, 2023 and decided to rerun the full set of unit tests for develop-ref with this GHA run. Check the number of obs in nam.20210311.t00z.prepbufr.tm00.pbl.nc. If it's still 401, then there is more debugging to be done. If it's 392 (which we've been getting since), then let's just update develop-ref to move past this difference.

I note that we did run the following commands:

docker run -it --rm -v /Volumes/d1/projects/MET/MET_unit_test/MET_test_input:/data dtcenter/met-dev:develop-ref /bin/bash
pb2nc /data/obs_data/prepbufr/nam.20210311.t00z.prepbufr.tm00 nam.20210311.t00z.prepbufr.tm00.pbl.test.nc MET-develop-ref/internal/test_unit/config/PB2NCConfig_pbl -log run_pb2nc.log -v 8

And the PB2NC output file there contained 392 (not 401).

UPDATE That GHA run produced 392 obs, not 401. So this issue cannot reasonably be replicated:

COMPARING pb2nc_indy/nam.20210311.t00z.prepbufr.tm00.pbl.nc
file1: /data/output/met_test_truth/pb2nc_indy/nam.20210311.t00z.prepbufr.tm00.pbl.nc
file2: /data/output/met_test_output/pb2nc_indy/nam.20210311.t00z.prepbufr.tm00.pbl.nc
ERROR: NetCDF headers differ:
7c7
< 	nobs = 401 ;
---
> 	nobs = 392 ;

Recommend just updating develop-ref to move past this difference, subject to any other changes that Howard would like to make.

@hsoh-u
Copy link
Collaborator

hsoh-u commented Oct 10, 2023

The github action against develop-ref has the same difference.

 /usr/bin/git -c protocol.version=2 fetch --no-tags --prune --progress --no-recurse-submodules --depth=1 origin +80afcac6e3906ac84b38ee678df640968c47b059:refs/remotes/origin/develop-ref

COMPARING pb2nc_indy/nam.20210311.t00z.prepbufr.tm00.pbl.nc
file1: /data/output/met_test_truth/pb2nc_indy/nam.20210311.t00z.prepbufr.tm00.pbl.nc
file2: /data/output/met_test_output/pb2nc_indy/nam.20210311.t00z.prepbufr.tm00.pbl.nc
ERROR: NetCDF headers differ:
7c7
< 	nobs = 401 ;
---
> 	nobs = 392 ;

@hsoh-u
Copy link
Collaborator

hsoh-u commented Oct 10, 2023

The PBL input for Fortran code is initialized here

hsoh-u pushed a commit that referenced this issue Oct 16, 2023
@JohnHalleyGotway JohnHalleyGotway changed the title Bugfix: Resolve missing derived HPBL obs Bugfix: Investigate missing derived HPBL obs Oct 17, 2023
@hsoh-u hsoh-u linked a pull request Oct 19, 2023 that will close this issue
15 tasks
@hsoh-u hsoh-u closed this as completed Oct 23, 2023
@JohnHalleyGotway JohnHalleyGotway changed the title Bugfix: Investigate missing derived HPBL obs Bugfix: Investigate unexpected number of derived HPBL observations in PB2NC Nov 17, 2023
JohnHalleyGotway added a commit that referenced this issue Jun 25, 2024
* Add user execute permissions to compile script - main_v11.1 (#2740)

* Changing -j to "-j 5" as the recommended value for MAKE_ARGS

* Per #2761, update the MET development environment after upgrading seneca to debian bookworm.

* Per #2761, define runtime python version for testing  rather than using the default version which no longer exists in /usr/local

* Per #2761, fix  setting ci-skip-all

* Per #2761, patching test_util.R to use the -C command line option for ncdiff.

* #2652 Added find_var_by_standard_name and separated common codes to find_xy_vars

* #2757 The SonarQube token and URL are replaced with the pre-defined strings, SONAR_TOKEN_VALUE and SONAR_SERVER_URL

* #2757 Get the email list from the environment variable MET_CRON_EMAIL_LIST__MET (or MET_CRON_EMAIL_LIST)

* #2757 The SonarQube token and URL are replaced by using the environment variable SONAR_TOKEN_VALUE and SONAR_SERVER_URL

* Bugfix #2760 main_v11.1 --enable-python (#2767)

* #2755 Added a header count and checking header count instead of using header id (hid)

* Bugfix #2782 main_v11.1 MASSDEN (#2784)

* Per #2782, update the multiple matching records warning message to include the table number for each record.

* Per #2782, update read_grib2_record_list() to parse the level values and aerosol information correctly for table 4.48.

* Per #2782 tweak variable naming convention.

* Removing ${MAKE_ARGS} in some locations

Removing ${MAKE_ARGS} from "make install" and "make test" for MET.  Removing "met" prefix from met.configure.log because we really need config.log for any detail.  It is confusing to have met.configure.log when that does not contain useful information.

* Adding -lnetcdf -lm to configure_lib_args for NetCDF-CXX

* Feature #2796 main_v11.1 gha node20 (#2798)

* Per #2796, update versions of actions to fix the node 16 to 20 warning message.

* Per #2796, port fixes for artifact name handling over from the develop branch to the main_v11.1 testing workflow. Also add the compilation_options.yml workflow since the workflows are being updated.

* Create 11.1.0_casper

* Recent changes to branch protection rules for the main_vX.Y branches have broken the logic of the update_truth.yml GHA workflow. Instead of submitting a PR to merge main_vX.Y into main_vX.Y-ref directly, use an intermediate update_truth_for_main_vX.Y branch.

* Update the pull request template to include a question about expected impacts to existing METplus Use Cases.

* Bugfix #2833 main_v11.1 azimuth (#2834)

* Per #2833, fix n-1 bug when defining the azimuth delta for range/azimuth grids.

* Per #2833, port fixes over from the bugfix_2833_develop_azimuth branch over to the main_v11.1 branch.

---------

Co-authored-by: MET Tools Test Account <met_test@seneca.rap.ucar.edu>

* Feature #2379 main_v11.1 sonarqube GHA (#2848)

* Per #2379, migrating largely the same changes for #2379 into the main_v11.1 branch. The difference is that --enable-all is not used since that is not a valid configuration option for MET version 11.1.0.

* Hotfix related to #2379. The sonar.newCode.referenceBranch and sonar.branch.name cannot be set to the same string! Only add the newCode definition when they differ.

* Feature #2379 main_v11.1 sonarqube updates (#2851)

* Feature #2379 main_v11.1 single_sq_project (#2866)

* Bugfix 2867 point2grid qc flag main v11.1 (#2874)

* #2867 Added point2grid_GOES_16_ADP_Enterprise_high and changed qc flags for ADP

* #2867 Added get_nc_att_values_

* #2867 Added get_nc_att_values

* #2867 Get the ADP QC flag values from the varibale attributes (support GOES16 Enterprise allgorithm) and apply them for QC Flags. Adjusted the ADP QC flags based on the variable QC values

---------

Co-authored-by: Howard Soh <hsoh@seneca.rap.ucar.edu>

* Bugfix 2867 point2grid qc flag main v11.1 (#2878)

* #2867 Added point2grid_GOES_16_ADP_Enterprise_high and changed qc flags for ADP

* #2867 Added get_nc_att_values_

* #2867 Added get_nc_att_values

* #2867 Get the ADP QC flag values from the varibale attributes (support GOES16 Enterprise allgorithm) and apply them for QC Flags. Adjusted the ADP QC flags based on the variable QC values

* #2867 Added adjusted confidnece counts

* #2867 Corretced indent

* #2867 Corretced indent

---------

Co-authored-by: Howard Soh <hsoh@seneca.rap.ucar.edu>

* #2884 No filtering by QC flags without -qc option (#2885)

Co-authored-by: Howard Soh <hsoh@seneca.rap.ucar.edu>

* Per #2659, making updates as proposed at the 20240516 MET Eng. Mtg. (#2896)

* Bugfix #2897 main_v11.1 python_valid_time (#2898)

* Per #2897, fix typos in 3 log messages. Also fix the bug in storing the valid time strings. The time string in vld_array should exactly correspond to the numeric unixtime values in vld_num_array. Therefore they need to be updated inside the same if block. The bug is that we were storing only the unique unixtime values but storing ALL of the valid time string, not just the unique ones.

* Per #2897, don’t waste time searching, just set vld_odd to n-1

* #2687 Saved the PBL input into the vector (#2903)

* #2904 Changed R path to R-4.4.0 (#2909)

Co-authored-by: Howard Soh <hsoh@seneca.rap.ucar.edu>

* Bugfix #2856 main_v11.1 ens_climo (#2917)

* Per #2856, reinitialize the climo cdf info pointer.

* Per #2856, update the testing.yml workflow dispatch option to let the comparison branch be manually defined.

* Revert "Per #2856, reinitialize the climo cdf info pointer."

This reverts commit 99b6bb3.

* Per #2856, reinitialize the climo cdf info pointer.

* Revert "Per #2856, update the testing.yml workflow dispatch option to let the comparison branch be manually defined."

This reverts commit 0ae224f.

* Per #2856, update the truth_data_version and input_data_version if main_vX.Y is not present in the branch name.

* Update set_job_controls.sh

* Update set_job_controls.sh

* Per #2856, update logic for counting the maximum number of PCT, PRC, PJC, and PSTD lines. Need to multiply by the number of climo cdf bins.

* Per #2856, add more error checking to avoid writing nan to .stat output files

* Bugfix #2841 main_v11.1 tang_rad_winds (#2920)

* Per #2841, unrelated, just removing a spurious character from a  log message.

* Per #2841, work in progress. Mostly just modifying whitespace and log messages so far

* Per #2841, running tc_rmw with only UGRD input causes the tool to hang. Adding break statements to prevent the hang.

* Per #2841, add new has_pressure_level() utility function and update tc_rmw to use it to deteremine whether or not the pressure dimension should be written the output rather than basing that off the number of levels being > 1.

* Per #1849, fix to radial and tangential winds.

* Per #2841, just changing a code comment about the longitude swap

* Per #2841, update variable names for consistency and understanding.

* Per #2841, the substantive bug fixed here is how we index into the U and V data in wind_ne_to_rt() using i_rev instead of i. Also, update the variable names for consistency and clarity..

* Per #2841, patch apparent copy/paste bug in tcrmw_grid.cc code.

* Per #2841, only changing whitespace.

* Per #2841, update logic in EarthRotation::set_tcrmw() to change the rotation of TCRMW grids from pointing north to pointing east.

* Per #2841, fix warning message

* Replace tab with spaces

* Per #2841, correct the units for the azimuth netcdf output variable

* Per #2841, reverse the x dimension of the rotated latlon grid to effectively switch from counterclockwise rotation to clockwise.

* Feature #2855 v11.1.1 (#2923)

* Per #2855, add v11.1.1 release notes.

* Per #2841, fix bold formatting of release notes.

* Update conf.py with actual 11.1.1 release date

---------

Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com>
Co-authored-by: Julie Prestopnik <jpresto@ucar.edu>
Co-authored-by: John Halley Gotway <johnhg@ucar.edu>
Co-authored-by: MET Tools Test Account <met_test@seneca.rap.ucar.edu>
Co-authored-by: Howard Soh <hsoh@seneca.rap.ucar.edu>
Co-authored-by: Howard Soh <hsoh@ucar.edu>
Co-authored-by: metplus-bot <97135045+metplus-bot@users.noreply.github.com>
JohnHalleyGotway added a commit that referenced this issue Sep 5, 2024
* Add user execute permissions to compile script - main_v11.1 (#2740)

* Changing -j to "-j 5" as the recommended value for MAKE_ARGS

* Per #2761, update the MET development environment after upgrading seneca to debian bookworm.

* Per #2761, define runtime python version for testing  rather than using the default version which no longer exists in /usr/local

* Per #2761, fix  setting ci-skip-all

* Per #2761, patching test_util.R to use the -C command line option for ncdiff.

* #2652 Added find_var_by_standard_name and separated common codes to find_xy_vars

* #2757 The SonarQube token and URL are replaced with the pre-defined strings, SONAR_TOKEN_VALUE and SONAR_SERVER_URL

* #2757 Get the email list from the environment variable MET_CRON_EMAIL_LIST__MET (or MET_CRON_EMAIL_LIST)

* #2757 The SonarQube token and URL are replaced by using the environment variable SONAR_TOKEN_VALUE and SONAR_SERVER_URL

* Bugfix #2760 main_v11.1 --enable-python (#2767)

* #2755 Added a header count and checking header count instead of using header id (hid)

* Bugfix #2782 main_v11.1 MASSDEN (#2784)

* Per #2782, update the multiple matching records warning message to include the table number for each record.

* Per #2782, update read_grib2_record_list() to parse the level values and aerosol information correctly for table 4.48.

* Per #2782 tweak variable naming convention.

* Removing ${MAKE_ARGS} in some locations

Removing ${MAKE_ARGS} from "make install" and "make test" for MET.  Removing "met" prefix from met.configure.log because we really need config.log for any detail.  It is confusing to have met.configure.log when that does not contain useful information.

* Adding -lnetcdf -lm to configure_lib_args for NetCDF-CXX

* Feature #2796 main_v11.1 gha node20 (#2798)

* Per #2796, update versions of actions to fix the node 16 to 20 warning message.

* Per #2796, port fixes for artifact name handling over from the develop branch to the main_v11.1 testing workflow. Also add the compilation_options.yml workflow since the workflows are being updated.

* Create 11.1.0_casper

* Recent changes to branch protection rules for the main_vX.Y branches have broken the logic of the update_truth.yml GHA workflow. Instead of submitting a PR to merge main_vX.Y into main_vX.Y-ref directly, use an intermediate update_truth_for_main_vX.Y branch.

* Update the pull request template to include a question about expected impacts to existing METplus Use Cases.

* Bugfix #2833 main_v11.1 azimuth (#2834)

* Per #2833, fix n-1 bug when defining the azimuth delta for range/azimuth grids.

* Per #2833, port fixes over from the bugfix_2833_develop_azimuth branch over to the main_v11.1 branch.

---------

Co-authored-by: MET Tools Test Account <met_test@seneca.rap.ucar.edu>

* Feature #2379 main_v11.1 sonarqube GHA (#2848)

* Per #2379, migrating largely the same changes for #2379 into the main_v11.1 branch. The difference is that --enable-all is not used since that is not a valid configuration option for MET version 11.1.0.

* Hotfix related to #2379. The sonar.newCode.referenceBranch and sonar.branch.name cannot be set to the same string! Only add the newCode definition when they differ.

* Feature #2379 main_v11.1 sonarqube updates (#2851)

* Feature #2379 main_v11.1 single_sq_project (#2866)

* Bugfix 2867 point2grid qc flag main v11.1 (#2874)

* #2867 Added point2grid_GOES_16_ADP_Enterprise_high and changed qc flags for ADP

* #2867 Added get_nc_att_values_

* #2867 Added get_nc_att_values

* #2867 Get the ADP QC flag values from the varibale attributes (support GOES16 Enterprise allgorithm) and apply them for QC Flags. Adjusted the ADP QC flags based on the variable QC values

---------

Co-authored-by: Howard Soh <hsoh@seneca.rap.ucar.edu>

* Bugfix 2867 point2grid qc flag main v11.1 (#2878)

* #2867 Added point2grid_GOES_16_ADP_Enterprise_high and changed qc flags for ADP

* #2867 Added get_nc_att_values_

* #2867 Added get_nc_att_values

* #2867 Get the ADP QC flag values from the varibale attributes (support GOES16 Enterprise allgorithm) and apply them for QC Flags. Adjusted the ADP QC flags based on the variable QC values

* #2867 Added adjusted confidnece counts

* #2867 Corretced indent

* #2867 Corretced indent

---------

Co-authored-by: Howard Soh <hsoh@seneca.rap.ucar.edu>

* #2884 No filtering by QC flags without -qc option (#2885)

Co-authored-by: Howard Soh <hsoh@seneca.rap.ucar.edu>

* Per #2659, making updates as proposed at the 20240516 MET Eng. Mtg. (#2896)

* Bugfix #2897 main_v11.1 python_valid_time (#2898)

* Per #2897, fix typos in 3 log messages. Also fix the bug in storing the valid time strings. The time string in vld_array should exactly correspond to the numeric unixtime values in vld_num_array. Therefore they need to be updated inside the same if block. The bug is that we were storing only the unique unixtime values but storing ALL of the valid time string, not just the unique ones.

* Per #2897, don’t waste time searching, just set vld_odd to n-1

* #2687 Saved the PBL input into the vector (#2903)

* #2904 Changed R path to R-4.4.0 (#2909)

Co-authored-by: Howard Soh <hsoh@seneca.rap.ucar.edu>

* Bugfix #2856 main_v11.1 ens_climo (#2917)

* Per #2856, reinitialize the climo cdf info pointer.

* Per #2856, update the testing.yml workflow dispatch option to let the comparison branch be manually defined.

* Revert "Per #2856, reinitialize the climo cdf info pointer."

This reverts commit 99b6bb3.

* Per #2856, reinitialize the climo cdf info pointer.

* Revert "Per #2856, update the testing.yml workflow dispatch option to let the comparison branch be manually defined."

This reverts commit 0ae224f.

* Per #2856, update the truth_data_version and input_data_version if main_vX.Y is not present in the branch name.

* Update set_job_controls.sh

* Update set_job_controls.sh

* Per #2856, update logic for counting the maximum number of PCT, PRC, PJC, and PSTD lines. Need to multiply by the number of climo cdf bins.

* Per #2856, add more error checking to avoid writing nan to .stat output files

* Bugfix #2841 main_v11.1 tang_rad_winds (#2920)

* Per #2841, unrelated, just removing a spurious character from a  log message.

* Per #2841, work in progress. Mostly just modifying whitespace and log messages so far

* Per #2841, running tc_rmw with only UGRD input causes the tool to hang. Adding break statements to prevent the hang.

* Per #2841, add new has_pressure_level() utility function and update tc_rmw to use it to deteremine whether or not the pressure dimension should be written the output rather than basing that off the number of levels being > 1.

* Per #1849, fix to radial and tangential winds.

* Per #2841, just changing a code comment about the longitude swap

* Per #2841, update variable names for consistency and understanding.

* Per #2841, the substantive bug fixed here is how we index into the U and V data in wind_ne_to_rt() using i_rev instead of i. Also, update the variable names for consistency and clarity..

* Per #2841, patch apparent copy/paste bug in tcrmw_grid.cc code.

* Per #2841, only changing whitespace.

* Per #2841, update logic in EarthRotation::set_tcrmw() to change the rotation of TCRMW grids from pointing north to pointing east.

* Per #2841, fix warning message

* Replace tab with spaces

* Per #2841, correct the units for the azimuth netcdf output variable

* Per #2841, reverse the x dimension of the rotated latlon grid to effectively switch from counterclockwise rotation to clockwise.

* Feature #2855 v11.1.1 (#2923)

* Per #2855, add v11.1.1 release notes.

* Per #2841, fix bold formatting of release notes.

* Update conf.py with actual 11.1.1 release date

* bugfix #2936 point2grid for tripolar GFS input (#2945)

* #2936 Support 1D lat/lon values

* #2936 Change obs_type to TYPE_NCCF if FileType_NcCF is given from the config file

---------

Co-authored-by: Howard Soh <hsoh@seneca.rap.ucar.edu>

* Bugfix #2958 main_v11.1 BAGSS SEDI CI (#2960)

---------

Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com>
Co-authored-by: Julie Prestopnik <jpresto@ucar.edu>
Co-authored-by: John Halley Gotway <johnhg@ucar.edu>
Co-authored-by: MET Tools Test Account <met_test@seneca.rap.ucar.edu>
Co-authored-by: Howard Soh <hsoh@seneca.rap.ucar.edu>
Co-authored-by: Howard Soh <hsoh@ucar.edu>
Co-authored-by: metplus-bot <97135045+metplus-bot@users.noreply.github.com>
JohnHalleyGotway added a commit that referenced this issue Sep 18, 2024
* Add user execute permissions to compile script - main_v11.1 (#2740)

* Changing -j to "-j 5" as the recommended value for MAKE_ARGS

* Per #2761, update the MET development environment after upgrading seneca to debian bookworm.

* Per #2761, define runtime python version for testing  rather than using the default version which no longer exists in /usr/local

* Per #2761, fix  setting ci-skip-all

* Per #2761, patching test_util.R to use the -C command line option for ncdiff.

* #2652 Added find_var_by_standard_name and separated common codes to find_xy_vars

* #2757 The SonarQube token and URL are replaced with the pre-defined strings, SONAR_TOKEN_VALUE and SONAR_SERVER_URL

* #2757 Get the email list from the environment variable MET_CRON_EMAIL_LIST__MET (or MET_CRON_EMAIL_LIST)

* #2757 The SonarQube token and URL are replaced by using the environment variable SONAR_TOKEN_VALUE and SONAR_SERVER_URL

* Bugfix #2760 main_v11.1 --enable-python (#2767)

* #2755 Added a header count and checking header count instead of using header id (hid)

* Bugfix #2782 main_v11.1 MASSDEN (#2784)

* Per #2782, update the multiple matching records warning message to include the table number for each record.

* Per #2782, update read_grib2_record_list() to parse the level values and aerosol information correctly for table 4.48.

* Per #2782 tweak variable naming convention.

* Removing ${MAKE_ARGS} in some locations

Removing ${MAKE_ARGS} from "make install" and "make test" for MET.  Removing "met" prefix from met.configure.log because we really need config.log for any detail.  It is confusing to have met.configure.log when that does not contain useful information.

* Adding -lnetcdf -lm to configure_lib_args for NetCDF-CXX

* Feature #2796 main_v11.1 gha node20 (#2798)

* Per #2796, update versions of actions to fix the node 16 to 20 warning message.

* Per #2796, port fixes for artifact name handling over from the develop branch to the main_v11.1 testing workflow. Also add the compilation_options.yml workflow since the workflows are being updated.

* Create 11.1.0_casper

* Recent changes to branch protection rules for the main_vX.Y branches have broken the logic of the update_truth.yml GHA workflow. Instead of submitting a PR to merge main_vX.Y into main_vX.Y-ref directly, use an intermediate update_truth_for_main_vX.Y branch.

* Update the pull request template to include a question about expected impacts to existing METplus Use Cases.

* Bugfix #2833 main_v11.1 azimuth (#2834)

* Per #2833, fix n-1 bug when defining the azimuth delta for range/azimuth grids.

* Per #2833, port fixes over from the bugfix_2833_develop_azimuth branch over to the main_v11.1 branch.

---------

Co-authored-by: MET Tools Test Account <met_test@seneca.rap.ucar.edu>

* Feature #2379 main_v11.1 sonarqube GHA (#2848)

* Per #2379, migrating largely the same changes for #2379 into the main_v11.1 branch. The difference is that --enable-all is not used since that is not a valid configuration option for MET version 11.1.0.

* Hotfix related to #2379. The sonar.newCode.referenceBranch and sonar.branch.name cannot be set to the same string! Only add the newCode definition when they differ.

* Feature #2379 main_v11.1 sonarqube updates (#2851)

* Feature #2379 main_v11.1 single_sq_project (#2866)

* Bugfix 2867 point2grid qc flag main v11.1 (#2874)

* #2867 Added point2grid_GOES_16_ADP_Enterprise_high and changed qc flags for ADP

* #2867 Added get_nc_att_values_

* #2867 Added get_nc_att_values

* #2867 Get the ADP QC flag values from the varibale attributes (support GOES16 Enterprise allgorithm) and apply them for QC Flags. Adjusted the ADP QC flags based on the variable QC values

---------

Co-authored-by: Howard Soh <hsoh@seneca.rap.ucar.edu>

* Bugfix 2867 point2grid qc flag main v11.1 (#2878)

* #2867 Added point2grid_GOES_16_ADP_Enterprise_high and changed qc flags for ADP

* #2867 Added get_nc_att_values_

* #2867 Added get_nc_att_values

* #2867 Get the ADP QC flag values from the varibale attributes (support GOES16 Enterprise allgorithm) and apply them for QC Flags. Adjusted the ADP QC flags based on the variable QC values

* #2867 Added adjusted confidnece counts

* #2867 Corretced indent

* #2867 Corretced indent

---------

Co-authored-by: Howard Soh <hsoh@seneca.rap.ucar.edu>

* #2884 No filtering by QC flags without -qc option (#2885)

Co-authored-by: Howard Soh <hsoh@seneca.rap.ucar.edu>

* Per #2659, making updates as proposed at the 20240516 MET Eng. Mtg. (#2896)

* Bugfix #2897 main_v11.1 python_valid_time (#2898)

* Per #2897, fix typos in 3 log messages. Also fix the bug in storing the valid time strings. The time string in vld_array should exactly correspond to the numeric unixtime values in vld_num_array. Therefore they need to be updated inside the same if block. The bug is that we were storing only the unique unixtime values but storing ALL of the valid time string, not just the unique ones.

* Per #2897, don’t waste time searching, just set vld_odd to n-1

* #2687 Saved the PBL input into the vector (#2903)

* #2904 Changed R path to R-4.4.0 (#2909)

Co-authored-by: Howard Soh <hsoh@seneca.rap.ucar.edu>

* Bugfix #2856 main_v11.1 ens_climo (#2917)

* Per #2856, reinitialize the climo cdf info pointer.

* Per #2856, update the testing.yml workflow dispatch option to let the comparison branch be manually defined.

* Revert "Per #2856, reinitialize the climo cdf info pointer."

This reverts commit 99b6bb3.

* Per #2856, reinitialize the climo cdf info pointer.

* Revert "Per #2856, update the testing.yml workflow dispatch option to let the comparison branch be manually defined."

This reverts commit 0ae224f.

* Per #2856, update the truth_data_version and input_data_version if main_vX.Y is not present in the branch name.

* Update set_job_controls.sh

* Update set_job_controls.sh

* Per #2856, update logic for counting the maximum number of PCT, PRC, PJC, and PSTD lines. Need to multiply by the number of climo cdf bins.

* Per #2856, add more error checking to avoid writing nan to .stat output files

* Bugfix #2841 main_v11.1 tang_rad_winds (#2920)

* Per #2841, unrelated, just removing a spurious character from a  log message.

* Per #2841, work in progress. Mostly just modifying whitespace and log messages so far

* Per #2841, running tc_rmw with only UGRD input causes the tool to hang. Adding break statements to prevent the hang.

* Per #2841, add new has_pressure_level() utility function and update tc_rmw to use it to deteremine whether or not the pressure dimension should be written the output rather than basing that off the number of levels being > 1.

* Per #1849, fix to radial and tangential winds.

* Per #2841, just changing a code comment about the longitude swap

* Per #2841, update variable names for consistency and understanding.

* Per #2841, the substantive bug fixed here is how we index into the U and V data in wind_ne_to_rt() using i_rev instead of i. Also, update the variable names for consistency and clarity..

* Per #2841, patch apparent copy/paste bug in tcrmw_grid.cc code.

* Per #2841, only changing whitespace.

* Per #2841, update logic in EarthRotation::set_tcrmw() to change the rotation of TCRMW grids from pointing north to pointing east.

* Per #2841, fix warning message

* Replace tab with spaces

* Per #2841, correct the units for the azimuth netcdf output variable

* Per #2841, reverse the x dimension of the rotated latlon grid to effectively switch from counterclockwise rotation to clockwise.

* Feature #2855 v11.1.1 (#2923)

* Per #2855, add v11.1.1 release notes.

* Per #2841, fix bold formatting of release notes.

* Update conf.py with actual 11.1.1 release date

* bugfix #2936 point2grid for tripolar GFS input (#2945)

* #2936 Support 1D lat/lon values

* #2936 Change obs_type to TYPE_NCCF if FileType_NcCF is given from the config file

---------

Co-authored-by: Howard Soh <hsoh@seneca.rap.ucar.edu>

* Bugfix #2958 main_v11.1 BAGSS SEDI CI (#2960)

* Bugfix #2958 main_v11.1 ORSS (#2972)

Co-authored-by: MET Tools Test Account <met_test@seneca.rap.ucar.edu>

---------

Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com>
Co-authored-by: Julie Prestopnik <jpresto@ucar.edu>
Co-authored-by: John Halley Gotway <johnhg@ucar.edu>
Co-authored-by: MET Tools Test Account <met_test@seneca.rap.ucar.edu>
Co-authored-by: Howard Soh <hsoh@seneca.rap.ucar.edu>
Co-authored-by: Howard Soh <hsoh@ucar.edu>
Co-authored-by: metplus-bot <97135045+metplus-bot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
alert: NEED ACCOUNT KEY Need to assign an account key to this issue component: build process Build process issue priority: high High Priority requestor: METplus Team METplus Development Team type: bug Fix something that is not working
Projects
Status: 🏁 Done
Development

Successfully merging a pull request may close this issue.

4 participants