Update develop-ref after #3057 #3060
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
* Per #2395, add new columns to VL1L2, VAL1L2, and VCNT line types for wind direction statistics. Work still in progress. * Per #2395, write the new VCNT columns to the output and document the additions to the VL1L2, VAL1L2, and VCNT columns. * Per #2395, add the definition of new statistics to Appendix G. * Per #2395, update file version history. * Per #2395, tweak warning message about zero wind vectors and update grid-stat and point-stat to log calls to the do_vl1l2() function. * Per #2395, refine the weights for wind direction stats, ignoring the undefined directions. * Update src/tools/core/stat_analysis/aggr_stat_line.cc * Update src/tools/core/stat_analysis/parse_stat_line.cc * Update src/tools/core/stat_analysis/aggr_stat_line.cc
… broken the logic of the update_truth.yml GHA workflow. Instead of submitting a PR to merge develop into develop-ref directly, use an intermediate update_truth_for_develop branch.
* Per #2280, update to support probability threshold strings like ==8, where 8 is the number of ensemble members, to create probability bins centered on the n/8 for n = 0 ... 8. * Per #2280, update docs about probability threshold settings. * Per #2280, use a loose tolerance when checking for consistent bin widths. * Per #2280, add a new unit test for grid_stat to demonstrate processing the output from gen_ens_prod. * Per #2280, when verifying NMEP probability forecasts, smooth the obs data first. * Per #2280, only request STAT output for the PCT line type to match unit_grid_stat.xml and minimize the new output files. * Per #2280, update config option docs. * Per #2280, update config option docs.
…qube_beta3_namespace
…ullptr Feature 2673 sonarqube beta4 nullptr
…eturn Feature 2673 sonarqube beta4 return
…qube_beta3_namespace
…idStatConfig_SEEPS config file needs to be updated with nc_pairs_flag.seeps = TRUE in order for the same output to be produced by the unit tests.
* Per #2966, add new solar_time() function to the vx_solar library. * Per #2966, add support for new solar_time masking type. Also make log messages for consistent and eliminate the warning about -thresh not being specified becuase its fine to not specify a threshold. * Per #2966, add a units attribute to the output NetCDF mask variable. * Per #2966, modify solar azimuth and altitude strings to make the log messages align well. * Per #2966, add gen_vx_mask unit test to demonstrate the solar_time masking type. * Per #2966, add documentation about the -solar_time option * Per #2966, reduce SonarQube code smells in gen_vx_mask * Per #2966, reduce SonarQube findings * Per #2966, support multiple mask types with the same mask field being supported in a single run. Still need to update the user's guide. * Per #2966, add UTC * Per #2966, update gen_vx_mask docs about supporting multiple -type options in a single run * Per #2966, update logic to fix using data masking twice, add a unit test to demonstrate, and update the mask_type attribute to include the magic string for the gridded data used for data masking. * Per #2966, adjust the logic slightly to revert to existing behavior where we only write the timing information of the input data to the gen_vx_mask output when no threshold was applied. This should reduce the number of diffs flagged by PR #3008 * Per #2966, update details about the -union, -intersection, and -symdiff options in the usage statement and documentation as recommended by @CPKalb. --------- Co-authored-by: MET Tools Test Account <met_test@seneca.rap.ucar.edu>
…ata/table_files/Makefile.in
* #3012 Initial release * #3012 Added search_3d_kdtree.cc * #3012 Initial release * #3012 Added check_llh_to_ecef and test_llh_to_ecef * #3012Added search_3d_kdtree_api * #3012 Reanmed point_lonlat to points_lonlat. Added points_XYZ & points_XYZ_km * #3012 Compare points_XYZ or points_latlon * #3012 Commented out latlonalt_to_xy & xy_to_latlonalt * #3012 Reanmed point_lonlat to points_lonlat. Added points_XYZ & points_XYZ_km. Added llh_to_ecef APIs * Per #3012, update the .gitignore settings to ignore the newly created test executables in the test_util/libcode/vx_grid directory. * Per #3012, changes to the vx_grid library to address SonarQube code smells in New Code and hopefully reduce the Overall number of code smells. * Per #3012, couple more minor SonarQube tweaks --------- Co-authored-by: Howard Soh <hsoh@seneca.rap.ucar.edu> Co-authored-by: MET Tools Test Account <met_test@seneca.rap.ucar.edu>
* Source code changes for multiple radii/thresholds in multivariate mode * Slight cleanup * added multiple thresh/radii functionality for the mvmode superobjects only situation * mvmode log message improvements * Mods to the mode users guide for multiple convolution thresholds and radii in multivariate mode * Added v12.1 * Minor bugfix and tweak * Per #2709, updates to fix SonarQube code smells. * Per #2709, fix compilation problem I introduced with SonarQube fixes. * Per #2709, eliminate SonarQube bug that was flagged * Per #2709, fix typo in docs --------- Co-authored-by: MET Tools Test Account <met_test@seneca.rap.ucar.edu>
* Per #3032, add data type column to all of the output tables * Per #3032, remove the first row from each output table since its info is repeated from the table name. Additional changes for consistency and accuracy in column names. * Update docs/Users_Guide/gsi-tools.rst Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu>
…' based on 1/9/2025 decision at the METplus-Analysis Tools project meeting. Also update MET to link to the METplus User's Guide.
…Integers (not Doubles).
* Per #3024, roll the version number from 12.0 to 12.1 prior to adding new columns to the existing GRAD line type. * Per #3024, return bad data for the square root of a negative number * Per #3024, update Grid-Stat to write 4 new columns to the GRAD line type. * Per #3024, tweak laplace MSE computation * Per #3024, add appendixC.rst equations for the new statistics * Per #3024, escape embedded underscores in equations only for latex formatting, not html * Per #3024, add reference to paper and summary of the new gradient stats * Per #3030, update docs by moving the gradient config entry from Grid-Stat to the common area and then referencing it in both Grid-Stat and Series-Analysis. * Per #3030, update Series-Analysis config file by adding the gradient dictionary and an entry for output_stats.gradient. Update the conf_info source code to parse them. Still need to update OTHER Series-Analysis config files and also update the logic in series_analysis.cc to compute GRAD statistics. * Per #3030, add descriptions of GRAD statistics for inclusion in the long_name attribute of the Series-Analysis output files. * Per #3030, add the GRADInfo::get_stat() member function to be called by Series-Analysis * Per #3030, fix parsing logic for the gradient dictionary. * Per #3030, add gradient entries to existing Series-Analysis config files * Per #3030, add GRADInfo::set_stat() member function * Per #3030, enhance Series-Analysis to aggregate the GRAD line type incrementally across multiple runs. However, this can only be done when requesting that 'ALL' GRAD columns be written. * Per #3030, reduce the volume of SonarQube code smells. * Updated syntax * Updated syntax * Updated syntax --------- Co-authored-by: MET Tools Test Account <met_test@seneca.rap.ucar.edu> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com>
…ve to MET_BASE (<install_loc>/share/met) and other files that are only in the MET repo are found relative to MET_TEST_BASE (MET/internal/test_unit). Also remove MET_BUILD_BASE env var (#3053)
* Per #1019, initial setup for supporting -format uscrn. It compiles but I still need to make it work for the variety of USCRN inputs. * Per #1019, add NumArray constructor using a vector of doubles. * Per #1019, saving progress after handling monthly, daily, and hourly input files. Need to complete support for other format types and handle the unit strings * Per #1019, consistent spacing. * Per #1019, tweak log messages so that the file being read is logged before it's actually read so that an error in parsing the data will indicate which file caused it. * Per #1019, update DataLine and LineDataFile classes to support parsing .csv files. Get rid of the unneeded Offsets vector. Add AllowEmptyColumns option to the DataLine class so that multiple delimiters in a row will be treated as separate columns. Since the default delim is whitespace, it makes sense that you'd want to parse multiple delims in a group. But for .csv files, each comma indicates a new column. * Per #1019, update USCRN handler code to support all 7 input variants, including .csv files. This required updates to the DataLine and LineDataFile classes to parse the .csv data properly. Still need to enhance ascii2nc to write units * Per #1019, add units to the Observation class. * Per #1019, add units string to the SummaryObs and SummaryKey classes. * Per #1019, update USCRN format to write units. Consider defining units for all the other ascii file types as well. * Per #1019, add StringArray::all_empty() member function to check for a list of all empty strings. This is used in ascii2nc to determine if observation units and descriptions should be written. * Per #1019, update library code to handle the independent writing of point observation descriptions. Previously, if units were present then descriptions (usually empty ones) were added. Now, units and descriptions and handled independently. * Per #1019, add descriptions for all USCRN observations, pulled from the USCRN website. * Per #1019, need to subtract 1900 from the year and 1 from the month to make this work. Seems like we should ADD these numbers where needed rather than subtracting them everywhere else! * Per #1019, get rid of USCRNHandler::_readHeaderInfo() since it compilicated the logic for ignoring the first line from csv files. * Per #1019, rather than always skipping the first line of USCRN csv files, just skip any lines where the station ID begins with 'WBAN'. That'll handle files being concatenated together and is simpler logic. * Per #1019, add an ascii2nc unit test for USCRN point observations. * Per #1019, doc-only change to indicate the prefix/suffix of the input USCRN files are used to the determine the specific format. * Per #1019, update USCRN code to no longer specify a default _qcOffset value since it conflicts with the initialization. While the GHA compiler is fine with it, the SonarQube one is not. These changes should enble to SonarQube build to complete. * Per #1019, for SonarQube replace ALL instances of push_back() with emplace_back() which SonarQube prefers for efficiency. * Per #1019, more changes to address SonarQube code smells and reduce the overall number of them lower than what's in the develop branch. * Per #1019, second pass through to further reduce SonarQube findings. * Per #1019, revert back to protected members in file_handler.h * Per #1019, one last round of minor SonarQube code smell remediation.
* Per #3047, modify the intel oneapi settings for compiler variable * Per #3047, modifications for ATLAS and ECKIT * Per 3047, creating a new development environment file for MET for seneca using the necessary values for the Intel oneAPI compilers on seneca * Per #3047, uploading config file used to install MET on seneca with Intel oneAPI * Per #3047, updated development file to use appropriate compiler settings * Per #3047, removing paths to compilers * Per #3047, removing paths to compilers * Per #3047, added BUFRLIB_NAME and GRIB2CLIB_NAME * Per #3047, adding paths back in for now * Per #3047, removing paths to compilers * Per #3047, commenting out source of setvars.sh and adding paths to older Intel oneAPI compilers * Per #2047, updates for compilation however a pb2nc unit test still fails * Per #3047, updating file for the 2024.2.1.100 compilers * Per #3047, updating some ADDTL_DIRs --------- Co-authored-by: Julie Prestopnik <jpresto@seneca.rap.ucar.edu>
* Per #3006, add new pair_stat tool as a full copy of the point_stat tool with all instances of point_stat renamed as pair_stat. * Per #3006, add pair_stat to the list of things for which no 'make test' command is run. * Per #3006, saving work in progress prior to seneca reboot * Per #3006, revert back to using FileType instead of GrdFileType. That change was not meaningful or warranted. * Per #3006, revert back to using FileType instead of GrdFileType. That change was not meaningful or warranted. * Per #3006, committing changes since the code is compiling. Added IODADataConfig_default file to store default settings for reading IODA data. * Per #3006, starting to tweak config options. Saving progress while it's successfully compiling * Per #3006, add fcst.pairs and obs.pairs config entries. * #3007 Added vx_ioda * #3007 Added vx_ioda * #3007 Added vx_ioda * #3007 Derived from IODADataConfInfo * #3007 Reduced the code smells (SonarQube findings) * #3007 Added station_value_base_t and point_pair_t * Initial release * #3007 Changed ack the location of nc_point_obs.set_nc_out_data * Changed station_value_base_t::clear() to station_value_base_t::clear_base() * Changed bAPI names * #3007 Reduced code smells * #3007 CLeanup * #3007 Cahnged API for IODADataConfInfo * #3007 Renamed ioda_file to ioda_reader * #3007 Corrected comment * #3007 Added -lvx_statistics again * #3007 Added get_nc_data(NcVar *, unixtime) * #3007 Cleanup * #3007 Added add_to_unixtime((unixtime) * #3007 Reduced the complexiity of read_time. Added read_time_as_number * #3007 Added read_time_as_number * #3007 Added add_to_unixtime(unixtime) * #3007 Cleanup * #3007 Set bad_data_int to qc_buf * #3007 Cleanup * Per #3006, define new GrdFileType::FileType_Pairs enumerated value to be used in the pair_stat tool. * Per #3006, update pair_stat to use the newly added GrdFileType::FileType_Pairs enumerated value. * #3007 Temporarily removed pair_stat * Per #3006, rerun bootstrap on seneca to incorporate the compilation of the vx_ioda library. * Per #3006, make docs build without warning * Per #3006, saving compiling state * Per #3006, use ConcatString instead of std::string for consistency. * Per #3006, work in progress * Unrelated to #3006, but fix typo in log message. * Per #3006, default_column_union was defined in 2 spots. Renaming one of them to avoid compilation conflict. * Per #3006, move StatHdrInfo out of aggr_stat_line.h/.cc and into vx_stat_out/stat_hdr_info.h/.cc. This make it available to both Stat-Analysis and the Pair-Stat tool to track the unique STAT headers elements read. * Per #3006, remove the unused land/topo/msg_type type config options from the pair_stat tool's configuration file and code that parses it. If needed, we can add it back in the future. * Per #3006, update VarInfoPairs::set_dict() to also call VarInfo::set_magic(). * Per #3006, since python_line.h lives in src/basic/vx_util, the vx_util library now also depends on the * Per #3006, saving off version that compiles before trying changes that may not. * #3007 Deleted commented out cpde * Changed data typo (float to double) * #3007 Resio;lved SonarQube finding * Per #3006, added logic to track ck unique header input columns like Stat-Analysis does. Committing the current state of this branch prior to merging in changes from the #3007 feature branch and completing development for the beta1 cycle. * Per #3006, fix indexing for vx_opt * Per #3006, error out for -format ioda and -format python * Per #3006, make -format ioda or -format python error out, but add unit tests demonstrating those errors. Also note this in the user's guide. * Per #3006, replace -outdir with -out and remove output_prefix config option. * Per #3006, remove output_prefix from Pair-Stat config files. * Per #3006, update unit_pair_stat.xml to use the -out option. * Per #3006, working version. However, the filtering by grid is not working all that well right now because it's based on G004. * Per #3006, expand the Pair-Stat example. * Per #3006, switch from using global 0.5 degree reference grid to 0.1 degree. This still isn't good enough though. Instead, we need to get rid of the reference grid altogether and keep track of the grid information separately for each mask. * Per #3006, fix for loop typo in 3 spots * Per #3006, remove one line from bad merge * Per #3006, SonarQube updates. * Per #3006, more SonarQube fixes * Per #3006, remove desctrutor as recommended by SonarQube --------- Co-authored-by: Howard Soh <hsoh@seneca.rap.ucar.edu> Co-authored-by: MET Tools Test Account <met_test@seneca.rap.ucar.edu>
|
Note that I confirmed that the differences flagged in this GHA testing workflow are all expected and consistent with the #3057 pull request. It is safe to update the truth data. |
JohnHalleyGotway
added a commit
that referenced
this pull request
Jan 28, 2025
JohnHalleyGotway
added a commit
that referenced
this pull request
Jan 31, 2025
* Per #3066, add Data-IO section about file lists and reference it in all tools that use them. * Per #3060, fix typo * Update docs/Users_Guide/data_io.rst Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * Update docs/Users_Guide/data_io.rst Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * Per #3066, update usage statements in the docs and application code based on PR feedback. * Per #3066, missed a handful of additional updates. --------- Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds pair_stat output and modifies existing ioda2nc output
Created by @JohnHalleyGotway