From b7d44d7c92284e4a5b03ce4462c33dc04dd2620a Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Fri, 12 Apr 2024 10:25:15 -0600 Subject: [PATCH] Update develop-ref after dtcenter/MET#2849 (#2552) * Fix for Dockerfile smell DL4000 (#2112) Co-authored-by: Giovanni Rosa * Changed "PROJECT" to "CYCLE" ASSIGNMENT * Feature 2115 update use cases (#2133) * Bugfix #2082 develop regrid.convert/censor_thresh/censor_val (#2140) fix #2082 main_v5.0 regrid.convert/censor_thresh/censor_val (#2101) * Bugfix #2137 develop PointStat -obs_valid_beg/end (#2141) * Feature #2143 5.1.0-beta2 release (#2144) * update release guide steps with new info based on changes to GitHub web interface * update version for development towards rc1 release * Feature #1977 ARGO use case (#2148) Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> * Feature develop cycle change (#2150) * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * Added paths to ignore --------- Co-authored-by: jprestop * Documentation: Update Truth Data instructions (#2154) * upgrade python to 3.10 for documentation builds to fix failured introduced by urllib3 (see https://github.com/urllib3/urllib3/issues/2168) * update workflow to update input test data to use branch from pull down or enter branch by hand if branch does not exist in dtcenter/METplus repo, e.g. if updating data for a PR coming from a fork * Bugfix #2161 develop PCPCombine additional field arguments in -subtract mode (#2162) * changed order of workflow event triggering info to more easily see the repository and commit sha that triggered the workflow * fix docker compose install and call -- needed due to new Alpine Linux release 3.18.0 that does not include docker-compose in the apk repository * Bugfix #2168 develop - StatAnalysis time shift (#2169) * Feature 1978 update pyreq doc (#2177) * Updating for 5.1.0 * Adding greater than or equal to signs * Feature 2106 release support (#2182) * Added a paragraph for Release Support Policy * Added a sentence about contacting the team through the METplus GitHub Discussions Forum. * Added Release Support Policy info * Per #2159, modify instructions to reflect that we only want a coordinated release for the official releases and updates to the coordinated release section for the bugfix releases. (#2183) * Feature 2147 workflow doc (#2185) * Add new images for github workflow * Added PR section * Modified the Find the GitHub Issue to be Find or Create a GitHub issue * Trying out image keyword compared with figure keyword * Trying out various widths * Added new figures * Reverting to figures from images after modifying figures * Adding the old images which are better quality * Trying out new indention * Working on formatting again * Reducing image widths * Update docs/Contributors_Guide/github_workflow.rst Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Update docs/Contributors_Guide/github_workflow.rst Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Update docs/Contributors_Guide/github_workflow.rst Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Update github_workflow.rst --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * added scripts to create new conda environment for PANDA-C use cases * Bugfix #2189 develop - spaces in complex thresholds (#2191) * added release notes for completed issues for rc1 * Feature 2011 tci from cesm fluxnet2015 netcdf (#2192) Co-authored-by: Mrinal Biswas Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Bugfix #2179 develop TCPairs fix -diag argument (#2187) * sort list of files to make command more readable and make logic more easily testable * fix unit test to expect alphabetical order of files * Feature #2197 MODE new multi-variate options (#2203) * added to release notes * added release notes for other items completed for rc1 release * Feature #1626 TCDiag Wrapper (#2201) * Feature #1610 first round of SonarQube fixes (#2200) * Feature #2195 create v5.1.0-rc1 release (#2204) * Update version for development towards official release * updated script to use metplotpy as base and install pygrib because conda was failing trying to install matplotlib in Docker for some strange reason * Feature 2188 use case panda c (#2202) * added first use case, need to finish docs and add to list * added documentation files, 3 new cases * added 5th use case, docs * final use case, corrected paths for output * Update GridStat_fcstGFS_obsERA5_lowAndTotalCloudFrac.py * removed chars, added space to last section * turn off 5/6 new air quality and comp use cases because they are causing the disk to fill up in the automated tests and only 1 is needed to test the new behavior. Reordered use case groups to be alphabetical * disabled other new air quality use case because the disk is still filling up from this case * Feature 2136 cross spectra (#2208) * per #2212, continue instead of return so other forecast leads will not be skipped * Feature #2210 MODE multi-variate variable rename (#2211) * Bugfix diff tests (#2217) * added change from NCEP produtil that was somehow not included in the changes pulled from their repo before starting SonarQube work * Feature #2228 v5.1.0-rc2 release (#2229) * update version after rc2 release to note development towards official 5.1.0 release * Feature #2188 move PANDA-C cases to clouds category and fix automated tests (#2237) Co-authored-by: j-opatz * Feature 2198 use case amdar pbl (#2206) Co-authored-by: Daniel Adriaansen Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Bugfix #2235 rename multivar_itensity to multivar_intensity_flag (#2236) * Feature #2230 MvMODE usecase doc updates (#2239) This updates the use-case documentation to go along with the multivariate MODE enhancements. * combine release notes from development releases for 5.1.0 release * fix indentation to prevent incorrect nesting of drop down menus * Bugfix #2241 Create directory containing -out_stat file (#2242) * Enhance diff logic to control decimal precision to prevent failures (#2243) * Bugfix #2245 use unique run ID to name logger instance (#2247) * per #2245, add METplusConfig class function that is called when object is deleted to close log handlers. This prevents OSError: [Errno 24] Too many open files from running all pytests * Feature #1626 TCDiag updates to wrapper and basic use case (#2248) Co-authored-by: Jonathan Vigh * move new use case into another group * Bugfix #2244 develop fix diff tests (#2254) * skip diff for specific png images that produce diffs occasionally but look the same visually * add 2 more keywords for use cases that occasionally flag small, unnoticeable diffs in png images * Feature #2253 update tests for diff_util (#2256) Co-authored-by: root * Feature #1974 Documentation: update coding standards section (#2260) * Feature #2253 conftest fixture to set pytest tmpdir (#2261) * Feature #2151 Documentation: Add quick search keywords METplotpy/calcpy use cases (#2259) * Feature #2138 CI enhance MET Docker tag override (#2258) * Updating development instructions. * fixed typo * fixed another typo * Feature #2207 Documentation - Updates to Contributor's Guide (#2263) Co-authored-by: Dan Adriaansen * Feature develop update install files (#2266) * Updating installation files * Removing run_commands.sh * Updated the file to add a MINICONDA_PATH * Adding files for jet * Adding orion file * Removing beta references * Corrected met version * Adding files for WCOSS2 machines * Removing rc1 from acorn file * Removing beta1 * Bugfix #1853 develop - PointStat don't require mask variables to be set (#2262) * update version of certifi to match auto-generated dependabot PR #2267 * feature #2253 more tests for diff_util.py (#2264) * update versions of METplus components for the next official release * updated release notes for 5.1.0 release * removed duplicate entries in release notes * Added EC2 instance recipee with S3 mounting ability (#2269) * Updating the Subsetting Tests by Marker section (#2272) * update release date for 5.1.0 release (#2276) * update version for development towards 6.0.0-beta1 release * added new use case that was missing * Bugfix #2279 develop - buoy station file from 2022 (#2280) * Feature 2253 series analysis tests (#2277) * Fix typo in pull_request_template.md * Make code coverage report available (#2287) * Use updated dtcenter/metplus-action-data-update tag that fixes bug that causes GHA disk to exceed capacity when too many data volumes are created. Use specific commit of coveralls python action to ensure it will not change * removed climatology keyword from use cases that aren't relevant that were accidentally added with PR #1984 * update readme with current information * Feature #2282 coord_release_docs (#2288) Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * continue workflow if coveralls step fails -- this typically happens on a pull request from a fork because the authentication fails * Feature 2253 system util tests (#2297) * Feature #2294 LSR use case (#2295) * Feature 2253 run util tests (#2307) * Release Guide - remove beta/rc tags/releases (#2309) * Add 'component: repository maintenance' label. Already ran the script to push this change and the NOAA OAR reporting label to all the METplus repos. * Hotfix for labels, adding a new one for 'component: input data' and also fixing the get_lablels.sh to search for up to 200 existing labels. Also work on the log messages. * Fix typo in comment * Feature #2283 time looping consolidation (#2311) Co-authored-by: John Halley Gotway * New issue template: Update Truth (#2332) Co-authored-by: John Halley Gotway * feature #2253 tests for run_util, 'Usage' bugfix (#2313) * Feature #2338 Debian 12 Conda environments (#2341) Co-authored-by: John Halley Gotway * Feature 1488 usecase satwinds (#2324) Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Feature #2283 include times (#2345) Co-authored-by: John Halley Gotway * feature #2253 tests for config_validate, mock logger in conftest (#2320) * Feature #2299 / #2310 Remove deprecated MET config env vars and TCMPRPlotter (#2347) Co-authored-by: John Halley Gotway * remove MODEL and OBTYPE to use defaults from MET config * Feature #2348 v6.0.0-beta1 release (#2351) * update version for development towards 6.0.0-beta2 release * copy libGL and libEGL files into docker image to prevent errors with geovista python package * run use that uses geovista to test * Feature #2156 release_acceptance_testing (#2352) * feature #2253 print error logs from conftest (#2358) * feature #2253 met_db_load tests (#2355) * Rename 5.1.0.lua_wcoss2 to 6.0.0.lua_wcoss2 * Update and rename 5.1.0_acorn to 6.0.0_acorn * Update 6.0.0_acorn * feature #2253 add tests for gfdl_tracker (#2354) Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Create 6.0.0_gaea * Update and rename 5.1.0_hera to 6.0.0_hera * Feature #2156 release_acceptance_testing take2 (#2361) Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> * Feature #2329 Docker info in Installation Chapter (#2366) * feature_2253_extract_tiles_tests (#2368) * Feature 2253 tc pairs tests (#2369) * Feature 2253 tc csv writer (#2373) * update requirements for building docs to get around build failure where python packages required to build RTD have disappeared from the automatically generated commands from RTD * fix ReadTheDocs requirements to include pillow which is a dependency of sphinx-gallery: see https://blog.readthedocs.com/defaulting-latest-build-tools/ for more info on why this was necessary * Feature #2340 TCDiag one_time_per_file_flag (#2374) * Update and rename 5.1.0_jet to 6.0.0_jet * Added libssh * prune docker files before running use cases to prevent running out of disk space when running use cases, ci-run-all-diff * Feature 2253 command builder tests (#2378) * Feature 2253 series analysis test (#2380) * Feature 2253 py embed test (#2379) * ignore cyclone plotter from code coverage report because it will be replaced with METplotpy logic * Feature 898 headers (#2389) * changing _ to - for header consistency * changing _ to - for header consistency * updating all headers for consistency * updating all headers for consistency and adding spacing The spacing doesn't seem to be required but it is the convention that we follow for headers. * updating all headers for consistency * updating headers for consistency and adding capital letters to headers * Using the overline ### to keep index consistent with other indexes * updating all headers for consistency * update requirements for building docs to get around build failure where python packages required to build RTD have disappeared from the automatically generated commands from RTD * updating all headers and some spacing for consistency * updating headers for consistency * changing to ### for consistency * Per #898, fixed the headers for the Release Guide part, chapters, and sections. * Duplicating changes in develop branch for requirements.txt file * updating headers * Per #2669, updated header formatting to resolve errors * Per #2669, udpating header * Per #2669, udpating headers * Per #2669, udpating header * Per #2669, updated header formatting * Per #2669, update header formatting * updating headers * Per #898, removed space in front of title * Capitalizing Please * changing to just italics to match standard formatting * indenting for consistent formatting * fixing italics again * changing from note to warning for consistency * updating headers, adding some capitalizing to the headers * fixing typo Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * This file was committed but never updated or used Per Minna, ok to delete. * Restructuring table of contents to make it more clear which guides are for users and which are for contributors * fixing formatting for clairity Co-authored-by: Julie Prestopnik --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> Co-authored-by: Julie Prestopnik * Feature #2349 upgrade instructions for deprecated MET config files (#2386) * Feature 2123 di doc update (#2392) Co-authored-by: Tracy * change log to list name of config variable , e.g. GRID_STAT_CONFIG_FILE, to easily see which variable should be removed * Feature 1667 doc conventions (#2397) * New additions based on the old branch feature_1667_updating_overview * Moving the release-notes reference label * Added label for METplus_glossary for use in Documentation conentions section. * Adding images for the Documentation Conventions section * Modifying wording and testing formatting in Internal Links section * Second take on formatting * Third attempt at formatting * Fourth attempt at formatting * Modified wording, sections, and formatting * Minor modifications * Added period * Changed Pretty Table to PrettyTable * Modify informationg about converting an existing table and adding images * Resolving errors * Reformatting * Moving placement of reference label * Attempting to fix table title * Fixed incorrect alignment * Made changes based on Lisa's suggestions * Made changes based on Lisa's suggestions * Made corrections * Made corrections * Per #1667, fixing typos * Per #1667, corrected text --------- Co-authored-by: Julie Prestopnik * Feature #2377 Log to terminal only (#2398) * Update conda envs to use conda-forge only (#2402) * rearrange MET config env var tables for GridStat so they are in the order that they appear in the wrapped MET config file * use mamba instead of conda and update version of netcdf python package used to 1.6.2 * skip s2s_mid_lat WeatherRegime use case that produces different results often * updated version of xesmf because <0.7.1 does not work with mamba * per #2412, fix bug in GenVxMask to put quotes around argument so a grid specification string with spaces will be an accepted value * downgrade version of esmf package to fix bug described in https://github.com/pangeo-data/xESMF/issues/246 * Feature #2219 SeriesAnalysis multiple input files (#2408) * Adding 3 new requestors to the list of common_labels.txt for NOAA/NHC, CSU/CIRA, and Australian BOM ci-skip-all * Feature 2405 match tables2wrapper (#2416) Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * per #2423, follow symbolic links when traversing directories to find files within a time window, ci-run-all-diff * Feature #2252 WaveletStat wrapper (#2427) Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> * add WaveletStat use case to group and temporarily disable TCDiag use case because changes to the wrapper are needed to fix it after changes to the MET tool were merged * update version number for beta2 release (#2431) * update version for dev towards beta3 * Feature #2371 Embed use case upgrade demo video into upgrade instructions (#2444) * fix failing use case tests -- install python packages dateutil and requests via apk instead of pip because the pip commands began failing -- see PEP668 https://peps.python.org/pep-0668/ -- also changed scripts to create conda environments for use case tests to install all packages in a single call to mamba instead of individual calls * remove commands to install geovista from source because it is now installed from conda * Feature #1989: Add OMI use case (#2457) Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Feature #2432 TCDiag new config changes (#2453) * move medium range 10 use case into group with 3-5 to speed up runs * Feature #2334 land_mask (and topo_mask) in PointStat (#2456) * added use cases with pygrib * Feature #2430 TCPairs consensus.diag_required and consensus.diag_min_req (#2439) * Quickfix cloud use case desc imgs (#2464) * added pics, updated desc * add last two imgs * Fixing spelling and capitalization * Feature 2454 doc overview conv (#2471) * adding documentation in different sections * adding grid table section * fixing links * grammar updates * Per #2454, updated sections and wording. * Per #2454, added a period to the end of a sentence. * Per #2454, fixing formatting * Per #2454, updating wording * adding a section for line breaks in a table * adding :code: information * trying to fix warning * take 2 * take 3 or 4 * maybe fixed * updating link * fixing web link again * web link saga continues * Changed "ReadTheDocs" to "Read the Docs" * Updated "main_v" references to be "main_v12.0" * Removed references to main_v*, replacing with raw RST It is not maintainable to have links to branches which will become old. Since we can avoid it by adding the raw RST in the documentation, I have removed all references to main_v* in favor of placing the raw RST in the documentation. * Modified the "Code in a Paragraph" section * Reworded for consistency within the document * Added back the link for Sphinx code blocks --------- Co-authored-by: Julie Prestopnik * add argument to workflow dispatch event so that MET DockerHub repo used for tests can be easily overridden to test changes in a MET PR before merging * Feature dtcenter/MET#2796 GHA Node20 deprecation warnings (#2473) * per dtcenter/MET#2796, update versions of actions to prevent deprecated node warnings in GHA runs * change arguments to workflow dispatch so they are no longer required -- these are not needed to be set when triggering by hand through the web interface * Feature dtcenter/MET#2796 develop - Fix error log artifact creation (#2475) * updated version of pillow to fix security vulnerability alerted by dependabot in PR #2477 * remove docker image after runtime image is created from metplus image and conda env image * turn on use case to test image removal * prune images if image tag doesn't exist -- it appears that if the image is built on the fly (when PR is coming from fork) then the tag exists, but if not, the image tag is set to * support commands that must run in the shell to see if || will work in docker image pruning step * try to fix image removal * Feature 2383 use case sat alt (#2480) * new docs, files for use case * new files * updating to run use case * updated python libraries, changed test env * trying new point logic * added to script for nan removal * redid Python script to take adv of new MET ability for nans * Update run status * removed unused settings * run image prune commands separately * changed shell back to false * split up use case groups so the same envs are used by a group to see if that resolves the disk space issues * turn off use cases * feature 2253 fix empty pytest logs (#2485) * added more commands to free up disk space as suggested in https://github.com/apache/flink/blob/master/tools/azure-pipelines/free_disk_space.sh, ci-run-all-cases * Feature 2406 redo usecase rrfs (#2488) * issue #2406 RRFS use case files * issue #2406 added usecase to tests * Issue #2406 added metplotpy and metcalcpy as dependencies * Feature #2460 allow missing input (#2493) * changed template to use datetime format that works on MacOS * update logic to only write a file list file if there are more than 1 files, updated unit tests to match new behavior, added exception handling to series analysis to prevent crash if file does not exist * use getraw instead of getstr to prevent crash if providing a filename template tag to override a config variable on the command line * Add optional argument to subset file function to always write a file list text file even if there is only 1 file found. Use this argument in UserScript wrapper so that the environment variables that contain paths to file list files are consistent in format for use in user scripts * enhanced function to support different output variable types * removed the need for overriding clear function in specific wrappers and added optional argument to skip clearing input file list * clean up formatting * per #2460, start to implement logic to prevent errors when some input files are not found * isolate logic to find input files into find_input_files functions. clean up those functions to return boolean instead of sometimes returning None or a list of files to be consistent * remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type * turn on use cases to test error handling * merge artifacts * run only failed cases * always run merge step * run on a case that will succeed to test error log merge step * only run error log merge step if there were 'Save error logs' jobs that succeeded * run cases that will fail * fix condition to merge error logs * run group that will succeed but have diffs - check error logs doesn't fail * testing - add use case group that will succeed but will cause diffs becaus there is no truth data - to confirm that the error log merge step behaves properly in this case * run 3 jobs, 2 should error, to confirm that error_logs is created properly * repeat diff no error test but with * per dtcenter/MET#2796, fix error log artifact creation by merging error logs if any of the 'Save error logs' steps ran successfully * run test to confirm diff does not cause merge error logs to fail * Revert "run test to confirm diff does not cause merge error logs to fail" This reverts commit ff2d1cac57c431a047ee250e9dae9b0a813a78ba. * run test to confirm error logs are merged properly when 2 use case groups have errors * try checking output variable as string instead of boolean * Revert "run test to confirm error logs are merged properly when 2 use case groups have errors" This reverts commit 8106666a73685e654e0146d4fed56f2382f1bfc7. * run test again * test again * move check for error logs for shell script and use github env vars * Revert "run test again" This reverts commit 7a0a99c6e7031c5dafb1177d4b4ca3f32a999dac. * break 2 use cases to test that error logs are still created properly * checkout repo to get script used to merge error logs * Revert "break 2 use cases to test that error logs are still created properly" This reverts commit cb6d0b46db353b4b4709183be2fe7e5ce64ff5ff. * test merge error log again on no error diff run * fix script * move merge error logic back to workflow * break 2 use cases to test that error logs are still created properly * Revert "break 2 use cases to test that error logs are still created properly" This reverts commit 82aa0e11096aace3ccc2c79cd631533fc6426900. * remove testing use case group * Revert "remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type" This reverts commit de3b4b03a45bb871c71e770ff9e602739d6b63d5. * clean up lines * update logic to check that python embedding is set up properly to only try to set file_type automatically if it is not already set and if the wrapper is a tool that supports multiple input files via python embedding (which require file_type to be set). also changed error if not set properly to warning and use PYTHON_NUMPY as a default * remove run_count increment before run_at_time_once - set closer to find_input_files so run count and missing input count are consistent * return boolean from find_input_files function to be consistent with other functions * per #2460, warn instead of error if missing inputs are allowed, track counters for number of runs and missing inputs * per #2460, added check to report error if allowed missing input threshold is met * run clear before running plot_data_plane * removed test group * report warning instead of error if ALLOW_MISSING_INPUTS is True * cleanup * change function to pytest fixture so it can be used by other test scripts * update ascii2nc test to process more than 1 time to ensure commands are built properly for each run * add unit tests to ensure missing input file logic works properly for ascii2nc and grid_stat * set variable to skip RuntimeFreq logic to find input files to prevent duplicate increment of run_count -- these will be removed when the wrapper has been updated to find files using RuntimeFreq logic * remove unneccesary error checking * cleanup * call function to handle input templates that need to be handled separately for each item in the comma-separated list (for UserScript and GridDiag only) * add time_info to ALL_FILES dictionaries to be consistent with other wrappers * clean up logging for reporting error when missing inputs exceeds threshold * added function to get files for a single run time to be consistent with other functions * skip increment of run_count when FIND_FILES=True and RuntimeFreq input file logic is skipped to prevent duplicate increments * added empty test files * remove redundant variables * view warnings on a failed test run * add more empty test files * added unit tests for missing input logic * remove MANDATORY setting for EnsembleStat and GenEnsProd and instead pass mandatory argument to call to find model files so warnings/errors are properly displayed for other inputs * cleanup * remove allow missing input logic from ExtractTiles wrapper * added functions to parse template/dir variables from config, removed explicit calls to read those variables from GridStat * remove error if more labels than inputs are provided (for UserScript and GridDiag only) -- extra labels will just be ignored * added required boolean for input templates * per #2460, change warning messages to debug when checking a list of DA offsets since it is common that a given offset will not always be found in the files * added tests for missing input logic for many wrappers * cleanup * fix increment of number of runs * skip missing input logic * change how required is handled for input templates * warn instead of error if missing input is allowed * remove increment of missing input counters because it is handled in RuntimeFreq * check status of input files and increment counters in overridden run_once_per_lead. remove increment of missing input counters because it is handled in run_once_per_lead * added unit tests for missing input logic * skip missing input logic * cleanup * cleanup, use fixture for tests, add unit tests for missing input, bypass missing input logic on wrappers that don't need it * removed file that is not needed * added unit tests for pb2nc to test -valid_beg/end arguments and changes to properly support any runtime frequencies * warn instead of error if allowing missing inputs * cleanup * implement changes to properly support all runtime frequencies for pb2nc. previously all files that match a wildcard will be used instead of selecting only files that fall within the specified time range. some functions moved into pb2nc wrapper will eventually be moved up so that they are used by all wrappers to be consistent * added unit tests that will fail until wrapper is updated * replace functions in RuntimeFreq wrapper used to find input files so they can be used by all wrappers, updated ioda2nc wrapper to find input files properly to fix tests * cleanup * removed mtd version of get_input_templates and added logic to RuntimeFreq's version to get the same behavior * added unit tests for MTD missing input checks * per #2491, add release notes for beta3 * Feature #2491 v6.0.0 beta3 (#2495) * update version for beta3 release * fixed typos in release notes * update version to note development towards beta4 release * Per suggestion from @JohnHalleyGotway, create intermediate branch for updating truth data to avoid branch protection rules. I added a step to delete the intermediate branch locally if it exists to prevent conflicts with the update * added quotes to prevent error in echo caused by parenthesis * fix incorrect command * Revert "fix incorrect command" This reverts commit e7dffb6b0b351ab1b4bca5b563c1f5beef7737a9. * Revert "added quotes to prevent error in echo caused by parenthesis" This reverts commit c1cb3c4f0d7851bea720a50fac6011cd381017dc. * Revert "Per suggestion from @JohnHalleyGotway, create intermediate branch for updating truth data to avoid branch protection rules. I added a step to delete the intermediate branch locally if it exists to prevent conflicts with the update" This reverts commit 525809dc3bd73ace969b046062967796035f4d86. * Hotfix: Allow symbolic link to run_metplus.py to run (#2500) * Adding use case tests * Changing test environment * Testing environment changes * Documentation update * Updating Documentation * Updating documentation for disk space failure * Added new use case category * Fixing use case test * Fixing bug in use case file * Testing s2s after data removal * add back use cases that were accidentally removed * fix incorrect use case added * Setting tests to false for merge * Testing for old use cases * Setting tests to false for merge * update tests to update develop data -- modified commands to create new use case category directory if it does not already exist, move step to remove old data to be completed just after new data is copied to vX.Y * Feature 2463 modify table (#2508) * creating test dropdown menus * fixing warnings * fixing warnings * fixing warnings 3 * fixing warnings 4 * Attempt to fix documentation errors * adding 2 more test dropdowns please note. There is still a message about WARNING: Duplicate explicit target name: "gridstat: cloud fractions with neighborhood and probabilities (pygrib)". John O will fix this. I should not touch it. * fixing spacing * trying to fix link * take 2 * Removing double underscores added earlier * moving dropdown menus * Adding version to dropdown menu title * fixing spacing * dropdowns date util, eofs, h5py * fixing formatting * fixing formatting * Per #2463, adding template for future entries * adding imageio, lxml & matplotlib * dropdown up to nc-time-axis * fixing spacing problems * Fixing broken s2s links and other incorrect links * Fixing spelling and capitalization * Removing the dash in front of 1.4 for nc-time-axis * Modifying formatting * adding dropdowns thru pylab * fixing problems * dropdowns thru scikit-learn * fixing spacing * final dropdowns thru yaml * fixing spacing * fixing loose ends * Per #2463, moving information to an Appendix and adding text and links in an overview * Per #2463, adding to index.rst * Per #2463, reworded language and updated Python 3.8 reference * Per #2463, fixing errors * Per #2463, made updates based on feedback at the METplus Engineering meeting. * removing tables, changing most METplus wrappers, version numbers. * trying to fix met_version * Per #2463, adding necessary code for substitutions * Per #2463, fixing syntax error * adding period * removing section 1.5 * Per #2463, replace old label reference with new label reference and updated text to reflect the move to drop down menus * read python version from file to replace in docs * fixed typo in variable name * adding python_version to the overview. --------- Co-authored-by: Julie Prestopnik Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * per #2509, automate MET version used in documentation to be X+6.Y.0 of METplus version * Bugfix #2520 ASCII2NC file window issue and redundant wrapper initialization (#2522) * per PyCharm documentation, only ignore workspace.xml idea file and commit the rest of the .idea files to version control * per #2520, create function to get METplus wrapper class without initializing it and use that function to read list of deprecated env vars to prevent redundant initialization of wrappers that can cause unintended side effects * handle file_window variables consistently by using CommandBuilder function * comment out optional config variables that previously caused a failure when unset * Update update_truth.md Fix typo in the update_truth issue template in the develop branch. * Update update_truth.md Update wording in the update_truth issue template. * Feature #2530 dev timeline (#2532) * Per #2530, add a development timeline to the METplus Release Information section of the User's Guide. Also update the Release Guide instructions. * Per #2530, tweak the wording. * Update docs/Release_Guide/release_steps/update_release_notes_development.rst Co-authored-by: Julie Prestopnik --------- Co-authored-by: Julie Prestopnik * Feature #2537 develop sonarqube_gha (#2541) * Per #2537, add SonarQube workflow for METplus * Per #2537, update nightly build email list. * Per #2537, fix cut/paste error configure_sonarqube.sh * Per #2537, exclude test code from code coverage statistics. * Update the 6.0.0 Coordinated Release development timeline in release-notes.rst * Added optional key/value to use_case_groups.json to prevent a use case group from running to easily disable it temporarily. Disable short_range:14 use case until it can be fixed with #2551 * update pillow version based on recommendation from dependabot: https://github.com/dtcenter/METplus/security/dependabot/5 --------- Co-authored-by: metplus-bot <97135045+metplus-bot@users.noreply.github.com> Co-authored-by: Giovanni Rosa Co-authored-by: Giovanni Rosa Co-authored-by: jprestop Co-authored-by: Dan Adriaansen Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> Co-authored-by: mrinalbiswas Co-authored-by: Mrinal Biswas Co-authored-by: Christina Kalb Co-authored-by: j-opatz Co-authored-by: jason-english <73247785+jason-english@users.noreply.github.com> Co-authored-by: Tracy Hertneky <39317287+hertneky@users.noreply.github.com> Co-authored-by: Jonathan Vigh Co-authored-by: John Sharples <41682323+John-Sharples@users.noreply.github.com> Co-authored-by: root Co-authored-by: Hank Fisher Co-authored-by: John Halley Gotway Co-authored-by: reza-armuei <144857501+reza-armuei@users.noreply.github.com> Co-authored-by: Tracy Co-authored-by: Mallory Row Co-authored-by: bikegeek <3753118+bikegeek@users.noreply.github.com> --- .github/jobs/configure_sonarqube.sh | 65 +++++++++++++++ .github/jobs/get_use_cases_to_run.sh | 4 +- .github/parm/use_case_groups.json | 1 + .github/pull_request_template.md | 3 + .github/workflows/sonarqube.yml | 82 +++++++++++++++++++ .../continuous_integration.rst | 25 ++++++ .../update_release_notes_development.rst | 10 ++- .../update_release_notes_official.rst | 7 +- docs/Users_Guide/release-notes.rst | 21 ++++- docs/requirements.txt | 2 +- .../{environment => }/development.seneca | 0 .../sonarqube/environment/development.docker | 10 --- internal/scripts/sonarqube/run_nightly.sh | 7 +- internal/scripts/sonarqube/run_sonarqube.sh | 49 +++++++---- .../sonarqube/sonar-project.properties | 23 ++---- 15 files changed, 254 insertions(+), 55 deletions(-) create mode 100755 .github/jobs/configure_sonarqube.sh create mode 100644 .github/workflows/sonarqube.yml rename internal/scripts/sonarqube/{environment => }/development.seneca (100%) delete mode 100644 internal/scripts/sonarqube/environment/development.docker diff --git a/.github/jobs/configure_sonarqube.sh b/.github/jobs/configure_sonarqube.sh new file mode 100755 index 0000000000..c60e73485b --- /dev/null +++ b/.github/jobs/configure_sonarqube.sh @@ -0,0 +1,65 @@ +#!/bin/bash + +# Constants +SONAR_PROPERTIES_DIR=internal/scripts/sonarqube +SONAR_PROPERTIES=sonar-project.properties + +# Check that this is being run from the top-level METplus directory +if [ ! -e $SONAR_PROPERTIES_DIR/$SONAR_PROPERTIES ]; then + echo "ERROR: ${0} -> must be run from the top-level METplus directory" + exit 1 +fi + +# Check required environment variables +if [ -z ${SOURCE_BRANCH+x} ]; then + echo "ERROR: ${0} -> \$SOURCE_BRANCH not defined!" + exit 1 +fi +if [ -z ${WD_REFERENCE_BRANCH+x} ]; then + echo "ERROR: ${0} -> \$WD_REFERENCE_BRANCH not defined!" + exit 1 +fi +if [ -z ${SONAR_HOST_URL+x} ]; then + echo "ERROR: ${0} -> \$SONAR_HOST_URL not defined!" + exit 1 +fi +if [ -z ${SONAR_TOKEN+x} ]; then + echo "ERROR: ${0} -> \$SONAR_TOKEN not defined!" + exit 1 +fi + +# Define the version string +SONAR_PROJECT_VERSION=$(cat metplus/VERSION) + +# +# Define the $SONAR_REFERENCE_BRANCH as the +# - Target of any requests +# - Manual setting for workflow dispatch +# - Source branch for any pushes (e.g. develop) +# +if [ "$GITHUB_EVENT_NAME" == "pull_request" ]; then + export SONAR_REFERENCE_BRANCH=$GITHUB_BASE_REF +elif [ "$GITHUB_EVENT_NAME" == "workflow_dispatch" ]; then + export SONAR_REFERENCE_BRANCH=$WD_REFERENCE_BRANCH +else + export SONAR_REFERENCE_BRANCH=$SOURCE_BRANCH +fi + +# Configure the sonar-project.properties +[ -e $SONAR_PROPERTIES ] && rm $SONAR_PROPERTIES +sed -e "s|SONAR_PROJECT_KEY|METplus-GHA|" \ + -e "s|SONAR_PROJECT_NAME|METplus GHA|" \ + -e "s|SONAR_PROJECT_VERSION|$SONAR_PROJECT_VERSION|" \ + -e "s|SONAR_HOST_URL|$SONAR_HOST_URL|" \ + -e "s|SONAR_TOKEN|$SONAR_TOKEN|" \ + -e "s|SONAR_BRANCH_NAME|$SOURCE_BRANCH|" \ + $SONAR_PROPERTIES_DIR/$SONAR_PROPERTIES > $SONAR_PROPERTIES + +# Define new code when the source and reference branches differ +if [ "$SOURCE_BRANCH" != "$SONAR_REFERENCE_BRANCH" ]; then + echo "sonar.newCode.referenceBranch=${SONAR_REFERENCE_BRANCH}" >> $SONAR_PROPERTIES +fi + +echo "Contents of the $SONAR_PROPERTIES file:" +cat $SONAR_PROPERTIES + diff --git a/.github/jobs/get_use_cases_to_run.sh b/.github/jobs/get_use_cases_to_run.sh index d937f1f7a8..a70ec08ac0 100755 --- a/.github/jobs/get_use_cases_to_run.sh +++ b/.github/jobs/get_use_cases_to_run.sh @@ -18,10 +18,10 @@ if [ "$run_use_cases" == "true" ]; then # if only running new use cases, add to filter criteria if [ "$run_all_use_cases" == "false" ]; then echo "Only run use cases that are marked to run every time (run = true)" - matrix=$(jq '[.[] | select(.run == true) | (.category + ":" + .index_list)]' $use_case_groups_filepath) + matrix=$(jq '[.[] | select(.run == true) | select(.disabled != true) | (.category + ":" + .index_list)]' $use_case_groups_filepath) else echo Add all available use cases - matrix=$(jq '[.[] | (.category + ":" + .index_list)]' $use_case_groups_filepath) + matrix=$(jq '[.[] | select(.disabled != true) | (.category + ":" + .index_list)]' $use_case_groups_filepath) fi fi diff --git a/.github/parm/use_case_groups.json b/.github/parm/use_case_groups.json index 6963d1127e..ea51000c83 100644 --- a/.github/parm/use_case_groups.json +++ b/.github/parm/use_case_groups.json @@ -267,6 +267,7 @@ { "category": "short_range", "index_list": "14", + "disabled": true, "run": false }, { diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index dda54835fa..342abeb992 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -11,6 +11,9 @@ - [ ] Will this PR result in changes to the test suite? **[Yes or No]**
If **yes**, describe the new output and/or changes to the existing output:
+- [ ] Do these changes introduce new SonarQube findings? **[Yes or No]**
+If **yes**, please describe: + - [ ] Please complete this pull request review by **[Fill in date]**.
## Pull Request Checklist ## diff --git a/.github/workflows/sonarqube.yml b/.github/workflows/sonarqube.yml new file mode 100644 index 0000000000..b204179029 --- /dev/null +++ b/.github/workflows/sonarqube.yml @@ -0,0 +1,82 @@ +name: SonarQube Scan + +# Run SonarQube for Pull Requests and changes to the develop and main_vX.Y branches + +on: + + # Trigger analysis for pushes to develop and main_vX.Y branches + push: + branches: + - develop + - 'main_v**' + paths-ignore: + - 'docs/**' + - '.github/pull_request_template.md' + - '.github/ISSUE_TEMPLATE/**' + - '.github/labels/**' + - 'build_components/**' + - 'manage_externals/**' + - '**/README.md' + - '**/LICENSE.md' + + # Trigger analysis for pull requests to develop and main_vX.Y branches + pull_request: + types: [opened, synchronize, reopened] + branches: + - develop + - 'main_v**' + paths-ignore: + - 'docs/**' + - '.github/pull_request_template.md' + - '.github/ISSUE_TEMPLATE/**' + - '.github/labels/**' + - 'build_components/**' + - 'manage_externals/**' + - '**/README.md' + - '**/LICENSE.md' + + workflow_dispatch: + inputs: + reference_branch: + description: 'Reference Branch' + default: develop + type: string + +jobs: + sonarqube: + name: SonarQube Scan + runs-on: ubuntu-latest + + steps: + + - uses: actions/checkout@v4 + with: + # Disable shallow clones for better analysis + fetch-depth: 0 + + - name: Get branch name + id: get_branch_name + run: echo branch_name=${GITHUB_REF#refs/heads/} >> $GITHUB_OUTPUT + + - name: Configure SonarQube + run: .github/jobs/configure_sonarqube.sh + env: + SOURCE_BRANCH: ${{ steps.get_branch_name.outputs.branch_name }} + WD_REFERENCE_BRANCH: ${{ github.event.inputs.reference_branch }} + SONAR_HOST_URL: ${{ secrets.SONAR_HOST_URL }} + SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }} + + - name: SonarQube Scan + uses: sonarsource/sonarqube-scan-action@master + env: + SONAR_HOST_URL: ${{ secrets.SONAR_HOST_URL }} + SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }} + + - name: SonarQube Quality Gate check + id: sonarqube-quality-gate-check + uses: sonarsource/sonarqube-quality-gate-action@master + # Force to fail step after specific time. + timeout-minutes: 5 + env: + SONAR_HOST_URL: ${{ secrets.SONAR_HOST_URL }} + SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }} diff --git a/docs/Contributors_Guide/continuous_integration.rst b/docs/Contributors_Guide/continuous_integration.rst index b4283f9a97..f98fbb14f2 100644 --- a/docs/Contributors_Guide/continuous_integration.rst +++ b/docs/Contributors_Guide/continuous_integration.rst @@ -1191,6 +1191,31 @@ to run. Example:: The above example will run data_assimilation 0, 1, 2, and 4 in one job, then data_assimilation 3 in another job. +.. _cg-ci-disable_use_case: + +Disabling Use Cases +""""""""""""""""""" + +Sometimes use cases should not run in the automated test suite. +For example, changes to another repository may break a use case and prevent it +from running successfully until a fix can be applied. +In the meantime, to prevent the use case(s) from failing in the automated tests, +a use case group can be disabled by adding *"disabled": true* in +**use_case_groups.json**:: + + { + "category": "short_range", + "index_list": "14", + "disabled": true, + "run": true + }, + +It is recommended to add this key/value pair before the *run* key/value to avoid +having to add a comma to the end of the *run* value. + +If the *disabled* value is set to *true*, then the use case group will not run +even if the *run* attribute is set to *true*. + Run Use Cases ^^^^^^^^^^^^^ diff --git a/docs/Release_Guide/release_steps/update_release_notes_development.rst b/docs/Release_Guide/release_steps/update_release_notes_development.rst index 1baf3ef0f6..a49edc7f92 100644 --- a/docs/Release_Guide/release_steps/update_release_notes_development.rst +++ b/docs/Release_Guide/release_steps/update_release_notes_development.rst @@ -26,8 +26,16 @@ release. Open the following URL in a browser: * If you are creating a beta1 release, remove the previous version's release notes, i.e. for 3.0.0-beta1, remove all 2.Y.Z notes and start a 3.0.0 section with the format "Version X.Y.Z release notes (YYYYMMDD)". - + * If you are creating a betaX release, add a new betaX section above the betaX-1 release. +* For the METplus repository, update the **development timeline**. + + * If you are creating a beta1 release, add development timeline + information with approximate dates for planned development cycles. + + * For other development releaes, edit the actual release dates and planned + release dates for future development cycles, as needed. + * Commit changes and push to GitHub. diff --git a/docs/Release_Guide/release_steps/update_release_notes_official.rst b/docs/Release_Guide/release_steps/update_release_notes_official.rst index d7509609a8..5da7a676db 100644 --- a/docs/Release_Guide/release_steps/update_release_notes_official.rst +++ b/docs/Release_Guide/release_steps/update_release_notes_official.rst @@ -20,10 +20,13 @@ release. Open the following URL in a browser: (e.g. Enhancements, Bugfixes, Documentation, etc.) and modifying GitHub issue titles for consistency. The release notes should match the GitHub issue titles, when possible. - + * Use your best judgement to apply bold formatting for any major or important changes. * When creating the official release, combine the beta release sections into one section (i.e. "Version X.Y.Z release notes (YYYYMMDD)"). - + +* For the METplus repository, remove the **development timeline** information + prior to creating the official release. + * Commit changes and push to GitHub. diff --git a/docs/Users_Guide/release-notes.rst b/docs/Users_Guide/release-notes.rst index 0bd253b36d..5d33b75b0b 100644 --- a/docs/Users_Guide/release-notes.rst +++ b/docs/Users_Guide/release-notes.rst @@ -4,10 +4,23 @@ METplus Release Information *************************** -Users can view the :ref:`releaseTypes` section of -the Release Guide for descriptions of the development releases (including -beta releases and release candidates), official releases, and bugfix -releases for the METplus Components. +Users can view the :ref:`releaseTypes` section of the Release Guide +for descriptions of the development releases (including beta releases +and release candidates), official releases, and bugfix releases for +the METplus Components. + +.. _development_timeline: + +The **development timeline** for the METplus 6.0.0 Coordinated Release +is broken down into the following development cycles for each component: + +1. **Beta1** releases for the METplus components occurred around 2023-09-15. +2. **Beta2** releases for the METplus components occurred around 2023-11-14. +3. **Beta3** releases for the METplus components occurred around 2024-02-08. +4. **Beta4** releases are tentatively scheduled for 2024-04-17. +5. **Beta5** releases are tentatively scheduled for 2024-06-26. +6. **Release Candidate 1** releases have not yet been scheduled. +7. **Official Release** releases have not yet been scheduled. .. _components-release-notes: diff --git a/docs/requirements.txt b/docs/requirements.txt index 4ae7110210..0628125f64 100644 --- a/docs/requirements.txt +++ b/docs/requirements.txt @@ -1,5 +1,5 @@ sphinx==5.3.0 -pillow==10.2.0 +pillow==10.3.0 sphinx-gallery==0.14.0 sphinx-rtd-theme==1.3.0 sphinx-design==0.3.0 diff --git a/internal/scripts/sonarqube/environment/development.seneca b/internal/scripts/sonarqube/development.seneca similarity index 100% rename from internal/scripts/sonarqube/environment/development.seneca rename to internal/scripts/sonarqube/development.seneca diff --git a/internal/scripts/sonarqube/environment/development.docker b/internal/scripts/sonarqube/environment/development.docker deleted file mode 100644 index 62db756cc9..0000000000 --- a/internal/scripts/sonarqube/environment/development.docker +++ /dev/null @@ -1,10 +0,0 @@ -# Define the development environment for NCAR project machine seneca -# Based on settings in /usr/local/src/met/README.snat - -# Top-level MET project directory -MET_PROJ_DIR=`ls -1d /met/MET*` - -# SonarQube -#export SONARQUBE_DIR=/d1/projects/SonarQube/ -#export SONARQUBE_WRAPPER_BIN=$SONARQUBE_DIR/build-wrapper-linux-x86 -#export SONARQUBE_SCANNER_BIN=$SONARQUBE_DIR/sonar-scanner-4.6.2.2472-linux/bin diff --git a/internal/scripts/sonarqube/run_nightly.sh b/internal/scripts/sonarqube/run_nightly.sh index 5e41979cc8..98bcf376b5 100755 --- a/internal/scripts/sonarqube/run_nightly.sh +++ b/internal/scripts/sonarqube/run_nightly.sh @@ -20,8 +20,7 @@ #======================================================================= # Constants -#EMAIL_LIST="johnhg@ucar.edu hsoh@ucar.edu jpresto@ucar.edu linden@ucar.edu mccabe@ucar.edu" -EMAIL_LIST="johnhg@ucar.edu hsoh@ucar.edu mccabe@ucar.edu" +EMAIL_LIST="johnhg@ucar.edu hsoh@ucar.edu jpresto@ucar.edu mccabe@ucar.edu" KEEP_DAYS=5 function usage { @@ -39,7 +38,7 @@ SCRIPT_DIR=`dirname $0` if [[ ${0:0:1} != "/" ]]; then SCRIPT_DIR=$(pwd)/${SCRIPT_DIR}; fi # Define the development environment -ENV_FILE=${SCRIPT_DIR}/environment/development.`hostname` +ENV_FILE=${SCRIPT_DIR}/development.`hostname` if [[ ! -e ${ENV_FILE} ]]; then echo "$0: ERROR -> Development environment file missing: ${ENV_FILE}" exit 1 @@ -71,6 +70,4 @@ if [[ $? -ne 0 ]]; then exit 1 fi -# Convert SonarQube report from pdf to html - exit 0 diff --git a/internal/scripts/sonarqube/run_sonarqube.sh b/internal/scripts/sonarqube/run_sonarqube.sh index 4b856eab22..3a3669c37a 100755 --- a/internal/scripts/sonarqube/run_sonarqube.sh +++ b/internal/scripts/sonarqube/run_sonarqube.sh @@ -1,37 +1,48 @@ #!/bin/bash # -# Run SonarQube Source Code Analyzer on a specified revision of MET +# Run SonarQube Source Code Analyzer for METplus #======================================================================= # # This run_sonarqube.sh script will check out the specified version -# of MET and run the SonarQube Source Code Analyzer on it. First, +# of METplus and run the SonarQube Source Code Analyzer on it. First, # go to the directory where you would like the SCA output written and # then run: # # git clone https://github.com/dtcenter/METplus -# METplus/internal/scripts/sonarqube/run_sonarqube.sh name +# METplus/sonarqube/run_sonarqube.sh name # # Usage: run_sonarqube.sh name -# Test the specified branched version of MET: +# Test the specified branched version of METplus: # run_sonarqube.sh {branch name} -# Test the specified tagged version of MET: +# Test the specified tagged version of METplus: # run_sonarqube.sh {tag name} # #======================================================================= # Constants -GIT_REPO="https://github.com/dtcenter/METplus" +GIT_REPO_NAME=METplus +GIT_REPO="https://github.com/dtcenter/${GIT_REPO_NAME}" function usage { - echo - echo "USAGE: $(basename $0) name" - echo " where \"name\" specifies a branch, tag, or hash." - echo + echo + echo "USAGE: $(basename $0) name" + echo " where \"name\" specifies a branch, tag, or hash." + echo } # Check for arguments if [[ $# -lt 1 ]]; then usage; exit; fi +# Check that SONAR_TOKEN and SONAR_HOST_URL are defined +if [ -z ${SONAR_TOKEN} ]; then + echo "ERROR: SONAR_TOKEN must be set" + exit 1 +fi +if [ -z ${SONAR_HOST_URL} ]; then + echo "ERROR: SONAR_HOST_URL must be set" + exit 1 +fi + # Check that SONARQUBE_WRAPPER_BIN is defined if [ -z ${SONARQUBE_WRAPPER_BIN} ]; then which build-wrapper-linux-x86-64 2> /dev/null @@ -86,13 +97,12 @@ function run_command() { return ${STATUS} } - # Store the full path to the scripts directory SCRIPT_DIR=`dirname $0` if [[ ${0:0:1} != "/" ]]; then SCRIPT_DIR=$(pwd)/${SCRIPT_DIR}; fi # Clone repo into a sub-directory and checkout the requested version -REPO_DIR="METplus-${1}" +REPO_DIR="${GIT_REPO_NAME}-${1}" if [ -e ${REPO_DIR} ]; then run_command "rm -rf ${REPO_DIR}" @@ -101,14 +111,21 @@ run_command "git clone ${GIT_REPO} ${REPO_DIR}" run_command "cd ${REPO_DIR}" run_command "git checkout ${1}" +# Define the version string +SONAR_PROJECT_VERSION=$(cat metplus/VERSION) + SONAR_PROPERTIES=sonar-project.properties -# Copy sonar-project.properties for Python code +# Configure the sonar-project.properties [ -e $SONAR_PROPERTIES ] && rm $SONAR_PROPERTIES -cp -p $SCRIPT_DIR/sonar-project.properties $SONAR_PROPERTIES +sed -e "s|SONAR_PROJECT_KEY|METplus_NB|" \ + -e "s|SONAR_PROJECT_NAME|METplus Nightly Build|" \ + -e "s|SONAR_PROJECT_VERSION|$SONAR_PROJECT_VERSION|" \ + -e "s|SONAR_HOST_URL|$SONAR_HOST_URL|" \ + -e "s|SONAR_TOKEN|$SONAR_TOKEN|" \ + -e "s|SONAR_BRANCH_NAME|${1}|" \ + $SCRIPT_DIR/$SONAR_PROPERTIES > $SONAR_PROPERTIES # Run SonarQube scan for Python code run_command "${SONARQUBE_SCANNER_BIN}/sonar-scanner" -# Run SonarQube report generator to make a PDF file -#TODAY=`date +%Y%m%d` diff --git a/internal/scripts/sonarqube/sonar-project.properties b/internal/scripts/sonarqube/sonar-project.properties index 677c640bcf..d1ed593a85 100644 --- a/internal/scripts/sonarqube/sonar-project.properties +++ b/internal/scripts/sonarqube/sonar-project.properties @@ -1,17 +1,12 @@ -sonar.projectKey=org.sonarqube:METplus_NB -sonar.projectName=METplus Nightly Build -sonar.projectVersion=1.0 - +# Project and source code settings +sonar.projectKey=SONAR_PROJECT_KEY +sonar.projectName=SONAR_PROJECT_NAME +sonar.projectVersion=SONAR_PROJECT_VERSION +sonar.branch.name=SONAR_BRANCH_NAME sonar.sources=docs,internal,manage_externals,metplus,parm,produtil,ush - -# The build-wrapper output dir - -# Encoding of the source files +sonar.coverage.exclusions=internal/tests/** sonar.sourceEncoding=UTF-8 -#----- Default SonarQube server -#sonar.host.url=http://localhost:9000 -sonar.host.url=http://mandan:9000 - -sonar.login=met -sonar.password=met@sonar.ucar +# SonarQube server +sonar.host.url=SONAR_HOST_URL +sonar.token=SONAR_TOKEN