Feature 1926 gridstat openmp #1977

mo-mglover · 2021-11-19T14:18:55Z

Pull Request Testing

Describe testing already performed for these changes:

Scaling behaviour plots: given in slides linked of issue Enhance Grid-Stat to use OpenMP for efficient computation of neighborhood statistics by setting $OMP_NUM_THREADS #1926. Differencing of output stat files did not show any differences, and none are expected. If differences - particularly transient differences - are observed, this would be symptomatic of a race condition (a bug).
Recommend testing for the reviewer(s) to perform, including the location of input datasets, and any additional instructions:

OpenMP activated by setting MET_OPENMP=true, or equivalently to some random string. (Similar to MET_DEVELOPMENT in that regard.) At runtime, the user sets environment variable OMP_NUM_THREADS=<# threads>. If OMP_NUM_THREADS is left unset, then the behaviour is platform-dependent; it will likely default to the number of logical cores on the system (not 1).
Do these changes include sufficient documentation updates, ensuring that no errors or warnings exist in the build of the documentation? [Yes or No]
Do these changes include sufficient testing updates? [Yes or No]
Will this PR result in changes to the test suite? [Yes or No]

If yes, describe the new output and/or changes to the existing output:
Please complete this pull request review by [Fill in date].

Pull Request Checklist

See the METplus Workflow for details.

Complete the PR definition above.
Ensure the PR title matches the feature or bugfix branch name.
Define the PR metadata, as permissions allow.
Select: Reviewer(s)
Select: Organization level software support Project or Repository level development cycle Project
Select: Milestone as the version that will include these changes
After submitting the PR, select Linked issues with the original issue number.
After the PR is approved, merge your changes. If permissions do not allow this, request that the reviewer do the merge.
Close the linked issue and delete your feature or bugfix branch from GitHub.

mo-mglover · 2021-11-19T14:38:12Z

Opened the pull request anticipating discussion and a number of iterations.

I'm aware of a couple of areas that will need tweaking:

Documentation
Automatic tests

I've found the test suite config. files, but don't know how to modify them appropriately. Essentially, we would want to run the same tests on 1 and 2 threads, and check that the results are identical.

There's a further potential issue around the configure.ac file and how best to add in the relevant OpenMP compiler flag. While there does seem to be some movement towards standardising around -fopenmp, this is not true in general today. GNU: -fopenmp; Intel: -qopenmp; Cray: -homp.

JohnHalleyGotway · 2021-11-22T15:49:02Z

Thanks for submitting this PR Maff. FYI, I did the following:

assigned to myself for review
set Projects = MET-10.1.0-beta5
set Milestone = MET 10.1.0
set Linked issues = Enhance Grid-Stat to use OpenMP for efficient computation of neighborhood statistics by setting $OMP_NUM_THREADS #1926

I'm starting by running a regression test to compare feature_1926_gridstat_openmp to the develop branch. Assuming that flags no diffs, then I'll try compiling WITH openmp enabled and confirm that the output remains unchanged.

I agree with the additional work you listed in this comment.

JohnHalleyGotway · 2021-11-22T15:55:02Z

met/configure.ac

-AM_COND_IF([ENABLE_DEVELOPMENT], [: ${CXXFLAGS="-g"}  ], [: ${CXXFLAGS=""}] )
-AM_COND_IF([ENABLE_DEVELOPMENT], [: ${FFLAGS="-g -O2"}], [: ${FFLAGS="-O2"}])
+AM_COND_IF([ENABLE_DEVELOPMENT], [: ${CFLAGS_OPT="-g -O2"}], [: ${CFLAGS_OPT="-O2"}])
+AM_COND_IF([ENABLE_DEVELOPMENT], [: ${CXXFLAGS_OPT="-g"}  ], [: ${CXXFLAGS_OPT="-O3"}] )


Was this change intentional... changing ${CXXFLAGS=""} to ${CXXFLAGS_OPT="-03"}?
I believe that when in "development" mode, we want to disable optimization for easier debugging.

Good spot! No, it wasn't intentional. Will put that right and push up the correction.

…MENT.

met/src/tools/core/grid_stat/grid_stat.cc

JohnHalleyGotway · 2021-11-22T16:06:12Z

@jprestop I added you as a reviewer of this PR as well since it impacts how the MET is compiled.
Note from Maff's comment above the OpenMP compiler flag to used is compiler-dependent:
GNU: -fopenmp; Intel: -qopenmp; Cray: -homp
Maff's current changes in configure.ac use -fopenmp.

You'll likely want to update the MET compilation script to include:
ENABLE_OPENMP and CFLAGS_OMP/CXXFLAGS_OMP/FFLAGS_OMP

JohnHalleyGotway · 2021-11-22T20:33:57Z

The unit_gen_vx_mask.xml tests failed for feature_1926_gridstat_openmp:

ERROR  : 
ERROR  : Out of memory reading GRIB2 data!  Exiting!
ERROR  : Check that MET and the GRIB2C library were compiled consistently, either with or without the -D__64BIT__ flag.
ERROR  : 
ERROR: /d1/projects/MET/MET_pull_requests/met-10.1.0/met-10.1.0_beta5/feature_1926/MET-feature_1926_gridstat_openmp/test/perl/unit.pl unit_gen_vx_mask.xml failed.
ERROR: Command returned with non-zero status (1): test/bin/unit_test.sh

The failure is caused by differences in the compiler flags:

egrep "CFLAGS=|CXXFLAGS=|FFLAGS=" MET-develop/met/config.log
CFLAGS='-g -DUNDERSCORE -fPIC -D__64BIT__'
CXXFLAGS='-g -DUNDERSCORE -fPIC -D__64BIT__'
FFLAGS='-g -O2'

versus the feature branch:

egrep "CFLAGS=|CXXFLAGS=|FFLAGS=" MET-feature_1926_gridstat_openmp/met/config.log
CFLAGS='-g -O2 '
CXXFLAGS='-g '
FFLAGS='-g -O2 '

The CFLAGS env var was set when configure was called:

ac_cv_env_CFLAGS_set=set
ac_cv_env_CFLAGS_value='-g -DUNDERSCORE -fPIC -D__64BIT__'

However configure.ac is overriding the input values starting at this line:

MET/met/configure.ac

Line 1079 in 3ff12fe

CFLAGS="$CFLAGS_OPT $CFLAGS_OMP"

Trouble is I don't fully understand the automake syntax used on this line:

MET/met/configure.ac

Line 1070 in 4e28ae7

AM_COND_IF([ENABLE_DEVELOPMENT], [: ${CFLAGS="-g -O2"}], [: ${CFLAGS="-O2"}])

Perhaps that only sets CFLAGS is they are not already set?

I recommend we handle the openmp options similar to how the GRIB2 and Python options are handled. Consistent with other optional compilation options, I recommend we use an environment variable named "MET_OPENMP", which if set, triggers ENABLE_OPENMP to be set in configure.ac.

I'll make some changes to configure.ac directly in the feature_1926_gridstat_openmp branch. If it breaks everything, we can always back them out.

…_DEVELOPMENT to being like GRIB2, with --enable-grib2."

…laceholder for --enable-openmp so that running make test works.

…_OPENMP for the moment.

mo-mglover · 2021-11-26T17:17:52Z

Modified system for setting -fopenmp is much better than the way I'd done it. I've cherry-picked that and merged into this branch. Also fixed the linking problem (-fopenmp is also needed by the linker); that takes away the need to add -lgomp by hand for the GNU compiler.

Co-authored-by: johnhg <johnhg@ucar.edu>

jprestop · 2021-11-30T22:54:05Z

@JohnHalleyGotway
Is there anything I need to test before this PR is approved? I see that you stated in an earlier comment:

I'm starting by running a regression test to compare feature_1926_gridstat_openmp to the develop branch. Assuming that flags no diffs, then I'll try compiling WITH openmp enabled and confirm that the output remains unchanged.

So I wasn't sure if you needed me to do any further testing or not.

I also wanted to make sure that I understand what I need to do with the compile script. It looks like an --enable-openmp will need to be added in the same manner as we either compile or don’t compile with --enable-grib2, but do I also need to add the following compilation flags for:
GNU: -fopenmp;
Intel: -qopenmp;
Cray: -homp
or is that handled in the autoconf setup if --enable-openmp is selected?

JohnHalleyGotway · 2021-12-02T21:35:32Z

@mo-mglover when $OMP_NUM_THREADS is not set, grid_stat runs on my Mac using 12 threads. I presume it's using all available threads by default. But I don't know how that is determined. I was expecting it to run on a single thread by default. The latter default seems a little "safer" so that leaving $OMP_NUM_THREADS unset yields the exact same behavior as previous versions.

But is there some standard here? How do you think this should be handled?

…d produce the same result as the previous test named grid_stat_GRIB2_NAM_RTMA.

…s seems like a pretty obvious typo that was causing a warning message to be written to the log file. While this will change the unit test results, I'm confident it was the original intention of this Grid-Stat configuration.

…calling the AC_OPENMP() macro. And that macro compile WITH openmp support by default. It can be disabled using --disable-openmp.

JohnHalleyGotway · 2021-12-08T20:31:38Z

met/src/basic/vx_util/handle_openmp.cc

+#pragma omp single
+  {
+     mlog << Debug(2) << "OpenMP running on " 
+       << omp_get_num_threads() << " thread(s).\n";


Suggested change

<< omp_get_num_threads() << " thread(s).\n";

<< omp_get_num_threads() << " thread(s).\n";

Just lining code up.

met/src/basic/vx_util/Makefile.am

JohnHalleyGotway · 2021-12-08T20:58:24Z

@mo-mglover, adding init_openmp() to the vx_util library is an excellent idea! Thanks for doing that. That'll make it much easier to add openmp support to other applications.

As for log message levels, I do think that DEBUG(2) is appropriate. For users running via the METplus wrappers, we can make sure OMP_NUM_THREADS is set there.

I had considered whether we'd want to provide a command line option to override the OMP_NUM_THREADS environment variable. But when running MET on the same platform, you'd probably want OMP_NUM_THREADS set to the same value for each call. If a need arises if the future, we could always add that later.

@jprestop is testing the compilation of the feature branch with Intel compilers to confirm that the autoconf settings pick the right compilation flags.

jprestop · 2021-12-09T00:02:51Z

@JohnHalleyGotway

I was able to get a successful build of the MET feature_1926_gridstat_openmp branch on kiowa at /d1/projects/MET/MET_pull_requests/met-10.1.0/met-10.1.0_beta5/feature_1926_intel, however, "-qopenmp" was NOT used but rather "-fopenmp" was used:

checking for /usr/local/intel-18.0.3/bin/icc option to support OpenMP... -fopenmp

I sent the output to build.log in that directory and can see that -fopenmp was used throughout. It looks like we need to revisit this issue.

jprestop · 2021-12-09T21:24:01Z

@mo-mglover

@JohnHalleyGotway and I reviewed the compilation of openmpi with the Intel compilers and reviewed the flag selection for PGI.

For Intel, it seems that -fopenmp is the same as using -qopenmp. Running "icc --help" shows:

-qopenmp  enable the compiler to generate multi-threaded code based on the
          OpenMP* directives (same as -fopenmp)

from the config.log file for compiling MET we see that -fopenmp was chosen:

checking for /usr/local/intel-18.0.3/bin/icc option to support OpenMP... -fopenmp

For PGI, it looks as if -mp was the chosen flag, which is expected. In config.log:

OPENMP_CFLAGS='-mp'

Based on these tests, we confirm that the autoconf settings pick the right compilation flags for the GNU, Intel, and PGI compilers.

@JohnHalleyGotway and @jprestop Need to add documentation on setting the OMP_NUM_THREADS runtime environment variable.

JohnHalleyGotway · 2021-12-13T18:13:02Z

Feedback in the NOAA User's Telecon on 12/13/2021 recommended that we:

Leave the Debug(2) message level in place but make it a bit longer:

DEBUG 2: OMP_NUM_THREADS is not set. Defaulting to 1 thread. Recommend setting OMP_NUM_THREADS for faster runtimes.

Add a new section to the MET User's Guide about the use of OpenMP, describing OMP_NUM_THREADS and listing the portions of the code that are parallelized.

Add a new 4.1 section for "Runtime Environment Variables" and create 4.1.1, 4.1.2, and 4.1.3 to discuss MET_OBS_ERROR_TABLE, MET_GRIB_TABLES, and OMP_NUM_THREADS.

@jprestop will make these changes and @JohnHalleyGotway will review/update.

Co-authored-by: johnhg <johnhg@ucar.edu>

…e info.

mo-mglover · 2021-12-16T19:41:47Z

@JohnHalleyGotway I've tackled (1) and (2), and pushed up the changes, but I've haven't been able to "compile" the RST. (I'm getting errors that seem to be related to the sphinx setup at my end, rather than the RST itself.)

I wasn't sure quite what was involved with the last paragraph. I've just noticed that @jprestop is tagged on that comment.

JohnHalleyGotway · 2021-12-16T19:51:04Z

Excellent @mo-mglover! Thanks for making these changes. @jprestop is the lead on documentation across all the METplus components. In the MET development meeting today, we discussed how/where to incorporate the OpenMP details into the User's Guide. So the next step is @jprestop integrating the details you provided in openmp.rst. After that, we'll be ready to merge this PR. Thanks for your work on this and patience.

…nfo from openmp.rst

…Runtime Environment Variables section in config_options.rst and removed openmp from index.rst

jprestop · 2021-12-22T00:14:32Z

Thank you @mo-mglover for all of your work on this and for writing up the documentation as well. It is very much appreciated. @JohnHalleyGotway, I added the documentation to the config_options.rst file in a new section called "Runtime Environment Variables" as we discussed. There are two things to note:

In addition to MET_OBS_ERROR_TABLE, MET_GRIB_TABLES, and OMP_NUM_THREADS, there was also MET_BASE, so that is listed in 4.1.1. Please let me know if that doesn't belong there.
Unfortunately, the Sphinx documentation for MET does not currently follow the standard (see below). As a result, the sections under OMP_NUM_THREADS "Introduction", "Setting the number of threads", "Which code is parallelized?", and "Thread Binding" are simply bolded. This work is for another time, however. I created a new issue for this task.

https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html
# with overline, for parts
* with overline, for chapters
=, for sections
-, for subsections
^, for subsubsections
“, for paragraphs

I believe everything is in order for your final review, @JohnHalleyGotway.

jprestop

I have added the documentation to the config_options.rst file in a new section called "Runtime Environment Variables" as we discussed. In addition to MET_OBS_ERROR_TABLE, MET_GRIB_TABLES, and OMP_NUM_THREADS, there was also MET_BASE, so that is listed in 4.1.1. Please let me know if that doesn't belong there. @JohnHalleyGotway I believe everything is in order for your final review.

JohnHalleyGotway

I approve of these changes.

On 1/12/22, I merged the latest changes from the develop branch into this feature branch and reran the regression test. Wanted to make sure there were no conflicts with recent development. The test flagged several differences but all of them are related to other development for which we have not yet updated the reference version.

Details can be found in kiowa:/d1/projects/MET/MET_pull_requests/met-10.1.0/met-10.1.0_beta5/feature_1926

egrep -i "file1:|file2:|ERROR:" test_regression.log | egrep -B 2 ERROR:

These files are added by this PR, generated using 2 processors.

ERROR: folder MET-develop/test_output missing 13 files
    grid_stat/grid_stat_GRIB2_NAM_RTMA_NP2_120000L_20120409_120000V_cnt.txt 
    grid_stat/grid_stat_GRIB2_NAM_RTMA_NP2_120000L_20120409_120000V_ctc.txt 
    grid_stat/grid_stat_GRIB2_NAM_RTMA_NP2_120000L_20120409_120000V_cts.txt 
    grid_stat/grid_stat_GRIB2_NAM_RTMA_NP2_120000L_20120409_120000V_eclv.txt 
    grid_stat/grid_stat_GRIB2_NAM_RTMA_NP2_120000L_20120409_120000V_fho.txt 
    grid_stat/grid_stat_GRIB2_NAM_RTMA_NP2_120000L_20120409_120000V_nbrcnt.txt 
    grid_stat/grid_stat_GRIB2_NAM_RTMA_NP2_120000L_20120409_120000V_nbrctc.txt 
    grid_stat/grid_stat_GRIB2_NAM_RTMA_NP2_120000L_20120409_120000V_nbrcts.txt 
    grid_stat/grid_stat_GRIB2_NAM_RTMA_NP2_120000L_20120409_120000V_pairs.nc 
    grid_stat/grid_stat_GRIB2_NAM_RTMA_NP2_120000L_20120409_120000V_sl1l2.txt 
    grid_stat/grid_stat_GRIB2_NAM_RTMA_NP2_120000L_20120409_120000V.stat 
    grid_stat/grid_stat_GRIB2_NAM_RTMA_NP2_120000L_20120409_120000V_vcnt.txt 
    grid_stat/grid_stat_GRIB2_NAM_RTMA_NP2_120000L_20120409_120000V_vl1l2.txt

All of these many, many changes are caused by a small change in this PR that @JohnHalleyGotway made, switching "min 1" to "min 3". That causes INTERP_MTHD = MIN_SQAURE with INTERP_PNTS = 9 to appear in the output, as expected.

file1: MET-feature_1926_gridstat_openmp/test_output/grid_stat/grid_stat_GRIB2_NAM_RTMA_120000L_20120409_120000V_cnt.txt
file2: MET-develop/test_output/grid_stat/grid_stat_GRIB2_NAM_RTMA_120000L_20120409_120000V_cnt.txt
ERROR: header information mismatch in column INTERP_MTHD
ERROR: failed tests for CNT: hdr
file1: MET-feature_1926_gridstat_openmp/test_output/grid_stat/grid_stat_GRIB2_NAM_RTMA_120000L_20120409_120000V_ctc.txt
file2: MET-develop/test_output/grid_stat/grid_stat_GRIB2_NAM_RTMA_120000L_20120409_120000V_ctc.txt
ERROR: header information mismatch in column INTERP_MTHD
ERROR: failed tests for CTC: hdr
file1: MET-feature_1926_gridstat_openmp/test_output/grid_stat/grid_stat_GRIB2_NAM_RTMA_120000L_20120409_120000V_cts.txt
file2: MET-develop/test_output/grid_stat/grid_stat_GRIB2_NAM_RTMA_120000L_20120409_120000V_cts.txt
ERROR: header information mismatch in column INTERP_MTHD
ERROR: failed tests for CTS: hdr
file1: MET-feature_1926_gridstat_openmp/test_output/grid_stat/grid_stat_GRIB2_NAM_RTMA_120000L_20120409_120000V_eclv.txt
file2: MET-develop/test_output/grid_stat/grid_stat_GRIB2_NAM_RTMA_120000L_20120409_120000V_eclv.txt
ERROR: header information mismatch in column INTERP_MTHD
ERROR: failed tests for ECLV#66: hdr
file1: MET-feature_1926_gridstat_openmp/test_output/grid_stat/grid_stat_GRIB2_NAM_RTMA_120000L_20120409_120000V_fho.txt
file2: MET-develop/test_output/grid_stat/grid_stat_GRIB2_NAM_RTMA_120000L_20120409_120000V_fho.txt
ERROR: header information mismatch in column INTERP_MTHD
ERROR: failed tests for FHO: hdr
--
file1: MET-feature_1926_gridstat_openmp/test_output/grid_stat/grid_stat_GRIB2_NAM_RTMA_120000L_20120409_120000V_pairs.nc
file2: MET-develop/test_output/grid_stat/grid_stat_GRIB2_NAM_RTMA_120000L_20120409_120000V_pairs.nc
ERROR: NetCDF headers differ:
file1: MET-feature_1926_gridstat_openmp/test_output/grid_stat/grid_stat_GRIB2_NAM_RTMA_120000L_20120409_120000V_sl1l2.txt
file2: MET-develop/test_output/grid_stat/grid_stat_GRIB2_NAM_RTMA_120000L_20120409_120000V_sl1l2.txt
ERROR: header information mismatch in column INTERP_MTHD
ERROR: failed tests for SL1L2: hdr
file1: MET-feature_1926_gridstat_openmp/test_output/grid_stat/grid_stat_GRIB2_NAM_RTMA_120000L_20120409_120000V.stat
file2: MET-develop/test_output/grid_stat/grid_stat_GRIB2_NAM_RTMA_120000L_20120409_120000V.stat
ERROR: header information mismatch in column INTERP_MTHD
ERROR: header information mismatch in column INTERP_MTHD
ERROR: header information mismatch in column INTERP_MTHD
ERROR: header information mismatch in column INTERP_MTHD
ERROR: header information mismatch in column INTERP_MTHD
ERROR: header information mismatch in column INTERP_MTHD
ERROR: header information mismatch in column INTERP_MTHD
ERROR: header information mismatch in column INTERP_MTHD
ERROR: failed tests for CNT: hdr
ERROR: failed tests for CTC: hdr
ERROR: failed tests for CTS: hdr
ERROR: failed tests for ECLV#66: hdr
ERROR: failed tests for FHO: hdr
ERROR: failed tests for SL1L2: hdr
ERROR: failed tests for VCNT: hdr
ERROR: failed tests for VL1L2: hdr
file1: MET-feature_1926_gridstat_openmp/test_output/grid_stat/grid_stat_GRIB2_NAM_RTMA_120000L_20120409_120000V_vcnt.txt
file2: MET-develop/test_output/grid_stat/grid_stat_GRIB2_NAM_RTMA_120000L_20120409_120000V_vcnt.txt
ERROR: header information mismatch in column INTERP_MTHD
ERROR: failed tests for VCNT: hdr
file1: MET-feature_1926_gridstat_openmp/test_output/grid_stat/grid_stat_GRIB2_NAM_RTMA_120000L_20120409_120000V_vl1l2.txt
file2: MET-develop/test_output/grid_stat/grid_stat_GRIB2_NAM_RTMA_120000L_20120409_120000V_vl1l2.txt
ERROR: header information mismatch in column INTERP_MTHD
ERROR: failed tests for VL1L2: hdr

These files are added/modified/fixed by MET#1810

ERROR: folder MET-feature_1926_gridstat_openmp/test_output missing 2 files
    tc_gen/tc_gen_2021_shape_pct.txt 
    tc_gen/tc_gen_2021_shape.stat
---
file1: MET-feature_1926_gridstat_openmp/test_output/gen_vx_mask/SHAPE_mask.nc
file2: MET-develop/test_output/gen_vx_mask/SHAPE_mask.nc
ERROR: found    21 differences in var shape_mask                             - max abs: 1 
---
file1: MET-feature_1926_gridstat_openmp/test_output/tc_gen/tc_gen_prob_pjc.txt
file2: MET-develop/test_output/tc_gen/tc_gen_prob_pjc.txt
ERROR: line type PCT#39 not found in stat2
ERROR: line type PJC#55 not found in stat1
ERROR: failed tests for PCT#39: lty
ERROR: failed tests for PJC#55: lty
file1: MET-feature_1926_gridstat_openmp/test_output/tc_gen/tc_gen_prob_prc.txt
file2: MET-develop/test_output/tc_gen/tc_gen_prob_prc.txt
ERROR: line type PCT#39 not found in stat2
ERROR: line type PRC#39 not found in stat1
ERROR: failed tests for PCT#39: lty
ERROR: failed tests for PRC#39: lty
--
file1: MET-feature_1926_gridstat_openmp/test_output/tc_gen/tc_gen_prob.stat
file2: MET-develop/test_output/tc_gen/tc_gen_prob.stat
ERROR: line type PJC#55 not found in stat1
ERROR: line type PRC#39 not found in stat1
ERROR: differing number of rows 9 vs. 3 for row type PCT#39 between versions 10_1 vs. 10_1 
ERROR: failed tests for PJC#55: lty
ERROR: failed tests for PRC#39: lty
ERROR: failed tests for PCT#39: nrow

#2016) Co-authored-by: Julie Prestopnik <jpresto@seneca.rap.ucar.edu> Co-authored-by: johnhg <johnhg@ucar.edu> Co-authored-by: Seth Linden <linden@kiowa.rap.ucar.edu> Co-authored-by: John Halley Gotway <johnhg@ucar.edu> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: Howard Soh <hsoh@kiowa.rap.ucar.edu> Co-authored-by: John Halley Gotway <johnhg@kiowa.rap.ucar.edu> Co-authored-by: jprestop <jpresto@ucar.edu> Co-authored-by: Seth Linden <linden@ucar.edu> Co-authored-by: hsoh-u <hsoh@ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> Co-authored-by: John Halley Gotway <johnhg@seneca.rap.ucar.edu> Co-authored-by: MET Tools Test Account <met_test@seneca.rap.ucar.edu> Co-authored-by: mo-mglover <78152252+mo-mglover@users.noreply.github.com>

* Per #1906, modify code to pass entire path to make_temp_file_name instead of only the filename so that the function can tell whether or not the file exists. (#1952) Co-authored-by: Julie Prestopnik <jpresto@seneca.rap.ucar.edu> * Feature 1761 percent thresh (#1956) * Per issue #1761 in set_perc() adding code to get FBIAS numeric value, like 1.0 or 0.9, etc. SL * Per issue #1761: in set_perc(), modified actual percentile calculation at end to use the extracted FBIAS numeric value (float). SL * Per issue #1761: modified the check on the perc_thresh_freq_bias, just has to be > 0 now. SL * Per issue #1761: cleaned up code in set_perc(). SL * Per #1761, updates to Simple_Node::set_perc() to handle variable frequency bias amounts. Changes include: - Reverting the formatting of this back to how it originally was in the develop branch. In general, just match the formatting of the existing file, so as the minimize the number of difference lines. - Add logic to adjust the percentile to be found based on the requested FBIAS value. Multiplying or dividing the percentile by the FBIAS value depends on the inequality type and whether we're bias adjusting the forecast or observation data. - Adjust the log messages slightly. Please be aware that I'm not totally confident in these changes. They warrant much more testing. This logic is very, very confusing. * Per #1761, call compute_percentile() when double-checking the percentile values. * Per #1761, remove unused variable. * Per #1761, add warning for percentiles > 100. * Per #1761. In set_perc(), after testing cleaned up code. SL * Per issue #1761: adding new config file for testing dynamic FBIAS values. SL * Per issue #1761: added new unit test for dynamic FBIAS values when running grid_stat. SL * Per issue #1761, modified FBIAS section to indicated that the user can use dynamic values that are not 1.0. SL * Update met/docs/Users_Guide/config_options.rst Co-authored-by: johnhg <johnhg@ucar.edu> * Update met/docs/Users_Guide/config_options.rst Co-authored-by: johnhg <johnhg@ucar.edu> * Update met/docs/Users_Guide/config_options.rst Co-authored-by: johnhg <johnhg@ucar.edu> * Update test/config/GridStatConfig_fbias_perc_thresh Co-authored-by: johnhg <johnhg@ucar.edu> * Update test/config/GridStatConfig_fbias_perc_thresh Co-authored-by: johnhg <johnhg@ucar.edu> * Update test/config/GridStatConfig_fbias_perc_thresh Co-authored-by: johnhg <johnhg@ucar.edu> * Update test/config/GridStatConfig_fbias_perc_thresh Co-authored-by: johnhg <johnhg@ucar.edu> * Per issue #1761, set nc_pairs_flag = FALSE. SL Co-authored-by: Seth Linden <linden@kiowa.rap.ucar.edu> Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * Feature 1905 ens_ctrl (#1955) Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> * Hotfix after merging changing for #1761. Updating the list of expected output files. * Per #1905, committing a hotfix directly to the develop branch. Reverting the logic for computing the ensemble range back to what it was previously. The new version produced very slight differences in the 6-th or 7-th decimal place when compared to previous results. There's not good reason for these changes which were caused by the order of operations in casting from doubles to floats. Reverting back to the old logic prevents diffs for anyone else downstream and is the prudent choice. * Feature 1957 ascii2nc_python (#1958) * Feature 1949 cf netcdf documentaton (#1951) * #1949 Added CF compliant NetCDF into data IO * #1949 Added commas * #1948 Some corrections for typo and added the links for CF attributes * #1948 Added Performance with NetCDF input data * #1949 Corrected tyoe and applied Juloie's suggestions Co-authored-by: Howard Soh <hsoh@kiowa.rap.ucar.edu> * Feature 1968 ens_ctrl (#1969) * Feature 1809 gen prob (#1967) * Feature 1970 v10.1.0-beta4 (#1973) * Fix tiny typo in plot_data_plane usage statement. * Hotfix for the develop branch to get met-10.1.0-beta4 compiling on WCOSS. The intel compiler does not allow adjacent >> characters when defining maps. * Updating the beta4 release date from 11/16 to 11/17 since the beta4 release needs to be recreated. * Update the MET PR template to list expected changes up front. * Bugfix 1976 develop cdist (#1979) * #1936 Excludes precip3hr, precip6hr, precip12hr, and precip10min from required variables for MESONET * Per #1985, correcting typo found while creating V10.0.1 in the develop branch as well. * Per #1985, VERY minor change to fix alignment of titles in Point-Stat... updating the develop branch. * #1936 Added an unit test for newer MESONET file * Feature 1989 sort (#1990) * Feature 1991 VCNT (#1992) * Feature 1993 grid_mask (#1994) * #1844 Added vx_pointdata_python * #1844 Added vx_pointdata_python to PYTHON_LIBS & make file for vx_pointdata_python * #1844 Added vx_pointdata_python to SUBDIRS * #1844 Initial release * #1844 Added met_point_data.cc & met_point_data.h * #1844 NcHeaderData is renamed to MetPointHeader and moved to met_point_dataq.h * #1844 NcPointObsData is derived from MetPointObsData. Many methods are moved to the base class MetPointObsData * #1844 Moved varianbles and methods to the base class MetPointData * #1844 override the obs_data pointer to the derived class * #1844 Added pyobject_as_bool & pyobject_as_string_array * #1844 Cleanup include statements and addpointdata_python.h if python is enabled * #18443 Support python embedding * #1844 Initial release * #1844 Make buf_size const * Add GitHub Actions workflow to trigger METplus testing workflow (#2002) * changed names of inputs to send to METplus to match changes to METplus repo * #1965 Excludes the duplicated variable names from multiple input files * #1965 Processed with the raw variable names and update them when the output is saved. * #1965 Added unit test ioda2nc_same_input * #1965 Processed with the raw variable names and update them when the output is saved. * #1965 Added unit test ioda2nc_same_input * #1965 initialize var_idx * Update the development environment for seneca to add /usr/local/nco/bin and /usr/local/netcdf/bin to the path. Also define MET_TEST_RSCRIPT to point to a new enough version of RSCRIPT that has the ncdf4 package 1.17 or later. * send email address of user who triggered event to METplus workflow as input * Fixup the Rscript diffing logic to support running the nightly build on seneca. The problem here is that the ncdiff -x -v command line options that worked in version 4.7.0 NO LONGER work in 4.9.2. Added a check to skip over the time_bounds variable when processing the ncdiff output. * No real changes here. Just formatting * Feature 1974 message_type_group_map (#1999) Co-authored-by: Howard Soh <hsoh@kiowa.rap.ucar.edu> Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * Feature 1996 uninitialized variables (#2004) Co-authored-by: Howard Soh <hsoh@kiowa.rap.ucar.edu> Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * Feature 1695 ensemble single file (gen_ens_prod) (#2001) Co-authored-by: johnhg <johnhg@ucar.edu> * Feature 1965 NB faile with time summary by ioda2nc (#2008) Co-authored-by: Howard Soh <hsoh@kiowa.rap.ucar.edu> * Feature 1810 shapes (#2005) * #1844 Added log message for use_var_id * #1844 Get use_var_id for python embedding * #1844 Initial release * #1844 Added met_point_obs.py and read_met_point_obs.py * #1844 Added 4 unit test for python embedding of MET point data * More changes to test_util.R to account for differences between 4.7.0 and 4.9.2 of ncdiff. The earlier version reported 0 diffs for the time_bounds, lat, lon, latitude, and longitude variables while the newer version reports their raw values. Simply ignore these variables for now. This isn't a great solution but when we reimplement this testing logic in python, we can address this issue then. * #1844 Added python embedding for point observation data * Feature 1926 gridstat openmp (#1977) Co-authored-by: johnhg <johnhg@ucar.edu> Co-authored-by: John Halley Gotway <johnhg@ucar.edu> Co-authored-by: Julie Prestopnik <jpresto@seneca.rap.ucar.edu> * feature 1695 ensemble single file (ensemble_stat) (#2007) Co-authored-by: johnhg <johnhg@ucar.edu> * Committing directly to the develop branch. Removing many, many instances of MET_BUILD_BASE and a couple instances of MET_BASE from the examples listed in Appendix A. The inconsistent use of these variables will only cause confusion. Removing them will help avoid that confusion. Big picture... MET_BASE defines the installed 'share/met' directory. It can be used to reference the location of map or polyline files. MET_BUILD_BASE is only used in the context of the test scripts. There, it's defined as the top-level source code directory in which the code was compiled. Its used to reference the location of sample data files or Rscript included in the tarball. I left a couple references to these variables in Appendix A where thier use is not confusing. But I removed all instances which are used to the define the location of the 'bin' directory. Instead, let's just assume the MET binaries are already in their path... or the user knows where to find them. * feature 2011 v10.1.0-beta5 (#2014) * feature 1695 fix issues with ensemble changes (#2012) Co-authored-by: John Halley Gotway <johnhg@ucar.edu> Co-authored-by: John Halley Gotway <johnhg@kiowa.rap.ucar.edu> * Hotfix for #1695 committed directly to the develop branch. The new Ensemble-Stat config files added for this issue don't define a random number generator seed. This results in different output for each run and prevents the regression test from producing repeatable results. Defining the seed here to fix that. * Adding George to email list for the nightly build. * Fix the links for the met-10.1.0-beta5 release notes. * #1844 Make met_point_obs as abstract class * #1844 correctedb for loop end condition on processing obs bdata * Feature 1546 unit_tests (#2021) * #2020 Added SonarQube related varibales * #2020 Initial release * #2015 Avoid the repeasted debug message if derived varibales are disabled * #1996 Initialize right and left * #1966 Call clear() instead of reset memory for Header variable * #1966 Make sure the levels from variable does not exceed the maxLevel * #1966 Removed unreachable codes * #1966 Do not write into NetcDF if empty data * #2015 Corrected "retained or derived" count * Feature 1546 CI testing (#2029) Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * fixed broken workflow by setting environment variable expected for docker build script * add missing docker tag from push command * added correct docker tag value * fixed incorrect truth data version for updating -ref reference data * Updating develop with changes to the unit tests names. Renaming unit_test.yml to a more generic testing.yml name since it compiles AND/OR runs unit tests. Also make the job names more concise so its easier to see what's running in the action summary window on GitHub. * fixed directory to copy truth data into -- copy command was copying the content of the directory, not including the directory itself * Per #1907, added warning about switch from Ensemble-Stat to Gen-Ens-Prod (#2032) Co-authored-by: Julie Prestopnik <jpresto@seneca.rap.ucar.edu> * 2028 Check null for header data * Removed un-reachable code * #2028 Check if the pointer (d) is null * #2015 Changed warning to debug message * #2028 Passing two dimensionl cur and dim array * #2028 Set obs_data * Feature 2003 briercl (#2034) Co-authored-by: jprestop <jpresto@ucar.edu> Co-authored-by: Julie Prestopnik <jpresto@seneca.rap.ucar.edu> Co-authored-by: Seth Linden <linden@ucar.edu> Co-authored-by: Seth Linden <linden@kiowa.rap.ucar.edu> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: hsoh-u <hsoh@ucar.edu> Co-authored-by: Howard Soh <hsoh@kiowa.rap.ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> Co-authored-by: John Halley Gotway <johnhg@seneca.rap.ucar.edu> Co-authored-by: MET Tools Test Account <met_test@seneca.rap.ucar.edu> Co-authored-by: mo-mglover <78152252+mo-mglover@users.noreply.github.com> Co-authored-by: John Halley Gotway <johnhg@kiowa.rap.ucar.edu> Co-authored-by: Howard Soh <hsoh@seneca.rap.ucar.edu>

* Per #1906, modify code to pass entire path to make_temp_file_name instead of only the filename so that the function can tell whether or not the file exists. (#1952) Co-authored-by: Julie Prestopnik <jpresto@seneca.rap.ucar.edu> * Feature 1761 percent thresh (#1956) * Per issue #1761 in set_perc() adding code to get FBIAS numeric value, like 1.0 or 0.9, etc. SL * Per issue #1761: in set_perc(), modified actual percentile calculation at end to use the extracted FBIAS numeric value (float). SL * Per issue #1761: modified the check on the perc_thresh_freq_bias, just has to be > 0 now. SL * Per issue #1761: cleaned up code in set_perc(). SL * Per #1761, updates to Simple_Node::set_perc() to handle variable frequency bias amounts. Changes include: - Reverting the formatting of this back to how it originally was in the develop branch. In general, just match the formatting of the existing file, so as the minimize the number of difference lines. - Add logic to adjust the percentile to be found based on the requested FBIAS value. Multiplying or dividing the percentile by the FBIAS value depends on the inequality type and whether we're bias adjusting the forecast or observation data. - Adjust the log messages slightly. Please be aware that I'm not totally confident in these changes. They warrant much more testing. This logic is very, very confusing. * Per #1761, call compute_percentile() when double-checking the percentile values. * Per #1761, remove unused variable. * Per #1761, add warning for percentiles > 100. * Per #1761. In set_perc(), after testing cleaned up code. SL * Per issue #1761: adding new config file for testing dynamic FBIAS values. SL * Per issue #1761: added new unit test for dynamic FBIAS values when running grid_stat. SL * Per issue #1761, modified FBIAS section to indicated that the user can use dynamic values that are not 1.0. SL * Update met/docs/Users_Guide/config_options.rst Co-authored-by: johnhg <johnhg@ucar.edu> * Update met/docs/Users_Guide/config_options.rst Co-authored-by: johnhg <johnhg@ucar.edu> * Update met/docs/Users_Guide/config_options.rst Co-authored-by: johnhg <johnhg@ucar.edu> * Update test/config/GridStatConfig_fbias_perc_thresh Co-authored-by: johnhg <johnhg@ucar.edu> * Update test/config/GridStatConfig_fbias_perc_thresh Co-authored-by: johnhg <johnhg@ucar.edu> * Update test/config/GridStatConfig_fbias_perc_thresh Co-authored-by: johnhg <johnhg@ucar.edu> * Update test/config/GridStatConfig_fbias_perc_thresh Co-authored-by: johnhg <johnhg@ucar.edu> * Per issue #1761, set nc_pairs_flag = FALSE. SL Co-authored-by: Seth Linden <linden@kiowa.rap.ucar.edu> Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * Feature 1905 ens_ctrl (#1955) Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> * Hotfix after merging changing for #1761. Updating the list of expected output files. * Per #1905, committing a hotfix directly to the develop branch. Reverting the logic for computing the ensemble range back to what it was previously. The new version produced very slight differences in the 6-th or 7-th decimal place when compared to previous results. There's not good reason for these changes which were caused by the order of operations in casting from doubles to floats. Reverting back to the old logic prevents diffs for anyone else downstream and is the prudent choice. * Feature 1957 ascii2nc_python (#1958) * Feature 1949 cf netcdf documentaton (#1951) * #1949 Added CF compliant NetCDF into data IO * #1949 Added commas * #1948 Some corrections for typo and added the links for CF attributes * #1948 Added Performance with NetCDF input data * #1949 Corrected tyoe and applied Juloie's suggestions Co-authored-by: Howard Soh <hsoh@kiowa.rap.ucar.edu> * Feature 1968 ens_ctrl (#1969) * Feature 1809 gen prob (#1967) * Feature 1970 v10.1.0-beta4 (#1973) * Fix tiny typo in plot_data_plane usage statement. * Hotfix for the develop branch to get met-10.1.0-beta4 compiling on WCOSS. The intel compiler does not allow adjacent >> characters when defining maps. * Updating the beta4 release date from 11/16 to 11/17 since the beta4 release needs to be recreated. * Update the MET PR template to list expected changes up front. * Bugfix 1976 develop cdist (#1979) * #1936 Excludes precip3hr, precip6hr, precip12hr, and precip10min from required variables for MESONET * Per #1985, correcting typo found while creating V10.0.1 in the develop branch as well. * Per #1985, VERY minor change to fix alignment of titles in Point-Stat... updating the develop branch. * #1936 Added an unit test for newer MESONET file * Feature 1989 sort (#1990) * Feature 1991 VCNT (#1992) * Feature 1993 grid_mask (#1994) * #1844 Added vx_pointdata_python * #1844 Added vx_pointdata_python to PYTHON_LIBS & make file for vx_pointdata_python * #1844 Added vx_pointdata_python to SUBDIRS * #1844 Initial release * #1844 Added met_point_data.cc & met_point_data.h * #1844 NcHeaderData is renamed to MetPointHeader and moved to met_point_dataq.h * #1844 NcPointObsData is derived from MetPointObsData. Many methods are moved to the base class MetPointObsData * #1844 Moved varianbles and methods to the base class MetPointData * #1844 override the obs_data pointer to the derived class * #1844 Added pyobject_as_bool & pyobject_as_string_array * #1844 Cleanup include statements and addpointdata_python.h if python is enabled * #18443 Support python embedding * #1844 Initial release * #1844 Make buf_size const * Add GitHub Actions workflow to trigger METplus testing workflow (#2002) * changed names of inputs to send to METplus to match changes to METplus repo * #1965 Excludes the duplicated variable names from multiple input files * #1965 Processed with the raw variable names and update them when the output is saved. * #1965 Added unit test ioda2nc_same_input * #1965 Processed with the raw variable names and update them when the output is saved. * #1965 Added unit test ioda2nc_same_input * #1965 initialize var_idx * Update the development environment for seneca to add /usr/local/nco/bin and /usr/local/netcdf/bin to the path. Also define MET_TEST_RSCRIPT to point to a new enough version of RSCRIPT that has the ncdf4 package 1.17 or later. * send email address of user who triggered event to METplus workflow as input * Fixup the Rscript diffing logic to support running the nightly build on seneca. The problem here is that the ncdiff -x -v command line options that worked in version 4.7.0 NO LONGER work in 4.9.2. Added a check to skip over the time_bounds variable when processing the ncdiff output. * No real changes here. Just formatting * Feature 1974 message_type_group_map (#1999) Co-authored-by: Howard Soh <hsoh@kiowa.rap.ucar.edu> Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * Feature 1996 uninitialized variables (#2004) Co-authored-by: Howard Soh <hsoh@kiowa.rap.ucar.edu> Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * Feature 1695 ensemble single file (gen_ens_prod) (#2001) Co-authored-by: johnhg <johnhg@ucar.edu> * Feature 1965 NB faile with time summary by ioda2nc (#2008) Co-authored-by: Howard Soh <hsoh@kiowa.rap.ucar.edu> * Feature 1810 shapes (#2005) * #1844 Added log message for use_var_id * #1844 Get use_var_id for python embedding * #1844 Initial release * #1844 Added met_point_obs.py and read_met_point_obs.py * #1844 Added 4 unit test for python embedding of MET point data * More changes to test_util.R to account for differences between 4.7.0 and 4.9.2 of ncdiff. The earlier version reported 0 diffs for the time_bounds, lat, lon, latitude, and longitude variables while the newer version reports their raw values. Simply ignore these variables for now. This isn't a great solution but when we reimplement this testing logic in python, we can address this issue then. * #1844 Added python embedding for point observation data * Feature 1926 gridstat openmp (#1977) Co-authored-by: johnhg <johnhg@ucar.edu> Co-authored-by: John Halley Gotway <johnhg@ucar.edu> Co-authored-by: Julie Prestopnik <jpresto@seneca.rap.ucar.edu> * feature 1695 ensemble single file (ensemble_stat) (#2007) Co-authored-by: johnhg <johnhg@ucar.edu> * Committing directly to the develop branch. Removing many, many instances of MET_BUILD_BASE and a couple instances of MET_BASE from the examples listed in Appendix A. The inconsistent use of these variables will only cause confusion. Removing them will help avoid that confusion. Big picture... MET_BASE defines the installed 'share/met' directory. It can be used to reference the location of map or polyline files. MET_BUILD_BASE is only used in the context of the test scripts. There, it's defined as the top-level source code directory in which the code was compiled. Its used to reference the location of sample data files or Rscript included in the tarball. I left a couple references to these variables in Appendix A where thier use is not confusing. But I removed all instances which are used to the define the location of the 'bin' directory. Instead, let's just assume the MET binaries are already in their path... or the user knows where to find them. * feature 2011 v10.1.0-beta5 (#2014) * feature 1695 fix issues with ensemble changes (#2012) Co-authored-by: John Halley Gotway <johnhg@ucar.edu> Co-authored-by: John Halley Gotway <johnhg@kiowa.rap.ucar.edu> * Hotfix for #1695 committed directly to the develop branch. The new Ensemble-Stat config files added for this issue don't define a random number generator seed. This results in different output for each run and prevents the regression test from producing repeatable results. Defining the seed here to fix that. * Adding George to email list for the nightly build. * Fix the links for the met-10.1.0-beta5 release notes. * #1844 Make met_point_obs as abstract class * #1844 correctedb for loop end condition on processing obs bdata * Feature 1546 unit_tests (#2021) * #2020 Added SonarQube related varibales * #2020 Initial release * #2015 Avoid the repeasted debug message if derived varibales are disabled * #1996 Initialize right and left * #1966 Call clear() instead of reset memory for Header variable * #1966 Make sure the levels from variable does not exceed the maxLevel * #1966 Removed unreachable codes * #1966 Do not write into NetcDF if empty data * #2015 Corrected "retained or derived" count * Feature 1546 CI testing (#2029) Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * fixed broken workflow by setting environment variable expected for docker build script * add missing docker tag from push command * added correct docker tag value * fixed incorrect truth data version for updating -ref reference data * Updating develop with changes to the unit tests names. Renaming unit_test.yml to a more generic testing.yml name since it compiles AND/OR runs unit tests. Also make the job names more concise so its easier to see what's running in the action summary window on GitHub. * fixed directory to copy truth data into -- copy command was copying the content of the directory, not including the directory itself * Per #1907, added warning about switch from Ensemble-Stat to Gen-Ens-Prod (#2032) Co-authored-by: Julie Prestopnik <jpresto@seneca.rap.ucar.edu> * 2028 Check null for header data * Removed un-reachable code * #2028 Check if the pointer (d) is null * #2015 Changed warning to debug message * #2028 Passing two dimensionl cur and dim array * #2028 Set obs_data * Feature 2003 briercl (#2034) * do not run testing workflow if changes are isolated to the met/docs directory * Per #2013, changed source and README. ci-run-unit (#2039) * Feature 1055 read rot latlon (#2041) * CI: output summary of differences in GHA log to easily see results without having to download the log artifact * #2027 Added a debug message * #2027 ci-run-unit Changed wind thres * #2027 ci-run-unit Increased the precision to debug message * print end of script after error summary, ci-run-unit * #2027 Increased the buffer and cheking the status of sprintf * Feature 1998 standard sections (#2038) * fixing section headers to be consistent with METplus #1998 * fixing section headers to be consistent with METplus #1998 * fixing section headers to be consistent with METplus #1998 missed one * fixing section headers to be consistent with METplus #1998 try again * fixing section headers to be consistent with METplus #1998 I had it right the first time * fixing section headers to be consistent with METplus #1998 * fixing section headers to be consistent with METplus #1998 fixing mistake * fixing section headers to be consistent with METplus #1998 * fixing section headers to be consistent with METplus #1998 * fixing section headers to be consistent with METplus #1998 * fixing section headers to be consistent with METplus #1998 * fixing section header underscore to dash #1998 * updating TOC vs ref 1998 * updating TOC vs ref 1998 completed * trying to fix link #1998 * trying to fix link adding reference back in #1998 * updating TOC again underscore vs dash #1998 * updating TOC again underscore vs dash #1998 * updating TOC vs ref 1998 try * standardizing TOC sections #1998 * standardizing TOC sections #1998 * standardizing TOC sections #1998 * standardizing TOC sections #1998 * standardizing TOC sections #1998 * standardizing TOC sections #1998 * standardizing TOC sections #1998 take 2 * standardizing TOC sections #1998 take 3 * re-standardizing TOC sections #1998 * re-standardizing TOC sections #1998 * re-standardizing TOC sections #1998 * re-standardizing TOC sections #1998 * standardizing TOC sections #1998 * standardizing TOC sections #1998 * standardizing TOC sections #1998 * standardizing TOC sections #1998 * standardizing TOC sections #1998 * standardizing TOC sections #1998 * standardizing TOC sections #1998 * standardizing TOC sections #1998 * standardizing TOC sections #1998 * standardizing TOC sections #1998 * standardizing TOC sections #1998 * standardizing TOC sections #1998 * standardizing TOC sections #1998 * standardizing TOC sections #1998 * standardizing TOC sections #1998 * standardizing TOC sections #1998 * standardizing TOC sections #1998 * standardizing TOC sections #1998 * updating questions into the TOC. test #1998 * updating questions into the TOC. #1998 * standardizing TOC sections #1998 * standardizing TOC sections #1998 fixing typo * standardizing TOC sections #1998 fixing typo * standardizing TOC sections #1998 fixing typo * standardizing TOC sections #1998 fixing typo * standardizing TOC sections #1998 fixing typo * standardizing TOC sections #1998 fixing typo * cleaning up questions. removing section title from questions. * adding a period #1989. * adding the line breaks back in to match other formatting within MET #1989. * Changed some "^^^" to "----" * Changed "###" to "***" Co-authored-by: jprestop <jpresto@ucar.edu> * Update met_stats.cc #2047 Removed the debug message * Update met_stats.cc Removing variable that was set but never used because SonarQube might complain about that, like Fortify does. * Feature 2040 parse timing (#2048) Co-authored-by: Julie Prestopnik <jpresto@seneca.rap.ucar.edu> * CI: Add logic to auto update input data (#2046) * Bugfix 2045 develop hira (#2049) Co-authored-by: jprestop <jpresto@ucar.edu> Co-authored-by: Julie Prestopnik <jpresto@seneca.rap.ucar.edu> Co-authored-by: Seth Linden <linden@ucar.edu> Co-authored-by: Seth Linden <linden@kiowa.rap.ucar.edu> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: hsoh-u <hsoh@ucar.edu> Co-authored-by: Howard Soh <hsoh@kiowa.rap.ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> Co-authored-by: John Halley Gotway <johnhg@seneca.rap.ucar.edu> Co-authored-by: MET Tools Test Account <met_test@seneca.rap.ucar.edu> Co-authored-by: mo-mglover <78152252+mo-mglover@users.noreply.github.com> Co-authored-by: John Halley Gotway <johnhg@kiowa.rap.ucar.edu> Co-authored-by: Howard Soh <hsoh@seneca.rap.ucar.edu> Co-authored-by: davidalbo <dave@ucar.edu> Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com>

mo-mglover added 5 commits November 18, 2021 09:43

Feature 1926. Introduce OpenMP in GridStat.

ed6934b

Feature 1926. Add MET_OPENMP flag to automake setup.

86c1a50

Feature 1926. Indentation tweaks.

b3a063a

Feature 1926. Revert temporary C++17 temporary file code tweak.

e0257c6

Feature 1926. Add _OPENMP ifdef for omp.h.

7d81d41

JohnHalleyGotway self-requested a review November 22, 2021 15:41

JohnHalleyGotway added this to the MET 10.1.0 milestone Nov 22, 2021

JohnHalleyGotway linked an issue Nov 22, 2021 that may be closed by this pull request

Enhance Grid-Stat to use OpenMP for efficient computation of neighborhood statistics by setting $OMP_NUM_THREADS #1926

Closed

21 tasks

JohnHalleyGotway reviewed Nov 22, 2021

View reviewed changes

JohnHalleyGotway requested a review from jprestop November 22, 2021 15:56

Feature 1926. Revert tweak to CXX optimisation level with MET_DEVELOP…

3ff12fe

…MENT.

JohnHalleyGotway reviewed Nov 22, 2021

View reviewed changes

met/src/tools/core/grid_stat/grid_stat.cc Outdated Show resolved Hide resolved

JohnHalleyGotway and others added 3 commits November 26, 2021 16:10

Per #1926, switch the OpenMP configuration option from being like MET…

9a045e0

…_DEVELOPMENT to being like GRIB2, with --enable-grib2."

Per #1926, just like --enable-grib2 and --enable-python, add a test p…

4c176c4

…laceholder for --enable-openmp so that running make test works.

Feature #1926: Ensure OpenMP flag is passed to the linker. Revert to …

7043995

…_OPENMP for the moment.

Feature #1926. Write thread count via logging, not cout.

81c98b9

Co-authored-by: johnhg <johnhg@ucar.edu>

mo-mglover force-pushed the feature_1926_gridstat_openmp branch from dd8d8df to 81c98b9 Compare November 30, 2021 09:23

Feature #1926: Add note about --enable-openmp in installation.rst

1abb2be

JohnHalleyGotway added 4 commits December 2, 2021 14:38

Per #1926, adding a call to Grid-Stat to run on 2 threads. This shoul…

d6969c6

…d produce the same result as the previous test named grid_stat_GRIB2_NAM_RTMA.

Per #1926, tweak the openmp log message slightly.

4b2982f

Per #1926, switch from manual --enable-openmp support in autoconf to …

3c1a5db

…calling the AC_OPENMP() macro. And that macro compile WITH openmp support by default. It can be disabled using --disable-openmp.

JohnHalleyGotway reviewed Dec 8, 2021

View reviewed changes

met/src/basic/vx_util/Makefile.am Outdated Show resolved Hide resolved

mo-mglover and others added 2 commits December 16, 2021 13:52

Update met/src/basic/vx_util/handle_openmp.cc

e0d7840

Co-authored-by: johnhg <johnhg@ucar.edu>

Update met/src/basic/vx_util/Makefile.am

78e4354

Co-authored-by: johnhg <johnhg@ucar.edu>

JohnHalleyGotway mentioned this pull request Dec 16, 2021

Support setting the OMP_NUM_THREADS environment variable. dtcenter/METplus#1320

Closed

21 tasks

Feature #1926: Lengthen OMP_NUM_THREADS debug message, plus user guid…

b2a6c1e

…e info.

Julie Prestopnik added 4 commits December 17, 2021 16:50

Set up the Runtime Environment Variables Section and started adding i…

f39313d

…nfo from openmp.rst

Updated documenation and fixed typos

63b1fce

removed openmp.rst in favor of having the information contained in a …

3e89e56

…Runtime Environment Variables section in config_options.rst and removed openmp from index.rst

Fixed typo

83688f5

jprestop reviewed Jan 6, 2022

View reviewed changes

JohnHalleyGotway added 2 commits January 11, 2022 17:35

Merge branch 'develop' into feature_1926_gridstat_openmp

d73d537

Merge branch 'develop' into feature_1926_gridstat_openmp

821ca68

JohnHalleyGotway self-requested a review January 13, 2022 22:47

JohnHalleyGotway approved these changes Jan 13, 2022

View reviewed changes

JohnHalleyGotway merged commit de81526 into develop Jan 13, 2022

JohnHalleyGotway deleted the feature_1926_gridstat_openmp branch January 13, 2022 22:48

JohnHalleyGotway mentioned this pull request Jan 16, 2022

Update develop-ref after #1977, #2001, #2005, #2006, #2007, and #2012. #2016

Merged

12 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature 1926 gridstat openmp #1977

Feature 1926 gridstat openmp #1977

mo-mglover commented Nov 19, 2021 •

edited by JohnHalleyGotway

Loading

mo-mglover commented Nov 19, 2021 •

edited

Loading

JohnHalleyGotway commented Nov 22, 2021

JohnHalleyGotway Nov 22, 2021 •

edited

Loading

mo-mglover Nov 22, 2021

JohnHalleyGotway commented Nov 22, 2021 •

edited

Loading

JohnHalleyGotway commented Nov 22, 2021

mo-mglover commented Nov 26, 2021 •

edited

Loading

jprestop commented Nov 30, 2021 •

edited

Loading

JohnHalleyGotway commented Dec 2, 2021

JohnHalleyGotway Dec 8, 2021

JohnHalleyGotway commented Dec 8, 2021

jprestop commented Dec 9, 2021 •

edited

Loading

jprestop commented Dec 9, 2021 •

edited

Loading

JohnHalleyGotway commented Dec 13, 2021 •

edited

Loading

mo-mglover commented Dec 16, 2021 •

edited

Loading

JohnHalleyGotway commented Dec 16, 2021

jprestop commented Dec 22, 2021

jprestop left a comment

JohnHalleyGotway left a comment

	<< omp_get_num_threads() << " thread(s).\n";
	<< omp_get_num_threads() << " thread(s).\n";

Feature 1926 gridstat openmp #1977

Feature 1926 gridstat openmp #1977

Conversation

mo-mglover commented Nov 19, 2021 • edited by JohnHalleyGotway Loading

Pull Request Testing

Pull Request Checklist

mo-mglover commented Nov 19, 2021 • edited Loading

JohnHalleyGotway commented Nov 22, 2021

JohnHalleyGotway Nov 22, 2021 • edited Loading

Choose a reason for hiding this comment

mo-mglover Nov 22, 2021

Choose a reason for hiding this comment

JohnHalleyGotway commented Nov 22, 2021 • edited Loading

JohnHalleyGotway commented Nov 22, 2021

mo-mglover commented Nov 26, 2021 • edited Loading

jprestop commented Nov 30, 2021 • edited Loading

JohnHalleyGotway commented Dec 2, 2021

JohnHalleyGotway Dec 8, 2021

Choose a reason for hiding this comment

JohnHalleyGotway commented Dec 8, 2021

jprestop commented Dec 9, 2021 • edited Loading

jprestop commented Dec 9, 2021 • edited Loading

JohnHalleyGotway commented Dec 13, 2021 • edited Loading

mo-mglover commented Dec 16, 2021 • edited Loading

JohnHalleyGotway commented Dec 16, 2021

jprestop commented Dec 22, 2021

jprestop left a comment

Choose a reason for hiding this comment

JohnHalleyGotway left a comment

Choose a reason for hiding this comment

mo-mglover commented Nov 19, 2021 •

edited by JohnHalleyGotway

Loading

mo-mglover commented Nov 19, 2021 •

edited

Loading

JohnHalleyGotway Nov 22, 2021 •

edited

Loading

JohnHalleyGotway commented Nov 22, 2021 •

edited

Loading

mo-mglover commented Nov 26, 2021 •

edited

Loading

jprestop commented Nov 30, 2021 •

edited

Loading

jprestop commented Dec 9, 2021 •

edited

Loading

jprestop commented Dec 9, 2021 •

edited

Loading

JohnHalleyGotway commented Dec 13, 2021 •

edited

Loading

mo-mglover commented Dec 16, 2021 •

edited

Loading