From 94374473eed96f0472558247dedfb7c321d8cba0 Mon Sep 17 00:00:00 2001 From: jprestop Date: Tue, 26 Oct 2021 13:08:20 -0600 Subject: [PATCH 01/82] Per #1906, modify code to pass entire path to make_temp_file_name instead of only the filename so that the function can tell whether or not the file exists. (#1952) Co-authored-by: Julie Prestopnik --- met/src/basic/vx_config/config_file.cc | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/met/src/basic/vx_config/config_file.cc b/met/src/basic/vx_config/config_file.cc index 8b289a7487..a7b1377405 100644 --- a/met/src/basic/vx_config/config_file.cc +++ b/met/src/basic/vx_config/config_file.cc @@ -312,9 +312,10 @@ if ( empty(name) ) { DictionaryStack DS(*this); ConcatString temp_filename = get_tmp_dir(); - -temp_filename << "/" << make_temp_file_name("met_config", 0); - + +temp_filename << "/" << "met_config"; +temp_filename = make_temp_file_name(temp_filename.c_str(), 0); + recursive_envs(name, temp_filename.c_str()); bison_input_filename = (const char *) temp_filename.c_str(); @@ -422,9 +423,10 @@ if ( empty(s) ) { ofstream out; ConcatString temp_filename = get_tmp_dir(); -temp_filename << "/" << make_temp_file_name("config", ".temp"); - - out.open(temp_filename.c_str()); +temp_filename << "/" << "met_config"; +temp_filename = make_temp_file_name(temp_filename.c_str(), 0); + +out.open(temp_filename.c_str()); if ( ! out ) { From 8c446ffd61fb79224c753060ada2fa355c678018 Mon Sep 17 00:00:00 2001 From: Seth Linden Date: Wed, 3 Nov 2021 09:36:55 -0600 Subject: [PATCH 02/82] Feature 1761 percent thresh (#1956) * Per issue #1761 in set_perc() adding code to get FBIAS numeric value, like 1.0 or 0.9, etc. SL * Per issue #1761: in set_perc(), modified actual percentile calculation at end to use the extracted FBIAS numeric value (float). SL * Per issue #1761: modified the check on the perc_thresh_freq_bias, just has to be > 0 now. SL * Per issue #1761: cleaned up code in set_perc(). SL * Per #1761, updates to Simple_Node::set_perc() to handle variable frequency bias amounts. Changes include: - Reverting the formatting of this back to how it originally was in the develop branch. In general, just match the formatting of the existing file, so as the minimize the number of difference lines. - Add logic to adjust the percentile to be found based on the requested FBIAS value. Multiplying or dividing the percentile by the FBIAS value depends on the inequality type and whether we're bias adjusting the forecast or observation data. - Adjust the log messages slightly. Please be aware that I'm not totally confident in these changes. They warrant much more testing. This logic is very, very confusing. * Per #1761, call compute_percentile() when double-checking the percentile values. * Per #1761, remove unused variable. * Per #1761, add warning for percentiles > 100. * Per #1761. In set_perc(), after testing cleaned up code. SL * Per issue #1761: adding new config file for testing dynamic FBIAS values. SL * Per issue #1761: added new unit test for dynamic FBIAS values when running grid_stat. SL * Per issue #1761, modified FBIAS section to indicated that the user can use dynamic values that are not 1.0. SL * Update met/docs/Users_Guide/config_options.rst Co-authored-by: johnhg * Update met/docs/Users_Guide/config_options.rst Co-authored-by: johnhg * Update met/docs/Users_Guide/config_options.rst Co-authored-by: johnhg * Update test/config/GridStatConfig_fbias_perc_thresh Co-authored-by: johnhg * Update test/config/GridStatConfig_fbias_perc_thresh Co-authored-by: johnhg * Update test/config/GridStatConfig_fbias_perc_thresh Co-authored-by: johnhg * Update test/config/GridStatConfig_fbias_perc_thresh Co-authored-by: johnhg * Per issue #1761, set nc_pairs_flag = FALSE. SL Co-authored-by: Seth Linden Co-authored-by: John Halley Gotway --- met/docs/Users_Guide/config_options.rst | 16 +- met/src/basic/vx_config/config.tab.yy | 2 +- met/src/basic/vx_config/threshold.cc | 99 ++++++--- test/config/GridStatConfig_fbias_perc_thresh | 204 +++++++++++++++++++ test/xml/unit_perc_thresh.xml | 24 +++ 5 files changed, 314 insertions(+), 31 deletions(-) create mode 100644 test/config/GridStatConfig_fbias_perc_thresh diff --git a/met/docs/Users_Guide/config_options.rst b/met/docs/Users_Guide/config_options.rst index 23f67fecda..e6d2349cc4 100644 --- a/met/docs/Users_Guide/config_options.rst +++ b/met/docs/Users_Guide/config_options.rst @@ -93,12 +93,14 @@ The configuration file language supports the following data types: e.g. "5.0" and - "fcst.cat_thresh = ==FBIAS1;", MET applies the >5.0 threshold to the - observations and then chooses a forecast threshold which results in a - frequency bias of 1. + * "==FBIAS" for a user-specified frequency bias value. + e.g. "==FBIAS1" to automatically de-bias the data, "==FBIAS0.9" to select a low-bias threshold, or "==FBIAS1.1" to select a high-bias threshold. + This option must be used in + conjunction with a simple threshold in the other field. For example, + when "obs.cat_thresh = >5.0" and "fcst.cat_thresh = ==FBIAS1;", + MET applies the >5.0 threshold to the observations and then chooses a + forecast threshold which results in a frequency bias of 1. + The frequency bias can be any float value > 0.0. * "CDP" for climatological distribution percentile thresholds. These thresholds require that the climatological mean and standard @@ -119,7 +121,7 @@ The configuration file language supports the following data types: For example, "==CDP25" is automatically expanded to 4 percentile bins: >=CDP0&&=CDP25&&=CDP50&&=CDP75&&<=CDP100 - * When sample percentile thresholds of type SFP, SOP, SCP, or FBIAS1 are + * When sample percentile thresholds of type SFP, SOP, SCP, or FBIAS are requested, MET recomputes the actual percentile that the threshold represents. If the requested percentile and actual percentile differ by more than 5%, a warning message is printed. This may occur when the diff --git a/met/src/basic/vx_config/config.tab.yy b/met/src/basic/vx_config/config.tab.yy index d19715eb35..a89d87c6e8 100644 --- a/met/src/basic/vx_config/config.tab.yy +++ b/met/src/basic/vx_config/config.tab.yy @@ -1661,7 +1661,7 @@ if ( s->PT < 0 || s->PT > 100 ) { } -if ( s->Ptype == perc_thresh_freq_bias && !is_eq(s->PT, 1.0) ) { +if ( s->Ptype == perc_thresh_freq_bias && s->PT <= 0 ) { mlog << Error << "\ndo_simple_perc_thresh() -> " << "unsupported frequency bias percentile threshold!\n\n"; diff --git a/met/src/basic/vx_config/threshold.cc b/met/src/basic/vx_config/threshold.cc index cf3e9dd26a..bf317733ff 100644 --- a/met/src/basic/vx_config/threshold.cc +++ b/met/src/basic/vx_config/threshold.cc @@ -934,10 +934,11 @@ void Simple_Node::set_perc(const NumArray *fptr, const NumArray *optr, const Num { -int i, count; +int i; double ptile, diff; NumArray data; const NumArray * ptr = 0; +bool fbias_fcst = false; // // handle sample percentile types @@ -972,14 +973,16 @@ else if ( Ptype == perc_thresh_freq_bias ) { fthr->get_ptype() == no_perc_thresh_type && fthr->get_type() != thresh_complex ) { + fbias_fcst = false; + ptr = optr; op = fthr->get_type(); PT = fptr->compute_percentile(fthr->get_value(), is_inclusive(fthr->get_type())); mlog << Debug(3) - << "The forecast threshold \"" << fthr->get_str() - << "\" includes " << PT * 100.0 << "% of the data.\n"; + << "The forecast threshold value \"" << fthr->get_str() + << "\" represents the " << PT * 100.0 << "-th percentile.\n"; } @@ -991,14 +994,16 @@ else if ( Ptype == perc_thresh_freq_bias ) { othr->get_ptype() == no_perc_thresh_type && othr->get_type() != thresh_complex ) { + fbias_fcst = true; + ptr = fptr; op = othr->get_type(); PT = optr->compute_percentile(othr->get_value(), is_inclusive(othr->get_type())); mlog << Debug(3) - << "The observation threshold \"" << othr->get_str() - << "\" includes " << PT * 100.0 << "% of the data.\n"; + << "The observation threshold value \"" << othr->get_str() + << "\" represents the " << PT * 100.0 << "-th percentile.\n"; } @@ -1029,7 +1034,8 @@ else if ( Ptype == perc_thresh_freq_bias ) { PT *= 100.0; -} +} // end else if PT == perc_thresh_freq_bias + // // nothing to do // @@ -1082,7 +1088,7 @@ if ( data.n() == 0 ) { << " threshold \"" << s << "\" because no valid data was provided.\n\n"; - exit ( 1 ); + exit ( 1 ); } else { @@ -1094,6 +1100,64 @@ else { s.strip_paren(); abbr_s.strip_paren(); + // + // parse the frequency bias value from the threshold string + // + + if ( Ptype == perc_thresh_freq_bias ) { + + ConcatString fs = s; + + fs.replace("==FBIAS", " ", false); + + double fbias_val = atof(fs.c_str()); + + // + // range check requested bias value + // + + if ( fbias_val <= 0.0 ) { + + mlog << Error << "\nSimple_Node::set_perc() -> " + << "the requested frequency bias value (" << fbias_val + << ") must be > 0 in threshold \"" << s << "\".\n\n"; + + } + + // + // adjust PT by the requested frequency bias amount + // + + double PT_new; + + if ( fbias_fcst ) { + if ( op == thresh_le || op == thresh_lt ) PT_new = PT * fbias_val; + else if ( op == thresh_ge || op == thresh_gt ) PT_new = PT / fbias_val; + } + else { + if ( op == thresh_le || op == thresh_lt ) PT_new = PT / fbias_val; + else if ( op == thresh_ge || op == thresh_gt ) PT_new = PT * fbias_val; + } + + if ( PT_new > 100.0 ) { + mlog << Warning << "\nSimple_Node::set_perc() -> " + << "For " << (fbias_fcst ? "forecast" : "observation" ) + << " threshold \"" << s << "\" the required percentile of " + << PT_new << " exceeds the maximum possible value. " + << "Resetting to 100.\n\n"; + + PT_new = 100.0; + } + + mlog << Debug(3) + << "For " << (fbias_fcst ? "forecast" : "observation" ) + << " threshold \"" << s << "\" with type \"" << thresh_type_str[op] + << "\" update the requested percentile from " << PT << " to " + << PT_new << ".\n"; + + PT = PT_new; + } + // // compute the percentile and update the strings // @@ -1117,27 +1181,16 @@ else { // compute the actual percentile and check tolerance // - if ( op == thresh_le || op == thresh_ge || op == thresh_eq ) { - - for ( i=count=0; i " << "the requested percentile (" << PT - << "%) for threshold \"" << s + << ") for threshold \"" << s << "\" differs from the actual percentile (" - << ptile * 100.0 << ") by " << diff * 100.0 << "%.\n" + << ptile * 100.0 << ") by " << diff * 100.0 << ".\n" << "This is common for small samples or data that contains " << "ties.\n\n"; @@ -1146,8 +1199,8 @@ else { mlog << Debug(3) << "The requested percentile (" << PT - << "%) for threshold threshold \"" << s - << "\" includes " << ptile * 100.0 << "% of the data.\n"; + << ") for threshold \"" << s << "\" includes " + << ptile * 100.0 << "% of the data.\n"; } diff --git a/test/config/GridStatConfig_fbias_perc_thresh b/test/config/GridStatConfig_fbias_perc_thresh new file mode 100644 index 0000000000..1072db748a --- /dev/null +++ b/test/config/GridStatConfig_fbias_perc_thresh @@ -0,0 +1,204 @@ +//////////////////////////////////////////////////////////////////////////////// +// +// Grid-Stat configuration file. +// +// For additional information, please see the MET User's Guide. +// +//////////////////////////////////////////////////////////////////////////////// + +// +// Output model name to be written +// +model = "GFS"; + +// +// Output description to be written +// May be set separately in each "obs.field" entry +// +desc = "NA"; + +// +// Output observation type to be written +// +obtype = "ANALYS"; + +//////////////////////////////////////////////////////////////////////////////// + +// +// Verification grid +// +regrid = { + to_grid = NONE; + method = NEAREST; + width = 1; + vld_thresh = 0.5; + shape = SQUARE; +} + +//////////////////////////////////////////////////////////////////////////////// + +// +// May be set separately in each "field" entry +// +censor_thresh = []; +censor_val = []; +mpr_column = []; +mpr_thresh = []; +cat_thresh = []; +cnt_thresh = [ NA ]; +cnt_logic = UNION; +wind_thresh = [ NA ]; +wind_logic = UNION; +eclv_points = 0.05; +nc_pairs_var_name = ""; +nc_pairs_var_suffix = ""; +hss_ec_value = NA; +rank_corr_flag = FALSE; + +// +// Forecast and observation fields to be verified +// +fcst = { + + field = [ + { name = "TMP"; level = [ "P500" ]; cat_thresh = [ ==FBIAS0.9, ==FBIAS1.0, ==FBIAS1.1, ==FBIAS0.9, ==FBIAS1.0, ==FBIAS1.1, <243, <243, <243, >253, >253, >253 ]; } + ]; + +} +obs = { + + field = [ + { name = "TMP"; level = [ "P500" ]; cat_thresh = [ <243, <243, <243, >253, >253, >253, ==FBIAS0.9, ==FBIAS1.0, ==FBIAS1.1, ==FBIAS0.9, ==FBIAS1.0, ==FBIAS1.1 ]; } + ]; + +} + +//////////////////////////////////////////////////////////////////////////////// + +// +// Verification masking regions +// +mask = { + grid = [ "FULL" ]; + poly = [ ]; +} + +//////////////////////////////////////////////////////////////////////////////// + +// +// Confidence interval settings +// +ci_alpha = [ 0.05 ]; + +boot = { + interval = PCTILE; + rep_prop = 1.0; + n_rep = 0; + rng = "mt19937"; + seed = "1"; +} + +//////////////////////////////////////////////////////////////////////////////// + +// +// Data smoothing methods +// +interp = { + field = FCST; + vld_thresh = 1.0; + shape = SQUARE; + + type = [ + { method = NEAREST; width = 1; } + ]; +} + +//////////////////////////////////////////////////////////////////////////////// + +// +// Neighborhood methods +// +nbrhd = { + width = [ 3, 5 ]; + cov_thresh = [ >=0.5 ]; + vld_thresh = 1.0; + shape = SQUARE; +} + +//////////////////////////////////////////////////////////////////////////////// + +// +// Fourier decomposition +// +fourier = { + wave_1d_beg = []; + wave_1d_end = []; +} + +//////////////////////////////////////////////////////////////////////////////// + +// +// Gradient statistics +// May be set separately in each "obs.field" entry +// +gradient = { + dx = [ 1 ]; + dy = [ 1 ]; +} + +//////////////////////////////////////////////////////////////////////////////// + +// +// Distance Map statistics +// May be set separately in each "obs.field" entry +// +distance_map = { + baddeley_p = 2; + baddeley_max_dist = NA; + fom_alpha = 0.1; + zhu_weight = 0.5; + beta_value(n) = n * n / 2.0; +} + +//////////////////////////////////////////////////////////////////////////////// + +// +// Statistical output types +// +output_flag = { + fho = NONE; + ctc = BOTH; + cts = BOTH; + mctc = NONE; + mcts = NONE; + cnt = NONE; + sl1l2 = NONE; + sal1l2 = NONE; + vl1l2 = NONE; + val1l2 = NONE; + vcnt = NONE; + pct = NONE; + pstd = NONE; + pjc = NONE; + prc = NONE; + eclv = NONE; + nbrctc = NONE; + nbrcts = NONE; + nbrcnt = NONE; + grad = NONE; + dmap = NONE; +} + +// +// NetCDF matched pairs output file +// +nc_pairs_flag = FALSE; + +//////////////////////////////////////////////////////////////////////////////// + +grid_weight_flag = NONE; +tmp_dir = "/tmp"; +output_prefix = "${OUTPUT_PREFIX}"; +version = "V10.1.0"; + +//////////////////////////////////////////////////////////////////////////////// diff --git a/test/xml/unit_perc_thresh.xml b/test/xml/unit_perc_thresh.xml index bc2ef52a83..23e1c83aea 100644 --- a/test/xml/unit_perc_thresh.xml +++ b/test/xml/unit_perc_thresh.xml @@ -84,6 +84,30 @@ + + + + + + &MET_BIN;/grid_stat + + OUTPUT_PREFIX PERC_THRESH_FBIAS + + \ + &DATA_DIR_MODEL;/grib2/gfs/gfs_2012040900_F024.grib2 \ + &DATA_DIR_MODEL;/grib2/gfsanl/gfsanl_4_20120410_0000_000.grb2 \ + &CONFIG_DIR;/GridStatConfig_fbias_perc_thresh \ + -outdir &OUTPUT_DIR;/perc_thresh -v 3 + + + &OUTPUT_DIR;/perc_thresh/grid_stat_PERC_THRESH_FBIAS_240000L_20120410_000000V.stat + &OUTPUT_DIR;/perc_thresh/grid_stat_PERC_THRESH_FBIAS_240000L_20120410_000000V_cts.txt + &OUTPUT_DIR;/perc_thresh/grid_stat_PERC_THRESH_FBIAS_240000L_20120410_000000V_cnt.txt + &OUTPUT_DIR;/perc_thresh/grid_stat_PERC_THRESH_FBIAS_240000L_20120410_000000V_sl1l2.txt + &OUTPUT_DIR;/perc_thresh/grid_stat_PERC_THRESH_FBIAS_240000L_20120410_000000V_pairs.nc + + + From d60924c9524ad034a7c6c785b0adbb11ab864feb Mon Sep 17 00:00:00 2001 From: johnhg Date: Wed, 3 Nov 2021 10:58:38 -0600 Subject: [PATCH 03/82] Feature 1905 ens_ctrl (#1955) Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> --- met/docs/Users_Guide/ensemble-stat.rst | 19 +- met/docs/Users_Guide/reformat_grid.rst | 2 +- met/src/basic/vx_util/num_array.cc | 53 ++ met/src/basic/vx_util/num_array.h | 4 + .../vx_statistics/pair_data_ensemble.cc | 60 ++- .../vx_statistics/pair_data_ensemble.h | 4 + .../tools/core/ensemble_stat/ensemble_stat.cc | 463 +++++++++--------- .../tools/core/ensemble_stat/ensemble_stat.h | 18 +- .../ensemble_stat/ensemble_stat_conf_info.cc | 7 +- .../ensemble_stat/ensemble_stat_conf_info.h | 4 +- .../tools/other/gen_ens_prod/gen_ens_prod.cc | 3 +- test/config/EnsembleStatConfig_MASK_SID | 2 +- test/xml/unit_ensemble_stat.xml | 28 ++ 13 files changed, 394 insertions(+), 273 deletions(-) diff --git a/met/docs/Users_Guide/ensemble-stat.rst b/met/docs/Users_Guide/ensemble-stat.rst index a39424d289..1c131a9580 100644 --- a/met/docs/Users_Guide/ensemble-stat.rst +++ b/met/docs/Users_Guide/ensemble-stat.rst @@ -69,6 +69,7 @@ The usage statement for the Ensemble Stat tool is shown below: [-grid_obs file] [-point_obs file] [-ens_mean file] + [-ctrl file] [-obs_valid_beg time] [-obs_valid_end time] [-outdir path] @@ -92,23 +93,23 @@ Optional arguments for ensemble_stat 4. To produce ensemble statistics using gridded observations, use the **-grid_obs file** option to specify a gridded observation file. This option may be used multiple times if your observations are in several files. +5. To produce ensemble statistics using point observations, use the **-point_obs file** option to specify a NetCDF point observation file. This option may be used multiple times if your observations are in several files. -5. To produce ensemble statistics using point observations, use the **-point_obs file** to specify a NetCDF point observation file. This option may be used multiple times if your observations are in several files. +6. To override the simple ensemble mean value of the input ensemble members for the ECNT, SSVAR, and ORANK line types, the **-ens_mean file** option specifies an ensemble mean model data file. This option replaces the **-ssvar_mean file** option from earlier versions of MET. +7. The **-ctrl file** option specifies an ensemble control member data file. The control member is included in the computation of the ensemble mean but excluded from the spread. -6. To override the simple ensemble mean value of the input ensemble members for the ECNT, SSVAR, and ORANK line types, the **-ens_mean file** specifies an ensemble mean model data file. This option replaces the **-ssvar_mean file** from earlier versions of MET. +8. To filter point observations by time, use **-obs_valid_beg time** in YYYYMMDD[_HH[MMSS]] format to set the beginning of the matching observation time window. -7. To filter point observations by time, use **-obs_valid_beg time** in YYYYMMDD[_HH[MMSS]] format to set the beginning of the matching observation time window. +9. As above, use **-obs_valid_end time** in YYYYMMDD[_HH[MMSS]] format to set the end of the matching observation time window. -8. As above, use **-obs_valid_end time** in YYYYMMDD[_HH[MMSS]] format to set the end of the matching observation time window. +10. Specify the **-outdir path** option to override the default output directory (./). -9. Specify the **-outdir path** option to override the default output directory (./). +11. The **-log** file outputs log messages to the specified file. -10. The **-log** file outputs log messages to the specified file. +12. The **-v level** option indicates the desired level of verbosity. The value of "level" will override the default setting of 2. Setting the verbosity to 0 will make the tool run with no log messages, while increasing the verbosity will increase the amount of logging. -11. The **-v level** option indicates the desired level of verbosity. The value of "level" will override the default setting of 2. Setting the verbosity to 0 will make the tool run with no log messages, while increasing the verbosity will increase the amount of logging. - -12. The **-compress level** option indicates the desired level of compression (deflate level) for NetCDF variables. The valid level is between 0 and 9. The value of "level" will override the default setting of 0 from the configuration file or the environment variable MET_NC_COMPRESS. Setting the compression level to 0 will make no compression for the NetCDF output. Lower number is for fast compression and higher number is for better compression. +13. The **-compress level** option indicates the desired level of compression (deflate level) for NetCDF variables. The valid level is between 0 and 9. The value of "level" will override the default setting of 0 from the configuration file or the environment variable MET_NC_COMPRESS. Setting the compression level to 0 will make no compression for the NetCDF output. Lower number is for fast compression and higher number is for better compression. An example of the ensemble_stat calling sequence is shown below: diff --git a/met/docs/Users_Guide/reformat_grid.rst b/met/docs/Users_Guide/reformat_grid.rst index 191ef4e1a4..3f67a377a7 100644 --- a/met/docs/Users_Guide/reformat_grid.rst +++ b/met/docs/Users_Guide/reformat_grid.rst @@ -74,7 +74,7 @@ Required arguments for the pcp_combine Optional arguments for pcp_combine ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -3. The **-field string** option defines the data to be extracted from the input files. Use this option when processing fields other than **APCP** or non-GRIB files. This option may be used multiple times and output will be created for each. +3. The **-field string** option defines the data to be extracted from the input files. Use this option when processing fields other than **APCP** or non-GRIB files. It can be used multiple times and output will be created for each. In general, the field string should include the **name** and **level** of the requested data and be enclosed in single quotes. It is processed as an inline configuration file and may also include data filtering, censoring, and conversion options. For example, use **-field ‘name=”ACPCP”; level=”A6”; convert(x)=x/25.4;’** to read 6-hourly accumulated convective precipitation from a GRIB file and convert from millimeters to inches. 4. The **-name list** option is a comma-separated list of output variable names which override the default choices. If specified, the number of names must match the number of variables to be written to the output file. diff --git a/met/src/basic/vx_util/num_array.cc b/met/src/basic/vx_util/num_array.cc index b681b901ee..446bf29034 100644 --- a/met/src/basic/vx_util/num_array.cc +++ b/met/src/basic/vx_util/num_array.cc @@ -1157,6 +1157,59 @@ double NumArray::wmean_fisher(const NumArray &wgt) const } + +//////////////////////////////////////////////////////////////////////// + + +double NumArray::variance(int skip_index) const + +{ + + if(n() == 0) return ( bad_data_double ); + + int j, count; + double s, s_sq, var; + + s = s_sq = 0.0; + count = 0; + + for(j=0; j 1) { + var = (s_sq - s*s/(double) count)/((double) (count - 1)); + if(is_eq(var, 0.0)) var = 0.0; + } + else { + var = bad_data_double; + } + + return(var); + +} + + +//////////////////////////////////////////////////////////////////////// + + +double NumArray::stdev(int skip_index) const + +{ + + double v = variance(skip_index); + + if ( !is_bad_data(v) ) v = sqrt(v); + + return(v); + +} + + //////////////////////////////////////////////////////////////////////// // diff --git a/met/src/basic/vx_util/num_array.h b/met/src/basic/vx_util/num_array.h index 207b512acd..623f7e39c8 100644 --- a/met/src/basic/vx_util/num_array.h +++ b/met/src/basic/vx_util/num_array.h @@ -21,6 +21,7 @@ #include #include "concat_string.h" +#include "is_bad_data.h" //////////////////////////////////////////////////////////////////////// @@ -107,6 +108,9 @@ class NumArray { double mean_sqrt() const; double mean_fisher() const; + double variance(int skip_index = bad_data_int) const; + double stdev(int skip_index = bad_data_int) const; + double wmean(const NumArray &) const; double wmean_sqrt(const NumArray &) const; double wmean_fisher(const NumArray &) const; diff --git a/met/src/libcode/vx_statistics/pair_data_ensemble.cc b/met/src/libcode/vx_statistics/pair_data_ensemble.cc index 5782cf6a0d..4811f66147 100644 --- a/met/src/libcode/vx_statistics/pair_data_ensemble.cc +++ b/met/src/libcode/vx_statistics/pair_data_ensemble.cc @@ -108,6 +108,7 @@ void PairDataEnsemble::clear() { n_ens = 0; n_pair = 0; + ctrl_index = bad_data_int; skip_const = false; skip_ba.clear(); @@ -121,6 +122,7 @@ void PairDataEnsemble::clear() { esum_na.clear(); esumsq_na.clear(); + esumn_na.clear(); mn_na.clear(); mn_oerr_na.clear(); @@ -170,6 +172,7 @@ void PairDataEnsemble::extend(int n) { var_plus_oerr_na.extend (n); esum_na.extend (n); esumsq_na.extend (n); + esumn_na.extend (n); mn_na.extend (n); mn_oerr_na.extend (n); @@ -222,6 +225,7 @@ void PairDataEnsemble::assign(const PairDataEnsemble &pd) { ign_na = pd.ign_na; pit_na = pd.pit_na; n_pair = pd.n_pair; + ctrl_index = pd.ctrl_index; skip_const = pd.skip_const; skip_ba = pd.skip_ba; @@ -231,6 +235,7 @@ void PairDataEnsemble::assign(const PairDataEnsemble &pd) { esum_na = pd.esum_na; esumsq_na = pd.esumsq_na; + esumn_na = pd.esumn_na; mn_na = pd.mn_na; mn_oerr_na = pd.mn_oerr_na; @@ -290,12 +295,14 @@ void PairDataEnsemble::add_ens_var_sums(int i_obs, double v) { if(i_obs >= esum_na.n()) { esum_na.add(0.0); esumsq_na.add(0.0); + esumn_na.add(0.0); } // Track sums of the raw ensemble member values if(!is_bad_data(v)) { esum_na.inc(i_obs, v); esumsq_na.inc(i_obs, v*v); + esumn_na.inc(i_obs, 1); } return; @@ -416,17 +423,17 @@ void PairDataEnsemble::compute_pair_vals(const gsl_rng *rng_ptr) { else { // Compute the variance of the unperturbed ensemble members - var_unperturbed = compute_variance(esum_na[i], esumsq_na[i], v_na[i]); + var_unperturbed = compute_variance(esum_na[i], esumsq_na[i], esumn_na[i]); var_na.add(var_unperturbed); // Process the observation error information. ObsErrorEntry * e = (has_obs_error() ? obs_error_entry[i] : 0); if(e) { - // Compute perturbed ensemble mean and variance - cur_ens.compute_mean_variance(mean, var_perturbed); - mn_oerr_na.add(mean); - var_oerr_na.add(var_perturbed); + // Compute perturbed ensemble mean and variance + // Exclude the control member from the variance + mn_oerr_na.add(cur_ens.mean()); + var_oerr_na.add(cur_ens.variance(ctrl_index)); // Compute the variance plus observation error variance. var_plus_oerr_na.add(var_unperturbed + @@ -467,8 +474,12 @@ void PairDataEnsemble::compute_pair_vals(const gsl_rng *rng_ptr) { crps_emp_na.add(compute_crps_emp(o_na[i], cur_ens)); crpscl_emp_na.add(compute_crps_emp(o_na[i], cur_clm)); + // Ensemble mean and standard deviation + // Exclude the control member from the standard deviation + mean = cur_ens.mean(); + stdev = cur_ens.stdev(ctrl_index); + // Store Gaussian CRPS stats - cur_ens.compute_mean_stdev(mean, stdev); crps_gaus_na.add(compute_crps_gaus(o_na[i], mean, stdev)); crpscl_gaus_na.add(compute_crps_gaus(o_na[i], cmn_na[i], csd_na[i])); ign_na.add(compute_ens_ign(o_na[i], mean, stdev)); @@ -613,7 +624,7 @@ struct ssvar_bin_comp { void PairDataEnsemble::compute_ssvar() { int i, j; - double mn, var; + double var; ssvar_bin_map bins; NumArray cur; @@ -644,21 +655,22 @@ void PairDataEnsemble::compute_ssvar() { if(skip_ba[i]) continue; // Store ensemble values for the current observation - for(j=0, cur.erase(); jflag) { @@ -1781,6 +1796,21 @@ void VxPairDataEnsemble::calc_obs_summary() { //////////////////////////////////////////////////////////////////////// +void VxPairDataEnsemble::set_ctrl_index(int index) { + + for(int i=0; i < n_msg_typ; i++){ + for(int j=0; j < n_mask; j++){ + for(int k=0; k < n_interp; k++){ + pd[i][j][k].ctrl_index = index; + } + } + } + + return; +} + +//////////////////////////////////////////////////////////////////////// + void VxPairDataEnsemble::set_skip_const(bool tf) { for(int i=0; i < n_msg_typ; i++){ diff --git a/met/src/libcode/vx_statistics/pair_data_ensemble.h b/met/src/libcode/vx_statistics/pair_data_ensemble.h index e9c2417e49..bee6b11838 100644 --- a/met/src/libcode/vx_statistics/pair_data_ensemble.h +++ b/met/src/libcode/vx_statistics/pair_data_ensemble.h @@ -90,6 +90,7 @@ class PairDataEnsemble : public PairBase { int n_ens; // Number of ensemble members int n_pair; // Number of valid pairs, n_obs - sum(skip_ba) + int ctrl_index; // Index of the control member bool skip_const; // Skip cases where the observation and // all ensemble members are constant BoolArray skip_ba; // Flag for each observation [n_obs] @@ -106,6 +107,7 @@ class PairDataEnsemble : public PairBase { NumArray esum_na; // Sum of unperturbed ensemble values [n_obs] NumArray esumsq_na; // Sum of unperturbed ensemble squared values [n_obs] + NumArray esumn_na; // Count of ensemble values [n_obs] NumArray mn_na; // Ensemble mean value [n_obs] NumArray mn_oerr_na; // Mean of perturbed members [n_obs] @@ -293,6 +295,8 @@ class VxPairDataEnsemble { void calc_obs_summary(); + void set_ctrl_index(int); + void set_skip_const(bool); }; diff --git a/met/src/tools/core/ensemble_stat/ensemble_stat.cc b/met/src/tools/core/ensemble_stat/ensemble_stat.cc index 6dfc98dee3..783691945b 100644 --- a/met/src/tools/core/ensemble_stat/ensemble_stat.cc +++ b/met/src/tools/core/ensemble_stat/ensemble_stat.cc @@ -61,6 +61,7 @@ // statistics in the ECNT line type. // 031 09/13/21 Seth Linden Changed obs_qty to obs_qty_inc. // Added code for obs_qty_exc. +// 032 10/07/21 Halley Gotway MET #1905 Add -ctrl option. // //////////////////////////////////////////////////////////////////////// @@ -124,7 +125,7 @@ static void do_rps (const EnsembleStatVxOpt &, const PairDataEnsemble *); static void clear_counts(); -static void track_counts(int, const DataPlane &); +static void track_counts(int, const DataPlane &, bool); static ConcatString get_ens_mn_var_name(int); @@ -135,10 +136,12 @@ static void setup_table (AsciiTable &); static void build_outfile_name(unixtime, const char *, ConcatString &); static void write_ens_nc(int, DataPlane &); -static void write_ens_var_float(int, float *, DataPlane &, +static void write_ens_var_float(int, float *, const DataPlane &, const char *, const char *); -static void write_ens_var_int(int, int *, DataPlane &, +static void write_ens_var_int(int, int *, const DataPlane &, const char *, const char *); +static void write_ens_data_plane(int, const DataPlane &, const DataPlane &, + const char *, const char *); static void write_orank_nc(PairDataEnsemble &, DataPlane &, int, int, int); static void write_orank_var_float(int, int, int, float *, DataPlane &, @@ -146,8 +149,8 @@ static void write_orank_var_float(int, int, int, float *, DataPlane &, static void write_orank_var_int(int, int, int, int *, DataPlane &, const char *, const char *); -static void add_var_att_local(VarInfo *, NcVar *, bool is_int, DataPlane &, - const char *, const char *); +static void add_var_att_local(VarInfo *, NcVar *, bool is_int, + const DataPlane &, const char *, const char *); static void finish_txt_files(); static void clean_up(); @@ -157,6 +160,7 @@ static void usage(); static void set_grid_obs(const StringArray &); static void set_point_obs(const StringArray &); static void set_ens_mean(const StringArray & a); +static void set_ctrl_file(const StringArray &); static void set_obs_valid_beg(const StringArray &); static void set_obs_valid_end(const StringArray &); static void set_outdir(const StringArray &); @@ -215,20 +219,18 @@ void process_command_line(int argc, char **argv) { cline.set_usage(usage); // - // add the options function calls + // add the option function calls + // quietly support deprecated -ssvar_mean option // - cline.add(set_grid_obs, "-grid_obs", 1); - cline.add(set_point_obs, "-point_obs", 1); - cline.add(set_ens_mean, "-ens_mean", 1); + cline.add(set_grid_obs, "-grid_obs", 1); + cline.add(set_point_obs, "-point_obs", 1); + cline.add(set_ens_mean, "-ens_mean", 1); + cline.add(set_ctrl_file, "-ctrl", 1); cline.add(set_obs_valid_beg, "-obs_valid_beg", 1); cline.add(set_obs_valid_end, "-obs_valid_end", 1); - cline.add(set_outdir, "-outdir", 1); - cline.add(set_compress, "-compress", 1); - - // - // quietly support deprecated -ssvar_mean option - // - cline.add(set_ens_mean, "-ssvar_mean", 1); + cline.add(set_outdir, "-outdir", 1); + cline.add(set_compress, "-compress", 1); + cline.add(set_ens_mean, "-ssvar_mean", 1); // // parse the command line @@ -291,6 +293,13 @@ void process_command_line(int argc, char **argv) { exit(1); } + // Prepend the control member, if specified + if(ctrl_file.nonempty()) { + ctrl_index = 0; + ens_file_list.insert(ctrl_index, ctrl_file.c_str()); + n_ens++; + } + // Check that the end_ut >= beg_ut if(obs_valid_beg_ut != (unixtime) 0 && obs_valid_end_ut != (unixtime) 0 && @@ -370,18 +379,19 @@ void process_command_line(int argc, char **argv) { shc.set_model(conf_info.model.c_str()); // Allocate arrays to store threshold counts - thresh_count_na = new NumArray [conf_info.get_max_n_thresh()]; - thresh_nbrhd_count_na = new NumArray * [conf_info.get_max_n_thresh()]; + thresh_cnt_na = new NumArray [conf_info.get_max_n_thresh()]; + thresh_nbrhd_cnt_na = new NumArray * [conf_info.get_max_n_thresh()]; for(i=0; i " + << "the control member file is not valid.\n\n"; + exit(1); + } + + // Otherwise, continue + continue; + } // Read the current field if(!get_data_plane(ens_file_list[j].c_str(), etype, - conf_info.ens_info[i], ens_dp, true)) continue; + conf_info.ens_info[i], ens_dp, true)) { + + // Error out if the control member data cannot be read + if(j==ctrl_index) { + mlog << Error << "\nprocess_ensemble() -> " + << "can't find \"" << conf_info.ens_info[i]->magic_str() + << "\" data in the control member file.\n\n"; + exit(1); + } + + // Otherwise, continue + continue; + } // Create a NetCDF file to store the ensemble output if(nc_out == (NcFile *) 0) { @@ -788,7 +821,7 @@ void process_ensemble() { } // Apply current data to the running sums and counts - track_counts(i, ens_dp); + track_counts(i, ens_dp, j == ctrl_index); // Keep track of the maximum initialization time if(is_bad_data(max_init_ut) || ens_dp.init() > max_init_ut) { @@ -802,8 +835,7 @@ void process_ensemble() { write_ens_nc(i, ens_dp); // Store the ensemble mean output file - ens_mean_file = - out_nc_file_list[out_nc_file_list.n() - 1]; + ens_mean_file = out_nc_file_list[out_nc_file_list.n() - 1]; } // end for i @@ -825,7 +857,7 @@ void process_vx() { if(point_obs_file_list.n() == 0 && grid_obs_file_list.n() == 0) { mlog << Error << "\nprocess_vx() -> " - << " when \"fcst.field\" is non-empty, you must use " + << "when \"fcst.field\" is non-empty, you must use " << "\"-point_obs\" and/or \"-grid_obs\" to specify the " << "verifying observations.\n\n"; exit(1); @@ -835,7 +867,7 @@ void process_vx() { conf_info.process_masks(grid); // Setup the PairDataEnsemble objects - conf_info.set_vx_pd(n_vx_vld); + conf_info.set_vx_pd(n_vx_vld, ctrl_index); // Process the point observations if(point_obs_flag) process_point_vx(); @@ -1080,7 +1112,8 @@ int process_point_ens(int i_ens, int &n_miss) { ens_mean_file : ens_mean_user); mlog << Debug(2) << "\n" << sep_str << "\n\n" - << "Processing " << file_type << " file: " << ens_file << "\n"; + << "Processing " << file_type << " file: " << ens_file + << (i_ens == ctrl_index ? " (control)\n" : "\n"); // Loop through each of the fields to be verified and extract // the forecast fields for verification @@ -1651,6 +1684,7 @@ void process_grid_vx() { pd_all.clear(); pd_all.set_ens_size(n_vx_vld[i]); pd_all.set_climo_cdf_info(conf_info.vx_opt[i].cdf_info); + pd_all.ctrl_index = conf_info.vx_opt[i].vx_pd.pd[0][0][0].ctrl_index; pd_all.skip_const = conf_info.vx_opt[i].vx_pd.pd[0][0][0].skip_const; // Apply the current mask to the fields and compute the pairs @@ -1813,7 +1847,7 @@ void process_grid_scores(int i_vx, ObsErrorEntry *e = (ObsErrorEntry *) 0; // Allocate memory in one big chunk based on grid size - pd.extend(grid.nx()*grid.ny()); + pd.extend(nxy); // Climatology flags bool emn_flag = (emn_dp.nx() == obs_dp.nx() && @@ -1880,7 +1914,8 @@ void process_grid_scores(int i_vx, pd.add_ens(j-n_miss, fcst_dp[j](x, y)); // Store the unperturbed ensemble value - pd.add_ens_var_sums(i, fraw_dp[j](x, y)); + // Exclude the control member from the variance + if(j != ctrl_index) pd.add_ens_var_sums(i, fraw_dp[j](x, y)); } } // end for j } // end for i @@ -1955,46 +1990,21 @@ void do_rps(const EnsembleStatVxOpt &vx_opt, //////////////////////////////////////////////////////////////////////// void clear_counts() { - int i, j, k; + int i, j; - // Allocate memory in one big chunk based on grid size, if needed - count_na.extend(nxy); - min_na.extend(nxy); - max_na.extend(nxy); - sum_na.extend(nxy); - sum_sq_na.extend(nxy); - for(i=0; i= MaxBuf[i] || is_bad_data(MaxBuf[i])) MaxBuf[i] = v; + // Ensemble sum + sum_na.buf()[i] += v; - SumBuf[i] += v; - SumSqBuf[i] += v*v; + // Ensemble min and max + if(v <= min_na.buf()[i] || is_bad_data(min_na.buf()[i])) min_na.buf()[i] = v; + if(v >= max_na.buf()[i] || is_bad_data(max_na.buf()[i])) max_na.buf()[i] = v; - for(j=0; j 0 @@ -2051,23 +2062,21 @@ void track_counts(int i_vx, const DataPlane &dp) { DataPlane frac_dp; // Loop over thresholds - for(i=0; i 0) thresh_nbrhd_count_na[i][j].inc(k, 1); + if(frac_dp.data()[k] > 0) thresh_nbrhd_cnt_na[i][j].inc(k, 1); } // end for k } // end for j @@ -2332,45 +2341,33 @@ void build_outfile_name(unixtime ut, const char *suffix, ConcatString &str) { //////////////////////////////////////////////////////////////////////// -void write_ens_nc(int i_ens, DataPlane &dp) { +void write_ens_nc(int i_var, DataPlane &ens_dp) { int i, j, k, l; double t, v; char type_str[max_str_len]; - DataPlane prob_dp, smooth_dp; - - // Arrays for storing ensemble data - float *ens_mean = (float *) 0; - float *ens_stdev = (float *) 0; - float *ens_minus = (float *) 0; - float *ens_plus = (float *) 0; - float *ens_min = (float *) 0; - float *ens_max = (float *) 0; - float *ens_range = (float *) 0; - int *ens_vld = (int *) 0; - float *ens_prob = (float *) 0; + DataPlane prob_dp, nbrhd_dp; // Allocate memory for storing ensemble data - ens_mean = new float [grid.nx()*grid.ny()]; - ens_stdev = new float [grid.nx()*grid.ny()]; - ens_minus = new float [grid.nx()*grid.ny()]; - ens_plus = new float [grid.nx()*grid.ny()]; - ens_min = new float [grid.nx()*grid.ny()]; - ens_max = new float [grid.nx()*grid.ny()]; - ens_range = new float [grid.nx()*grid.ny()]; - ens_vld = new int [grid.nx()*grid.ny()]; - ens_prob = new float [grid.nx()*grid.ny()]; + float * ens_mean = new float [nxy]; + float * ens_stdev = new float [nxy]; + float * ens_minus = new float [nxy]; + float * ens_plus = new float [nxy]; + float * ens_min = new float [nxy]; + float * ens_max = new float [nxy]; + float * ens_range = new float [nxy]; + int * ens_vld = new int [nxy]; // Store the threshold for the ratio of valid data points t = conf_info.vld_data_thresh; // Store the data - for(i=0; i 0) var_str << "_" << cs; // Construct the variable name ens_var_name << cs_erase - << conf_info.ens_info[i_ens]->name_attr() << "_" - << conf_info.ens_info[i_ens]->level_attr() + << conf_info.ens_info[i_var]->name_attr() << "_" + << conf_info.ens_info[i_var]->level_attr() << var_str << "_" << type_str; // Skip variable names that have already been written @@ -2625,16 +2590,16 @@ void write_ens_var_float(int i_ens, float *ens_data, DataPlane &dp, // if(strcmp(type_str, "ENS_MEAN") == 0) { name_str << cs_erase - << conf_info.ens_info[i_ens]->name_attr(); + << conf_info.ens_info[i_var]->name_attr(); } else { name_str << cs_erase - << conf_info.ens_info[i_ens]->name_attr() << "_" + << conf_info.ens_info[i_var]->name_attr() << "_" << type_str; } // Add the variable attributes - add_var_att_local(conf_info.ens_info[i_ens], &ens_var, false, dp, + add_var_att_local(conf_info.ens_info[i_var], &ens_var, false, dp, name_str.c_str(), long_name_str); // Write the data @@ -2650,20 +2615,20 @@ void write_ens_var_float(int i_ens, float *ens_data, DataPlane &dp, //////////////////////////////////////////////////////////////////////// -void write_ens_var_int(int i_ens, int *ens_data, DataPlane &dp, +void write_ens_var_int(int i_var, int *ens_data, const DataPlane &dp, const char *type_str, const char *long_name_str) { NcVar ens_var; ConcatString ens_var_name, var_str, name_str, cs; // Append nc_pairs_var_str config file entry - cs = conf_info.ens_var_str[i_ens]; + cs = conf_info.ens_var_str[i_var]; if(cs.length() > 0) var_str << "_" << cs; // Construct the variable name ens_var_name << cs_erase - << conf_info.ens_info[i_ens]->name_attr() << "_" - << conf_info.ens_info[i_ens]->level_attr() + << conf_info.ens_info[i_var]->name_attr() << "_" + << conf_info.ens_info[i_var]->level_attr() << var_str << "_" << type_str; // Skip variable names that have already been written @@ -2678,11 +2643,11 @@ void write_ens_var_int(int i_ens, int *ens_data, DataPlane &dp, // Construct the variable name attribute name_str << cs_erase - << conf_info.ens_info[i_ens]->name_attr() << "_" + << conf_info.ens_info[i_var]->name_attr() << "_" << type_str; // Add the variable attributes - add_var_att_local(conf_info.ens_info[i_ens], &ens_var, true, dp, + add_var_att_local(conf_info.ens_info[i_var], &ens_var, true, dp, name_str.c_str(), long_name_str); // Write the data @@ -2698,6 +2663,26 @@ void write_ens_var_int(int i_ens, int *ens_data, DataPlane &dp, //////////////////////////////////////////////////////////////////////// +void write_ens_data_plane(int i_var, const DataPlane &ens_dp, const DataPlane &dp, + const char *type_str, const char *long_name_str) { + + // Allocate memory for this data + float *ens_data = new float [nxy]; + + // Store the data in an array of floats + for(int i=0; i + + &MET_BIN;/ensemble_stat + + CENSOR_THRESH + CENSOR_VAL + SKIP_CONST FALSE + OUTPUT_PREFIX MASK_SID_CTRL + CONFIG_DIR &CONFIG_DIR; + + \ + 5 \ + &DATA_DIR_MODEL;/grib1/arw-fer-gep5/arw-fer-gep5_2012040912_F024.grib \ + &DATA_DIR_MODEL;/grib1/arw-sch-gep2/arw-sch-gep2_2012040912_F024.grib \ + &DATA_DIR_MODEL;/grib1/arw-sch-gep6/arw-sch-gep6_2012040912_F024.grib \ + &DATA_DIR_MODEL;/grib1/arw-tom-gep3/arw-tom-gep3_2012040912_F024.grib \ + &DATA_DIR_MODEL;/grib1/arw-tom-gep7/arw-tom-gep7_2012040912_F024.grib \ + &CONFIG_DIR;/EnsembleStatConfig_MASK_SID \ + -ctrl &DATA_DIR_MODEL;/grib1/arw-fer-gep1/arw-fer-gep1_2012040912_F024.grib \ + -point_obs &OUTPUT_DIR;/ascii2nc/gauge_2012041012_24hr.nc \ + -outdir &OUTPUT_DIR;/ensemble_stat -v 1 + + + &OUTPUT_DIR;/ensemble_stat/ensemble_stat_MASK_SID_CTRL_20120410_120000V.stat + &OUTPUT_DIR;/ensemble_stat/ensemble_stat_MASK_SID_CTRL_20120410_120000V_orank.txt + &OUTPUT_DIR;/ensemble_stat/ensemble_stat_MASK_SID_CTRL_20120410_120000V_ens.nc + + + &MET_BIN;/ensemble_stat From 3b13ce102db384e8c8b05d5092f2e28c8dd2f9cd Mon Sep 17 00:00:00 2001 From: John Halley Gotway Date: Wed, 3 Nov 2021 13:42:36 -0600 Subject: [PATCH 04/82] Hotfix after merging changing for #1761. Updating the list of expected output files. --- test/xml/unit_perc_thresh.xml | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/test/xml/unit_perc_thresh.xml b/test/xml/unit_perc_thresh.xml index 23e1c83aea..893dedf3cf 100644 --- a/test/xml/unit_perc_thresh.xml +++ b/test/xml/unit_perc_thresh.xml @@ -101,10 +101,8 @@ &OUTPUT_DIR;/perc_thresh/grid_stat_PERC_THRESH_FBIAS_240000L_20120410_000000V.stat + &OUTPUT_DIR;/perc_thresh/grid_stat_PERC_THRESH_FBIAS_240000L_20120410_000000V_ctc.txt &OUTPUT_DIR;/perc_thresh/grid_stat_PERC_THRESH_FBIAS_240000L_20120410_000000V_cts.txt - &OUTPUT_DIR;/perc_thresh/grid_stat_PERC_THRESH_FBIAS_240000L_20120410_000000V_cnt.txt - &OUTPUT_DIR;/perc_thresh/grid_stat_PERC_THRESH_FBIAS_240000L_20120410_000000V_sl1l2.txt - &OUTPUT_DIR;/perc_thresh/grid_stat_PERC_THRESH_FBIAS_240000L_20120410_000000V_pairs.nc From 5131042053ffdc94c6c5fe6ed9a64ccd70775e96 Mon Sep 17 00:00:00 2001 From: John Halley Gotway Date: Wed, 3 Nov 2021 15:54:06 -0600 Subject: [PATCH 05/82] Per #1905, committing a hotfix directly to the develop branch. Reverting the logic for computing the ensemble range back to what it was previously. The new version produced very slight differences in the 6-th or 7-th decimal place when compared to previous results. There's not good reason for these changes which were caused by the order of operations in casting from doubles to floats. Reverting back to the old logic prevents diffs for anyone else downstream and is the prudent choice. --- met/src/tools/core/ensemble_stat/ensemble_stat.cc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/met/src/tools/core/ensemble_stat/ensemble_stat.cc b/met/src/tools/core/ensemble_stat/ensemble_stat.cc index 783691945b..d712c96abd 100644 --- a/met/src/tools/core/ensemble_stat/ensemble_stat.cc +++ b/met/src/tools/core/ensemble_stat/ensemble_stat.cc @@ -2387,8 +2387,8 @@ void write_ens_nc(int i_var, DataPlane &ens_dp) { ens_min[i] = (float) min_na[i]; ens_max[i] = (float) max_na[i]; v = max_na[i] - min_na[i]; - ens_range[i] = (is_eq(max_na[i], min_na[i]) ? - 0.0 : (float) max_na[i] - min_na[i]); + if(is_eq(v, 0.0)) v = 0; + ens_range[i] = (float) v; } } // end for i From 4ff28d192d07fb61c3271cea4d62b8b59b84ef6b Mon Sep 17 00:00:00 2001 From: johnhg Date: Thu, 4 Nov 2021 08:53:32 -0600 Subject: [PATCH 06/82] Feature 1957 ascii2nc_python (#1958) --- met/src/tools/other/ascii2nc/ascii2nc.cc | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/met/src/tools/other/ascii2nc/ascii2nc.cc b/met/src/tools/other/ascii2nc/ascii2nc.cc index a8f74f9daf..4d876793b8 100644 --- a/met/src/tools/other/ascii2nc/ascii2nc.cc +++ b/met/src/tools/other/ascii2nc/ascii2nc.cc @@ -116,7 +116,7 @@ static ASCIIFormat ascii_format = ASCIIFormat_None; //////////////////////////////////////////////////////////////////////// // Variables for command line arguments -static vector< ConcatString > asfile_list; +static vector asfile_list; static ConcatString ncfile; static ConcatString config_filename(replace_path(DEFAULT_CONFIG_FILENAME)); @@ -234,10 +234,13 @@ int main(int argc, char *argv[]) { file_handler->setMessageTypeMap(config_info.getMessageTypeMap()); // - // Process the files. If a configuration file was specified, do any - // extra processing specified. + // Read the input files // - file_handler->readAsciiFiles(asfile_list); + if(!file_handler->readAsciiFiles(asfile_list)) { + mlog << Error << "\n" << program_name << "-> " + << "encountered an error while reading input files!\n\n"; + exit(1); + } // // Summarize the observations, if directed. We need to use a different From aee6269263380bcf1e0b1c91f44e9435c299f24e Mon Sep 17 00:00:00 2001 From: hsoh-u Date: Mon, 15 Nov 2021 09:01:58 -0700 Subject: [PATCH 07/82] Feature 1949 cf netcdf documentaton (#1951) * #1949 Added CF compliant NetCDF into data IO * #1949 Added commas * #1948 Some corrections for typo and added the links for CF attributes * #1948 Added Performance with NetCDF input data * #1949 Corrected tyoe and applied Juloie's suggestions Co-authored-by: Howard Soh --- met/docs/Users_Guide/data_io.rst | 112 +++++++++++++++++++++++++++++++ 1 file changed, 112 insertions(+) diff --git a/met/docs/Users_Guide/data_io.rst b/met/docs/Users_Guide/data_io.rst index 8b6d41ea95..dcca8763f9 100644 --- a/met/docs/Users_Guide/data_io.rst +++ b/met/docs/Users_Guide/data_io.rst @@ -18,6 +18,118 @@ Input point observation files in PrepBUFR format are available through NCEP. The Tropical cyclone forecasts and observations are typically provided in a specific ATCF (Automated Tropical Cyclone Forecasting) ASCII format, in A-deck, B-deck, and E-deck files. +Requirements for CF Compliant NetCDF +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The MET tools use following attributes and variables for input CF Compliant NetCDF data. + +1. The global attribute "Conventions". + +2. The "`standard_name `_" and "`units `_" attributes for coordinate variables. The "`axis `_" attribute ("T" or "time") must exist as the time variable if the "standard_name" attribute does not exist. + +3. The "`coordinates `_" attribute for the data variables. It contains the coordinate variable names. + +4. The "`grid_mapping `_" attribute for the data variables for projections and the matching grid mapping variable (optional for the latitude_longitude projection). + +5. The gridded data should be evenly spaced horizontally and vertically. + +6. (Optional) the "`forecast_reference_time `_" variable for init_time. + +MET processes the CF-Compliant gridded NetCDF files with the projection information. The CF-Compliant NetCDF is defined by the global attribute "Conventions" whose value begins with "CF-" ("CF-"). The global attribute "Conventions" is mandatory. MET accepts the variation of this attribute ("conventions" and "CONVENTIONS"). The value should be started with "CF-" and followed by the version number. MET accepts the attribute value that begins with "CF " ("CF" and a space instead of a hyphen) or "COARDS". + +The grid mapping variable contains the projection information. The grid mapping variable can be found by looking at the variable attribute "grid_mapping" from the data variables. The "standard_name" attribute is used to filter out the coordinate variables like time, latitude, and longitude variables. The value of the "grid_mapping" attribute is the name of the grid mapping variable. Four projections are supported with grid mapping variables: latitude_longitude, lambert_conformal_conic, polar_stereographic, and geostationary. In case of the latitude_longitude projection, the latitude and longitude variable names should be the same as the dimension names and the "units" attribute should be valid. + +Here are examples for the grid mapping variable ("edr" is the data variable): + +**Example 1: grid mapping for latitude_longitude projection** + +.. code-block:: none + + float edr(time, z, lat, lon) ; + edr:units = "m^(2/3) s^-1" ; + edr:long_name = "Median eddy dissipation rate" ; + edr:coordinates = "lat lon" ; + edr:_FillValue = -9999.f ; + edr:grid_mapping = "grid_mapping" ; + int grid_mapping ; + grid_mapping:grid_mapping_name = "latitude_longitude" ; + grid_mapping:semi_major_axis = 6371000. ; + grid_mapping:inverse_flattening = 0 ; + + +**Example 2: grid mapping for lambert_conformal_conic projection** + +.. code-block:: none + + float edr(time, z, y, x) ; + edr:units = "m^(2/3) s^-1" ; + edr:long_name = "Eddy dissipation rate" ; + edr:coordinates = "lat lon" ; + edr:_FillValue = -9999.f ; + edr:grid_mapping = "grid_mapping" ; + int grid_mapping ; + grid_mapping:grid_mapping_name = "lambert_conformal_conic" ; + grid_mapping:standard_parallel = 25. ; + grid_mapping:longitude_of_central_meridian = -95. ; + grid_mapping:latitude_of_projection_origin = 25. ; + grid_mapping:false_easting = 0 ; + grid_mapping:false_northing = 0 ; + grid_mapping:GRIB_earth_shape = "spherical" ; + grid_mapping:GRIB_earth_shape_code = 0 ; + +When the grid mapping variable is not available, MET detects the latitude_longitude projection in following order: + +1. the lat/lon projection from the dimensions + +2. the lat/lon projection from the "coordinates" attribute from the data variable + +3. the lat/lon projection from the latitude and longitude variables by the "standard_name" attribute + +MET is looking for variables with the same name as the dimension and checking the "units" attribute to find the latitude and longitude variables. The valid "units" strings are listed in the table below. MET accepts the variable "tlat" and "tlon" if the dimension names are "nlat" and "nlon”. + +If there are no latitude and longitude variables from dimensions, MET gets coordinate variable names from the "coordinates" attribute. The matching coordinate variables should have the proper "units" attribute. + +MET gets the time, latitude, and longitude variables by looking at the standard name: "time", "latitude", and "longitude" as the last option. + +MET gets the valid time from the time variable and the "forecast_reference_time" variable for the init_time. If the time variable does not exist, it can come from the file name. MET supports only two cases: + +1. TRMM_3B42_3hourly_filename (3B42...7.G3.nc) + +2. TRMM_3B42_daily_filename (3B42_daily...
.7.G3.nc) + +.. list-table:: Valid strings for the "units" attribute. + :widths: auto + :header-rows: 1 + + * - time + - latitude + - longitude + * - "seconds since YYYY-MM-DD HH:MM:SS", + "minutes since YYYY-MM-DD HH:MM:SS", + "hours since YYYY-MM-DD HH:MM:SS", + "days since YYYY-MM-DD HH:MM:SS", + Accepts "Y", "YY", "YYY", "M", "D", "HH", and "HH:MM". + "HH:MM:SS" is optional + - "degrees_north", + "degree_north", + "degree_N", + "degrees_N", + "degreeN", + "degreesN" + - "degrees_east", + "degree_east", + "degree_E", + "degrees_E", + "degreeE", + "degreesE" + +Performance with NetCDF input data +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +There is no limitation on the NetCDF file size. The size of the data variables matters more than the file size. The NetCDF API loads the metadata first upon opening the NetCDF file. It's similar for accessing data variables. There are two API calls: getting the metadata and getting the actual data. The memory is allocated and consumed at the second API call (getting the actual data). + +The dimensions of the data variables matter. MET requests the NetCDF data needs based on: 1) loading and processing a data plane, and 2) loading and processing the next data plane. This means an extra step for slicing with one more dimension in the NetCDF input data. The performance is quite different if the compression is enabled with high resolution data. NetCDF does compression per variable. The variables can have different compression levels (0 to 9). A value of 0 means no compression, and 9 is the highest level of compression possible. The number for decompression is the same between one more and one less dimension NetCDF input files (combined VS separated). The difference is the amount of data to be decompressed which requires more memory. For example, let's assume the time dimension is 30. NetCDF data with one less dimension (no time dimension) does decompression 30 times for nx by ny dataset. NetCDF with one more dimension does compression 30 times for 30 by nx by ny dataset and slicing for target time offset. So it's better to have multiple NetCDF files with one less dimension than a big file with bigger variable data if compressed. If the compression is not enabled, the file size will be much bigger requiring more disk space. + .. _Intermediate data formats: Intermediate data formats From 867360f7fc52c62a752873bf69775973e7936a8a Mon Sep 17 00:00:00 2001 From: johnhg Date: Mon, 15 Nov 2021 14:41:55 -0700 Subject: [PATCH 08/82] Feature 1968 ens_ctrl (#1969) --- met/docs/Users_Guide/ensemble-stat.rst | 2 +- met/docs/Users_Guide/gen-ens-prod.rst | 2 +- met/src/tools/core/ensemble_stat/ensemble_stat.cc | 9 +++++++++ met/src/tools/other/gen_ens_prod/gen_ens_prod.cc | 15 ++++++++++++--- 4 files changed, 23 insertions(+), 5 deletions(-) diff --git a/met/docs/Users_Guide/ensemble-stat.rst b/met/docs/Users_Guide/ensemble-stat.rst index 1c131a9580..266519da4f 100644 --- a/met/docs/Users_Guide/ensemble-stat.rst +++ b/met/docs/Users_Guide/ensemble-stat.rst @@ -97,7 +97,7 @@ Optional arguments for ensemble_stat 6. To override the simple ensemble mean value of the input ensemble members for the ECNT, SSVAR, and ORANK line types, the **-ens_mean file** option specifies an ensemble mean model data file. This option replaces the **-ssvar_mean file** option from earlier versions of MET. -7. The **-ctrl file** option specifies an ensemble control member data file. The control member is included in the computation of the ensemble mean but excluded from the spread. +7. The **-ctrl file** option specifies an ensemble control member data file. The control member is included in the computation of the ensemble mean but excluded from the spread. The control file should not appear in the list of ensemble member files. 8. To filter point observations by time, use **-obs_valid_beg time** in YYYYMMDD[_HH[MMSS]] format to set the beginning of the matching observation time window. diff --git a/met/docs/Users_Guide/gen-ens-prod.rst b/met/docs/Users_Guide/gen-ens-prod.rst index 69cf1d3c73..029d256ad2 100644 --- a/met/docs/Users_Guide/gen-ens-prod.rst +++ b/met/docs/Users_Guide/gen-ens-prod.rst @@ -67,7 +67,7 @@ Required arguments gen_ens_prod Optional arguments for gen_ens_prod ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -4. The **-ctrl file** option specifies the input file for the ensemble control member. Data for this member is included in the computation of the ensemble mean, but excluded from the spread. +4. The **-ctrl file** option specifies the input file for the ensemble control member. Data for this member is included in the computation of the ensemble mean, but excluded from the spread. The control file should not appear in the **-ens** list of ensemble member files. 5. The **-log** file outputs log messages to the specified file. diff --git a/met/src/tools/core/ensemble_stat/ensemble_stat.cc b/met/src/tools/core/ensemble_stat/ensemble_stat.cc index d712c96abd..fe5d160157 100644 --- a/met/src/tools/core/ensemble_stat/ensemble_stat.cc +++ b/met/src/tools/core/ensemble_stat/ensemble_stat.cc @@ -62,6 +62,7 @@ // 031 09/13/21 Seth Linden Changed obs_qty to obs_qty_inc. // Added code for obs_qty_exc. // 032 10/07/21 Halley Gotway MET #1905 Add -ctrl option. +// 032 11/15/21 Halley Gotway MET #1968 Ensemble -ctrl error check. // //////////////////////////////////////////////////////////////////////// @@ -295,6 +296,14 @@ void process_command_line(int argc, char **argv) { // Prepend the control member, if specified if(ctrl_file.nonempty()) { + + if(ens_file_list.has(ctrl_file)) { + mlog << Error << "\nprocess_command_line() -> " + << "the ensemble control file should not appear in the list " + << "of ensemble member files:\n" << ctrl_file << "\n\n"; + exit(1); + } + ctrl_index = 0; ens_file_list.insert(ctrl_index, ctrl_file.c_str()); n_ens++; diff --git a/met/src/tools/other/gen_ens_prod/gen_ens_prod.cc b/met/src/tools/other/gen_ens_prod/gen_ens_prod.cc index 4fc7491c86..c7b48a7331 100644 --- a/met/src/tools/other/gen_ens_prod/gen_ens_prod.cc +++ b/met/src/tools/other/gen_ens_prod/gen_ens_prod.cc @@ -12,9 +12,10 @@ // // Description: // -// Mod# Date Name Description -// ---- ---- ---- ----------- -// 000 09/10/21 Halley Gotway Initial version (MET #1904). +// Mod# Date Name Description +// ---- ---- ---- ----------- +// 000 09/10/21 Halley Gotway MET #1904 Initial version. +// 001 11/15/21 Halley Gotway MET #1968 Ensemble -ctrl error check. // //////////////////////////////////////////////////////////////////////// @@ -201,6 +202,14 @@ void process_command_line(int argc, char **argv) { mlog << Debug(1) << "Ensemble Control: " << (ctrl_file.empty() ? "None" : ctrl_file.c_str()) << "\n"; + // Check for control in the list of ensemble files + if(ctrl_file.nonempty() && ens_files.has(ctrl_file)) { + mlog << Error << "\nprocess_command_line() -> " + << "the ensemble control file should not appear in the list " + << "of ensemble member files:\n" << ctrl_file << "\n\n"; + exit(1); + } + // Check for missing non-python ensemble files for(i=0; i Date: Tue, 16 Nov 2021 11:14:18 -0700 Subject: [PATCH 09/82] Feature 1809 gen prob (#1967) --- met/data/config/TCGenConfig_default | 11 +- .../table_files/met_header_columns_V10.1.txt | 2 +- met/docs/Users_Guide/tc-gen.rst | 143 ++- .../libcode/vx_tc_util/Makefile.am | 2 + .../libcode/vx_tc_util/test_read_prob.cc | 2 +- met/src/basic/vx_config/config_constants.h | 1 + met/src/basic/vx_util/stat_column_defs.h | 1 + met/src/libcode/vx_statistics/met_stats.cc | 21 + met/src/libcode/vx_statistics/met_stats.h | 1 + met/src/libcode/vx_tc_util/Makefile.am | 1 + met/src/libcode/vx_tc_util/atcf_offsets.h | 15 +- met/src/libcode/vx_tc_util/atcf_prob_line.cc | 6 +- met/src/libcode/vx_tc_util/prob_gen_info.cc | 241 +++++ met/src/libcode/vx_tc_util/prob_gen_info.h | 98 ++ met/src/libcode/vx_tc_util/prob_info_array.cc | 168 ++- met/src/libcode/vx_tc_util/prob_info_array.h | 18 +- met/src/libcode/vx_tc_util/prob_info_base.cc | 21 +- met/src/libcode/vx_tc_util/prob_info_base.h | 7 +- met/src/libcode/vx_tc_util/prob_rirw_info.cc | 15 +- met/src/libcode/vx_tc_util/prob_rirw_info.h | 4 +- met/src/tools/tc_utils/tc_gen/tc_gen.cc | 976 +++++++++++++++--- met/src/tools/tc_utils/tc_gen/tc_gen.h | 20 +- .../tools/tc_utils/tc_gen/tc_gen_conf_info.cc | 197 +++- .../tools/tc_utils/tc_gen/tc_gen_conf_info.h | 63 +- met/src/tools/tc_utils/tc_pairs/Makefile.am | 1 + met/src/tools/tc_utils/tc_pairs/tc_pairs.cc | 40 +- test/config/TCGenConfig_prob | 288 ++++++ test/hdr/met_10_1.hdr | 2 +- test/xml/unit_tc_gen.xml | 28 +- 29 files changed, 2103 insertions(+), 290 deletions(-) create mode 100644 met/src/libcode/vx_tc_util/prob_gen_info.cc create mode 100644 met/src/libcode/vx_tc_util/prob_gen_info.h create mode 100644 test/config/TCGenConfig_prob diff --git a/met/data/config/TCGenConfig_default b/met/data/config/TCGenConfig_default index 9ed75cc2c2..839d8bd21d 100644 --- a/met/data/config/TCGenConfig_default +++ b/met/data/config/TCGenConfig_default @@ -212,6 +212,11 @@ ops_method_flag = TRUE; // //////////////////////////////////////////////////////////////////////////////// +// +// Probability of genesis thresholds +// +prob_genesis_thresh = ==0.25; + // // Confidence interval alpha value // @@ -222,8 +227,12 @@ ci_alpha = 0.05; // output_flag = { fho = NONE; - ctc = BOTH; + ctc = BOTH; cts = BOTH; + pct = NONE; + pstd = NONE; + pjc = NONE; + prc = NONE; genmpr = NONE; } diff --git a/met/data/table_files/met_header_columns_V10.1.txt b/met/data/table_files/met_header_columns_V10.1.txt index c0218275ec..1dbd3c2849 100644 --- a/met/data/table_files/met_header_columns_V10.1.txt +++ b/met/data/table_files/met_header_columns_V10.1.txt @@ -28,7 +28,7 @@ V10.1 : STAT : SSVAR : VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG FCST_VALID V10.1 : STAT : VAL1L2 : VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG FCST_VALID_END OBS_LEAD OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_UNITS FCST_LEV OBS_VAR OBS_UNITS OBS_LEV OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH COV_THRESH ALPHA LINE_TYPE TOTAL UFABAR VFABAR UOABAR VOABAR UVFOABAR UVFFABAR UVOOABAR V10.1 : STAT : VL1L2 : VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG FCST_VALID_END OBS_LEAD OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_UNITS FCST_LEV OBS_VAR OBS_UNITS OBS_LEV OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH COV_THRESH ALPHA LINE_TYPE TOTAL UFBAR VFBAR UOBAR VOBAR UVFOBAR UVFFBAR UVOOBAR F_SPEED_BAR O_SPEED_BAR V10.1 : STAT : VCNT : VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG FCST_VALID_END OBS_LEAD OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_UNITS FCST_LEV OBS_VAR OBS_UNITS OBS_LEV OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH COV_THRESH ALPHA LINE_TYPE TOTAL FBAR FBAR_BCL FBAR_BCU OBAR OBAR_BCL OBAR_BCU FS_RMS FS_RMS_BCL FS_RMS_BCU OS_RMS OS_RMS_BCL OS_RMS_BCU MSVE MSVE_BCL MSVE_BCU RMSVE RMSVE_BCL RMSVE_BCU FSTDEV FSTDEV_BCL FSTDEV_BCU OSTDEV OSTDEV_BCL OSTDEV_BCU FDIR FDIR_BCL FDIR_BCU ODIR ODIR_BCL ODIR_BCU FBAR_SPEED FBAR_SPEED_BCL FBAR_SPEED_BCU OBAR_SPEED OBAR_SPEED_BCL OBAR_SPEED_BCU VDIFF_SPEED VDIFF_SPEED_BCL VDIFF_SPEED_BCU VDIFF_DIR VDIFF_DIR_BCL VDIFF_DIR_BCU SPEED_ERR SPEED_ERR_BCL SPEED_ERR_BCU SPEED_ABSERR SPEED_ABSERR_BCL SPEED_ABSERR_BCU DIR_ERR DIR_ERR_BCL DIR_ERR_BCU DIR_ABSERR DIR_ABSERR_BCL DIR_ABSERR_BCU -V10.1 : STAT : GENMPR : VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG FCST_VALID_END OBS_LEAD OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_UNITS FCST_LEV OBS_VAR OBS_UNITS OBS_LEV OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH COV_THRESH ALPHA LINE_TYPE TOTAL INDEX STORM_ID AGEN_INIT AGEN_FHR AGEN_LAT AGEN_LON AGEN_DLAND BGEN_LAT BGEN_LON BGEN_DLAND GEN_DIST GEN_TDIFF INIT_TDIFF DEV_CAT OPS_CAT +V10.1 : STAT : GENMPR : VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG FCST_VALID_END OBS_LEAD OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_UNITS FCST_LEV OBS_VAR OBS_UNITS OBS_LEV OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH COV_THRESH ALPHA LINE_TYPE TOTAL INDEX STORM_ID PROB_LEAD PROB_VAL AGEN_INIT AGEN_FHR AGEN_LAT AGEN_LON AGEN_DLAND BGEN_LAT BGEN_LON BGEN_DLAND GEN_DIST GEN_TDIFF INIT_TDIFF DEV_CAT OPS_CAT V10.1 : STAT : SSIDX : VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG FCST_VALID_END OBS_LEAD OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_UNITS FCST_LEV OBS_VAR OBS_UNITS OBS_LEV OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH COV_THRESH ALPHA LINE_TYPE FCST_MODEL REF_MODEL N_INIT N_TERM N_VLD SS_INDEX V10.1 : MODE : OBJ : VERSION MODEL N_VALID GRID_RES DESC FCST_LEAD FCST_VALID FCST_ACCUM OBS_LEAD OBS_VALID OBS_ACCUM FCST_RAD FCST_THR OBS_RAD OBS_THR FCST_VAR FCST_UNITS FCST_LEV OBS_VAR OBS_UNITS OBS_LEV OBTYPE OBJECT_ID OBJECT_CAT CENTROID_X CENTROID_Y CENTROID_LAT CENTROID_LON AXIS_ANG LENGTH WIDTH AREA AREA_THRESH CURVATURE CURVATURE_X CURVATURE_Y COMPLEXITY INTENSITY_10 INTENSITY_25 INTENSITY_50 INTENSITY_75 INTENSITY_90 INTENSITY_USER INTENSITY_SUM CENTROID_DIST BOUNDARY_DIST CONVEX_HULL_DIST ANGLE_DIFF ASPECT_DIFF AREA_RATIO INTERSECTION_AREA UNION_AREA SYMMETRIC_DIFF INTERSECTION_OVER_AREA CURVATURE_RATIO COMPLEXITY_RATIO PERCENTILE_INTENSITY_RATIO INTEREST diff --git a/met/docs/Users_Guide/tc-gen.rst b/met/docs/Users_Guide/tc-gen.rst index 61209566dc..b67f1a26b9 100644 --- a/met/docs/Users_Guide/tc-gen.rst +++ b/met/docs/Users_Guide/tc-gen.rst @@ -6,14 +6,18 @@ TC-Gen Tool Introduction ____________ -The TC-Gen tool provides verification of tropical cyclone genesis forecasts in ATCF file format. Producing reliable tropical cyclone genesis forecasts is an important metric for global numerical weather prediction models. This tool ingests deterministic model output post-processed by a genesis tracking software (e.g. GFDL vortex tracker) and ATCF format reference dataset(s) (e.g. Best Track analysis and CARQ operational tracks) and outputs categorical counts and statistics. The capability to modify the spatial and temporal tolerances that define a "hit" forecast is included to give users the ability to condition the criteria based on model performance and/or conduct sensitivity analyses. Statistical aspects are outlined in :numref:`tc-gen_stat_aspects` and practical aspects of the TC-Gen tool are described in :numref:`tc-gen_practical_info`. +The TC-Gen tool provides verification of deterministic and probabilistic tropical cyclone genesis forecasts in the ATCF file format. Producing reliable tropical cyclone genesis forecasts is an important metric for global numerical weather prediction models. This tool ingests deterministic model output post-processed by a genesis tracking software (e.g. GFDL vortex tracker), ATCF edeck files containing probability of genesis forecasts, and ATCF reference track dataset(s) (e.g. Best Track analysis and CARQ operational tracks). It writes categorical counts and statistics. The capability to modify the spatial and temporal tolerances when matching forecasts to reference genesis events, as well as scoring those matched pairs, gives users the ability to condition the criteria based on model performance and/or conduct sensitivity analyses. Statistical aspects are outlined in :numref:`tc-gen_stat_aspects` and practical aspects of the TC-Gen tool are described in :numref:`tc-gen_practical_info`. .. _tc-gen_stat_aspects: Statistical aspects ___________________ -The TC-Gen tool populates a contingency tables with hits, misses, and false alarms. As with other extreme events (where the event occurs much less frequently than the non-event), the correct negative category is not computed the non-events would dominate the contingency table. Therefore, only statistics that do not include correct negatives should be considered for this tool. The following CTS statistics are relevant: Base rate (BASER), Mean forecast (FMEAN), Frequency Bias (FBIAS), Probability of Detection (PODY), False Alarm Ratio (FAR), Critical Success Index (CSI), Gilbert Skill Score (GSS), Extreme Dependency Score (EDS), Symmetric Extreme Dependency Score (SEDS), Bias Adjusted Gilbert Skill Score (BAGSS). +The TC-Gen tool processes both deterministic and probabilistic forecasts. For deterministic forecasts specified using the **-track** command line option, it identifies genesis events in both the forecasts and reference datasets, typically Best tracks. It applies user-specified configuration options to pair up the forecast and reference genesis events and categorize each pair as a hit, miss, or false alarm. + +As with other extreme events (where the event occurs much less frequently than the non-event), the correct negative category is not computed since the non-events would dominate the contingency table. Therefore, only statistics that do not include correct negatives should be considered for this tool. The following CTS statistics are relevant: Base rate (BASER), Mean forecast (FMEAN), Frequency Bias (FBIAS), Probability of Detection (PODY), False Alarm Ratio (FAR), Critical Success Index (CSI), Gilbert Skill Score (GSS), Extreme Dependency Score (EDS), Symmetric Extreme Dependency Score (SEDS), Bias Adjusted Gilbert Skill Score (BAGSS). + +For probabilistic forecasts specified using the **-edeck** command line option, it identifies genesis events in the reference dataset. It applies user-specified configuration options to pair the forecast probabilities to the reference genesis events. These pairs are added to an Nx2 probabilistic contingency table. If the reference genesis event occurs within in the predicted time window, the pair is counted in the observation-yes column. Otherwise, it is added to the observation-no column. Other considerations for interpreting the output of the TC-Gen tool involve the size of the contingency table output. The size of the contingency table will change depending on the number of matches. Additionally, the number of misses is based on the forecast duration and interval (specified in the configuration file). This change is due to the number of model opportunities to forecast the event, which is determined by the specified duration/interval. @@ -34,8 +38,8 @@ The usage statement for tc_gen is shown below: .. code-block:: none Usage: tc_gen - -genesis path - -track path + -genesis source and/or -edeck source + -track source -config file [-out base] [-log file] @@ -46,50 +50,76 @@ TC-Gen has three required arguments and accepts optional ones. Required arguments for tc_gen ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -1. The **-genesis path** argument is the path to one or more ATCF or fort.66 (see documentation listed below) files generated by the Geophysical Fluid Dynamics Laboratory (GFDL) Vortex Tracker when run in tcgen mode or an ASCII file list or a top-level directory containing them. The **-genesis** option must be used at least once. The required file format is described in the "Output formats" section of the `GFDL Vortex Tracker users guide. `_ +1. The **-genesis source** argument is the path to one or more ATCF or fort.66 (see documentation listed below) files generated by the Geophysical Fluid Dynamics Laboratory (GFDL) Vortex Tracker when run in tcgen mode or an ASCII file list or a top-level directory containing them. The required file format is described in the "Output formats" section of the `GFDL Vortex Tracker users guide. `_ + +2. The **-edeck source** argument is the path to one or more ATCF edeck files, an ASCII file list containing them, or a top-level directory with files matching the regular expression ".dat". The probability of genesis are read from each edeck input file and verified against at the **-track** data. The **-genesis** or **-edeck** option must be used at least once. -2. The **-track path** argument is one or more ATCF reference track files or an ASCII file list or top-level directory containing them, with files ending in ".dat". This tool processes either Best track data from bdeck files, or operational track data (e.g. CARQ) from adeck files, or both. Providing both bdeck and adeck files will result in a richer dataset to match with the **-genesis** files. Both adeck and bdeck data should be provided using the **-track** option. The **-track** option must be used at least once. +3. The **-track source** argument is one or more ATCF reference track files or an ASCII file list or top-level directory containing them, with files ending in ".dat". This tool processes either Best track data from bdeck files, or operational track data (e.g. CARQ) from adeck files, or both. Providing both bdeck and adeck files will result in a richer dataset to match with the **-genesis** files. Both adeck and bdeck data should be provided using the **-track** option. The **-track** option must be used at least once. -3. The **-config** file argument indicates the name of the configuration file to be used. The contents of the configuration file are discussed below. +4. The **-config** file argument indicates the name of the configuration file to be used. The contents of the configuration file are discussed below. Optional arguments for tc_gen ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -4. The **-out base** argument indicates the path of the output file base. This argument overrides the default output file base (./tc_gen) +5. The **-out base** argument indicates the path of the output file base. This argument overrides the default output file base (./tc_gen) -5. The **-log file** option directs output and errors to the specified log file. All messages will be written to that file as well as standard out and error. Thus, users can save the messages without having to redirect the output on the command line. The default behavior is no log file. +6. The **-log file** option directs output and errors to the specified log file. All messages will be written to that file as well as standard out and error. Thus, users can save the messages without having to redirect the output on the command line. The default behavior is no log file. -6. The **-v level** option indicates the desired level of verbosity. The contents of "level" will override the default setting of 2. Setting the verbosity to 0 will make the tool run with no log messages, while increasing the verbosity above 1 will increase the amount of logging. +7. The **-v level** option indicates the desired level of verbosity. The contents of "level" will override the default setting of 2. Setting the verbosity to 0 will make the tool run with no log messages, while increasing the verbosity above 1 will increase the amount of logging. -The TC-Gen tool implements the following logic: +Scoring Logic +^^^^^^^^^^^^^ -* Parse the forecast genesis data and identify forecast genesis events separately for each model present. +The TC-Gen tool implements the following logic: * Parse the Best and operational track data, and identify Best track genesis events. Note that Best tracks with a cyclone number greater than 50 are automatically discarded from the analysis. Large cyclone numbers are used for pre-season testing or to track invests prior to a storm actually forming. Running this tool at verbosity level 6 (-v 6) prints details about which tracks are discarded. -* Loop over the filters defined in the configuration file and apply the following logic for each. +* For **-track** inputs: + + * Parse the forecast genesis data and identify forecast genesis events separately for each model present. - * For each Best track genesis event meeting the filter critera, determine the initialization and lead times for which the model had an opportunity to forecast that genesis event. Store an unmatched genesis pair for each case. + * Loop over the filters defined in the configuration file and apply the following logic for each. + + * For each Best track genesis event meeting the filter critera, determine the initialization and lead times for which the model had an opportunity to forecast that genesis event. Store an unmatched genesis pair for each case. - * For each forecast genesis event, search for a matching Best track. A configurable boolean option controls whether all Best track points are considered for a match or only the single Best track genesis point. A match occurs if the Best track point valid time is within a configurable window around the forecast genesis time and the Best track point location is within a configurable radius of the forecast genesis location. If a Best track match is found, store the storm ID. + * For each forecast genesis event, search for a matching Best track. A configurable boolean option controls whether all Best track points are considered for a match or only the single Best track genesis point. A match occurs if the Best track point valid time is within a configurable window around the forecast genesis time and the Best track point location is within a configurable radius of the forecast genesis location. If a Best track match is found, store the storm ID. - * In no Best track match is found, apply the same logic to search the operational track points with lead time of 0 hours. If an operational match is found, store the storm ID. + * If no Best track match is found, apply the same logic to search the operational track points with lead time of 0 hours. If an operational match is found, store the storm ID. - * If a matching storm ID is found, match the forecast genesis event to the Best track genesis event for that storm ID. + * If a matching storm ID is found, match the forecast genesis event to the Best track genesis event for that storm ID. - * If no matching storm ID is found, store an unmatched pair for the genesis forecast. + * If no matching storm ID is found, store an unmatched pair for the genesis forecast. - * Loop through the genesis pairs and populate contingency tables using two methods, the developement (dev) and operational (ops) methods. For each pair, if the forecast genesis event is unmatched, score it as a dev and ops FALSE ALARM. If the Best track genesis event is unmatched, score it as a dev and ops MISS. Score each matched genesis pair as follows: + * Loop through the genesis pairs and populate contingency tables using two methods, the development (dev) and operational (ops) methods. For each pair, if the forecast genesis event is unmatched, score it as a dev and ops FALSE ALARM. If the Best track genesis event is unmatched, score it as a dev and ops MISS. Score each matched genesis pair as follows: - * If the forecast initialization time is at or after the Best track genesis event, DISCARD this case and exclude it from the statistics. + * If the forecast initialization time is at or after the Best track genesis event, DISCARD this case and exclude it from the statistics. - * Compute the difference between the forecast and Best track genesis events in time and space. If they are both within the configurable tolerance, score it as a dev HIT. If not, score it as a dev FALSE ALARM. + * Compute the difference between the forecast and Best track genesis events in time and space. If they are both within the configurable tolerance, score it as a dev HIT. If not, score it as a dev FALSE ALARM. - * Compute the difference between the Best track genesis time and model initialization time. If it is within the configurable tolerance, score it as an ops HIT. If not, score it as an ops FALSE ALARM. + * Compute the difference between the Best track genesis time and model initialization time. If it is within the configurable tolerance, score it as an ops HIT. If not, score it as an ops FALSE ALARM. + + * Do not count any CORRECT NEGATIVES. + + * Report the contingency table hits, misses, and false alarms separately for each forecast model and configuration file filter. The development (dev) scoring method is indicated in the output as *GENESIS_DEV* while the operational (ops) scoring method is indicated as *GENESIS_OPS*. + +* For **-edeck** inputs: + + * Parse the ATCF edeck files. Ignore any lines not containing "GN" and "genFcst", which indicate a genesis probability forecast. Also, ignore any lines which do not contain a predicted genesis location (latitude and longitude) or genesis time. + + * Loop over the filters defined in the configuration file and apply the following logic for each. - * Do not count any CORRECT NEGATIVES. + * Subset the genesis probability forecasts based on the current filter criteria. Typically, genesis probability forecast are provided for multiple lead times. Create separate Nx2 probabilistic contingency tables for each unique combination of predicted lead time and model name. + + * For each genesis probability forecast, search for a matching Best track. A configurable boolean option controls whether all Best track points are considered for a match or only the single Best track genesis point. A match occurs if the Best track point valid time is within a configurable window around the forecast genesis time and the Best track point location is within a configurable radius of the forecast genesis location. If a Best track match is found, store the storm ID. + + * If no Best track match is found, apply the same logic to search the operational track points with lead time of 0 hours. If an operational match is found, store the storm ID. + + * If no matching storm ID is found, add the unmatched forecast to the observation-no column of the Nx2 probabilistic contingency table. + + * If a matching storm ID is found, check whether that storm's genesis occurred within the predicted time window: between the forecast initialization time and the predicted lead time. If so, add the matched forecast to the observation-yes column. If not, add it to observation-no column. + + * Report the Nx2 probabilistic contingency table counts and statistics for each forecast model, lead time, and configuration file filter. These counts and statistics are identified in the output files as *PROB_GENESIS*. -* Report the contingency table hits, misses, and false alarms separately for each forecast model and configuration file filter. The development (dev) scoring method is indicated in the output as *GENESIS_DEV* while the operational (ops) scoring method is indicated as *GENESIS_OPS*. tc_gen configuration file ~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -381,18 +411,31 @@ The **nc_pairs_grid** entry is a string which defines the grid to be used for th ______________________ +.. code-block:: none + + prob_genesis_thresh = ==0.25; + +The **prob_genesis_thresh** entry defines the probability thresholds used to create the output Nx2 contingency table when verifying edeck probability of genesis forecasts. The default is probability bins of width 0.25. These probabilities may be specified as a list (>0.00,>0.25,>0.50,>0.75,>1.00) or using shorthand notation (==0.25) for bins of equal width. + +______________________ + .. code-block:: none ci_alpha = 0.05; output_flag = { - fho = BOTH; - ctc = BOTH; - cts = BOTH; + fho = BOTH; + ctc = BOTH; + cts = BOTH; + pct = NONE; + pstd = NONE; + pjc = NONE; + prc = NONE; + genmpr = NONE; } dland_file = "MET_BASE/tc_data/dland_global_tenth_degree.nc"; version = "VN.N"; -The configuration options listed above are common to many MET tools and are described in :numref:`config_options`. Note that TC-Gen writes output for 2x2 contingency tables to the **FHO, CTC**, and **CTS** line types. +The configuration options listed above are common to many MET tools and are described in :numref:`config_options`. TC-Gen writes output for 2x2 contingency tables to the **FHO**, **CTC**, and **CTS** line types when verifying deterministic genesis forecasts specified using the **-track** command line option. TC-Gen writes output for Nx2 probabilistic contingency tables to the **PCT**, **PSTD**, **PJC**, and **PRC** line types when verifying the probability of genesis forecasts specified using the **-edeck** command line option. Note that the **genmpr** line type is specific to TC-Gen and describes individual genesis matched pairs. tc_gen output ~~~~~~~~~~~~~ @@ -440,7 +483,7 @@ TC-Gen produces output in STAT and, optionally, ASCII and NetCDF formats. The AS - Maximum Best track valid time in YYYYMMDD_HHMMSS format * - 10 - FCST_VAR - - Genesis methodology + - Genesis methodology (GENESIS_DEV, GENESIS_OPS, or PROB_GENESIS) * - 11 - FCST_UNITS - Does not apply and is set to NA @@ -449,7 +492,7 @@ TC-Gen produces output in STAT and, optionally, ASCII and NetCDF formats. The AS - Does not apply and is set to NA * - 13 - OBS_VAR - - Genesis methodology + - Genesis methodology (GENESIS_DEV, GENESIS_OPS, or PROB_GENESIS) * - 14 - OBS_UNITS - Does not apply and is set to NA @@ -515,44 +558,50 @@ TC-Gen produces output in STAT and, optionally, ASCII and NetCDF formats. The AS - STORM_ID - BBCCYYYY designation of storm (basin, cyclone number, and year) * - 28 + - PROB_LEAD + - Lead time in HHH format for the predicted probability of genesis (only for **-edeck** inputs) + * - 29 + - PROB_VAL + - Predicted probability of genesis (only for **-edeck** inputs) + * - 30 - AGEN_INIT - Forecast initialization time - * - 29 + * - 31 - AGEN_FHR - Forecast hour of genesis event - * - 30 + * - 32 - AGEN_LAT - Latitude position of the forecast genesis event - * - 31 + * - 33 - AGEN_LON - Longitude position of the forecast genesis event - * - 32 + * - 34 - AGEN_DLAND - Forecast genesis event distance to land (nm) - * - 33 + * - 35 - BGEN_LAT - Latitude position of the verifying Best track genesis event - * - 34 + * - 36 - BGEN_LON - Longitude position of the verifying Best track genesis event - * - 35 + * - 37 - BGEN_DLAND - Best track genesis event distance to land (nm) - * - 36 + * - 38 - GEN_DIST - - Distance between the forecast and Best track genesis events (km) - * - 37 + - Distance between the forecast and Best track genesis events (km) (only for **-track** inputs) + * - 39 - GEN_TDIFF - - Forecast minus Best track genesis time in HHMMSS format - * - 38 + - Forecast minus Best track genesis time in HHMMSS format (only for **-track** inputs) + * - 40 - INIT_TDIFF - - Best track genesis minus forecast initialization time in HHMMSS format - * - 39 + - Best track genesis minus forecast initialization time in HHMMSS format (only for **-track** inputs) + * - 41 - DEV_CAT - - Development methodology category (FYOY, FYON, FNOY, or DISCARD) - * - 40 + - Category for the development methodology (FYOY, FYON, FNOY, or DISCARD) (only for **-track** inputs) + * - 42 - OPS_CAT - - Operational methodology category (FYOY, FYON, FNOY, or DISCARD) + - Category for the operational methodology (FYOY, FYON, FNOY, or DISCARD for **-track** inputs and FYOY or FYON for **-edeck** inputs) .. _table_TG_var_NetCDF_matched_pair_out: diff --git a/met/internal_tests/libcode/vx_tc_util/Makefile.am b/met/internal_tests/libcode/vx_tc_util/Makefile.am index a5fc96c5d0..a1d0fed6c5 100644 --- a/met/internal_tests/libcode/vx_tc_util/Makefile.am +++ b/met/internal_tests/libcode/vx_tc_util/Makefile.am @@ -43,6 +43,7 @@ test_read_LDADD = -lvx_stat_out \ -lvx_config \ -lvx_gsl_prob \ -lvx_cal \ + -lvx_nav \ -lvx_util \ -lvx_math \ -lvx_color \ @@ -75,6 +76,7 @@ test_read_prob_LDADD = -lvx_stat_out \ -lvx_config \ -lvx_gsl_prob \ -lvx_cal \ + -lvx_nav \ -lvx_util \ -lvx_math \ -lvx_color \ diff --git a/met/internal_tests/libcode/vx_tc_util/test_read_prob.cc b/met/internal_tests/libcode/vx_tc_util/test_read_prob.cc index 402876f47c..eea9a30e2d 100644 --- a/met/internal_tests/libcode/vx_tc_util/test_read_prob.cc +++ b/met/internal_tests/libcode/vx_tc_util/test_read_prob.cc @@ -53,7 +53,7 @@ int main(int argc, char *argv[]) { while(f >> p_line) { // Add the current line to the array of probs - probs.add(p_line); + probs.add(p_line, bad_data_double, false); // Increment the line count count++; diff --git a/met/src/basic/vx_config/config_constants.h b/met/src/basic/vx_config/config_constants.h index a46559ca02..01c598bd72 100644 --- a/met/src/basic/vx_config/config_constants.h +++ b/met/src/basic/vx_config/config_constants.h @@ -1097,6 +1097,7 @@ static const char conf_key_ops_hit_window[] = "ops_hit_window"; static const char conf_key_discard_init_post_genesis_flag[] = "discard_init_post_genesis_flag"; static const char conf_key_dev_method_flag[] = "dev_method_flag"; static const char conf_key_ops_method_flag[] = "ops_method_flag"; +static const char conf_key_prob_genesis_thresh[] = "prob_genesis_thresh"; static const char conf_key_fcst_fy_oy[] = "fcst_fy_oy"; static const char conf_key_fcst_fy_on[] = "fcst_fy_on"; static const char conf_key_fcst_tracks[] = "fcst_tracks"; diff --git a/met/src/basic/vx_util/stat_column_defs.h b/met/src/basic/vx_util/stat_column_defs.h index b8ddc583bc..9b21762513 100644 --- a/met/src/basic/vx_util/stat_column_defs.h +++ b/met/src/basic/vx_util/stat_column_defs.h @@ -320,6 +320,7 @@ static const char * ssidx_columns [] = { static const char * genmpr_columns [] = { "TOTAL", "INDEX", "STORM_ID", + "PROB_LEAD", "PROB_VAL", "AGEN_INIT", "AGEN_FHR", "AGEN_LAT", "AGEN_LON", "AGEN_DLAND", "BGEN_LAT", "BGEN_LON", "BGEN_DLAND", diff --git a/met/src/libcode/vx_statistics/met_stats.cc b/met/src/libcode/vx_statistics/met_stats.cc index 2975b03432..3880ab204a 100644 --- a/met/src/libcode/vx_statistics/met_stats.cc +++ b/met/src/libcode/vx_statistics/met_stats.cc @@ -2424,6 +2424,27 @@ void PCTInfo::allocate_n_alpha(int i) { //////////////////////////////////////////////////////////////////////// +void PCTInfo::set_fthresh(const ThreshArray &ta) { + + // Expand the probability thresholds, as needed + fthresh = string_to_prob_thresh(ta.get_str().c_str()); + + // Validate the threshold settings + check_prob_thresh(fthresh, true); + + // Store the values in an array + NumArray prob_vals; + for(int i=0; i " << "skipping ATCF line type (" << atcflinetype_to_string(Type) << ")\n"; diff --git a/met/src/libcode/vx_tc_util/prob_gen_info.cc b/met/src/libcode/vx_tc_util/prob_gen_info.cc new file mode 100644 index 0000000000..c0db7ce9fb --- /dev/null +++ b/met/src/libcode/vx_tc_util/prob_gen_info.cc @@ -0,0 +1,241 @@ +// *=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=* +// ** Copyright UCAR (c) 1992 - 2021 +// ** University Corporation for Atmospheric Research (UCAR) +// ** National Center for Atmospheric Research (NCAR) +// ** Research Applications Lab (RAL) +// ** P.O.Box 3000, Boulder, Colorado, 80307-3000, USA +// *=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=* + +//////////////////////////////////////////////////////////////////////// + +using namespace std; + +#include +#include +#include +#include +#include +#include + +#include "nav.h" + +#include "prob_gen_info.h" +#include "atcf_offsets.h" + +//////////////////////////////////////////////////////////////////////// +// +// Code for class ProbGenInfo +// +//////////////////////////////////////////////////////////////////////// + +ProbGenInfo::ProbGenInfo() { + + init_from_scratch(); +} + +//////////////////////////////////////////////////////////////////////// + +ProbGenInfo::~ProbGenInfo() { + + clear(); +} + +//////////////////////////////////////////////////////////////////////// + +ProbGenInfo::ProbGenInfo(const ProbGenInfo & t) { + + init_from_scratch(); + + assign(t); +} + +//////////////////////////////////////////////////////////////////////// + +ProbGenInfo & ProbGenInfo::operator=(const ProbGenInfo & t) { + + if(this == &t) return(*this); + + assign(t); + + return(*this); +} + +//////////////////////////////////////////////////////////////////////// + +void ProbGenInfo::init_from_scratch() { + + clear(); + + return; +} + +//////////////////////////////////////////////////////////////////////// + +void ProbGenInfo::clear() { + + ProbInfoBase::clear(); + + Initials.clear(); + GenOrDis.clear(); + GenesisTime = (unixtime) 0; + GenesisLead = 0; + + return; +} + +//////////////////////////////////////////////////////////////////////// + +void ProbGenInfo::dump(ostream &out, int indent_depth) const { + Indent prefix(indent_depth); + + ProbInfoBase::dump(out, indent_depth); + + out << prefix << "Initials = \"" << Initials.contents() << "\"\n"; + out << prefix << "GenOrDis = \"" << GenOrDis.contents() << "\"\n"; + out << prefix << "GenesisTime = " << unix_to_yyyymmdd_hhmmss(GenesisTime) << "\n"; + out << prefix << "GenesisLead = " << sec_to_hhmmss(GenesisLead) << "\n"; + + out << flush; + + return; + +} + +//////////////////////////////////////////////////////////////////////// + +ConcatString ProbGenInfo::serialize() const { + ConcatString s; + + s << ProbInfoBase::serialize() + << ", ProbGenInfo: " + << "Initials = \"" << Initials << "\"" + << ", GenOrDis = \"" << GenOrDis << "\"" + << ", GenesisTime = " << unix_to_yyyymmdd_hhmmss(GenesisTime) + << ", GenesisLead = " << sec_to_hhmmss(GenesisLead) << "\n"; + + return(s); +} + +//////////////////////////////////////////////////////////////////////// + +ConcatString ProbGenInfo::serialize_r(int n, int indent_depth) const { + ConcatString s; + + s << ProbInfoBase::serialize_r(n, indent_depth); + + return(s); +} + +//////////////////////////////////////////////////////////////////////// + +void ProbGenInfo::assign(const ProbGenInfo &p) { + + clear(); + + ProbInfoBase::assign(p); + + Initials = p.Initials; + GenOrDis = p.GenOrDis; + GenesisTime = p.GenesisTime; + GenesisLead = p.GenesisLead; + + return; +} + +//////////////////////////////////////////////////////////////////////// + +int ProbGenInfo::genesis_fhr() const { + return(is_bad_data(GenesisLead) ? + bad_data_int : + nint((double) GenesisLead/sec_per_hour)); +} + +//////////////////////////////////////////////////////////////////////// + +void ProbGenInfo::initialize(const ATCFProbLine &l, double dland) { + + clear(); + + ProbInfoBase::initialize(l, dland); + + Initials = l.get_item(ProbGenInitialsOffset); + GenOrDis = l.get_item(ProbGenOrDisOffset); + + // Store an empty string as unixtime 0 + GenesisTime = (l.get_item(ProbGenTimeOffset).empty() ? + (unixtime) 0 : + parse_time(l.get_item(ProbGenTimeOffset).c_str())); + GenesisLead = (GenesisTime == 0 ? bad_data_int : + GenesisTime - InitTime); + + return; +} + +//////////////////////////////////////////////////////////////////////// + +bool ProbGenInfo::is_match(const ATCFProbLine &l) const { + + if(!ProbInfoBase::is_match(l)) return(false); + + unixtime gen_ut = (l.get_item(ProbGenTimeOffset).empty() ? + (unixtime) 0 : + parse_time(l.get_item(ProbGenTimeOffset).c_str())); + + return(GenesisTime == gen_ut); +} + +//////////////////////////////////////////////////////////////////////// + +bool ProbGenInfo::add(const ATCFProbLine &l, double dland, bool check_dup) { + + // Check for duplicates + if(check_dup) { + if(has(l)) { + mlog << Warning + << "\nProbGenInfo::add(const ATCFProbLine &l, bool check_dup) -> " + << "skipping duplicate ATCF line:\n" + << l.get_line() << "\n\n"; + return(false); + } + } + + // Initialize the header information, if necessary + if(Type == NoATCFLineType) initialize(l, dland); + + // Check for matching header information + if(!is_match(l)) return(false); + + // Add probability information + NProb++; + Prob.add(l.prob()); + ProbItem.add(l.prob_item()); + + // Store the ATCFProbLine that was just added + if(check_dup) ProbLines.add(l.get_line()); + + return(true); +} + +//////////////////////////////////////////////////////////////////////// + +bool ProbGenInfo::is_match(const TrackPoint &p, const double rad, + const int beg, const int end) const { + + // Check for matching in time and space + return(p.valid() >= (GenesisTime + beg) && + p.valid() <= (GenesisTime + end) && + gc_dist(Lat, Lon, p.lat(), p.lon()) <= rad); +} + +//////////////////////////////////////////////////////////////////////// + +bool ProbGenInfo::is_match(const GenesisInfo &gi, const double rad, + const int beg, const int end) const { + + // Input genesis point + const TrackPoint *p = gi.genesis(); + + return(p ? is_match(*p, rad, beg, end) : false); +} + +//////////////////////////////////////////////////////////////////////// diff --git a/met/src/libcode/vx_tc_util/prob_gen_info.h b/met/src/libcode/vx_tc_util/prob_gen_info.h new file mode 100644 index 0000000000..af97bb03fa --- /dev/null +++ b/met/src/libcode/vx_tc_util/prob_gen_info.h @@ -0,0 +1,98 @@ +// *=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=* +// ** Copyright UCAR (c) 1992 - 2021 +// ** University Corporation for Atmospheric Research (UCAR) +// ** National Center for Atmospheric Research (NCAR) +// ** Research Applications Lab (RAL) +// ** P.O.Box 3000, Boulder, Colorado, 80307-3000, USA +// *=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=* + +//////////////////////////////////////////////////////////////////////// + +#ifndef __VX_PROB_GEN_INFO_H__ +#define __VX_PROB_GEN_INFO_H__ + +//////////////////////////////////////////////////////////////////////// + +#include + +#include "prob_info_base.h" +#include "genesis_info.h" + +#include "vx_util.h" + +//////////////////////////////////////////////////////////////////////// +// +// ProbGenInfo class stores probability of rapid intensification. +// +//////////////////////////////////////////////////////////////////////// + +class ProbGenInfo : public ProbInfoBase { + + private: + + void init_from_scratch(); + void assign(const ProbGenInfo &); + + // Probability of Genesis specific values + ConcatString Initials; + ConcatString GenOrDis; + unixtime GenesisTime; + int GenesisLead; + + public: + + ProbGenInfo(); + ~ProbGenInfo(); + ProbGenInfo(const ProbGenInfo &); + ProbGenInfo & operator=(const ProbGenInfo &); + + void clear(); + + void dump(ostream &, int = 0) const; + ConcatString serialize() const; + ConcatString serialize_r(int, int = 0) const; + + // + // set stuff + // + + void set_best_gen(const GenesisInfo *); + + // + // get stuff + // + + const ConcatString & initials() const; + const ConcatString & gen_or_dis() const; + unixtime genesis_time() const; + int genesis_lead() const; + int genesis_fhr() const; + const GenesisInfo * best_gen() const; + + // + // do stuff + // + + void initialize(const ATCFProbLine &, double); + bool is_match (const ATCFProbLine &) const; + bool add (const ATCFProbLine &, double, bool check_dup = false); + + bool is_match (const TrackPoint &, + const double, const int, const int) const; + bool is_match (const GenesisInfo &, + const double, const int, const int) const; + +}; + +//////////////////////////////////////////////////////////////////////// + +inline const ConcatString & ProbGenInfo::initials() const { return(Initials); } +inline const ConcatString & ProbGenInfo::gen_or_dis() const { return(GenOrDis); } +inline unixtime ProbGenInfo::genesis_time() const { return(GenesisTime); } +inline int ProbGenInfo::genesis_lead() const { return(GenesisLead); } + +//////////////////////////////////////////////////////////////////////// + +#endif /* __VX_PROB_GEN_INFO_H__ */ + +//////////////////////////////////////////////////////////////////////// diff --git a/met/src/libcode/vx_tc_util/prob_info_array.cc b/met/src/libcode/vx_tc_util/prob_info_array.cc index 5a8e91844b..64809049ec 100644 --- a/met/src/libcode/vx_tc_util/prob_info_array.cc +++ b/met/src/libcode/vx_tc_util/prob_info_array.cc @@ -72,8 +72,9 @@ void ProbInfoArray::init_from_scratch() { void ProbInfoArray::clear() { - // Erase the entire vector + // Erase the entire vectors ProbRIRW.erase(ProbRIRW.begin(), ProbRIRW.end()); + ProbGen.erase(ProbGen.begin(), ProbGen.end()); return; } @@ -82,15 +83,23 @@ void ProbInfoArray::clear() { void ProbInfoArray::dump(ostream &out, int indent_depth) const { Indent prefix(indent_depth); + int i; out << prefix << "ProbInfoArray:\n" - << prefix << "NProbRIRW = " << n_prob_rirw() << "\n"; + << prefix << "NProbRIRW = " << n_prob_rirw() << "\n"; - for(unsigned int i=0; i 0 ) { + s << prefix << serialize() << ", ProbRIRW:\n"; + } + for(i=0; i 0 ) { + s << prefix << serialize() << ", ProbGen:\n"; + } + + for(i=0; i= n_probs())) { - mlog << Error - << "\nProbInfoBase * ProbInfoArray::operator[] -> " + mlog << Error << "\nProbInfoBase * ProbInfoArray::operator[] -> " << "range check error for index value " << n << "\n\n"; exit(1); } - // Return a base pointer to the n-th probability - // Need to revise for each new probability vector + // Get a base pointer to the n-th probability + const ProbInfoBase *ptr; + + if(ProbRIRW.size() > 0 && n < ProbRIRW.size()) { + ptr = &ProbRIRW[n]; + } + else { + n -= ProbRIRW.size(); + ptr = &ProbGen[n]; + } - return(&ProbRIRW[n]); + return(ptr); } //////////////////////////////////////////////////////////////////////// -const ProbRIRWInfo & ProbInfoArray::prob_rirw(int n) const { +ProbRIRWInfo & ProbInfoArray::prob_rirw(int n) { // Check range if((n < 0) || (n >= (int) ProbRIRW.size())) { - mlog << Error - << "\nProbRIRWInfo & ProbInfoArray::prob_rirw(int) -> " + mlog << Error << "\nProbRIRWInfo & ProbInfoArray::prob_rirw(int) -> " << "range check error for index value " << n << "\n\n"; exit(1); } @@ -171,14 +195,49 @@ const ProbRIRWInfo & ProbInfoArray::prob_rirw(int n) const { //////////////////////////////////////////////////////////////////////// -bool ProbInfoArray::add(const ATCFProbLine &l, bool check_dup) { +ProbGenInfo & ProbInfoArray::prob_gen(int n) { + + // Check range + if((n < 0) || (n >= (int) ProbGen.size())) { + mlog << Error << "\nProbGenInfo & ProbInfoArray::prob_gen(int) -> " + << "range check error for index value " << n << "\n\n"; + exit(1); + } + + return(ProbGen[n]); +} + +//////////////////////////////////////////////////////////////////////// + +int ProbInfoArray::n_technique() const { + StringArray sa; + + // Count the number of unique technique names + for(int i=0; i 100) { - mlog << Debug(4) - << "\nbool ProbInfoArray::add() -> " + mlog << Debug(4) + << "bool ProbInfoArray::add() -> " << "skipping probability value (" << l.prob() - << ") outside of range (0, 100).\n\n"; + << ") outside of range (0, 100).\n"; return(false); } @@ -187,27 +246,61 @@ bool ProbInfoArray::add(const ATCFProbLine &l, bool check_dup) { case(ATCFLineType_ProbRI): - // Check for no entries or a mismatch with the latest entry - if( ProbRIRW.size() == 0 || - (ProbRIRW.size() > 0 && - !ProbRIRW[ProbRIRW.size()-1].add(l, check_dup))) { - - // Store new entry + // Add line to an existing entry + if(ProbRIRW.size() > 0 && + ProbRIRW[ProbRIRW.size()-1].add(l, dland, check_dup)) { + status = true; + } + // Add a new entry + else { ProbRIRWInfo ri; - ri.add(l, check_dup); + ri.add(l, dland, check_dup); ProbRIRW.push_back(ri); + status = true; + } + break; + + case(ATCFLineType_ProbGN): + + // Add line to an existing entry + if(ProbGen.size() > 0 && + ProbGen[ProbGen.size()-1].add(l, dland, check_dup)) { + status = true; + } + // Add a new entry + else { + ProbGenInfo gi; + gi.add(l, dland, check_dup); + + // Check for the expected genesis type and predicted location + if(gi.gen_or_dis() != "genFcst") { + mlog << Debug(4) + << "bool ProbInfoArray::add() -> " + << "skipping ATCF " << atcflinetype_to_string(ATCFLineType_ProbGN) + << " line with non-genesis probability type (" + << gi.gen_or_dis() << " != genFcst).\n"; + } + else if(is_bad_data(gi.lat()) || is_bad_data(gi.lon())) { + mlog << Debug(4) + << "bool ProbInfoArray::add() -> " + << "skipping ATCF " << atcflinetype_to_string(ATCFLineType_ProbGN) + << " line with no predicted genesis location.\n"; + } + else { + ProbGen.push_back(gi); + status = true; + } } break; default: - mlog << Warning - << "\nbool ProbInfoArray::add() -> " + mlog << Warning << "\nbool ProbInfoArray::add() -> " << "unexpected ATCF line type (" << atcflinetype_to_string(l.type()) << ")\n\n"; - return(false); + status = false; } - return(true); + return(status); } //////////////////////////////////////////////////////////////////////// @@ -218,3 +311,10 @@ void ProbInfoArray::add(const ProbRIRWInfo &rirw) { } //////////////////////////////////////////////////////////////////////// + +void ProbInfoArray::add(const ProbGenInfo &gi) { + ProbGen.push_back(gi); + return; +} + +//////////////////////////////////////////////////////////////////////// diff --git a/met/src/libcode/vx_tc_util/prob_info_array.h b/met/src/libcode/vx_tc_util/prob_info_array.h index 1228b91880..881cf3dcfb 100644 --- a/met/src/libcode/vx_tc_util/prob_info_array.h +++ b/met/src/libcode/vx_tc_util/prob_info_array.h @@ -19,6 +19,7 @@ #include "atcf_prob_line.h" #include "prob_info_base.h" #include "prob_rirw_info.h" +#include "prob_gen_info.h" #include "vx_util.h" @@ -36,6 +37,7 @@ class ProbInfoArray { void assign(const ProbInfoArray &); vector ProbRIRW; + vector ProbGen; public: @@ -58,21 +60,27 @@ class ProbInfoArray { const ProbInfoBase * operator[](int) const; int n_prob_rirw() const; - const ProbRIRWInfo & prob_rirw(int) const; + ProbRIRWInfo & prob_rirw(int); + + int n_prob_gen() const; + ProbGenInfo & prob_gen(int); + + int n_technique() const; // // do stuff // - bool add(const ATCFProbLine &, bool check_dup = false); + bool add(const ATCFProbLine &, double dland, bool check_dup = false); void add(const ProbRIRWInfo &); - + void add(const ProbGenInfo &); }; //////////////////////////////////////////////////////////////////////// -inline int ProbInfoArray::n_probs() const { return(ProbRIRW.size()); } -inline int ProbInfoArray::n_prob_rirw() const { return(ProbRIRW.size()); } +inline int ProbInfoArray::n_probs() const { return(ProbRIRW.size() + ProbGen.size()); } +inline int ProbInfoArray::n_prob_rirw() const { return(ProbRIRW.size()); } +inline int ProbInfoArray::n_prob_gen() const { return(ProbGen.size()); } //////////////////////////////////////////////////////////////////////// diff --git a/met/src/libcode/vx_tc_util/prob_info_base.cc b/met/src/libcode/vx_tc_util/prob_info_base.cc index f8d3bb791e..fb4147226c 100644 --- a/met/src/libcode/vx_tc_util/prob_info_base.cc +++ b/met/src/libcode/vx_tc_util/prob_info_base.cc @@ -79,6 +79,7 @@ void ProbInfoBase::clear() { ValidTime = (unixtime) 0; Lat = bad_data_double; Lon = bad_data_double; + DLand = bad_data_double; NProb = 0; Prob.clear(); ProbItem.clear(); @@ -101,6 +102,7 @@ void ProbInfoBase::dump(ostream &out, int indent_depth) const { out << prefix << "ValidTime = \"" << (ValidTime > 0 ? unix_to_yyyymmdd_hhmmss(ValidTime).text() : na_str) << "\"\n"; out << prefix << "Lat = " << Lat << "\n"; out << prefix << "Lon = " << Lon << "\n"; + out << prefix << "DLand = " << DLand << "\n"; out << prefix << "NProb = " << NProb << "\n"; out << prefix << "Prob:" << "\n"; Prob.dump(out, indent_depth+1); @@ -130,6 +132,7 @@ ConcatString ProbInfoBase::serialize() const { << ", ValidTime = \"" << (ValidTime > 0 ? unix_to_yyyymmdd_hhmmss(ValidTime).text() : na_str) << "\"" << ", Lat = " << Lat << ", Lon = " << Lon + << ", DLand = " << DLand << ", NProb = " << NProb; return(s); @@ -155,6 +158,14 @@ ConcatString ProbInfoBase::serialize_r(int n, int indent_depth) const { //////////////////////////////////////////////////////////////////////// +void ProbInfoBase::set_dland(double d) { + DLand = d; + + return; +} + +//////////////////////////////////////////////////////////////////////// + void ProbInfoBase::assign(const ProbInfoBase &t) { clear(); @@ -168,6 +179,7 @@ void ProbInfoBase::assign(const ProbInfoBase &t) { ValidTime = t.ValidTime; Lat = t.Lat; Lon = t.Lon; + DLand = t.DLand; NProb = t.NProb; Prob = t.Prob; ProbItem = t.ProbItem; @@ -178,7 +190,7 @@ void ProbInfoBase::assign(const ProbInfoBase &t) { //////////////////////////////////////////////////////////////////////// -void ProbInfoBase::initialize(const ATCFProbLine &l) { +void ProbInfoBase::initialize(const ATCFProbLine &l, double dland) { clear(); @@ -192,6 +204,7 @@ void ProbInfoBase::initialize(const ATCFProbLine &l) { ValidTime = l.valid(); Lat = l.lat(); Lon = l.lon(); + DLand = dland; return; } @@ -206,7 +219,6 @@ bool ProbInfoBase::is_match(const ATCFProbLine &l) const { Cyclone == l.cyclone_number() && Technique == l.technique() && InitTime == l.warning_time() && - ValidTime == l.valid() && Lat == l.lat() && Lon == l.lon()); } @@ -239,7 +251,7 @@ bool ProbInfoBase::is_match(const TrackInfo &t) const { //////////////////////////////////////////////////////////////////////// -bool ProbInfoBase::add(const ATCFProbLine &l, bool check_dup) { +bool ProbInfoBase::add(const ATCFProbLine &l, double dland, bool check_dup) { // Check for duplicates if(check_dup) { @@ -253,7 +265,7 @@ bool ProbInfoBase::add(const ATCFProbLine &l, bool check_dup) { } // Initialize the header information, if necessary - if(Type == NoATCFLineType) initialize(l); + if(Type == NoATCFLineType) initialize(l, dland); // Check for matching header information if(!is_match(l)) return(false); @@ -297,6 +309,7 @@ void ProbInfoBase::set(const TCStatLine &l) { ValidTime = l.valid(); Lat = atof(l.get_item("ALAT")); Lon = atof(l.get_item("ALON")); + DLand = atof(l.get_item("ADLAND")); NProb = atoi(l.get_item("N_THRESH")); for(int i=1; i<=NProb; i++) { cs << cs_erase << "PROB_" << i; diff --git a/met/src/libcode/vx_tc_util/prob_info_base.h b/met/src/libcode/vx_tc_util/prob_info_base.h index 4f416db319..fff06def24 100644 --- a/met/src/libcode/vx_tc_util/prob_info_base.h +++ b/met/src/libcode/vx_tc_util/prob_info_base.h @@ -50,6 +50,7 @@ class ProbInfoBase { // Location information double Lat; double Lon; + double DLand; // Probability information int NProb; @@ -93,6 +94,7 @@ class ProbInfoBase { int valid_hour() const; double lat() const; double lon() const; + double dland() const; int n_prob() const; double prob(int i) const; double prob_item(int i) const; @@ -101,11 +103,11 @@ class ProbInfoBase { // do stuff // - virtual void initialize(const ATCFProbLine &); + virtual void initialize(const ATCFProbLine &, double); virtual bool is_match (const ATCFProbLine &) const; bool is_match (const TrackInfo &) const; bool has (const ATCFProbLine &) const; - virtual bool add (const ATCFProbLine &, bool check_dup = false); + virtual bool add (const ATCFProbLine &, double, bool check_dup = false); virtual void set (const TCStatLine &); }; @@ -123,6 +125,7 @@ inline unixtime ProbInfoBase::valid() const { return(ValidT inline int ProbInfoBase::valid_hour() const { return(unix_to_sec_of_day(ValidTime)); } inline double ProbInfoBase::lat() const { return(Lat); } inline double ProbInfoBase::lon() const { return(Lon); } +inline double ProbInfoBase::dland() const { return(DLand); } inline int ProbInfoBase::n_prob() const { return(NProb); } inline double ProbInfoBase::prob(int i) const { return(Prob[i]); } inline double ProbInfoBase::prob_item(int i) const { return(ProbItem[i]); } diff --git a/met/src/libcode/vx_tc_util/prob_rirw_info.cc b/met/src/libcode/vx_tc_util/prob_rirw_info.cc index edbd318334..7799e591d3 100644 --- a/met/src/libcode/vx_tc_util/prob_rirw_info.cc +++ b/met/src/libcode/vx_tc_util/prob_rirw_info.cc @@ -149,11 +149,11 @@ int ProbRIRWInfo::rirw_window() const { //////////////////////////////////////////////////////////////////////// -void ProbRIRWInfo::initialize(const ATCFProbLine &l) { +void ProbRIRWInfo::initialize(const ATCFProbLine &l, double dland) { clear(); - ProbInfoBase::initialize(l); + ProbInfoBase::initialize(l, dland); Value = parse_int(l.get_item(ProbRIRWValueOffset).c_str()); Initials = l.get_item(ProbRIRWInitialsOffset); @@ -169,14 +169,15 @@ bool ProbRIRWInfo::is_match(const ATCFProbLine &l) const { if(!ProbInfoBase::is_match(l)) return(false); - return(Value == parse_int(l.get_item(ProbRIRWValueOffset).c_str()) && - RIRWBeg == parse_int(l.get_item(ProbRIRWBegOffset).c_str()) && - RIRWEnd == parse_int(l.get_item(ProbRIRWEndOffset).c_str())); + return(ValidTime == l.valid() && + Value == parse_int(l.get_item(ProbRIRWValueOffset).c_str()) && + RIRWBeg == parse_int(l.get_item(ProbRIRWBegOffset).c_str()) && + RIRWEnd == parse_int(l.get_item(ProbRIRWEndOffset).c_str())); } //////////////////////////////////////////////////////////////////////// -bool ProbRIRWInfo::add(const ATCFProbLine &l, bool check_dup) { +bool ProbRIRWInfo::add(const ATCFProbLine &l, double dland, bool check_dup) { // Check for duplicates if(check_dup) { @@ -190,7 +191,7 @@ bool ProbRIRWInfo::add(const ATCFProbLine &l, bool check_dup) { } // Initialize the header information, if necessary - if(Type == NoATCFLineType) initialize(l); + if(Type == NoATCFLineType) initialize(l, dland); // Check for matching header information if(!is_match(l)) return(false); diff --git a/met/src/libcode/vx_tc_util/prob_rirw_info.h b/met/src/libcode/vx_tc_util/prob_rirw_info.h index 9f47615dbf..2f04fc4baf 100644 --- a/met/src/libcode/vx_tc_util/prob_rirw_info.h +++ b/met/src/libcode/vx_tc_util/prob_rirw_info.h @@ -69,9 +69,9 @@ class ProbRIRWInfo : public ProbInfoBase { // do stuff // - void initialize(const ATCFProbLine &); + void initialize(const ATCFProbLine &, double); bool is_match (const ATCFProbLine &) const; - bool add (const ATCFProbLine &, bool check_dup = false); + bool add (const ATCFProbLine &, double, bool check_dup = false); void set (const TCStatLine &); }; diff --git a/met/src/tools/tc_utils/tc_gen/tc_gen.cc b/met/src/tools/tc_utils/tc_gen/tc_gen.cc index bfb3960226..fb05b4c743 100644 --- a/met/src/tools/tc_utils/tc_gen/tc_gen.cc +++ b/met/src/tools/tc_utils/tc_gen/tc_gen.cc @@ -20,6 +20,7 @@ // 003 12/31/20 Halley Gotway Add NetCDF output for MET #1430 // 004 01/14/21 Halley Gotway Add GENMPR output for MET #1597 // 005 04/02/21 Halley Gotway Refinements for MET #1714 +// 006 11/04/21 Halley Gotway Add -edeck for MET #1809 // //////////////////////////////////////////////////////////////////////// @@ -56,19 +57,24 @@ using namespace std; //////////////////////////////////////////////////////////////////////// static void process_command_line (int, char **); -static void process_genesis (); +static void score_track_genesis (const GenesisInfoArray &, + const TrackInfoArray &); +static void score_genesis_prob (const GenesisInfoArray &, + const TrackInfoArray &); static void get_atcf_files (const StringArray &, const StringArray &, const char *, StringArray &, StringArray &); -static void process_fcst_tracks (const StringArray &, +static void process_genesis (const StringArray &, const StringArray &, GenesisInfoArray &); -static void process_best_tracks (const StringArray &, +static void process_tracks (const StringArray &, const StringArray &, GenesisInfoArray &, TrackInfoArray &); - +static void process_edecks (const StringArray &, + const StringArray &, + ProbInfoArray &); static void get_genesis_pairs (const TCGenVxOpt &, const ConcatString &, const GenesisInfoArray &, @@ -79,25 +85,44 @@ static void get_genesis_pairs (const TCGenVxOpt &, static void do_genesis_ctc (const TCGenVxOpt &, PairDataGenesis &, GenCTCInfo &); +static void do_probgen_pct (const TCGenVxOpt &, + ProbInfoArray &, + const GenesisInfoArray &, + const TrackInfoArray &, + ProbGenPCTInfo &); static int find_genesis_match (const GenesisInfo &, const GenesisInfoArray &, const TrackInfoArray &, bool, double, int, int); +static int find_probgen_match (const ProbGenInfo &, + const GenesisInfoArray &, + const TrackInfoArray &, + bool, double, int, int); -static void setup_txt_files (int, int); +static void setup_txt_files (int, int, int); static void setup_table (AsciiTable &); static void setup_nc_file (); -static void write_stats (const PairDataGenesis &, +static void write_ctc_stats (const PairDataGenesis &, GenCTCInfo &); -static void write_genmpr_row (StatHdrColumns &, +static void write_ctc_genmpr_row (StatHdrColumns &, const PairDataGenesis &, STATOutputType, AsciiTable &, int &, AsciiTable &, int &); -static void write_genmpr_cols (const PairDataGenesis &, int, +static void write_ctc_genmpr_cols(const PairDataGenesis &, int, AsciiTable &, int, int); + +static void write_pct_stats (ProbGenPCTInfo &); +static void write_pct_genmpr_row (StatHdrColumns &, + ProbGenPCTInfo &, int, + STATOutputType, + AsciiTable &, int &, + AsciiTable &, int &); +static void write_pct_genmpr_cols(ProbGenPCTInfo &, int, int, + AsciiTable &, int, int); + static void write_nc (GenCTCInfo &); static void finish_txt_files (); @@ -105,6 +130,7 @@ static void usage (); static void set_source (const StringArray &, const char *, StringArray &, StringArray &); static void set_genesis (const StringArray &); +static void set_edeck (const StringArray &); static void set_track (const StringArray &); static void set_config (const StringArray &); static void set_out (const StringArray &); @@ -119,8 +145,43 @@ int main(int argc, char *argv[]) { // Process the command line arguments process_command_line(argc, argv); - // Identify and process genesis events and write output - process_genesis(); + // Process the verifying BEST and operational tracks + StringArray track_files, track_files_model_suffix; + GenesisInfoArray best_ga; + TrackInfoArray oper_ta; + + // Get the list of verifing track files + get_atcf_files(track_source, track_model_suffix, atcf_reg_exp, + track_files, track_files_model_suffix); + + mlog << Debug(2) + << "Processing " << track_files.n() + << " verifying track files.\n"; + process_tracks(track_files, track_files_model_suffix, + best_ga, oper_ta); + + // Score genesis events and write output + if(genesis_source.n() > 0) { + score_track_genesis(best_ga, oper_ta); + } + + // Score EDECK genesis probabilities and write output + if(edeck_source.n() > 0) { + score_genesis_prob(best_ga, oper_ta); + } + + // Finish output files + finish_txt_files(); + + // Close the NetCDF output file + if(nc_out) { + + // List the NetCDF file after it is finished + mlog << Debug(1) << "Output file: " << out_nc_file << "\n"; + + delete nc_out; + nc_out = (NcFile *) 0; + } return(0); } @@ -143,6 +204,7 @@ void process_command_line(int argc, char **argv) { // Add function calls for the arguments cline.add(set_genesis, "-genesis", -1); + cline.add(set_edeck, "-edeck", -1); cline.add(set_track, "-track", -1); cline.add(set_config, "-config", 1); cline.add(set_out, "-out", 1); @@ -154,21 +216,29 @@ void process_command_line(int argc, char **argv) { for(i=genesis_model_suffix.n(); i " - << "the \"-genesis\", \"-track\", and \"-config\" command " + if(genesis_source.n() == 0 && edeck_source.n() == 0) { + mlog << Error << "\nprocess_command_line(int argc, char **argv) -> " + << "at least one of the \"-genesis\" or \"-edeck\" command " << "line options are required\n\n"; usage(); } + // Check for the minimum number of arguments + if(track_source.n() == 0 || config_file.length() == 0) { + mlog << Error << "\nprocess_command_line(int argc, char **argv) -> " + << "the \"-track\" and \"-config\" command line options " + << "are required\n\n"; + usage(); + } + // List the input genesis track files for(i=0; i model_ga_map; map::iterator it; PairDataGenesis pairs; - GenCTCInfo ctc_info; + GenCTCInfo genesis_ctc; // Get the list of genesis track files get_atcf_files(genesis_source, genesis_model_suffix, atcf_gen_reg_exp, @@ -221,25 +298,15 @@ void process_genesis() { mlog << Debug(2) << "Processing " << genesis_files.n() << " forecast genesis track files.\n"; - process_fcst_tracks(genesis_files, genesis_files_model_suffix, - fcst_ga); - - // Get the list of verifing track files - get_atcf_files(track_source, track_model_suffix, atcf_reg_exp, - track_files, track_files_model_suffix); - - mlog << Debug(2) - << "Processing " << track_files.n() - << " verifying track files.\n"; - process_best_tracks(track_files, track_files_model_suffix, - best_ga, oper_ta); + process_genesis(genesis_files, genesis_files_model_suffix, + fcst_ga); // Setup output files based on the number of techniques present // and possible pairs. int n_time = (conf_info.FcstSecEnd - conf_info.FcstSecBeg) / (conf_info.InitFreqHr*sec_per_hour) + 1; int n_pair = best_ga.n() * n_time + fcst_ga.n(); - setup_txt_files(fcst_ga.n_technique(), n_pair); + setup_txt_files(fcst_ga.n_technique(), 1, n_pair); // If requested, setup the NetCDF output file if(!conf_info.NcInfo.all_false()) setup_nc_file(); @@ -272,16 +339,16 @@ void process_genesis() { // Store the current genesis event model_ga_map[model].add(fcst_ga[j]); - } // end j + } // end for j - // Process the genesis events for each model. + // Process the genesis events for each model for(j=0,it=model_ga_map.begin(); it!=model_ga_map.end(); it++,j++) { // Initialize - ctc_info.clear(); - ctc_info.Model = it->first; - ctc_info.set_vx_opt(&conf_info.VxOpt[i], - &conf_info.NcOutGrid); + genesis_ctc.clear(); + genesis_ctc.Model = it->first; + genesis_ctc.set_vx_opt(&conf_info.VxOpt[i], + &conf_info.NcOutGrid); mlog << Debug(2) << "[Filter " << i+1 << " (" << conf_info.VxOpt[i].Desc @@ -297,33 +364,107 @@ void process_genesis() { best_ga, oper_ta, pairs); // Do the categorical verification - do_genesis_ctc(conf_info.VxOpt[i], pairs, ctc_info); + do_genesis_ctc(conf_info.VxOpt[i], pairs, genesis_ctc); // Write the statistics output - write_stats(pairs, ctc_info); + write_ctc_stats(pairs, genesis_ctc); // Write NetCDF output fields if(!conf_info.VxOpt[i].NcInfo.all_false()) { - write_nc(ctc_info); + write_nc(genesis_ctc); } } // end for j } // end for i n_vx - // Finish output files - finish_txt_files(); + return; +} - // Close the NetCDF output file - if(nc_out) { +//////////////////////////////////////////////////////////////////////// - // List the NetCDF file after it is finished - mlog << Debug(1) << "Output file: " << out_nc_file << "\n"; +void score_genesis_prob(const GenesisInfoArray &best_ga, + const TrackInfoArray &oper_ta) { + int i, j, n, max_n_prob, n_pair; + StringArray edeck_files, edeck_files_model_suffix; + ProbInfoArray fcst_pa, empty_pa; + ConcatString model; + map model_pa_map; + map::iterator it; + ProbGenPCTInfo probgen_pct; - delete nc_out; - nc_out = (NcFile *) 0; + // Get the list of EDECK files + get_atcf_files(edeck_source, edeck_model_suffix, atcf_reg_exp, + edeck_files, edeck_files_model_suffix); + + mlog << Debug(2) + << "Processing " << edeck_files.n() + << " forecast EDECK files.\n"; + process_edecks(edeck_files, edeck_files_model_suffix, + fcst_pa); + + // Count up the total number of probabilities + for(i=0,max_n_prob=0,n_pair=0; i max_n_prob) max_n_prob = n; + n_pair += n; } + // Setup output files based on the number of techniques + setup_txt_files(fcst_pa.n_technique(), max_n_prob, n_pair); + + // Process each verification filter + for(i=0; i 0 && + !conf_info.VxOpt[i].Model.has(model)) continue; + + // Add a new map entry, if necessary + if(model_pa_map.count(model) == 0) { + empty_pa.clear(); + model_pa_map[model] = empty_pa; + } + + // Store the current genesis event + model_pa_map[model].add(fcst_pa.prob_gen(j)); + + } // end for j + + // Process the genesis probabilities for each model + for(j=0,it=model_pa_map.begin(); it!=model_pa_map.end(); it++,j++) { + + mlog << Debug(2) + << "[Filter " << i+1 << " (" << conf_info.VxOpt[i].Desc + << ") " << ": Model " << j+1 << "] " << "For " << it->first + << " model, comparing " << it->second.n_prob_gen() + << " probability of genesis forecasts to " << best_ga.n() << " " + << conf_info.BestEventInfo.Technique << " and " + << oper_ta.n() << " " << conf_info.OperTechnique + << " tracks.\n"; + + // Do the probabilistic verification + do_probgen_pct(conf_info.VxOpt[i], it->second, + best_ga, oper_ta, probgen_pct); + + // Write the statistics output + write_pct_stats(probgen_pct); + + } // end for j + } // end for i return; } @@ -598,9 +739,63 @@ void do_genesis_ctc(const TCGenVxOpt &vx_opt, //////////////////////////////////////////////////////////////////////// -int find_genesis_match(const GenesisInfo &fcst_gi, +void do_probgen_pct(const TCGenVxOpt &vx_opt, + ProbInfoArray &model_pa, + const GenesisInfoArray &best_ga, + const TrackInfoArray &oper_ta, + ProbGenPCTInfo &pgi) { + int i, i_bga, j, time_diff; + bool is_event; + const GenesisInfo *bgi; + + // Initialize + pgi.clear(); + pgi.set_vx_opt(&vx_opt); + + // Score each of the probability forecasts + for(i=0; igenesis_time() - + model_pa.prob_gen(i).init(); + is_event = time_diff >= 0 && + time_diff <= (model_pa.prob_gen(i).prob_item(j) * sec_per_hour); + } + else { + is_event = false; + } + + // Store pair info + pgi.add(model_pa.prob_gen(i), j, bgi, is_event); + + } // end for j + } // end for i + + return; +} + + +//////////////////////////////////////////////////////////////////////// + +int find_genesis_match(const GenesisInfo &fcst_gi, const GenesisInfoArray &bga, - const TrackInfoArray &ota, + const TrackInfoArray &ota, bool point2track, double rad, int beg, int end) { int i, j, i_best, i_oper; @@ -684,6 +879,94 @@ int find_genesis_match(const GenesisInfo &fcst_gi, return(i_best); } +//////////////////////////////////////////////////////////////////////// + +int find_probgen_match(const ProbGenInfo &prob_gi, + const GenesisInfoArray &bga, + const TrackInfoArray &ota, + bool point2track, double rad, + int beg, int end) { + int i, j, i_best, i_oper; + + ConcatString case_cs; + case_cs << prob_gi.technique() << " " + << unix_to_yyyymmdd_hhmmss(prob_gi.init()) + << " initialization, " + << unix_to_yyyymmdd_hhmmss(prob_gi.genesis_time()) + << " forecast genesis at (" << prob_gi.lat() << ", " + << prob_gi.lon() << ")"; + + // Search for a BEST track genesis match + for(i=0, i_best=bad_data_int; + i " + mlog << Error << "\nprocess_genesis() -> " << "unable to open file \"" << files[i] << "\"\n\n"; exit(1); } @@ -842,10 +1125,10 @@ void process_fcst_tracks(const StringArray &files, //////////////////////////////////////////////////////////////////////// -void process_best_tracks(const StringArray &files, - const StringArray &model_suffix, - GenesisInfoArray &best_ga, - TrackInfoArray &oper_ta) { +void process_tracks(const StringArray &files, + const StringArray &model_suffix, + GenesisInfoArray &best_ga, + TrackInfoArray &oper_ta) { int i, i_bga, n_lines; ConcatString suffix, gen_basin, case_cs, storm_id; StringArray best_tech, oper_tech; @@ -871,8 +1154,7 @@ void process_best_tracks(const StringArray &files, // Open the current file if(!f.open(files[i].c_str())) { - mlog << Error - << "\nprocess_best_tracks() -> " + mlog << Error << "\nprocess_tracks() -> " << "unable to open file \"" << files[i] << "\"\n\n"; exit(1); } @@ -976,7 +1258,7 @@ void process_best_tracks(const StringArray &files, i--; } else { - mlog << Warning << "\nprocess_best_tracks() -> " + mlog << Warning << "\nprocess_tracks() -> " << case_cs << "neither " << best_ga[i_bga].storm_id() << " nor " << best_gi.storm_id() << " matches the basin!\n\n"; @@ -997,6 +1279,82 @@ void process_best_tracks(const StringArray &files, mlog << Debug(2) << "Found " << best_ga.n() << " BEST genesis events.\n"; + // Dump out very verbose output + if(mlog.verbosity_level() > 6) { + mlog << Debug(6) << best_ga.serialize_r() << "\n"; + } + // Dump out track info + else { + for(i=0; i " + << "unable to open file \"" << files[i] << "\"\n\n"; + exit(1); + } + + // Set metadata pointer + suffix = model_suffix[i]; + line.set_tech_suffix(&suffix); + + // Process the input track lines + while(f >> line) { + + // Skip off-hour track points + if((line.valid_hour() % valid_freq_sec) != 0) continue; + + // Only process genesis probability lines + if(line.type() == ATCFLineType_ProbGN) { + dland = conf_info.compute_dland(line.lat(), -1.0*line.lon()); + if(probs.add(line, dland, false)) n_lines++; + } + } + + // Close the current file + f.close(); + + } // end for i + + // Dump out the total number of lines + mlog << Debug(3) + << "Read a total of " << n_lines << " " + << atcflinetype_to_string(ATCFLineType_ProbGN) + << " lines from " << files.n() << " input files.\n"; + + // Dump out very verbose output + if(mlog.verbosity_level() >= 6) { + mlog << Debug(6) << probs.serialize_r() << "\n"; + } + return; } @@ -1006,70 +1364,153 @@ void process_best_tracks(const StringArray &files, // //////////////////////////////////////////////////////////////////////// -void setup_txt_files(int n_model, int n_pair) { - int i, n_rows, n_cols; +void setup_txt_files(int n_model, int max_n_prob, int n_pair) { + int i, n_rows, n_cols, stat_rows, stat_cols, n_prob; - // Check to see if the text files have already been set up - if(stat_at.nrows() > 0 || stat_at.ncols() > 0) return; + // Check to see if the stat file stream has already been setup + bool init_from_scratch = (stat_out == (ofstream *) 0); - // Initialize file stream - stat_out = (ofstream *) 0; + // Get the maximum number of probability thresholds + n_prob = conf_info.get_max_n_prob_thresh(); - // Build the file name - stat_file << out_base << stat_file_ext; + // Compute the number of rows/cols needs for each file type + for(i=0, stat_rows=0, stat_cols=0; iOutputMap.at(stat_fho), + gci.VxOpt->output_map(stat_fho), stat_at, i_stat_row, txt_at[i_fho], i_txt_row[i_fho]); } @@ -1172,7 +1613,7 @@ void write_stats(const PairDataGenesis &gpd, shc.set_fcst_var(genesis_ops_name); shc.set_obs_var (genesis_ops_name); write_fho_row(shc, gci.CTSOps, - gci.VxOpt->OutputMap.at(stat_fho), + gci.VxOpt->output_map(stat_fho), stat_at, i_stat_row, txt_at[i_fho], i_txt_row[i_fho]); } @@ -1185,7 +1626,7 @@ void write_stats(const PairDataGenesis &gpd, shc.set_fcst_var(genesis_dev_name); shc.set_obs_var (genesis_dev_name); write_ctc_row(shc, gci.CTSDev, - gci.VxOpt->OutputMap.at(stat_ctc), + gci.VxOpt->output_map(stat_ctc), stat_at, i_stat_row, txt_at[i_ctc], i_txt_row[i_ctc]); } @@ -1194,7 +1635,7 @@ void write_stats(const PairDataGenesis &gpd, shc.set_fcst_var(genesis_ops_name); shc.set_obs_var (genesis_ops_name); write_ctc_row(shc, gci.CTSOps, - gci.VxOpt->OutputMap.at(stat_ctc), + gci.VxOpt->output_map(stat_ctc), stat_at, i_stat_row, txt_at[i_ctc], i_txt_row[i_ctc]); } @@ -1210,7 +1651,7 @@ void write_stats(const PairDataGenesis &gpd, shc.set_fcst_var(genesis_dev_name); shc.set_obs_var (genesis_dev_name); write_cts_row(shc, gci.CTSDev, - gci.VxOpt->OutputMap.at(stat_cts), + gci.VxOpt->output_map(stat_cts), stat_at, i_stat_row, txt_at[i_cts], i_txt_row[i_cts]); } @@ -1222,7 +1663,7 @@ void write_stats(const PairDataGenesis &gpd, shc.set_fcst_var(genesis_ops_name); shc.set_obs_var (genesis_ops_name); write_cts_row(shc, gci.CTSOps, - gci.VxOpt->OutputMap.at(stat_cts), + gci.VxOpt->output_map(stat_cts), stat_at, i_stat_row, txt_at[i_cts], i_txt_row[i_cts]); } @@ -1232,21 +1673,22 @@ void write_stats(const PairDataGenesis &gpd, if(gci.VxOpt->output_map(stat_genmpr) != STATOutputType_None) { shc.set_fcst_var(genesis_name); shc.set_obs_var (genesis_name); - write_genmpr_row(shc, gpd, - gci.VxOpt->OutputMap.at(stat_genmpr), - stat_at, i_stat_row, - txt_at[i_genmpr], i_txt_row[i_genmpr]); + write_ctc_genmpr_row(shc, gpd, + gci.VxOpt->output_map(stat_genmpr), + stat_at, i_stat_row, + txt_at[i_genmpr], i_txt_row[i_genmpr]); } return; } + //////////////////////////////////////////////////////////////////////// -void write_genmpr_row(StatHdrColumns &shc, - const PairDataGenesis &gpd, - STATOutputType out_type, - AsciiTable &stat_at, int &stat_row, - AsciiTable &txt_at, int &txt_row) { +void write_ctc_genmpr_row(StatHdrColumns &shc, + const PairDataGenesis &gpd, + STATOutputType out_type, + AsciiTable &stat_at, int &stat_row, + AsciiTable &txt_at, int &txt_row) { int i; unixtime ut; @@ -1254,6 +1696,8 @@ void write_genmpr_row(StatHdrColumns &shc, shc.set_line_type(stat_genmpr_str); // Not Applicable + shc.set_fcst_thresh(na_str); + shc.set_obs_thresh(na_str); shc.set_alpha(bad_data_double); // Write a line for each matched pair @@ -1262,7 +1706,7 @@ void write_genmpr_row(StatHdrColumns &shc, // Pointers for current case const GenesisInfo* fgi = gpd.fcst_gen(i); const GenesisInfo* bgi = gpd.best_gen(i); - + // Store timing info shc.set_fcst_lead_sec(gpd.lead_time(i)); ut = (fgi ? fgi->genesis_time() : bgi->genesis_time()); @@ -1277,7 +1721,7 @@ void write_genmpr_row(StatHdrColumns &shc, write_header_cols(shc, stat_at, stat_row); // Write the data columns - write_genmpr_cols(gpd, i, stat_at, stat_row, n_header_columns); + write_ctc_genmpr_cols(gpd, i, stat_at, stat_row, n_header_columns); // If requested, copy row to the text file if(out_type == STATOutputType_Both) { @@ -1296,16 +1740,17 @@ void write_genmpr_row(StatHdrColumns &shc, //////////////////////////////////////////////////////////////////////// - void write_genmpr_cols(const PairDataGenesis &gpd, int i, - AsciiTable &at, int r, int c) { + void write_ctc_genmpr_cols(const PairDataGenesis &gpd, int i, + AsciiTable &at, int r, int c) { // Pointers for current case const GenesisInfo* fgi = gpd.fcst_gen(i); const GenesisInfo* bgi = gpd.best_gen(i); - + // // Genesis Matched Pairs (GENMPR): // TOTAL, INDEX, STORM_ID, + // PROB_LEAD, PROB_VAL, // AGEN_INIT, AGEN_FHR, // AGEN_LAT, AGEN_LON, AGEN_DLAND, // BGEN_LAT, BGEN_LON, BGEN_DLAND, @@ -1322,43 +1767,49 @@ void write_genmpr_row(StatHdrColumns &shc, at.set_entry(r, c+2, // Best track Storm ID gpd.best_storm_id(i)); - at.set_entry(r, c+3, // Fcst genesis initialization time + at.set_entry(r, c+3, // Probability lead time + na_str); + + at.set_entry(r, c+4, // Probability value + na_str); + + at.set_entry(r, c+5, // Fcst genesis initialization time fgi ? unix_to_yyyymmdd_hhmmss(fgi->init()) : na_str); - at.set_entry(r, c+4, // Fcst genesis hour + at.set_entry(r, c+6, // Fcst genesis hour fgi ? fgi->genesis_fhr() : bad_data_int); - at.set_entry(r, c+5, // Fcst track latitude + at.set_entry(r, c+7, // Fcst track latitude fgi ? fgi->lat() : bad_data_double); - at.set_entry(r, c+6, // Fcst track longitude + at.set_entry(r, c+8, // Fcst track longitude fgi ? fgi->lon() : bad_data_double); - at.set_entry(r, c+7, // Fcst track distance to land + at.set_entry(r, c+9, // Fcst track distance to land fgi ? fgi->dland() : bad_data_double); - at.set_entry(r, c+8, // Best track latitude + at.set_entry(r, c+10, // Best track latitude bgi ? bgi->lat() : bad_data_double); - at.set_entry(r, c+9, // Best track longitude + at.set_entry(r, c+11, // Best track longitude bgi ? bgi->lon() : bad_data_double); - at.set_entry(r, c+10, // Best track distance to land + at.set_entry(r, c+12, // Best track distance to land bgi ? bgi->dland() : bad_data_double); - at.set_entry(r, c+11, // Genesis distance + at.set_entry(r, c+13, // Genesis distance gpd.gen_diff(i).DevDist); - at.set_entry(r, c+12, // Genesis time difference + at.set_entry(r, c+14, // Genesis time difference sec_to_hhmmss(gpd.gen_diff(i).DevDSec)); - at.set_entry(r, c+13, // Genesis - Init time + at.set_entry(r, c+15, // Genesis - Init time sec_to_hhmmss(gpd.gen_diff(i).OpsDSec)); - at.set_entry(r, c+14, // Development category + at.set_entry(r, c+16, // Development category genesispaircategory_to_string(gpd.gen_diff(i).DevCategory)); - at.set_entry(r, c+15, // Operational category + at.set_entry(r, c+17, // Operational category genesispaircategory_to_string(gpd.gen_diff(i).OpsCategory)); return; @@ -1366,6 +1817,222 @@ void write_genmpr_row(StatHdrColumns &shc, //////////////////////////////////////////////////////////////////////// +void write_pct_stats(ProbGenPCTInfo &pgi) { + int i, lead_hr, lead_sec; + + // Setup header columns + shc.set_model(pgi.Model.c_str()); + shc.set_desc(pgi.VxOpt->Desc.c_str()); + shc.set_obtype(conf_info.BestEventInfo.Technique.c_str()); + shc.set_mask(pgi.VxOpt->VxMaskName.empty() ? + na_str : pgi.VxOpt->VxMaskName.c_str()); + shc.set_fcst_var(prob_genesis_name); + shc.set_obs_var (prob_genesis_name); + + // Write results for each lead time + for(i=0; ioutput_map(stat_pct) != STATOutputType_None) { + write_pct_row(shc, pgi.PCTMap[lead_hr], + pgi.VxOpt->output_map(stat_pct), + 1, 1, stat_at, i_stat_row, + txt_at[i_pct], i_txt_row[i_pct]); + } + + // Write PSTD output + if(pgi.VxOpt->output_map(stat_pstd) != STATOutputType_None) { + pgi.PCTMap[lead_hr].compute_stats(); + pgi.PCTMap[lead_hr].compute_ci(); + write_pstd_row(shc, pgi.PCTMap[lead_hr], + pgi.VxOpt->output_map(stat_pstd), + 1, 1, stat_at, i_stat_row, + txt_at[i_pstd], i_txt_row[i_pstd]); + } + + // Write PJC output + if(pgi.VxOpt->output_map(stat_pjc) != STATOutputType_None) { + write_pct_row(shc, pgi.PCTMap[lead_hr], + pgi.VxOpt->output_map(stat_pjc), + 1, 1, stat_at, i_stat_row, + txt_at[i_pjc], i_txt_row[i_pjc]); + } + + // Write PRC output + if(pgi.VxOpt->output_map(stat_pjc) != STATOutputType_None) { + write_pct_row(shc, pgi.PCTMap[lead_hr], + pgi.VxOpt->output_map(stat_pjc), + 1, 1, stat_at, i_stat_row, + txt_at[i_prc], i_txt_row[i_prc]); + } + + // Write out GENMPR + if(pgi.VxOpt->output_map(stat_genmpr) != STATOutputType_None) { + shc.set_fcst_var(prob_genesis_name); + shc.set_obs_var (prob_genesis_name); + write_pct_genmpr_row(shc, pgi, lead_hr, + pgi.VxOpt->output_map(stat_genmpr), + stat_at, i_stat_row, + txt_at[i_genmpr], i_txt_row[i_genmpr]); + } + + } // end for i + + return; +} + +//////////////////////////////////////////////////////////////////////// + +void write_pct_genmpr_row(StatHdrColumns &shc, + ProbGenPCTInfo &pgi, + int lead_hr, + STATOutputType out_type, + AsciiTable &stat_at, int &stat_row, + AsciiTable &txt_at, int &txt_row) { + int i; + unixtime ut; + + // GENMPR line type + shc.set_line_type(stat_genmpr_str); + + // Not Applicable + shc.set_fcst_thresh(na_str); + shc.set_obs_thresh(na_str); + shc.set_alpha(bad_data_double); + + // Write a line for each matched pair + for(i=0; igenesis_lead()); + ut = fgi->genesis_time(); + shc.set_fcst_valid_beg(ut); + shc.set_fcst_valid_end(ut); + shc.set_obs_lead_sec(bad_data_int); + ut = (bgi ? bgi->genesis_time() : ut); + shc.set_obs_valid_beg(ut); + shc.set_obs_valid_end(ut); + + // Write the header columns + write_header_cols(shc, stat_at, stat_row); + + // Write the data columns + write_pct_genmpr_cols(pgi, lead_hr, i, + stat_at, stat_row, n_header_columns); + + // If requested, copy row to the text file + if(out_type == STATOutputType_Both) { + copy_ascii_table_row(stat_at, stat_row, txt_at, txt_row); + + // Increment the text row counter + txt_row++; + } + + // Increment the STAT row counter + stat_row++; + + } + + return; +} + +//////////////////////////////////////////////////////////////////////// + + void write_pct_genmpr_cols(ProbGenPCTInfo &pgi, + int lead_hr, int index, + AsciiTable &at, int r, int c) { + + // Pointers for current case + const ProbGenInfo *fgi = pgi.FcstGenMap[lead_hr][index]; + const GenesisInfo *bgi = pgi.BestGenMap[lead_hr][index]; + + int i_prob = pgi.FcstIdxMap[lead_hr][index]; + + // + // Genesis Matched Pairs (GENMPR): + // TOTAL, INDEX, STORM_ID, + // PROB_LEAD, PROB_VAL, + // AGEN_INIT, AGEN_FHR, + // AGEN_LAT, AGEN_LON, AGEN_DLAND, + // BGEN_LAT, BGEN_LON, BGEN_DLAND, + // GEN_DIST, GEN_TDIFF, INIT_TDIFF, + // DEV_CAT, OPS_CAT + // + + at.set_entry(r, c+0, // Total number of pairs + (int) pgi.FcstGenMap[lead_hr].size()); + + at.set_entry(r, c+1, // Index of current pair + index+1); + + at.set_entry(r, c+2, // Best track Storm ID + (bgi ? bgi->storm_id() : na_str)); + + at.set_entry(r, c+3, // Probability lead time + fgi->prob_item(i_prob)); + + at.set_entry(r, c+4, // Probability value + fgi->prob(i_prob)); + + at.set_entry(r, c+5, // Fcst genesis initialization time + unix_to_yyyymmdd_hhmmss(fgi->init())); + + at.set_entry(r, c+6, // Fcst genesis hour + fgi->genesis_fhr()); + + at.set_entry(r, c+7, // Fcst genesis latitude + fgi->lat()); + + at.set_entry(r, c+8, // Fcst genesis longitude + fgi->lon()); + + at.set_entry(r, c+9, // Fcst genesis distance to land + fgi->dland()); + + at.set_entry(r, c+10, // Best track latitude + bgi ? bgi->lat() : bad_data_double); + + at.set_entry(r, c+11, // Best track longitude + bgi ? bgi->lon() : bad_data_double); + + at.set_entry(r, c+12, // Best track distance to land + bgi ? bgi->dland() : bad_data_double); + + at.set_entry(r, c+13, // Genesis distance + na_str); + + at.set_entry(r, c+14, // Genesis time difference + na_str); + + at.set_entry(r, c+15, // Genesis - Init time + na_str); + + at.set_entry(r, c+16, // Development category + na_str); + + at.set_entry(r, c+17, // Operational category + (pgi.BestEvtMap[lead_hr][index] ? "FYOY" : "FYON" )); + + return; + } + +//////////////////////////////////////////////////////////////////////// + void write_nc(GenCTCInfo &gci) { int i; ConcatString var_name, long_name; @@ -1547,19 +2214,24 @@ void usage() { << ") ***\n\n" << "Usage: " << program_name << "\n" - << "\t-genesis path\n" - << "\t-track path\n" + << "\t-genesis source and/or -edeck source\n" + << "\t-track source\n" << "\t-config file\n" << "\t[-out base]\n" << "\t[-log file]\n" << "\t[-v level]\n\n" - << "\twhere\t\"-genesis path\" is one or more ATCF genesis " + << "\twhere\t\"-genesis source\" is one or more ATCF genesis " << "files, an ASCII file list containing them, or a top-level " << "directory with files matching the regular expression \"" - << atcf_gen_reg_exp << "\" (required).\n" + << atcf_gen_reg_exp << "\" (required if no -edeck).\n" - << "\t\t\"-track path\" is one or more ATCF track " + << "\t\t\"-edeck source\" is one or more ensemble model output " + << "files, an ASCII file list containing them, or a top-level " + << "directory with files matching the regular expression \"" + << atcf_reg_exp << "\" (required if no -genesis).\n" + + << "\t\t\"-track source\" is one or more ATCF track " << "files, an ASCII file list containing them, or a top-level " << "directory with files matching the regular expression \"" << atcf_reg_exp << "\" for the verifying BEST and operational " @@ -1648,6 +2320,12 @@ void set_genesis(const StringArray & a) { //////////////////////////////////////////////////////////////////////// +void set_edeck(const StringArray & a) { + set_source(a, "edeck", edeck_source, edeck_model_suffix); +} + +//////////////////////////////////////////////////////////////////////// + void set_track(const StringArray & a) { set_source(a, "track", track_source, track_model_suffix); } diff --git a/met/src/tools/tc_utils/tc_gen/tc_gen.h b/met/src/tools/tc_utils/tc_gen/tc_gen.h index a531652c32..19461c5941 100644 --- a/met/src/tools/tc_utils/tc_gen/tc_gen.h +++ b/met/src/tools/tc_utils/tc_gen/tc_gen.h @@ -60,22 +60,27 @@ static const char * default_config_filename = // Header columns static const char **txt_columns[n_txt] = { - fho_columns, ctc_columns, cts_columns, genmpr_columns + fho_columns, ctc_columns, cts_columns, + pct_columns, pstd_columns, pjc_columns, + prc_columns, genmpr_columns }; // Length of header columns static const int n_txt_columns[n_txt] = { - n_fho_columns, n_ctc_columns, n_cts_columns, n_genmpr_columns + n_fho_columns, n_ctc_columns, n_cts_columns, + n_pct_columns, n_pstd_columns, n_pjc_columns, + n_prc_columns, n_genmpr_columns }; // Text file abbreviations static const char *txt_file_abbr[n_txt] = { - "fho", "ctc", "cts", "genmpr" + "fho", "ctc", "cts", "pct", "pstd", "pjc", "prc", "genmpr" }; -const ConcatString genesis_name ("GENESIS"); -const ConcatString genesis_dev_name("GENESIS_DEV"); -const ConcatString genesis_ops_name("GENESIS_OPS"); +const ConcatString genesis_name ("GENESIS"); +const ConcatString genesis_dev_name ("GENESIS_DEV"); +const ConcatString genesis_ops_name ("GENESIS_OPS"); +const ConcatString prob_genesis_name("PROB_GENESIS"); // Maximum Best track cyclone number to be processed // Cyclone numbers > 50 are for testing or invests @@ -89,7 +94,8 @@ static const int max_best_cyclone_number = 50; // Input files static StringArray genesis_source, genesis_model_suffix; -static StringArray track_source, track_model_suffix; +static StringArray edeck_source, edeck_model_suffix; +static StringArray track_source, track_model_suffix; static ConcatString config_file; static TCGenConfInfo conf_info; diff --git a/met/src/tools/tc_utils/tc_gen/tc_gen_conf_info.cc b/met/src/tools/tc_utils/tc_gen/tc_gen_conf_info.cc index 02f75d16d7..0592c667b4 100644 --- a/met/src/tools/tc_utils/tc_gen/tc_gen_conf_info.cc +++ b/met/src/tools/tc_utils/tc_gen/tc_gen_conf_info.cc @@ -164,6 +164,7 @@ void TCGenVxOpt::clear() { OpsHitBeg = OpsHitEnd = bad_data_int; DiscardFlag = false; DevFlag = OpsFlag = false; + ProbGenThresh.clear(); CIAlpha = bad_data_double; OutputMap.clear(); NcInfo.clear(); @@ -288,7 +289,12 @@ void TCGenVxOpt::process_config(Dictionary &dict) { << " must be set to true!\n\n"; exit(1); } - + + // Conf: prob_genesis_thresh + ProbGenThresh = dict.lookup_thresh_array(conf_key_prob_genesis_thresh); + ProbGenThresh = string_to_prob_thresh(ProbGenThresh.get_str().c_str()); + check_prob_thresh(ProbGenThresh); + // Conf: ci_alpha CIAlpha = dict.lookup_double(conf_key_ci_alpha); @@ -508,6 +514,62 @@ bool TCGenVxOpt::is_keeper(const GenesisInfo &gi) const { //////////////////////////////////////////////////////////////////////// +bool TCGenVxOpt::is_keeper(const ProbGenInfo &gi) const { + bool keep = true; + + // ATCF ID processed elsewhere + + // Check storm id + if(StormId.n() > 0 && + !has_storm_id(StormId, gi.basin(), gi.cyclone(), gi.init())) + keep = false; + + // Check storm name: no included in genesis probabilities + + // Initialization time + if((InitBeg > 0 && InitBeg > gi.init()) || + (InitEnd > 0 && InitEnd < gi.init()) || + (InitInc.n() > 0 && !InitInc.has(gi.init())) || + (InitExc.n() > 0 && InitExc.has(gi.init()))) + keep = false; + + // Initialization hours + if(InitHour.n() > 0 && !InitHour.has(gi.init_hour())) + keep = false; + + // Lead and valid times: + // ProbGenInfo objects can contain multiple lead/valid times. + // Do not filter by them here. + + // Poly masking + if(VxPolyMask.n_points() > 0 && + !VxPolyMask.latlon_is_inside(gi.lat(), gi.lon())) + keep = false; + + // Area masking + if(!VxAreaMask.is_empty()) { + double x, y; + VxGridMask.latlon_to_xy(gi.lat(), -1.0*gi.lon(), x, y); + if(x < 0 || x >= VxGridMask.nx() || + y < 0 || y >= VxGridMask.ny()) { + keep = false; + } + else { + keep = VxAreaMask(nint(x), nint(y)); + } + } + + // Distance to land + if((DLandThresh.get_type() != no_thresh_type) && + (is_bad_data(gi.dland()) || !DLandThresh.check(gi.dland()))) + keep = false; + + // Return the keep status + return(keep); +} + +//////////////////////////////////////////////////////////////////////// + STATOutputType TCGenVxOpt::output_map(STATLineType t) const { return(OutputMap.at(t)); } @@ -823,6 +885,18 @@ STATOutputType TCGenConfInfo::output_map(STATLineType t) const { return(OutputMap.at(t)); } +//////////////////////////////////////////////////////////////////////// + +int TCGenConfInfo::get_max_n_prob_thresh() const { + int i, n; + + for(i=0,n=0; iProbGenThresh); + DefaultPCT.allocate_n_alpha(1); + DefaultPCT.alpha[0] = VxOpt->CIAlpha; + + return; +} + +//////////////////////////////////////////////////////////////////////// + +void ProbGenPCTInfo::add(const ProbGenInfo &fgi, int index, + const GenesisInfo *bgi, bool is_event) { + int i; + unixtime ut; + + // Store the model name + if(Model.empty()) Model = fgi.technique(); + + // Track the range of forecast initalization times + ut = fgi.init(); + if(InitBeg == 0 || InitBeg > ut) InitBeg = ut; + if(InitEnd == 0 || InitEnd < ut) InitEnd = ut; + + // Track the range of verifying BEST genesis events + if(bgi) { + ut = bgi->genesis_time(); + if(BestBeg == 0 || BestBeg > ut) BestBeg = ut; + if(BestEnd == 0 || BestEnd < ut) BestEnd = ut; + } + + // Current lead time and probability value + int lead_hr = nint(fgi.prob_item(index)); + double prob = fgi.prob(index) / 100.0; + + // Add new map entries, if needed + if(!LeadTimes.has(lead_hr)) { + + LeadTimes.add(lead_hr); + vector empty_fgi; + vector empty_idx; + vector empty_bgi; + vector empty_evt; + + PCTMap [lead_hr] = DefaultPCT; + FcstGenMap[lead_hr] = empty_fgi; + FcstIdxMap[lead_hr] = empty_idx; + BestGenMap[lead_hr] = empty_bgi; + BestEvtMap[lead_hr] = empty_evt; + } + + // Update map entries + FcstGenMap[lead_hr].push_back(&fgi); + FcstIdxMap[lead_hr].push_back(index); + BestGenMap[lead_hr].push_back(bgi); + BestEvtMap[lead_hr].push_back(is_event); + + // Increment counts + if(is_event) PCTMap[lead_hr].pct.inc_event (prob); + else PCTMap[lead_hr].pct.inc_nonevent(prob); + + return; +} + +//////////////////////////////////////////////////////////////////////// diff --git a/met/src/tools/tc_utils/tc_gen/tc_gen_conf_info.h b/met/src/tools/tc_utils/tc_gen/tc_gen_conf_info.h index a02a62e562..fe1b3c3f57 100644 --- a/met/src/tools/tc_utils/tc_gen/tc_gen_conf_info.h +++ b/met/src/tools/tc_utils/tc_gen/tc_gen_conf_info.h @@ -29,16 +29,24 @@ static const int i_fho = 0; static const int i_ctc = 1; static const int i_cts = 2; -static const int i_genmpr = 3; +static const int i_pct = 3; +static const int i_pstd = 4; +static const int i_pjc = 5; +static const int i_prc = 6; +static const int i_genmpr = 7; -static const int n_txt = 4; +static const int n_txt = 8; // Text file type static const STATLineType txt_file_type[n_txt] = { stat_fho, // 0 stat_ctc, // 1 stat_cts, // 2 - stat_genmpr // 3 + stat_pct, // 3 + stat_pstd, // 4 + stat_pjc, // 5 + stat_prc, // 6 + stat_genmpr // 7 }; // Names for output data plane types @@ -78,7 +86,6 @@ struct TCGenNcOutInfo { bool do_best_fy_oy; bool do_best_fn_oy; - ////////////////////////////////////////////////////////////////// TCGenNcOutInfo(); @@ -145,6 +152,7 @@ class TCGenVxOpt { bool DiscardFlag, DevFlag, OpsFlag; // Output file options + ThreshArray ProbGenThresh; double CIAlpha; map OutputMap; TCGenNcOutInfo NcInfo; @@ -161,6 +169,7 @@ class TCGenVxOpt { void parse_nc_info(Dictionary &); bool is_keeper(const GenesisInfo &) const; + bool is_keeper(const ProbGenInfo &) const; STATOutputType output_map(STATLineType) const; }; @@ -242,6 +251,9 @@ class TCGenConfInfo { int compression_level(); STATOutputType output_map(STATLineType) const; + + // Maximum across all verification tasks + int get_max_n_prob_thresh() const; }; //////////////////////////////////////////////////////////////////////// @@ -303,6 +315,49 @@ class GenCTCInfo { //////////////////////////////////////////////////////////////////////// +class ProbGenPCTInfo { + + private: + + void init_from_scratch(); + + PCTInfo DefaultPCT; + + public: + + ProbGenPCTInfo(); + ~ProbGenPCTInfo(); + + ////////////////////////////////////////////////////////////////// + + ConcatString Model; + unixtime InitBeg, InitEnd; + unixtime BestBeg, BestEnd; + const TCGenVxOpt* VxOpt; + IntArray LeadTimes; + + // Map of lead times to PCT tables + map PCTMap; + + // Map of lead times to vectors of pair info + map> FcstGenMap; + map> FcstIdxMap; + map> BestGenMap; + map> BestEvtMap; + + ////////////////////////////////////////////////////////////////// + + void clear(); + + void set_vx_opt(const TCGenVxOpt *); + + void add(const ProbGenInfo &, int, + const GenesisInfo *, bool); + +}; + +//////////////////////////////////////////////////////////////////////// + #endif /* __TC_GEN_CONF_INFO_H__ */ //////////////////////////////////////////////////////////////////////// diff --git a/met/src/tools/tc_utils/tc_pairs/Makefile.am b/met/src/tools/tc_utils/tc_pairs/Makefile.am index d5d4436698..1dd9e1275f 100644 --- a/met/src/tools/tc_utils/tc_pairs/Makefile.am +++ b/met/src/tools/tc_utils/tc_pairs/Makefile.am @@ -38,6 +38,7 @@ tc_pairs_LDADD = -lvx_stat_out \ -lvx_gsl_prob \ -lvx_pb_util \ -lvx_cal \ + -lvx_nav \ -lvx_util \ -lvx_math \ -lvx_color \ diff --git a/met/src/tools/tc_utils/tc_pairs/tc_pairs.cc b/met/src/tools/tc_utils/tc_pairs/tc_pairs.cc index 7a70c0736a..5bf073959e 100644 --- a/met/src/tools/tc_utils/tc_pairs/tc_pairs.cc +++ b/met/src/tools/tc_utils/tc_pairs/tc_pairs.cc @@ -437,7 +437,7 @@ void process_edecks(const TrackInfoArray &bdeck_tracks) { // Filter the EDECK tracks using the config file information mlog << Debug(2) - << "Filtering " << edeck_probs.n_probs() + << "Filtering " << edeck_probs.n_prob_rirw() << " probabilities based on config file settings.\n"; filter_probs(edeck_probs); @@ -446,11 +446,11 @@ void process_edecks(const TrackInfoArray &bdeck_tracks) { // mlog << Debug(2) - << "Matching " << edeck_probs.n_probs() + << "Matching " << edeck_probs.n_prob_rirw() << " EDECK probabilities to " << bdeck_tracks.n() << " BDECK tracks.\n"; - for(i=0; idland()); + cur_ri.set_bdland(compute_dland(cur_ri.blat(), -1.0*cur_ri.blon())); // Store the current pair prob_rirw_pairs.add(cur_ri); @@ -640,6 +640,7 @@ void process_prob_files(const StringArray &files, const StringArray &model_suffix, ProbInfoArray &probs) { int i, cur_read, cur_add, tot_read, tot_add; + double dland; LineDataFile f; ConcatString suffix; ATCFProbLine line; @@ -683,10 +684,13 @@ void process_prob_files(const StringArray &files, // Check the keep status if(!is_keeper(&line)) continue; - // Attempt to add the current line to ProbInfoArray - if(probs.add(line, conf_info.CheckDup)) { - cur_add++; - tot_add++; + // Only process probability of RI lines + if(line.type() == ATCFLineType_ProbRI) { + dland = compute_dland(line.lat(), -1.0*line.lon()); + if(probs.add(line, dland, conf_info.CheckDup)) { + cur_add++; + tot_add++; + } } } @@ -708,7 +712,7 @@ void process_prob_files(const StringArray &files, // Dump out the track information mlog << Debug(3) - << "Identified " << probs.n_probs() << " probabilities.\n"; + << "Identified " << probs.n_prob_rirw() << " probabilities.\n"; // Dump out very verbose output if(mlog.verbosity_level() >= 5) { @@ -717,9 +721,9 @@ void process_prob_files(const StringArray &files, } // Dump out track info else { - for(i=0; iserialize() << "\n"; } } @@ -928,7 +932,7 @@ void filter_probs(ProbInfoArray &probs) { // Loop through the pairs and determine which should be retained // The is_keeper() function has already filtered by model, storm id, // basin, cyclone, initialization time, and initialization hour. - for(i=0; i 0 && @@ -977,11 +981,11 @@ void filter_probs(ProbInfoArray &probs) { // Print summary filtering info mlog << Debug(3) - << "Total probabilities read = " << p.n_probs() << "\n" - << "Total probabilities kept = " << probs.n_probs() << "\n" - << "Rejected for valid time = " << n_vld << "\n" - << "Rejected for init mask = " << n_mask_init << "\n" - << "Rejected for valid mask = " << n_mask_vld << "\n"; + << "Total probabilities read = " << p.n_prob_rirw() << "\n" + << "Total probabilities kept = " << probs.n_prob_rirw() << "\n" + << "Rejected for valid time = " << n_vld << "\n" + << "Rejected for init mask = " << n_mask_init << "\n" + << "Rejected for valid mask = " << n_mask_vld << "\n"; return; } diff --git a/test/config/TCGenConfig_prob b/test/config/TCGenConfig_prob new file mode 100644 index 0000000000..db075b4479 --- /dev/null +++ b/test/config/TCGenConfig_prob @@ -0,0 +1,288 @@ +//////////////////////////////////////////////////////////////////////////////// +// +// TC-Gen configuration file. +// +// For additional information, please see the MET User's Guide. +// +//////////////////////////////////////////////////////////////////////////////// + +// +// ATCF file format reference: +// http://www.nrlmry.navy.mil/atcf_web/docs/database/new/abrdeck.html +// + +//////////////////////////////////////////////////////////////////////////////// +// +// Genesis event definition criteria +// +//////////////////////////////////////////////////////////////////////////////// + +// +// Model initialization frequency in hours, starting at 0 +// +init_freq = 6; + +// +// Valid hour frequency to be analyzed in hours, starting at 0 +// +valid_freq = 6; + +// +// Forecast hours to be searched for genesis events +// +fcst_hr_window = { + beg = 6; + end = 120; +} + +// +// Minimum track duration for genesis event in hours +// +min_duration = 12; + +// +// Forecast genesis event criteria. Defined as tracks reaching the specified +// intensity category, maximum wind speed threshold, and minimum sea-level +// pressure threshold. The forecast genesis time is the valid time of the first +// track point where all of these criteria are met. +// +fcst_genesis = { + vmax_thresh = NA; + mslp_thresh = NA; +} + +// +// BEST track genesis event criteria. Defined as tracks reaching the specified +// intensity category, maximum wind speed threshold, and minimum sea-level +// pressure threshold. The BEST track genesis time is the valid time of the +// first track point where all of these criteria are met. +// +best_genesis = { + technique = "BEST"; + category = [ "TD", "TS" ]; + vmax_thresh = NA; + mslp_thresh = NA; +} + +// +// Operational track technique name +// +oper_technique = "CARQ"; + +//////////////////////////////////////////////////////////////////////////////// +// +// Track filtering options +// May be specified separately in each filter array entry. +// +//////////////////////////////////////////////////////////////////////////////// + +// +// Array of dictionaries containing the track filtering options +// If empty, a single filter is defined using the top-level settings. +// +filter = [ + { + desc = "ALL"; + } +]; + +// +// Description written to output DESC column +// +desc = "ALL"; + +// +// Forecast ATCF ID's +// If empty, all ATCF ID's found will be processed. +// Statistics will be generated separately for each ATCF ID. +// +model = []; + +// +// BEST and operational track storm identifiers +// +storm_id = []; + +// +// BEST and operational track storm names +// +storm_name = []; + +// +// Forecast and operational initialization times to include or exclude +// +init_beg = ""; +init_end = ""; +init_inc = []; +init_exc = []; + +// +// Forecast, BEST, and operational valid time window +// +valid_beg = ""; +valid_end = ""; + +// +// Forecast and operational initialization hours +// +init_hour = []; + +// +// Forecast and operational lead times in hours +// +lead = []; + +// +// Spatial masking region (path to gridded data file or polyline file) +// +vx_mask = ""; + +// +// Spatial masking of hurricane basin names from the basin_file +// +basin_mask = []; + +// +// Distance to land threshold +// +dland_thresh = NA; + +//////////////////////////////////////////////////////////////////////////////// +// +// Matching and scoring options +// May be specified separately in each filter array entry. +// +//////////////////////////////////////////////////////////////////////////////// + +// +// Genesis matching logic. Compare the forecast genesis point to all points in +// the Best track (TRUE) or the single Best track genesis point (FALSE). +// +genesis_match_point_to_track = FALSE; + +// +// Radius in km to search for a matching genesis event +// +genesis_match_radius = 500; + +// +// Time window in hours, relative to the model genesis time, to search for a +// matching Best track point +// +genesis_match_window = { + beg = -6; + end = 6; +} + +// +// Radius in km for a development scoring method hit +// +dev_hit_radius = 500; + +// +// Time window in hours, relative to the model genesis time, for a development +// scoring method hit +// +dev_hit_window = { + beg = -24; + end = 24; +} + +// +// Time window in hours for the Best track genesis minus model initialization +// time difference for an operational scoring method hit +// +ops_hit_window = { + beg = 0; + end = 48; +} + +// +// Discard genesis forecasts for initializations at or after the matching +// BEST track genesis time +// +discard_init_post_genesis_flag = TRUE; + +// +// Scoring methods to be applied +// +dev_method_flag = TRUE; +ops_method_flag = TRUE; + +//////////////////////////////////////////////////////////////////////////////// +// +// Output options +// May be specified separately in each filter array entry. +// +//////////////////////////////////////////////////////////////////////////////// + +// +// Probability of genesis thresholds +// +prob_genesis_thresh = ==0.25; + +// +// Confidence interval alpha value +// +ci_alpha = 0.05; + +// +// Statistical output types +// +output_flag = { + fho = NONE; + ctc = NONE; + cts = NONE; + pct = BOTH; + pstd = BOTH; + pjc = BOTH; + prc = BOTH; + genmpr = BOTH; +} + +// +// NetCDF genesis pair counts +// +nc_pairs_flag = FALSE; + +// +// Specify which track points should be counted by thresholding the track point +// valid time minus genesis time difference. +// +valid_minus_genesis_diff_thresh = NA; + +// +// Count unique BEST track genesis event locations (TRUE) versus counting the +// location for all pairs (FALSE). +// +best_unique_flag = TRUE; + +//////////////////////////////////////////////////////////////////////////////// +// +// Global settings +// May only be specified once. +// +//////////////////////////////////////////////////////////////////////////////// + +// +// Specify the NetCDF output of the gen_dland tool containing a gridded +// representation of the minimum distance to land. +// +dland_file = "MET_BASE/tc_data/dland_global_tenth_degree.nc"; + +// +// Specify the NetCDF file containing a gridded representation of the +// global basins. +// +basin_file = "MET_BASE/tc_data/basin_global_tenth_degree.nc"; + +// +// NetCDF genesis pairs grid +// +nc_pairs_grid = "G003"; + +// +// Indicate a version number for the contents of this configuration file. +// The value should generally not be modified. +// +version = "V10.1.0"; diff --git a/test/hdr/met_10_1.hdr b/test/hdr/met_10_1.hdr index 641ac25813..84f5b5a950 100644 --- a/test/hdr/met_10_1.hdr +++ b/test/hdr/met_10_1.hdr @@ -28,7 +28,7 @@ SSVAR : VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG FCST_VALID_END OBS_L VL1L2 : VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG FCST_VALID_END OBS_LEAD OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_UNITS FCST_LEV OBS_VAR OBS_UNITS OBS_LEV OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH COV_THRESH ALPHA LINE_TYPE TOTAL UFBAR VFBAR UOBAR VOBAR UVFOBAR UVFFBAR UVOOBAR F_SPEED_BAR O_SPEED_BAR VAL1L2 : VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG FCST_VALID_END OBS_LEAD OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_UNITS FCST_LEV OBS_VAR OBS_UNITS OBS_LEV OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH COV_THRESH ALPHA LINE_TYPE TOTAL UFABAR VFABAR UOABAR VOABAR UVFOABAR UVFFABAR UVOOABAR VCNT : VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG FCST_VALID_END OBS_LEAD OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_UNITS FCST_LEV OBS_VAR OBS_UNITS OBS_LEV OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH COV_THRESH ALPHA LINE_TYPE TOTAL FBAR FBAR_BCL FBAR_BCU OBAR OBAR_BCL OBAR_BCU FS_RMS FS_RMS_BCL FS_RMS_BCU OS_RMS OS_RMS_BCL OS_RMS_BCU MSVE MSVE_BCL MSVE_BCU RMSVE RMSVE_BCL RMSVE_BCU FSTDEV FSTDEV_BCL FSTDEV_BCU OSTDEV OSTDEV_BCL OSTDEV_BCU FDIR FDIR_BCL FDIR_BCU ODIR ODIR_BCL ODIR_BCU FBAR_SPEED FBAR_SPEED_BCL FBAR_SPEED_BCU OBAR_SPEED OBAR_SPEED_BCL OBAR_SPEED_BCU VDIFF_SPEED VDIFF_SPEED_BCL VDIFF_SPEED_BCU VDIFF_DIR VDIFF_DIR_BCL VDIFF_DIR_BCU SPEED_ERR SPEED_ERR_BCL SPEED_ERR_BCU SPEED_ABSERR SPEED_ABSERR_BCL SPEED_ABSERR_BCU DIR_ERR DIR_ERR_BCL DIR_ERR_BCU DIR_ABSERR DIR_ABSERR_BCL DIR_ABSERR_BCU -GENMPR : VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG FCST_VALID_END OBS_LEAD OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_UNITS FCST_LEV OBS_VAR OBS_UNITS OBS_LEV OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH COV_THRESH ALPHA LINE_TYPE TOTAL INDEX STORM_ID AGEN_INIT AGEN_FHR AGEN_LAT AGEN_LON AGEN_DLAND BGEN_LAT BGEN_LON BGEN_DLAND GEN_DIST GEN_TDIFF INIT_TDIFF DEV_CAT OPS_CAT +GENMPR : VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG FCST_VALID_END OBS_LEAD OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_UNITS FCST_LEV OBS_VAR OBS_UNITS OBS_LEV OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH COV_THRESH ALPHA LINE_TYPE TOTAL INDEX STORM_ID PROB_LEAD PROB_VAL AGEN_INIT AGEN_FHR AGEN_LAT AGEN_LON AGEN_DLAND BGEN_LAT BGEN_LON BGEN_DLAND GEN_DIST GEN_TDIFF INIT_TDIFF DEV_CAT OPS_CAT SSIDX : VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG FCST_VALID_END OBS_LEAD OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_UNITS FCST_LEV OBS_VAR OBS_UNITS OBS_LEV OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH COV_THRESH ALPHA LINE_TYPE FCST_MODEL REF_MODEL N_INIT N_TERM N_VLD SS_INDEX MODE_SOA : VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG FCST_VALID_END OBS_LEAD OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_UNITS FCST_LEV OBS_VAR OBS_UNITS OBS_LEV OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH COV_THRESH ALPHA LINE_TYPE N_VALID GRID_RES OBJECT_ID OBJECT_CAT CENTROID_X CENTROID_Y CENTROID_LAT CENTROID_LON AXIS_ANG LENGTH WIDTH AREA AREA_THRESH CURVATURE CURVATURE_X CURVATURE_Y COMPLEXITY INTENSITY_10 INTENSITY_25 INTENSITY_50 INTENSITY_75 INTENSITY_90 INTENSITY_50 INTENSITY_SUM MODE_POA : VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG FCST_VALID_END OBS_LEAD OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_UNITS FCST_LEV OBS_VAR OBS_UNITS OBS_LEV OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH COV_THRESH ALPHA LINE_TYPE N_VALID GRID_RES OBJECT_ID OBJECT_CAT CENTROID_DIST BOUNDARY_DIST CONVEX_HULL_DIST ANGLE_DIFF ASPECT_DIFF AREA_RATIO INTERSECTION_AREA UNION_AREA SYMMETRIC_DIFF INTERSECTION_OVER_AREA CURVATURE_RATIO COMPLEXITY_RATIO PERCENTILE_INTENSITY_RATIO INTEREST diff --git a/test/xml/unit_tc_gen.xml b/test/xml/unit_tc_gen.xml index d1aa664166..86245745d8 100644 --- a/test/xml/unit_tc_gen.xml +++ b/test/xml/unit_tc_gen.xml @@ -6,7 +6,7 @@ - + ]> @@ -17,8 +17,8 @@ &MET_BIN;/tc_gen \ - -genesis &DATA_DIR;/suite1/2016*/genesis*2016* \ - -track &DATA_DIR;/atcf/2016 \ + -genesis &DATA_DIR;/genesis/suite1/2016*/genesis*2016* \ + -track &DATA_DIR;/genesis/atcf/2016 \ -config &CONFIG_DIR;/TCGenConfig_2016 \ -out &OUTPUT_DIR;/tc_gen/tc_gen_2016 \ -log &OUTPUT_DIR;/tc_gen/tc_gen_2016.log \ @@ -31,4 +31,26 @@ + + + + &MET_BIN;/tc_gen + \ + -edeck &DATA_DIR;/edeck/eal152020*dat \ + -track &DATA_DIR;/bdeck/bal152020.dat \ + -config &CONFIG_DIR;/TCGenConfig_prob \ + -out &OUTPUT_DIR;/tc_gen/tc_gen_prob \ + -log &OUTPUT_DIR;/tc_gen/tc_gen_prob.log \ + -v 4 + + + &OUTPUT_DIR;/tc_gen/tc_gen_prob.stat + &OUTPUT_DIR;/tc_gen/tc_gen_prob_pct.txt + &OUTPUT_DIR;/tc_gen/tc_gen_prob_pstd.txt + &OUTPUT_DIR;/tc_gen/tc_gen_prob_pjc.txt + &OUTPUT_DIR;/tc_gen/tc_gen_prob_prc.txt + &OUTPUT_DIR;/tc_gen/tc_gen_prob_genmpr.txt + + + From 13996c21d45e92b12483a70f074bf3e2ec0eb441 Mon Sep 17 00:00:00 2001 From: johnhg Date: Tue, 16 Nov 2021 14:57:47 -0700 Subject: [PATCH 10/82] Feature 1970 v10.1.0-beta4 (#1973) --- met/docs/Users_Guide/release-notes.rst | 25 +++++++++++++++++++++++-- met/docs/conf.py | 4 ++-- 2 files changed, 25 insertions(+), 4 deletions(-) diff --git a/met/docs/Users_Guide/release-notes.rst b/met/docs/Users_Guide/release-notes.rst index af9fb5168b..68524d323b 100644 --- a/met/docs/Users_Guide/release-notes.rst +++ b/met/docs/Users_Guide/release-notes.rst @@ -5,8 +5,29 @@ When applicable, release notes are followed by the GitHub issue number which describes the bugfix, enhancement, or new feature: `MET GitHub issues. `_ -MET Version 10.1.0-beta3 release notes (|release_date|) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +MET Version 10.1.0-beta4 release notes (20211116) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +* Enhancements: + + * **Add logic to Ensemble-Stat to handle an ensemble control member** (`#1905 `_). + * Enhance Ensemble-Stat and Gen-Ens-Prod to error out if the control member also appears in the list of ensemble members (`#1968 `_). + * **Enhance TC-Gen to verify genesis probabilities from ATCF e-deck files** (`#1809 `_). + * Modify the STAT-Analysis GO Index configuration file (`#1945 `_). + * **Support percentile thresholds for frequency bias not equal to 1 (e.g. ==FBIAS0.9)** (`#1761 `_). + * Reimplement the NumArray class based on an STL template (`#1899 `_). + +* Bugfixes: + + * Fix bug with the incrementing of numbers in temporary file names (`#1906 `_). + * Fix ascii2nc to check the return status when reading ASCII input files (`#1957 `_). + +* Documentation: + + * Enhance the documentation with meta-data that is expected by MET for netCDF (`#1949 `_). + +MET Version 10.1.0-beta3 release notes (20211006) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ * New tools: diff --git a/met/docs/conf.py b/met/docs/conf.py index 7a625f0c9e..b06f98568c 100644 --- a/met/docs/conf.py +++ b/met/docs/conf.py @@ -20,11 +20,11 @@ project = 'MET' author = 'UCAR/NCAR, NOAA, CSU/CIRA, and CU/CIRES' author_list = 'Halley Gotway, J., K. Newman, H. Soh, J. Opatz, T. Jensen, J. Prestopnik, L. Goodrich, D. Fillmore, B. Brown, R. Bullock, T. Fowler' -version = '10.1.0-beta3' +version = '10.1.0-beta4' verinfo = version release = f'{version}' release_year = '2021' -release_date = f'{release_year}-10-06' +release_date = f'{release_year}-11-16' copyright = f'{release_year}, {author}' # -- General configuration --------------------------------------------------- From dae9db7a95acae2d8faa130733fcc39eedddb83e Mon Sep 17 00:00:00 2001 From: John Halley Gotway Date: Wed, 17 Nov 2021 11:34:04 -0700 Subject: [PATCH 11/82] Fix tiny typo in plot_data_plane usage statement. --- met/src/tools/other/plot_data_plane/plot_data_plane.cc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/met/src/tools/other/plot_data_plane/plot_data_plane.cc b/met/src/tools/other/plot_data_plane/plot_data_plane.cc index 80251a7ccf..101eb51c0e 100644 --- a/met/src/tools/other/plot_data_plane/plot_data_plane.cc +++ b/met/src/tools/other/plot_data_plane/plot_data_plane.cc @@ -289,7 +289,7 @@ void usage() { << "\t[-v level]\n\n" << "\twhere\t\"input_filename\" is the name of a " - << " gridded data file to be plotted (required).\n" + << "gridded data file to be plotted (required).\n" << "\t\t\"output_filename\" is the name of the output " << "PostScript file to be written (required).\n" From 9bd9946d5e5c2be1b64ffeb7daf9ca30e65f2faf Mon Sep 17 00:00:00 2001 From: John Halley Gotway Date: Wed, 17 Nov 2021 16:02:31 -0700 Subject: [PATCH 12/82] Hotfix for the develop branch to get met-10.1.0-beta4 compiling on WCOSS. The intel compiler does not allow adjacent >> characters when defining maps. --- met/src/tools/tc_utils/tc_gen/tc_gen_conf_info.h | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/met/src/tools/tc_utils/tc_gen/tc_gen_conf_info.h b/met/src/tools/tc_utils/tc_gen/tc_gen_conf_info.h index fe1b3c3f57..b3512c0d8c 100644 --- a/met/src/tools/tc_utils/tc_gen/tc_gen_conf_info.h +++ b/met/src/tools/tc_utils/tc_gen/tc_gen_conf_info.h @@ -340,10 +340,10 @@ class ProbGenPCTInfo { map PCTMap; // Map of lead times to vectors of pair info - map> FcstGenMap; - map> FcstIdxMap; - map> BestGenMap; - map> BestEvtMap; + map > FcstGenMap; + map > FcstIdxMap; + map > BestGenMap; + map > BestEvtMap; ////////////////////////////////////////////////////////////////// From e0b659dc5bab88c3263c6f95046ae37a67119443 Mon Sep 17 00:00:00 2001 From: John Halley Gotway Date: Wed, 17 Nov 2021 16:16:23 -0700 Subject: [PATCH 13/82] Updating the beta4 release date from 11/16 to 11/17 since the beta4 release needs to be recreated. --- met/docs/Users_Guide/release-notes.rst | 2 +- met/docs/conf.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/met/docs/Users_Guide/release-notes.rst b/met/docs/Users_Guide/release-notes.rst index 68524d323b..eebd4f9e28 100644 --- a/met/docs/Users_Guide/release-notes.rst +++ b/met/docs/Users_Guide/release-notes.rst @@ -5,7 +5,7 @@ When applicable, release notes are followed by the GitHub issue number which describes the bugfix, enhancement, or new feature: `MET GitHub issues. `_ -MET Version 10.1.0-beta4 release notes (20211116) +MET Version 10.1.0-beta4 release notes (20211117) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ * Enhancements: diff --git a/met/docs/conf.py b/met/docs/conf.py index b06f98568c..c04f0ceeb3 100644 --- a/met/docs/conf.py +++ b/met/docs/conf.py @@ -24,7 +24,7 @@ verinfo = version release = f'{version}' release_year = '2021' -release_date = f'{release_year}-11-16' +release_date = f'{release_year}-11-17' copyright = f'{release_year}, {author}' # -- General configuration --------------------------------------------------- From 817d9db840f7e8434ab6c9dde14cd4aab414fe6c Mon Sep 17 00:00:00 2001 From: John Halley Gotway Date: Fri, 19 Nov 2021 15:26:31 -0700 Subject: [PATCH 14/82] Update the MET PR template to list expected changes up front. --- .github/pull_request_template.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index 450355440e..c2984bffdf 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -1,3 +1,11 @@ +## Expected Differences ## + +- [ ] Do these changes introduce new tools, command line arguments, or configuration file options? **[Yes or No]**
+If **yes**, please describe:
+ +- [ ] Do these changes modify the structure of existing or add new output data types (e.g. statistic line types or NetCDF variables)? **[Yes or No]**
+If **yes**, please describe:
+ ## Pull Request Testing ## - [ ] Describe testing already performed for these changes:
From 4e28ae7f11b90e0cac989e53ee7357ab63d2c6c9 Mon Sep 17 00:00:00 2001 From: johnhg Date: Mon, 22 Nov 2021 12:40:28 -0700 Subject: [PATCH 15/82] Bugfix 1976 develop cdist (#1979) --- .../tools/other/mode_time_domain/3d_att.cc | 26 ++++++++++++------- 1 file changed, 16 insertions(+), 10 deletions(-) diff --git a/met/src/tools/other/mode_time_domain/3d_att.cc b/met/src/tools/other/mode_time_domain/3d_att.cc index 0fc640ff05..6c10fadfda 100644 --- a/met/src/tools/other/mode_time_domain/3d_att.cc +++ b/met/src/tools/other/mode_time_domain/3d_att.cc @@ -1261,7 +1261,8 @@ moments = mask.calc_3d_moments(); if ( moments.N == 0 ) { - mlog << Error << "\n\n calc_3d_single_atts() -> empty object!\n\n"; + mlog << Error << "\n\ncalc_3d_single_atts() -> " + << "empty object!\n\n"; exit ( 1 ); @@ -1322,16 +1323,22 @@ for (j=0,x_old=0,y_old=0; j 0 ) { - } else { - dist += calc_2d_dist(xbar_2d, ybar_2d, x_old, y_old, *grid); - } // else + } + + // + // update old (x, y) with current + // + + x_old = xbar_2d; + y_old = ybar_2d; } @@ -1347,7 +1354,8 @@ values = new float [Vol]; if ( !values ) { - mlog << Error << "\n\n calc_3d_single_atts() -> memory allocation error\n\n"; + mlog << Error << "\n\ncalc_3d_single_atts() -> " + << "memory allocation error\n\n"; exit ( 1 ); @@ -1390,8 +1398,6 @@ a.Ptile_User = percentile_f(values, n, (double) (a.Ptile_Value/100.0)); if ( values ) { delete [] values; values = 0; } -// a.dump(cout); - return ( a ); } From 39a5233d324a735410036a30aa7edc7360e2a2af Mon Sep 17 00:00:00 2001 From: Howard Soh Date: Mon, 29 Nov 2021 13:47:45 -0700 Subject: [PATCH 16/82] #1936 Excludes precip3hr, precip6hr, precip12hr, and precip10min from required variables for MESONET --- met/src/tools/other/madis2nc/madis2nc.cc | 26 +++++++++--------------- 1 file changed, 10 insertions(+), 16 deletions(-) diff --git a/met/src/tools/other/madis2nc/madis2nc.cc b/met/src/tools/other/madis2nc/madis2nc.cc index b1192f566f..6f48a30ccc 100644 --- a/met/src/tools/other/madis2nc/madis2nc.cc +++ b/met/src/tools/other/madis2nc/madis2nc.cc @@ -84,7 +84,7 @@ static void clean_up(); static void setup_netcdf_out(int nhdr); static bool get_filtered_nc_data(NcVar var, float *data, const long dim, - const long cur, const char *var_name); + const long cur, const char *var_name, bool required=true); static bool get_filtered_nc_data_2d(NcVar var, int *data, const long *dim, const long *cur, const char *var_name, bool count_bad=false); static bool get_filtered_nc_data_2d(NcVar var, float *data, const long *dim, @@ -415,7 +415,7 @@ void setup_netcdf_out(int nhdr) { static bool get_filtered_nc_data(NcVar var, float *data, const long dim, const long cur, - const char *var_name) { + const char *var_name, bool required) { bool status = false; float in_fill_value; @@ -439,15 +439,13 @@ static bool get_filtered_nc_data(NcVar var, float *data, << "Fail to read data [" << var_name << "].\n\n"; } } - else { + else if (required) { mlog << Error << "\n" << method_name << "Can not read a NetCDF data because the variable [" << var_name << "] is missing.\n\n"; } - if (!status) { - for (int idx=0; idx Date: Wed, 1 Dec 2021 12:42:39 -0700 Subject: [PATCH 17/82] Per #1985, correcting typo found while creating V10.0.1 in the develop branch as well. --- met/docs/Users_Guide/point-stat.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/met/docs/Users_Guide/point-stat.rst b/met/docs/Users_Guide/point-stat.rst index 926ad1f0c5..311332574f 100644 --- a/met/docs/Users_Guide/point-stat.rst +++ b/met/docs/Users_Guide/point-stat.rst @@ -745,7 +745,7 @@ The first set of header columns are common to all of the output files generated .. _table_PS_format_info_CNT: -.. list-table:: Format information for CNT(Continuous Statistics) output line type. +.. list-table:: Format information for CNT (Continuous Statistics) output line type. :widths: auto :header-rows: 2 @@ -804,7 +804,7 @@ The first set of header columns are common to all of the output files generated .. _table_PS_format_info_CNT_cont: -.. list-table:: Format information for CNT(Continuous Statistics) output line type continued from above table +.. list-table:: Format information for CNT (Continuous Statistics) output line type continued from above table :widths: auto :header-rows: 2 @@ -1299,7 +1299,7 @@ The first set of header columns are common to all of the output files generated .. _table_PS_format_info_VCNT: -.. list-table:: Format information for VAL1L2 (Vector Anomaly Partial Sums) output line type. Note that each statistic (except TOTAL) is followed by two columns giving bootstrap confidence intervals. These confidence intervals are not currently calculated for this release of MET, but will be in future releases. +.. list-table:: Format information for VCNT (Vector Continuous Statistics) output line type. Note that each statistic (except TOTAL) is followed by two columns giving bootstrap confidence intervals. These confidence intervals are not currently calculated for this release of MET, but will be in future releases. :widths: auto :header-rows: 2 From 84f8c7fb311fe632a89870274bf6ce74d1cc40a7 Mon Sep 17 00:00:00 2001 From: John Halley Gotway Date: Wed, 1 Dec 2021 12:54:12 -0700 Subject: [PATCH 18/82] Per #1985, VERY minor change to fix alignment of titles in Point-Stat... updating the develop branch. --- met/docs/Users_Guide/point-stat.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/met/docs/Users_Guide/point-stat.rst b/met/docs/Users_Guide/point-stat.rst index 311332574f..e9cc3fdca8 100644 --- a/met/docs/Users_Guide/point-stat.rst +++ b/met/docs/Users_Guide/point-stat.rst @@ -698,7 +698,7 @@ The first set of header columns are common to all of the output files generated .. _table_PS_format_info_CTS_cont: -.. list-table:: Format information for CTS (Contingency Table Statistics) output line type, continued from above +.. list-table:: Format information for CTS (Contingency Table Statistics) output line type, continued from above :widths: auto :header-rows: 2 @@ -804,7 +804,7 @@ The first set of header columns are common to all of the output files generated .. _table_PS_format_info_CNT_cont: -.. list-table:: Format information for CNT (Continuous Statistics) output line type continued from above table +.. list-table:: Format information for CNT (Continuous Statistics) output line type continued from above table :widths: auto :header-rows: 2 From d35190e8c1ad5b3d60b7e5a2863ece30bc0d15c0 Mon Sep 17 00:00:00 2001 From: Howard Soh Date: Wed, 1 Dec 2021 13:50:10 -0700 Subject: [PATCH 19/82] #1936 Added an unit test for newer MESONET file --- test/xml/unit_madis2nc.xml | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/test/xml/unit_madis2nc.xml b/test/xml/unit_madis2nc.xml index 0b0020e8ba..802eebc3c7 100644 --- a/test/xml/unit_madis2nc.xml +++ b/test/xml/unit_madis2nc.xml @@ -130,6 +130,18 @@ + + &MET_BIN;/madis2nc + \ + &DATA_DIR_OBS;/madis/mesonet/mesonet_20170101_0000.nc \ + &OUTPUT_DIR;/madis2nc/mesonet_20170101_0000_F000.nc \ + -type mesonet -mask_grid G207 -v 2 + + + &OUTPUT_DIR;/madis2nc/mesonet_20170101_0000_F000.nc + + + &MET_BIN;/madis2nc \ From 8c29f09d21ea73ce755027e18192b638d1a591bc Mon Sep 17 00:00:00 2001 From: johnhg Date: Mon, 13 Dec 2021 11:08:49 -0700 Subject: [PATCH 20/82] Feature 1989 sort (#1990) --- met/data/table_files/Makefile.am | 2 +- .../{grib2_af_gpp.txt => grib2_gpp_af.txt} | 0 met/src/basic/vx_util/get_filenames.cc | 12 +++++++++++- 3 files changed, 12 insertions(+), 2 deletions(-) rename met/data/table_files/{grib2_af_gpp.txt => grib2_gpp_af.txt} (100%) diff --git a/met/data/table_files/Makefile.am b/met/data/table_files/Makefile.am index 354ba05876..ab8a87b130 100644 --- a/met/data/table_files/Makefile.am +++ b/met/data/table_files/Makefile.am @@ -87,9 +87,9 @@ tablefiles_DATA = \ grib1_ncep_141_7.txt \ grib1_ncep_2_7.txt \ grib1_lidar2nc.txt \ - grib2_af_gpp.txt \ grib2_all.txt \ grib2_bom.txt \ + grib2_gpp_af.txt \ grib2_mrms.txt \ grib2_ndfd.txt diff --git a/met/data/table_files/grib2_af_gpp.txt b/met/data/table_files/grib2_gpp_af.txt similarity index 100% rename from met/data/table_files/grib2_af_gpp.txt rename to met/data/table_files/grib2_gpp_af.txt diff --git a/met/src/basic/vx_util/get_filenames.cc b/met/src/basic/vx_util/get_filenames.cc index d0067fa437..f52d31ae7e 100644 --- a/met/src/basic/vx_util/get_filenames.cc +++ b/met/src/basic/vx_util/get_filenames.cc @@ -112,11 +112,14 @@ struct stat sbuf; if ( S_ISDIR(sbuf.st_mode) ) { // - // process directory + // get filenames from the directory and sort them + // to make the order consistent across platforms // b = get_filenames_from_dir(search_dir.c_str(), prefix, suffix); + b.sort(); + a.add(b); b.clear(); @@ -198,10 +201,17 @@ while ( (entry = readdir(directory)) != NULL ) { } + // + // get filenames from the directory and sort them + // to make the order consistent across platforms + // + if ( S_ISDIR(sbuf.st_mode) ) { b = get_filenames_from_dir(entry_path, prefix, suffix); + b.sort(); + a.add(b); b.clear(); From b93f4eae10e5061cd78a07b6307f36b26191b20a Mon Sep 17 00:00:00 2001 From: johnhg Date: Mon, 13 Dec 2021 12:08:56 -0700 Subject: [PATCH 21/82] Feature 1991 VCNT (#1992) --- met/src/tools/core/grid_stat/grid_stat.cc | 30 ++++++++++++--------- met/src/tools/core/point_stat/point_stat.cc | 10 +++---- 2 files changed, 23 insertions(+), 17 deletions(-) diff --git a/met/src/tools/core/grid_stat/grid_stat.cc b/met/src/tools/core/grid_stat/grid_stat.cc index 380df877b5..9b2447818a 100644 --- a/met/src/tools/core/grid_stat/grid_stat.cc +++ b/met/src/tools/core/grid_stat/grid_stat.cc @@ -85,8 +85,7 @@ // output line types. // 034 05/10/16 Halley Gotway Add grid weighting. // 035 05/20/16 Prestopnik J Removed -version (now in command_line.cc) -// 036 02/14/17 Win MET-621 enhancement support- additional -// nc_pairs_flag 'apply_mask' +// 036 02/14/17 Win MET#621 Add nc_pairs_flag.apply_mask // 037 05/15/17 Prestopnik P Add shape for regrid, nbrhd and interp // 038 06/26/17 Halley Gotway Add ECLV line type. // 039 08/18/17 Halley Gotway Add fourier decomposition. @@ -109,6 +108,7 @@ // 051 03/28/21 Halley Gotway Add mpr_column and mpr_thresh // filtering options. // 052 05/28/21 Halley Gotway Add MCTS HSS_EC output. +// 053 12/11/21 Halley Gotway MET #1991 Fix VCNT output. // //////////////////////////////////////////////////////////////////////// @@ -930,12 +930,13 @@ void process_scores() { do_cnt_sl1l2(conf_info.vx_opt[i], &pd); } - // Compute VL1L2 and VAL1L2 partial sums for UGRD,VGRD + // Compute VL1L2 and VAL1L2 partial sums for UGRD and VGRD if(!conf_info.vx_opt[i].fcst_info->is_prob() && conf_info.vx_opt[i].fcst_info->is_v_wind() && conf_info.vx_opt[i].fcst_info->uv_index() >= 0 && (conf_info.vx_opt[i].output_flag[i_vl1l2] != STATOutputType_None || - conf_info.vx_opt[i].output_flag[i_val1l2] != STATOutputType_None) ) { + conf_info.vx_opt[i].output_flag[i_val1l2] != STATOutputType_None || + conf_info.vx_opt[i].output_flag[i_vcnt] != STATOutputType_None)) { // Store the forecast variable name shc.set_fcst_var(ugrd_vgrd_abbr_str); @@ -1006,7 +1007,6 @@ void process_scores() { // Write out VL1L2 if(conf_info.vx_opt[i].output_flag[i_vl1l2] != STATOutputType_None && vl1l2_info[m].vcount > 0) { - write_vl1l2_row(shc, vl1l2_info[m], conf_info.vx_opt[i].output_flag[i_vl1l2], stat_at, i_stat_row, @@ -1016,18 +1016,15 @@ void process_scores() { // Write out VAL1L2 if(conf_info.vx_opt[i].output_flag[i_val1l2] != STATOutputType_None && vl1l2_info[m].vacount > 0) { - write_val1l2_row(shc, vl1l2_info[m], conf_info.vx_opt[i].output_flag[i_val1l2], stat_at, i_stat_row, txt_at[i_val1l2], i_txt_row[i_val1l2]); } - // Write out VCNT if(conf_info.vx_opt[i].output_flag[i_vcnt] != STATOutputType_None && vl1l2_info[m].vcount > 0) { - write_vcnt_row(shc, vl1l2_info[m], conf_info.vx_opt[i].output_flag[i_vcnt], stat_at, i_stat_row, @@ -1677,12 +1674,13 @@ void process_scores() { do_cnt_sl1l2(conf_info.vx_opt[i], &pd); } - // Compute VL1L2 and VAL1L2 partial sums for UGRD,VGRD + // Compute VL1L2 and VAL1L2 partial sums for UGRD and VGRD if(!conf_info.vx_opt[i].fcst_info->is_prob() && conf_info.vx_opt[i].fcst_info->is_v_wind() && conf_info.vx_opt[i].fcst_info->uv_index() >= 0 && (conf_info.vx_opt[i].output_flag[i_vl1l2] != STATOutputType_None || - conf_info.vx_opt[i].output_flag[i_val1l2] != STATOutputType_None) ) { + conf_info.vx_opt[i].output_flag[i_val1l2] != STATOutputType_None || + conf_info.vx_opt[i].output_flag[i_vcnt] != STATOutputType_None)) { // Store the forecast variable name shc.set_fcst_var(ugrd_vgrd_abbr_str); @@ -1754,7 +1752,6 @@ void process_scores() { // Write out VL1L2 if(conf_info.vx_opt[i].output_flag[i_vl1l2] != STATOutputType_None && vl1l2_info[m].vcount > 0) { - write_vl1l2_row(shc, vl1l2_info[m], conf_info.vx_opt[i].output_flag[i_vl1l2], stat_at, i_stat_row, @@ -1764,12 +1761,21 @@ void process_scores() { // Write out VAL1L2 if(conf_info.vx_opt[i].output_flag[i_val1l2] != STATOutputType_None && vl1l2_info[m].vacount > 0) { - write_val1l2_row(shc, vl1l2_info[m], conf_info.vx_opt[i].output_flag[i_val1l2], stat_at, i_stat_row, txt_at[i_val1l2], i_txt_row[i_val1l2]); } + + // Write out VCNT + if(conf_info.vx_opt[i].output_flag[i_vcnt] != STATOutputType_None && + vl1l2_info[m].vcount > 0) { + write_vcnt_row(shc, vl1l2_info[m], + conf_info.vx_opt[i].output_flag[i_vcnt], + stat_at, i_stat_row, + txt_at[i_vcnt], i_txt_row[i_vcnt]); + } + } // end for m // Reset the forecast variable name diff --git a/met/src/tools/core/point_stat/point_stat.cc b/met/src/tools/core/point_stat/point_stat.cc index 549e6fce9f..7e78611c59 100644 --- a/met/src/tools/core/point_stat/point_stat.cc +++ b/met/src/tools/core/point_stat/point_stat.cc @@ -98,6 +98,7 @@ // 047 08/23/21 Seth Linden Add ORANK line type for HiRA. // 048 09/13/21 Seth Linden Changed obs_qty to obs_qty_inc. // Added code for obs_qty_exc. +// 049 12/11/21 Halley Gotway MET#1991 Fix VCNT output. // //////////////////////////////////////////////////////////////////////// @@ -1055,12 +1056,13 @@ void process_scores() { do_cnt_sl1l2(conf_info.vx_opt[i], pd_ptr); } - // Compute VL1L2 and VAL1L2 partial sums for UGRD,VGRD + // Compute VL1L2 and VAL1L2 partial sums for UGRD and VGRD if(!conf_info.vx_opt[i].vx_pd.fcst_info->is_prob() && conf_info.vx_opt[i].vx_pd.fcst_info->is_v_wind() && conf_info.vx_opt[i].vx_pd.fcst_info->uv_index() >= 0 && (conf_info.vx_opt[i].output_flag[i_vl1l2] != STATOutputType_None || - conf_info.vx_opt[i].output_flag[i_val1l2] != STATOutputType_None) ) { + conf_info.vx_opt[i].output_flag[i_val1l2] != STATOutputType_None || + conf_info.vx_opt[i].output_flag[i_vcnt] != STATOutputType_None)) { // Store the forecast variable name shc.set_fcst_var(ugrd_vgrd_abbr_str); @@ -1117,8 +1119,7 @@ void process_scores() { txt_at[i_val1l2], i_txt_row[i_val1l2]); } - - // Write out VCNT + // Write out VCNT if(conf_info.vx_opt[i].output_flag[i_vcnt] != STATOutputType_None && vl1l2_info[m].vcount > 0) { write_vcnt_row(shc, vl1l2_info[m], @@ -1127,7 +1128,6 @@ void process_scores() { txt_at[i_vcnt], i_txt_row[i_vcnt]); } - } // end for m // Reset the forecast variable name From 49fb555e1523ed7e9b61772ca05789977b7b90e1 Mon Sep 17 00:00:00 2001 From: johnhg Date: Mon, 13 Dec 2021 13:37:38 -0700 Subject: [PATCH 22/82] Feature 1993 grid_mask (#1994) --- .../tools/other/gen_vx_mask/gen_vx_mask.cc | 52 +++++++++++++++---- test/xml/unit_gen_vx_mask.xml | 35 +++++++++++++ 2 files changed, 77 insertions(+), 10 deletions(-) diff --git a/met/src/tools/other/gen_vx_mask/gen_vx_mask.cc b/met/src/tools/other/gen_vx_mask/gen_vx_mask.cc index 3b220f8e75..1fd7bb9bc2 100644 --- a/met/src/tools/other/gen_vx_mask/gen_vx_mask.cc +++ b/met/src/tools/other/gen_vx_mask/gen_vx_mask.cc @@ -25,7 +25,8 @@ // 007 04/08/19 Halley Gotway Add percentile thresholds. // 008 04/06/20 Halley Gotway Generalize input_grid option. // 009 06/01/21 Seth Linden Change -type from optional to required. -// 010 08/30/21 Halley Gotway MET #1891 fix input and mask fields. +// 010 08/30/21 Halley Gotway MET#1891 Fix input and mask fields. +// 011 12/13/21 Halley Gotway MET#1993 Fix -type grid. // //////////////////////////////////////////////////////////////////////// @@ -163,26 +164,26 @@ void process_command_line(int argc, char **argv) { void process_input_grid(DataPlane &dp) { - // Parse info.name as a white-space separated string + // Parse the input grid as a white-space separated string StringArray sa; sa.parse_wsss(input_gridname); // Search for a named grid if(sa.n() == 1 && find_grid_by_name(sa[0].c_str(), grid)) { mlog << Debug(3) - << "Use the grid named \"" << input_gridname << "\".\n"; + << "Use input grid named \"" << input_gridname << "\".\n"; } // Parse grid definition else if(sa.n() > 1 && parse_grid_def(sa, grid)) { mlog << Debug(3) - << "Use the grid defined by string \"" << input_gridname + << "Use input grid defined by string \"" << input_gridname << "\".\n"; } // Extract the grid from a gridded data file else { mlog << Debug(3) - << "Use the grid defined by file \"" << input_gridname + << "Use input grid defined by file \"" << input_gridname << "\".\n"; // Read the input grid and data plane, if requested @@ -249,13 +250,44 @@ void process_mask_file(DataPlane &dp) { mask_type == MaskType_Lon) { } - // Otherwise, process the mask file as a gridded data file + // Otherwise, process the mask file as a named grid, grid specification + // string or gridded data file else { - // Read the mask grid and data plane, if requested - get_data_plane(mask_filename, mask_field_str, - mask_type == MaskType_Data, - dp, grid_mask); + // For the grid mask type, support named grids and grid + // specification strings + if(mask_type == MaskType_Grid) { + + // Parse the mask file as a white-space separated string + StringArray sa; + sa.parse_wsss(mask_filename); + + // Search for a named grid + if(sa.n() == 1 && find_grid_by_name(sa[0].c_str(), grid_mask)) { + mlog << Debug(3) + << "Use mask grid named \"" << mask_filename << "\".\n"; + } + // Parse grid definition + else if(sa.n() > 1 && parse_grid_def(sa, grid_mask)) { + mlog << Debug(3) + << "Use mask grid defined by string \"" << mask_filename + << "\".\n"; + } + } + + // Parse as a gridded data file if not already set + if(grid_mask.nxy() == 0) { + + // Extract the grid from a gridded data file + mlog << Debug(3) + << "Use mask grid defined by file \"" << mask_filename + << "\".\n"; + + // Read the mask grid and data plane, if requested + get_data_plane(mask_filename, mask_field_str, + mask_type == MaskType_Data, + dp, grid_mask); + } mlog << Debug(2) << "Parsed Mask Grid:\t" << grid_mask.name() diff --git a/test/xml/unit_gen_vx_mask.xml b/test/xml/unit_gen_vx_mask.xml index 8ada4b6649..6416d0a048 100644 --- a/test/xml/unit_gen_vx_mask.xml +++ b/test/xml/unit_gen_vx_mask.xml @@ -125,6 +125,41 @@ + + + + + + &MET_BIN;/gen_vx_mask + \ + G004 G130 \ + &OUTPUT_DIR;/gen_vx_mask/GRID_NAMED_GRIDS_mask.nc \ + -type grid -name G130_grid \ + -v 1 + + + &OUTPUT_DIR;/gen_vx_mask/GRID_NAMED_GRIDS_mask.nc + + + + + + + + + &MET_BIN;/gen_vx_mask + \ + "latlon 720 361 -90 0 0.5 0.5" \ + "latlon 200 100 -40 -50 0.5 0.5" \ + &OUTPUT_DIR;/gen_vx_mask/GRID_SPEC_STRINGS_mask.nc \ + -type grid -name grid_spec \ + -v 1 + + + &OUTPUT_DIR;/gen_vx_mask/GRID_SPEC_STRINGS_mask.nc + + + From dd0a68b22bc3a9b2d899c1446dc939c8a3b67653 Mon Sep 17 00:00:00 2001 From: Howard Soh Date: Tue, 28 Dec 2021 15:28:37 -0700 Subject: [PATCH 23/82] #1844 Added vx_pointdata_python --- met/Make-include | 2 ++ 1 file changed, 2 insertions(+) diff --git a/met/Make-include b/met/Make-include index dfa6bd5faa..42f1d5f3b3 100644 --- a/met/Make-include +++ b/met/Make-include @@ -25,6 +25,7 @@ MET_CPPFLAGS = -I${top_builddir}/src/basic/vx_cal \ -I${top_builddir}/src/libcode/vx_nc_util \ -I${top_builddir}/src/libcode/vx_pb_util \ -I${top_builddir}/src/libcode/vx_plot_util \ + -I${top_builddir}/src/libcode/vx_pointdata_python \ -I${top_builddir}/src/libcode/vx_ps \ -I${top_builddir}/src/libcode/vx_pxm \ -I${top_builddir}/src/libcode/vx_render \ @@ -67,6 +68,7 @@ MET_LDFLAGS = -L${top_builddir}/src/basic/vx_cal \ -L${top_builddir}/src/libcode/vx_nc_util \ -L${top_builddir}/src/libcode/vx_pb_util \ -L${top_builddir}/src/libcode/vx_plot_util \ + -L${top_builddir}/src/libcode/vx_pointdata_python \ -L${top_builddir}/src/libcode/vx_ps \ -L${top_builddir}/src/libcode/vx_pxm \ -L${top_builddir}/src/libcode/vx_render \ From 2b61506d5e6e75162ebf13dcd5bbc99bf3afcf27 Mon Sep 17 00:00:00 2001 From: Howard Soh Date: Tue, 28 Dec 2021 15:31:27 -0700 Subject: [PATCH 24/82] #1844 Added vx_pointdata_python to PYTHON_LIBS & make file for vx_pointdata_python --- met/configure.ac | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/met/configure.ac b/met/configure.ac index 2fc599e47a..a20d682eff 100644 --- a/met/configure.ac +++ b/met/configure.ac @@ -1036,7 +1036,7 @@ if test "x$ENABLE_PYTHON" = "xyes"; then AC_DEFINE([ENABLE_PYTHON], [], ["build python embedding"]) AC_MSG_NOTICE([python embedding will be compiled]) CPPFLAGS="${CPPFLAGS} -DWITH_PYTHON" - PYTHON_LIBS="-lvx_data2d_python -lvx_python3_utils ${MET_PYTHON_LD}" + PYTHON_LIBS="-lvx_data2d_python -lvx_pointdata_python -lvx_python3_utils ${MET_PYTHON_LD}" else AC_MSG_NOTICE([python embedding will not be compiled]) PYTHON_LIBS= @@ -1233,6 +1233,7 @@ AC_CONFIG_FILES([Makefile src/libcode/vx_summary/Makefile src/libcode/vx_python3_utils/Makefile src/libcode/vx_data2d_python/Makefile + src/libcode/vx_pointdata_python/Makefile src/tools/Makefile src/tools/core/Makefile src/tools/core/ensemble_stat/Makefile From b4a4e18d509ee39458e8af8adc3f018e485a2a14 Mon Sep 17 00:00:00 2001 From: Howard Soh Date: Tue, 28 Dec 2021 15:32:05 -0700 Subject: [PATCH 25/82] #1844 Added vx_pointdata_python to SUBDIRS --- met/src/libcode/Makefile.am | 1 + 1 file changed, 1 insertion(+) diff --git a/met/src/libcode/Makefile.am b/met/src/libcode/Makefile.am index 76817f7749..8ed5d82c73 100644 --- a/met/src/libcode/Makefile.am +++ b/met/src/libcode/Makefile.am @@ -27,6 +27,7 @@ SUBDIRS = vx_grid \ if ENABLE_PYTHON SUBDIRS += vx_python3_utils SUBDIRS += vx_data2d_python + SUBDIRS += vx_pointdata_python endif From ba9f81e22fe63c0a4cc7621f0db2f818f453ca49 Mon Sep 17 00:00:00 2001 From: Howard Soh Date: Tue, 28 Dec 2021 15:33:59 -0700 Subject: [PATCH 26/82] #1844 Initial release --- met/src/libcode/vx_nc_obs/met_point_data.cc | 437 ++++++++++++++++++++ met/src/libcode/vx_nc_obs/met_point_data.h | 179 ++++++++ 2 files changed, 616 insertions(+) create mode 100644 met/src/libcode/vx_nc_obs/met_point_data.cc create mode 100644 met/src/libcode/vx_nc_obs/met_point_data.h diff --git a/met/src/libcode/vx_nc_obs/met_point_data.cc b/met/src/libcode/vx_nc_obs/met_point_data.cc new file mode 100644 index 0000000000..6cc15a7e81 --- /dev/null +++ b/met/src/libcode/vx_nc_obs/met_point_data.cc @@ -0,0 +1,437 @@ +// *=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=* +// ** Copyright UCAR (c) 1992 - 2021 +// ** University Corporation for Atmospheric Research (UCAR) +// ** National Center for Atmospheric Research (NCAR) +// ** Research Applications Lab (RAL) +// ** P.O.Box 3000, Boulder, Colorado, 80307-3000, USA +// *=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=* + + +//////////////////////////////////////////////////////////////////////// + + +using namespace std; + +#include +#include +#include +#include +#include +#include +#include + +#include "vx_log.h" +#include "is_bad_data.h" + +#include "met_point_data.h" +//#include "nc_point_obs.h" + +//////////////////////////////////////////////////////////////////////// + + + // + // Code for class MetPointData + // + + +//////////////////////////////////////////////////////////////////////// + +MetPointData::MetPointData() { + // Derived class should set obs_data + init_from_scratch(); +} + +//////////////////////////////////////////////////////////////////////// + +MetPointData::~MetPointData() { + clear(); +} + +//////////////////////////////////////////////////////////////////////// + +void MetPointData::init_from_scratch() { + nobs = 0; + nhdr = 0; + qty_length = 0; + + use_var_id = false; + use_arr_vars = false; +} + + +//////////////////////////////////////////////////////////////////////// + +//void MetPointData::allocate() { +// obs_data.allocate(); +//} + +//////////////////////////////////////////////////////////////////////// + +void MetPointData::clear() { + if (obs_data) obs_data->clear(); + header_data.clear(); +} + +//////////////////////////////////////////////////////////////////////// + +bool MetPointData::get_header(int header_offset, float hdr_arr[HDR_ARRAY_LEN], + ConcatString &hdr_typ_str, ConcatString &hdr_sid_str, + ConcatString &hdr_vld_str) { + int hdr_idx; + + // Read the corresponding header array for this observation + + hdr_arr[0] = header_data.lat_array[header_offset]; + hdr_arr[1] = header_data.lon_array[header_offset]; + hdr_arr[2] = header_data.elv_array[header_offset]; + + // Read the corresponding header type for this observation + hdr_idx = use_arr_vars ? header_offset : header_data.typ_idx_array[header_offset]; + hdr_typ_str = header_data.typ_array[hdr_idx]; + + // Read the corresponding header Station ID for this observation + hdr_idx = use_arr_vars ? header_offset : header_data.sid_idx_array[header_offset]; + hdr_sid_str = header_data.sid_array[hdr_idx]; + + // Read the corresponding valid time for this observation + hdr_idx = use_arr_vars ? header_offset : header_data.vld_idx_array[header_offset]; + hdr_vld_str = header_data.vld_array[hdr_idx]; + + return true; +} + +//////////////////////////////////////////////////////////////////////// + +bool MetPointData::get_header_type(int header_offset, int hdr_typ_arr[HDR_TYPE_ARR_LEN]) { + int hdr_idx; + // Read the header integer types + hdr_typ_arr[0] = (header_data.prpt_typ_array.n() > header_offset ? + header_data.prpt_typ_array[header_offset] : bad_data_int); + hdr_typ_arr[1] = (header_data.irpt_typ_array.n() > header_offset ? + header_data.irpt_typ_array[header_offset] : bad_data_int); + hdr_typ_arr[2] = (header_data.inst_typ_array.n() > header_offset ? + header_data.inst_typ_array[header_offset] : bad_data_int); + + return true; +} + +//////////////////////////////////////////////////////////////////////// + +bool MetPointData::get_lats(float *hdr_lats) { + for (int idx=0; idxobs_cnt = obs_cnt; +} + + + + +//////////////////////////////////////////////////////////////////////// + + + // + // Code for class MetPointDataPython + // + + +//////////////////////////////////////////////////////////////////////// + +MetPointDataPython::MetPointDataPython() { + obs_data = new MetPointObsData(); + init_from_scratch(); +} + +//////////////////////////////////////////////////////////////////////// + +MetPointDataPython::MetPointDataPython(MetPointDataPython &d) { + init_from_scratch(); + //obs_data = d.get_point_obs_data(); + //header_data = d.get_header_data(); + obs_data->assign(*d.get_point_obs_data()); + header_data.assign(*d.get_header_data()); +cout << " DEBUG HS MetPointData(MetPointData &d) is called \n"; +cout << " DEBUG HS MetPointData(MetPointData &d) &header_data.lat_array=" << &(header_data.lat_array) << "\n"; +cout << " DEBUG HS MetPointData(MetPointData &d) header_data.lat_array.n()=" << header_data.lat_array.n() << "\n"; +cout << " DEBUG HS MetPointData(MetPointData &d) header_data.lon_array.n()=" << header_data.lon_array.n() << "\n"; +cout << " DEBUG HS MetPointData(MetPointData &d) header_data.elv_array.n()=" << header_data.elv_array.n() << "\n"; +cout << " DEBUG HS MetPointData(MetPointData &d) header_data.typ_idx_array.n()=" << header_data.typ_idx_array.n() << "\n"; +cout << " DEBUG HS MetPointData(MetPointData &d) header_data.sid_idx_array.n()=" << header_data.sid_idx_array.n() << "\n"; +cout << " DEBUG HS MetPointData(MetPointData &d) header_data.vld_idx_array.n()=" << header_data.vld_idx_array.n() << "\n"; +} + + +//////////////////////////////////////////////////////////////////////// + +MetPointDataPython::~MetPointDataPython() { + clear(); +} + +//////////////////////////////////////////////////////////////////////// + +//void MetPointDataPython::init_from_scratch() { +// nobs = 0; +// nhdr = 0; +// qty_length = 0; +// +// use_var_id = false; +// use_arr_vars = false; +//} + + +//////////////////////////////////////////////////////////////////////// + +void MetPointDataPython::allocate(int obs_cnt) { + set_obs_cnt(obs_cnt); + obs_data->allocate(); +} + +//////////////////////////////////////////////////////////////////////// + + + +/////////////////////////////////////////////////////////////////////////////// +// struct MetPointObsData + +MetPointObsData::MetPointObsData(): + obs_cnt(0), + obs_ids((int *)0), + obs_hids((int *)0), + obs_qids((int *)0), + obs_lvls((float *)0), + obs_hgts((float *)0), + obs_vals((float *)0), + obs_arr((float *)0), + is_obs_array(false) +{ +} + + +/////////////////////////////////////////////////////////////////////////////// + +void MetPointObsData::allocate() { + if (is_obs_array) obs_arr = new float[obs_cnt*OBS_ARRAY_LEN]; // nobs * 5 + else { + obs_ids = new int[obs_cnt]; // grib_code or var_id + obs_hids = new int[obs_cnt]; + obs_qids = new int[obs_cnt]; + obs_lvls = new float[obs_cnt]; + obs_hgts = new float[obs_cnt]; + obs_vals = new float[obs_cnt]; + } +} + +/////////////////////////////////////////////////////////////////////////////// + +void MetPointObsData::assign(MetPointObsData &o) { + obs_cnt = o.obs_cnt; + is_obs_array = o.is_obs_array; + + clear(); + allocate(); + if (is_obs_array) + for (int idx=0; idx + +#include "nc_utils.h" + + +//////////////////////////////////////////////////////////////////////// + +struct MetPointHeader { + int typ_len; + int sid_len; + int vld_len; + int strl_len; + int strll_len; + int min_vld_time; + int max_vld_time; + int hdr_count; + int hdr_type_count; + + StringArray typ_array; + StringArray sid_array; + StringArray vld_array; // string array for valid time + IntArray vld_num_array; // number array for valid time + IntArray typ_idx_array; + IntArray sid_idx_array; + IntArray vld_idx_array; // index to vld_array/vld_num_array + NumArray lat_array; + NumArray lon_array; + NumArray elv_array; + IntArray prpt_typ_array; + IntArray irpt_typ_array; + IntArray inst_typ_array; + + MetPointHeader(); + void assign(MetPointHeader &h); + void clear(); + void reset_counters(); +}; + +//////////////////////////////////////////////////////////////////////// + +struct MetPointObsData { + int obs_cnt; + bool is_obs_array; + + int *obs_ids; // grib_code or var_id + int *obs_hids; + int *obs_qids; + float *obs_lvls; + float *obs_hgts; + float *obs_vals; + float *obs_arr; // nobs * 5 + StringArray var_names; + StringArray qty_names; + + MetPointObsData(); + void allocate(); + void assign(MetPointObsData &d); + void clear(); + void clear_numbers(); + void clear_strings(); + bool fill_obs_buf(int buf_size, int offset, float *obs_arr, int *qty_idx_arr); + float get_obs_val(int index); +}; + + +class MetPointData { + + protected: + + int nobs; + int nhdr; + int qty_length; + bool use_var_id; + bool use_arr_vars; + + MetPointHeader header_data; + MetPointObsData *obs_data = 0; + + void init_from_scratch(); + + public: + + MetPointData(); + ~MetPointData(); + + //void close(); + void clear(); + + int get_buf_size(); + int get_hdr_cnt(); + int get_grib_code_or_var_index(const float obs_arr[OBS_ARRAY_LEN]); + MetPointHeader *get_header_data(); + bool get_header(int header_offset, float hdr_arr[HDR_ARRAY_LEN], + ConcatString &hdr_typ_str, ConcatString &hdr_sid_str, + ConcatString &hdr_vld_str); + int get_header_offset(const float obs_arr[OBS_ARRAY_LEN]); + bool get_header_type(int header_offset, int hdr_typ_arr[HDR_TYPE_ARR_LEN]); + bool get_lats(float *hdr_lats); + bool get_lons(float *hdr_lons); + int get_obs_cnt(); + MetPointObsData *get_point_obs_data(); + StringArray get_qty_data(); + StringArray get_var_names(); + + bool is_same_obs_values(const float obs_arr1[OBS_ARRAY_LEN], const float obs_arr2[OBS_ARRAY_LEN]); + bool is_using_var_id(); + + void set_grib_code_or_var_index(float obs_arr[OBS_ARRAY_LEN], int grib_code); + void set_hdr_cnt(int hdr_cnt); + void set_obs_cnt(int obs_cnt); + // variables + + // data + +}; // MetPointDataBase + + + +class MetPointDataPython : public MetPointData { + + protected: + + public: + + MetPointDataPython(); + MetPointDataPython(MetPointDataPython &d); + ~MetPointDataPython(); + + void allocate(int obs_cnt); + void set_use_var_id(bool new_use_var_id); + + // variables + + // data + +}; // MetPointData + +//////////////////////////////////////////////////////////////////////// + +inline MetPointHeader *MetPointData::get_header_data() { return &header_data; } +inline int MetPointData::get_buf_size() { return OBS_BUFFER_SIZE; } +inline int MetPointData::get_grib_code_or_var_index(const float obs_arr[OBS_ARRAY_LEN]) { return obs_arr[1]; }; +inline int MetPointData::get_hdr_cnt() { return nhdr; } +inline int MetPointData::get_header_offset(const float obs_arr[OBS_ARRAY_LEN]) { return obs_arr[0]; }; +inline int MetPointData::get_obs_cnt() { return nobs; } +inline MetPointObsData *MetPointData::get_point_obs_data() { return obs_data; } +inline StringArray MetPointData::get_qty_data() { return obs_data->qty_names; } +inline StringArray MetPointData::get_var_names() { return obs_data->var_names; } +inline bool MetPointData::is_using_var_id() { return use_var_id; } +inline void MetPointData::set_grib_code_or_var_index(float obs_arr[OBS_ARRAY_LEN], int grib_code) { obs_arr[1] = grib_code; } +inline void MetPointDataPython::set_use_var_id(bool new_use_var_id) { use_var_id = new_use_var_id; } + +//////////////////////////////////////////////////////////////////////// + + +#endif /* __MET_POINT_DATA_H__ */ + + +//////////////////////////////////////////////////////////////////////// + From abef2b798c1851f0cbd8cbe42b98d3a621c60631 Mon Sep 17 00:00:00 2001 From: Howard Soh Date: Tue, 28 Dec 2021 15:34:46 -0700 Subject: [PATCH 27/82] #1844 Added met_point_data.cc & met_point_data.h --- met/src/libcode/vx_nc_obs/Makefile.am | 1 + 1 file changed, 1 insertion(+) diff --git a/met/src/libcode/vx_nc_obs/Makefile.am b/met/src/libcode/vx_nc_obs/Makefile.am index ba2fb8c910..22f0871fab 100644 --- a/met/src/libcode/vx_nc_obs/Makefile.am +++ b/met/src/libcode/vx_nc_obs/Makefile.am @@ -12,6 +12,7 @@ include ${top_srcdir}/Make-include noinst_LIBRARIES = libvx_nc_obs.a libvx_nc_obs_a_SOURCES = \ + met_point_data.cc met_point_data.h \ nc_obs_util.cc nc_obs_util.h \ nc_point_obs.cc nc_point_obs.h \ nc_point_obs_in.cc nc_point_obs_in.h \ From 6c36d79297baabfd741feda5236ce8442e05c85f Mon Sep 17 00:00:00 2001 From: Howard Soh Date: Tue, 28 Dec 2021 15:46:13 -0700 Subject: [PATCH 28/82] #1844 NcHeaderData is renamed to MetPointHeader and moved to met_point_dataq.h --- met/src/libcode/vx_nc_obs/nc_obs_util.h | 60 +++---------------------- 1 file changed, 6 insertions(+), 54 deletions(-) diff --git a/met/src/libcode/vx_nc_obs/nc_obs_util.h b/met/src/libcode/vx_nc_obs/nc_obs_util.h index 4985639c09..d3d07f3007 100644 --- a/met/src/libcode/vx_nc_obs/nc_obs_util.h +++ b/met/src/libcode/vx_nc_obs/nc_obs_util.h @@ -17,6 +17,7 @@ using namespace netCDF; #include "nc_summary.h" +#include "met_point_data.h" //////////////////////////////////////////////////////////////////////// @@ -78,39 +79,6 @@ struct NcDataBuffer { //////////////////////////////////////////////////////////////////////// -struct NcHeaderData { - bool valid_point_obs; - int typ_len; - int sid_len; - int vld_len; - int strl_len; - int strll_len; - int min_vld_time; - int max_vld_time; - int hdr_count; - int hdr_type_count; - - StringArray typ_array; - StringArray sid_array; - StringArray vld_array; - IntArray vld_num_array; - IntArray typ_idx_array; - IntArray sid_idx_array; - IntArray vld_idx_array; - NumArray lat_array; - NumArray lon_array; - NumArray elv_array; - IntArray prpt_typ_array; - IntArray irpt_typ_array; - IntArray inst_typ_array; - - NcHeaderData(); - void clear(); - void reset_counters(); -}; - -//////////////////////////////////////////////////////////////////////// - struct NetcdfObsVars { bool attr_agl ; bool attr_pb2nc ; @@ -167,7 +135,7 @@ struct NetcdfObsVars { void create_hdr_vars (NcFile *f_out, const int hdr_count); void create_obs_vars (NcFile *f_out); void create_obs_name_vars (NcFile *f_out, const int var_count, const int unit_count); - void create_table_vars (NcFile *f_out, NcHeaderData &hdr_data, NcDataBuffer &data_buffer); + void create_table_vars (NcFile *f_out, MetPointHeader &hdr_data, NcDataBuffer &data_buffer); void create_pb_hdrs (NcFile *f_out, const int hdr_count); NcDim create_var_obs_var (NcFile *f_out, int var_count); @@ -175,41 +143,25 @@ struct NetcdfObsVars { int get_obs_index(); void read_dims_vars(NcFile *f_in); - void read_header_data(NcHeaderData &hdr_data); + void read_header_data(MetPointHeader &hdr_data); bool read_obs_data(int buf_size, int offset, int qty_len, float *obs_arr, int *qty_idx_arr, char *obs_qty_buf); - void read_pb_hdr_data(NcHeaderData &hdr_data); + void read_pb_hdr_data(MetPointHeader &hdr_data); void write_header_to_nc(NcDataBuffer &data_buf, const int buf_size, const bool is_pb = false); void write_obs_buffer(NcDataBuffer &data_buffer, const int buf_size); void write_obs_var_names(StringArray &obs_names); void write_obs_var_units(StringArray &units); void write_obs_var_descriptions(StringArray &descriptions); - void write_table_vars(NcHeaderData &hdr_data, NcDataBuffer &data_buffer); + void write_table_vars(MetPointHeader &hdr_data, NcDataBuffer &data_buffer); }; // NetcdfObsVars //////////////////////////////////////////////////////////////////////// -struct NcPointObsData { - int obs_cnt; - bool is_obs_array; - - int *obs_ids; // grib_code or var_id - int *obs_hids; - int *obs_qids; - float *obs_lvls; - float *obs_hgts; - float *obs_vals; - float *obs_arr; // nobs * 5 - StringArray var_names; - StringArray qty_names; +struct NcPointObsData : public MetPointObsData { NcPointObsData(); - void clear(); - void clear_numbers(); - void clear_strings(); - float get_obs_val(int index); bool read_obs_data_numbers(NetcdfObsVars obs_vars, bool stop=true); bool read_obs_data_table_lookups(NetcdfObsVars obs_vars, bool stop=true); }; From 3f0de5e7e45c47e0fe9615b2c964003b4a8fcb69 Mon Sep 17 00:00:00 2001 From: Howard Soh Date: Tue, 28 Dec 2021 15:47:58 -0700 Subject: [PATCH 29/82] #1844 NcPointObsData is derived from MetPointObsData. Many methods are moved to the base class MetPointObsData --- met/src/libcode/vx_nc_obs/nc_obs_util.cc | 121 +---------------------- 1 file changed, 5 insertions(+), 116 deletions(-) diff --git a/met/src/libcode/vx_nc_obs/nc_obs_util.cc b/met/src/libcode/vx_nc_obs/nc_obs_util.cc index 70ea844d49..39a59266d2 100644 --- a/met/src/libcode/vx_nc_obs/nc_obs_util.cc +++ b/met/src/libcode/vx_nc_obs/nc_obs_util.cc @@ -68,78 +68,12 @@ void NcDataBuffer::reset_counters() { /////////////////////////////////////////////////////////////////////////////// // struct NcPointObsData -NcPointObsData::NcPointObsData(): - obs_cnt(0), - obs_ids((int *)0), - obs_hids((int *)0), - obs_qids((int *)0), - obs_lvls((float *)0), - obs_hgts((float *)0), - obs_vals((float *)0), - obs_arr((float *)0), - is_obs_array(false) +NcPointObsData::NcPointObsData() { } /////////////////////////////////////////////////////////////////////////////// -void NcPointObsData::clear() { - obs_cnt = 0; - is_obs_array = false; - - clear_numbers(); - clear_strings(); -} - -/////////////////////////////////////////////////////////////////////////////// - -void NcPointObsData::clear_numbers() { - if (0 != obs_ids) { - delete [] obs_ids; - obs_ids = (int *)0; - } - if (0 != obs_hids) { - delete [] obs_hids; - obs_hids = (int *)0; - } - if (0 != obs_qids) { - delete [] obs_qids; - obs_qids = (int *)0; - } - if (0 != obs_lvls) { - delete [] obs_lvls; - obs_lvls = (float *)0; - } - if (0 != obs_hgts) { - delete [] obs_hgts; - obs_hgts = (float *)0; - } - if (0 != obs_vals) { - delete [] obs_vals; - obs_vals = (float *)0; - } - if (0 != obs_arr) { - delete [] obs_arr; - obs_arr = (float *)0; - } -} - -/////////////////////////////////////////////////////////////////////////////// - -void NcPointObsData::clear_strings() { - var_names.clear(); - qty_names.clear(); -} - -/////////////////////////////////////////////////////////////////////////////// - -float NcPointObsData::get_obs_val(int index) { - float obs_val = (is_obs_array ? obs_arr[index*obs_cnt+4] : obs_vals[index]); - return obs_val; -} - -/////////////////////////////////////////////////////////////////////////////// - bool NcPointObsData::read_obs_data_numbers(NetcdfObsVars obs_vars, bool stop) { bool succeed = true; const char* method_name = "NcPointObsData::read_obs_data_numbers()"; @@ -434,7 +368,7 @@ void NetcdfObsVars::create_pb_hdrs (NcFile *f_out, const int hdr_count) { /////////////////////////////////////////////////////////////////////////////// -void NetcdfObsVars::create_table_vars (NcFile *f_out, NcHeaderData &hdr_data, +void NetcdfObsVars::create_table_vars (NcFile *f_out, MetPointHeader &hdr_data, NcDataBuffer &data_buffer) { const string method_name = " create_table_vars()"; @@ -603,7 +537,7 @@ void NetcdfObsVars::read_dims_vars(NcFile *f_in) { //////////////////////////////////////////////////////////////////////// -void NetcdfObsVars::read_header_data(NcHeaderData &hdr_data) { +void NetcdfObsVars::read_header_data(MetPointHeader &hdr_data) { bool is_valid_obs_nc = true; long nhdr_count = get_dim_size(&hdr_dim); int strl_len = get_dim_size(&strl_dim); @@ -970,7 +904,7 @@ bool NetcdfObsVars::read_obs_data(int buf_size, int offset, //////////////////////////////////////////////////////////////////////// -void NetcdfObsVars::read_pb_hdr_data(NcHeaderData &hdr_data) { +void NetcdfObsVars::read_pb_hdr_data(MetPointHeader &hdr_data) { const char *method_name = "get_nc_pb_hdr_data() -> "; int pb_hdr_count = get_dim_size(&pb_hdr_dim); @@ -1166,7 +1100,7 @@ void NetcdfObsVars::write_header_to_nc(NcDataBuffer &data_buf, /////////////////////////////////////////////////////////////////////////////// -void NetcdfObsVars::write_table_vars (NcHeaderData &hdr_data, NcDataBuffer &data_buffer) +void NetcdfObsVars::write_table_vars (MetPointHeader &hdr_data, NcDataBuffer &data_buffer) { mlog << Debug(5) << "write_table_vars() is called. valid hdr_typ_tbl_var: " << !IS_INVALID_NC(hdr_typ_tbl_var) << "\n"; @@ -1261,51 +1195,6 @@ void NetcdfObsVars::write_obs_var_descriptions(StringArray &descriptions) { //////////////////////////////////////////////////////////////////////// // End of NetcdfObsVars -/////////////////////////////////////////////////////////////////////////////// - -// struct NcHeaderData - -NcHeaderData::NcHeaderData() -{ - reset_counters(); -} - -/////////////////////////////////////////////////////////////////////////////// - -void NcHeaderData::clear() { - reset_counters(); - - typ_array.clear(); - sid_array.clear(); - vld_array.clear(); - vld_num_array.clear(); - typ_idx_array.clear(); - sid_idx_array.clear(); - vld_idx_array.clear(); - lat_array.clear(); - lon_array.clear(); - elv_array.clear(); - prpt_typ_array.clear(); - irpt_typ_array.clear(); - inst_typ_array.clear(); -} - -/////////////////////////////////////////////////////////////////////////////// - -void NcHeaderData::reset_counters() { - valid_point_obs = false; - typ_len = 0; - sid_len = 0; - vld_len = 0; - strl_len = 0; - strll_len = 0; - hdr_count = 0; - - min_vld_time = -1; - max_vld_time = -1; -} - - /////////////////////////////////////////////////////////////////////////////// long count_nc_headers(vector< Observation > &observations) From 6c50d500ffb0b21f2b29221a8e12fbbcda212702 Mon Sep 17 00:00:00 2001 From: Howard Soh Date: Tue, 28 Dec 2021 15:52:54 -0700 Subject: [PATCH 30/82] #1844 Moved varianbles and methods to the base class MetPointData --- met/src/libcode/vx_nc_obs/nc_point_obs.cc | 92 ++--------------------- met/src/libcode/vx_nc_obs/nc_point_obs.h | 42 +---------- 2 files changed, 7 insertions(+), 127 deletions(-) diff --git a/met/src/libcode/vx_nc_obs/nc_point_obs.cc b/met/src/libcode/vx_nc_obs/nc_point_obs.cc index b665e9d2e9..0869f3a31e 100644 --- a/met/src/libcode/vx_nc_obs/nc_point_obs.cc +++ b/met/src/libcode/vx_nc_obs/nc_point_obs.cc @@ -36,6 +36,7 @@ using namespace std; //////////////////////////////////////////////////////////////////////// MetNcPointObs::MetNcPointObs() { + obs_data = new NcPointObsData(); init_from_scratch(); } @@ -48,104 +49,21 @@ MetNcPointObs::~MetNcPointObs() { //////////////////////////////////////////////////////////////////////// void MetNcPointObs::init_from_scratch() { - obs_nc = (NcFile *) 0; + MetPointData::init_from_scratch(); - nobs = 0; - nhdr = 0; - qty_length = 0; keep_nc = false; - use_var_id = false; - use_arr_vars = false; - - return; + obs_nc = (NcFile *) 0; } //////////////////////////////////////////////////////////////////////// void MetNcPointObs::close() { + MetPointData::clear(); + if ( !keep_nc && obs_nc ) { delete obs_nc; obs_nc = (NcFile *) 0; } - - obs_data.clear(); - header_data.clear(); - return; -} - -//////////////////////////////////////////////////////////////////////// - -int MetNcPointObs::get_qty_length() { - qty_length = get_nc_string_length(&obs_vars.obs_qty_tbl_var); - return qty_length; -} - -//////////////////////////////////////////////////////////////////////// - -bool MetNcPointObs::get_header(int header_offset, float hdr_arr[HDR_ARRAY_LEN], - ConcatString &hdr_typ_str, ConcatString &hdr_sid_str, ConcatString &hdr_vld_str) { - int hdr_idx; - - // Read the corresponding header array for this observation - hdr_arr[0] = header_data.lat_array[header_offset]; - hdr_arr[1] = header_data.lon_array[header_offset]; - hdr_arr[2] = header_data.elv_array[header_offset]; - - // Read the corresponding header type for this observation - hdr_idx = use_arr_vars ? header_offset : header_data.typ_idx_array[header_offset]; - hdr_typ_str = header_data.typ_array[hdr_idx]; - - // Read the corresponding header Station ID for this observation - hdr_idx = use_arr_vars ? header_offset : header_data.sid_idx_array[header_offset]; - hdr_sid_str = header_data.sid_array[hdr_idx]; - - // Read the corresponding valid time for this observation - hdr_idx = use_arr_vars ? header_offset : header_data.vld_idx_array[header_offset]; - hdr_vld_str = header_data.vld_array[hdr_idx]; - - return true; -} - -//////////////////////////////////////////////////////////////////////// - -bool MetNcPointObs::get_header_type(int header_offset, int hdr_typ_arr[HDR_TYPE_ARR_LEN]) { - int hdr_idx; - // Read the header integer types - hdr_typ_arr[0] = (header_data.prpt_typ_array.n() > header_offset ? - header_data.prpt_typ_array[header_offset] : bad_data_int); - hdr_typ_arr[1] = (header_data.irpt_typ_array.n() > header_offset ? - header_data.irpt_typ_array[header_offset] : bad_data_int); - hdr_typ_arr[2] = (header_data.inst_typ_array.n() > header_offset ? - header_data.inst_typ_array[header_offset] : bad_data_int); - - return true; -} - -//////////////////////////////////////////////////////////////////////// - -bool MetNcPointObs::get_lats(float *hdr_lats) { - for (int idx=0; idx Date: Tue, 28 Dec 2021 15:54:30 -0700 Subject: [PATCH 31/82] #1844 override the obs_data pointer to the derived class --- met/src/libcode/vx_nc_obs/nc_point_obs_in.cc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/met/src/libcode/vx_nc_obs/nc_point_obs_in.cc b/met/src/libcode/vx_nc_obs/nc_point_obs_in.cc index 3345edb5a7..3a91441deb 100644 --- a/met/src/libcode/vx_nc_obs/nc_point_obs_in.cc +++ b/met/src/libcode/vx_nc_obs/nc_point_obs_in.cc @@ -94,7 +94,7 @@ bool MetNcPointObsIn::read_obs_data(int buf_size, int start, float *obs_arr_bloc bool MetNcPointObsIn::read_obs_data_numbers() { bool status = false; if( IS_VALID_NC_P(obs_nc) ) { - status = obs_data.read_obs_data_numbers(obs_vars); + status = ((NcPointObsData *)obs_data)->read_obs_data_numbers(obs_vars); } return status; } @@ -104,7 +104,7 @@ bool MetNcPointObsIn::read_obs_data_numbers() { bool MetNcPointObsIn::read_obs_data_table_lookups() { bool status = false; if( IS_VALID_NC_P(obs_nc) ) { - status = obs_data.read_obs_data_table_lookups(obs_vars); + status = ((NcPointObsData *)obs_data)->read_obs_data_table_lookups(obs_vars); } return status; } From b053f714ec4873d4a63d82930142e4393beb5744 Mon Sep 17 00:00:00 2001 From: Howard Soh Date: Tue, 28 Dec 2021 15:55:59 -0700 Subject: [PATCH 32/82] #1844 Added pyobject_as_bool & pyobject_as_string_array --- .../libcode/vx_python3_utils/python3_util.cc | 36 ++++++++++++++++++- .../libcode/vx_python3_utils/python3_util.h | 2 ++ 2 files changed, 37 insertions(+), 1 deletion(-) diff --git a/met/src/libcode/vx_python3_utils/python3_util.cc b/met/src/libcode/vx_python3_utils/python3_util.cc index fe56b6f6b2..ab17acd7b8 100644 --- a/met/src/libcode/vx_python3_utils/python3_util.cc +++ b/met/src/libcode/vx_python3_utils/python3_util.cc @@ -12,7 +12,6 @@ using namespace std; #include #include "vx_log.h" -#include "concat_string.h" #include "vx_math.h" #include "python3_util.h" @@ -125,6 +124,18 @@ return ( k ); //////////////////////////////////////////////////////////////////////// +bool pyobject_as_bool (PyObject * obj) + +{ + +return ( 1 == PyObject_IsTrue(obj) ); + +} + + +//////////////////////////////////////////////////////////////////////// + + double pyobject_as_double (PyObject * obj) { @@ -215,6 +226,29 @@ return ( s ); //////////////////////////////////////////////////////////////////////// +StringArray pyobject_as_string_array (PyObject * obj) + +{ + +StringArray a; +ConcatString s; +PyObject *item = 0; + +int size = PyList_Size (obj); +for (int idx=0; idx Date: Tue, 28 Dec 2021 16:06:28 -0700 Subject: [PATCH 33/82] #1844 Cleanup include statements and addpointdata_python.h if python is enabled --- met/src/tools/other/plot_point_obs/plot_point_obs.h | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/met/src/tools/other/plot_point_obs/plot_point_obs.h b/met/src/tools/other/plot_point_obs/plot_point_obs.h index 7fec0e19f0..0bf33b0679 100644 --- a/met/src/tools/other/plot_point_obs/plot_point_obs.h +++ b/met/src/tools/other/plot_point_obs/plot_point_obs.h @@ -43,12 +43,13 @@ using namespace netCDF; #include "plot_point_obs_conf_info.h" -#include "vx_data2d_factory.h" -#include "vx_grid.h" #include "vx_util.h" #include "vx_stat_out.h" #include "vx_gsl_prob.h" #include "nc_utils.h" +#ifdef WITH_PYTHON +#include "pointdata_python.h" +#endif //////////////////////////////////////////////////////////////////////// // From e397fa12eb11df51efb22cbcae1fb0830b0b142d Mon Sep 17 00:00:00 2001 From: Howard Soh Date: Tue, 28 Dec 2021 16:14:03 -0700 Subject: [PATCH 34/82] #18443 Support python embedding --- .../tools/core/ensemble_stat/ensemble_stat.cc | 110 ++++++-- met/src/tools/core/point_stat/point_stat.cc | 115 +++++--- .../other/plot_point_obs/plot_point_obs.cc | 143 ++++++---- met/src/tools/other/point2grid/point2grid.cc | 254 ++++++++++++++---- 4 files changed, 458 insertions(+), 164 deletions(-) diff --git a/met/src/tools/core/ensemble_stat/ensemble_stat.cc b/met/src/tools/core/ensemble_stat/ensemble_stat.cc index fe5d160157..422c8d6f84 100644 --- a/met/src/tools/core/ensemble_stat/ensemble_stat.cc +++ b/met/src/tools/core/ensemble_stat/ensemble_stat.cc @@ -92,6 +92,11 @@ using namespace std; #include "nc_obs_util.h" #include "nc_point_obs_in.h" +#ifdef WITH_PYTHON +#include "data2d_nc_met.h" +#include "pointdata_python.h" +#endif + //////////////////////////////////////////////////////////////////////// static void process_command_line (int, char **); @@ -992,25 +997,59 @@ void process_point_obs(int i_nc) { << point_obs_file_list[i_nc] << "\n"; // Open the observation file as a NetCDF file. + bool status; + bool use_var_id = true; + bool use_arr_vars = false; + bool use_python = false; MetNcPointObsIn nc_point_obs; - if(!nc_point_obs.open(point_obs_file_list[i_nc].c_str())) { - nc_point_obs.close(); + MetPointData *met_point_obs = 0; +#ifdef WITH_PYTHON + MetPythonPointDataFile met_point_file; + string python_command = point_obs_file_list[i_nc]; + bool use_xarray = (0 == python_command.find(conf_val_python_xarray)); + use_python = use_xarray || (0 == python_command.find(conf_val_python_numpy)); + if (use_python) { + int offset = python_command.find("="); + if (offset == std::string::npos) { + mlog << Error << "\n" << method_name + << "trouble parsing the python command " << python_command << ".\n\n"; + exit(1); + } - mlog << Warning << "\n" << method_name - << "can't open observation netCDF file: " - << point_obs_file_list[i_nc] << "\n\n"; - return; - } + if(!met_point_file.open(python_command.substr(offset+1).c_str(), use_xarray)) { + met_point_file.close(); + mlog << Error << "\n" << method_name + << "trouble getting point observation file from python command " + << python_command << ".\n\n"; + exit(1); + } - // Read the dimensions and variables - nc_point_obs.read_dim_headers(); - nc_point_obs.check_nc(point_obs_file_list[i_nc].c_str(), method_name); // exit if missing dims/vars - nc_point_obs.read_obs_data_table_lookups(); + met_point_obs = met_point_file.get_met_point_data(); + } + else { +#endif + if(!nc_point_obs.open(point_obs_file_list[i_nc].c_str())) { + nc_point_obs.close(); + + mlog << Warning << "\n" << method_name + << "can't open observation netCDF file: " + << point_obs_file_list[i_nc] << "\n\n"; + return; + } + + // Read the dimensions and variables + nc_point_obs.read_dim_headers(); + nc_point_obs.check_nc(point_obs_file_list[i_nc].c_str(), method_name); // exit if missing dims/vars + nc_point_obs.read_obs_data_table_lookups(); + met_point_obs = (MetPointData *)&nc_point_obs; + use_var_id = nc_point_obs.is_using_var_id(); + use_arr_vars = nc_point_obs.is_using_obs_arr(); +#ifdef WITH_PYTHON + } +#endif - int hdr_count = nc_point_obs.get_hdr_cnt(); - int obs_count = nc_point_obs.get_obs_cnt(); - bool use_var_id = nc_point_obs.is_using_var_id(); - bool use_arr_vars = nc_point_obs.is_using_obs_arr(); + int hdr_count = met_point_obs->get_hdr_cnt(); + int obs_count = met_point_obs->get_obs_cnt(); mlog << Debug(2) << "Searching " << (obs_count) << " observations from " << (hdr_count) @@ -1028,17 +1067,23 @@ void process_point_obs(int i_nc) { ConcatString obs_qty_str; ConcatString var_name; StringArray var_names; - StringArray obs_qty_array = nc_point_obs.get_qty_data(); - if(use_var_id) var_names = nc_point_obs.get_var_names(); + StringArray obs_qty_array = met_point_obs->get_qty_data(); + if(use_var_id) var_names = met_point_obs->get_var_names(); for(int i_start=0; i_start DEF_NC_BUFFER_SIZE) - ? DEF_NC_BUFFER_SIZE : (obs_count-i_start); + int buf_size2 = (obs_count-i_start); + if (buf_size2 > DEF_NC_BUFFER_SIZE) buf_size2 = DEF_NC_BUFFER_SIZE; + +#ifdef WITH_PYTHON + if (use_python) + status = met_point_obs->get_point_obs_data()->fill_obs_buf( + buf_size2, i_start, (float *)obs_arr_block, obs_qty_idx_block); + else +#endif + status = nc_point_obs.read_obs_data(buf_size2, i_start, (float *)obs_arr_block, + obs_qty_idx_block, (char *)0); + if (!status) exit(1); - if (!nc_point_obs.read_obs_data(buf_size, i_start, (float *)obs_arr_block, - obs_qty_idx_block, (char *)0)) { - exit(1); - } // Process each observation in the file for(int i_offset=0; i_offsetget_header_offset(obs_arr); // Range check the header offset if(headerOffset < 0 || headerOffset >= hdr_count) { @@ -1065,16 +1110,21 @@ void process_point_obs(int i_nc) { // Read the corresponding header array for this observation // - the corresponding header type, header Station ID, and valid time - nc_point_obs.get_header(headerOffset, hdr_arr, hdr_typ_str, - hdr_sid_str, hdr_vld_str); +#ifdef WITH_PYTHON + if (use_python) + met_point_obs->get_header(headerOffset, hdr_arr, hdr_typ_str, hdr_sid_str, hdr_vld_str); + else +#endif + nc_point_obs.get_header(headerOffset, hdr_arr, hdr_typ_str, + hdr_sid_str, hdr_vld_str); // Read the header integer types - nc_point_obs.get_header_type(headerOffset, hdr_typ_arr); + met_point_obs->get_header_type(headerOffset, hdr_typ_arr); // Convert string to a unixtime hdr_ut = timestring_to_unix(hdr_vld_str.c_str()); - int grib_code = nc_point_obs.get_grib_code_or_var_index(obs_arr); + int grib_code = met_point_obs->get_grib_code_or_var_index(obs_arr); if (use_var_id && grib_code < var_names.n()) { var_name = var_names[grib_code]; } @@ -1097,6 +1147,10 @@ void process_point_obs(int i_nc) { } // end for i_start // Deallocate and clean up +#ifdef WITH_PYTHON + if (use_python) met_point_file.close(); + else +#endif nc_point_obs.close(); return; diff --git a/met/src/tools/core/point_stat/point_stat.cc b/met/src/tools/core/point_stat/point_stat.cc index 549e6fce9f..4578d91a33 100644 --- a/met/src/tools/core/point_stat/point_stat.cc +++ b/met/src/tools/core/point_stat/point_stat.cc @@ -126,6 +126,11 @@ using namespace std; #include "nc_obs_util.h" #include "nc_point_obs_in.h" +#ifdef WITH_PYTHON +#include "data2d_nc_met.h" +#include "pointdata_python.h" +#endif + //////////////////////////////////////////////////////////////////////// #define BUFFER_SIZE (DEF_NC_BUFFER_SIZE/2) @@ -671,33 +676,67 @@ void process_obs_file(int i_nc) { // Open the observation file as a NetCDF file. // The observation file must be in NetCDF format as the // output of the PB2NC or ASCII2NC tool. + bool status; + bool use_var_id = true; + bool use_arr_vars = false; + bool use_python = false; MetNcPointObsIn nc_point_obs; - if( !nc_point_obs.open(obs_file[i_nc].c_str()) ) { - nc_point_obs.close(); + MetPointData *met_point_obs = 0; +#ifdef WITH_PYTHON + MetPythonPointDataFile met_point_file; + string python_command = obs_file[i_nc]; + bool use_xarray = (0 == python_command.find(conf_val_python_xarray)); + use_python = use_xarray || (0 == python_command.find(conf_val_python_numpy)); + if (use_python) { + int offset = python_command.find("="); + if (offset == std::string::npos) { + mlog << Error << "\n" << method_name + << "trouble parsing the python command " << python_command << ".\n\n"; + exit(1); + } - mlog << Warning << "\n" << method_name - << "can't open observation netCDF file: " - << obs_file[i_nc] << "\n\n"; - return; - } + if(!met_point_file.open(python_command.substr(offset+1).c_str(), use_xarray)) { + met_point_file.close(); + mlog << Error << "\n" << method_name + << "trouble getting point observation file from python command " + << python_command << ".\n\n"; + exit(1); + } - nc_point_obs.read_dim_headers(); - nc_point_obs.check_nc(obs_file[i_nc].c_str(), method_name); - nc_point_obs.read_obs_data_table_lookups(); + met_point_obs = met_point_file.get_met_point_data(); + } + else { +#endif + if( !nc_point_obs.open(obs_file[i_nc].c_str()) ) { + nc_point_obs.close(); + + mlog << Warning << "\n" << method_name + << "can't open observation netCDF file: " + << obs_file[i_nc] << "\n\n"; + return; + } + + nc_point_obs.read_dim_headers(); + nc_point_obs.check_nc(obs_file[i_nc].c_str(), method_name); + nc_point_obs.read_obs_data_table_lookups(); + met_point_obs = (MetPointData *)&nc_point_obs; + use_var_id = nc_point_obs.is_using_var_id(); + use_arr_vars = nc_point_obs.is_using_obs_arr(); +#ifdef WITH_PYTHON + } +#endif - bool use_var_id = nc_point_obs.is_using_var_id(); - int hdr_count = nc_point_obs.get_hdr_cnt(); - int obs_count = nc_point_obs.get_obs_cnt(); + int hdr_count = met_point_obs->get_hdr_cnt(); + int obs_count = met_point_obs->get_obs_cnt(); mlog << Debug(2) << "Searching " << obs_count << " observations from " << hdr_count << " messages.\n"; ConcatString var_name(""); - bool use_arr_vars = nc_point_obs.is_using_obs_arr(); StringArray var_names; - StringArray obs_qty_array = nc_point_obs.get_qty_data(); - if( use_var_id ) var_names = nc_point_obs.get_var_names(); + StringArray obs_qty_array = met_point_obs->get_qty_data(); + if( use_var_id ) var_names = met_point_obs->get_var_names(); int buf_size = ((obs_count > BUFFER_SIZE) ? BUFFER_SIZE : (obs_count)); int obs_qty_idx_block[buf_size]; @@ -706,17 +745,22 @@ void process_obs_file(int i_nc) { // Process each observation in the file int str_length, block_size; for(int i_block_start_idx=0; i_block_start_idx BUFFER_SIZE) block_size = BUFFER_SIZE; - - if (!nc_point_obs.read_obs_data(block_size, i_block_start_idx, - (float *)obs_arr_block, - obs_qty_idx_block, (char *)0)) { - exit(1); - } + int block_size2 = (obs_count - i_block_start_idx); + if (block_size2 > BUFFER_SIZE) block_size2 = BUFFER_SIZE; + +#ifdef WITH_PYTHON + if (use_python) + status = met_point_obs->get_point_obs_data()->fill_obs_buf( + block_size2, i_block_start_idx, (float *)obs_arr_block, obs_qty_idx_block); + else +#endif + status = nc_point_obs.read_obs_data(block_size2, i_block_start_idx, + (float *)obs_arr_block, + obs_qty_idx_block, (char *)0); + if (!status) exit(1); int hdr_idx; - for(int i_block_idx=0; i_block_idxget_header_offset(obs_arr); // Range check the header offset if(headerOffset < 0 || headerOffset >= hdr_count) { @@ -740,11 +784,16 @@ void process_obs_file(int i_nc) { // Read the corresponding header array for this observation // - the corresponding header type, header Station ID, and valid time - nc_point_obs.get_header(headerOffset, hdr_arr, hdr_typ_str, - hdr_sid_str, hdr_vld_str); +#ifdef WITH_PYTHON + if (use_python) + met_point_obs->get_header(headerOffset, hdr_arr, hdr_typ_str, hdr_sid_str, hdr_vld_str); + else +#endif + nc_point_obs.get_header(headerOffset, hdr_arr, hdr_typ_str, + hdr_sid_str, hdr_vld_str); // Store the variable name - int org_grib_code = nc_point_obs.get_grib_code_or_var_index(obs_arr); + int org_grib_code = met_point_obs->get_grib_code_or_var_index(obs_arr); int grib_code = org_grib_code; if (use_var_id && grib_code < var_names.n()) { var_name = var_names[grib_code]; @@ -773,7 +822,7 @@ void process_obs_file(int i_nc) { // and at the same vertical level. if(vflag && is_vgrd) { - if(!nc_point_obs.is_same_obs_values(obs_arr, prev_obs_arr)) { + if(!met_point_obs->is_same_obs_values(obs_arr, prev_obs_arr)) { mlog << Error << "\n" << method_name << "for observation index " << i_obs << ", when computing VL1L2 and/or VAL1L2 vector winds " @@ -801,12 +850,16 @@ void process_obs_file(int i_nc) { grid, var_name.c_str()); } - nc_point_obs.set_grib_code_or_var_index(obs_arr, org_grib_code); + met_point_obs->set_grib_code_or_var_index(obs_arr, org_grib_code); } } // end for i_block_start_idx // Deallocate and clean up +#ifdef WITH_PYTHON + if (use_python) met_point_file.close(); + else +#endif nc_point_obs.close(); return; diff --git a/met/src/tools/other/plot_point_obs/plot_point_obs.cc b/met/src/tools/other/plot_point_obs/plot_point_obs.cc index 1ba1b9938b..af45a93641 100644 --- a/met/src/tools/other/plot_point_obs/plot_point_obs.cc +++ b/met/src/tools/other/plot_point_obs/plot_point_obs.cc @@ -45,26 +45,11 @@ using namespace std; #include "plot_point_obs.h" -#include "nc_utils.h" -#include "vx_log.h" -#include "data_plane.h" #include "data_plane_plot.h" -#include "write_netcdf.h" -#include "vx_data2d.h" -#include "vx_data2d_factory.h" -#include "vx_data2d_grib.h" -#include "vx_data2d_nc_met.h" -#include "vx_data2d_nc_pinterp.h" -#include "vx_util.h" -#include "vx_cal.h" -#include "vx_grid.h" -#include "vx_math.h" -#include "vx_color.h" #include "vx_ps.h" #include "vx_pxm.h" #include "vx_render.h" #include "vx_plot_util.h" -#include "nc_obs_util.h" #include "nc_point_obs_in.h" //////////////////////////////////////////////////////////////////////// @@ -149,42 +134,76 @@ int main(int argc, char *argv[]) { void process_point_obs(const char *point_obs_filename) { int h, v, i_obs; - const char *method_name = "\nprocess_point_obs() -> "; - const char *method_name_s = "\nprocess_point_obs() "; + const char *method_name = "process_point_obs() -> "; + const char *method_name_s = "process_point_obs() "; // Open the netCDF point observation file mlog << Debug(1) << "Reading point observation file: " << point_obs_filename << "\n"; + bool status; + bool use_var_id = true; + bool use_obs_arr = false; + + bool use_python = false; MetNcPointObsIn nc_point_obs; - if(!nc_point_obs.open(point_obs_filename)) { - nc_point_obs.close(); + MetPointData *met_point_obs = 0; +#ifdef WITH_PYTHON + MetPythonPointDataFile met_point_file; + string python_command = point_obs_filename; + bool use_xarray = (0 == python_command.find(conf_val_python_xarray)); + use_python = use_xarray || (0 == python_command.find(conf_val_python_numpy)); + if (use_python) { + int offset = python_command.find("="); + if (offset == std::string::npos) { + mlog << Error << "\n" << method_name + << "trouble parsing the python command " << python_command << ".\n\n"; + exit(1); + } - mlog << Error << "\n" << method_name - << "trouble opening point observation file " - << point_obs_filename << ".\n\n"; + if(!met_point_file.open(python_command.substr(offset+1).c_str(), use_xarray)) { + met_point_file.close(); + mlog << Error << "\n" << method_name + << "trouble getting point observation file from python command " + << python_command << ".\n\n"; + exit(1); + } - exit(1); + met_point_obs = met_point_file.get_met_point_data(); } + else +#endif + { + if(!nc_point_obs.open(point_obs_filename)) { + nc_point_obs.close(); - // Read the dimensions and variables - nc_point_obs.read_dim_headers(); - nc_point_obs.check_nc(point_obs_filename, method_name_s); // exit if missing dims/vars - nc_point_obs.read_obs_data(); + mlog << Error << "\n" << method_name + << "trouble opening point observation file " + << point_obs_filename << ".\n\n"; - bool use_var_id = nc_point_obs.is_using_var_id(); + exit(1); + } - // Print warning about ineffective command line arguments - if(use_var_id && ivar.n() != 0) { - mlog << Warning << "\n" << method_name << "-gc option is ignored!\n\n"; - } - if(!use_var_id && svar.n() != 0) { - mlog << Warning << "\n" << method_name << "-obs_var option is ignored!\n\n"; - } + // Read the dimensions and variables + nc_point_obs.read_dim_headers(); + nc_point_obs.check_nc(point_obs_filename, method_name_s); // exit if missing dims/vars + nc_point_obs.read_obs_data(); + use_var_id = nc_point_obs.is_using_var_id(); + use_obs_arr = nc_point_obs.is_using_obs_arr(); - long nhdr_count = nc_point_obs.get_hdr_cnt(); - long nobs_count = nc_point_obs.get_obs_cnt(); + // Print warning about ineffective command line arguments + if(use_var_id && ivar.n() != 0) { + mlog << Warning << "\n" << method_name << "-gc option is ignored!\n\n"; + } + if(!use_var_id && svar.n() != 0) { + mlog << Warning << "\n" << method_name << "-obs_var option is ignored!\n\n"; + } + met_point_obs = (MetPointData *)&nc_point_obs; + } + + long nhdr_count = met_point_obs->get_hdr_cnt(); + long nobs_count = met_point_obs->get_obs_cnt(); mlog << Debug(2) << "Processing " << nobs_count << " observations at " << nhdr_count << " locations.\n"; @@ -192,8 +211,6 @@ void process_point_obs(const char *point_obs_filename) { // Get the corresponding header: // message type, staton_id, valid_time, and lat/lon/elv - bool use_obs_arr = nc_point_obs.is_using_obs_arr(); - int buf_size = (nobs_count > DEF_NC_BUFFER_SIZE) ? DEF_NC_BUFFER_SIZE : nobs_count; // Allocate space to store the data @@ -202,8 +219,17 @@ void process_point_obs(const char *point_obs_filename) { int obs_qty_block[buf_size]; float obs_arr_block[buf_size][OBS_ARRAY_LEN]; - if(use_var_id) var_list = nc_point_obs.get_var_names(); - qty_list = nc_point_obs.get_qty_data(); + if(use_var_id) var_list = met_point_obs->get_var_names(); + qty_list = met_point_obs->get_qty_data(); + + if (0 == qty_list.n()) { + mlog << Error << "\n" << method_name << "Missing quality marks\n\n"; + exit (1); + } + if (use_var_id && (0 == var_list.n())) { + mlog << Debug(1) << method_name << "Missing variable names\ns\n"; + exit (1); + } ConcatString hdr_typ_str; ConcatString hdr_sid_str; @@ -211,16 +237,21 @@ void process_point_obs(const char *point_obs_filename) { ConcatString obs_qty_str; for(int i_start=0; i_start DEF_NC_BUFFER_SIZE) ? - DEF_NC_BUFFER_SIZE : (nobs_count-i_start); - - if (!nc_point_obs.read_obs_data(buf_size, i_start, (float *)obs_arr_block, - obs_qty_block, (char *)0)) { - exit(1); - } + int buf_size2 = ((nobs_count-i_start) > DEF_NC_BUFFER_SIZE) ? + DEF_NC_BUFFER_SIZE : (nobs_count-i_start); + +#ifdef WITH_PYTHON + if (use_python) + status = met_point_obs->get_point_obs_data()->fill_obs_buf( + buf_size2, i_start, (float *)obs_arr_block, obs_qty_block); + else +#endif + status = nc_point_obs.read_obs_data(buf_size2, i_start, (float *)obs_arr_block, + obs_qty_block, (char *)0); + if (!status) exit(1); int typ_idx, sid_idx, vld_idx; - for(int i_offset=0; i_offsetget_header(h, hdr_arr, hdr_typ_str, hdr_sid_str, hdr_vld_str); + else +#endif + nc_point_obs.get_header(h, hdr_arr, hdr_typ_str, hdr_sid_str, hdr_vld_str); // Store data in an observation object Observation cur_obs( @@ -263,12 +299,15 @@ void process_point_obs(const char *point_obs_filename) { } // end for i_start // Clean up +#ifdef WITH_PYTHON + if (use_python) met_point_file.close(); + else +#endif nc_point_obs.close(); return; } - //////////////////////////////////////////////////////////////////////// void create_plot() { @@ -470,7 +509,7 @@ void create_plot() { if(use_flate) plot.end_flate(); mlog << Debug(2) - << "Finished plotting " << plot_count << " locations.\n" + << "Finished plotting " << plot_count << " locations for " << ps_file << ".\n" << "Skipped " << skip_count << " locations off the grid.\n"; plot.showpage(); diff --git a/met/src/tools/other/point2grid/point2grid.cc b/met/src/tools/other/point2grid/point2grid.cc index 7b840f08d9..f806023c11 100644 --- a/met/src/tools/other/point2grid/point2grid.cc +++ b/met/src/tools/other/point2grid/point2grid.cc @@ -43,6 +43,11 @@ using namespace std; #include "point2grid_conf_info.h" +#ifdef WITH_PYTHON +#include "data2d_nc_met.h" +#include "pointdata_python.h" +#endif + //////////////////////////////////////////////////////////////////////// static ConcatString program_name; @@ -53,6 +58,7 @@ static const int TYPE_OBS = 1; // MET Point Obs NetCDF (from xxx2nc) static const int TYPE_NCCF = 2; // CF NetCDF with time and lat/lon variables static const int TYPE_GOES = 5; static const int TYPE_GOES_ADP = 6; +static const int TYPE_PYTHON = 7; // MET Point Obs NetCDF from PYTHON static const InterpMthd DefaultInterpMthd = InterpMthd_UW_Mean; static const int DefaultInterpWdth = 2; @@ -120,14 +126,19 @@ static NcDim lon_dim ; static void process_command_line(int, char **); static void process_data_file(); static void process_point_file(NcFile *nc_in, MetConfig &config, - VarInfo *, const Grid fr_grid, const Grid to_grid); + VarInfo *, const Grid to_grid); +#ifdef WITH_PYTHON +static void process_point_python(string python_command, MetConfig &config, + VarInfo *vinfo, const Grid to_grid, bool use_xarray); +#endif static void process_point_nccf_file(NcFile *nc_in, MetConfig &config, - VarInfo *, Met2dDataFile *fr_mtddf, const Grid to_grid); + VarInfo *, Met2dDataFile *fr_mtddf, + const Grid to_grid); static void open_nc(const Grid &grid, const ConcatString run_cs); static void write_nc(const DataPlane &dp, const Grid &grid, const VarInfo *vinfo, const char *vname); static void write_nc_int(const DataPlane &dp, const Grid &grid, - const VarInfo *vinfo, const char *vname); + const VarInfo *vinfo, const char *vname); static void close_nc(); static void usage(); static void set_field(const StringArray &); @@ -265,11 +276,26 @@ void process_command_line(int argc, char **argv) { OutputFilename = cline[2]; // Check if the input file - if ( !file_exists(InputFilename.c_str()) ) { - mlog << Error << "\n" << method_name - << "file does not exist \"" << InputFilename << "\"\n\n"; - exit(1); + bool use_python = false; +#ifdef WITH_PYTHON + string python_command = InputFilename; + bool use_xarray = (0 == python_command.find(conf_val_python_xarray)); + use_python = use_xarray || (0 == python_command.find(conf_val_python_numpy)); + if (use_python) { + int offset = python_command.find("="); + if (offset == std::string::npos) { + mlog << Error << "\n" << method_name + << "trouble parsing the python command " << python_command << ".\n\n"; + exit(1); + } } + else +#endif + if ( !file_exists(InputFilename.c_str()) ) { + mlog << Error << "\n" << method_name + << "file does not exist \"" << InputFilename << "\"\n\n"; + exit(1); + } // Check for at least one configuration string if(FieldSA.n() < 1) { @@ -347,18 +373,42 @@ void process_data_file() { // Open the input file mlog << Debug(1) << "Reading data file: " << InputFilename << "\n"; - nc_in = open_ncfile(InputFilename.c_str()); - - // Get the obs type before opening NetCDF - int obs_type = get_obs_type(nc_in); - bool goes_data = (obs_type == TYPE_GOES || obs_type == TYPE_GOES_ADP); - - if (obs_type == TYPE_NCCF) setenv(nc_att_met_point_nccf, "yes", 1); - - // Read the input data file + bool goes_data = false; + bool use_python = false; + int obs_type = TYPE_UNKNOWN; Met2dDataFileFactory m_factory; Met2dDataFile *fr_mtddf = (Met2dDataFile *) 0; - fr_mtddf = m_factory.new_met_2d_data_file(InputFilename.c_str(), ftype); +#ifdef WITH_PYTHON + string python_command = InputFilename; + MetPointData *met_point_obs = 0; + bool use_xarray = (0 == python_command.find(conf_val_python_xarray)); + use_python = use_xarray || (0 == python_command.find(conf_val_python_numpy)); + if (use_python) { + int offset = python_command.find("="); + if (offset == std::string::npos) { + mlog << Error << "\n" << method_name + << "trouble parsing the python command " << python_command << ".\n\n"; + exit(1); + } + + python_command = python_command.substr(offset+1); + obs_type = TYPE_PYTHON; + fr_mtddf = new MetNcMetDataFile(); + } + else +#endif + { + nc_in = open_ncfile(InputFilename.c_str()); + + // Get the obs type before opening NetCDF + obs_type = get_obs_type(nc_in); + goes_data = (obs_type == TYPE_GOES || obs_type == TYPE_GOES_ADP); + + if (obs_type == TYPE_NCCF) setenv(nc_att_met_point_nccf, "yes", 1); + + // Read the input data file + fr_mtddf = m_factory.new_met_2d_data_file(InputFilename.c_str(), ftype); + } if(!fr_mtddf) { mlog << Error << "\n" << method_name @@ -384,7 +434,11 @@ void process_data_file() { // For python types read the first field to set the grid // Determine the "from" grid +#ifdef WITH_PYTHON + if (!use_python) fr_grid = fr_mtddf->grid(); +#else fr_grid = fr_mtddf->grid(); +#endif // Determine the "to" grid to_grid = parse_vx_grid(RGInfo, &fr_grid, &fr_grid); @@ -410,12 +464,17 @@ void process_data_file() { process_goes_file(nc_in, config, vinfo, fr_grid, to_grid); } else if (TYPE_OBS == obs_type) { - process_point_file(nc_in, config, vinfo, fr_grid, to_grid); + process_point_file(nc_in, config, vinfo, to_grid); } else if (TYPE_NCCF == obs_type) { process_point_nccf_file(nc_in, config, vinfo, fr_mtddf, to_grid); unsetenv(nc_att_met_point_nccf); } +#ifdef WITH_PYTHON + else if (TYPE_PYTHON == obs_type) { + process_point_python(python_command, config, vinfo, to_grid, use_xarray); + } +#endif else { mlog << Error << "\n" << method_name << "Please check the input file. Only supports GOES, MET point obs, " @@ -599,8 +658,8 @@ IntArray prepare_qc_array(const IntArray qc_flags, StringArray qc_tables) { //////////////////////////////////////////////////////////////////////// -void process_point_file(NcFile *nc_in, MetConfig &config, VarInfo *vinfo, - const Grid fr_grid, const Grid to_grid) { +void process_point_met_data(MetPointData *met_point_obs, MetConfig &config, VarInfo *vinfo, + const Grid to_grid) { int nhdr, nobs; int nx, ny, var_count, to_count, var_count2; int idx, hdr_idx; @@ -616,8 +675,8 @@ void process_point_file(NcFile *nc_in, MetConfig &config, VarInfo *vinfo, bool has_prob_thresh = !prob_cat_thresh.check(bad_data_double); unixtime requested_valid_time, valid_time; - static const char *method_name = "process_point_file() -> "; - static const char *method_name_s = "process_point_file()"; + static const char *method_name = "process_point_met_data() -> "; + static const char *method_name_s = "process_point_met_data()"; // Check for at least one configuration string if(FieldSA.n() < 1) { @@ -626,35 +685,29 @@ void process_point_file(NcFile *nc_in, MetConfig &config, VarInfo *vinfo, usage(); } - MetNcPointObsIn nc_point_obs; - nc_point_obs.set_netcdf(nc_in, true); - // Read the dimensions and variables - nc_point_obs.read_dim_headers(); - nc_point_obs.check_nc(GET_NC_NAME_P(nc_in).c_str(), method_name_s); // exit if missing dims/vars - // Read all obs data to compute the cell mapping - nc_point_obs.read_obs_data(); - NcHeaderData header_data = nc_point_obs.get_header_data(); - NcPointObsData obs_data = nc_point_obs.get_point_obs_data(); +// MetNcPointObsIn nc_point_obs; + MetPointHeader *header_data = met_point_obs->get_header_data(); + MetPointObsData *obs_data = met_point_obs->get_point_obs_data(); - nhdr = nc_point_obs.get_hdr_cnt(); - nobs = nc_point_obs.get_obs_cnt(); + nhdr = met_point_obs->get_hdr_cnt(); + nobs = met_point_obs->get_obs_cnt(); bool empty_input = (nhdr == 0 && nobs == 0); - bool use_var_id = nc_point_obs.is_using_var_id(); + bool use_var_id = met_point_obs->is_using_var_id(); float *hdr_lats = new float[nhdr]; float *hdr_lons = new float[nhdr]; IntArray var_index_array; IntArray valid_time_array; - StringArray qc_tables = nc_point_obs.get_qty_data(); - StringArray var_names = nc_point_obs.get_var_names(); - StringArray hdr_valid_times = header_data.vld_array; + StringArray qc_tables = met_point_obs->get_qty_data(); + StringArray var_names = met_point_obs->get_var_names(); + StringArray hdr_valid_times = header_data->vld_array; hdr_valid_times.sort(); - nc_point_obs.get_lats(hdr_lats); - nc_point_obs.get_lons(hdr_lons); + met_point_obs->get_lats(hdr_lats); + met_point_obs->get_lons(hdr_lons); // Check the message types - prepare_message_types(header_data.typ_array); + prepare_message_types(header_data->typ_array); // Check and read obs_vid and obs_var if exists bool success_to_read = true; @@ -732,7 +785,7 @@ void process_point_file(NcFile *nc_in, MetConfig &config, VarInfo *vinfo, else { bool not_found_grib_code = true; for (idx=0; idxobs_ids[idx]) { not_found_grib_code = false; break; } @@ -757,10 +810,10 @@ void process_point_file(NcFile *nc_in, MetConfig &config, VarInfo *vinfo, log_msg << "GRIB codes: "; IntArray grib_codes; for (idx=0; idxobs_ids[idx])) { + grib_codes.add(obs_data->obs_ids[idx]); if (0 < idx) log_msg << ", "; - log_msg << obs_data.obs_ids[idx]; + log_msg << obs_data->obs_ids[idx]; } } } @@ -836,29 +889,29 @@ void process_point_file(NcFile *nc_in, MetConfig &config, VarInfo *vinfo, var_count = var_count2 = to_count = 0; filtered_by_time = filtered_by_msg_type = filtered_by_qc = 0; for (idx=0; idx < nobs; idx++) { - if (var_idx_or_gc == obs_data.obs_ids[idx]) { + if (var_idx_or_gc == obs_data->obs_ids[idx]) { var_count2++; - hdr_idx = obs_data.obs_hids[idx]; + hdr_idx = obs_data->obs_hids[idx]; if (0 < valid_time_array.n() && - !valid_time_array.has(header_data.vld_idx_array[hdr_idx])) { + !valid_time_array.has(header_data->vld_idx_array[hdr_idx])) { filtered_by_time++; continue; } - if(!keep_message_type(header_data.typ_idx_array[hdr_idx])) { + if(!keep_message_type(header_data->typ_idx_array[hdr_idx])) { filtered_by_msg_type++; continue; } // Filter by QC flag - if (has_qc_flags && !qc_idx_array.has(obs_data.obs_qids[idx])) { + if (has_qc_flags && !qc_idx_array.has(obs_data->obs_qids[idx])) { filtered_by_qc++; continue; } var_index_array.add(idx); var_count++; - if (is_eq(obs_data.obs_vals[idx], 0.)) obs_count_zero_from++; + if (is_eq(obs_data->obs_vals[idx], 0.)) obs_count_zero_from++; else obs_count_non_zero_from++; } } @@ -869,7 +922,7 @@ void process_point_file(NcFile *nc_in, MetConfig &config, VarInfo *vinfo, } cellMapping = new IntArray[nx * ny]; if( get_grid_mapping(to_grid, cellMapping, var_index_array, - obs_data.obs_hids, hdr_lats, hdr_lons) ) { + obs_data->obs_hids, hdr_lats, hdr_lons) ) { int from_index; IntArray cellArray; NumArray dataArray; @@ -905,7 +958,7 @@ void process_point_file(NcFile *nc_in, MetConfig &config, VarInfo *vinfo, dataArray.extend(cellArray.n()); for (int dIdx=0; dIdxget_obs_val(from_index); if (is_bad_data(data_value)) continue; if(mlog.verbosity_level() >= 4) { @@ -1073,13 +1126,108 @@ void process_point_file(NcFile *nc_in, MetConfig &config, VarInfo *vinfo, delete [] hdr_lats; delete [] hdr_lons; + + return; +} + +//////////////////////////////////////////////////////////////////////// + +void process_point_file(NcFile *nc_in, MetConfig &config, VarInfo *vinfo, + const Grid to_grid) { + int nhdr, nobs; + int nx, ny, var_count, to_count, var_count2; + int idx, hdr_idx; + int var_idx_or_gc; + int filtered_by_time, filtered_by_msg_type, filtered_by_qc; + ConcatString vname, vname_cnt, vname_mask; + DataPlane fr_dp, to_dp; + DataPlane cnt_dp, mask_dp; + DataPlane prob_dp, prob_mask_dp; + NcVar var_obs_gc, var_obs_var; + + clock_t start_clock = clock(); + bool has_prob_thresh = !prob_cat_thresh.check(bad_data_double); + + unixtime requested_valid_time, valid_time; + static const char *method_name = "process_point_file() -> "; + static const char *method_name_s = "process_point_file()"; + + // Check for at least one configuration string + if(FieldSA.n() < 1) { + mlog << Error << "\n" << method_name + << "The -field option must be used at least once!\n\n"; + usage(); + } + + MetNcPointObsIn nc_point_obs; + nc_point_obs.set_netcdf(nc_in, true); + // Read the dimensions and variables + nc_point_obs.read_dim_headers(); + nc_point_obs.check_nc(GET_NC_NAME_P(nc_in).c_str(), method_name_s); // exit if missing dims/vars + // Read all obs data to compute the cell mapping + nc_point_obs.read_obs_data(); + process_point_met_data(&nc_point_obs, config, vinfo, to_grid); + nc_point_obs.close(); mlog << Debug(LEVEL_FOR_PERFORMANCE) << method_name << "took " << (clock()-start_clock)/double(CLOCKS_PER_SEC) << " seconds\n"; +} + +//////////////////////////////////////////////////////////////////////// + +#ifdef WITH_PYTHON + +void process_point_python(string python_command, MetConfig &config, VarInfo *vinfo, + const Grid to_grid, bool use_xarray) { + int nhdr, nobs; + int nx, ny, var_count, to_count, var_count2; + int idx, hdr_idx; + int var_idx_or_gc; + int filtered_by_time, filtered_by_msg_type, filtered_by_qc; + ConcatString vname, vname_cnt, vname_mask; + DataPlane fr_dp, to_dp; + DataPlane cnt_dp, mask_dp; + DataPlane prob_dp, prob_mask_dp; + NcVar var_obs_gc, var_obs_var; + + clock_t start_clock = clock(); + bool has_prob_thresh = !prob_cat_thresh.check(bad_data_double); + + unixtime requested_valid_time, valid_time; + static const char *method_name = "process_point_python() -> "; + static const char *method_name_s = "process_point_python()"; + + // Check for at least one configuration string + if(FieldSA.n() < 1) { + mlog << Error << "\n" << method_name + << "The -field option must be used at least once!\n\n"; + usage(); + } + + MetPythonPointDataFile met_point_file; + if(!met_point_file.open(python_command.c_str(), use_xarray)) { + met_point_file.close(); + + mlog << Error << "\n" << method_name + << "trouble getting point observation file from python command " + << python_command << "\n\n"; + + exit(1); + } + + MetPointData *met_point_obs = met_point_file.get_met_point_data(); + process_point_met_data(met_point_obs, config, vinfo, to_grid); + + met_point_file.close(); + + mlog << Debug(LEVEL_FOR_PERFORMANCE) << method_name << "took " + << (clock()-start_clock)/double(CLOCKS_PER_SEC) << " seconds\n"; + return; } +#endif //////////////////////////////////////////////////////////////////////// @@ -1097,7 +1245,7 @@ void process_point_nccf_file(NcFile *nc_in, MetConfig &config, bool opt_all_attrs = false; Grid fr_grid = fr_mtddf->grid(); int from_size = fr_grid.nx() * fr_grid.ny(); - static const char *method_name = "process_point_file_with_latlon() -> "; + static const char *method_name = "process_point_nccf_file() -> "; NcVar var_lat = get_nc_var_lat(nc_in); NcVar var_lon = get_nc_var_lon(nc_in); From 315d3b433551948b99acb7e3b87eed85cef10237 Mon Sep 17 00:00:00 2001 From: Howard Soh Date: Tue, 28 Dec 2021 16:14:24 -0700 Subject: [PATCH 35/82] #1844 Initial release --- .../libcode/vx_pointdata_python/Makefile.am | 19 + .../pointdata_from_array.cc | 659 ++++++++++++++++++ .../pointdata_from_array.h | 59 ++ .../vx_pointdata_python/pointdata_python.cc | 290 ++++++++ .../vx_pointdata_python/pointdata_python.h | 112 +++ .../vx_pointdata_python/python_pointdata.cc | 588 ++++++++++++++++ .../vx_pointdata_python/python_pointdata.h | 80 +++ 7 files changed, 1807 insertions(+) create mode 100644 met/src/libcode/vx_pointdata_python/Makefile.am create mode 100644 met/src/libcode/vx_pointdata_python/pointdata_from_array.cc create mode 100644 met/src/libcode/vx_pointdata_python/pointdata_from_array.h create mode 100644 met/src/libcode/vx_pointdata_python/pointdata_python.cc create mode 100644 met/src/libcode/vx_pointdata_python/pointdata_python.h create mode 100644 met/src/libcode/vx_pointdata_python/python_pointdata.cc create mode 100644 met/src/libcode/vx_pointdata_python/python_pointdata.h diff --git a/met/src/libcode/vx_pointdata_python/Makefile.am b/met/src/libcode/vx_pointdata_python/Makefile.am new file mode 100644 index 0000000000..c48c07e2df --- /dev/null +++ b/met/src/libcode/vx_pointdata_python/Makefile.am @@ -0,0 +1,19 @@ +## @start 1 +## Makefile.am -- Process this file with automake to produce Makefile.in +## @end 1 + +MAINTAINERCLEANFILES = Makefile.in + +# Include the project definitions + +include ${top_srcdir}/Make-include + +# The library + +noinst_LIBRARIES = libvx_pointdata_python.a +libvx_pointdata_python_a_SOURCES = \ + pointdata_python.h pointdata_python.cc \ + pointdata_from_array.h pointdata_from_array.cc \ + python_pointdata.h python_pointdata.cc +libvx_pointdata_python_a_CPPFLAGS = ${MET_CPPFLAGS} -I../vx_python2_utils ${MET_PYTHON_CC} $(MET_PYTHON_LD) + diff --git a/met/src/libcode/vx_pointdata_python/pointdata_from_array.cc b/met/src/libcode/vx_pointdata_python/pointdata_from_array.cc new file mode 100644 index 0000000000..b3d6cd84fe --- /dev/null +++ b/met/src/libcode/vx_pointdata_python/pointdata_from_array.cc @@ -0,0 +1,659 @@ +// *=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=* +// ** Copyright UCAR (c) 1992 - 2021 +// ** University Corporation for Atmospheric Research (UCAR) +// ** National Center for Atmospheric Research (NCAR) +// ** Research Applications Lab (RAL) +// ** P.O.Box 3000, Boulder, Colorado, 80307-3000, USA +// *=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=* + +//////////////////////////////////////////////////////////////////////// + + +#include "string.h" + +#include "vx_python3_utils.h" +#include "vx_statistics.h" +#include "check_endian.h" + +#include "pointdata_from_array.h" + +//////////////////////////////////////////////////////////////////////// + +static const int api_delug_level = 11; + +//////////////////////////////////////////////////////////////////////// + + +template +void load_numpy (void * buf, + const int n, + const int data_endian, + void (*shuf)(void *), + float * out) +{ + +bool need_swap = (shuf != 0) && (native_endian != data_endian); + +int j, x, y, r, c; +T * u = (T *) buf; +T value; + +for (j=0; j +void load_numpy (void * buf, + const int n, + const int data_endian, + void (*shuf)(void *), + int * out) +{ + +const char *method_name = "load_numpy(int *) "; +bool need_swap = (shuf != 0) && (native_endian != data_endian); + +int j; +T * u = (T *) buf; +T value; + +for (j=0; j +void load_numpy_int (void * buf, + const int n, + const int data_endian, + void (*shuf)(void *), + IntArray *out) +{ + +const char *method_name = "load_numpy_int(IntArray *) "; +bool need_swap = (shuf != 0) && (native_endian != data_endian); + +int j; +T * u = (T *) buf; +T value; + +out->extend(n); + +for (j=0; jadd((int)value); + + mlog << Debug(api_delug_level) << method_name << " [" << j << "] value=" << value << "\n"; +} // for j + + + +return; + +} + + +//////////////////////////////////////////////////////////////////////// + + +template +void load_numpy_num (void * buf, + const int n, + const int data_endian, + void (*shuf)(void *), + NumArray *out) +{ + +const char *method_name = "load_numpy_num(NumArray *) "; +bool need_swap = (shuf != 0) && (native_endian != data_endian); + +int j; +T * u = (T *) buf; +T value; + +out->extend(n); + +for (j=0; jadd((float)value); + + mlog << Debug(api_delug_level) << method_name << "[" << j << "] value=" << value << "\n"; +} // for j + + + +return; + +} + + +//////////////////////////////////////////////////////////////////////// + + +bool pointdata_from_np_array(Python3_Numpy & np, float * data_out) +{ + +const char *method_name = "pointdata_from_np_array(float) -> "; + + // + // make sure it's a 1D array + // + +if ( np.n_dims() != 1 ) { + + mlog << Error << "\n" << method_name + << "data array is not 1-dimensional! ... " + << "(dim = " << (np.n_dims()) << ")\n\n"; + + exit ( 1 ); + + +} + +int n = np.dim(0); + + + // + // load the data + // + +const ConcatString dtype = np.dtype(); + + // 1 byte integers + + if ( dtype == "|i1" ) load_numpy (np.buffer(), n, little_endian, 0, data_out); +else if ( dtype == "|u1" ) load_numpy (np.buffer(), n, little_endian, 0, data_out); + + // 2 byte integers + +else if ( dtype == " (np.buffer(), n, little_endian, shuffle_2, data_out); +else if ( dtype == " (np.buffer(), n, little_endian, shuffle_2, data_out); + +else if ( dtype == ">i2" ) load_numpy (np.buffer(), n, big_endian, shuffle_2, data_out); +else if ( dtype == ">u2" ) load_numpy (np.buffer(), n, big_endian, shuffle_2, data_out); + + // 4 byte integers + +else if ( dtype == " (np.buffer(), n, little_endian, shuffle_4, data_out); +else if ( dtype == " (np.buffer(), n, little_endian, shuffle_4, data_out); + +else if ( dtype == ">i4" ) load_numpy (np.buffer(), n, big_endian, shuffle_4, data_out); +else if ( dtype == ">u4" ) load_numpy (np.buffer(), n, big_endian, shuffle_4, data_out); + + // 8 byte integers + +else if ( dtype == " (np.buffer(), n, little_endian, shuffle_8, data_out); +else if ( dtype == " (np.buffer(), n, little_endian, shuffle_8, data_out); + +else if ( dtype == ">i8" ) load_numpy (np.buffer(), n, big_endian, shuffle_8, data_out); +else if ( dtype == ">u8" ) load_numpy (np.buffer(), n, big_endian, shuffle_8, data_out); + + // single precision floats + +else if ( dtype == " (np.buffer(), n, little_endian, shuffle_4, data_out); +else if ( dtype == ">f4" ) load_numpy (np.buffer(), n, big_endian, shuffle_4, data_out); + + // double precision floats + +else if ( dtype == " (np.buffer(), n, little_endian, shuffle_8, data_out); +else if ( dtype == ">f8" ) load_numpy (np.buffer(), n, big_endian, shuffle_8, data_out); + + // + // nope ... the only other numerical data type for numpy arrays + // is single or double precision complex numbers, and + // we're not supporting those at this time + // + +else { + + mlog << Error << "\n" << method_name + << "unsupported data type \"" << dtype << "\"\n\n"; + + exit ( 1 ); + +} + + //////////////////// + + + // + // done + // + +return ( true ); + +} + + +//////////////////////////////////////////////////////////////////////// + + +bool pointdata_from_np_array(Python3_Numpy & np, int * data_out) +{ + +const char *method_name = "pointdata_from_np_array(int) -> "; + + // + // make sure it's a 1D array + // + +if ( np.n_dims() != 1 ) { + + mlog << Error << "\n" << method_name + << "data array is not 1-dimensional! ... " + << "(dim = " << (np.n_dims()) << ")\n\n"; + + exit ( 1 ); + + +} + +int n = np.dim(0); + + + // + // load the data + // + +const ConcatString dtype = np.dtype(); + + // 1 byte integers + + if ( dtype == "|i1" ) load_numpy (np.buffer(), n, little_endian, 0, data_out); +else if ( dtype == "|u1" ) load_numpy (np.buffer(), n, little_endian, 0, data_out); + + // 2 byte integers + +else if ( dtype == " (np.buffer(), n, little_endian, shuffle_2, data_out); +else if ( dtype == " (np.buffer(), n, little_endian, shuffle_2, data_out); + +else if ( dtype == ">i2" ) load_numpy (np.buffer(), n, big_endian, shuffle_2, data_out); +else if ( dtype == ">u2" ) load_numpy (np.buffer(), n, big_endian, shuffle_2, data_out); + + // 4 byte integers + +else if ( dtype == " (np.buffer(), n, little_endian, shuffle_4, data_out); +else if ( dtype == " (np.buffer(), n, little_endian, shuffle_4, data_out); + +else if ( dtype == ">i4" ) load_numpy (np.buffer(), n, big_endian, shuffle_4, data_out); +else if ( dtype == ">u4" ) load_numpy (np.buffer(), n, big_endian, shuffle_4, data_out); + + // 8 byte integers + +else if ( dtype == " (np.buffer(), n, little_endian, shuffle_8, data_out); +else if ( dtype == " (np.buffer(), n, little_endian, shuffle_8, data_out); + +else if ( dtype == ">i8" ) load_numpy (np.buffer(), n, big_endian, shuffle_8, data_out); +else if ( dtype == ">u8" ) load_numpy (np.buffer(), n, big_endian, shuffle_8, data_out); + + // single precision floats + +else if ( dtype == " (np.buffer(), n, little_endian, shuffle_4, data_out); +else if ( dtype == ">f4" ) load_numpy (np.buffer(), n, big_endian, shuffle_4, data_out); + + // double precision floats + +else if ( dtype == " (np.buffer(), n, little_endian, shuffle_8, data_out); +else if ( dtype == ">f8" ) load_numpy (np.buffer(), n, big_endian, shuffle_8, data_out); + + // + // nope ... the only other numerical data type for numpy arrays + // is single or double precision complex numbers, and + // we're not supporting those at this time + // + +else { + + mlog << Error << "\n" << method_name + << "unsupported data type \"" << dtype << "\"\n\n"; + + exit ( 1 ); + +} + + //////////////////// + + + // + // done + // + +return ( true ); + +} + + +//////////////////////////////////////////////////////////////////////// + + +bool pointdata_from_np_array(Python3_Numpy & np, IntArray *data_out) +{ + +const char *method_name = "pointdata_from_np_array(IntArray) -> "; + + // + // make sure it's a 1D array + // + +if ( np.n_dims() != 1 ) { + + mlog << Error << "\n" << method_name + << "data array is not 1-dimensional! ... " + << "(dim = " << (np.n_dims()) << ")\n\n"; + + exit ( 1 ); + + +} + +int n = np.dim(0); + + + // + // load the data + // + +IntArray data; +const ConcatString dtype = np.dtype(); + + // 1 byte integers + + if ( dtype == "|i1" ) load_numpy_int (np.buffer(), n, little_endian, 0, data_out); +else if ( dtype == "|u1" ) load_numpy_int (np.buffer(), n, little_endian, 0, data_out); + + // 2 byte integers + +else if ( dtype == " (np.buffer(), n, little_endian, shuffle_2, data_out); +else if ( dtype == " (np.buffer(), n, little_endian, shuffle_2, data_out); + +else if ( dtype == ">i2" ) load_numpy_int (np.buffer(), n, big_endian, shuffle_2, data_out); +else if ( dtype == ">u2" ) load_numpy_int (np.buffer(), n, big_endian, shuffle_2, data_out); + + // 4 byte integers + +else if ( dtype == " (np.buffer(), n, little_endian, shuffle_4, data_out); +else if ( dtype == " (np.buffer(), n, little_endian, shuffle_4, data_out); + +else if ( dtype == ">i4" ) load_numpy_int (np.buffer(), n, big_endian, shuffle_4, data_out); +else if ( dtype == ">u4" ) load_numpy_int (np.buffer(), n, big_endian, shuffle_4, data_out); + + // 8 byte integers + +else if ( dtype == " (np.buffer(), n, little_endian, shuffle_8, data_out); +else if ( dtype == " (np.buffer(), n, little_endian, shuffle_8, data_out); + +else if ( dtype == ">i8" ) load_numpy_int (np.buffer(), n, big_endian, shuffle_8, data_out); +else if ( dtype == ">u8" ) load_numpy_int (np.buffer(), n, big_endian, shuffle_8, data_out); + + // single precision floats + +else if ( dtype == " (np.buffer(), n, little_endian, shuffle_4, data_out); +else if ( dtype == ">f4" ) load_numpy_int (np.buffer(), n, big_endian, shuffle_4, data_out); + + // double precision floats + +else if ( dtype == " (np.buffer(), n, little_endian, shuffle_8, data_out); +else if ( dtype == ">f8" ) load_numpy_int (np.buffer(), n, big_endian, shuffle_8, data_out); + + // + // nope ... the only other numerical data type for numpy arrays + // is single or double precision complex numbers, and + // we're not supporting those at this time + // + +else { + + mlog << Error << "\n" << method_name + << "unsupported data type \"" << dtype << "\"\n\n"; + + exit ( 1 ); + +} + + //////////////////// + + + // + // done + // + +return ( true ); + +} + + +//////////////////////////////////////////////////////////////////////// + + +bool pointdata_from_np_array(Python3_Numpy & np, NumArray *data_out) +{ + +const char *method_name = "pointdata_from_np_array(NumArray) -> "; + + // + // make sure it's a 1D array + // + +if ( np.n_dims() != 1 ) { + + mlog << Error << "\n" << method_name + << "data array is not 1-dimensional! ... " + << "(dim = " << (np.n_dims()) << ")\n\n"; + + exit ( 1 ); + + +} + +int n = np.dim(0); + + + // + // load the data + // + +NumArray data; +const ConcatString dtype = np.dtype(); + + // 1 byte integers + + if ( dtype == "|i1" ) load_numpy_num (np.buffer(), n, little_endian, 0, data_out); +else if ( dtype == "|u1" ) load_numpy_num (np.buffer(), n, little_endian, 0, data_out); + + // 2 byte integers + +else if ( dtype == " (np.buffer(), n, little_endian, shuffle_2, data_out); +else if ( dtype == " (np.buffer(), n, little_endian, shuffle_2, data_out); + +else if ( dtype == ">i2" ) load_numpy_num (np.buffer(), n, big_endian, shuffle_2, data_out); +else if ( dtype == ">u2" ) load_numpy_num (np.buffer(), n, big_endian, shuffle_2, data_out); + + // 4 byte integers + +else if ( dtype == " (np.buffer(), n, little_endian, shuffle_4, data_out); +else if ( dtype == " (np.buffer(), n, little_endian, shuffle_4, data_out); + +else if ( dtype == ">i4" ) load_numpy_num (np.buffer(), n, big_endian, shuffle_4, data_out); +else if ( dtype == ">u4" ) load_numpy_num (np.buffer(), n, big_endian, shuffle_4, data_out); + + // 8 byte integers + +else if ( dtype == " (np.buffer(), n, little_endian, shuffle_8, data_out); +else if ( dtype == " (np.buffer(), n, little_endian, shuffle_8, data_out); + +else if ( dtype == ">i8" ) load_numpy_num (np.buffer(), n, big_endian, shuffle_8, data_out); +else if ( dtype == ">u8" ) load_numpy_num (np.buffer(), n, big_endian, shuffle_8, data_out); + + // single precision floats + +else if ( dtype == " (np.buffer(), n, little_endian, shuffle_4, data_out); +else if ( dtype == ">f4" ) load_numpy_num (np.buffer(), n, big_endian, shuffle_4, data_out); + + // double precision floats + +else if ( dtype == " (np.buffer(), n, little_endian, shuffle_8, data_out); +else if ( dtype == ">f8" ) load_numpy_num (np.buffer(), n, big_endian, shuffle_8, data_out); + + // + // nope ... the only other numerical data type for numpy arrays + // is single or double precision complex numbers, and + // we're not supporting those at this time + // + +else { + + mlog << Error << "\n" << method_name + << "unsupported data type \"" << dtype << "\"\n\n"; + + exit ( 1 ); + +} + + //////////////////// + + + // + // done + // + +return ( true ); + +} + + +//////////////////////////////////////////////////////////////////////// + + +bool pointdata_from_str_array(PyObject *data_array, StringArray *data_out) +{ + +//const char *method_name = "pointdata_from_str_array(StringArray) -> "; + +StringArray a = pyobject_as_string_array(data_array); +data_out->clear(); +data_out->add(a); + + //////////////////// + + + // + // done + // + +return ( true ); + +} + + +//////////////////////////////////////////////////////////////////////// + + // + // we just grab the numpy array and the attributes dictionary + // + // from the xarray DataArray object, and then hand them + // + // off to pointdata_from_numpy_array + // + +bool pointdata_from_xarray(PyObject * data_array, float *data_out) +{ + +Python3_Numpy np; +PyObject *numpy_array = PyObject_GetAttrString(data_array, data_attr_name); + + ///////////////////// + +np.set(numpy_array); + +bool status = pointdata_from_np_array(np, data_out); + + // + // done + // + +return ( status ); + +} + + +//////////////////////////////////////////////////////////////////////// + + // + // we just grab the numpy array and the attributes dictionary + // + // from the xarray DataArray object, and then hand them + // + // off to pointdata_from_numpy_array + // + +bool pointdata_from_xarray(PyObject * data_array, int *data_out) +{ + +Python3_Numpy np; +PyObject *numpy_array = PyObject_GetAttrString(data_array, data_attr_name); + + + ///////////////////// + + +np.set(numpy_array); + +bool status = pointdata_from_np_array(np, data_out); + + // + // done + // + +return ( status ); + +} + + +//////////////////////////////////////////////////////////////////////// diff --git a/met/src/libcode/vx_pointdata_python/pointdata_from_array.h b/met/src/libcode/vx_pointdata_python/pointdata_from_array.h new file mode 100644 index 0000000000..bec9380705 --- /dev/null +++ b/met/src/libcode/vx_pointdata_python/pointdata_from_array.h @@ -0,0 +1,59 @@ +// *=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=* +// ** Copyright UCAR (c) 1992 - 2021 +// ** University Corporation for Atmospheric Research (UCAR) +// ** National Center for Atmospheric Research (NCAR) +// ** Research Applications Lab (RAL) +// ** P.O.Box 3000, Boulder, Colorado, 80307-3000, USA +// *=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=* + +//////////////////////////////////////////////////////////////////////// + + +#ifndef __MET_POINTDATA_FROM_ARRAY_H__ +#define __MET_POINTDATA_FROM_ARRAY_H__ + + +//////////////////////////////////////////////////////////////////////// + + +#include "met_point_data.h" + +#include "python3_dict.h" +#include "python3_numpy.h" + + +extern "C" { + +#include "Python.h" + +} + + +//////////////////////////////////////////////////////////////////////// + +static const char data_attr_name [] = "values"; + +//////////////////////////////////////////////////////////////////////// + + +extern bool pointdata_from_np_array(Python3_Numpy & np, int * data_out); +extern bool pointdata_from_np_array(Python3_Numpy & np, float * data_out); +extern bool pointdata_from_np_array(Python3_Numpy & np, IntArray *data_out); +extern bool pointdata_from_np_array(Python3_Numpy & np, NumArray *data_out); +extern bool pointdata_from_str_array(PyObject *data_array, StringArray *data_out); + + +extern bool pointdata_from_xarray(PyObject *data_array, int *data_out); +extern bool pointdata_from_xarray(PyObject *data_array, float *data_out); +extern bool pointdata_from_xarray(PyObject *data_array, IntArray &data_out); +extern bool pointdata_from_xarray(PyObject *data_array, NumArray &data_out); +extern bool pointdata_from_xarray(PyObject *data_array, StringArray &data_out); + +//////////////////////////////////////////////////////////////////////// + + +#endif /* __MET_POINTDATA_FROM_NUMPY_ARRAY_H__ */ + + +//////////////////////////////////////////////////////////////////////// + diff --git a/met/src/libcode/vx_pointdata_python/pointdata_python.cc b/met/src/libcode/vx_pointdata_python/pointdata_python.cc new file mode 100644 index 0000000000..846966a6f1 --- /dev/null +++ b/met/src/libcode/vx_pointdata_python/pointdata_python.cc @@ -0,0 +1,290 @@ +// *=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=* +// ** Copyright UCAR (c) 1992 - 2021 +// ** University Corporation for Atmospheric Research (UCAR) +// ** National Center for Atmospheric Research (NCAR) +// ** Research Applications Lab (RAL) +// ** P.O.Box 3000, Boulder, Colorado, 80307-3000, USA +// *=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=* + + +//////////////////////////////////////////////////////////////////////// + + +using namespace std; + +#include +#include +#include +#include + +#include "pointdata_python.h" +#include "pointdata_from_array.h" +#include "vx_python3_utils.h" + +#include "vx_math.h" +#include "vx_log.h" + + +//////////////////////////////////////////////////////////////////////// + + + // + // Code for class MetPythonPointDataFile + // + + +//////////////////////////////////////////////////////////////////////// + + +MetPythonPointDataFile::MetPythonPointDataFile() + +{ + +python_init_from_scratch(); + +} + + +//////////////////////////////////////////////////////////////////////// + + +MetPythonPointDataFile::~MetPythonPointDataFile() + +{ + +close(); + +} + + +//////////////////////////////////////////////////////////////////////// + + +MetPythonPointDataFile::MetPythonPointDataFile(const MetPythonPointDataFile &) + +{ + +mlog << Error << "\nMetPythonPointDataFile::MetPythonPointDataFile(const MetPythonPointDataFile &) -> " + << "should never be called!\n\n"; + +exit ( 1 ); + +} + + +//////////////////////////////////////////////////////////////////////// + + +MetPythonPointDataFile & MetPythonPointDataFile::operator=(const MetPythonPointDataFile &) + +{ + +mlog << Error << "\nMetPythonPointDataFile::operator=(const MetPythonPointDataFile &) -> " + << "should never be called!\n\n"; + +exit ( 1 ); + +return ( * this ); + +} + + +//////////////////////////////////////////////////////////////////////// + + +void MetPythonPointDataFile::python_init_from_scratch() + +{ + +PythonCommand.clear(); +//met_data.clear(); + +close(); + +return; + +} + + +//////////////////////////////////////////////////////////////////////// + + +void MetPythonPointDataFile::close() + +{ + +met_data.clear(); + + // + // Don't reset the Type field + // Don't reset the PythonCommand + // + +return; + +} + + +//////////////////////////////////////////////////////////////////////// + +/* +void MetPythonPointDataFile::set_type(const GrdFileType t) + +{ + +Type = t; + +return; + +} +*/ + +//////////////////////////////////////////////////////////////////////// + + +bool MetPythonPointDataFile::open(const char * cur_command, bool use_xarray) + +{ + +close(); + +ConcatString full_path, file_name; +int i, file_argc; +char **file_argv = (char **) 0; // allocated +StringArray sa; +const char *method_name = "MetPythonPointDataFile::open() "; + + // + // Store the PythonCommand that is being run + // + +PythonCommand = cur_command; + + // + // parse and store argc and argv + // + +sa = PythonCommand.split(" "); + +file_argc = sa.n_elements(); + +if ( file_argc > 0 ) { + file_argv = new char * [ file_argc ]; + char a_var_name[512+1]; + + for ( i=0; idump(out, depth + 1); + +} else { + + out << prefix << "No Grid!\n"; + +} +*/ + // + // done + // + +out.flush(); + +return; + +} + + +//////////////////////////////////////////////////////////////////////// +/* + +bool MetPythonPointDataFile::data_ok(int x, int y) const + +{ + +//const double value = get(x, y); + +//return ( !is_bad_data(value) ); +return true; + +} + + +//////////////////////////////////////////////////////////////////////// + + +void MetPythonPointDataFile::data_minmax(double & data_min, double & data_max) const + +{ + +//Plane.data_range(data_min, data_max); + +return; + +} +*/ + +//////////////////////////////////////////////////////////////////////// + + +MetPointDataPython *MetPythonPointDataFile::get_met_point_data() + +{ + +return &met_data; + +} + + +//////////////////////////////////////////////////////////////////////// + diff --git a/met/src/libcode/vx_pointdata_python/pointdata_python.h b/met/src/libcode/vx_pointdata_python/pointdata_python.h new file mode 100644 index 0000000000..22b32c7f52 --- /dev/null +++ b/met/src/libcode/vx_pointdata_python/pointdata_python.h @@ -0,0 +1,112 @@ +// *=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=* +// ** Copyright UCAR (c) 1992 - 2021 +// ** University Corporation for Atmospheric Research (UCAR) +// ** National Center for Atmospheric Research (NCAR) +// ** Research Applications Lab (RAL) +// ** P.O.Box 3000, Boulder, Colorado, 80307-3000, USA +// *=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=* + + +//////////////////////////////////////////////////////////////////////// + + +#ifndef __MET_VX_POINTDATA_PYTHON_H__ +#define __MET_VX_POINTDATA_PYTHON_H__ + + +//////////////////////////////////////////////////////////////////////// + + +#include "data_plane.h" +#include "data_class.h" +#include "two_to_one.h" + +#include "python_pointdata.h" // the main access point +#include "pointdata_from_array.h" // takes an NumPy array or xarray DataArray + +#include "global_python.h" + + +//////////////////////////////////////////////////////////////////////// + + +//class MetPythonPointDataFile : public Met2dDataFile { +class MetPythonPointDataFile { + + private: + + void python_init_from_scratch(); + + MetPythonPointDataFile(const MetPythonPointDataFile &); + MetPythonPointDataFile & operator=(const MetPythonPointDataFile &); + + ConcatString PythonCommand; + + MetPointDataPython met_data; + + //VarInfoPython VInfo; + + //GrdFileType Type; // FileType_Python_Xarray or FileType_Python_Numpy + + public: + + MetPythonPointDataFile(); + ~MetPythonPointDataFile(); + + + // + // set stuff + // + + //void set_type(const GrdFileType); + + // + // get stuff + // + +// GrdFileType file_type() const; + +// double operator () (int x, int y) const; + + //double get (int x, int y) const; + +// bool data_ok (int x, int y) const; + +// void data_minmax (double & data_min, double & data_max) const; + + // + // do stuff + // + + bool open(const char * cur_command, bool use_xarray); + + void close(); + + + void dump (ostream &, int depth = 0) const; + + //MetPointData get_met_point_data(); + MetPointDataPython *get_met_point_data(); + + //int data_plane_array(VarInfo &, DataPlaneArray &); + + //bool met_point_data(MetPointData &); + +}; + + +//////////////////////////////////////////////////////////////////////// + + +//inline double MetPythonPointDataFile::operator () (int x, int y) const { return ( get(x, y) ); } +//inline GrdFileType MetPythonPointDataFile::file_type () const { return ( Type ); } + + +//////////////////////////////////////////////////////////////////////// + + +#endif /* __MET_VX_POINTDATA_PYTHON_H__ */ + + +//////////////////////////////////////////////////////////////////////// + diff --git a/met/src/libcode/vx_pointdata_python/python_pointdata.cc b/met/src/libcode/vx_pointdata_python/python_pointdata.cc new file mode 100644 index 0000000000..0712bbb13a --- /dev/null +++ b/met/src/libcode/vx_pointdata_python/python_pointdata.cc @@ -0,0 +1,588 @@ +// *=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=* +// ** Copyright UCAR (c) 1992 - 2021 +// ** University Corporation for Atmospheric Research (UCAR) +// ** National Center for Atmospheric Research (NCAR) +// ** Research Applications Lab (RAL) +// ** P.O.Box 3000, Boulder, Colorado, 80307-3000, USA +// *=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=* + +//////////////////////////////////////////////////////////////////////// + + +#include "vx_python3_utils.h" +#include "python_pointdata.h" +#include "pointdata_from_array.h" +#include "vx_util.h" + + +#include "global_python.h" +#include "wchar_argv.h" + +//////////////////////////////////////////////////////////////////////// + +//////////////////////////////////////////////////////////////////////// + + +static bool straight_python_point_data(const char * script_name, + int script_argc, char ** script_argv, + const bool use_xarray, MetPointDataPython &met_pd_out); + +//////////////////////////////////////////////////////////////////////// + +template +static void set_array_from_python(PyObject *python_data, const char *python_key, T *out, bool required=true) { + const char *method_name = "set_array_from_python(T *) -> "; + PyObject *numpy_array_obj = PyDict_GetItemString (python_data, python_key); + if (numpy_array_obj) { + Python3_Numpy np; + np.set(numpy_array_obj); + pointdata_from_np_array(np, out); + mlog << Debug(7) << method_name + << "get the point data for " << python_key << " from python object\n"; + } + else { + if (required) { + mlog << Error << "\n" << method_name + << "error getting the point data by the key (" << python_key << ") from python object\n\n"; + exit (1); + } + else mlog << Debug(3) << method_name + << "not exists the point data (" << python_key << ") from python object\n"; + } +} + +/* +static void set_array_from_python(PyObject *python_data, const char *python_key, int *out, bool required=true) { + const char *method_name = "set_array_from_python(T *) -> "; + PyObject *numpy_array_obj = PyDict_GetItemString (python_data, python_key); + if (numpy_array_obj) { + Python3_Numpy np; +mlog << Debug(7) << method_name + << "get the point data for " << python_key << " from python object - before\n"; + np.set(numpy_array_obj); +mlog << Debug(7) << method_name + << "get the point data for " << python_key << " from python object - after np.set\n"; + pointdata_from_np_array(np, out); + mlog << Debug(7) << method_name + << "get the point data for " << python_key << " from python object\n"; + } + else { + if (required) { + mlog << Error << "\n" << method_name + << "error getting the point data by the key (" << python_key << ") from python object\n\n"; + exit (1); + } + else mlog << Debug(3) << method_name + << "not exists the point data (" << python_key << ") from python object\n"; + } +} +*/ + +/* +static void set_array_from_python(PyObject *python_data, const char *python_key, int *out, bool required=true) { + const char *method_name = "set_array_from_python(int *) -> "; + PyObject *numpy_array_obj = PyDict_GetItemString (python_data, python_key); + if (numpy_array_obj) { + Python3_Numpy np; + np.set(numpy_array_obj); + pointdata_from_np_array(np, out); + } + else if (required) { + mlog << Error << "\nset_array_from_python(int *) -> " + << "error getting member (" << python_key << ") from python object\"\n\n"; + exit (1); + } + else mlog << Debug(3) << method_name + << "not exists the point data (" << python_key << ") from python object\n"; +} + +//////////////////////////////////////////////////////////////////////// + +static void set_array_from_python(PyObject *python_data, const char *python_key, float *out, bool required=true) { + const char *method_name = "set_array_from_python(float *) -> "; + PyObject *numpy_array_obj = PyDict_GetItemString (python_data, python_key); + if (numpy_array_obj) { + Python3_Numpy np; + np.set(numpy_array_obj); + pointdata_from_np_array(np, out); + mlog << Debug(7) << method_name + << "get the point data for " << python_key << " from python object\n"; + } + else if (required) { + mlog << Error << "\n" << method_name + << "error getting member (" << python_key << ") from python object\"\n\n"; + exit (1); + } + else mlog << Debug(3) << method_name + << "not exists the point data (" << python_key << ") from python object\n"; +} +*/ +//////////////////////////////////////////////////////////////////////// + +/* +static void set_met_array_from_python(PyObject *python_data, const char *python_key, IntArray *out, bool required=true) { + const char *method_name = "set_met_array_from_python(IntArray *) -> "; + PyObject *numpy_array_obj = PyDict_GetItemString (python_data, python_key); + if (numpy_array_obj) { + Python3_Numpy np; + np.set(numpy_array_obj); + pointdata_from_np_array(np, out); + mlog << Debug(7) << method_name + << "get the point data for " << python_key << " from python object\n"; + } + else if (required) { + mlog << Error << "\n" << method_name + << "error getting member (" << python_key << ") from python object\"\n\n"; + exit (1); + } + else mlog << Debug(3) << method_name + << "not exists the point data (" << python_key << ") from python object\n"; +} +*/ +/* +//////////////////////////////////////////////////////////////////////// + +static void set_met_array_from_python(PyObject *python_data, const char *python_key, NumArray &out) { + PyObject *numpy_array_obj = PyDict_GetItemString (python_data, python_key); + if (numpy_array_obj) { + Python3_Numpy np; + np.set(numpy_array_obj); + pointdata_from_np_array(np, out); + } + else { + mlog << Error << "\nset_array_from_python(NumArray) -> " + << "error getting member (" << python_key << ") from python object\"\n\n"; + exit (1); + } +} +*/ +//////////////////////////////////////////////////////////////////////// + +static void set_str_array_from_python(PyObject *python_data, const char *python_key, StringArray *out) { + const char *method_name = "set_met_array_from_python(StringArray *) -> "; + PyObject *str_array_obj = PyDict_GetItemString (python_data, python_key); + if (str_array_obj) { + pointdata_from_str_array(str_array_obj, out); + mlog << Debug(7) << method_name + << "get the point data for " << python_key << " from python object\n"; + } + else { + mlog << Error << "\n" << method_name + << "error getting member (" << python_key << ") from python object\"\n\n"; + exit (1); + } +} + + +//////////////////////////////////////////////////////////////////////// + + +bool python_point_data(const char * script_name, int script_argc, char ** script_argv, + const bool use_xarray, MetPointDataPython &met_pd_out) + +{ + +bool status = straight_python_point_data(script_name, script_argc, script_argv, + use_xarray, met_pd_out); + +return ( status ); + +} + +//////////////////////////////////////////////////////////////////////// + + +bool straight_python_point_data(const char * script_name, int script_argc, char ** script_argv, + const bool use_xarray, MetPointDataPython &met_pd_out) +{ + +int int_value; +PyObject * module_obj = 0; +PyObject * module_dict_obj = 0; +PyObject * python_key = 0; +PyObject * python_value = 0; +PyObject * numpy_array_obj = 0; +PyObject * python_met_point_data = 0; +ConcatString cs, user_dir, user_base; +const char *method_name = "straight_python_point_data -> "; +const char *method_name_s = "straight_python_point_data()"; + +mlog << Debug(3) << "Running user's python script (" + << script_name << ").\n"; + +cs = script_name; +user_dir = cs.dirname(); +user_base = cs.basename(); + +Wchar_Argv wa; + +wa.set(script_argc, script_argv); + + // + // if the global python object has already been initialized, + // we need to reload the module + // + +bool do_reload = GP.is_initialized; + +GP.initialize(); + + // + // start up the python interpreter + // + +if ( PyErr_Occurred() ) { + + PyErr_Print(); + + mlog << Warning << "\n" << method_name + << "an error occurred initializing python\n\n"; + + return ( false ); + +} + + // + // set the arguments + // + +run_python_string("import os"); +run_python_string("import sys"); + +ConcatString command; + +command << cs_erase + << "sys.path.append(\"" + << user_dir + << "\")"; + +run_python_string(command.text()); + +if ( script_argc > 0 ) { + + PySys_SetArgv (wa.wargc(), wa.wargv()); + +} + + // + // import the python script as a module + // + +module_obj = PyImport_ImportModule (user_base.c_str()); + + // + // if needed, reload the module + // + +if ( do_reload ) { + + module_obj = PyImport_ReloadModule (module_obj); + +} + +if ( PyErr_Occurred() ) { + + PyErr_Print(); + + mlog << Warning << "\n" << method_name + << "an error occurred importing module \"" + << script_name << "\"\n\n"; + + return ( false ); + +} + +if ( ! module_obj ) { + + mlog << Warning << "\n" << method_name + << "error running python script \"" + << script_name << "\"\n\n"; + + return ( false ); + +} + + // + // get the namespace for the module (as a dictionary) + // + +module_dict_obj = PyModule_GetDict (module_obj); + + // + // get handles to the objects of interest from the module_dict + // + +python_met_point_data = PyDict_GetItemString (module_dict_obj, python_key_point_data); + +python_value = PyDict_GetItemString (python_met_point_data, python_use_var_id); + +bool use_var_id = pyobject_as_bool(python_value); +met_pd_out.set_use_var_id(use_var_id); + + +python_value = PyDict_GetItemString (python_met_point_data, python_key_nhdr); + +int_value = pyobject_as_int(python_value); +met_pd_out.set_hdr_cnt(int_value); + +python_value = PyDict_GetItemString (python_met_point_data, python_key_nobs); +int_value = pyobject_as_int(python_value); + +met_pd_out.allocate(int_value); + +MetPointObsData *obs_data = met_pd_out.get_point_obs_data(); +MetPointHeader *header_data = met_pd_out.get_header_data(); + +if ( use_xarray ) { + + PyObject * data_array_obj = 0; + + // look up the data array variable name from the dictionary + + data_array_obj = PyDict_GetItemString (python_met_point_data, numpy_array_hdr_typ); + + if ( ! data_array_obj ) { + + mlog << Warning << "\n" << method_name + << "trouble reading data from \"" + << script_name << "\"\n\n"; + + return ( false ); + } + + //pointdata_from_xarray(data_array_obj, &header_data.hdr_typ); + +} else { // numpy array & dict + + // look up the data array variable name from the dictionary + +/* + set_met_array_from_python(python_met_point_data, numpy_array_hdr_typ, header_data->typ_idx_array); + set_met_array_from_python(python_met_point_data, numpy_array_hdr_sid, header_data->sid_idx_array); + set_met_array_from_python(python_met_point_data, numpy_array_hdr_vld, header_data->vld_idx_array); + //IntArray vld_num_array; // number array for valid time + + set_met_array_from_python(python_met_point_data, numpy_array_hdr_lat, header_data->lat_array); + set_met_array_from_python(python_met_point_data, numpy_array_hdr_lon, header_data->lon_array); + set_met_array_from_python(python_met_point_data, numpy_array_hdr_elv, header_data->elv_array); +*/ + set_array_from_python(python_met_point_data, numpy_array_hdr_typ, &header_data->typ_idx_array); + set_array_from_python(python_met_point_data, numpy_array_hdr_sid, &header_data->sid_idx_array); + set_array_from_python(python_met_point_data, numpy_array_hdr_vld, &header_data->vld_idx_array); + set_array_from_python(python_met_point_data, numpy_array_hdr_lat, &header_data->lat_array); + set_array_from_python(python_met_point_data, numpy_array_hdr_lon, &header_data->lon_array); + set_array_from_python(python_met_point_data, numpy_array_hdr_elv, &header_data->elv_array); + + set_str_array_from_python(python_met_point_data, numpy_array_hdr_typ_table, &header_data->typ_array); + set_str_array_from_python(python_met_point_data, numpy_array_hdr_sid_table, &header_data->sid_array); + set_str_array_from_python(python_met_point_data, numpy_array_hdr_vld_table, &header_data->vld_array); + set_array_from_python(python_met_point_data, numpy_array_prpt_typ_table, &header_data->prpt_typ_array, false); + set_array_from_python(python_met_point_data, numpy_array_irpt_typ_table, &header_data->irpt_typ_array, false); + set_array_from_python(python_met_point_data, numpy_array_inst_typ_table, &header_data->inst_typ_array, false); + + +// if ( !numpy_array_obj ) { +// +// mlog << Warning << "\n" << method_name +// << "trouble reading data from \"" +// << script_name << "\"\n\n"; +// +//// return ( false ); +// } +/* +*/ + set_array_from_python(python_met_point_data, numpy_array_obs_qty, obs_data->obs_qids); + set_array_from_python(python_met_point_data, numpy_array_obs_hid, obs_data->obs_hids); + set_array_from_python(python_met_point_data, numpy_array_obs_vid, obs_data->obs_ids); + set_array_from_python(python_met_point_data, numpy_array_obs_lvl, obs_data->obs_lvls); + set_array_from_python(python_met_point_data, numpy_array_obs_hgt, obs_data->obs_hgts); + set_array_from_python(python_met_point_data, numpy_array_obs_val, obs_data->obs_vals); + + set_str_array_from_python(python_met_point_data, numpy_array_obs_qty_table, &obs_data->qty_names); + set_str_array_from_python(python_met_point_data, numpy_array_obs_var_table, &obs_data->var_names); + + if(mlog.verbosity_level()>=point_data_debug_level) print_met_data(obs_data, header_data, method_name_s); + +} + + // + // done + // + +return ( true ); + +} + + +//////////////////////////////////////////////////////////////////////// + +void print_met_data(MetPointObsData *obs_data, MetPointHeader *header_data, + const char *caller, int debug_level) { + int log_count, count; + const int min_count = 20; + const char *method_name = "print_met_data() "; + + mlog << Debug(debug_level) << "\n" << method_name << "by " << caller << "\n" + << " obs_data.obs_cnt = " << obs_data->obs_cnt << "\n" + << " header_data.hdr_count = " << header_data->hdr_count << " type=" + << header_data->typ_idx_array.n() << ", sid=" + << header_data->sid_idx_array.n() << ", valid=" + << header_data->vld_idx_array.n() << ", lat=" + << header_data->lat_array.n() << ", lon=" + << header_data->lon_array.n() << ", elv=" + << header_data->elv_array.n() << ", message_type=" + << header_data->typ_array.n() << ", station_id=" + << header_data->sid_array.n() << ", valid_time=" + << header_data->vld_array.n() << ", prpt=" + << header_data->prpt_typ_array.n() << ", irpt=" + << header_data->irpt_typ_array.n() << ", inst=" + << header_data->inst_typ_array.n() << " &header_data=" << header_data + << "\n"; + + log_count = (header_data->hdr_count > min_count) ? min_count : header_data->hdr_count; + mlog << Debug(debug_level) << method_name + << "header_data: message_type,station_id,time_time,lat,lon.elv\n"; + for (int idx=0; idxsid_idx_array[idx] << ", " + << header_data->vld_idx_array[idx] << ", " + << header_data->lat_array[idx] << ", " + << header_data->lon_array[idx] << ", " + << header_data->elv_array[idx] << "\n"; + } + if (header_data->hdr_count > log_count) { + log_count = header_data->hdr_count - min_count; + if (log_count < min_count) log_count = min_count; + else mlog << Debug(debug_level) + << " header_data[...] = ...\n"; + for (int idx=log_count; idxhdr_count; idx++) { + mlog << Debug(debug_level) + << " header_data[" << idx << "] = " + << header_data->typ_idx_array[idx] << ", " + << header_data->sid_idx_array[idx] << ", " + << header_data->vld_idx_array[idx] << ", " + << header_data->lat_array[idx] << ", " + << header_data->lon_array[idx] << ", " + << header_data->elv_array[idx] << "\n"; + } + } + if (header_data->typ_array.n() > 0) { + mlog << Debug(debug_level) << "\n"; + count = header_data->typ_array.n(); + for (int idx=0; idxsid_array.n() > 0) { + mlog << Debug(debug_level) << "\n"; + count = header_data->sid_array.n(); + log_count = (count > min_count) ? min_count : count; + for (int idx=0; idx log_count) { + log_count = count - min_count; + if (log_count < min_count) log_count = min_count; + else mlog << Debug(debug_level) + << " station_id[...] = ...\n"; + for (int idx=log_count; idxvld_array.n() > 0) { + mlog << Debug(debug_level) << "\n"; + count = header_data->vld_array.n(); + log_count = (count > min_count) ? min_count : count; + for (int idx=0; idx log_count) { + log_count = count - min_count; + if (log_count < min_count) log_count = min_count; + else mlog << Debug(debug_level) + << " valid time[...] = ...\n"; + for (int idx=log_count; idxprpt_typ_array.n() > 0) { + mlog << Debug(debug_level) << "\n"; + for (int idx=0; idxprpt_typ_array.n(); idx++) + mlog << Debug(debug_level) + << " prpt_type[" << idx << "] = " << header_data->prpt_typ_array[idx] << "\n"; + } + if (header_data->irpt_typ_array.n() > 0) { + mlog << Debug(debug_level) << "\n"; + for (int idx=0; idxirpt_typ_array.n(); idx++) + mlog << Debug(debug_level) + << " irpt_type[" << idx << "] = " << header_data->irpt_typ_array[idx] << "\n"; + } + if (header_data->inst_typ_array.n() > 0) { + mlog << Debug(debug_level) << "\n"; + for (int idx=0; idxinst_typ_array.n(); idx++) + mlog << Debug(debug_level) + << " inst_type[" << idx << "] = " << header_data->inst_typ_array[idx] << "\n"; + } + + log_count = (obs_data->obs_cnt > min_count) ? min_count : obs_data->obs_cnt; + mlog << Debug(debug_level) << "\n" << method_name + << "obs_data: hid,vid.level,height,value,qty\n"; + for (int idx=0; idxobs_ids[idx] << ", " + << obs_data->obs_lvls[idx] << ", " + << obs_data->obs_hgts[idx] << ", " + << obs_data->obs_vals[idx] << ", " + << obs_data->obs_qids[idx] << "\n"; + } + if (obs_data->obs_cnt > log_count) { + log_count = obs_data->obs_cnt - min_count; + if (log_count < min_count) log_count = min_count; + else mlog << Debug(debug_level) + << " obs_data[...] = ...\n"; + for (int idx=log_count; idxobs_cnt; idx++) { + mlog << Debug(debug_level) + << " obs_data[" << idx << "] = " + << obs_data->obs_hids[idx] << ", " + << obs_data->obs_ids[idx] << ", " + << obs_data->obs_lvls[idx] << ", " + << obs_data->obs_hgts[idx] << ", " + << obs_data->obs_vals[idx] << ", " + << obs_data->obs_qids[idx] << "\n"; + } + } + + if (obs_data->var_names.n() > 0) { + mlog << Debug(debug_level) << "\n"; + for (int idx=0; idxvar_names.n(); idx++) + mlog << Debug(debug_level) + << " var_names[" << idx << "] = " << obs_data->var_names[idx] << "\n"; + } + else mlog << Debug(debug_level) + << " var_names is empty!!!\n"; + if (obs_data->qty_names.n() > 0) { + mlog << Debug(debug_level) << "\n"; + for (int idx=0; idxqty_names.n(); idx++) + mlog << Debug(debug_level) + << " qty_names[" << idx << "] = " << obs_data->qty_names[idx] << "\n"; + } + else mlog << Debug(debug_level) + << " qty_names is empty!!!\n"; + + mlog << Debug(debug_level) << "Done " << method_name << "by " << caller << "\n\n"; + +} + +//////////////////////////////////////////////////////////////////////// diff --git a/met/src/libcode/vx_pointdata_python/python_pointdata.h b/met/src/libcode/vx_pointdata_python/python_pointdata.h new file mode 100644 index 0000000000..5be19111f0 --- /dev/null +++ b/met/src/libcode/vx_pointdata_python/python_pointdata.h @@ -0,0 +1,80 @@ +// *=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=* +// ** Copyright UCAR (c) 1992 - 2021 +// ** University Corporation for Atmospheric Research (UCAR) +// ** National Center for Atmospheric Research (NCAR) +// ** Research Applications Lab (RAL) +// ** P.O.Box 3000, Boulder, Colorado, 80307-3000, USA +// *=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=* + +//////////////////////////////////////////////////////////////////////// + + +#ifndef __PYTHON_POINTDATA__ +#define __PYTHON_POINTDATA__ + + +//////////////////////////////////////////////////////////////////////// + + +#include "met_point_data.h" + + +extern "C" { + +#include "Python.h" + +} + + +//////////////////////////////////////////////////////////////////////// + +static const char python_key_point_data [] = "met_point_data"; + +static const char python_key_nhdr [] = "nhdr"; +//static const char python_key_npbhdr [] = "npbhdr"; +static const char python_use_var_id [] = "use_var_id"; +static const char numpy_array_hdr_typ [] = "hdr_typ"; // message type IDs +static const char numpy_array_hdr_sid [] = "hdr_sid"; // station IDs +static const char numpy_array_hdr_vld [] = "hdr_vld"; // valid time IDs +static const char numpy_array_hdr_lat [] = "hdr_lat"; +static const char numpy_array_hdr_lon [] = "hdr_lon"; +static const char numpy_array_hdr_elv [] = "hdr_elv"; +static const char numpy_array_hdr_typ_table [] = "hdr_typ_table"; // message type list +static const char numpy_array_hdr_sid_table [] = "hdr_sid_table"; // station ID list +static const char numpy_array_hdr_vld_table [] = "hdr_vld_table"; // valid time list +static const char numpy_array_prpt_typ_table[] = "prpt_typ_table"; +static const char numpy_array_irpt_typ_table[] = "irpt_typ_table"; +static const char numpy_array_inst_typ_table[] = "inst_typ_table"; + +static const char python_key_nobs [] = "nobs"; +static const char numpy_array_obs_qty [] = "obs_qty"; // quality_id +static const char numpy_array_obs_hid [] = "obs_hid"; // header id +static const char numpy_array_obs_vid [] = "obs_vid"; // variable id or grib code +static const char numpy_array_obs_lvl [] = "obs_lvl"; +static const char numpy_array_obs_hgt [] = "obs_hgt"; +static const char numpy_array_obs_val [] = "obs_val"; +static const char numpy_array_obs_qty_table [] = "obs_qty_table"; +static const char numpy_array_obs_var_table [] = "obs_var_table"; // variable names or grib codes as string + +static const int point_data_debug_level = 10; + + +//////////////////////////////////////////////////////////////////////// + + +extern bool python_point_data(const char * script_name, int script_argc, char ** script_argv, + const bool use_xarray, MetPointDataPython &met_pd_out); +//extern bool python_point_data(const char *python_command, const bool use_xarray, +// MetPointData & po_out); +extern void print_met_data(MetPointObsData *obs_data, MetPointHeader *header_data, + const char *caller, int debug_level=point_data_debug_level); + + +//////////////////////////////////////////////////////////////////////// + + +#endif /* __PYTHON_POINTDATA__ */ + + +//////////////////////////////////////////////////////////////////////// + From 95a65ed174f397d9a8005c8e84ee3b5a23b2eefd Mon Sep 17 00:00:00 2001 From: Howard Soh Date: Wed, 29 Dec 2021 14:07:18 -0700 Subject: [PATCH 36/82] #1844 Make buf_size const --- met/src/tools/core/ensemble_stat/ensemble_stat.cc | 4 ++-- met/src/tools/core/point_stat/point_stat.cc | 14 +++++++------- 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/met/src/tools/core/ensemble_stat/ensemble_stat.cc b/met/src/tools/core/ensemble_stat/ensemble_stat.cc index 422c8d6f84..f84e557b97 100644 --- a/met/src/tools/core/ensemble_stat/ensemble_stat.cc +++ b/met/src/tools/core/ensemble_stat/ensemble_stat.cc @@ -1055,7 +1055,7 @@ void process_point_obs(int i_nc) { << " observations from " << (hdr_count) << " header messages.\n"; - int buf_size = ((obs_count > DEF_NC_BUFFER_SIZE) ? DEF_NC_BUFFER_SIZE : (obs_count)); + const int buf_size = ((obs_count > DEF_NC_BUFFER_SIZE) ? DEF_NC_BUFFER_SIZE : (obs_count)); int obs_qty_idx_block[buf_size]; float obs_arr_block[buf_size][OBS_ARRAY_LEN]; @@ -1072,7 +1072,7 @@ void process_point_obs(int i_nc) { for(int i_start=0; i_start DEF_NC_BUFFER_SIZE) buf_size2 = DEF_NC_BUFFER_SIZE; + if (buf_size2 > buf_size) buf_size2 = buf_size; #ifdef WITH_PYTHON if (use_python) diff --git a/met/src/tools/core/point_stat/point_stat.cc b/met/src/tools/core/point_stat/point_stat.cc index 4578d91a33..f56a95bfe7 100644 --- a/met/src/tools/core/point_stat/point_stat.cc +++ b/met/src/tools/core/point_stat/point_stat.cc @@ -738,29 +738,29 @@ void process_obs_file(int i_nc) { StringArray obs_qty_array = met_point_obs->get_qty_data(); if( use_var_id ) var_names = met_point_obs->get_var_names(); - int buf_size = ((obs_count > BUFFER_SIZE) ? BUFFER_SIZE : (obs_count)); + const int buf_size = ((obs_count > BUFFER_SIZE) ? BUFFER_SIZE : (obs_count)); int obs_qty_idx_block[buf_size]; float obs_arr_block[buf_size][OBS_ARRAY_LEN]; // Process each observation in the file int str_length, block_size; - for(int i_block_start_idx=0; i_block_start_idx BUFFER_SIZE) block_size2 = BUFFER_SIZE; + for(int i_block_start_idx=0; i_block_start_idx buf_size) block_size = buf_size; #ifdef WITH_PYTHON if (use_python) status = met_point_obs->get_point_obs_data()->fill_obs_buf( - block_size2, i_block_start_idx, (float *)obs_arr_block, obs_qty_idx_block); + block_size, i_block_start_idx, (float *)obs_arr_block, obs_qty_idx_block); else #endif - status = nc_point_obs.read_obs_data(block_size2, i_block_start_idx, + status = nc_point_obs.read_obs_data(block_size, i_block_start_idx, (float *)obs_arr_block, obs_qty_idx_block, (char *)0); if (!status) exit(1); int hdr_idx; - for(int i_block_idx=0; i_block_idx Date: Wed, 5 Jan 2022 16:25:52 -0700 Subject: [PATCH 37/82] Add GitHub Actions workflow to trigger METplus testing workflow (#2002) --- .github/jobs/build_and_push_docker_image.sh | 40 +++++++++++++++++ .../build_docker_and_trigger_metplus.yml | 45 +++++++++++++++++++ met/.gitignore | 1 + 3 files changed, 86 insertions(+) create mode 100755 .github/jobs/build_and_push_docker_image.sh create mode 100644 .github/workflows/build_docker_and_trigger_metplus.yml diff --git a/.github/jobs/build_and_push_docker_image.sh b/.github/jobs/build_and_push_docker_image.sh new file mode 100755 index 0000000000..766b67cc41 --- /dev/null +++ b/.github/jobs/build_and_push_docker_image.sh @@ -0,0 +1,40 @@ +#! /bin/bash + +# utility function to run command get log the time it took to run +function time_command { + local start_seconds=$SECONDS + echo "RUNNING: $*" + "$@" + local error=$? + + local duration=$(( SECONDS - start_seconds )) + echo "TIMING: Command took `printf '%02d' $(($duration / 60))`:`printf '%02d' $(($duration % 60))` (MM:SS): '$*'" + if [ ${error} -ne 0 ]; then + echo "ERROR: '$*' exited with status = ${error}" + fi + return $error +} + +prefix=refs/heads/ +branch_name=${GITHUB_REF#"$prefix"} +DOCKERHUB_TAG=dtcenter/met:${branch_name} + +DOCKERFILE_DIR=${GITHUB_WORKSPACE}/scripts/docker + +echo "::group::Docker Build Command" +time_command docker build -t ${DOCKERHUB_TAG} \ + --build-arg SOURCE_BRANCH=$branch_name \ + $DOCKERFILE_DIR +echo "::endgroup::" + +# skip docker push if credentials are not set +if [ -z ${DOCKER_USERNAME+x} ] || [ -z ${DOCKER_PASSWORD+x} ]; then + echo "DockerHub credentials not set. Skipping docker push" + exit 0 +fi + +echo "$DOCKER_PASSWORD" | docker login --username "$DOCKER_USERNAME" --password-stdin + +echo "::group::Docker Push Command" +time_command docker push ${DOCKERHUB_TAG} +echo "::endgroup::" diff --git a/.github/workflows/build_docker_and_trigger_metplus.yml b/.github/workflows/build_docker_and_trigger_metplus.yml new file mode 100644 index 0000000000..76a7dadb80 --- /dev/null +++ b/.github/workflows/build_docker_and_trigger_metplus.yml @@ -0,0 +1,45 @@ +name: Build Docker Image and Trigger METplus Workflow + +on: + push: + branches: + - develop + paths-ignore: + - 'met/docs/**' + +jobs: + build_met_docker: + name: Handle Docker Image + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v2 + - name: Build and Push Docker Image + run: .github/jobs/build_and_push_docker_image.sh + env: + DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }} + DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }} + trigger_metplus: + name: Trigger METplus testing workflow + runs-on: ubuntu-latest + needs: build_met_docker + steps: + - name: Print GitHub values for reference + env: + GITHUB_CONTEXT: ${{ toJson(github) }} + run: echo "$GITHUB_CONTEXT" + - uses: actions/github-script@v5 + with: + github-token: ${{ secrets.METPLUS_BOT_TOKEN }} + script: | + await github.rest.actions.createWorkflowDispatch({ + owner: 'dtcenter', + repo: 'METplus', + workflow_id: 'testing.yml', + ref: 'develop', + inputs: { + repo_name: '${{ github.repository }}', + docker_tag: '${{ github.ref }}', + actor: '${{ github.actor }}', + commit: '${{ github.sha }}' + }, + }); diff --git a/met/.gitignore b/met/.gitignore index 28829cf0bb..1afea10016 100644 --- a/met/.gitignore +++ b/met/.gitignore @@ -17,6 +17,7 @@ bin share Makefile Makefile.in +make.log # tilda files generated by emacs *~ From 1c981f4276dcccbf614177e02326d549b742bcea Mon Sep 17 00:00:00 2001 From: George McCabe <23407799+georgemccabe@users.noreply.github.com> Date: Wed, 5 Jan 2022 16:54:38 -0700 Subject: [PATCH 38/82] changed names of inputs to send to METplus to match changes to METplus repo --- .github/workflows/build_docker_and_trigger_metplus.yml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/build_docker_and_trigger_metplus.yml b/.github/workflows/build_docker_and_trigger_metplus.yml index 76a7dadb80..02b6bca831 100644 --- a/.github/workflows/build_docker_and_trigger_metplus.yml +++ b/.github/workflows/build_docker_and_trigger_metplus.yml @@ -37,9 +37,9 @@ jobs: workflow_id: 'testing.yml', ref: 'develop', inputs: { - repo_name: '${{ github.repository }}', - docker_tag: '${{ github.ref }}', + repository: '${{ github.repository }}', + ref: '${{ github.ref }}', actor: '${{ github.actor }}', - commit: '${{ github.sha }}' + sha: '${{ github.sha }}' }, }); From 0273cf162c3ef63e2b4989382f69a4ab1be0c268 Mon Sep 17 00:00:00 2001 From: Howard Soh Date: Fri, 7 Jan 2022 10:44:38 -0700 Subject: [PATCH 39/82] #1965 Excludes the duplicated variable names from multiple input files --- met/src/tools/other/ioda2nc/ioda2nc.cc | 72 +++++++++++++++----------- 1 file changed, 41 insertions(+), 31 deletions(-) diff --git a/met/src/tools/other/ioda2nc/ioda2nc.cc b/met/src/tools/other/ioda2nc/ioda2nc.cc index 772af80a72..79d96a6434 100644 --- a/met/src/tools/other/ioda2nc/ioda2nc.cc +++ b/met/src/tools/other/ioda2nc/ioda2nc.cc @@ -369,7 +369,8 @@ void process_ioda_file(int i_pb) { IntArray diff_file_times; int diff_file_time_count; StringArray variables_big_nlevels; - static const char *method_name = "process_ioda_file() "; + static const char *method_name = "process_ioda_file() ->"; + static const char *method_name_s = "process_ioda_file() "; bool apply_grid_mask = (conf_info.grid_mask.nx() > 0 && conf_info.grid_mask.ny() > 0); @@ -388,7 +389,7 @@ void process_ioda_file(int i_pb) { // Check for a valid file if(IS_INVALID_NC_P(f_in)) { - mlog << Error << "\n" << method_name << "-> " + mlog << Error << "\n" << method_name << "can't open input NetCDF file \"" << ioda_files[i_pb] << "\" for reading.\n\n"; delete f_in; @@ -444,7 +445,7 @@ void process_ioda_file(int i_pb) { dim_names, metadata_vars); if(!is_netcdf_ready) { - mlog << Error << "\n" << method_name << "-> " + mlog << Error << "\n" << method_name << "Please check the IODA file (required dimensions or meta variables are missing).\n\n"; delete f_in; f_in = (NcFile *) 0; @@ -460,7 +461,7 @@ void process_ioda_file(int i_pb) { if(dim_names.has("nrecs")) nrecs = get_dim_value(f_in, "nrecs", false); else { nrecs = nvars * nlocs; - mlog << Debug(3) << "\n" << method_name << "-> " + mlog << Debug(3) << "\n" << method_name << "nrecs dimension does not exist, so computed\n"; } NcVar in_hdr_vld_var = get_var(f_in, "datetime@MetaData"); @@ -472,10 +473,10 @@ void process_ioda_file(int i_pb) { else { NcDim datetime_dim = get_nc_dim(&in_hdr_vld_var, 1); ndatetime = IS_VALID_NC(datetime_dim) ? get_dim_size(&datetime_dim) : nstring; - mlog << Debug(3) << "\n" << method_name << "-> " + mlog << Debug(3) << "\n" << method_name << "ndatetime dimension does not exist!\n"; } - mlog << Debug(5) << method_name << "-> dimensions: nvars=" << nvars << ", nlocs=" << nlocs + mlog << Debug(5) << method_name << "dimensions: nvars=" << nvars << ", nlocs=" << nlocs << ", nrecs=" << nrecs << ", nstring=" << nstring << ", ndatetime=" << ndatetime << "\n"; npbmsg_total = npbmsg = nlocs; @@ -515,17 +516,17 @@ void process_ioda_file(int i_pb) { } if(!get_nc_data(&in_hdr_lat_var, hdr_lat_arr, nlocs)) { - mlog << Error << "\n" << method_name << " -> " + mlog << Error << "\n" << method_name << "trouble getting latitude\n\n"; exit(1); } if(!get_nc_data(&in_hdr_lon_var, hdr_lon_arr, nlocs)) { - mlog << Error << "\n" << method_name << " -> " + mlog << Error << "\n" << method_name << "trouble getting longitude\n\n"; exit(1); } if(!get_nc_data(&in_hdr_vld_var, hdr_vld_block, lengths, offsets)) { - mlog << Error << "\n" << method_name << " -> " + mlog << Error << "\n" << method_name << "trouble getting datetime\n\n"; exit(1); } @@ -533,7 +534,6 @@ void process_ioda_file(int i_pb) { StringArray raw_var_names; if(do_all_vars || obs_var_names.n() == 0) raw_var_names = obs_value_vars; else raw_var_names = obs_var_names; - if(obs_var_names.n() > 0) obs_var_names.clear(); NcVar obs_var, qc_var; ConcatString unit_attr; @@ -543,8 +543,8 @@ void process_ioda_file(int i_pb) { int *qc_data = new int[nlocs]; float *obs_data = new float[nlocs]; ConcatString obs_var_name = raw_var_names[idx] + "@ObsValue"; - - mlog << Debug(7) << method_name << " -> " + + mlog << Debug(7) << method_name << "processing \"" << obs_var_name << "\" variable!\n"; obs_var = get_var(f_in, obs_var_name.c_str()); v_qc_data.push_back(qc_data); @@ -556,13 +556,16 @@ void process_ioda_file(int i_pb) { get_var_units(&obs_var, unit_attr); get_att_value_string(&obs_var, "long_name", desc_attr); } - obs_var_units.add(unit_attr); - obs_var_descs.add(desc_attr); - + // Replace the input variable name to the output variable name ConcatString new_name = name_map[raw_var_names[idx]]; - if (0 < new_name.length()) obs_var_names.add(new_name); - else obs_var_names.add(raw_var_names[idx]); + if (0 >= new_name.length()) new_name = raw_var_names[idx]; + // Filter out the same variable names from multiple input files + if (!obs_var_names.has(new_name)) { + obs_var_names.add(new_name); + obs_var_units.add(unit_attr); + obs_var_descs.add(desc_attr); + } } // Initialize counts @@ -573,7 +576,7 @@ void process_ioda_file(int i_pb) { bool showed_progress = false; if(mlog.verbosity_level() >= debug_level_for_performance) { end_t = clock(); - mlog << Debug(debug_level_for_performance) << " PERF: " << method_name << " " + mlog << Debug(debug_level_for_performance) << " PERF: " << method_name_s << " " << (end_t-start_t)/double(CLOCKS_PER_SEC) << " seconds for preparing\n"; start_t = clock(); @@ -612,7 +615,7 @@ void process_ioda_file(int i_pb) { char valid_time[ndatetime+1]; m_strncpy(valid_time, (const char *)(hdr_vld_block + (i_read * ndatetime)), - ndatetime, method_name, "valid_time", true); + ndatetime, method_name_s, "valid_time", true); valid_time[ndatetime] = 0; msg_ut = yyyymmddThhmmss_to_unix(valid_time); @@ -662,7 +665,7 @@ void process_ioda_file(int i_pb) { if(has_msg_type) { int buf_len = sizeof(modified_hdr_typ); - m_strncpy(hdr_typ, hdr_msg_types+(i_read*nstring), nstring, method_name, "hdr_typ"); + m_strncpy(hdr_typ, hdr_msg_types+(i_read*nstring), nstring, method_name_s, "hdr_typ"); m_rstrip(hdr_typ, nstring); // If the message type is not listed in the configuration @@ -675,22 +678,22 @@ void process_ioda_file(int i_pb) { if(0 < message_type_map.count((string)hdr_typ)) { ConcatString mappedMessageType = message_type_map[(string)hdr_typ]; - mlog << Debug(6) << "\n" << method_name << " -> " + mlog << Debug(6) << "\n" << method_name << "Switching report type \"" << hdr_typ << "\" to message type \"" << mappedMessageType << "\".\n"; if(mappedMessageType.length() < HEADER_STR_LEN) buf_len = HEADER_STR_LEN; m_strncpy(modified_hdr_typ, mappedMessageType.c_str(), buf_len, - method_name, "modified_hdr_typ"); + method_name_s, "modified_hdr_typ"); } else { - m_strncpy(modified_hdr_typ, hdr_typ, buf_len, method_name, "modified_hdr_typ2"); + m_strncpy(modified_hdr_typ, hdr_typ, buf_len, method_name_s, "modified_hdr_typ2"); } modified_hdr_typ[buf_len-1] = 0; } if(has_station_id) { char tmp_sid[nstring+1]; - m_strncpy(tmp_sid, hdr_station_ids+(i_read*nstring), nstring, method_name, "tmp_sid"); + m_strncpy(tmp_sid, hdr_station_ids+(i_read*nstring), nstring, method_name_s, "tmp_sid"); m_rstrip(tmp_sid, nstring); hdr_sid = tmp_sid; } @@ -775,7 +778,16 @@ void process_ioda_file(int i_pb) { n_hdr_obs = 0; for(idx=0; idx= out_name.length()) out_name = raw_var_names[idx]; + if (!obs_var_names.has(out_name, var_idx)) { + mlog << Warning << "\n" << method_name + << "Skip the variable " << out_name << " (" << raw_var_names[idx] + << ") at " << ioda_files[i_pb] << "\n\n"; + continue; + } + obs_arr[1] = var_idx; obs_arr[2] = obs_pres_arr[i_read]; obs_arr[3] = obs_hght_arr[i_read]; obs_arr[4] = v_obs_data[idx][i_read]; @@ -795,8 +807,6 @@ void process_ioda_file(int i_pb) { // store the header data and increment the IODA record // counter if(n_hdr_obs > 0) { - //nc_point_obs.add_header(modified_hdr_typ, hdr_sid.c_str(), hdr_vld_ut, - // hdr_lat, hdr_lon, hdr_elv); i_msg++; } else { @@ -839,12 +849,12 @@ void process_ioda_file(int i_pb) { << n_file_obs << "\n"; if(npbmsg == rej_vld && 0 < rej_vld) { - mlog << Warning << "\n" << method_name << " -> " + mlog << Warning << "\n" << method_name << "All records were filtered out by valid time.\n" << "\tPlease adjust time range with \"-valid_beg\" and \"-valid_end\".\n" << "\tmin/max obs time from IODA file: " << min_time_str << " and " << max_time_str << ".\n" - << "\ttime range: " << start_time_str << " and " << end_time_str << ".\n"; + << "\ttime range: " << start_time_str << " and " << end_time_str << ".\n\n"; } else { mlog << Debug(1) << "Obs time between " << unix_to_yyyymmdd_hhmmss(min_msg_ut) @@ -882,13 +892,13 @@ void process_ioda_file(int i_pb) { if(mlog.verbosity_level() >= debug_level_for_performance) { method_end = clock(); - cout << " PERF: " << method_name << " " + cout << " PERF: " << method_name_s << (method_end-method_start)/double(CLOCKS_PER_SEC) << " seconds\n"; } if(i_msg <= 0) { - mlog << Warning << "\n" << method_name << " -> " + mlog << Warning << "\n" << method_name << "No IODA records retained from file: " << ioda_files[i_pb] << "\n\n"; } From ab1199b8eecec4fe14b0226e8d00ea8744c86bfb Mon Sep 17 00:00:00 2001 From: Howard Soh Date: Tue, 11 Jan 2022 12:18:14 -0700 Subject: [PATCH 40/82] #1965 Processed with the raw variable names and update them when the output is saved. --- met/src/tools/other/ioda2nc/ioda2nc.cc | 27 ++++++++++++-------------- 1 file changed, 12 insertions(+), 15 deletions(-) diff --git a/met/src/tools/other/ioda2nc/ioda2nc.cc b/met/src/tools/other/ioda2nc/ioda2nc.cc index 79d96a6434..9c7df80b1a 100644 --- a/met/src/tools/other/ioda2nc/ioda2nc.cc +++ b/met/src/tools/other/ioda2nc/ioda2nc.cc @@ -538,7 +538,6 @@ void process_ioda_file(int i_pb) { NcVar obs_var, qc_var; ConcatString unit_attr; ConcatString desc_attr; - map name_map = conf_info.getObsVarMap(); for(idx=0; idx= new_name.length()) new_name = raw_var_names[idx]; + ConcatString raw_name = raw_var_names[idx]; // Filter out the same variable names from multiple input files - if (!obs_var_names.has(new_name)) { - obs_var_names.add(new_name); + if (!obs_var_names.has(raw_name)) { + obs_var_names.add(raw_name); obs_var_units.add(unit_attr); obs_var_descs.add(desc_attr); } @@ -779,12 +777,10 @@ void process_ioda_file(int i_pb) { n_hdr_obs = 0; for(idx=0; idx= out_name.length()) out_name = raw_var_names[idx]; - if (!obs_var_names.has(out_name, var_idx)) { + if (!obs_var_names.has(raw_var_names[idx], var_idx)) { mlog << Warning << "\n" << method_name - << "Skip the variable " << out_name << " (" << raw_var_names[idx] - << ") at " << ioda_files[i_pb] << "\n\n"; + << "Skip the variable " << raw_var_names[idx] + << " at " << ioda_files[i_pb] << "\n\n"; continue; } obs_arr[1] = var_idx; @@ -930,13 +926,14 @@ void write_netcdf_hdr_data() { StringArray nc_var_name_arr; StringArray nc_var_unit_arr; StringArray nc_var_desc_arr; - const long var_count = obs_var_names.n(); - const long units_count = obs_var_units.n(); + map name_map = conf_info.getObsVarMap(); - for(int i=0; i= new_name.length()) new_name = obs_var_names[i]; + nc_var_name_arr.add(new_name); } - for(int i=0; i Date: Tue, 11 Jan 2022 12:19:50 -0700 Subject: [PATCH 41/82] #1965 Added unit test ioda2nc_same_input --- test/xml/unit_ioda2nc.xml | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/test/xml/unit_ioda2nc.xml b/test/xml/unit_ioda2nc.xml index 0006da34d8..a44463c6e1 100644 --- a/test/xml/unit_ioda2nc.xml +++ b/test/xml/unit_ioda2nc.xml @@ -100,4 +100,24 @@ + + &MET_BIN;/ioda2nc + + STATION_ID "KEKA" + MASK_GRID + MASK_POLY + MESSAGE_TYPE + + \ + &DATA_DIR_OBS;/ioda/ioda.NC001007.2020031012.nc \ + &OUTPUT_DIR;/ioda2nc/ioda.NC001007.2020031012.same_input.nc \ + -config &CONFIG_DIR;/IODA2NCConfig_mask \ + -iodafile &DATA_DIR_OBS;/ioda/ioda.NC001007.2020031012.nc \ + -v 7 + + + &OUTPUT_DIR;/ioda2nc/ioda.NC001007.2020031012.same_input.nc + + + From e48c5ff4ade259a591ebdd535a0adad3bc1170a0 Mon Sep 17 00:00:00 2001 From: Howard Soh Date: Tue, 11 Jan 2022 12:26:53 -0700 Subject: [PATCH 42/82] #1965 Processed with the raw variable names and update them when the output is saved. --- met/src/tools/other/ioda2nc/ioda2nc.cc | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/met/src/tools/other/ioda2nc/ioda2nc.cc b/met/src/tools/other/ioda2nc/ioda2nc.cc index 9c7df80b1a..1d27aa5ec1 100644 --- a/met/src/tools/other/ioda2nc/ioda2nc.cc +++ b/met/src/tools/other/ioda2nc/ioda2nc.cc @@ -926,14 +926,16 @@ void write_netcdf_hdr_data() { StringArray nc_var_name_arr; StringArray nc_var_unit_arr; StringArray nc_var_desc_arr; + const long var_count = obs_var_names.n(); + const long units_count = obs_var_units.n(); map name_map = conf_info.getObsVarMap(); - for(int i=0; i= new_name.length()) new_name = obs_var_names[i]; nc_var_name_arr.add(new_name); } - for(int i=0; i Date: Tue, 11 Jan 2022 12:28:20 -0700 Subject: [PATCH 43/82] #1965 Added unit test ioda2nc_same_input --- test/xml/unit_ioda2nc.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/xml/unit_ioda2nc.xml b/test/xml/unit_ioda2nc.xml index a44463c6e1..b519a08d4d 100644 --- a/test/xml/unit_ioda2nc.xml +++ b/test/xml/unit_ioda2nc.xml @@ -113,7 +113,7 @@ &OUTPUT_DIR;/ioda2nc/ioda.NC001007.2020031012.same_input.nc \ -config &CONFIG_DIR;/IODA2NCConfig_mask \ -iodafile &DATA_DIR_OBS;/ioda/ioda.NC001007.2020031012.nc \ - -v 7 + -v 2 &OUTPUT_DIR;/ioda2nc/ioda.NC001007.2020031012.same_input.nc From ee761439084ed3a23133f0cc9c7ffb442376dafa Mon Sep 17 00:00:00 2001 From: hsoh-u Date: Tue, 11 Jan 2022 18:02:40 -0700 Subject: [PATCH 44/82] #1965 initialize var_idx --- met/src/tools/other/ioda2nc/ioda2nc.cc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/met/src/tools/other/ioda2nc/ioda2nc.cc b/met/src/tools/other/ioda2nc/ioda2nc.cc index 1d27aa5ec1..323cd83447 100644 --- a/met/src/tools/other/ioda2nc/ioda2nc.cc +++ b/met/src/tools/other/ioda2nc/ioda2nc.cc @@ -776,7 +776,7 @@ void process_ioda_file(int i_pb) { n_hdr_obs = 0; for(idx=0; idx Date: Wed, 12 Jan 2022 10:05:21 -0700 Subject: [PATCH 45/82] Update the development environment for seneca to add /usr/local/nco/bin and /usr/local/netcdf/bin to the path. Also define MET_TEST_RSCRIPT to point to a new enough version of RSCRIPT that has the ncdf4 package 1.17 or later. --- scripts/environment/development.seneca | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/scripts/environment/development.seneca b/scripts/environment/development.seneca index d6e13f0543..325e80ed3b 100644 --- a/scripts/environment/development.seneca +++ b/scripts/environment/development.seneca @@ -40,7 +40,13 @@ export LDFLAGS="${LDFLAGS} -L${MET_JASPER}/lib" export MET_TEST_INPUT=${MET_PROJ_DIR}/MET_test_data/unit_test export MET_FONT_DIR=${MET_TEST_INPUT}/fonts +# Define Rscript to use a version with the ncdf4 package 1.17 or later +export MET_TEST_RSCRIPT=/usr/local/R-4.1.2/bin/Rscript + # This is a cron script -- create the shell environment for this job -export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:\ - /usr/bin/X11:/opt/bin:${MET_NETCDF}/bin" +# - NCO is for ncdiff +# - NetCDF is for ncdump. +export PATH="/usr/local/nco/bin:/usr/local/netcdf/bin:\ + /usr/local/sbin:/usr/local/bin:/usr/sbin:\ + /usr/bin:/sbin:/bin:/usr/bin/X11:/opt/bin" From 58988bffa134be48d9320dd3b8a681ff933201a3 Mon Sep 17 00:00:00 2001 From: George McCabe <23407799+georgemccabe@users.noreply.github.com> Date: Wed, 12 Jan 2022 11:57:32 -0700 Subject: [PATCH 46/82] send email address of user who triggered event to METplus workflow as input --- .github/workflows/build_docker_and_trigger_metplus.yml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/.github/workflows/build_docker_and_trigger_metplus.yml b/.github/workflows/build_docker_and_trigger_metplus.yml index 02b6bca831..070fa071a6 100644 --- a/.github/workflows/build_docker_and_trigger_metplus.yml +++ b/.github/workflows/build_docker_and_trigger_metplus.yml @@ -40,6 +40,7 @@ jobs: repository: '${{ github.repository }}', ref: '${{ github.ref }}', actor: '${{ github.actor }}', - sha: '${{ github.sha }}' + sha: '${{ github.sha }}', + pusher_email: '${{ github.event.pusher.email }}' }, }); From b26788310d7ce3ff98d1c1063742fe898b13ac86 Mon Sep 17 00:00:00 2001 From: John Halley Gotway Date: Wed, 12 Jan 2022 12:41:58 -0700 Subject: [PATCH 47/82] Fixup the Rscript diffing logic to support running the nightly build on seneca. The problem here is that the ncdiff -x -v command line options that worked in version 4.7.0 NO LONGER work in 4.9.2. Added a check to skip over the time_bounds variable when processing the ncdiff output. --- test/R_test/test_util.R | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/test/R_test/test_util.R b/test/R_test/test_util.R index d00616209a..6dea9afd11 100644 --- a/test/R_test/test_util.R +++ b/test/R_test/test_util.R @@ -753,6 +753,11 @@ compareNc = function(nc1, nc2, verb, strict=0, delta=-1, comp_var=0){ # for each variable present in the file, check for differences for(strVar in names(ncFileD$var)){ + # Skip the time_bounds variable + # Note: Running "ncdiff -x -v time_bounds" successfully excludes that variable + # in version 4.7.0. However, it fails in version 4.9.2. + if (strVar == "time_bounds"){ next; } + # check the variable attributes for differences listAtt1 = ncatt_get(ncFile1, varid=strVar); listAtt1Nam = names(listAtt1); From 96536f54e361957670aa1e8673ad9dfe53436067 Mon Sep 17 00:00:00 2001 From: MET Tools Test Account Date: Wed, 12 Jan 2022 12:50:24 -0700 Subject: [PATCH 48/82] No real changes here. Just formatting --- test/R_test/test_util.R | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/R_test/test_util.R b/test/R_test/test_util.R index 6dea9afd11..d7491fa966 100644 --- a/test/R_test/test_util.R +++ b/test/R_test/test_util.R @@ -755,7 +755,7 @@ compareNc = function(nc1, nc2, verb, strict=0, delta=-1, comp_var=0){ # Skip the time_bounds variable # Note: Running "ncdiff -x -v time_bounds" successfully excludes that variable - # in version 4.7.0. However, it fails in version 4.9.2. + # in version 4.7.0. However, it fails in version 4.9.2. if (strVar == "time_bounds"){ next; } # check the variable attributes for differences From fa41760ba04ce6c20322c877dc3d4b272dccdc3a Mon Sep 17 00:00:00 2001 From: hsoh-u Date: Wed, 12 Jan 2022 14:12:03 -0700 Subject: [PATCH 49/82] Feature 1974 message_type_group_map (#1999) Co-authored-by: Howard Soh Co-authored-by: John Halley Gotway --- met/docs/Users_Guide/point-stat.rst | 2 +- met/src/libcode/vx_statistics/pair_data_ensemble.cc | 4 ++-- met/src/libcode/vx_statistics/pair_data_point.cc | 8 ++++---- 3 files changed, 7 insertions(+), 7 deletions(-) diff --git a/met/docs/Users_Guide/point-stat.rst b/met/docs/Users_Guide/point-stat.rst index e9cc3fdca8..7a9966511c 100644 --- a/met/docs/Users_Guide/point-stat.rst +++ b/met/docs/Users_Guide/point-stat.rst @@ -22,7 +22,7 @@ Interpolation/matching methods This section provides information about the various methods available in MET to match gridded model output to point observations. Matching in the vertical and horizontal are completed separately using different methods. -In the vertical, if forecasts and observations are at the same vertical level, then they are paired as-is. If any discrepancy exists between the vertical levels, then the forecasts are interpolated to the level of the observation. The vertical interpolation is done in the natural log of pressure coordinates, except for specific humidity, which is interpolated using the natural log of specific humidity in the natural log of pressure coordinates. Vertical interpolation for heights above ground are done linear in height coordinates. When forecasts are for the surface, no interpolation is done. They are matched to observations with message types that are mapped to **SURFACE** in the **message_type_group_map** configuration option. By default, the surface message types include ADPSFC, SFCSHP, and MSONET. +In the vertical, if forecasts and observations are at the same vertical level, then they are paired as-is. If any discrepancy exists between the vertical levels, then the forecasts are interpolated to the level of the observation. The vertical interpolation is done in the natural log of pressure coordinates, except for specific humidity, which is interpolated using the natural log of specific humidity in the natural log of pressure coordinates. Vertical interpolation for heights above ground are done linear in height coordinates. When forecasts are for the surface, no interpolation is done. They are matched to observations with message types that are mapped to **SURFACE** in the **message_type_group_map** configuration option. By default, the surface message types include ADPSFC, SFCSHP, and MSONET. The regular expression is applied to the message type list at the message_type_group_map. The derived message types from the time summary ("ADPSFC_MIN_hhmmss" and "ADPSFC_MAX_hhmmss") are accepted as "ADPSFC". To match forecasts and observations in the horizontal plane, the user can select from a number of methods described below. Many of these methods require the user to define the width of the forecast grid W, around each observation point P, that should be considered. In addition, the user can select the interpolation shape, either a SQUARE or a CIRCLE. For example, a square of width 2 defines the 2 x 2 set of grid points enclosing P, or simply the 4 grid points closest to P. A square of width of 3 defines a 3 x 3 square consisting of 9 grid points centered on the grid point closest to P. :numref:`point_stat_fig1` provides illustration. The point P denotes the observation location where the interpolated value is calculated. The interpolation width W, shown is five. diff --git a/met/src/libcode/vx_statistics/pair_data_ensemble.cc b/met/src/libcode/vx_statistics/pair_data_ensemble.cc index 4811f66147..6f6471b63a 100644 --- a/met/src/libcode/vx_statistics/pair_data_ensemble.cc +++ b/met/src/libcode/vx_statistics/pair_data_ensemble.cc @@ -1442,7 +1442,7 @@ void VxPairDataEnsemble::add_point_obs(float *hdr_arr, int *hdr_typ_arr, // falls within the requested range. else { - if(!msg_typ_sfc.has(hdr_typ_str) && + if(!msg_typ_sfc.reg_exp_match(hdr_typ_str) && (obs_hgt < obs_info_grib->level().lower() || obs_hgt > obs_info_grib->level().upper())) { return; @@ -1481,7 +1481,7 @@ void VxPairDataEnsemble::add_point_obs(float *hdr_arr, int *hdr_typ_arr, // set the observation level value to bad data so that it's not used in the // duplicate logic. if(obs_info->level().type() == LevelType_Vert && - msg_typ_sfc.has(hdr_typ_str)) { + msg_typ_sfc.reg_exp_match(hdr_typ_str)) { obs_lvl = bad_data_double; } diff --git a/met/src/libcode/vx_statistics/pair_data_point.cc b/met/src/libcode/vx_statistics/pair_data_point.cc index 6cfa81de6c..d26e6c8c89 100644 --- a/met/src/libcode/vx_statistics/pair_data_point.cc +++ b/met/src/libcode/vx_statistics/pair_data_point.cc @@ -984,7 +984,7 @@ void VxPairDataPoint::add_point_obs(float *hdr_arr, const char *hdr_typ_str, } // Check for a large topography difference - if(sfc_info.topo_ptr && msg_typ_sfc.has(hdr_typ_str)) { + if(sfc_info.topo_ptr && msg_typ_sfc.reg_exp_match(hdr_typ_str)) { // Interpolate model topography to observation location double topo = compute_horz_interp( @@ -1051,7 +1051,7 @@ void VxPairDataPoint::add_point_obs(float *hdr_arr, const char *hdr_typ_str, // falls within the requested range. else { - if(!msg_typ_sfc.has(hdr_typ_str) && + if(!msg_typ_sfc.reg_exp_match(hdr_typ_str) && (obs_hgt < obs_info->level().lower() || obs_hgt > obs_info->level().upper())) { rej_lvl++; @@ -1105,7 +1105,7 @@ void VxPairDataPoint::add_point_obs(float *hdr_arr, const char *hdr_typ_str, // type, set the observation level value to bad data so that it's not // used in the duplicate logic. if(obs_info->level().type() == LevelType_Vert && - msg_typ_sfc.has(hdr_typ_str)) { + msg_typ_sfc.reg_exp_match(hdr_typ_str)) { obs_lvl = bad_data_double; } @@ -1208,7 +1208,7 @@ void VxPairDataPoint::add_point_obs(float *hdr_arr, const char *hdr_typ_str, // For surface verification, apply land/sea and topo masks if((sfc_info.land_ptr || sfc_info.topo_ptr) && - (msg_typ_sfc.has(hdr_typ_str))) { + (msg_typ_sfc.reg_exp_match(hdr_typ_str))) { bool is_land = msg_typ_lnd.has(hdr_typ_str); From 551f92b618d21868a7f9a5e7cabb29c8827018c7 Mon Sep 17 00:00:00 2001 From: hsoh-u Date: Wed, 12 Jan 2022 14:15:56 -0700 Subject: [PATCH 50/82] Feature 1996 uninitialized variables (#2004) Co-authored-by: Howard Soh Co-authored-by: John Halley Gotway --- met/src/basic/vx_config/threshold.cc | 2 +- met/src/basic/vx_log/concat_string.cc | 21 ------------- met/src/basic/vx_log/concat_string.h | 8 ----- met/src/basic/vx_log/string_array.cc | 1 + met/src/basic/vx_log/string_array.h | 2 -- met/src/libcode/vx_nc_util/nc_utils.cc | 9 ++++-- met/src/libcode/vx_statistics/pair_base.cc | 1 + met/src/libcode/vx_tc_util/vx_tc_nc_util.cc | 1 + met/src/tools/tc_utils/tc_stat/tc_stat.cc | 35 +++++++++++---------- 9 files changed, 30 insertions(+), 50 deletions(-) diff --git a/met/src/basic/vx_config/threshold.cc b/met/src/basic/vx_config/threshold.cc index bf317733ff..78147f1071 100644 --- a/met/src/basic/vx_config/threshold.cc +++ b/met/src/basic/vx_config/threshold.cc @@ -1128,7 +1128,7 @@ else { // adjust PT by the requested frequency bias amount // - double PT_new; + double PT_new = 0.; if ( fbias_fcst ) { if ( op == thresh_le || op == thresh_lt ) PT_new = PT * fbias_val; diff --git a/met/src/basic/vx_log/concat_string.cc b/met/src/basic/vx_log/concat_string.cc index 2f4e6550dd..198f17ff66 100644 --- a/met/src/basic/vx_log/concat_string.cc +++ b/met/src/basic/vx_log/concat_string.cc @@ -50,8 +50,6 @@ ConcatString::ConcatString() init_from_scratch(); -set_alloc_inc(default_cs_alloc_inc); - } @@ -64,8 +62,6 @@ ConcatString::ConcatString(int _alloc_inc) init_from_scratch(); -set_alloc_inc(_alloc_inc); - } @@ -237,23 +233,6 @@ void ConcatString::assign(const ConcatString & c) memcpy(FloatFormat, c.FloatFormat, sizeof(FloatFormat)); Precision = c.Precision; - AllocInc = c.AllocInc; -} - - -//////////////////////////////////////////////////////////////////////// - - -void ConcatString::set_alloc_inc(int _alloc_inc) - -{ - -if ( _alloc_inc < min_cs_alloc_inc ) _alloc_inc = min_cs_alloc_inc; - -AllocInc = _alloc_inc; - -return; - } diff --git a/met/src/basic/vx_log/concat_string.h b/met/src/basic/vx_log/concat_string.h index 29903f8e40..03424effbc 100644 --- a/met/src/basic/vx_log/concat_string.h +++ b/met/src/basic/vx_log/concat_string.h @@ -71,10 +71,6 @@ class ConcatString { void assign(const ConcatString &); - void extend(int); - - int AllocInc; - int Precision; char FloatFormat[16]; @@ -102,8 +98,6 @@ class ConcatString { // set stuff // - void set_alloc_inc(int); - void set_precision(int); // @@ -207,8 +201,6 @@ inline int ConcatString::precision() const { return ( Precision ); } inline const char * ConcatString::float_format() const { return ( FloatFormat ); } -inline int ConcatString::alloc_inc() const { return ( AllocInc ); } - inline bool ConcatString::empty() const { return ( s ? s->empty() : true ); } inline bool ConcatString::nonempty() const { return ( s ? !s->empty() : false ); } diff --git a/met/src/basic/vx_log/string_array.cc b/met/src/basic/vx_log/string_array.cc index 21419fbe6c..57f90dac49 100644 --- a/met/src/basic/vx_log/string_array.cc +++ b/met/src/basic/vx_log/string_array.cc @@ -131,6 +131,7 @@ void StringArray::init_from_scratch() { IgnoreCase = 0; +MaxLength = 0; clear(); diff --git a/met/src/basic/vx_log/string_array.h b/met/src/basic/vx_log/string_array.h index 6563f68e2e..3662719197 100644 --- a/met/src/basic/vx_log/string_array.h +++ b/met/src/basic/vx_log/string_array.h @@ -37,8 +37,6 @@ class StringArray { std::vector s; - int Nalloc; - int MaxLength; bool IgnoreCase; diff --git a/met/src/libcode/vx_nc_util/nc_utils.cc b/met/src/libcode/vx_nc_util/nc_utils.cc index 50c3ff778e..8d1505875d 100644 --- a/met/src/libcode/vx_nc_util/nc_utils.cc +++ b/met/src/libcode/vx_nc_util/nc_utils.cc @@ -129,6 +129,9 @@ bool get_att_value(const NcAtt *att, double &att_val) { int get_att_value_int(const NcAtt *att) { int value = bad_data_int; static const char *method_name = "get_att_value_int(NcAtt) -> "; + + if (IS_INVALID_NC_P(att)) return value; + switch (att->getType().getId()) { case NC_BYTE: ncbyte b_value; @@ -221,7 +224,7 @@ bool get_att_value_chars(const NcAtt *att, ConcatString &value) { long long get_att_value_llong(const NcAtt *att) { long long value = bad_data_int; - att->getValues(&value); + if (IS_VALID_NC_P(att)) att->getValues(&value); return value; } @@ -229,7 +232,7 @@ long long get_att_value_llong(const NcAtt *att) { double get_att_value_double(const NcAtt *att) { double value = bad_data_double; - att->getValues(&value); + if (IS_VALID_NC_P(att)) att->getValues(&value); return value; } @@ -1658,6 +1661,8 @@ bool get_nc_data(NcVar *var, float *data) { fill_value = get_att_value_char(att_fill_value); } + var->getVar(packed_data); + if (unsigned_value) { int value; int positive_cnt = 0; diff --git a/met/src/libcode/vx_statistics/pair_base.cc b/met/src/libcode/vx_statistics/pair_base.cc index 45bf5696a5..a8f2f2ff83 100644 --- a/met/src/libcode/vx_statistics/pair_base.cc +++ b/met/src/libcode/vx_statistics/pair_base.cc @@ -71,6 +71,7 @@ void PairBase::clear() { msg_typ.clear(); msg_typ_vals.clear(); + interp_wdth = 0; interp_mthd = InterpMthd_None; interp_shape = GridTemplateFactory::GridTemplate_None; diff --git a/met/src/libcode/vx_tc_util/vx_tc_nc_util.cc b/met/src/libcode/vx_tc_util/vx_tc_nc_util.cc index 7bd51ffbbe..40a6feedc1 100644 --- a/met/src/libcode/vx_tc_util/vx_tc_nc_util.cc +++ b/met/src/libcode/vx_tc_util/vx_tc_nc_util.cc @@ -512,6 +512,7 @@ void write_tc_azi_mean_data(NcFile* nc_out, const TcrmwGrid& grid, data_azi_mean = new double[grid.range_n()]; for(int ir = 0; ir < grid.range_n(); ir++) { + data_azi_mean[ir] = 0.; for(int ia = 0; ia < grid.azimuth_n(); ia++) { int i = ir * grid.azimuth_n() + ia; int i_rev = (grid.range_n() - ir - 1) * grid.azimuth_n() + ia; diff --git a/met/src/tools/tc_utils/tc_stat/tc_stat.cc b/met/src/tools/tc_utils/tc_stat/tc_stat.cc index 5595e02b93..e1793ec194 100644 --- a/met/src/tools/tc_utils/tc_stat/tc_stat.cc +++ b/met/src/tools/tc_utils/tc_stat/tc_stat.cc @@ -165,6 +165,7 @@ void process_jobs() { ConcatString jobstring; int i, n_jobs; TCLineCounts n; + const char *method_name = "process_jobs() -> "; // Open the output file open_out_file(); @@ -193,27 +194,29 @@ void process_jobs() { // Allocate a new job cur_job = factory.new_tc_stat_job(jobstring.c_str()); + if (cur_job) { + // Set the job output file stream + cur_job->JobOut = tc_stat_out; - // Set the job output file stream - cur_job->JobOut = tc_stat_out; + // Set the output precision + cur_job->set_precision(conf_info.Conf.output_precision()); - // Set the output precision - cur_job->set_precision(conf_info.Conf.output_precision()); + // Serialize the current job + mlog << Debug(2) + << "\nProcessing Job " << i+1 << ": " + << cur_job->serialize() << "\n"; - // Serialize the current job - mlog << Debug(2) - << "\nProcessing Job " << i+1 << ": " - << cur_job->serialize() << "\n"; + // Initialize counts + memset(&n, 0, sizeof(TCLineCounts)); - // Initialize counts - memset(&n, 0, sizeof(TCLineCounts)); + // Do the job + cur_job->do_job(tcst_files, n); - // Do the job - cur_job->do_job(tcst_files, n); - - mlog << Debug(2) - << "Job " << i+1 << " used " << n.NKeep << " out of " - << n.NRead << " lines read.\n"; + mlog << Debug(2) << method_name + << "Job " << i+1 << " used " << n.NKeep << " out of " + << n.NRead << " lines read.\n"; + } + else mlog << Debug(1) << method_name << "job is missing\n"; mlog << Debug(3) << "Total lines read = " << n.NRead << "\n" From dbc7d10ec9f82a1a8885c940d0dd5dcb8c8c41d4 Mon Sep 17 00:00:00 2001 From: George McCabe <23407799+georgemccabe@users.noreply.github.com> Date: Wed, 12 Jan 2022 15:30:46 -0700 Subject: [PATCH 51/82] Feature 1695 ensemble single file (gen_ens_prod) (#2001) Co-authored-by: johnhg --- met/data/config/GenEnsProdConfig_default | 7 + met/docs/Users_Guide/gen-ens-prod.rst | 46 +++- met/src/basic/vx_config/config_constants.h | 2 + met/src/basic/vx_config/config_file.cc | 2 +- met/src/basic/vx_config/config_util.cc | 16 ++ met/src/basic/vx_config/config_util.h | 1 + met/src/basic/vx_config/dictionary.cc | 29 ++- met/src/basic/vx_config/dictionary.h | 5 +- met/src/basic/vx_log/file_fxns.h | 1 - .../ensemble_stat/ensemble_stat_conf_info.h | 1 + .../tools/other/gen_ens_prod/gen_ens_prod.cc | 212 +++++++++-------- .../tools/other/gen_ens_prod/gen_ens_prod.h | 2 +- .../gen_ens_prod/gen_ens_prod_conf_info.cc | 223 ++++++++++++++++-- .../gen_ens_prod/gen_ens_prod_conf_info.h | 48 +++- .../other/mode_graphics/plot_mode_field.cc | 2 +- scripts/environment/development.seneca | 2 +- test/config/GenEnsProdConfig | 7 + test/config/GenEnsProdConfig_single_file_grib | 149 ++++++++++++ test/config/GenEnsProdConfig_single_file_nc | 149 ++++++++++++ test/xml/unit_gen_ens_prod.xml | 56 ++++- 20 files changed, 819 insertions(+), 141 deletions(-) create mode 100644 test/config/GenEnsProdConfig_single_file_grib create mode 100644 test/config/GenEnsProdConfig_single_file_nc diff --git a/met/data/config/GenEnsProdConfig_default b/met/data/config/GenEnsProdConfig_default index fbe81a00f4..09746b626e 100644 --- a/met/data/config/GenEnsProdConfig_default +++ b/met/data/config/GenEnsProdConfig_default @@ -57,6 +57,13 @@ ens = { ]; } +// +// IDs for ensemble members +// Only set if processing a single file with all ensembles +// +ens_member_ids = []; +control_id = ""; + //////////////////////////////////////////////////////////////////////////////// // diff --git a/met/docs/Users_Guide/gen-ens-prod.rst b/met/docs/Users_Guide/gen-ens-prod.rst index 029d256ad2..459a9778f1 100644 --- a/met/docs/Users_Guide/gen-ens-prod.rst +++ b/met/docs/Users_Guide/gen-ens-prod.rst @@ -67,7 +67,7 @@ Required arguments gen_ens_prod Optional arguments for gen_ens_prod ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -4. The **-ctrl file** option specifies the input file for the ensemble control member. Data for this member is included in the computation of the ensemble mean, but excluded from the spread. The control file should not appear in the **-ens** list of ensemble member files. +4. The **-ctrl file** option specifies the input file for the ensemble control member. Data for this member is included in the computation of the ensemble mean, but excluded from the spread. The control file should not appear in the **-ens** list of ensemble member files (unless processing a single file that contains all ensemble members). 5. The **-log** file outputs log messages to the specified file. @@ -134,6 +134,50 @@ For each dictionary entry in the **field** array, give the name and vertical or _______________________ +.. code-block:: none + + ens_member_ids = []; + control_id = ""; + +The **ens_member_ids** array is only used if reading a single file that contains all ensemble members. +It should contain a list of string identifiers that are substituted into the **ens** dictionary fields +to determine which data to read from the file. +The length of the array determines how many ensemble members will be processed for a given field. +Each value in the array will replace the text **MET_ENS_MEMBER_ID**. + +**NetCDF Example:** + +.. code-block:: none + + ens = { + field = [ + { + name = "fcst"; + level = "(MET_ENS_MEMBER_ID,0,*,*)"; + } + ]; + } + +**GRIB Example:** + +.. code-block:: none + + ens = { + field = [ + { + name = "fcst"; + level = "L0"; + GRIB_ens = "+MET_ENS_MEMBER_ID"; + } + ]; + } + +**control_id** is a string that is substituted in the same way as the **ens_member_ids** values +to read a control member. This value is only used when the **-ctrl** command line argument is +used. The value should not be found in the **ens_member_ids** array. + +_______________________ + .. code-block:: none nbrhd_prob = { diff --git a/met/src/basic/vx_config/config_constants.h b/met/src/basic/vx_config/config_constants.h index 01c598bd72..01f03b3840 100644 --- a/met/src/basic/vx_config/config_constants.h +++ b/met/src/basic/vx_config/config_constants.h @@ -632,6 +632,8 @@ static const char conf_key_eclv_points[] = "eclv_points"; static const char conf_key_var_name_map[] = "var_name_map"; static const char conf_key_metadata_map[] = "metadata_map"; static const char conf_key_missing_thresh[] = "missing_thresh"; +static const char conf_key_control_id[] = "control_id"; +static const char conf_key_ens_member_ids[] = "ens_member_ids"; // // Entries to override file metadata diff --git a/met/src/basic/vx_config/config_file.cc b/met/src/basic/vx_config/config_file.cc index a7b1377405..1e229d5aa9 100644 --- a/met/src/basic/vx_config/config_file.cc +++ b/met/src/basic/vx_config/config_file.cc @@ -252,7 +252,7 @@ ConcatString MetConfig::get_tmp_dir() // Use the MET_TMP_DIR environment variable, if set. if(!get_env("MET_TMP_DIR", tmp_dir)) { const DictionaryEntry * _e = lookup(conf_key_tmp_dir); - if ( LastLookupStatus ) tmp_dir = *(_e->string_value()); + if ( LastLookupStatus ) tmp_dir = _e->string_value(); else tmp_dir = default_tmp_dir; } diff --git a/met/src/basic/vx_config/config_util.cc b/met/src/basic/vx_config/config_util.cc index 934bec72b9..1218e2940d 100644 --- a/met/src/basic/vx_config/config_util.cc +++ b/met/src/basic/vx_config/config_util.cc @@ -2965,3 +2965,19 @@ ConcatString wavelettype_to_string(WaveletType type) { } /////////////////////////////////////////////////////////////////////////////// + +StringArray parse_conf_ens_member_ids(Dictionary *dict) { + const char *method_name = "parse_conf_ens_member_ids() -> "; + + StringArray sa = parse_conf_string_array(dict, conf_key_ens_member_ids, method_name); + + if(sa.n() > 0) { + mlog << Debug(4) << method_name + << "Ensemble Member IDs \"" << conf_key_ens_member_ids << "\" list contains " + << sa.n() << " entries.\n"; + } + + return(sa); +} + +/////////////////////////////////////////////////////////////////////////////// diff --git a/met/src/basic/vx_config/config_util.h b/met/src/basic/vx_config/config_util.h index 5e83b2c6aa..ff9d38e7d4 100644 --- a/met/src/basic/vx_config/config_util.h +++ b/met/src/basic/vx_config/config_util.h @@ -73,6 +73,7 @@ extern map parse_conf_filter_attr_map(Dictionary *dict); extern void parse_conf_range_int(Dictionary *dict, int &beg, int &end); extern void parse_conf_range_double(Dictionary *dict, double &beg, double &end); +extern StringArray parse_conf_ens_member_ids(Dictionary *dict); extern void check_mask_names(const StringArray &); diff --git a/met/src/basic/vx_config/dictionary.cc b/met/src/basic/vx_config/dictionary.cc index 55ef59a0ac..f090916ace 100644 --- a/met/src/basic/vx_config/dictionary.cc +++ b/met/src/basic/vx_config/dictionary.cc @@ -762,21 +762,30 @@ return ( Bval ); //////////////////////////////////////////////////////////////////////// -const ConcatString * DictionaryEntry::string_value() const +const ConcatString DictionaryEntry::string_value() const { -if ( Type != StringType ) { + if ( Type != StringType ) { - mlog << Error - << "\nDictionaryEntry::string_value() -> bad type\n\n"; + mlog << Error + << "\nDictionaryEntry::string_value() -> bad type\n\n"; - exit ( 1 ); + exit ( 1 ); -} + } + + ConcatString sub_text = ConcatString(*Text); + ConcatString cur_env_val; + if ( get_env(met_ens_member_id, cur_env_val) ) { + if(!cur_env_val.empty()) { + sub_text.replace(met_ens_member_id, cur_env_val.c_str(), false); + } + + } -return ( Text ); + return ( sub_text ); } @@ -1816,7 +1825,7 @@ if ( !Entry || !is_correct_type ) { } -return ( *(Entry->string_value()) ); +return ( Entry->string_value() ); } @@ -1874,7 +1883,7 @@ if ( !Entry || !is_correct_type ) { if ( Entry->type() == StringType ) { - array.add( *(Entry->string_value()) ); + array.add( Entry->string_value() ); return ( array ); @@ -1911,7 +1920,7 @@ if ( Dict->n_entries() > 0 ) { for (int i=0; in_entries(); i++) { - array.add(*(*Dict)[i]->string_value()); + array.add((*Dict)[i]->string_value()); } diff --git a/met/src/basic/vx_config/dictionary.h b/met/src/basic/vx_config/dictionary.h index a39949be04..3a90e047e5 100644 --- a/met/src/basic/vx_config/dictionary.h +++ b/met/src/basic/vx_config/dictionary.h @@ -40,6 +40,9 @@ class Dictionary; // forward reference //////////////////////////////////////////////////////////////////////// +static const char met_ens_member_id [] = "MET_ENS_MEMBER_ID"; + +//////////////////////////////////////////////////////////////////////// class DictionaryEntry { @@ -122,7 +125,7 @@ class DictionaryEntry { int n_args () const; - const ConcatString * string_value () const; + const ConcatString string_value () const; Dictionary * dict_value () const; diff --git a/met/src/basic/vx_log/file_fxns.h b/met/src/basic/vx_log/file_fxns.h index 4b868bbdbf..f5d571962d 100644 --- a/met/src/basic/vx_log/file_fxns.h +++ b/met/src/basic/vx_log/file_fxns.h @@ -22,7 +22,6 @@ #include #include "concat_string.h" -#include "file_fxns.h" //////////////////////////////////////////////////////////////////////// diff --git a/met/src/tools/core/ensemble_stat/ensemble_stat_conf_info.h b/met/src/tools/core/ensemble_stat/ensemble_stat_conf_info.h index 7148109bba..df46f8ecf7 100644 --- a/met/src/tools/core/ensemble_stat/ensemble_stat_conf_info.h +++ b/met/src/tools/core/ensemble_stat/ensemble_stat_conf_info.h @@ -95,6 +95,7 @@ class EnsembleStatVxOpt { VxPairDataEnsemble vx_pd; // Ensemble pair data ConcatString var_str; // nc_pairs_var_str string + ConcatString control_id; // Control ID int beg_ds; // Begin observation time window offset int end_ds; // End observation time window offset diff --git a/met/src/tools/other/gen_ens_prod/gen_ens_prod.cc b/met/src/tools/other/gen_ens_prod/gen_ens_prod.cc index c7b48a7331..1f634c38b2 100644 --- a/met/src/tools/other/gen_ens_prod/gen_ens_prod.cc +++ b/met/src/tools/other/gen_ens_prod/gen_ens_prod.cc @@ -53,17 +53,17 @@ static void process_ensemble(); static bool get_data_plane(const char *, GrdFileType, VarInfo *, DataPlane &); static void clear_counts(); -static void track_counts(int, const DataPlane &, bool, +static void track_counts(EnsVarInfo *, const DataPlane &, bool, const DataPlane &, const DataPlane &); static void setup_nc_file(); -static void write_ens_nc(int, int, const DataPlane &, +static void write_ens_nc(EnsVarInfo *, int, const DataPlane &, const DataPlane &, const DataPlane &); -static void write_ens_var_float(int, float *, const DataPlane &, +static void write_ens_var_float(EnsVarInfo *, float *, const DataPlane &, const char *, const char *); -static void write_ens_var_int(int, int *, const DataPlane &, +static void write_ens_var_int(EnsVarInfo *, int *, const DataPlane &, const char *, const char *); -static void write_ens_data_plane(int, const DataPlane &, const DataPlane &, +static void write_ens_data_plane(EnsVarInfo *, const DataPlane &, const DataPlane &, const char *, const char *); static void add_var_att_local(VarInfo *, NcVar *, bool is_int, @@ -136,8 +136,8 @@ void process_command_line(int argc, char **argv) { if(cline.n() != 0) usage(); // Check that the required arguments have been set - n_ens = ens_files.n(); - if(ens_files.n() == 0) { + n_ens_files = ens_files.n(); + if(n_ens_files == 0) { mlog << Error << "\nprocess_command_line() -> " << "the ensemble file list must be set using the " << "\"-ens\" option.\n\n"; @@ -181,7 +181,7 @@ void process_command_line(int argc, char **argv) { etype = ens_mtddf->file_type(); // Process the configuration - conf_info.process_config(etype); + conf_info.process_config(etype, &ens_files); // Allocate arrays to store threshold counts thresh_cnt_na = new NumArray [conf_info.get_max_n_cat()]; @@ -193,8 +193,8 @@ void process_command_line(int argc, char **argv) { // List the input ensemble files mlog << Debug(1) << "Ensemble Files[" - << ens_files.n() << "]:\n"; - for(i=0; i " << "the ensemble control file should not appear in the list " << "of ensemble member files:\n" << ctrl_file << "\n\n"; @@ -211,7 +211,7 @@ void process_command_line(int argc, char **argv) { } // Check for missing non-python ensemble files - for(i=0; i " @@ -224,6 +224,12 @@ void process_command_line(int argc, char **argv) { } } + if(conf_info.control_id.nonempty() && ctrl_file.empty()) { + mlog << Warning << "\nprocess_command_line() -> " + << "control_id is set in the config file but " + << "control file is not provided with -ctrl argument\n\n"; + } + // Deallocate memory for data files if(ens_mtddf) { delete ens_mtddf; ens_mtddf = (Met2dDataFile *) 0; } @@ -238,7 +244,7 @@ void process_grid(const Grid &fcst_grid) { // Parse regridding logic RegridInfo ri; if(conf_info.get_n_var() > 0) { - ri = conf_info.ens_info[0]->regrid(); + ri = conf_info.ens_input[0]->get_var_info()->regrid(); } else { mlog << Error << "\nprocess_grid() -> " @@ -308,33 +314,45 @@ bool get_data_plane(const char *infile, GrdFileType ftype, //////////////////////////////////////////////////////////////////////// void process_ensemble() { - int i_var, i_file, n_ens_vld; + int i_var, i_ens, n_ens_vld, n_ens_inputs; bool need_reset; DataPlane ens_dp, ctrl_dp, cmn_dp, csd_dp; unixtime max_init_ut = bad_data_ll; + VarInfo * var_info; + ConcatString ens_file; // Loop through each of the ensemble fields to be processed - for(i_var=0; i_var::const_iterator var_it = conf_info.ens_input.begin(); + for(i_var=0; var_it != conf_info.ens_input.end(); var_it++, i_var++) { + + // Need to reinitialize counts and sums for each ensemble field + need_reset = true; + var_info = (*var_it)->get_var_info(); mlog << Debug(2) << "\n" << sep_str << "\n\n" << "Processing ensemble field: " - << conf_info.ens_info[i_var]->magic_str() << "\n"; + << (*var_it)->raw_magic_str << "\n"; - // Need to reinitialize counts and sums for each ensemble field - need_reset = true; + n_ens_inputs = (*var_it)->inputs_n(); + for(i_ens=n_ens_vld=0; i_ens < n_ens_inputs; i_ens++) { - // Loop through the ensemble member files - for(i_file=0, n_ens_vld=0; i_fileget_file(i_ens); + var_info = (*var_it)->get_var_info(i_ens); // Skip bad data files - if(!ens_file_vld[i_file]) continue; + if(!ens_file_vld[(*var_it)->get_file_index(i_ens)]) continue; + + mlog << Debug(3) << "\n" + << "Reading field: " + << var_info->magic_str() << "\n"; // Read data and track the valid data count - if(!get_data_plane(ens_files[i_file].c_str(), etype, - conf_info.ens_info[i_var], ens_dp)) { + if(!get_data_plane(ens_file.c_str(), etype, + var_info, ens_dp)) { mlog << Warning << "\nprocess_ensemble() -> " - << "ensemble field \"" << conf_info.ens_info[i_var]->magic_str() - << "\" not found in file \"" << ens_files[i_file] << "\"\n\n"; + << "ensemble field \"" << var_info->magic_str() + << "\" not found in file \"" << ens_file << "\"\n\n"; continue; } else { @@ -360,19 +378,24 @@ void process_ensemble() { // Read ensemble control member data, if provided if(ctrl_file.nonempty()) { + VarInfo * ctrl_info = (*var_it)->get_ctrl(i_ens); + + mlog << Debug(3) << "\n" + << "Reading control field: " + << ctrl_info->magic_str() << "\n"; // Error out if missing if(!get_data_plane(ctrl_file.c_str(), etype, - conf_info.ens_info[i_var], ctrl_dp)) { + ctrl_info, ctrl_dp)) { mlog << Error << "\nprocess_ensemble() -> " << "control member ensemble field \"" - << conf_info.ens_info[i_var]->magic_str() + << ctrl_info->magic_str() << "\" not found in file \"" << ctrl_file << "\"\n\n"; exit(1); } // Apply current data to the running sums and counts - track_counts(i_var, ctrl_dp, true, cmn_dp, csd_dp); + track_counts(*var_it, ctrl_dp, true, cmn_dp, csd_dp); } // end if ctrl_file @@ -381,26 +404,27 @@ void process_ensemble() { << " control member, " << (cmn_dp.is_empty() ? 0 : 1) << " climatology mean, and " << (csd_dp.is_empty() ? 0 : 1) << " climatology standard deviation field(s) for \"" - << conf_info.ens_info[i_var]->magic_str() << "\".\n"; + << var_info->magic_str() << "\".\n"; } // end if need_reset // Apply current data to the running sums and counts - track_counts(i_var, ens_dp, false, cmn_dp, csd_dp); + track_counts(*var_it, ens_dp, false, cmn_dp, csd_dp); // Keep track of the maximum initialization time if(is_bad_data(max_init_ut) || ens_dp.init() > max_init_ut) { max_init_ut = ens_dp.init(); } - } // end for i_file + } // end for it // Check for too much missing data - if(((double) n_ens_vld/n_ens) < conf_info.vld_ens_thresh) { + if(((double) n_ens_vld/n_ens_inputs) < conf_info.vld_ens_thresh) { mlog << Error << "\nprocess_ensemble() -> " - << n_ens - n_ens_vld << " of " << n_ens - << " missing fields for \"" << conf_info.ens_info[i_var]->magic_str() - << "\" exceeds the maximum allowable specified by \"" + << n_ens_vld << " of " << n_ens_inputs + << " (" << (double)n_ens_vld/n_ens_inputs << ")" + << " fields found for \"" << (*var_it)->get_var_info()->magic_str() + << "\" does not meet the threshold specified by \"" << conf_key_ens_ens_thresh << "\" (" << conf_info.vld_ens_thresh << ") in the configuration file.\n\n"; exit(1); @@ -408,9 +432,9 @@ void process_ensemble() { // Write out the ensemble information to a NetCDF file ens_dp.set_init(max_init_ut); - write_ens_nc(i_var, n_ens_vld, ens_dp, cmn_dp, csd_dp); + write_ens_nc(*var_it, n_ens_vld, ens_dp, cmn_dp, csd_dp); - } // end for i_var + } // end for var_it return; } @@ -441,14 +465,14 @@ void clear_counts() { //////////////////////////////////////////////////////////////////////// -void track_counts(int i_var, const DataPlane &ens_dp, bool is_ctrl, +void track_counts(EnsVarInfo * ens_info, const DataPlane &ens_dp, bool is_ctrl, const DataPlane &cmn_dp, const DataPlane &csd_dp) { int i, j, k; double ens, cmn, csd; // Ensemble thresholds - const int n_thr = conf_info.cat_ta[i_var].n(); - SingleThresh *thr_buf = conf_info.cat_ta[i_var].buf(); + const int n_thr = ens_info->cat_ta.n(); + SingleThresh *thr_buf = ens_info->cat_ta.buf(); // Increment counts for each grid point for(i=0; i 0 - if(conf_info.nc_info[i_var].do_nmep) { + if(ens_info->nc_info.do_nmep) { DataPlane frac_dp; // Loop over thresholds @@ -543,7 +567,7 @@ void setup_nc_file() { lon_dim = add_dim(nc_out, "lon", (long) grid.nx()); // Add the lat/lon variables - if(conf_info.nc_info[0].do_latlon) { + if(conf_info.ens_input[0]->nc_info.do_latlon) { write_netcdf_latlon(nc_out, &lat_dim, &lon_dim, grid); } @@ -552,7 +576,7 @@ void setup_nc_file() { //////////////////////////////////////////////////////////////////////// -void write_ens_nc(int i_var, int n_ens_vld, +void write_ens_nc(EnsVarInfo * ens_info, int n_ens_vld, const DataPlane &ens_dp, const DataPlane &cmn_dp, const DataPlane &csd_dp) { @@ -605,69 +629,69 @@ void write_ens_nc(int i_var, int n_ens_vld, } // end for i // Add the ensemble mean, if requested - if(conf_info.nc_info[i_var].do_mean) { - write_ens_var_float(i_var, ens_mean, ens_dp, + if(ens_info->nc_info.do_mean) { + write_ens_var_float(ens_info, ens_mean, ens_dp, "ENS_MEAN", "Ensemble Mean"); } // Add the ensemble standard deviation, if requested - if(conf_info.nc_info[i_var].do_stdev) { - write_ens_var_float(i_var, ens_stdev, ens_dp, + if(ens_info->nc_info.do_stdev) { + write_ens_var_float(ens_info, ens_stdev, ens_dp, "ENS_STDEV", "Ensemble Standard Deviation"); } // Add the ensemble mean minus one standard deviation, if requested - if(conf_info.nc_info[i_var].do_minus) { - write_ens_var_float(i_var, ens_minus, ens_dp, + if(ens_info->nc_info.do_minus) { + write_ens_var_float(ens_info, ens_minus, ens_dp, "ENS_MINUS", "Ensemble Mean Minus 1 Standard Deviation"); } // Add the ensemble mean plus one standard deviation, if requested - if(conf_info.nc_info[i_var].do_plus) { - write_ens_var_float(i_var, ens_plus, ens_dp, + if(ens_info->nc_info.do_plus) { + write_ens_var_float(ens_info, ens_plus, ens_dp, "ENS_PLUS", "Ensemble Mean Plus 1 Standard Deviation"); } // Add the ensemble minimum value, if requested - if(conf_info.nc_info[i_var].do_min) { - write_ens_var_float(i_var, ens_min, ens_dp, + if(ens_info->nc_info.do_min) { + write_ens_var_float(ens_info, ens_min, ens_dp, "ENS_MIN", "Ensemble Minimum"); } // Add the ensemble maximum value, if requested - if(conf_info.nc_info[i_var].do_max) { - write_ens_var_float(i_var, ens_max, ens_dp, + if(ens_info->nc_info.do_max) { + write_ens_var_float(ens_info, ens_max, ens_dp, "ENS_MAX", "Ensemble Maximum"); } // Add the ensemble range, if requested - if(conf_info.nc_info[i_var].do_range) { - write_ens_var_float(i_var, ens_range, ens_dp, + if(ens_info->nc_info.do_range) { + write_ens_var_float(ens_info, ens_range, ens_dp, "ENS_RANGE", "Ensemble Range"); } // Add the ensemble valid data count, if requested - if(conf_info.nc_info[i_var].do_vld) { - write_ens_var_int(i_var, ens_vld, ens_dp, + if(ens_info->nc_info.do_vld) { + write_ens_var_int(ens_info, ens_vld, ens_dp, "ENS_VLD", "Ensemble Valid Data Count"); } // Add the ensemble relative frequencies and neighborhood probabilities, if requested - if(conf_info.nc_info[i_var].do_freq || - conf_info.nc_info[i_var].do_nep) { + if(ens_info->nc_info.do_freq || + ens_info->nc_info.do_nep) { prob_dp.set_size(grid.nx(), grid.ny()); // Loop through each threshold - for(i=0; icat_ta.n(); i++) { // Initialize prob_dp.erase(); @@ -680,15 +704,15 @@ void write_ens_nc(int i_var, int n_ens_vld, } // Write ensemble relative frequency - if(conf_info.nc_info[i_var].do_freq) { + if(ens_info->nc_info.do_freq) { snprintf(type_str, sizeof(type_str), "ENS_FREQ_%s", - conf_info.cat_ta[i_var][i].get_abbr_str().contents().c_str()); - write_ens_data_plane(i_var, prob_dp, ens_dp, type_str, + ens_info->cat_ta[i].get_abbr_str().contents().c_str()); + write_ens_data_plane(ens_info, prob_dp, ens_dp, type_str, "Ensemble Relative Frequency"); } // Process the neighborhood ensemble probability - if(conf_info.nc_info[i_var].do_nep) { + if(ens_info->nc_info.do_nep) { GaussianInfo info; // Loop over the neighborhoods @@ -701,10 +725,10 @@ void write_ens_nc(int i_var, int n_ens_vld, // Write neighborhood ensemble probability snprintf(type_str, sizeof(type_str), "ENS_NEP_%s_%s%i", - conf_info.cat_ta[i_var][i].get_abbr_str().contents().c_str(), + ens_info->cat_ta[i].get_abbr_str().contents().c_str(), interpmthd_to_string(InterpMthd_Nbrhd).c_str(), conf_info.nbrhd_prob.width[j]*conf_info.nbrhd_prob.width[j]); - write_ens_data_plane(i_var, nbrhd_dp, ens_dp, type_str, + write_ens_data_plane(ens_info, nbrhd_dp, ens_dp, type_str, "Neighborhood Ensemble Probability"); } // end for k } // end if do_nep @@ -712,12 +736,12 @@ void write_ens_nc(int i_var, int n_ens_vld, } // end if // Add the neighborhood maximum ensemble probabilities, if requested - if(conf_info.nc_info[i_var].do_nmep) { + if(ens_info->nc_info.do_nmep) { prob_dp.set_size(grid.nx(), grid.ny()); // Loop through each threshold - for(i=0; icat_ta.n(); i++) { // Loop through each neigbhorhood size for(j=0; jcat_ta[i].get_abbr_str().contents().c_str(), interpmthd_to_string(InterpMthd_Nbrhd).c_str(), conf_info.nbrhd_prob.width[j]*conf_info.nbrhd_prob.width[j], conf_info.nmep_smooth.method[k].c_str(), conf_info.nmep_smooth.width[k]*conf_info.nmep_smooth.width[k]); - write_ens_data_plane(i_var, nbrhd_dp, ens_dp, type_str, + write_ens_data_plane(ens_info, nbrhd_dp, ens_dp, type_str, "Neighborhood Maximum Ensemble Probability"); } // end for k } // end for j @@ -757,26 +781,26 @@ void write_ens_nc(int i_var, int n_ens_vld, } // end if do_nep // Write the climo mean field, if requested - if(conf_info.nc_info[i_var].do_climo && !cmn_dp.is_empty()) { - write_ens_data_plane(i_var, cmn_dp, ens_dp, + if(ens_info->nc_info.do_climo && !cmn_dp.is_empty()) { + write_ens_data_plane(ens_info, cmn_dp, ens_dp, "CLIMO_MEAN", "Climatology mean"); } // Write the climo stdev field, if requested - if(conf_info.nc_info[i_var].do_climo && !csd_dp.is_empty()) { - write_ens_data_plane(i_var, csd_dp, ens_dp, + if(ens_info->nc_info.do_climo && !csd_dp.is_empty()) { + write_ens_data_plane(ens_info, csd_dp, ens_dp, "CLIMO_STDEV", "Climatology standard deviation"); } // Write the climo distribution percentile thresholds, if requested - if(conf_info.nc_info[i_var].do_climo_cdp && + if(ens_info->nc_info.do_climo_cdp && !cmn_dp.is_empty() && !csd_dp.is_empty()) { DataPlane cdp_dp; vector simp; - conf_info.cat_ta[i_var].get_simple_nodes(simp); + ens_info->cat_ta.get_simple_nodes(simp); // Process all CDP thresholds except 0 and 100 for(vector::iterator it = simp.begin(); @@ -787,7 +811,7 @@ void write_ens_nc(int i_var, int n_ens_vld, snprintf(type_str, sizeof(type_str), "CLIMO_CDP%i", nint(it->pvalue())); cdp_dp = normal_cdf_inv(it->pvalue()/100.0, cmn_dp, csd_dp); - write_ens_data_plane(i_var, cdp_dp, ens_dp, + write_ens_data_plane(ens_info, cdp_dp, ens_dp, type_str, "Climatology distribution percentile"); } @@ -809,20 +833,20 @@ void write_ens_nc(int i_var, int n_ens_vld, //////////////////////////////////////////////////////////////////////// -void write_ens_var_float(int i_var, float *ens_data, const DataPlane &dp, +void write_ens_var_float(EnsVarInfo * ens_info, float *ens_data, const DataPlane &dp, const char *type_str, const char *long_name_str) { NcVar ens_var; ConcatString ens_var_name, var_str, name_str, cs; // Append nc_var_str config file entry - cs = conf_info.nc_var_str[i_var]; + cs = ens_info->nc_var_str; if(cs.length() > 0) var_str << "_" << cs; // Construct the variable name ens_var_name << cs_erase - << conf_info.ens_info[i_var]->name_attr() << "_" - << conf_info.ens_info[i_var]->level_attr() + << ens_info->get_var_info()->name_attr() << "_" + << ens_info->get_var_info()->level_attr() << var_str << "_" << type_str; // Skip variable names that have already been written @@ -840,16 +864,16 @@ void write_ens_var_float(int i_var, float *ens_data, const DataPlane &dp, // if(strcmp(type_str, "ENS_MEAN") == 0) { name_str << cs_erase - << conf_info.ens_info[i_var]->name_attr(); + << ens_info->get_var_info()->name_attr(); } else { name_str << cs_erase - << conf_info.ens_info[i_var]->name_attr() << "_" + << ens_info->get_var_info()->name_attr() << "_" << type_str; } // Add the variable attributes - add_var_att_local(conf_info.ens_info[i_var], &ens_var, false, dp, + add_var_att_local(ens_info->get_var_info(), &ens_var, false, dp, name_str.c_str(), long_name_str); // Write the data @@ -865,20 +889,20 @@ void write_ens_var_float(int i_var, float *ens_data, const DataPlane &dp, //////////////////////////////////////////////////////////////////////// -void write_ens_var_int(int i_var, int *ens_data, const DataPlane &dp, +void write_ens_var_int(EnsVarInfo * ens_info, int *ens_data, const DataPlane &dp, const char *type_str, const char *long_name_str) { NcVar ens_var; ConcatString ens_var_name, var_str, name_str, cs; // Append nc_var_str config file entry - cs = conf_info.nc_var_str[i_var]; + cs = ens_info->nc_var_str; if(cs.length() > 0) var_str << "_" << cs; // Construct the variable name ens_var_name << cs_erase - << conf_info.ens_info[i_var]->name_attr() << "_" - << conf_info.ens_info[i_var]->level_attr() + << ens_info->get_var_info()->name_attr() << "_" + << ens_info->get_var_info()->level_attr() << var_str << "_" << type_str; // Skip variable names that have already been written @@ -892,11 +916,11 @@ void write_ens_var_int(int i_var, int *ens_data, const DataPlane &dp, // Construct the variable name attribute name_str << cs_erase - << conf_info.ens_info[i_var]->name_attr() << "_" + << ens_info->get_var_info()->name_attr() << "_" << type_str; // Add the variable attributes - add_var_att_local(conf_info.ens_info[i_var], &ens_var, true, dp, + add_var_att_local(ens_info->get_var_info(), &ens_var, true, dp, name_str.c_str(), long_name_str); // Write the data @@ -912,7 +936,7 @@ void write_ens_var_int(int i_var, int *ens_data, const DataPlane &dp, //////////////////////////////////////////////////////////////////////// -void write_ens_data_plane(int i_var, const DataPlane &ens_dp, const DataPlane &dp, +void write_ens_data_plane(EnsVarInfo * ens_info, const DataPlane &ens_dp, const DataPlane &dp, const char *type_str, const char *long_name_str) { // Allocate memory for this data @@ -922,7 +946,7 @@ void write_ens_data_plane(int i_var, const DataPlane &ens_dp, const DataPlane &d for(int i=0; i::const_iterator it; + vector::const_iterator var_it = ens_input.begin(); // Clear, erase, and initialize members model.clear(); desc.clear(); - for(it = ens_info.begin(); it != ens_info.end(); it++) { - if(*it) { delete *it; } + + for(; var_it != ens_input.end(); var_it++) { + + if(*var_it) { delete *var_it; } + } - ens_info.clear(); + ens_input.clear(); cdf_info.clear(); - cat_ta.clear(); - nc_var_str.clear(); nbrhd_prob.clear(); nmep_smooth.clear(); vld_ens_thresh = bad_data_double; vld_data_thresh = bad_data_double; - nc_info.clear(); version.clear(); // Reset counts @@ -100,12 +100,18 @@ void GenEnsProdConfInfo::read_config(const ConcatString default_file_name, //////////////////////////////////////////////////////////////////////// -void GenEnsProdConfInfo::process_config(GrdFileType etype) { - int i; +void GenEnsProdConfInfo::process_config(GrdFileType etype, StringArray * ens_files) { + int i, j; VarInfoFactory info_factory; Dictionary *edict = (Dictionary *) 0; Dictionary i_edict; InterpMthd mthd; + VarInfo * next_var; + + int n_ens_files = ens_files->n(); + + // Unset MET_ENS_MEMBER_ID in case it is set by the user + unsetenv(met_ens_member_id); // Dump the contents of the config file if(mlog.verbosity_level() >= 5) conf.dump(cout); @@ -122,6 +128,38 @@ void GenEnsProdConfInfo::process_config(GrdFileType etype) { // Conf: desc desc = parse_conf_string(&conf, conf_key_desc); + // Conf: ens_member_ids + ens_member_ids = parse_conf_ens_member_ids(&conf); + + // Conf: control_id + control_id = parse_conf_string(&conf, conf_key_control_id, false); + + // Error check ens_member_ids and ensemble file list + if(ens_member_ids.n() > 1) { + + // Only a single file should be provided if using ens_member_ids + if(n_ens_files > 1) { + mlog << Error << "\nGenEnsProdConfInfo::process_config() -> " + << "The \"" << conf_key_ens_member_ids << "\" " + << "must be empty if more than " + << "one file is provided.\n\n"; + exit(1); + } + + // If control ID is set, it cannot be found in ens_member_ids + if(!control_id.empty() && ens_member_ids.has(control_id)) { + mlog << Error << "\nGenEnsProdConfInfo::process_config() -> " + << "control_id (" << control_id << ") must not be found " + << "in ens_member_ids\n\n"; + exit(1); + } + } + + // If no ensemble member IDs were provided, add an empty string + if(ens_member_ids.n() == 0) { + ens_member_ids.add(""); + } + // Conf: ens.field edict = conf.lookup_array(conf_key_ens_field); @@ -136,41 +174,87 @@ void GenEnsProdConfInfo::process_config(GrdFileType etype) { // Parse the ensemble field information for(i=0,max_n_cat=0; iset_dict(i_edict); + // get VarInfo magic string without substituted values + ens_info->raw_magic_str = raw_magic_str(i_edict); - // Dump the contents of the current VarInfo - if(mlog.verbosity_level() >= 5) { - mlog << Debug(5) - << "Parsed ensemble field number " << i+1 << ":\n"; - ens_info[i]->dump(cout); + // Loop over ensemble member IDs to substitute + for(j=0; jset_dict(i_edict); + + // Dump the contents of the current VarInfo + if(mlog.verbosity_level() >= 5) { + mlog << Debug(5) + << "Parsed ensemble field number " << i+1 + << " (" << j+1 << "):\n"; + next_var->dump(cout); + } + + InputInfo input_info; + input_info.var_info = next_var; + input_info.file_index = 0; + input_info.file_list = ens_files; + ens_info->add_input(input_info); + + // Add InputInfo to ens info list for each ensemble file provided + // set var_info to NULL to note first VarInfo should be used + for(int k=1; kadd_input(input_info); + } // end for k + + } // end for j + + // Get field info for control member if set + if(!control_id.empty()) { + + // Set environment variable for ens member ID + setenv(met_ens_member_id, control_id.c_str(), 1); + + // Allocate new VarInfo object + next_var = info_factory.new_var_info(etype); + + // Set the current dictionary + next_var->set_dict(i_edict); + + ens_info->set_ctrl(next_var); } // Conf: nc_var_str - nc_var_str.add(parse_conf_string(&i_edict, conf_key_nc_var_str, false)); + ens_info->nc_var_str =parse_conf_string(&i_edict, conf_key_nc_var_str, false); // Conf: cat_thresh - cat_ta.push_back(i_edict.lookup_thresh_array(conf_key_cat_thresh)); + ens_info->cat_ta = i_edict.lookup_thresh_array(conf_key_cat_thresh); // Dump the contents of the current thresholds if(mlog.verbosity_level() >= 5) { mlog << Debug(5) << "Parsed thresholds for ensemble field number " << i+1 << ":\n"; - cat_ta[i].dump(cout); + ens_info->cat_ta.dump(cout); } // Keep track of the maximum number of thresholds - if(cat_ta[i].n() > max_n_cat) max_n_cat = cat_ta[i].n(); + if(ens_info->cat_ta.n() > max_n_cat) max_n_cat = ens_info->cat_ta.n(); // Conf: ensemble_flag - nc_info.push_back(parse_nc_info(&i_edict)); - } + ens_info->nc_info = parse_nc_info(&i_edict); + + ens_input.push_back(ens_info); + } // end for i // Conf: ens.ens_thresh vld_ens_thresh = conf.lookup_double(conf_key_ens_ens_thresh); @@ -360,3 +444,94 @@ void GenEnsProdNcOutInfo::set_all_true() { } //////////////////////////////////////////////////////////////////////// + + +//////////////////////////////////////////////////////////////////////// +// +// Code for class EnsVarInfo +// +//////////////////////////////////////////////////////////////////////// + +EnsVarInfo::EnsVarInfo() { + ctrl_info = NULL; +} + +//////////////////////////////////////////////////////////////////////// + +EnsVarInfo::~EnsVarInfo() { + vector::const_iterator it; + for(it = inputs.begin(); it != inputs.end(); it++) { + if((*it).var_info) { delete (*it).var_info; } + } + + if(ctrl_info) { delete ctrl_info; } +} + +//////////////////////////////////////////////////////////////////////// + +void EnsVarInfo::add_input(InputInfo input) { + inputs.push_back(input); +} + +//////////////////////////////////////////////////////////////////////// + +int EnsVarInfo::inputs_n() { + return inputs.size(); +} + +//////////////////////////////////////////////////////////////////////// + +void EnsVarInfo::set_ctrl(VarInfo * ctrl) { + ctrl_info = ctrl; +} + +//////////////////////////////////////////////////////////////////////// + +VarInfo * EnsVarInfo::get_ctrl(int index) { + if(ctrl_info) { + return ctrl_info; + } + return inputs[index].var_info; +} + +//////////////////////////////////////////////////////////////////////// + +VarInfo * EnsVarInfo::get_var_info(int index) { + if(inputs[index].var_info) { + return inputs[index].var_info; + } + return inputs[0].var_info; +} + +//////////////////////////////////////////////////////////////////////// + +ConcatString EnsVarInfo::get_file(int index) { + int file_index = inputs[index].file_index; + return (*inputs[index].file_list)[file_index]; +} + +//////////////////////////////////////////////////////////////////////// + +int EnsVarInfo::get_file_index(int index) { + return inputs[index].file_index; +} + +//////////////////////////////////////////////////////////////////////// + +ConcatString raw_magic_str(Dictionary i_edict) { + ConcatString magic_str; + + ConcatString name = i_edict.lookup_string("name"); + ConcatString level = i_edict.lookup_string("level"); + + if(level.nonempty() && level[0] != '(') { + magic_str << name << "/" << level; + } else { + magic_str << name << level; + } + + return magic_str; + +} + +//////////////////////////////////////////////////////////////////////// diff --git a/met/src/tools/other/gen_ens_prod/gen_ens_prod_conf_info.h b/met/src/tools/other/gen_ens_prod/gen_ens_prod_conf_info.h index 2ec6014824..d0d5e60bbe 100644 --- a/met/src/tools/other/gen_ens_prod/gen_ens_prod_conf_info.h +++ b/met/src/tools/other/gen_ens_prod/gen_ens_prod_conf_info.h @@ -51,6 +51,43 @@ struct GenEnsProdNcOutInfo { void set_all_true(); }; + +//////////////////////////////////////////////////////////////////////// + +struct InputInfo { + VarInfo * var_info; // Variable information to read + int file_index; // Index in file_list of file to read + StringArray * file_list; // Array of files (unallocated) +}; + +//////////////////////////////////////////////////////////////////////// + +class EnsVarInfo { + + private: + vector inputs; // Vector of InputInfo + VarInfo * ctrl_info; // Field info for control member + public: + EnsVarInfo(); + ~EnsVarInfo(); + + void add_input(InputInfo); + int inputs_n(); + + void set_ctrl(VarInfo *); + VarInfo * get_ctrl(int); + + // Get VarInfo from first InputInfo if requested VarInfo is NULL + VarInfo * get_var_info(int index=0); + ConcatString get_file(int index=0); + int get_file_index(int index=0); + + ConcatString nc_var_str; // Ensemble variable name strings + ThreshArray cat_ta; // Ensemble categorical thresholds + GenEnsProdNcOutInfo nc_info; // Ensemble product outputs + ConcatString raw_magic_str; // Magic string w/o var substitution +}; + //////////////////////////////////////////////////////////////////////// class GenEnsProdConfInfo { @@ -76,12 +113,11 @@ class GenEnsProdConfInfo { // Data parsed from the Gen-Ens-Prod configuration object ConcatString model; // Model name ConcatString desc; // Description + ConcatString control_id; // Control ID - vector ens_info; // Array of VarInfo pointers (allocated) + vector ens_input; // Vector of EnsVarInfo pointers (allocated) vector cdf_info; // Array of climo CDF info objects - vector cat_ta; // Array for ensemble categorical thresholds - StringArray nc_var_str; // Array of ensemble variable name strings - vector nc_info; // Array of ensemble product outputs + StringArray ens_member_ids; // Array of ensemble member ID strings NbrhdInfo nbrhd_prob; // Neighborhood probability definition InterpInfo nmep_smooth; // Neighborhood maximum smoothing information @@ -96,7 +132,7 @@ class GenEnsProdConfInfo { void clear(); void read_config (const ConcatString, const ConcatString); - void process_config(GrdFileType); + void process_config(GrdFileType, StringArray *); GenEnsProdNcOutInfo parse_nc_info(Dictionary *); @@ -116,6 +152,8 @@ inline int GenEnsProdConfInfo::get_compression_level() { return(conf.nc_compress //////////////////////////////////////////////////////////////////////// +ConcatString raw_magic_str(Dictionary i_edict); + #endif /* __GEN_ENS_PROD_CONF_INFO_H__ */ //////////////////////////////////////////////////////////////////////// diff --git a/met/src/tools/other/mode_graphics/plot_mode_field.cc b/met/src/tools/other/mode_graphics/plot_mode_field.cc index 8d0894f154..ab77a6bb4c 100644 --- a/met/src/tools/other/mode_graphics/plot_mode_field.cc +++ b/met/src/tools/other/mode_graphics/plot_mode_field.cc @@ -1111,7 +1111,7 @@ if ( e->type() != StringType ) { } -s = *(e->string_value()); +s = e->string_value(); return ( s ); diff --git a/scripts/environment/development.seneca b/scripts/environment/development.seneca index 325e80ed3b..5b3497c0c1 100644 --- a/scripts/environment/development.seneca +++ b/scripts/environment/development.seneca @@ -48,5 +48,5 @@ export MET_TEST_RSCRIPT=/usr/local/R-4.1.2/bin/Rscript # - NetCDF is for ncdump. export PATH="/usr/local/nco/bin:/usr/local/netcdf/bin:\ /usr/local/sbin:/usr/local/bin:/usr/sbin:\ - /usr/bin:/sbin:/bin:/usr/bin/X11:/opt/bin" + /usr/bin:/sbin:/bin:/usr/bin/X11:/opt/bin:$PATH" diff --git a/test/config/GenEnsProdConfig b/test/config/GenEnsProdConfig index 33fcc2337f..3c4af297ef 100644 --- a/test/config/GenEnsProdConfig +++ b/test/config/GenEnsProdConfig @@ -74,6 +74,13 @@ ens = { ]; } +// +// IDs for ensemble members +// Only set if processing a single file with all ensembles +// +ens_member_ids = []; +control_id = ""; + //////////////////////////////////////////////////////////////////////////////// // diff --git a/test/config/GenEnsProdConfig_single_file_grib b/test/config/GenEnsProdConfig_single_file_grib new file mode 100644 index 0000000000..cc5edda640 --- /dev/null +++ b/test/config/GenEnsProdConfig_single_file_grib @@ -0,0 +1,149 @@ +//////////////////////////////////////////////////////////////////////////////// +// +// Gen-Ens-Prod configuration file. +// +// For additional information, please see the MET User's Guide. +// +//////////////////////////////////////////////////////////////////////////////// + +// +// Output model name to be written +// +model = "GEFS"; + +// +// Output description to be written +// May be set separately in each "obs.field" entry +// +desc = "NA"; + +//////////////////////////////////////////////////////////////////////////////// + +// +// Verification grid +// May be set separately in each "field" entry +// +regrid = { + to_grid = NONE; + method = NEAREST; + width = 1; + vld_thresh = 0.5; + shape = SQUARE; +} + +//////////////////////////////////////////////////////////////////////////////// + +// +// May be set separately in each "field" entry +// +censor_thresh = []; +censor_val = []; +cat_thresh = []; +nc_var_str = ""; + +// +// Ensemble fields to be processed +// +ens = { + ens_thresh = 1.0; + vld_thresh = 1.0; + + field = [ + { + name = "PRMSL"; + level = "L0"; + lead_time = "06"; + GRIB_ens = "MET_ENS_MEMBER_ID"; + } + ]; +} + +// +// IDs for ensemble members +// Only set if processing a single file with all ensembles +// +ens_member_ids = ["+1", "+2", "+3", "+4", "+5", "+6", "+7", "+8", "+9", "+10", "+11", "+12", "+13", "+14", "+15", "+16", "+17", "+18", "+19", "+20"]; +control_id = "hi_res_ctl"; + +//////////////////////////////////////////////////////////////////////////////// + +// +// Neighborhood ensemble probabilities +// +nbrhd_prob = { + width = [ 5 ]; + shape = CIRCLE; + vld_thresh = 0.0; +} + +// +// NMEP smoothing methods +// +nmep_smooth = { + vld_thresh = 0.0; + shape = CIRCLE; + gaussian_dx = 81.27; + gaussian_radius = 120; + type = [ + { + method = GAUSSIAN; + width = 1; + } + ]; +} + +//////////////////////////////////////////////////////////////////////////////// + +// +// Climatology data +// +climo_mean = { + + file_name = []; + field = []; + + regrid = { + method = NEAREST; + width = 1; + vld_thresh = 0.5; + shape = SQUARE; + } + + time_interp_method = DW_MEAN; + day_interval = 31; + hour_interval = 6; +} + +climo_stdev = climo_mean; +climo_stdev = { + file_name = []; +} + +//////////////////////////////////////////////////////////////////////////////// + +// +// Ensemble product output types +// May be set separately in each "ens.field" entry +// +ensemble_flag = { + latlon = TRUE; + mean = TRUE; + stdev = TRUE; + minus = TRUE; + plus = TRUE; + min = TRUE; + max = TRUE; + range = TRUE; + vld_count = TRUE; + frequency = TRUE; + nep = FALSE; + nmep = FALSE; + climo = FALSE; + climo_cdp = FALSE; +} + +//////////////////////////////////////////////////////////////////////////////// + +version = "V10.1.0"; + +//////////////////////////////////////////////////////////////////////////////// diff --git a/test/config/GenEnsProdConfig_single_file_nc b/test/config/GenEnsProdConfig_single_file_nc new file mode 100644 index 0000000000..1049dc6a3a --- /dev/null +++ b/test/config/GenEnsProdConfig_single_file_nc @@ -0,0 +1,149 @@ +//////////////////////////////////////////////////////////////////////////////// +// +// Gen-Ens-Prod configuration file. +// +// For additional information, please see the MET User's Guide. +// +//////////////////////////////////////////////////////////////////////////////// + +// +// Output model name to be written +// +model = "WRF"; + +// +// Output description to be written +// May be set separately in each "obs.field" entry +// +desc = "NA"; + +//////////////////////////////////////////////////////////////////////////////// + +// +// Verification grid +// May be set separately in each "field" entry +// +regrid = { + to_grid = NONE; + method = NEAREST; + width = 1; + vld_thresh = 0.5; + shape = SQUARE; +} + +//////////////////////////////////////////////////////////////////////////////// + +// +// May be set separately in each "field" entry +// +censor_thresh = []; +censor_val = []; +cat_thresh = []; +nc_var_str = ""; + +// +// Ensemble fields to be processed +// +ens = { + file_type = NETCDF_NCCF; + ens_thresh = 1.0; + vld_thresh = 1.0; + + field = [ + { + name = "fcst"; + level = "(MET_ENS_MEMBER_ID,0,*,*)"; + cat_thresh = [ >0.0, >=5.0 ]; + } + ]; +} + +// +// IDs for ensemble members +// Only set if processing a single file with all ensembles +// +ens_member_ids = ["1", "2", "3", "4", "5", "6", "7", "8", "9", "10", "11", "12", "13", "14", "15", "16", "17", "18", "19", "20", "21", "22", "23"]; +control_id = "0"; + +//////////////////////////////////////////////////////////////////////////////// + +// +// Neighborhood ensemble probabilities +// +nbrhd_prob = { + width = [ 5 ]; + shape = CIRCLE; + vld_thresh = 0.0; +} + +// +// NMEP smoothing methods +// +nmep_smooth = { + vld_thresh = 0.0; + shape = CIRCLE; + gaussian_dx = 81.27; + gaussian_radius = 120; + type = [ + { + method = GAUSSIAN; + width = 1; + } + ]; +} + +//////////////////////////////////////////////////////////////////////////////// + +// +// Climatology data +// +climo_mean = ens; +climo_mean = { + + file_name = []; + + regrid = { + method = NEAREST; + width = 1; + vld_thresh = 0.5; + shape = SQUARE; + } + + time_interp_method = DW_MEAN; + day_interval = 31; + hour_interval = 6; +} + +climo_stdev = climo_mean; +climo_stdev = { + file_name = []; +} + +//////////////////////////////////////////////////////////////////////////////// + +// +// Ensemble product output types +// May be set separately in each "ens.field" entry +// +ensemble_flag = { + latlon = TRUE; + mean = TRUE; + stdev = TRUE; + minus = TRUE; + plus = TRUE; + min = TRUE; + max = TRUE; + range = TRUE; + vld_count = TRUE; + frequency = TRUE; + nep = FALSE; + nmep = FALSE; + climo = FALSE; + climo_cdp = FALSE; +} + +//////////////////////////////////////////////////////////////////////////////// + +version = "V10.1.0"; + +//////////////////////////////////////////////////////////////////////////////// diff --git a/test/xml/unit_gen_ens_prod.xml b/test/xml/unit_gen_ens_prod.xml index 9d3f2b0c75..91f2321431 100644 --- a/test/xml/unit_gen_ens_prod.xml +++ b/test/xml/unit_gen_ens_prod.xml @@ -75,4 +75,58 @@ - + + &MET_BIN;/gen_ens_prod + \ + -ens &DATA_DIR_MODEL;/CPC_NMME/CFSv2.tmp2m.198201.fcst.nc \ + -config &CONFIG_DIR;/GenEnsProdConfig_single_file_nc \ + -out &OUTPUT_DIR;/gen_ens_prod/gen_ens_prod_SINGLE_FILE_NC_NO_CTRL.nc \ + -v 2 + + + &OUTPUT_DIR;/gen_ens_prod/gen_ens_prod_SINGLE_FILE_NC_NO_CTRL.nc + + + + + &MET_BIN;/gen_ens_prod + \ + -ens &DATA_DIR_MODEL;/CPC_NMME/CFSv2.tmp2m.198201.fcst.nc \ + -config &CONFIG_DIR;/GenEnsProdConfig_single_file_nc \ + -ctrl &DATA_DIR_MODEL;/CPC_NMME/CFSv2.tmp2m.198201.fcst.nc \ + -out &OUTPUT_DIR;/gen_ens_prod/gen_ens_prod_SINGLE_FILE_NC_WITH_CTRL.nc \ + -v 2 + + + &OUTPUT_DIR;/gen_ens_prod/gen_ens_prod_SINGLE_FILE_NC_WITH_CTRL.nc + + + + + &MET_BIN;/gen_ens_prod + \ + -ens &DATA_DIR_MODEL;/grib2/gefs/enspost_grb2.t00z.prmsl \ + -config &CONFIG_DIR;/GenEnsProdConfig_single_file_grib \ + -out &OUTPUT_DIR;/gen_ens_prod/gen_ens_prod_SINGLE_FILE_GRIB_NO_CTRL.nc \ + -v 2 + + + &OUTPUT_DIR;/gen_ens_prod/gen_ens_prod_SINGLE_FILE_GRIB_NO_CTRL.nc + + + + + &MET_BIN;/gen_ens_prod + \ + -ens &DATA_DIR_MODEL;/grib2/gefs/enspost_grb2.t00z.prmsl \ + -config &CONFIG_DIR;/GenEnsProdConfig_single_file_grib \ + -ctrl &DATA_DIR_MODEL;/grib2/gefs/enspost_grb2.t00z.prmsl \ + -out &OUTPUT_DIR;/gen_ens_prod/gen_ens_prod_SINGLE_FILE_GRIB_WITH_CTRL.nc \ + -v 2 + + + &OUTPUT_DIR;/gen_ens_prod/gen_ens_prod_SINGLE_FILE_GRIB_WITH_CTRL.nc + + + + From be7910eaa6bc3a8f4191ae501d0e9095880ec0ce Mon Sep 17 00:00:00 2001 From: hsoh-u Date: Wed, 12 Jan 2022 16:21:10 -0700 Subject: [PATCH 52/82] Feature 1965 NB faile with time summary by ioda2nc (#2008) Co-authored-by: Howard Soh --- met/src/tools/other/ioda2nc/ioda2nc.cc | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/met/src/tools/other/ioda2nc/ioda2nc.cc b/met/src/tools/other/ioda2nc/ioda2nc.cc index 323cd83447..df1830c084 100644 --- a/met/src/tools/other/ioda2nc/ioda2nc.cc +++ b/met/src/tools/other/ioda2nc/ioda2nc.cc @@ -369,7 +369,7 @@ void process_ioda_file(int i_pb) { IntArray diff_file_times; int diff_file_time_count; StringArray variables_big_nlevels; - static const char *method_name = "process_ioda_file() ->"; + static const char *method_name = "process_ioda_file() -> "; static const char *method_name_s = "process_ioda_file() "; bool apply_grid_mask = (conf_info.grid_mask.nx() > 0 && @@ -961,7 +961,9 @@ void addObservation(const float *obs_arr, const ConcatString &hdr_typ, else obs_qty.format("%d", nint(quality_mark)); int var_index = obs_arr[1]; + map name_map = conf_info.getObsVarMap(); string var_name = obs_var_names[var_index]; + string out_name = name_map[var_name]; Observation obs = Observation(hdr_typ.text(), hdr_sid.text(), hdr_vld, @@ -969,7 +971,7 @@ void addObservation(const float *obs_arr, const ConcatString &hdr_typ, obs_qty.text(), var_index, obs_arr[2], obs_arr[3], obs_arr[4], - var_name); + (0addObservationObj(obs); From 1e13dea8e927ca9982bac3c2ec5c97bde1ef1cf6 Mon Sep 17 00:00:00 2001 From: johnhg Date: Wed, 12 Jan 2022 20:12:20 -0700 Subject: [PATCH 53/82] Feature 1810 shapes (#2005) --- met/docs/Users_Guide/tc-gen.rst | 73 ++- met/src/basic/vx_config/Makefile.am | 1 - met/src/libcode/vx_color/Makefile.am | 1 - met/src/libcode/vx_gis/Makefile.am | 28 +- met/src/libcode/vx_gis/dbf_file.cc | 400 +++++++++++++++- met/src/libcode/vx_gis/dbf_file.h | 78 +++- met/src/libcode/vx_gis/shp_array.h | 2 + met/src/libcode/vx_gis/shp_file.cc | 5 - met/src/libcode/vx_gis/shp_file.h | 5 - met/src/libcode/vx_gis/shp_poly_record.cc | 21 + met/src/libcode/vx_gis/shp_poly_record.h | 11 +- met/src/libcode/vx_gis/vx_gis.h | 29 ++ met/src/libcode/vx_statistics/Makefile.am | 1 + .../vx_statistics}/grid_closed_poly.cc | 226 ++++------ .../vx_statistics}/grid_closed_poly.h | 38 +- met/src/libcode/vx_statistics/vx_statistics.h | 1 + met/src/libcode/vx_tc_util/Makefile.am | 1 + met/src/libcode/vx_tc_util/gen_shape_info.cc | 377 ++++++++++++++++ met/src/libcode/vx_tc_util/gen_shape_info.h | 149 ++++++ met/src/libcode/vx_tc_util/vx_tc_util.h | 1 + met/src/tools/core/grid_stat/grid_stat.cc | 2 +- met/src/tools/core/point_stat/point_stat.cc | 2 +- met/src/tools/other/gen_vx_mask/Makefile.am | 6 +- .../tools/other/gen_vx_mask/gen_vx_mask.cc | 75 +-- met/src/tools/tc_utils/tc_gen/Makefile.am | 1 + met/src/tools/tc_utils/tc_gen/tc_gen.cc | 426 ++++++++++++++++-- met/src/tools/tc_utils/tc_gen/tc_gen.h | 16 +- .../tools/tc_utils/tc_gen/tc_gen_conf_info.cc | 154 ++++++- .../tools/tc_utils/tc_gen/tc_gen_conf_info.h | 33 +- test/config/TCGenConfig_shape | 292 ++++++++++++ test/xml/unit_tc_gen.xml | 19 + 31 files changed, 2102 insertions(+), 372 deletions(-) create mode 100644 met/src/libcode/vx_gis/vx_gis.h rename met/src/{tools/other/gen_vx_mask => libcode/vx_statistics}/grid_closed_poly.cc (53%) rename met/src/{tools/other/gen_vx_mask => libcode/vx_statistics}/grid_closed_poly.h (76%) create mode 100644 met/src/libcode/vx_tc_util/gen_shape_info.cc create mode 100644 met/src/libcode/vx_tc_util/gen_shape_info.h create mode 100644 test/config/TCGenConfig_shape diff --git a/met/docs/Users_Guide/tc-gen.rst b/met/docs/Users_Guide/tc-gen.rst index b67f1a26b9..74271ec9e1 100644 --- a/met/docs/Users_Guide/tc-gen.rst +++ b/met/docs/Users_Guide/tc-gen.rst @@ -6,19 +6,25 @@ TC-Gen Tool Introduction ____________ -The TC-Gen tool provides verification of deterministic and probabilistic tropical cyclone genesis forecasts in the ATCF file format. Producing reliable tropical cyclone genesis forecasts is an important metric for global numerical weather prediction models. This tool ingests deterministic model output post-processed by a genesis tracking software (e.g. GFDL vortex tracker), ATCF edeck files containing probability of genesis forecasts, and ATCF reference track dataset(s) (e.g. Best Track analysis and CARQ operational tracks). It writes categorical counts and statistics. The capability to modify the spatial and temporal tolerances when matching forecasts to reference genesis events, as well as scoring those matched pairs, gives users the ability to condition the criteria based on model performance and/or conduct sensitivity analyses. Statistical aspects are outlined in :numref:`tc-gen_stat_aspects` and practical aspects of the TC-Gen tool are described in :numref:`tc-gen_practical_info`. +The TC-Gen tool provides verification of deterministic and probabilistic tropical cyclone genesis forecasts in the ATCF file and shapefile formats. Producing reliable tropical cyclone genesis forecasts is an important metric for global numerical weather prediction models. This tool ingests deterministic model output post-processed by genesis tracking software (e.g. GFDL vortex tracker), ATCF edeck files containing probability of genesis forecasts, operational shapefile warning areas, and ATCF reference track dataset(s) (e.g. Best Track analysis and CARQ operational tracks). It writes categorical counts and statistics. The capability to modify the spatial and temporal tolerances when matching forecasts to reference genesis events, as well as scoring those matched pairs, gives users the ability to condition the criteria based on model performance and/or conduct sensitivity analyses. Statistical aspects are outlined in :numref:`tc-gen_stat_aspects` and practical aspects of the TC-Gen tool are described in :numref:`tc-gen_practical_info`. .. _tc-gen_stat_aspects: Statistical aspects ___________________ -The TC-Gen tool processes both deterministic and probabilistic forecasts. For deterministic forecasts specified using the **-track** command line option, it identifies genesis events in both the forecasts and reference datasets, typically Best tracks. It applies user-specified configuration options to pair up the forecast and reference genesis events and categorize each pair as a hit, miss, or false alarm. +The TC-Gen tool processes both deterministic and probabilistic forecasts. + +For deterministic forecasts specified using the **-track** command line option, it identifies genesis events in both the forecasts and reference datasets, typically Best tracks. It applies user-specified configuration options to pair up the forecast and reference genesis events and categorize each pair as a hit, miss, or false alarm. As with other extreme events (where the event occurs much less frequently than the non-event), the correct negative category is not computed since the non-events would dominate the contingency table. Therefore, only statistics that do not include correct negatives should be considered for this tool. The following CTS statistics are relevant: Base rate (BASER), Mean forecast (FMEAN), Frequency Bias (FBIAS), Probability of Detection (PODY), False Alarm Ratio (FAR), Critical Success Index (CSI), Gilbert Skill Score (GSS), Extreme Dependency Score (EDS), Symmetric Extreme Dependency Score (SEDS), Bias Adjusted Gilbert Skill Score (BAGSS). For probabilistic forecasts specified using the **-edeck** command line option, it identifies genesis events in the reference dataset. It applies user-specified configuration options to pair the forecast probabilities to the reference genesis events. These pairs are added to an Nx2 probabilistic contingency table. If the reference genesis event occurs within in the predicted time window, the pair is counted in the observation-yes column. Otherwise, it is added to the observation-no column. +For warning area shapefiles specified using the **-shape** command line option, it processes metadata from the corresponding database files. The database file is assumed to exist at exactly the same path as the shapefile, but with a ".dbf" suffix instead of ".shp". Note that only shapefiles exactly following the NOAA National Hurricane Center's (NHC) "gtwo_areas_YYYYMMDDHHMM.shp" file naming and corresonding metadata conventions are supported. For each shapefile record, the database file defines up to three corresponding probability values. The first percentage is interpreted as the probability of genesis inside the shape within 48 hours. The second and, if provided, third percentages are interpreted as the 120-hour and 168-hour probabilities, respectively. Care is taken to identify and either ignore or update duplicate shapes found in the input. + +The shapes are then subset based on the filtering criteria in the configuration file. For each probability and shape, the reference genesis events are searched for a match within the defined time window. These pairs are added to an Nx2 probabilistic contingency table. The probabilistic contingeny tables and statistics are computed and reported separately for filter defined and lead hour encountered in the input. + Other considerations for interpreting the output of the TC-Gen tool involve the size of the contingency table output. The size of the contingency table will change depending on the number of matches. Additionally, the number of misses is based on the forecast duration and interval (specified in the configuration file). This change is due to the number of model opportunities to forecast the event, which is determined by the specified duration/interval. Care should be taken when interpreting the statistics for filtered data. In some cases, variables (e.g. storm name) are only available in either the forecast or reference datasets, rather than both. When filtering on a field that is only present in one dataset, the contingency table counts will be impacted. Similarly, the initialization field only impacts the model forecast data. If the valid time (which will impact the reference dataset) isn't also specified, the forecasts will be filtered and matched such that the number of misses will erroneously increase. See :numref:`tc-gen_practical_info` for more detail. @@ -28,7 +34,7 @@ Care should be taken when interpreting the statistics for filtered data. In some Practical information _____________________ -This section describes how to configure and run the TC-Gen tool. The TC-Gen tool identifies tropical cyclone genesis events in both genesis forecasts and ATCF track datasets. It applies configurable logic to process the forecast and observed genesis events, classify them, and populate a contingency table with hits, misses, and false alarms. It writes the categorical counts and statistics to the output file(s). The tool can be configured to apply one or more sets of filtering criteria in a single run. The following sections describe the usage statement, required arguments, and optional arguments for tc_gen. +This section describes how to configure and run the TC-Gen tool. The following sections describe the usage statement, required arguments, and optional arguments for tc_gen. tc_gen usage ~~~~~~~~~~~~ @@ -38,7 +44,9 @@ The usage statement for tc_gen is shown below: .. code-block:: none Usage: tc_gen - -genesis source and/or -edeck source + -genesis source + -edeck source + -shape source -track source -config file [-out base] @@ -52,20 +60,24 @@ Required arguments for tc_gen 1. The **-genesis source** argument is the path to one or more ATCF or fort.66 (see documentation listed below) files generated by the Geophysical Fluid Dynamics Laboratory (GFDL) Vortex Tracker when run in tcgen mode or an ASCII file list or a top-level directory containing them. The required file format is described in the "Output formats" section of the `GFDL Vortex Tracker users guide. `_ -2. The **-edeck source** argument is the path to one or more ATCF edeck files, an ASCII file list containing them, or a top-level directory with files matching the regular expression ".dat". The probability of genesis are read from each edeck input file and verified against at the **-track** data. The **-genesis** or **-edeck** option must be used at least once. +2. The **-edeck source** argument is the path to one or more ATCF edeck files, an ASCII file list containing them, or a top-level directory with files matching the regular expression ".dat". The probability of genesis are read from each edeck input file and verified against at the **-track** data. -3. The **-track source** argument is one or more ATCF reference track files or an ASCII file list or top-level directory containing them, with files ending in ".dat". This tool processes either Best track data from bdeck files, or operational track data (e.g. CARQ) from adeck files, or both. Providing both bdeck and adeck files will result in a richer dataset to match with the **-genesis** files. Both adeck and bdeck data should be provided using the **-track** option. The **-track** option must be used at least once. +3. The **-shape source** argument is the path to one or more NHC genesis warning area shapefiles, an ASCII file list containing them, or a top-level directory with files matching the regular expression "gtwo_areas.*.shp". The genesis warning areas and corresponding 2, 5, and 7 day probability values area verified against the **-track** data. -4. The **-config** file argument indicates the name of the configuration file to be used. The contents of the configuration file are discussed below. +Note: The **-genesis**, **-edeck**, or **-shape** options must be used at least once. + +4. The **-track source** argument is one or more ATCF reference track files or an ASCII file list or top-level directory containing them, with files ending in ".dat". This tool processes either Best track data from bdeck files, or operational track data (e.g. CARQ) from adeck files, or both. Providing both bdeck and adeck files will result in a richer dataset to match with the **-genesis** files. Both adeck and bdeck data should be provided using the **-track** option. The **-track** option must be used at least once. + +5. The **-config** file argument indicates the name of the configuration file to be used. The contents of the configuration file are discussed below. Optional arguments for tc_gen ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -5. The **-out base** argument indicates the path of the output file base. This argument overrides the default output file base (./tc_gen) +6. The **-out base** argument indicates the path of the output file base. This argument overrides the default output file base (./tc_gen) -6. The **-log file** option directs output and errors to the specified log file. All messages will be written to that file as well as standard out and error. Thus, users can save the messages without having to redirect the output on the command line. The default behavior is no log file. +7. The **-log file** option directs output and errors to the specified log file. All messages will be written to that file as well as standard out and error. Thus, users can save the messages without having to redirect the output on the command line. The default behavior is no log file. -7. The **-v level** option indicates the desired level of verbosity. The contents of "level" will override the default setting of 2. Setting the verbosity to 0 will make the tool run with no log messages, while increasing the verbosity above 1 will increase the amount of logging. +8. The **-v level** option indicates the desired level of verbosity. The contents of "level" will override the default setting of 2. Setting the verbosity to 0 will make the tool run with no log messages, while increasing the verbosity above 1 will increase the amount of logging. Scoring Logic ^^^^^^^^^^^^^ @@ -120,6 +132,29 @@ The TC-Gen tool implements the following logic: * Report the Nx2 probabilistic contingency table counts and statistics for each forecast model, lead time, and configuration file filter. These counts and statistics are identified in the output files as *PROB_GENESIS*. +* For **-shape** inputs: + + * For each input shapefile, parse the timestamp from the "gtwo_areas_YYYYMMDDHHMM.shp" naming convention, and error out otherwise. Round the timestamp to the nearest synoptic time (e.g. 00, 06, 12, 18) and store that as the issuance time. + + * Open the shapefile and corresponding database file. Process each record. + + * For each record, extract the shape and metadata which defines the basin and 2, 5, and 7 day probabilities. + + * Check if this shape is a duplicate that has already been processed. If it is an exact duplicate, with the same basin, file timestamp, issue time, and min/max lat/lon values, ignore it. If the file timestamp is older than the existing shape, also ignore it. If the file timestamp is newer than the existing shape, replace the existing shape with the new one. + + * Loop over the filters defined in the configuration file and apply the following logic for each. + + * Subset the list of genesis shapes based on the current filter criteria. + + * Search the Best track genesis events to see if any occurred inside the shape within 7 days of the issuance time. If multiple genesis events occurred, choose the one closest to the issuance time. + + * If not found, score each probability as a miss. + + * If found, further check the 2 and 5 day time windows to classify each probability as a hit or miss. + + * Add each probability pair to an Nx2 probabilistic contingency table, tracking results separately for each lead time. + + * Report the Nx2 probabilistic contingency table counts and statistics for each lead time. These counts and statistics are identified in the output files as *GENESIS_SHAPE*. tc_gen configuration file ~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -239,6 +274,8 @@ ______________________ The **init_beg**, **init_end**, **init_inc**, and **init_exc** entries define strings in YYYYMMDD[_HH[MMSS]] format which defines which forecast and operational tracks initializations to be processed. If left empty, all tracks will be used. Otherwise, only those tracks whose initialization time meets all the criteria will be processed. The initialization time must fall between **init_beg**, and **init_end**, must appear in **init_inc** inclusion list, and must not appear in the **init_exc** exclusion list. Note that these settings only apply to the forecast and operational tracks, not the Best tracks, for which the initialization time is undefined. Care should be given when interpreting the contingency table results for filtered data. +For genesis shapes, these options are used to filter the warning issuance time. + ______________________ .. code-block:: none @@ -257,6 +294,8 @@ ______________________ The **init_hour** and **lead** entries are arrays of strings in HH[MMSS] format defining which forecast tracks should be included. If left empty, all tracks will be used. Otherwise, only those forecast tracks whose initialization hour and lead times appear in the list will be used. Note that these settings only apply to the forecast tracks, not the Best tracks, for which the initialization time is undefined. Care should be given when interpreting the contingency table results for filtered data. +For genesis shapes, the **init_hour** option is used to filter the warning issuance hour. + ______________________ .. code-block:: none @@ -265,6 +304,8 @@ ______________________ The **vx_mask** entry is a string defining the path to a Lat/Lon polyline file or a gridded data file that MET can read to subset the results spatially. If specified, only those genesis events whose Lat/Lon location falls within the specified area will be included. +If specified for genesis shapes, the lat/lon of the central location of the shape will be checked. The central location is computed as the average of the min/max lat/lon values of the shape points. + ______________________ .. code-block:: none @@ -275,6 +316,8 @@ The **basin_mask** entry is an array of strings listing tropical cycline basin a The **vx_mask** and **basin_mask** names are concatenated and written to the **VX_MASK** output column. +If **vx_mask** is not specified for genesis shapes and **basin_mask** is, the basin name is extracted from the shapefile metadata and compared to the **basin_mask** list. + ______________________ .. code-block:: none @@ -415,7 +458,7 @@ ______________________ prob_genesis_thresh = ==0.25; -The **prob_genesis_thresh** entry defines the probability thresholds used to create the output Nx2 contingency table when verifying edeck probability of genesis forecasts. The default is probability bins of width 0.25. These probabilities may be specified as a list (>0.00,>0.25,>0.50,>0.75,>1.00) or using shorthand notation (==0.25) for bins of equal width. +The **prob_genesis_thresh** entry defines the probability thresholds used to create the output Nx2 contingency table when verifying edeck probability of genesis forecasts and probabilistic shapefile warning areas. The default is probability bins of width 0.25. These probabilities may be specified as a list (>0.00,>0.25,>0.50,>0.75,>1.00) or using shorthand notation (==0.25) for bins of equal width. ______________________ @@ -435,12 +478,12 @@ ______________________ dland_file = "MET_BASE/tc_data/dland_global_tenth_degree.nc"; version = "VN.N"; -The configuration options listed above are common to many MET tools and are described in :numref:`config_options`. TC-Gen writes output for 2x2 contingency tables to the **FHO**, **CTC**, and **CTS** line types when verifying deterministic genesis forecasts specified using the **-track** command line option. TC-Gen writes output for Nx2 probabilistic contingency tables to the **PCT**, **PSTD**, **PJC**, and **PRC** line types when verifying the probability of genesis forecasts specified using the **-edeck** command line option. Note that the **genmpr** line type is specific to TC-Gen and describes individual genesis matched pairs. +The configuration options listed above are common to many MET tools and are described in :numref:`config_options`. TC-Gen writes output for 2x2 contingency tables to the **FHO**, **CTC**, and **CTS** line types when verifying deterministic genesis forecasts specified using the **-track** command line option. TC-Gen writes output for Nx2 probabilistic contingency tables to the **PCT**, **PSTD**, **PJC**, and **PRC** line types when verifying the probability of genesis forecasts specified using the **-edeck** command line option and probabilistic shapefiles using the **-shape** command line option. Note that the **genmpr** line type is specific to TC-Gen and describes individual genesis matched pairs. tc_gen output ~~~~~~~~~~~~~ -TC-Gen produces output in STAT and, optionally, ASCII and NetCDF formats. The ASCII output duplicates the STAT output but has the data organized by line type. The output files are created based on the **-out** command line argument. The default output base name, **./tc_gen** writes output files in the current working directory named **tc_gen.stat** and, optionally, **tc_gen_fho.txt, tc_gen_ctc.txt**, **tc_gen_cts.txt**, **tc_gen_genmpr.txt**, and **tc_gen_pairs.nc**. The format of the STAT and ASCII output of the TC-Gen tool matches the output of other MET tools with the exception of the genesis matched pair line type. Please refer to the tables in :numref:`point_stat-output` for a description of the common output line types. The genesis matched pair line type and NetCDF output file are described below. +TC-Gen produces output in STAT and, optionally, ASCII and NetCDF formats. The ASCII output duplicates the STAT output but has the data organized by line type. The output files are created based on the **-out** command line argument. The default output base name, **./tc_gen** writes output files in the current working directory named **tc_gen.stat** and, optionally, **tc_gen_pairs.nc** and **tc_gen_{TYPE}.txt** for each of the supported output line types. These output files can easily be redirected to another location using the **-out** command line option. The format of the STAT and ASCII output of the TC-Gen tool matches the output of other MET tools with the exception of the genesis matched pair line type. Please refer to the tables in :numref:`point_stat-output` for a description of the common output line types. The genesis matched pair line type and NetCDF output file are described below. .. _table_TG_header_info_tg_outputs: @@ -483,7 +526,7 @@ TC-Gen produces output in STAT and, optionally, ASCII and NetCDF formats. The AS - Maximum Best track valid time in YYYYMMDD_HHMMSS format * - 10 - FCST_VAR - - Genesis methodology (GENESIS_DEV, GENESIS_OPS, or PROB_GENESIS) + - Genesis methodology (GENESIS_DEV, GENESIS_OPS, PROB_GENESIS, or GENESIS_SHAPE) * - 11 - FCST_UNITS - Does not apply and is set to NA @@ -492,7 +535,7 @@ TC-Gen produces output in STAT and, optionally, ASCII and NetCDF formats. The AS - Does not apply and is set to NA * - 13 - OBS_VAR - - Genesis methodology (GENESIS_DEV, GENESIS_OPS, or PROB_GENESIS) + - Genesis methodology (GENESIS_DEV, GENESIS_OPS, PROB_GENESIS, or GENESIS_SHAPE) * - 14 - OBS_UNITS - Does not apply and is set to NA diff --git a/met/src/basic/vx_config/Makefile.am b/met/src/basic/vx_config/Makefile.am index bce95fa29c..b4d2d52ce6 100644 --- a/met/src/basic/vx_config/Makefile.am +++ b/met/src/basic/vx_config/Makefile.am @@ -11,7 +11,6 @@ include ${top_srcdir}/Make-include # Yacc/lex flags AM_YFLAGS = --defines=config.tab.h -p config -#AM_LFLAGS = --prefix=config --outfile=lex.yy.c # The library diff --git a/met/src/libcode/vx_color/Makefile.am b/met/src/libcode/vx_color/Makefile.am index 0035825c40..8c147693d7 100644 --- a/met/src/libcode/vx_color/Makefile.am +++ b/met/src/libcode/vx_color/Makefile.am @@ -11,7 +11,6 @@ include ${top_srcdir}/Make-include # YACC/LEX flags AM_YFLAGS = --defines=color_parser_yacc.h -p color -#AM_LFLAGS = --prefix=color --outfile=lex.yy.c # The library diff --git a/met/src/libcode/vx_gis/Makefile.am b/met/src/libcode/vx_gis/Makefile.am index fcc506f49f..f214481071 100644 --- a/met/src/libcode/vx_gis/Makefile.am +++ b/met/src/libcode/vx_gis/Makefile.am @@ -12,24 +12,16 @@ include ${top_srcdir}/Make-include noinst_LIBRARIES = libvx_gis.a -libvx_gis_a_SOURCES = shapetype_to_string.h \ - shapetype_to_string.cc \ - dbf_file.cc \ - dbf_file.h \ - shp_array.h \ - shp_file.cc \ - shp_file.h \ - shp_point_record.cc \ - shp_point_record.h \ - shp_poly_record.cc \ - shp_poly_record.h \ - shp_types.h \ - shx_file.cc \ - shx_file.h - - - - +libvx_gis_a_SOURCES = \ + shapetype_to_string.cc shapetype_to_string.h \ + dbf_file.cc dbf_file.h \ + shx_file.cc shx_file.h \ + shp_file.cc shp_file.h \ + shp_types.h \ + shp_array.h \ + shp_point_record.cc shp_point_record.h \ + shp_poly_record.cc shp_poly_record.h \ + vx_gis.h libvx_gis_a_CPPFLAGS = ${MET_CPPFLAGS} # If we are in development mode, generate the "to_string" files and diff --git a/met/src/libcode/vx_gis/dbf_file.cc b/met/src/libcode/vx_gis/dbf_file.cc index eb0fdc921f..0a5049d150 100644 --- a/met/src/libcode/vx_gis/dbf_file.cc +++ b/met/src/libcode/vx_gis/dbf_file.cc @@ -1,4 +1,3 @@ - // *=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=* // ** Copyright UCAR (c) 1992 - 2021 // ** University Corporation for Atmospheric Research (UCAR) @@ -9,7 +8,6 @@ //////////////////////////////////////////////////////////////////////// - using namespace std; #include @@ -311,15 +309,12 @@ unsigned char buf[32]; // seek to the first record // -// pos = pos_first_record; - pos = 32; -if ( lseek(fd, pos, SEEK_SET) < 0 ) { +if ( ::lseek(fd, pos, SEEK_SET) < 0 ) { - mlog << Error - << "\n\n DbfHeader::set_subrecords(int) -> lseek error ... " - << strerror(errno) << "\n\n"; + mlog << Error << "\nDbfHeader::set_subrecords(int) -> " + << "lseek error ... " << strerror(errno) << "\n\n"; exit ( 1 ); @@ -339,8 +334,8 @@ for (j=0; j read error ... n_read = " << n_read << "\n\n"; + mlog << Error << "\nDbfHeader::set_subrecords(int) -> " + << "read error ... n_read = " << n_read << "\n\n"; exit ( 1 ); @@ -372,8 +367,8 @@ DbfSubRecord * DbfHeader::lookup_subrec(const char * text) const if ( empty(text) ) { - mlog << Error - << "\n\n DbfHeader::set_subrecords(const char *) -> empty string!\n\n"; + mlog << Error << "\nDbfHeader::set_subrecords(const char *) -> " + << "empty string!\n\n"; exit ( 1 ); @@ -552,7 +547,6 @@ out << prefix << "field_flags = " << ((int) field_flags) << '\n'; out << prefix << "autoinc_next_value = " << autoinc_next_value << '\n'; out << prefix << "autoinc_step_value = " << autoinc_step_value << '\n'; - // // done // @@ -589,9 +583,6 @@ memcpy(&autoinc_next_value, buf + 19, 4); autoinc_step_value = (int) (buf[23]); - - - // // done // @@ -651,7 +642,6 @@ for (j=0; j<(h.n_subrecs); ++j) { } - // for (k=0; k " + << "should never be called!\n\n"; + +exit ( 1 ); + +} + + +//////////////////////////////////////////////////////////////////////// + + +DbfFile & DbfFile::operator=(const DbfFile &) + +{ + +mlog << Error << "\nDbfFile::operator=(const DbfFile &) -> " + << "should never be called!\n\n"; + +exit ( 1 ); + +return ( * this ); + +} + + +//////////////////////////////////////////////////////////////////////// + + +void DbfFile::init_from_scratch() + +{ + +fd = -1; + +close(); + +return; + +} + + +//////////////////////////////////////////////////////////////////////// + + +void DbfFile::close() + +{ + +if ( fd >= 0 ) ::close(fd); + +fd = -1; + +memset(&Header, 0, sizeof(Header)); + +Filename.clear(); + +At_Eof = false; + + +return; + +} + + +//////////////////////////////////////////////////////////////////////// + + +const char * DbfFile::filename() const + +{ + +if ( Filename.empty() ) return ( (const char *) 0 ); + +return ( Filename.text() ); + +} + + +//////////////////////////////////////////////////////////////////////// + + +bool DbfFile::open(const char * path) + +{ + +size_t n_read, bytes; +int j; + +close(); + +if ( (fd = met_open(path, O_RDONLY)) < 0 ) { + + fd = -1; + + return ( false ); + +} + +Filename = path; + + // + // read the file header + // + +const size_t buf_size = 65536; +unsigned char buf[buf_size]; + +bytes = 32; + +if((n_read = ::read(fd, buf, bytes)) != bytes) { + mlog << Warning << "\nDbfFile::open(const char * path) -> " + << "trouble reading file header from dbf file \"" + << path << "\"\n\n"; + close(); + + return(false); +} + +Header.set_header(buf); + + // + // subrecords + // + +Header.set_subrecords(fd); + + // + // done + // + +return ( true ); + +} + +//////////////////////////////////////////////////////////////////////// + + +int DbfFile::position() const + +{ + +if ( ! is_open() ) { + + mlog << Error << "\nDbfFile::position() -> " + << "no file open!\n\n"; + + exit ( 1 ); + +} + +const int pos = ::lseek(fd, 0, SEEK_CUR); + +if ( pos < 0 ) { + + mlog << Error << "\nDbfFile::position() const -> " + << "lseek error ... " << strerror(errno) << "\n\n"; + + exit ( 1 ); + +} + + +return ( pos ); + +} + + +//////////////////////////////////////////////////////////////////////// + + +void DbfFile::lseek(int offset, int whence) + +{ + + +if ( ! is_open() ) { + + mlog << Error << "\nDbfFile::lseek() -> " + << "no file open!\n\n"; + + exit ( 1 ); + +} + + +if ( ::lseek(fd, offset, whence) < 0 ) { + + mlog << Error << "\nDbfFile::lseek() -> " + << "lseek(2) failed! ... " << strerror(errno) << "\n\n"; + + exit ( 1 ); + +} + +return; + +} + + +//////////////////////////////////////////////////////////////////////// + + +bool DbfFile::read(unsigned char * buf, int n_bytes) + +{ + +if ( ! is_open() ) { + + mlog << Error << "\nDbfFile::read() -> " + << "no file open!\n\n"; + + exit ( 1 ); + +} + +int n_read; + +n_read = ::read(fd, buf, n_bytes); + +if ( n_read < 0 ) { // some kind of error + + mlog << Error << "\nDbfFile::read() -> " + << "read error on dbf file \"" << Filename << "\"\n\n"; + + exit ( 1 ); + +} + +if ( n_read == 0 ) return ( false ); + +if ( n_read == n_bytes ) return ( true ); + +return ( false ); // gotta return something + +} + + +//////////////////////////////////////////////////////////////////////// + + +StringArray DbfFile::subrecord_names() + +{ + +StringArray sa; + + // + // subrecords (if any) + // + +if ( Header.subrec ) { + + for (int j=0; j (Header.n_records) ) { + + mlog << Error << "\nDbfFile::subrecord_values(int i_rec) -> " + << "requested record index (" << i_rec << ") out of range (0," + << Header.n_records << ")!\n\n"; + + exit ( 1 ); +} + +bytes = Header.record_length; + +if ( ::lseek(fd, Header.pos_first_record + i_rec * bytes, SEEK_SET) < 0 ) { + + mlog << Error << "\nDbfFile::subrecord_values(int i_rec) -> " + << "lseek error\n\n"; + + exit ( 1 ); + +} + + // + // read the requested record into the buffer + // + +n_read = ::read(fd, buf, bytes < buf_size ? bytes : buf_size); + +if ( n_read != bytes ) { + + mlog << Error << "\nDbfFile::subrecord_values(int i_rec) -> " + << "read error ... n_read = " << n_read << "\n\n"; + + exit ( 1 ); + +} + +if ( Header.record_length < buf_size) buf[Header.record_length] = 0; + +std::string s = (const char *) buf+1; // skip first byte + + // + // parse each subrecord value + // + +for (j=0,pos=0; j<(Header.n_subrecs); ++j) { + + cs = s.substr(pos, Header.subrec[j].field_length); + cs.ws_strip(); + sa.add(cs); + + pos += Header.subrec[j].field_length; + +} + +return ( sa ); + +} + + +//////////////////////////////////////////////////////////////////////// diff --git a/met/src/libcode/vx_gis/dbf_file.h b/met/src/libcode/vx_gis/dbf_file.h index 6bc2a354f4..7e60e681e5 100644 --- a/met/src/libcode/vx_gis/dbf_file.h +++ b/met/src/libcode/vx_gis/dbf_file.h @@ -6,17 +6,13 @@ // ** P.O.Box 3000, Boulder, Colorado, 80307-3000, USA // *=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=* - //////////////////////////////////////////////////////////////////////// - #ifndef __VX_SHAPEFILES_DBF_FILE_H__ #define __VX_SHAPEFILES_DBF_FILE_H__ - //////////////////////////////////////////////////////////////////////// - // // Got info on the dbf file format at // @@ -27,16 +23,12 @@ // http://web.archive.org/web/20150323061445/http://ulisse.elettra.trieste.it/services/doc/dbase/DBFstruct.htm // - //////////////////////////////////////////////////////////////////////// - struct DbfSubRecord; // forward reference - //////////////////////////////////////////////////////////////////////// - class DbfHeader { private: @@ -85,10 +77,8 @@ class DbfHeader { }; - //////////////////////////////////////////////////////////////////////// - class DbfSubRecord { private: @@ -133,19 +123,83 @@ class DbfSubRecord { }; +//////////////////////////////////////////////////////////////////////// + +void dump_record(ostream &, const int depth, const unsigned char * buf, const DbfHeader &); //////////////////////////////////////////////////////////////////////// +class DbfFile { -void dump_record(ostream &, const int depth, const unsigned char * buf, const DbfHeader &); + private: + + DbfFile(const DbfFile &); + DbfFile & operator=(const DbfFile &); + + protected: + + void init_from_scratch(); + + int fd; + + bool At_Eof; + + DbfHeader Header; + + ConcatString Filename; + + public: + + DbfFile(); + ~DbfFile(); + + bool open(const char * path); + + void close(); + + // + // set stuff + // + + // + // get stuff + // + + const DbfHeader * header() const; + + const char * filename() const; + + bool at_eof() const; + + int position() const; // offset in bytes from beginning of file + + bool is_open() const; + + // + // do stuff + // + + void lseek(int offset, int whence = SEEK_SET); // just like lseek(2) + + bool read(unsigned char * buf, int nbytes); + + StringArray subrecord_names(); + + StringArray subrecord_values(int i_rec); + +}; //////////////////////////////////////////////////////////////////////// +inline const DbfHeader * DbfFile::header() const { return ( &Header ); } -#endif /* __VX_SHAPEFILES_DBF_FILE_H__ */ +inline bool DbfFile::at_eof() const { return ( At_Eof ); } +inline bool DbfFile::is_open() const { return ( fd >= 0 ); } //////////////////////////////////////////////////////////////////////// +#endif /* __VX_SHAPEFILES_DBF_FILE_H__ */ +//////////////////////////////////////////////////////////////////////// diff --git a/met/src/libcode/vx_gis/shp_array.h b/met/src/libcode/vx_gis/shp_array.h index 419d1ca5bf..bfeeba578b 100644 --- a/met/src/libcode/vx_gis/shp_array.h +++ b/met/src/libcode/vx_gis/shp_array.h @@ -97,6 +97,8 @@ class Shp_Array { int n() const { return ( Nelements ); } + int n_elements() const { return ( Nelements ); } + // // do stuff // diff --git a/met/src/libcode/vx_gis/shp_file.cc b/met/src/libcode/vx_gis/shp_file.cc index 14417c0048..1f311fce2a 100644 --- a/met/src/libcode/vx_gis/shp_file.cc +++ b/met/src/libcode/vx_gis/shp_file.cc @@ -1,8 +1,3 @@ - - -//////////////////////////////////////////////////////////////////////// - - // *=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=* // ** Copyright UCAR (c) 1992 - 2021 // ** University Corporation for Atmospheric Research (UCAR) diff --git a/met/src/libcode/vx_gis/shp_file.h b/met/src/libcode/vx_gis/shp_file.h index 77b2c48418..7a362d9e52 100644 --- a/met/src/libcode/vx_gis/shp_file.h +++ b/met/src/libcode/vx_gis/shp_file.h @@ -1,8 +1,3 @@ - - -//////////////////////////////////////////////////////////////////////// - - // *=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=* // ** Copyright UCAR (c) 1992 - 2021 // ** University Corporation for Atmospheric Research (UCAR) diff --git a/met/src/libcode/vx_gis/shp_poly_record.cc b/met/src/libcode/vx_gis/shp_poly_record.cc index b540b78779..b5876253ab 100644 --- a/met/src/libcode/vx_gis/shp_poly_record.cc +++ b/met/src/libcode/vx_gis/shp_poly_record.cc @@ -50,6 +50,27 @@ static SmartBuffer Buf; //////////////////////////////////////////////////////////////////////// +void ShpPolyRecord::clear() + +{ + + memset(&rh, 0, sizeof(rh)); + memset(&bbox, 0, sizeof(bbox)); + + shape_type = bad_data_int; + + n_parts = 0; + n_points = 0; + + parts.clear(); + points.clear(); + +} + + +//////////////////////////////////////////////////////////////////////// + + void ShpPolyRecord::set(unsigned char * buf) { diff --git a/met/src/libcode/vx_gis/shp_poly_record.h b/met/src/libcode/vx_gis/shp_poly_record.h index faad26478d..18131cadaa 100644 --- a/met/src/libcode/vx_gis/shp_poly_record.h +++ b/met/src/libcode/vx_gis/shp_poly_record.h @@ -30,13 +30,6 @@ #include "shp_array.h" -//////////////////////////////////////////////////////////////////////// - - -// static const int max_shp_poly_parts = 600; -// static const int max_shp_poly_points = (1 << 17); - - //////////////////////////////////////////////////////////////////////// @@ -63,6 +56,8 @@ struct ShpPolyRecord { // this should really be a class, not a struct // + void clear(); + double x_min() const; double x_max() const; @@ -81,7 +76,7 @@ struct ShpPolyRecord { // this should really be a class, not a struct bool is_closed() const; - void toggle_longitudes(); + void toggle_longitudes(); }; diff --git a/met/src/libcode/vx_gis/vx_gis.h b/met/src/libcode/vx_gis/vx_gis.h new file mode 100644 index 0000000000..40ade039ec --- /dev/null +++ b/met/src/libcode/vx_gis/vx_gis.h @@ -0,0 +1,29 @@ +// *=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=* +// ** Copyright UCAR (c) 1992 - 2021 +// ** University Corporation for Atmospheric Research (UCAR) +// ** National Center for Atmospheric Research (NCAR) +// ** Research Applications Lab (RAL) +// ** P.O.Box 3000, Boulder, Colorado, 80307-3000, USA +// *=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=* + +//////////////////////////////////////////////////////////////////////// + +#ifndef __VX_GIS_H__ +#define __VX_GIS_H__ + +//////////////////////////////////////////////////////////////////////// + +#include "dbf_file.h" +#include "shx_file.h" +#include "shp_file.h" +#include "shp_types.h" +#include "shp_array.h" +#include "shapetype_to_string.h" +#include "shp_point_record.h" +#include "shp_poly_record.h" + +//////////////////////////////////////////////////////////////////////// + +#endif // __VX_GIS_H__ + +//////////////////////////////////////////////////////////////////////// diff --git a/met/src/libcode/vx_statistics/Makefile.am b/met/src/libcode/vx_statistics/Makefile.am index 8d3a183cbd..bf1cd72c74 100644 --- a/met/src/libcode/vx_statistics/Makefile.am +++ b/met/src/libcode/vx_statistics/Makefile.am @@ -13,6 +13,7 @@ include ${top_srcdir}/Make-include noinst_LIBRARIES = libvx_statistics.a libvx_statistics_a_SOURCES = \ apply_mask.cc apply_mask.h \ + grid_closed_poly.cc grid_closed_poly.h \ compute_ci.cc compute_ci.h \ contable.cc contable.h \ contable_stats.cc \ diff --git a/met/src/tools/other/gen_vx_mask/grid_closed_poly.cc b/met/src/libcode/vx_statistics/grid_closed_poly.cc similarity index 53% rename from met/src/tools/other/gen_vx_mask/grid_closed_poly.cc rename to met/src/libcode/vx_statistics/grid_closed_poly.cc index a39c63457a..e010cca11b 100644 --- a/met/src/tools/other/gen_vx_mask/grid_closed_poly.cc +++ b/met/src/libcode/vx_statistics/grid_closed_poly.cc @@ -8,7 +8,7 @@ /////////////////////////////////////////////////////////////////////////////// // -// Filename: polyline.cc +// Filename: grid_closed_poly.cc // // Description: // @@ -34,257 +34,187 @@ using namespace std; #include "vx_util.h" #include "grid_closed_poly.h" - /////////////////////////////////////////////////////////////////////////////// // // Code for class GridClosedPoly // /////////////////////////////////////////////////////////////////////////////// - -GridClosedPoly::GridClosedPoly() - -{ - -gcp_init_from_scratch(); - +GridClosedPoly::GridClosedPoly() { + gcp_init_from_scratch(); } - /////////////////////////////////////////////////////////////////////////////// - -GridClosedPoly::~GridClosedPoly() - -{ - -} - +GridClosedPoly::~GridClosedPoly() { } /////////////////////////////////////////////////////////////////////////////// +void GridClosedPoly::clear() { -void GridClosedPoly::clear() - -{ + Polyline::clear(); -Polyline::clear(); - -u_min = u_max = v_min = v_max = 0.0; - -return; + u_min = u_max = v_min = v_max = 0.0; + return; } - /////////////////////////////////////////////////////////////////////////////// +GridClosedPoly::GridClosedPoly(const GridClosedPoly & g) { -GridClosedPoly::GridClosedPoly(const GridClosedPoly & g) - -{ - -gcp_init_from_scratch(); - -gcp_assign(g); + gcp_init_from_scratch(); + gcp_assign(g); } - /////////////////////////////////////////////////////////////////////////////// +GridClosedPoly & GridClosedPoly::operator=(const GridClosedPoly & g) { -GridClosedPoly & GridClosedPoly::operator=(const GridClosedPoly & g) - -{ + if(this == &g) return(*this); -if ( this == &g ) return ( * this ); - -gcp_assign(g); - - -return ( * this ); + gcp_assign(g); + return(*this); } - /////////////////////////////////////////////////////////////////////////////// +void GridClosedPoly::gcp_init_from_scratch() { -void GridClosedPoly::gcp_init_from_scratch() - -{ - -u_min = u_max = v_min = v_max = 0.0; - -return; + u_min = u_max = v_min = v_max = 0.0; + return; } - /////////////////////////////////////////////////////////////////////////////// +void GridClosedPoly::gcp_assign(const GridClosedPoly & g) { -void GridClosedPoly::gcp_assign(const GridClosedPoly & g) + Polyline::assign(g); -{ + u_min = g.u_min; + u_max = g.u_max; -Polyline::assign(g); - -u_min = g.u_min; -u_max = g.u_max; - -v_min = g.v_min; -v_max = g.v_max; - -return; + v_min = g.v_min; + v_max = g.v_max; + return; } - /////////////////////////////////////////////////////////////////////////////// +void GridClosedPoly::add_point(double uu, double vv) { -void GridClosedPoly::add_point(double uu, double vv) + if(n_points == 0) { + u_min = u_max = uu; + v_min = v_max = vv; + } + else { + if(uu < u_min) u_min = uu; + if(uu > u_max) u_max = uu; -{ + if(vv < v_min) v_min = vv; + if(vv > v_max) v_max = vv; + } -if ( n_points == 0 ) { - - u_min = u_max = uu; v_min = v_max = vv; - -} else { - - if ( uu < u_min ) u_min = uu; - if ( uu > u_max ) u_max = uu; - - if ( vv < v_min ) v_min = vv; - if ( vv > v_max ) v_max = vv; - -} - -Polyline::add_point(uu, vv); - - -return; + Polyline::add_point(uu, vv); + return; } - /////////////////////////////////////////////////////////////////////////////// -int GridClosedPoly::is_inside(double u_test, double v_test) const - -{ +int GridClosedPoly::is_inside(double u_test, double v_test) const { // // test bounding box // -if ( u_test < u_min ) return ( 0 ); -if ( u_test > u_max ) return ( 0 ); + if(u_test < u_min) return(0); + if(u_test > u_max) return(0); -if ( v_test < v_min ) return ( 0 ); -if ( v_test > v_max ) return ( 0 ); + if(v_test < v_min) return(0); + if(v_test > v_max) return(0); // // test polyline // -const int status = Polyline::is_inside(u_test, v_test); - -return ( status ); + const int status = Polyline::is_inside(u_test, v_test); + return(status); } - /////////////////////////////////////////////////////////////////////////////// // // Code for class GridClosedPolyArray // /////////////////////////////////////////////////////////////////////////////// +bool GridClosedPolyArray::is_inside(double u_test, double v_test) const { -bool GridClosedPolyArray::is_inside(double u_test, double v_test) const - -{ + if(Nelements == 0) return(false); -if ( Nelements == 0 ) return ( false ); - -int j, status; + int j, status; // - // if the test point is inside **ANY** of the element polylines, return true + // return true if the test point is inside **ANY** of the polylines // + for(j=0; jis_inside(u_test, v_test); + status = e[j]->is_inside(u_test, v_test); - if ( status != 0 ) return ( true ); - -} + if(status != 0) return(true); + } - -return ( false ); + return(false); } - /////////////////////////////////////////////////////////////////////////////// +void GridClosedPolyArray::set(const ShpPolyRecord &r, const Grid &grid) { -void GridClosedPolyArray::set(const ShpPolyRecord & r, const Grid & grid) - -{ - -clear(); - -int i, j, k, m; -int start, stop; -double lat, lon; -double x, y; -GridClosedPoly g; - + clear(); -if ( r.n_parts == 0 ) return; + int i, j, k, m; + int start, stop; + double lat, lon; + double x, y; + GridClosedPoly p; + for(j=0; j<(r.n_parts); j++) { -for (j=0; j<(r.n_parts); ++j) { + p.clear(); - g.clear(); + start = r.start_index(j); + stop = r.stop_index(j); - start = r.start_index(j); + i = stop - start + 1; - stop = r.stop_index(j); + for(k=0; k #include "polyline.h" @@ -33,13 +32,9 @@ #include "shp_poly_record.h" #include "vx_grid.h" - /////////////////////////////////////////////////////////////////////////////// - -class GridClosedPoly : public Polyline - -{ +class GridClosedPoly : public Polyline { protected: @@ -47,8 +42,7 @@ class GridClosedPoly : public Polyline void gcp_assign(const GridClosedPoly &); - // bounding box info - + // bounding box limits double u_min, u_max; double v_min, v_max; @@ -61,36 +55,28 @@ class GridClosedPoly : public Polyline void clear(); - int is_inside(double u_test, double v_test) const; // test bounding box first + // tests bounding box first for efficiency + int is_inside(double u_test, double v_test) const; - void add_point(double, double); // updates bounding box + // updates bounding box for each new point + void add_point(double, double); }; - /////////////////////////////////////////////////////////////////////////////// - -class GridClosedPolyArray : public NCRR_Array - -{ +class GridClosedPolyArray : public NCRR_Array { public: bool is_inside(double u_test, double v_test) const; - void set (const ShpPolyRecord &, const Grid &); + void set(const ShpPolyRecord &, const Grid &); }; - /////////////////////////////////////////////////////////////////////////////// - -#endif // __DATA2D_UTIL_GCP_H__ - +#endif // __GRID_CLOSED_POLY_H__ /////////////////////////////////////////////////////////////////////////////// - - - diff --git a/met/src/libcode/vx_statistics/vx_statistics.h b/met/src/libcode/vx_statistics/vx_statistics.h index db1d6685d1..11c06f68e8 100644 --- a/met/src/libcode/vx_statistics/vx_statistics.h +++ b/met/src/libcode/vx_statistics/vx_statistics.h @@ -19,6 +19,7 @@ #include "apply_mask.h" +#include "grid_closed_poly.h" #include "compute_ci.h" #include "contable.h" #include "met_stats.h" diff --git a/met/src/libcode/vx_tc_util/Makefile.am b/met/src/libcode/vx_tc_util/Makefile.am index 733ef9cd6b..390eb1e5ae 100644 --- a/met/src/libcode/vx_tc_util/Makefile.am +++ b/met/src/libcode/vx_tc_util/Makefile.am @@ -24,6 +24,7 @@ libvx_tc_util_a_SOURCES = \ prob_gen_info.h prob_gen_info.cc \ prob_rirw_pair_info.h prob_rirw_pair_info.cc \ genesis_info.cc genesis_info.h \ + gen_shape_info.cc gen_shape_info.h \ pair_data_genesis.cc pair_data_genesis.h \ tc_hdr_columns.cc tc_hdr_columns.h \ tc_columns.cc tc_columns.h \ diff --git a/met/src/libcode/vx_tc_util/gen_shape_info.cc b/met/src/libcode/vx_tc_util/gen_shape_info.cc new file mode 100644 index 0000000000..f51b92c4e7 --- /dev/null +++ b/met/src/libcode/vx_tc_util/gen_shape_info.cc @@ -0,0 +1,377 @@ +// *=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=* +// ** Copyright UCAR (c) 1992 - 2021 +// ** University Corporation for Atmospheric Research (UCAR) +// ** National Center for Atmospheric Research (NCAR) +// ** Research Applications Lab (RAL) +// ** P.O.Box 3000, Boulder, Colorado, 80307-3000, USA +// *=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=* + +//////////////////////////////////////////////////////////////////////// + +using namespace std; + +#include +#include +#include +#include +#include +#include + +#include "gen_shape_info.h" + +//////////////////////////////////////////////////////////////////////// +// +// Code for class GenShapeInfo +// +//////////////////////////////////////////////////////////////////////// + +GenShapeInfo::GenShapeInfo() { + clear(); +} + +//////////////////////////////////////////////////////////////////////// + +GenShapeInfo::~GenShapeInfo() { + clear(); +} + +//////////////////////////////////////////////////////////////////////// + +GenShapeInfo::GenShapeInfo(const GenShapeInfo &g) { + clear(); + assign(g); +} + +//////////////////////////////////////////////////////////////////////// + +GenShapeInfo & GenShapeInfo::operator=(const GenShapeInfo &g) { + + if(this == &g) return(*this); + + assign(g); + + return(*this); +} + +//////////////////////////////////////////////////////////////////////// + +void GenShapeInfo::clear() { + + Basin.clear(); + + FileTime = (unixtime) 0; + IssueTime = (unixtime) 0; + + Poly.clear(); + LeadSec.clear(); + ProbVal.clear(); + + return; +} + +//////////////////////////////////////////////////////////////////////// + +ConcatString GenShapeInfo::serialize() const { + ConcatString s; + + s << "GenShapeInfo: " + << "Basin = \"" << Basin << "\"" + << ", FileTime = \"" << (FileTime > 0 ? + unix_to_yyyymmdd_hhmmss(FileTime).text() : na_str) << "\"" + << ", IssueTime = \"" << (IssueTime > 0 ? + unix_to_yyyymmdd_hhmmss(IssueTime).text() : na_str) << "\"" + << ", NPoints = " << Poly.n_points + << ", Lat = " << Poly.y_min() << " to " << Poly.y_max() + << ", Lon = " << Poly.x_min() << " to " << Poly.x_max(); + + return(s); + +} + +//////////////////////////////////////////////////////////////////////// + +void GenShapeInfo::assign(const GenShapeInfo &gsi) { + + clear(); + + Basin = gsi.Basin; + + FileTime = gsi.FileTime; + IssueTime = gsi.IssueTime; + + Poly = gsi.Poly; + LeadSec = gsi.LeadSec; + ProbVal = gsi.ProbVal; + + return; +} + +//////////////////////////////////////////////////////////////////////// + +void GenShapeInfo::set_basin(const char *s) { + Basin = s; + return; +} + +//////////////////////////////////////////////////////////////////////// + +void GenShapeInfo::set_time(unixtime file_ut) { + + FileTime = file_ut; + + // Round the file timestamp to the nearest synoptic time (00, 06, 12, 18) + int h3 = 3*sec_per_hour; + int h6 = 6*sec_per_hour; + IssueTime = ( (int) (file_ut + h3) / h6 ) * h6; + + return; +} + +//////////////////////////////////////////////////////////////////////// + +void GenShapeInfo::set_poly(const ShpPolyRecord &rec) { + Poly = rec; + return; +} + +//////////////////////////////////////////////////////////////////////// + +void GenShapeInfo::add_prob(int sec, double prob) { + + // Check range of values + if(sec < 0 || prob < 0.0 || prob > 1.0) { + mlog << Error << "\nGenShapeInfo::add_prob() -> " + << "unexpected lead time (" << sec + << ") or probability value (" << prob + << ")!\n\n"; + exit(1); + } + + LeadSec.add(sec); + ProbVal.add(prob); + + return; +} + +//////////////////////////////////////////////////////////////////////// + +const ShpPolyRecord &GenShapeInfo::poly() const { + return(Poly); +} + +//////////////////////////////////////////////////////////////////////// + +double GenShapeInfo::center_lat() const { + return((Poly.y_min() + Poly.y_max())/2.0); +} + +//////////////////////////////////////////////////////////////////////// + +double GenShapeInfo::center_lon() const { + return((Poly.x_min() + Poly.x_max())/2.0); +} + +//////////////////////////////////////////////////////////////////////// + +int GenShapeInfo::n_prob() const { + return(ProbVal.n()); +} + +//////////////////////////////////////////////////////////////////////// + +int GenShapeInfo::lead_sec(int i) const { + + // Check range of values + if(i < 0 || i > LeadSec.n()) { + mlog << Error << "\nGenShapeInfo::lead_sec(int) -> " + << "range check error (" << i << ")!\n\n"; + exit(1); + } + + return(LeadSec[i]); +} + +//////////////////////////////////////////////////////////////////////// + +double GenShapeInfo::prob_val(int i) const { + + // Check range of values + if(i < 0 || i > ProbVal.n()) { + mlog << Error << "\nGenShapeInfo::prob_val(int) -> " + << "range check error (" << i << ")!\n\n"; + exit(1); + } + + return(ProbVal[i]); +} + +//////////////////////////////////////////////////////////////////////// +// +// Code for class GenShapeInfoArray +// +//////////////////////////////////////////////////////////////////////// + +GenShapeInfoArray::GenShapeInfoArray() { + + init_from_scratch(); +} + +//////////////////////////////////////////////////////////////////////// + +GenShapeInfoArray::~GenShapeInfoArray() { + + clear(); +} + +//////////////////////////////////////////////////////////////////////// + +GenShapeInfoArray::GenShapeInfoArray(const GenShapeInfoArray & t) { + + init_from_scratch(); + + assign(t); +} + +//////////////////////////////////////////////////////////////////////// + +GenShapeInfoArray & GenShapeInfoArray::operator=(const GenShapeInfoArray & t) { + + if(this == &t) return(*this); + + assign(t); + + return(*this); +} + +//////////////////////////////////////////////////////////////////////// +// +// Define equality as duplicates with the same file time +// +//////////////////////////////////////////////////////////////////////// + +bool GenShapeInfo::operator==(const GenShapeInfo & gsi) const { + + return(is_duplicate(gsi) && FileTime == gsi.FileTime); +} + + +//////////////////////////////////////////////////////////////////////// +// +// Duplicates have the same basin, issue time, number of points, +// and range of lat/lons +// +//////////////////////////////////////////////////////////////////////// + +bool GenShapeInfo::is_duplicate(const GenShapeInfo & gsi) const { + + return(Basin == gsi.Basin && + IssueTime == gsi.IssueTime && + Poly.n_points == gsi.Poly.n_points && + Poly.x_min() == gsi.Poly.x_min() && + Poly.x_max() == gsi.Poly.x_max() && + Poly.y_min() == gsi.Poly.y_min() && + Poly.y_max() == gsi.Poly.y_max()); +} + + +//////////////////////////////////////////////////////////////////////// + +void GenShapeInfoArray::init_from_scratch() { + + clear(); + + return; +} + +//////////////////////////////////////////////////////////////////////// + +void GenShapeInfoArray::clear() { + + GenShape.clear(); + + return; +} + +//////////////////////////////////////////////////////////////////////// + +void GenShapeInfoArray::assign(const GenShapeInfoArray &ga) { + + clear(); + + for(int i=0; i= GenShape.size())) { + mlog << Error << "\nGenShapeInfoArray::operator[](int) -> " + << "range check error for index value " << n << "\n\n"; + exit(1); + } + + return(GenShape[n]); +} + +//////////////////////////////////////////////////////////////////////// + +bool GenShapeInfoArray::add(const GenShapeInfo &gsi, bool check_dup) { + + // Check for duplicates: + // - Skip exact duplicates + // - Replace older duplicates with newer ones + // - Skip older duplicates + if(check_dup) { + + // Loop over existing shapes + for(int i=0; i +#include + +#include "vx_cal.h" +#include "vx_math.h" +#include "vx_util.h" +#include "vx_gis.h" + +//////////////////////////////////////////////////////////////////////// +// +// GenShapeInfo class stores information about genesis shapefiles. +// +//////////////////////////////////////////////////////////////////////// + +class GenShapeInfo { + + private: + + void assign(const GenShapeInfo &); + + // Shapefile information + ConcatString Basin; + unixtime FileTime; + unixtime IssueTime; + + ShpPolyRecord Poly; + + IntArray LeadSec; + NumArray ProbVal; + + public: + + GenShapeInfo(); + ~GenShapeInfo(); + GenShapeInfo(const GenShapeInfo &); + + GenShapeInfo & operator=(const GenShapeInfo &); + bool operator==(const GenShapeInfo &) const; + bool is_duplicate(const GenShapeInfo &) const; + + void clear(); + ConcatString serialize() const; + + // + // set stuff + // + + void set_basin(const char *); + void set_time(unixtime); + void set_poly(const ShpPolyRecord &); + void add_prob(int, double); + + // + // get stuff + // + + const ConcatString & basin() const; + + unixtime file_time() const; + unixtime issue_time() const; + int issue_hour() const; + + const ShpPolyRecord & poly() const; + + double center_lat() const; + double center_lon() const; + + int n_prob() const; + int lead_sec(int) const; + double prob_val(int) const; + + // + // do stuff + // +}; + +//////////////////////////////////////////////////////////////////////// + +inline const ConcatString & GenShapeInfo::basin() const { return(Basin); } + +inline unixtime GenShapeInfo::file_time() const { return(FileTime); } +inline unixtime GenShapeInfo::issue_time() const { return(IssueTime); } +inline int GenShapeInfo::issue_hour() const { return(unix_to_sec_of_day(IssueTime)); } + +//////////////////////////////////////////////////////////////////////// +// +// GenShapeInfoArray class stores an array of GenShapeInfo objects. +// +//////////////////////////////////////////////////////////////////////// + +class GenShapeInfoArray { + + private: + + void init_from_scratch(); + void assign(const GenShapeInfoArray &); + + vector GenShape; + + public: + + GenShapeInfoArray(); + ~GenShapeInfoArray(); + GenShapeInfoArray(const GenShapeInfoArray &); + GenShapeInfoArray & operator=(const GenShapeInfoArray &); + + void clear(); + + ConcatString serialize() const; + + // + // set stuff + // + + bool add(const GenShapeInfo &, bool check_dup = false); + bool has(const GenShapeInfo &) const; + + // + // get stuff + // + + const GenShapeInfo & operator[](int) const; + int n() const; +}; + +//////////////////////////////////////////////////////////////////////// + +inline int GenShapeInfoArray::n() const { return(GenShape.size()); } + +//////////////////////////////////////////////////////////////////////// + +#endif /* __VX_GEN_SHAPE_INFO_H__ */ + +//////////////////////////////////////////////////////////////////////// diff --git a/met/src/libcode/vx_tc_util/vx_tc_util.h b/met/src/libcode/vx_tc_util/vx_tc_util.h index 76e5448429..89a4434432 100644 --- a/met/src/libcode/vx_tc_util/vx_tc_util.h +++ b/met/src/libcode/vx_tc_util/vx_tc_util.h @@ -24,6 +24,7 @@ #include "prob_rirw_info.h" #include "prob_rirw_pair_info.h" #include "genesis_info.h" +#include "gen_shape_info.h" #include "tc_columns.h" #include "tc_hdr_columns.h" #include "tc_stat_line.h" diff --git a/met/src/tools/core/grid_stat/grid_stat.cc b/met/src/tools/core/grid_stat/grid_stat.cc index 9b2447818a..f420e568c4 100644 --- a/met/src/tools/core/grid_stat/grid_stat.cc +++ b/met/src/tools/core/grid_stat/grid_stat.cc @@ -85,7 +85,7 @@ // output line types. // 034 05/10/16 Halley Gotway Add grid weighting. // 035 05/20/16 Prestopnik J Removed -version (now in command_line.cc) -// 036 02/14/17 Win MET#621 Add nc_pairs_flag.apply_mask +// 036 02/14/17 Win MET #621 Add nc_pairs_flag.apply_mask // 037 05/15/17 Prestopnik P Add shape for regrid, nbrhd and interp // 038 06/26/17 Halley Gotway Add ECLV line type. // 039 08/18/17 Halley Gotway Add fourier decomposition. diff --git a/met/src/tools/core/point_stat/point_stat.cc b/met/src/tools/core/point_stat/point_stat.cc index 7e78611c59..f62f3ae511 100644 --- a/met/src/tools/core/point_stat/point_stat.cc +++ b/met/src/tools/core/point_stat/point_stat.cc @@ -98,7 +98,7 @@ // 047 08/23/21 Seth Linden Add ORANK line type for HiRA. // 048 09/13/21 Seth Linden Changed obs_qty to obs_qty_inc. // Added code for obs_qty_exc. -// 049 12/11/21 Halley Gotway MET#1991 Fix VCNT output. +// 049 12/11/21 Halley Gotway MET #1991 Fix VCNT output. // //////////////////////////////////////////////////////////////////////// diff --git a/met/src/tools/other/gen_vx_mask/Makefile.am b/met/src/tools/other/gen_vx_mask/Makefile.am index f8b773183b..5eb4938ea7 100644 --- a/met/src/tools/other/gen_vx_mask/Makefile.am +++ b/met/src/tools/other/gen_vx_mask/Makefile.am @@ -12,8 +12,7 @@ include ${top_srcdir}/Make-include bin_PROGRAMS = gen_vx_mask -gen_vx_mask_SOURCES = gen_vx_mask.cc \ - grid_closed_poly.cc +gen_vx_mask_SOURCES = gen_vx_mask.cc gen_vx_mask_CPPFLAGS = ${MET_CPPFLAGS} gen_vx_mask_LDFLAGS = ${MET_LDFLAGS} gen_vx_mask_LDADD = -lvx_shapedata \ @@ -51,5 +50,4 @@ gen_vx_mask_LDADD = -lvx_shapedata \ -lvx_log \ -lm -lnetcdf_c++4 -lnetcdf -lgsl -lgslcblas -EXTRA_DIST = gen_vx_mask.h \ - grid_closed_poly.h +EXTRA_DIST = gen_vx_mask.h diff --git a/met/src/tools/other/gen_vx_mask/gen_vx_mask.cc b/met/src/tools/other/gen_vx_mask/gen_vx_mask.cc index 1fd7bb9bc2..ed5ac52802 100644 --- a/met/src/tools/other/gen_vx_mask/gen_vx_mask.cc +++ b/met/src/tools/other/gen_vx_mask/gen_vx_mask.cc @@ -25,8 +25,8 @@ // 007 04/08/19 Halley Gotway Add percentile thresholds. // 008 04/06/20 Halley Gotway Generalize input_grid option. // 009 06/01/21 Seth Linden Change -type from optional to required. -// 010 08/30/21 Halley Gotway MET#1891 Fix input and mask fields. -// 011 12/13/21 Halley Gotway MET#1993 Fix -type grid. +// 010 08/30/21 Halley Gotway MET #1891 Fix input and mask fields. +// 011 12/13/21 Halley Gotway MET #1993 Fix -type grid. // //////////////////////////////////////////////////////////////////////// @@ -549,10 +549,8 @@ void apply_poly_mask(DataPlane & dp) { bool inside; double lat, lon; - n_in = 0; - // Check each grid point being inside the polyline - for(x=0; x 0) { + mlog << Debug(2) + << "Scoring track genesis forecasts.\n"; score_track_genesis(best_ga, oper_ta); } // Score EDECK genesis probabilities and write output if(edeck_source.n() > 0) { + mlog << Debug(2) + << "Scoring probability of genesis forecasts.\n"; score_genesis_prob(best_ga, oper_ta); } + // Score genesis shapefiles and write output + if(shape_source.n() > 0) { + mlog << Debug(2) + << "Scoring genesis shapefiles.\n"; + score_genesis_shape(best_ga); + } + // Finish output files finish_txt_files(); @@ -205,6 +233,7 @@ void process_command_line(int argc, char **argv) { // Add function calls for the arguments cline.add(set_genesis, "-genesis", -1); cline.add(set_edeck, "-edeck", -1); + cline.add(set_shape, "-shape", -1); cline.add(set_track, "-track", -1); cline.add(set_config, "-config", 1); cline.add(set_out, "-out", 1); @@ -224,10 +253,10 @@ void process_command_line(int argc, char **argv) { } // Check for the minimum number of arguments - if(genesis_source.n() == 0 && edeck_source.n() == 0) { + if((genesis_source.n() + edeck_source.n() + shape_source.n()) == 0) { mlog << Error << "\nprocess_command_line(int argc, char **argv) -> " - << "at least one of the \"-genesis\" or \"-edeck\" command " - << "line options are required\n\n"; + << "at least one of the \"-genesis\", \"-edeck\", or \"-shape\" " + << "command line options are required\n\n"; usage(); } @@ -255,6 +284,13 @@ void process_command_line(int argc, char **argv) { << ", Model Suffix: " << edeck_model_suffix[i] << "\n"; } + // List the input shapefiles + for(i=0; igenesis_time() - fcst_sa[i].issue_time()) < + (fcst_sa[i].lead_sec(j)); + + if(is_event) { + case_cs << "MATCHES " + << unix_to_yyyymmdd_hhmmss(bgi->genesis_time()) + << " BEST track " << bgi->storm_id() + << " genesis at (" << bgi->lat() << ", " + << bgi->lon() << ").\n"; + } + else { + case_cs << "has NO MATCH in the BEST track.\n"; + } + } + // No Best track match + else { + case_cs << "has NO MATCH in the BEST track.\n"; + } + + mlog << Debug(5) << case_cs; + + // Store pair info + pgi.add_genshape(fcst_sa[i], j, bgi, is_event); + + } // end for j + } // end for i + + return; +} //////////////////////////////////////////////////////////////////////// @@ -966,6 +1131,48 @@ int find_probgen_match(const ProbGenInfo &prob_gi, return(i_best); } +//////////////////////////////////////////////////////////////////////// + +int find_genshape_match(const GenShapeInfo &gsi, + const GenesisInfoArray &bga) { + int i, i_bga; + double x, y; + unixtime min_ut; + + // Time window + unixtime beg_ut = gsi.issue_time(); + unixtime end_ut = beg_ut + shape_prob_search_sec; + + // Load the shape + GridClosedPolyArray p; + p.set(gsi.poly(), conf_info.DLandGrid); + + // Search Best track genesis events for a match + for(i=0,i_bga=bad_data_int; i end_ut) continue; + + // Second, check the polyline + conf_info.DLandGrid.latlon_to_xy(bga[i].lat(), -1.0*bga[i].lon(), x, y); + if(p.is_inside(x, y)) { + + // First match found + if(is_bad_data(i_bga)) { + i_bga = i; + min_ut = bga[i].genesis_time(); + } + // Better match found + else if(bga[i].genesis_time() < min_ut) { + i_bga = i; + min_ut = bga[i].genesis_time(); + } + } + } // end for i + + return(i_bga); +} //////////////////////////////////////////////////////////////////////// @@ -1358,6 +1565,140 @@ void process_edecks(const StringArray &files, return; } +//////////////////////////////////////////////////////////////////////// + +void process_shapes(const StringArray &files, + GenShapeInfoArray &shapes) { + int i, j, k, n_rec, total_probs; + unixtime file_ut; + ConcatString shp_file_name, dbf_file_name; + StringArray sa; + ConcatString cs; + GenShapeInfo gsi; + DbfFile dbf_file; + ShpFile shp_file; + ShpPolyRecord poly_rec; + + // Initialize + total_probs = 0; + + // Process each shapefile + for(i=0; i " + << "the specified shapefile (" << shp_file_name + << ") and corresponding database file (" << dbf_file_name + << ") must both exists!\n\n"; + } + + // Extract the issue time from the filename: + // gtwo_areas_YYYYMMDDHHMM.shp + sa = shp_file_name.basename().split("_."); + if(sa.n() >= 3 && sa[2].length() >= 12) { + cs << cs_erase + << sa[2].substr(0, 8).c_str() << "_" + << sa[2].substr(8, 4).c_str() << "00"; + file_ut = yyyymmdd_hhmmss_to_unix(cs.c_str()); + } + else { + mlog << Error << "\nprocess_shapes() -> " + << "can't extract the issue time from \"" << shp_file_name + << "\" since it does not match the regular expression \"" + << gen_shp_reg_exp << "\"!\n\n"; + exit(1); + } + + // Open the dbf and shp files + dbf_file.open(dbf_file_name.c_str()); + shp_file.open(shp_file_name.c_str()); + + // Store the number of records + n_rec = dbf_file.header()->n_records; + + // Check expected shape types + const ShapeType shape_type = (ShapeType) (shp_file.header()->shape_type); + if(shape_type != shape_type_polygon) { + mlog << Error << "\nprocess_shapes() -> " + << "shapefile type \"" << shapetype_to_string(shape_type) + << "\" in file \"" << shp_file_name + << "\" is not supported!\n\n"; + exit(1); + } + + mlog << Debug(4) << "[File " << i+1 << " of " << files.n() << "]: " + << "Found " << n_rec << " records in file \"" << shp_file_name << "\".\n"; + + // Process each shape + for(j=0; j " + << "hit shp file EOF before reading all records!\n\n"; + exit(1); + } + + // Read the current shape and metadata + shp_file >> poly_rec; + poly_rec.toggle_longitudes(); + sa = dbf_file.subrecord_values(j); + + // Initialize GenShapeInfo + gsi.clear(); + gsi.set_time(file_ut); + gsi.set_basin(string_to_basin_abbr(sa[0]).c_str()); + gsi.set_poly(poly_rec); + + // Parse probabilities from the subrecord values + for(k=0; k= max_n_shape_prob) { + mlog << Warning << "\nprocess_shapes() -> " + << "unexpected number of shapefile probabilities (" + << gsi.n_prob() << ") in record " << j+1 + << " of file \"" << dbf_file_name + << "\"!\n\n"; + continue; + } + + // Store the probability info + gsi.add_prob(shape_prob_lead_hr[gsi.n_prob()]*sec_per_hour, + atoi(sa[k].c_str())/100.0); + } + } // end for k + + // Store this shape, check for duplicates + if(shapes.add(gsi, true)) { + mlog << Debug(5) << "Add new " << gsi.serialize() << "\n"; + total_probs += gsi.n_prob(); + } + + } // end for j + + // Close files + dbf_file.close(); + shp_file.close(); + + } // end for i + + mlog << Debug(3) << "Found a total of " << total_probs + << " probabilities in " << shapes.n() << " genesis shapes from " + << files.n() << " input files.\n"; + + return; +} + //////////////////////////////////////////////////////////////////////// // // Setup the output ASCII files @@ -1391,7 +1732,7 @@ void setup_txt_files(int n_model, int max_n_prob, int n_pair) { break; // Nx2 probabilistic contingency table output: - // 1 header + 1 vx method * # models * #probs * # filters + // 1 header + 1 vx method * # models * # probs * # filters case(i_pct): case(i_pstd): case(i_pjc): @@ -1818,16 +2159,19 @@ void write_ctc_genmpr_row(StatHdrColumns &shc, //////////////////////////////////////////////////////////////////////// void write_pct_stats(ProbGenPCTInfo &pgi) { + + // Check for no data to write + if(pgi.PCTMap.size() == 0) return; + int i, lead_hr, lead_sec; // Setup header columns shc.set_model(pgi.Model.c_str()); shc.set_desc(pgi.VxOpt->Desc.c_str()); shc.set_obtype(conf_info.BestEventInfo.Technique.c_str()); - shc.set_mask(pgi.VxOpt->VxMaskName.empty() ? - na_str : pgi.VxOpt->VxMaskName.c_str()); - shc.set_fcst_var(prob_genesis_name); - shc.set_obs_var (prob_genesis_name); + shc.set_mask(pgi.VxMask.c_str()); + shc.set_fcst_var(pgi.VarName.c_str()); + shc.set_obs_var (pgi.VarName.c_str()); // Write results for each lead time for(i=0; ioutput_map(stat_pjc) != STATOutputType_None) { - write_pct_row(shc, pgi.PCTMap[lead_hr], + write_pjc_row(shc, pgi.PCTMap[lead_hr], pgi.VxOpt->output_map(stat_pjc), 1, 1, stat_at, i_stat_row, txt_at[i_pjc], i_txt_row[i_pjc]); @@ -1871,22 +2215,19 @@ void write_pct_stats(ProbGenPCTInfo &pgi) { // Write PRC output if(pgi.VxOpt->output_map(stat_pjc) != STATOutputType_None) { - write_pct_row(shc, pgi.PCTMap[lead_hr], - pgi.VxOpt->output_map(stat_pjc), + write_prc_row(shc, pgi.PCTMap[lead_hr], + pgi.VxOpt->output_map(stat_prc), 1, 1, stat_at, i_stat_row, txt_at[i_prc], i_txt_row[i_prc]); } // Write out GENMPR if(pgi.VxOpt->output_map(stat_genmpr) != STATOutputType_None) { - shc.set_fcst_var(prob_genesis_name); - shc.set_obs_var (prob_genesis_name); write_pct_genmpr_row(shc, pgi, lead_hr, pgi.VxOpt->output_map(stat_genmpr), stat_at, i_stat_row, txt_at[i_genmpr], i_txt_row[i_genmpr]); } - } // end for i return; @@ -1912,10 +2253,10 @@ void write_pct_genmpr_row(StatHdrColumns &shc, shc.set_alpha(bad_data_double); // Write a line for each matched pair - for(i=0; i 50 are for testing or invests static const int max_best_cyclone_number = 50; +// 2, 5, and 7 days shapefile probabilities +static const int max_n_shape_prob = 3; +static const int shape_prob_lead_hr[max_n_shape_prob] = { + 48, 120, 168 +}; +static const int shape_prob_search_sec = 168*sec_per_hour; + //////////////////////////////////////////////////////////////////////// // // Variables for Command Line Arguments @@ -95,6 +100,7 @@ static const int max_best_cyclone_number = 50; // Input files static StringArray genesis_source, genesis_model_suffix; static StringArray edeck_source, edeck_model_suffix; +static StringArray shape_source; static StringArray track_source, track_model_suffix; static ConcatString config_file; static TCGenConfInfo conf_info; diff --git a/met/src/tools/tc_utils/tc_gen/tc_gen_conf_info.cc b/met/src/tools/tc_utils/tc_gen/tc_gen_conf_info.cc index 0592c667b4..a39a7fc06c 100644 --- a/met/src/tools/tc_utils/tc_gen/tc_gen_conf_info.cc +++ b/met/src/tools/tc_utils/tc_gen/tc_gen_conf_info.cc @@ -150,6 +150,7 @@ void TCGenVxOpt::clear() { ValidBeg = ValidEnd = (unixtime) 0; InitHour.clear(); Lead.clear(); + VxMaskConf.clear(); VxMaskName.clear(); VxPolyMask.clear(); VxGridMask.clear(); @@ -228,8 +229,9 @@ void TCGenVxOpt::process_config(Dictionary &dict) { } // Conf: vx_mask - if(nonempty(dict.lookup_string(conf_key_vx_mask).c_str())) { - file_name = replace_path(dict.lookup_string(conf_key_vx_mask)); + VxMaskConf = dict.lookup_string(conf_key_vx_mask); + if(VxMaskConf.nonempty()) { + file_name = replace_path(VxMaskConf); mlog << Debug(2) << "Masking File: " << file_name << "\n"; parse_poly_mask(file_name, VxPolyMask, VxGridMask, VxAreaMask, VxMaskName); @@ -359,7 +361,8 @@ void TCGenVxOpt::process_basin_mask(const Grid &basin_grid, if(!basin_abbr.has(VxBasinMask[i], j)) { mlog << Error << "\nTCGenConfInfo::process_basin_mask() -> " << "\"" << VxBasinMask[i] - << "\" is not a valid basin name!\n\n"; + << "\" is not a valid basin name (" + << write_css(basin_abbr) << ")!\n\n"; exit(1); } @@ -570,6 +573,60 @@ bool TCGenVxOpt::is_keeper(const ProbGenInfo &gi) const { //////////////////////////////////////////////////////////////////////// +bool TCGenVxOpt::is_keeper(const GenShapeInfo &gsi) const { + bool keep = true; + + // ATCF ID and storm name do not apply + + // Initialization time + if((InitBeg > 0 && InitBeg > gsi.issue_time()) || + (InitEnd > 0 && InitEnd < gsi.issue_time()) || + (InitInc.n() > 0 && !InitInc.has(gsi.issue_time())) || + (InitExc.n() > 0 && InitExc.has(gsi.issue_time()))) + keep = false; + + // Initialization hours + if(InitHour.n() > 0 && !InitHour.has(gsi.issue_hour())) + keep = false; + + // Lead and valid times: + // GenShapeInfo objects contain multiple lead/valid times. + // Do not filter by them here. + + // If VxMaskConf set, filter spatially by the center of the shape + if(keep && VxMaskConf.nonempty()) { + + // Poly masking: use center lat/lon + if(VxPolyMask.n_points() > 0 && + !VxPolyMask.latlon_is_inside(gsi.center_lat(), gsi.center_lon())) + keep = false; + + // Area masking: use center lat/lon + if(!VxAreaMask.is_empty()) { + double x, y; + VxGridMask.latlon_to_xy(gsi.center_lat(), -1.0*gsi.center_lon(), x, y); + if(x < 0 || x >= VxGridMask.nx() || + y < 0 || y >= VxGridMask.ny()) { + keep = false; + } + else { + keep = VxAreaMask(nint(x), nint(y)); + } + } + } + // Otherwise, if VxBasinMask set, filter by the GenShapeArea basin + else if(keep && VxBasinMask.n() > 0) { + keep = VxBasinMask.has(gsi.basin()); + } + + // Distance to land does not apply + + // Return the keep status + return(keep); +} + +//////////////////////////////////////////////////////////////////////// + STATOutputType TCGenVxOpt::output_map(STATLineType t) const { return(OutputMap.at(t)); } @@ -1242,12 +1299,16 @@ void ProbGenPCTInfo::clear() { DefaultPCT.clear(); Model.clear(); + VarName.clear(); + VxMask.clear(); + InitBeg = InitEnd = (unixtime) 0; BestBeg = BestEnd = (unixtime) 0; PCTMap.clear(); - FcstGenMap.clear(); - FcstIdxMap.clear(); + ProbGenMap.clear(); + GenShapeMap.clear(); + ProbIdxMap.clear(); BestGenMap.clear(); BestEvtMap.clear(); @@ -1276,16 +1337,18 @@ void ProbGenPCTInfo::set_vx_opt(const TCGenVxOpt *vx_opt) { //////////////////////////////////////////////////////////////////////// -void ProbGenPCTInfo::add(const ProbGenInfo &fgi, int index, - const GenesisInfo *bgi, bool is_event) { - int i; +void ProbGenPCTInfo::add_probgen(const ProbGenInfo &pgi, int index, + const GenesisInfo *bgi, bool is_event) { unixtime ut; - // Store the model name - if(Model.empty()) Model = fgi.technique(); + // Store the model and variable names + if(Model.empty()) Model = pgi.technique(); + if(VarName.empty()) VarName = prob_genesis_name; + if(VxMask.empty()) VxMask = (VxOpt->VxMaskName.empty() ? + na_str : VxOpt->VxMaskName); // Track the range of forecast initalization times - ut = fgi.init(); + ut = pgi.init(); if(InitBeg == 0 || InitBeg > ut) InitBeg = ut; if(InitEnd == 0 || InitEnd < ut) InitEnd = ut; @@ -1297,28 +1360,28 @@ void ProbGenPCTInfo::add(const ProbGenInfo &fgi, int index, } // Current lead time and probability value - int lead_hr = nint(fgi.prob_item(index)); - double prob = fgi.prob(index) / 100.0; + int lead_hr = nint(pgi.prob_item(index)); + double prob = pgi.prob(index) / 100.0; // Add new map entries, if needed if(!LeadTimes.has(lead_hr)) { LeadTimes.add(lead_hr); - vector empty_fgi; + vector empty_pgi; vector empty_idx; vector empty_bgi; vector empty_evt; PCTMap [lead_hr] = DefaultPCT; - FcstGenMap[lead_hr] = empty_fgi; - FcstIdxMap[lead_hr] = empty_idx; + ProbGenMap[lead_hr] = empty_pgi; + ProbIdxMap[lead_hr] = empty_idx; BestGenMap[lead_hr] = empty_bgi; BestEvtMap[lead_hr] = empty_evt; } // Update map entries - FcstGenMap[lead_hr].push_back(&fgi); - FcstIdxMap[lead_hr].push_back(index); + ProbGenMap[lead_hr].push_back(&pgi); + ProbIdxMap[lead_hr].push_back(index); BestGenMap[lead_hr].push_back(bgi); BestEvtMap[lead_hr].push_back(is_event); @@ -1330,3 +1393,58 @@ void ProbGenPCTInfo::add(const ProbGenInfo &fgi, int index, } //////////////////////////////////////////////////////////////////////// + +void ProbGenPCTInfo::add_genshape(const GenShapeInfo &gsi, int index, + const GenesisInfo *bgi, bool is_event) { + unixtime ut; + int lead_hr; + + // Store the model and variable names + if(Model.empty()) Model = "OPER"; + if(VarName.empty()) VarName = genesis_shape_name; + if(VxMask.empty()) VxMask = (VxOpt->VxMaskName.empty() ? + na_str : VxOpt->VxMaskName); + + // Track the range of forecast issue times + ut = gsi.issue_time(); + if(InitBeg == 0 || InitBeg > ut) InitBeg = ut; + if(InitEnd == 0 || InitEnd < ut) InitEnd = ut; + + // Track the range of verifying BEST genesis events + if(bgi) { + ut = bgi->genesis_time(); + if(BestBeg == 0 || BestBeg > ut) BestBeg = ut; + if(BestEnd == 0 || BestEnd < ut) BestEnd = ut; + } + + // Add new map entries, if needed + lead_hr = gsi.lead_sec(index)/sec_per_hour; + if(!LeadTimes.has(lead_hr)) { + + LeadTimes.add(lead_hr); + vector empty_gsi; + vector empty_idx; + vector empty_bgi; + vector empty_evt; + + PCTMap [lead_hr] = DefaultPCT; + GenShapeMap[lead_hr] = empty_gsi; + ProbIdxMap[lead_hr] = empty_idx; + BestGenMap[lead_hr] = empty_bgi; + BestEvtMap[lead_hr] = empty_evt; + } + + // Update map entries + GenShapeMap[lead_hr].push_back(&gsi); + ProbIdxMap[lead_hr].push_back(index); + BestGenMap[lead_hr].push_back(bgi); + BestEvtMap[lead_hr].push_back(is_event); + + // Increment counts + if(is_event) PCTMap[lead_hr].pct.inc_event (gsi.prob_val(index)); + else PCTMap[lead_hr].pct.inc_nonevent(gsi.prob_val(index)); + + return; +} + +//////////////////////////////////////////////////////////////////////// diff --git a/met/src/tools/tc_utils/tc_gen/tc_gen_conf_info.h b/met/src/tools/tc_utils/tc_gen/tc_gen_conf_info.h index b3512c0d8c..487ce9245b 100644 --- a/met/src/tools/tc_utils/tc_gen/tc_gen_conf_info.h +++ b/met/src/tools/tc_utils/tc_gen/tc_gen_conf_info.h @@ -49,6 +49,14 @@ static const STATLineType txt_file_type[n_txt] = { stat_genmpr // 7 }; +// Output data type names + +static const string genesis_name ("GENESIS"); +static const string genesis_dev_name ("GENESIS_DEV"); +static const string genesis_ops_name ("GENESIS_OPS"); +static const string prob_genesis_name ("PROB_GENESIS"); +static const string genesis_shape_name("GENESIS_SHAPE"); + // Names for output data plane types static const string fgen_str = "fcst_genesis"; @@ -129,6 +137,7 @@ class TCGenVxOpt { NumArray Lead; // Spatial masking information + ConcatString VxMaskConf; ConcatString VxMaskName; MaskPoly VxPolyMask; Grid VxGridMask; @@ -168,8 +177,9 @@ class TCGenVxOpt { const StringArray &); void parse_nc_info(Dictionary &); - bool is_keeper(const GenesisInfo &) const; - bool is_keeper(const ProbGenInfo &) const; + bool is_keeper(const GenesisInfo &) const; + bool is_keeper(const ProbGenInfo &) const; + bool is_keeper(const GenShapeInfo &) const; STATOutputType output_map(STATLineType) const; }; @@ -331,6 +341,9 @@ class ProbGenPCTInfo { ////////////////////////////////////////////////////////////////// ConcatString Model; + ConcatString VarName; + ConcatString VxMask; + unixtime InitBeg, InitEnd; unixtime BestBeg, BestEnd; const TCGenVxOpt* VxOpt; @@ -340,10 +353,11 @@ class ProbGenPCTInfo { map PCTMap; // Map of lead times to vectors of pair info - map > FcstGenMap; - map > FcstIdxMap; - map > BestGenMap; - map > BestEvtMap; + map > ProbGenMap; + map > GenShapeMap; + map > ProbIdxMap; + map > BestGenMap; + map > BestEvtMap; ////////////////////////////////////////////////////////////////// @@ -351,8 +365,11 @@ class ProbGenPCTInfo { void set_vx_opt(const TCGenVxOpt *); - void add(const ProbGenInfo &, int, - const GenesisInfo *, bool); + void add_probgen(const ProbGenInfo &, int, + const GenesisInfo *, bool); + + void add_genshape(const GenShapeInfo &, int, + const GenesisInfo *, bool); }; diff --git a/test/config/TCGenConfig_shape b/test/config/TCGenConfig_shape new file mode 100644 index 0000000000..767349741e --- /dev/null +++ b/test/config/TCGenConfig_shape @@ -0,0 +1,292 @@ +//////////////////////////////////////////////////////////////////////////////// +// +// TC-Gen configuration file. +// +// For additional information, please see the MET User's Guide. +// +//////////////////////////////////////////////////////////////////////////////// + +// +// ATCF file format reference: +// http://www.nrlmry.navy.mil/atcf_web/docs/database/new/abrdeck.html +// + +//////////////////////////////////////////////////////////////////////////////// +// +// Genesis event definition criteria +// +//////////////////////////////////////////////////////////////////////////////// + +// +// Model initialization frequency in hours, starting at 0 +// +init_freq = 6; + +// +// Valid hour frequency to be analyzed in hours, starting at 0 +// +valid_freq = 6; + +// +// Forecast hours to be searched for genesis events +// +fcst_hr_window = { + beg = 6; + end = 120; +} + +// +// Minimum track duration for genesis event in hours +// +min_duration = 12; + +// +// Forecast genesis event criteria. Defined as tracks reaching the specified +// intensity category, maximum wind speed threshold, and minimum sea-level +// pressure threshold. The forecast genesis time is the valid time of the first +// track point where all of these criteria are met. +// +fcst_genesis = { + vmax_thresh = NA; + mslp_thresh = NA; +} + +// +// BEST track genesis event criteria. Defined as tracks reaching the specified +// intensity category, maximum wind speed threshold, and minimum sea-level +// pressure threshold. The BEST track genesis time is the valid time of the +// first track point where all of these criteria are met. +// +best_genesis = { + technique = "BEST"; + category = [ "TD", "TS" ]; + vmax_thresh = NA; + mslp_thresh = NA; +} + +// +// Operational track technique name +// +oper_technique = "CARQ"; + +//////////////////////////////////////////////////////////////////////////////// +// +// Track filtering options +// May be specified separately in each filter array entry. +// +//////////////////////////////////////////////////////////////////////////////// + +// +// Array of dictionaries containing the track filtering options +// If empty, a single filter is defined using the top-level settings. +// +filter = [ + { desc = "SUMMER_INIT"; init_beg = "20210601"; init_end = "20210901"; }, + { desc = "00Z_12Z_INIT"; init_hour = [ "00", "12" ]; }, + { desc = "06Z_18Z_INIT"; init_hour = [ "06", "18" ]; }, + { desc = "AL_EP_BASIN"; basin_mask = ["AL", "EP" ]; }, + { desc = "AL_BASIN"; basin_mask = "AL"; }, + { desc = "EP_BASIN"; basin_mask = "EP"; } +]; + +// +// Description written to output DESC column +// +desc = "ALL"; + +// +// Forecast ATCF ID's +// If empty, all ATCF ID's found will be processed. +// Statistics will be generated separately for each ATCF ID. +// +model = []; + +// +// BEST and operational track storm identifiers +// +storm_id = []; + +// +// BEST and operational track storm names +// +storm_name = []; + +// +// Forecast and operational initialization times to include or exclude +// +init_beg = ""; +init_end = ""; +init_inc = []; +init_exc = []; + +// +// Forecast, BEST, and operational valid time window +// +valid_beg = ""; +valid_end = ""; + +// +// Forecast and operational initialization hours +// +init_hour = []; + +// +// Forecast and operational lead times in hours +// +lead = []; + +// +// Spatial masking region (path to gridded data file or polyline file) +// +vx_mask = ""; + +// +// Spatial masking of hurricane basin names from the basin_file +// +basin_mask = []; + +// +// Distance to land threshold +// +dland_thresh = NA; + +//////////////////////////////////////////////////////////////////////////////// +// +// Matching and scoring options +// May be specified separately in each filter array entry. +// +//////////////////////////////////////////////////////////////////////////////// + +// +// Genesis matching logic. Compare the forecast genesis point to all points in +// the Best track (TRUE) or the single Best track genesis point (FALSE). +// +genesis_match_point_to_track = TRUE; + +// +// Radius in km to search for a matching genesis event +// +genesis_match_radius = 500; + +// +// Time window in hours, relative to the model genesis time, to search for a +// matching Best track point +// +genesis_match_window = { + beg = 0; + end = 0; +} + +// +// Radius in km for a development scoring method hit +// +dev_hit_radius = 500; + +// +// Time window in hours, relative to the model genesis time, for a development +// scoring method hit +// +dev_hit_window = { + beg = -24; + end = 24; +} + +// +// Time window in hours for the Best track genesis minus model initialization +// time difference for an operational scoring method hit +// +ops_hit_window = { + beg = 0; + end = 48; +} + +// +// Discard genesis forecasts for initializations at or after the matching +// BEST track genesis time +// +discard_init_post_genesis_flag = TRUE; + +// +// Scoring methods to be applied +// +dev_method_flag = TRUE; +ops_method_flag = TRUE; + +//////////////////////////////////////////////////////////////////////////////// +// +// Output options +// May be specified separately in each filter array entry. +// +//////////////////////////////////////////////////////////////////////////////// + +// +// Probability of genesis thresholds +// +prob_genesis_thresh = ==0.25; + +// +// Confidence interval alpha value +// +ci_alpha = 0.05; + +// +// Statistical output types +// +output_flag = { + fho = NONE; + ctc = NONE; + cts = NONE; + pct = BOTH; + pstd = STAT; + pjc = STAT; + prc = STAT; + genmpr = NONE; +} + + +// +// NetCDF genesis pair counts +// +nc_pairs_flag = FALSE; + +// +// Specify which track points should be counted by thresholding the track point +// valid time minus genesis time difference. +// +valid_minus_genesis_diff_thresh = NA; + +// +// Count unique BEST track genesis event locations (TRUE) versus counting the +// location for all pairs (FALSE). +// +best_unique_flag = TRUE; + +//////////////////////////////////////////////////////////////////////////////// +// +// Global settings +// May only be specified once. +// +//////////////////////////////////////////////////////////////////////////////// + +// +// Specify the NetCDF output of the gen_dland tool containing a gridded +// representation of the minimum distance to land. +// +dland_file = "MET_BASE/tc_data/dland_global_tenth_degree.nc"; + +// +// Specify the NetCDF file containing a gridded representation of the +// global basins. +// +basin_file = "MET_BASE/tc_data/basin_global_tenth_degree.nc"; + +// +// NetCDF genesis pairs grid +// +nc_pairs_grid = "G003"; + +// +// Indicate a version number for the contents of this configuration file. +// The value should generally not be modified. +// +version = "V10.1.0"; diff --git a/test/xml/unit_tc_gen.xml b/test/xml/unit_tc_gen.xml index 86245745d8..6146f1d3ec 100644 --- a/test/xml/unit_tc_gen.xml +++ b/test/xml/unit_tc_gen.xml @@ -53,4 +53,23 @@ + + + + &MET_BIN;/tc_gen + \ + -shape &DATA_DIR;/genesis/shape/atl \ + -shape &DATA_DIR;/genesis/shape/epac \ + -track &DATA_DIR;/genesis/atcf/2021 \ + -config &CONFIG_DIR;/TCGenConfig_shape \ + -out &OUTPUT_DIR;/tc_gen/tc_gen_2021_shape \ + -log &OUTPUT_DIR;/tc_gen/tc_gen_2021_shape.log \ + -v 3 + + + &OUTPUT_DIR;/tc_gen/tc_gen_2021_shape.stat + &OUTPUT_DIR;/tc_gen/tc_gen_2021_shape_pct.txt + + + From edd61fd49f4aa3b484a95124ac8423075c5e567e Mon Sep 17 00:00:00 2001 From: Howard Soh Date: Wed, 12 Jan 2022 20:59:16 -0700 Subject: [PATCH 54/82] #1844 Added log message for use_var_id --- met/src/libcode/vx_pointdata_python/python_pointdata.cc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/met/src/libcode/vx_pointdata_python/python_pointdata.cc b/met/src/libcode/vx_pointdata_python/python_pointdata.cc index 0712bbb13a..43196acc5f 100644 --- a/met/src/libcode/vx_pointdata_python/python_pointdata.cc +++ b/met/src/libcode/vx_pointdata_python/python_pointdata.cc @@ -318,7 +318,7 @@ python_value = PyDict_GetItemString (python_met_point_data, python_use_var_id); bool use_var_id = pyobject_as_bool(python_value); met_pd_out.set_use_var_id(use_var_id); - +mlog << Debug(9) << method_name << "use_var_id: \"" << use_var_id << "\" from python. is_using_var_id(): " << met_pd_out.is_using_var_id() << "\n"; python_value = PyDict_GetItemString (python_met_point_data, python_key_nhdr); From 6cae6b526cc31364e9f8e0347a0938ac70981729 Mon Sep 17 00:00:00 2001 From: Howard Soh Date: Wed, 12 Jan 2022 21:00:10 -0700 Subject: [PATCH 55/82] #1844 Get use_var_id for python embedding --- met/src/tools/other/plot_point_obs/plot_point_obs.cc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/met/src/tools/other/plot_point_obs/plot_point_obs.cc b/met/src/tools/other/plot_point_obs/plot_point_obs.cc index af45a93641..243f4d0c29 100644 --- a/met/src/tools/other/plot_point_obs/plot_point_obs.cc +++ b/met/src/tools/other/plot_point_obs/plot_point_obs.cc @@ -189,7 +189,6 @@ void process_point_obs(const char *point_obs_filename) { nc_point_obs.check_nc(point_obs_filename, method_name_s); // exit if missing dims/vars nc_point_obs.read_obs_data(); - use_var_id = nc_point_obs.is_using_var_id(); use_obs_arr = nc_point_obs.is_using_obs_arr(); // Print warning about ineffective command line arguments @@ -219,6 +218,7 @@ void process_point_obs(const char *point_obs_filename) { int obs_qty_block[buf_size]; float obs_arr_block[buf_size][OBS_ARRAY_LEN]; + use_var_id = met_point_obs->is_using_var_id(); if(use_var_id) var_list = met_point_obs->get_var_names(); qty_list = met_point_obs->get_qty_data(); From aaa1010f44b62150009b6dee88de7a952efe4f69 Mon Sep 17 00:00:00 2001 From: Howard Soh Date: Wed, 12 Jan 2022 21:01:36 -0700 Subject: [PATCH 56/82] #1844 Initial release --- met/scripts/python/met_point_obs.py | 169 +++++++++++++++++++++++ met/scripts/python/read_met_point_obs.py | 136 ++++++++++++++++++ 2 files changed, 305 insertions(+) create mode 100755 met/scripts/python/met_point_obs.py create mode 100755 met/scripts/python/read_met_point_obs.py diff --git a/met/scripts/python/met_point_obs.py b/met/scripts/python/met_point_obs.py new file mode 100755 index 0000000000..b4e1e59e30 --- /dev/null +++ b/met/scripts/python/met_point_obs.py @@ -0,0 +1,169 @@ +''' +Created on Nov 10, 2021 + +@author: hsoh +''' + + +import numpy as np + +COUNT_SHOW = 30 + +MET_PYTHON_OBS_ARGS = "MET_POINT_PYTHON_ARGS" + +class met_point_obs(object): + ''' + classdocs + ''' + python_prefix = 'PYTHON_POINT_RAW' + + def __init__(self, use_var_id=True): + ''' + Constructor + ''' + self.raw_data = None + self.use_var_id = use_var_id # True if variable index, False if GRIB code + + # Header + self.nhdr = 0 + self.npbhdr = 0 + self.nhdr_typ = 0 # type table + self.nhdr_sid = 0 #station_uid table + self.nhdr_vld = 0 # valid time strings + self.hdr_typ = [] # (nhdr) + self.hdr_sid = [] # (nhdr) + self.hdr_vld = [] # (nhdr) + self.hdr_lat = [] # (nhdr) + self.hdr_lon = [] # (nhdr) + self.hdr_elv = [] # (nhdr) + self.hdr_typ_table = [] # (nhdr_typ, mxstr2) + self.hdr_sid_table = [] # (nhdr_sid, mxstr2) + self.hdr_vld_table = [] # (nhdr_vld, mxstr) + self.hdr_prpt_typ = [] # optional + self.hdr_irpt_typ = [] # optional + self.hdr_inst_typ = [] # optional + + #Observation data + self.nobs = 0 + self.nobs_qty = 0 + self.nobs_var = 0 + self.obs_qty = [] # (nobs_qty ) + self.obs_hid = [] # (nobs) + self.obs_vid = [] # (nobs) veriable index or GRIB code + self.obs_lvl = [] # (nobs) + self.obs_hgt = [] # (nobs) + self.obs_val = [] # (nobs) + self.obs_qty_table = [] # (nobs_qty, mxstr) + self.obs_var_table = [] # (nobs_var, mxstr2) optional if GRIB code at self.obs_vid + + #def convert(self): + # pass + + def get_point_data(self): + if self.nhdr <= 0: + self.nhdr = len(self.hdr_lat) + if self.nobs <= 0: + self.nobs = len(self.obs_val) + if self.nhdr_typ <= 0: + self.nhdr_typ = len(self.hdr_typ_table) + if self.nhdr_sid <= 0: + self.nhdr_sid = len(self.hdr_sid_table) + if self.nhdr_vld <= 0: + self.nhdr_vld = len(self.hdr_vld_table) + if self.npbhdr <= 0: + self.npbhdr = len(self.hdr_prpt_typ) + if self.nobs_qty <= 0: + self.nobs_qty = len(self.obs_qty_table) + if self.nobs_var <= 0: + self.nobs_var = len(self.obs_var_table) + return self.__dict__ + + @staticmethod + def is_python_script(arg_value): + return arg_value.startswith(met_point_obs.python_prefix) + + @staticmethod + def get_python_script(arg_value): + return arg_value[len(met_point_obs.python_prefix)+1:] + + @staticmethod + def print_data(key, data_array, show_count=COUNT_SHOW): + if isinstance(data_array, list): + data_len = len(data_array) + if show_count >= data_len: + print(" {k:10s}: {v}".format(k=key, v= data_array)) + else: + end_offset = int(show_count/2) + print(" {k:10s}: count={v}".format(k=key, v=data_len)) + print(" {k:10s}[0:{o}] {v}".format(k=key, v=data_array[:end_offset], o=end_offset)) + print(" {k:10s}[{s}:{e}]: {v}".format(k=key, v='...', s=end_offset+1, e=data_len-end_offset-1)) + print(" {k:10s}[{s}:{e}]: {v}".format(k=key, v= data_array[-end_offset:], s=(data_len-end_offset), e=(data_len-1))) + else: + print(" {k:10s}: {v}".format(k=key, v= data_array)) + + @staticmethod + def print_point_data(met_point_data): + print(' === MET point data by python embedding ===') + met_point_obs.print_data('nhdr', met_point_data['nhdr']) + met_point_obs.print_data('nobs' , met_point_data['nobs']) + met_point_obs.print_data('use_var_id',met_point_data['use_var_id']) + met_point_obs.print_data('hdr_typ',met_point_data['hdr_typ']) + met_point_obs.print_data('obs_hid',met_point_data['obs_hid']) + met_point_obs.print_data('obs_vid',met_point_data['obs_vid']) + met_point_obs.print_data('obs_val',met_point_data['obs_val']) + met_point_obs.print_data('obs_var_table',met_point_data['obs_var_table']) + met_point_obs.print_data('obs_qty_table',met_point_data['obs_qty_table']) + met_point_obs.print_data('hdr_typ_table',met_point_data['hdr_typ_table']) + met_point_obs.print_data('hdr_sid_table',met_point_data['hdr_sid_table']) + met_point_obs.print_data('hdr_vld_table',met_point_data['hdr_vld_table']) + print(' === MET point data by python embedding ===') + + +def get_sample_point_obs(): + point_obs_data = met_point_obs() + point_obs_data.hdr_typ = np.array([ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ]) + point_obs_data.hdr_sid = np.array([ 0, 0, 0, 0, 0, 1, 2, 3, 3, 1, 2, 2, 3, 0, 0, 0, 0, 0, 1, 2, 3, 3, 1, 2, 2, 3 ]) + point_obs_data.hdr_vld = np.array([ 0, 1, 2, 3, 4, 4, 3, 4, 3, 4, 5, 4, 3, 0, 1, 2, 3, 4, 4, 3, 4, 3, 4, 5, 4, 3 ]) + point_obs_data.hdr_lat = np.array([ 43., 43., 43., 43., 43., 43., 43., 43., 43., 46., 46., 46., 46., 43., 43., 43., 43., 43., 43., 43., 43., 43., 46., 46., 46., 46. ]) + point_obs_data.hdr_lon = np.array([ -89., -89., -89., -89., -89., -89., -89., -89., -89., -92., -92., -92., -92., -89., -89., -89., -89., -89., -89., -89., -89., -89., -92., -92., -92., -92. ]) + point_obs_data.hdr_elv = np.array([ 220., 220., 220., 220., 220., 220., 220., 220., 220., 220., 220., 220., 220., 220., 220., 220., 220., 220., 220., 220., 220., 220., 220., 220., 220., 220. ]) + + point_obs_data.obs_qid = np.array([ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ]) + point_obs_data.obs_hid = np.array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 25 ]) + point_obs_data.obs_vid = np.array([ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 ]) + point_obs_data.obs_lvl = np.array([ 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000. ]) + point_obs_data.obs_hgt = np.array([ 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2. ]) + point_obs_data.obs_val = np.array([ 292., 292.5, 293., 293.5, 294., 294.5, 295., 295.5, 296., 292., 293.4, 293., 296., 294., 92., 92.5, 93., 93.5, 94., 94.5, 95., 95.5, 96., 92., 93.4, 93., 96., 94. ]) + + point_obs_data.hdr_typ_table = [ "ADPSFC" ] + point_obs_data.hdr_sid_table = [ "001", "002", "003", "004" ] + point_obs_data.hdr_vld_table = [ + "20120409_115000", "20120409_115500", "20120409_120100", "20120409_120500", "20120409_121000", + "20120409_120000" ] + point_obs_data.obs_var_table = [ "TMP", "RH" ] + point_obs_data.obs_qty_table = [ "NA" ] + point_obs_data.nhdr = len(point_obs_data.hdr_lat) + point_obs_data.nobs = len(point_obs_data.obs_val) + #point_obs_data.met_point_data = point_obs_data + return point_obs_data + + +def main(): + met_point_data = get_sample_point_obs().get_point_data() + + print('All',met_point_data) + print(" nhdr: ",met_point_data['nhdr']) + print(" nobs: ",met_point_data['nobs']) + print('use_var_id: ',met_point_data['use_var_id']) + print('hdr_typ: ',met_point_data['hdr_typ']) + print('obs_vid: ',met_point_data['obs_vid']) + print('obs_val: ',met_point_data['obs_val']) + print('obs_var_table: ',met_point_data['obs_var_table']) + print('obs_qty_table: ',met_point_data['obs_qty_table']) + print('hdr_typ_table: ',met_point_data['hdr_typ_table']) + print('hdr_sid_table: ',met_point_data['hdr_sid_table']) + print('hdr_vld_table: ',met_point_data['hdr_vld_table']) + print('Done python scripot') + +if __name__ == '__main__': + main() diff --git a/met/scripts/python/read_met_point_obs.py b/met/scripts/python/read_met_point_obs.py new file mode 100755 index 0000000000..5a99f63a8e --- /dev/null +++ b/met/scripts/python/read_met_point_obs.py @@ -0,0 +1,136 @@ +''' +Created on Nov 10, 2021 + +@author: hsoh +''' + +import os +import sys +import time +import numpy as np +import netCDF4 as nc + +from met_point_obs import met_point_obs, get_sample_point_obs, MET_PYTHON_OBS_ARGS + +DO_PRINT_DATA = False +ARG_PRINT_DATA = 'show_data' + +class nc_point_obs(met_point_obs): + + @staticmethod + def get_num_array(dataset, var_name): + nc_var = dataset.variables.get(var_name, None) + return nc_var[:].filled(nc_var._FillValue if '_FillValue' in nc_var.ncattrs() else -9999) if nc_var else [] + + @staticmethod + def get_ncbyte_array_to_str(nc_var): + nc_str_data = nc_var[:] + if nc_var.datatype.name == 'bytes8': + nc_str_data = [ str(s.compressed(),"utf-8") for s in nc_var[:]] + return nc_str_data + + @staticmethod + def get_string_array(dataset, var_name): + nc_var = dataset.variables.get(var_name, None) + return nc_point_obs.get_ncbyte_array_to_str(nc_var) if nc_var else [] + + #def convert(self): + # pass + + def read_nectdf(self, nc_filename): + if os.path.exists(nc_filename): + dataset = nc.Dataset(nc_filename, 'r') + # Header + self.hdr_typ = dataset['hdr_typ'][:] + self.hdr_sid = dataset['hdr_sid'][:] + self.hdr_vld = dataset['hdr_vld'][:] + self.hdr_lat = dataset['hdr_lat'][:] + self.hdr_lon = dataset['hdr_lon'][:] + self.hdr_elv = dataset['hdr_elv'][:] + self.hdr_typ_table = self.get_string_array(dataset, 'hdr_typ_table') + self.hdr_sid_table = self.get_string_array(dataset, 'hdr_sid_table') + self.hdr_vld_table = self.get_string_array(dataset, 'hdr_vld_table') + + nc_var = dataset.variables.get('hdr_prpt_typ', None) + if nc_var: + self.hdr_prpt_typ = nc_var[:] + nc_var = dataset.variables.get('hdr_irpt_typ', None) + if nc_var: + self.hdr_irpt_typ = nc_var[:] + nc_var = dataset.variables.get('hdr_inst_typ', None) + if nc_var: + self.hdr_inst_typ =nc_var[:] + + #Observation data + self.nobs = 0 + self.hdr_sid = dataset['hdr_sid'][:] + #self.obs_qty = self.get_num_array(dataset, 'obs_qty') + self.obs_qty = np.array(dataset['obs_qty'][:]) + self.obs_hid = np.array(dataset['obs_hid'][:]) + self.obs_lvl = np.array(dataset['obs_lvl'][:]) + self.obs_hgt = np.array(dataset['obs_hgt'][:]) + self.obs_val = np.array(dataset['obs_val'][:]) + nc_var = dataset.variables.get('obs_vid', None) + if nc_var is None: + self.use_var_id = False + nc_var = dataset.variables.get('obs_gc', None) + else: + self.obs_var_table = self.get_string_array(dataset, 'obs_var') + if nc_var: + self.obs_vid = np.array(nc_var[:]) + + self.obs_qty_rwo = dataset['obs_qty'][:] + + self.obs_qty_table = self.get_string_array(dataset, 'obs_qty_table') + else: + print("==ERROR== netcdf file ({f}) does not exist".format(f=nc_filename)) + + + +perf_start_time = time.time() +perf_start_counter = time.perf_counter_ns() + +point_obs_data = None +if len(sys.argv) == 1: + point_obs_data = get_sample_point_obs() +else: + netcdf_filename = sys.argv[1] + if os.path.exists(netcdf_filename): + point_obs_data = nc_point_obs() + point_obs_data.read_nectdf(netcdf_filename) + elif met_point_obs.is_python_script(netcdf_filename): + python_script = met_point_obs.get_python_script(netcdf_filename) + python_args = [] + if len(sys.argv) > 2: + if ARG_PRINT_DATA == sys.argv[-1]: + python_args = sys.argv[2:-1] + else: + python_args = sys.argv[2:] + python_args_string = " ".join(python_args) if 0 < len(python_args) else '' + os.environ[MET_PYTHON_OBS_ARGS] = python_args_string + #os.system(python_command) + exec(open(python_script).read()) + print('------') + + if point_obs_data is None: + print("PYTHON: Fail to get point_obs_data by calling {s} from {m}".format( + s=python_script, m=sys.argv[0])) + if ARG_PRINT_DATA == sys.argv[-1]: + DO_PRINT_DATA = True + + point_obs_data = nc_point_obs() + point_obs_data.read_nectdf(netcdf_filename) + +if point_obs_data is not None: + met_point_data = point_obs_data.get_point_data() + met_point_data['met_point_data'] = point_obs_data + + if DO_PRINT_DATA: + met_point_obs.print_point_data(met_point_data) + +perf_end_time = time.time() +perf_end_counter = time.perf_counter_ns() +perf_duration = perf_end_time - perf_start_time +perf_duration_counter = (perf_end_counter - perf_start_counter) / 1000000000 + +print('Done python script {s} Took walltime: {t1} & perf: {t2} seconds'.format(s=sys.argv[0], t1=perf_duration, t2=perf_duration_counter)) From adf4928a8dd35933781c79d0cbe40c40cfdd0dcd Mon Sep 17 00:00:00 2001 From: Howard Soh Date: Wed, 12 Jan 2022 21:02:14 -0700 Subject: [PATCH 57/82] #1844 Added met_point_obs.py and read_met_point_obs.py --- met/scripts/python/Makefile.am | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/met/scripts/python/Makefile.am b/met/scripts/python/Makefile.am index 78dc7d88bc..368da3da7f 100644 --- a/met/scripts/python/Makefile.am +++ b/met/scripts/python/Makefile.am @@ -26,11 +26,13 @@ pythonscriptsdir = $(pkgdatadir)/python pythonscripts_DATA = \ + met_point_obs.py \ read_ascii_numpy.py \ read_ascii_numpy_grid.py \ read_ascii_xarray.py \ read_ascii_point.py \ - read_ascii_mpr.py + read_ascii_mpr.py \ + read_met_point_obs.py EXTRA_DIST = ${pythonscripts_DATA} From 0640d8ddfe39c33d70f76e19d9dd942812eebcbb Mon Sep 17 00:00:00 2001 From: Howard Soh Date: Wed, 12 Jan 2022 21:04:17 -0700 Subject: [PATCH 58/82] #1844 Added 4 unit test for python embedding of MET point data --- test/xml/unit_python.xml | 107 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 107 insertions(+) diff --git a/test/xml/unit_python.xml b/test/xml/unit_python.xml index 35993ed441..6b0850343d 100644 --- a/test/xml/unit_python.xml +++ b/test/xml/unit_python.xml @@ -446,4 +446,111 @@ + + &MET_BIN;/point2grid + \ + 'PYTHON_NUMPY=&MET_BASE;/python/read_met_point_obs.py &OUTPUT_DIR;/pb2nc/ndas.20120409.t12z.prepbufr.tm00.nc' \ + G212 \ + &OUTPUT_DIR;/python/pb2nc_TMP.nc \ + -field 'name="TMP"; level="*"; valid_time="20120409_120000"; censor_thresh=[ <0 ]; censor_val=[0];' \ + -name TEMP \ + -v 1 + + + &OUTPUT_DIR;/python/pb2nc_TMP.nc + + + + + &MET_BIN;/plot_point_obs + + TO_GRID NONE + + \ + &OUTPUT_DIR;/pb2nc/ndas.20120409.t12z.prepbufr.tm00.nc \ + &OUTPUT_DIR;/python/nam_and_ndas.20120409.t12z.prepbufr_CONFIG.ps \ + -point_obs &OUTPUT_DIR;/ascii2nc/trmm_2012040912_3hr.nc \ + -plot_grid &DATA_DIR_MODEL;/grib2/nam/nam_2012040900_F012.grib2 \ + -config &CONFIG_DIR;/PlotPointObsConfig \ + -title "NAM 2012040900 F12 vs NDAS 500mb RH and TRMM 3h > 0" \ + -v 3 + + + &OUTPUT_DIR;/python/nam_and_ndas.20120409.t12z.prepbufr_CONFIG.ps + + + + + &MET_BIN;/plot_point_obs + + TO_GRID NONE + + \ + 'PYTHON_NUMPY=&MET_BASE;/python/read_met_point_obs.py &OUTPUT_DIR;/pb2nc/ndas.20120409.t12z.prepbufr.tm00.nc' \ + &OUTPUT_DIR;/python/nam_and_ndas.20120409.t12z.prepbufr_CONFIG.ps \ + -point_obs 'PYTHON_NUMPY=&MET_BASE;/python/read_met_point_obs.py &OUTPUT_DIR;/ascii2nc/trmm_2012040912_3hr.nc' \ + -plot_grid &DATA_DIR_MODEL;/grib2/nam/nam_2012040900_F012.grib2 \ + -config &CONFIG_DIR;/PlotPointObsConfig \ + -title "NAM 2012040900 F12 vs NDAS 500mb RH and TRMM 3h > 0" \ + -v 3 + + + &OUTPUT_DIR;/python/nam_and_ndas.20120409.t12z.prepbufr_CONFIG.ps + + + + + echo "&DATA_DIR_MODEL;/grib1/arw-fer-gep1/arw-fer-gep1_2012040912_F024.grib \ + &DATA_DIR_MODEL;/grib1/arw-fer-gep5/arw-fer-gep5_2012040912_F024.grib \ + &DATA_DIR_MODEL;/grib1/arw-sch-gep2/arw-sch-gep2_2012040912_F024.grib \ + &DATA_DIR_MODEL;/grib1/arw-sch-gep6/arw-sch-gep6_2012040912_F024.grib \ + &DATA_DIR_MODEL;/grib1/arw-tom-gep3/arw-tom-gep3_2012040912_F024.grib \ + &DATA_DIR_MODEL;/grib1/arw-tom-gep7/arw-tom-gep7_2012040912_F024.grib" \ + > &OUTPUT_DIR;/python/ensemble_stat/input_file_list; \ + &MET_BIN;/ensemble_stat + + DESC NA + OBS_ERROR_FLAG FALSE + SKIP_CONST FALSE + OUTPUT_PREFIX FILE_LIST + + \ + &OUTPUT_DIR;/python/ensemble_stat/input_file_list \ + &CONFIG_DIR;/EnsembleStatConfig \ + -grid_obs &DATA_DIR_OBS;/laps/laps_2012041012_F000.grib \ + -point_obs 'PYTHON_NUMPY=&MET_BASE;/python/read_met_point_obs.py &OUTPUT_DIR;/ascii2nc/gauge_2012041012_24hr.nc' \ + -outdir &OUTPUT_DIR;/python/ensemble_stat -v 1 + + + &OUTPUT_DIR;/python/ensemble_stat/ensemble_stat_FILE_LIST_20120410_120000V.stat + &OUTPUT_DIR;/python/ensemble_stat/ensemble_stat_FILE_LIST_20120410_120000V_ecnt.txt + &OUTPUT_DIR;/python/ensemble_stat/ensemble_stat_FILE_LIST_20120410_120000V_rhist.txt + &OUTPUT_DIR;/python/ensemble_stat/ensemble_stat_FILE_LIST_20120410_120000V_phist.txt + &OUTPUT_DIR;/python/ensemble_stat/ensemble_stat_FILE_LIST_20120410_120000V_orank.txt + &OUTPUT_DIR;/python/ensemble_stat/ensemble_stat_FILE_LIST_20120410_120000V_ssvar.txt + &OUTPUT_DIR;/python/ensemble_stat/ensemble_stat_FILE_LIST_20120410_120000V_ens.nc + &OUTPUT_DIR;/python/ensemble_stat/ensemble_stat_FILE_LIST_20120410_120000V_orank.nc + + + + + &MET_BIN;/point_stat + + BEG_DS -1800 + END_DS 1800 + OUTPUT_PREFIX GRIB1_NAM_GDAS_WINDS + CONFIG_DIR &CONFIG_DIR; + CLIMO_FILE "&DATA_DIR_MODEL;/grib1/gfs/gfs_2012040900_F012_gNam.grib" + + \ + &DATA_DIR_MODEL;/grib1/nam/nam_2012040900_F012.grib \ + 'PYTHON_NUMPY=&MET_BASE;/python/read_met_point_obs.py &OUTPUT_DIR;/pb2nc/gdas1.20120409.t12z.prepbufr.nc' \ + &CONFIG_DIR;/PointStatConfig_WINDS \ + -outdir &OUTPUT_DIR;/python -v 1 + + + &OUTPUT_DIR;/python/point_stat_GRIB1_NAM_GDAS_WINDS_120000L_20120409_120000V.stat + + + From 37f4f321b07a1933aea85cb3120356605e0e2a72 Mon Sep 17 00:00:00 2001 From: John Halley Gotway Date: Wed, 12 Jan 2022 21:25:51 -0700 Subject: [PATCH 59/82] More changes to test_util.R to account for differences between 4.7.0 and 4.9.2 of ncdiff. The earlier version reported 0 diffs for the time_bounds, lat, lon, latitude, and longitude variables while the newer version reports their raw values. Simply ignore these variables for now. This isn't a great solution but when we reimplement this testing logic in python, we can address this issue then. --- test/R_test/test_util.R | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/test/R_test/test_util.R b/test/R_test/test_util.R index d00616209a..bcbeb14e71 100644 --- a/test/R_test/test_util.R +++ b/test/R_test/test_util.R @@ -753,6 +753,21 @@ compareNc = function(nc1, nc2, verb, strict=0, delta=-1, comp_var=0){ # for each variable present in the file, check for differences for(strVar in names(ncFileD$var)){ + # Skip the time_bounds, lat, and lon variables + # Note: Running "ncdiff -x -v time_bounds" successfully excludes that variable + # in version 4.7.0, but not in version 4.9.2. Similarly, version 4.7.0 + # returns 0 diffs for variables named lat and lon while version 4.9.2 + # returns the raw values rather than their diffs. + # For now, just ignore these variables. + if(strVar == "time_bounds" || + strVar == "lat" || + strVar == "latitude" || + strVar == "lon" || + strVar == "longitude" ){ + if( 1 <= verb ){ cat("ignored var", strVar, "\n"); } + next; + } + # check the variable attributes for differences listAtt1 = ncatt_get(ncFile1, varid=strVar); listAtt1Nam = names(listAtt1); From 4f57edd62d52d5496a881aecc696bf14d8a8d6a2 Mon Sep 17 00:00:00 2001 From: Howard Soh Date: Wed, 12 Jan 2022 22:37:42 -0700 Subject: [PATCH 60/82] #1844 Added python embedding for point observation data --- met/docs/Users_Guide/appendixF.rst | 50 +++++++++++++++++++++++++ met/docs/Users_Guide/ensemble-stat.rst | 2 +- met/docs/Users_Guide/plotting.rst | 15 ++++++++ met/docs/Users_Guide/point-stat.rst | 2 +- met/docs/Users_Guide/reformat_point.rst | 19 ++++++++++ 5 files changed, 86 insertions(+), 2 deletions(-) diff --git a/met/docs/Users_Guide/appendixF.rst b/met/docs/Users_Guide/appendixF.rst index 3a4046e9ea..586c264c9e 100644 --- a/met/docs/Users_Guide/appendixF.rst +++ b/met/docs/Users_Guide/appendixF.rst @@ -257,3 +257,53 @@ The **read_ascii_mpr.py** sample script can be found in: • MET installation directory in *MET_BASE/python*. • `MET GitHub repository `_ in *met/scripts/python*. + + +Python Embedding for Point Observations as input +________________________________________________ + + +The point2grid, plot_point_obs, ensemble_stat, and point_stat tools use MET point observation NetCDF. They support the python embedding by the prefix 'PYTHON_NUMPY=" and followed by a python script name instead of the MET point observastion NetCDF filename. The customized python script is expected to extend MET_BASE/python/met_point_obs.py and to produce the python variable, **met_point_data**, which is the dictionary of the MET point observation data. They are defined at MET_BASE/python/met_point_obs.py. + + +.. _pyembed-point-obs-data: + + +.. code-block:: none + + met_point_data = { + + 'use_var_id': Trur/False, # obs_vid are variable index if True, otherwise GRIB codes + + # Header data + 'nhdr': integer_value, # number of headers + 'pbhdr': integer_value, # number of PREPBUFR specific headers + 'nhdr_typ': integer_value, # number of message types + 'nhdr_sid': integer_value, # number of station IDs + 'nhdr_vld': integer_value, # number of valid times + 'hdr_typ': nympy_integer_array, # index of message type + 'hdr_sid': nympy_integer_array, # index of station ID + 'hdr_vld': nympy_integer_array, # index of valid time + 'hdr_lat': nympy_float_array, # latitude + 'hdr_lon': nympy_float_array, # longitude + 'hdr_elv': nympy_float_array, # station elevation + 'hdr_typ_table': string_value, # message types + 'hdr_sid_table': string_value, # station IDs + 'hdr_vld_table': string_value, # valid times "yyyymmdd_hhmmss" + 'hdr_prpt_typ': nympy_integer_array, # optional + 'hdr_irpt_typ': nympy_integer_array, # optional + 'hdr_inst_typ': nympy_integer_array, # optional + + # Observation data + 'nobs': integer_value, # number of observation + 'nobs_qty': integer_value # number of quality marks + 'nobs_var': integer_value # number of variable names + 'obs_qty': nympy_integer_array, # index of quality mark + 'obs_hid': nympy_integer_array, # index of header + 'obs_vid': nympy_integer_array, # index of veriable or GRIB code + 'obs_lvl': nympy_float_array, # pressure level + 'obs_hgt': nympy_float_array, # height of observation data + 'obs_val' nympy_float_array, # observatin value + 'obs_qty_table': string_array, # quality marks + 'obs_var_table': string_array, # variable names + } diff --git a/met/docs/Users_Guide/ensemble-stat.rst b/met/docs/Users_Guide/ensemble-stat.rst index 266519da4f..bf3bd54291 100644 --- a/met/docs/Users_Guide/ensemble-stat.rst +++ b/met/docs/Users_Guide/ensemble-stat.rst @@ -93,7 +93,7 @@ Optional arguments for ensemble_stat 4. To produce ensemble statistics using gridded observations, use the **-grid_obs file** option to specify a gridded observation file. This option may be used multiple times if your observations are in several files. -5. To produce ensemble statistics using point observations, use the **-point_obs file** option to specify a NetCDF point observation file. This option may be used multiple times if your observations are in several files. +5. To produce ensemble statistics using point observations, use the **-point_obs file** option to specify a NetCDF point observation file. This option may be used multiple times if your observations are in several files. The python embedding will be activated if the **file** begines with 'PYTHON_NUMPY=" and followed by a python script name. 6. To override the simple ensemble mean value of the input ensemble members for the ECNT, SSVAR, and ORANK line types, the **-ens_mean file** option specifies an ensemble mean model data file. This option replaces the **-ssvar_mean file** option from earlier versions of MET. diff --git a/met/docs/Users_Guide/plotting.rst b/met/docs/Users_Guide/plotting.rst index 7f748e7885..a96dca605a 100644 --- a/met/docs/Users_Guide/plotting.rst +++ b/met/docs/Users_Guide/plotting.rst @@ -8,6 +8,9 @@ __________________ This section describes how to check your data files using plotting utilities. Point observations can be plotted using the Plot-Point-Obs utility. A single model level can be plotted using the plot_data_plane utility. For object based evaluations, the MODE objects can be plotted using plot_mode_field. Occasionally, a post-processing or timing error can lead to errors in MET. These tools can assist the user by showing the data to be verified to ensure that times and locations match up as expected. +MET version 10.1 adds support for the passing point observations to plot_point_obs using a Python scriptAn example of running plot_point_obs with Python embedding is included below. + + plot_point_obs usage ~~~~~~~~~~~~~~~~~~~~ @@ -66,6 +69,18 @@ An example of the plot_point_obs calling sequence is shown below: In this example, the Plot-Point-Obs tool will process the input sample_pb.nc file and write a postscript file containing a plot to a file named sample_pb.ps. +This is an equivalent command with the python embedding. This is an example for the python embeddingt. + +.. code-block:: none + + plot_point_obs 'PYTHON_NUMPY=MET_BASE/python/read_met_point_obs.py sample_pb.nc' sample_data.ps + + +The user should replace the python script with the customized python script for the custom point observation data. + +Please refer to :numref:`Appendix F, Section %s ` for more details about Python embedding in MET. + + plot_point_obs configuration file ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/met/docs/Users_Guide/point-stat.rst b/met/docs/Users_Guide/point-stat.rst index 7a9966511c..ce7b74e905 100644 --- a/met/docs/Users_Guide/point-stat.rst +++ b/met/docs/Users_Guide/point-stat.rst @@ -286,7 +286,7 @@ Required arguments for point_stat Optional arguments for point_stat ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -4. The **-point_obs** file may be used to pass additional NetCDF point observation files to be used in the verification. +4. The **-point_obs** file may be used to pass additional NetCDF point observation files to be used in the verification. The python embedding will be activated if the **file** begines with 'PYTHON_NUMPY=" and followed by a python script name. 5. The **-obs_valid_beg** time option in YYYYMMDD[_HH[MMSS]] format sets the beginning of the observation matching time window, overriding the configuration file setting. diff --git a/met/docs/Users_Guide/reformat_point.rst b/met/docs/Users_Guide/reformat_point.rst index 999393b72d..9e41ca80dc 100644 --- a/met/docs/Users_Guide/reformat_point.rst +++ b/met/docs/Users_Guide/reformat_point.rst @@ -981,6 +981,8 @@ _______________ The Point2Grid tool takes point observations from a NetCDF output file from one of the four previously mentioned MET tools (ascii2nc, madis2nc, pb2nc, lidar2nc) and creates a gridded NetCDF file. The other point observations are GOES-16/17 input files in NetCDF format (especially, Aerosol Optical Depth. Future development will include support for reading input files not produced from MET tools. +MET version 10.1 adds support for the passing point observations to point2grid using a Python script with "input_filename" which begins with the "PYTHON_NUMPY= [arguments]" instead of MET point observation NetCDF input. An example of running point2grid with Python embedding is included below. + point2grid usage ~~~~~~~~~~~~~~~~ @@ -1013,6 +1015,8 @@ Required arguments for point2grid 1. The **input_filename** argument indicates the name of the input NetCDF file to be processed. Currently, only NetCDF files produced from the ascii2nc, madis2nc, pb2nc, and lidar2nc are supported. And AOD dataset from GOES16/17 are supported, too. Support for additional file types will be added in future releases. +The MET point observation NetCDF file name as **input_filename** argument is equivalent with "PYTHON_NUMPY=MET_BASE/python/read_met_point_obs.py netcdf_file name'. + 2. The **to_grid** argument defines the output grid as: (1) a named grid, (2) the path to a gridded data file, or (3) an explicit grid specification string. 3. The **output_filename** argument is the name of the output NetCDF file to be written. @@ -1065,6 +1069,21 @@ For the GOES-16 and GOES-17 data, the computing lat/long is time consuming. So t When processing GOES-16 data, the **-qc** option may also be used to specify the acceptable quality control flag values. The example above regrids the GOES-16 AOD values to NCEP Grid number 212 (which QC flags are high, medium, and low), writing to the output the maximum AOD value falling inside each grid box. +Here is an example of processing the same set of observations but using Python embedding instead: + + +.. code-block:: none + + point2grid \ + 'PYTHON_NUMPY=MET_BASE/python/read_met_point_obs.py ascii2nc_edr_hourly.20130827.nc' \ + G212 python_gridded_ascii_python.nc -config Point2GridConfig_edr \ + -field 'name="200"; level="*"; valid_time="20130827_205959";' -method MAX -v 1 + +The user should replace the python script with the customized python script for the custom point observation data. This is an example for the python embedding. + +Please refer to :numref:`Appendix F, Section %s ` for more details about Python embedding in MET. + + point2grid output ~~~~~~~~~~~~~~~~~ From de81526c4894e14ca0496eba9872462f8cc1dfd8 Mon Sep 17 00:00:00 2001 From: mo-mglover <78152252+mo-mglover@users.noreply.github.com> Date: Thu, 13 Jan 2022 22:48:26 +0000 Subject: [PATCH 61/82] Feature 1926 gridstat openmp (#1977) Co-authored-by: johnhg Co-authored-by: John Halley Gotway Co-authored-by: Julie Prestopnik --- met/configure.ac | 7 + met/docs/Users_Guide/config_options.rst | 119 ++++++++++-- met/docs/Users_Guide/installation.rst | 8 + met/src/basic/vx_util/Makefile.am | 1 + met/src/basic/vx_util/data_plane_util.cc | 170 ++++++++++-------- met/src/basic/vx_util/handle_openmp.cc | 52 ++++++ met/src/basic/vx_util/handle_openmp.h | 24 +++ .../tools/core/ensemble_stat/ensemble_stat.cc | 5 + met/src/tools/core/grid_stat/grid_stat.cc | 5 + .../tools/other/gen_ens_prod/gen_ens_prod.cc | 5 + test/config/GridStatConfig_rtma | 2 +- test/xml/unit_grid_stat.xml | 26 +++ 12 files changed, 336 insertions(+), 88 deletions(-) create mode 100644 met/src/basic/vx_util/handle_openmp.cc create mode 100644 met/src/basic/vx_util/handle_openmp.h diff --git a/met/configure.ac b/met/configure.ac index 2fc599e47a..7008748079 100644 --- a/met/configure.ac +++ b/met/configure.ac @@ -8,6 +8,13 @@ AC_CONFIG_HEADERS([config.h]) AM_INIT_AUTOMAKE([1.9 foreign]) +# OpenMP + +AC_OPENMP() + +CPPFLAGS="${CPPFLAGS} ${OPENMP_CFLAGS}" +LDFLAGS="${LDFLAGS} ${OPENMP_CFLAGS}" + # # Look for the NetCDF library # diff --git a/met/docs/Users_Guide/config_options.rst b/met/docs/Users_Guide/config_options.rst index e6d2349cc4..b56cb65d75 100644 --- a/met/docs/Users_Guide/config_options.rst +++ b/met/docs/Users_Guide/config_options.rst @@ -202,6 +202,12 @@ to use one configuration file rather than maintianing many very similar ones. An error in the syntax of a configuration file will result in an error from the MET tool stating the location of the parsing error. +Runtime Environment Variables +----------------------------- + +MET_BASE +^^^^^^^^ + The MET_BASE variable is defined in the code at compilation time as the path to the MET shared data. These are things like the default configuration files, common polygons and color scales. MET_BASE may be used in the MET configuration @@ -209,6 +215,9 @@ files when specifying paths and the appropriate path will be substituted in. If MET_BASE is defined as an environment variable, its value will be used instead of the one defined at compilation time. +MET_OBS_ERROR_TABLE +^^^^^^^^^^^^^^^^^^^ + The MET_OBS_ERROR_TABLE environment variable can be set to specify the location of an ASCII file defining observation error information. The default table can be found in the installed *share/met/table_files/obs_error_table.txt*. This @@ -236,6 +245,9 @@ values and randomly perturbed ensemble member values. Values less than MIN are reset to the mimimum value and values greater than MAX are reset to the maximum value. A value of NA indicates that the variable is unbounded. +MET_GRIB_TABLES +^^^^^^^^^^^^^^^ + The MET_GRIB_TABLES environment variable can be set to specify the location of custom GRIB tables. It can either be set to a specific file name or to a directory containing custom GRIB tables files. These file names must begin with @@ -289,9 +301,96 @@ References: | `NCEP WMO GRIB2 Documentation `_ | +OMP_NUM_THREADS +^^^^^^^^^^^^^^^ + +**Introduction** + +There are a number of different ways of parallelizing code. OpenMP offers +parallelism within a single shared-memory workstation or supercomputer node. +The programmer writes OpenMP directives into the code to parallelize +particular code regions. + +When a parallelized code region is reached, which we shall hereafter call a +parallel region, a number of threads are spawned and work is shared among them. +Running on different cores, this reduces the execution time. At the end of the +parallel region, the code returns to single-thread execution. + +A limited number of code regions are parallelized in MET. As a consequence, +there are limits to the overall speed gains acheivable. Only the parallel +regions of code will get faster with more threads, leaving the remaining +serial portions to dominate the runtime. + +Not all top-level executables use parallelized code. If OpenMP is available, +a log message will appear inviting the user to increase the number of threads +for faster runtimes. + +**Setting the number of threads** + +The number of threads is controlled by the environment variable +*OMP_NUM_THREADS* . For example, on a quad core machine, the user might choose +to run on 4 threads: + +.. code :: bash + + export OMP_NUM_THREADS=4 + +Alternatively, the variable may be specified as a prefix to the executable +itself. For example: + +.. code :: bash + + OMP_NUM_THREADS=4 + +The case where this variable remains unset is handled inside the code, which +defaults to a single thread. + +There are choices when deciding how many threads to use. To perform a single run +as fast as possible, it would likely be appropriate to use as many threads as +there are (physical) cores available on the specific system. However, it is not +a cast-iron guarantee that more threads will always translate into more speed. +In theory, there is a chance that running across multiple non-uniform memory +access (NUMA) regions may carry negative performance impacts. This has not been +observed in practice, however. + +A lower thread count is appropriate when time-to-solution is not so critical, +because cores remain idle when the code is not inside a parallel region. Fewer +threads typically means better resource utilization. + +**Which code is parallelized?** + +Regions of parallelized code are: + + * :code:`fractional_coverage (data_plane_util.cc)` + +Only the following top-level executables can presently benefit from OpenMP +parallelization: + + * :code:`grid_stat` + * :code:`ensemble_stat` + * :code:`grid_ens_prod` + +**Thread Binding** + +It is normally beneficial to bind threads to particular cores, sometimes called +*affinitization*. There are a few reasons for this, but at the very least it +guarantees that threads remain evenly distributed across the available cores. +Otherwise, the operating system may migrate threads between cores during a run. + +OpenMP provides some environment variables to handle this: :code:`OMP_PLACES` +and :code:`OMP_PROC_BIND`. We anticipate that the effect of setting only +:code:`OMP_PROC_BIND=true` would be neutral-to-positive. + +However, there are sometimes compiler-specific environment variables. Instead, +thread affinitization is sometimes handled by MPI launchers, since OpenMP is +often used in MPI codes to reduce intra-node communications. + +Where code is running in a production context, it is worth being familiar with +the binding / affinitization method on the particular system and building it +into any relevant scripting. Settings common to multiple tools -_________________________________ +--------------------------------- .. _exit_on_warning: @@ -2190,10 +2289,10 @@ are empty. Note: grib_code 11 is equivalent to obs_var "TMP". } Settings specific to individual tools -_____________________________________ +------------------------------------- EnsembleStatConfig_default -~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^^^ .. _ens: @@ -2505,7 +2604,7 @@ used for random assignment of ranks when they are tied. } MODEAnalysisConfig_default -~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^^^ MODE line options are used to create filters that determine which MODE output lines are read in and processed. The MODE line options are numerous. They @@ -2843,7 +2942,7 @@ MET User's Guide for a description of these attributes. MODEConfig_default -~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^ .. _quilt: @@ -3158,7 +3257,7 @@ much more flexible "regrid" option may be used instead. shift_right = 0; PB2NCConfig_default -~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^ The PB2NC tool filters out observations from PREPBUFR or BUFR files using the following criteria: @@ -3484,7 +3583,7 @@ stack (most quality controlled) or the bottom of the event stack (most raw). event_stack_flag = TOP; SeriesAnalysisConfig_default -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. _block_size: @@ -3539,7 +3638,7 @@ grid is large. } STATAnalysisConfig_default -~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^^^ .. _jobs: @@ -4008,7 +4107,7 @@ confidence intervals computed for the aggregated statistics. vif_flag = FALSE; WaveletStatConfig_default -~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^^ .. _grid_decomp_flag: @@ -4099,7 +4198,7 @@ similar to the "fcst_raw_plot" described in the "Settings common to multiple tools" section. WWMCARegridConfig_default -~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^^ .. _to_grid: diff --git a/met/docs/Users_Guide/installation.rst b/met/docs/Users_Guide/installation.rst index 963545ac2f..b4459fed0d 100644 --- a/met/docs/Users_Guide/installation.rst +++ b/met/docs/Users_Guide/installation.rst @@ -290,6 +290,14 @@ Enable compilation of the MODE-Graphics tool. Requires $MET_CAIRO and $MET_FREET Disable use of BLOCK4 in the compilation. Use this if you have trouble using PrepBUFR files. +.. code-block:: none + + --disable-openmp + +Disable compilation of OpenMP directives within the code which allows some code +regions to benefit from thread-parallel execution. Runtime environment variable +:code:`OMP_NUM_THREADS` controls the number of threads. + Run the configure script with the **-help** argument to see the full list of configuration options. Make Targets diff --git a/met/src/basic/vx_util/Makefile.am b/met/src/basic/vx_util/Makefile.am index 57b7a100db..ae466f0963 100644 --- a/met/src/basic/vx_util/Makefile.am +++ b/met/src/basic/vx_util/Makefile.am @@ -69,6 +69,7 @@ libvx_util_a_SOURCES = ascii_table.cc ascii_table.h \ GridOffset.h GridOffset.cc \ observation.h observation.cc \ stat_column_defs.h \ + handle_openmp.h handle_openmp.cc \ RectangularTemplate.h RectangularTemplate.cc $(OPT_PYTHON_SOURCES) libvx_util_a_CPPFLAGS = ${MET_CPPFLAGS} diff --git a/met/src/basic/vx_util/data_plane_util.cc b/met/src/basic/vx_util/data_plane_util.cc index 3f5b5f7278..39fbb67679 100644 --- a/met/src/basic/vx_util/data_plane_util.cc +++ b/met/src/basic/vx_util/data_plane_util.cc @@ -16,6 +16,10 @@ using namespace std; #include #include +#ifdef _OPENMP + #include "omp.h" +#endif + #include "data_plane_util.h" #include "interp_util.h" #include "two_to_one.h" @@ -247,83 +251,95 @@ void fractional_coverage(const DataPlane &dp, DataPlane &frac_dp, } } - // Build the grid template - GridTemplateFactory gtf; - GridTemplate* gt = gtf.buildGT(shape, width, wrap_lon); - - mlog << Debug(3) - << "Computing fractional coverage field using the " - << t.get_str() << " threshold and the " - << interpmthd_to_string(InterpMthd_Nbrhd) << "(" << gt->size() - << ") " << gt->getClassName() << " interpolation method.\n"; - - // Initialize the fractional coverage field - frac_dp = dp; - frac_dp.set_constant(bad_data_double); - - // Compute the fractional coverage meeting the threshold criteria - for(x=0; xgetFirstInGrid(x, y, dp.nx(), dp.ny()); - gp != NULL; - gp = gt->getNextInGrid()) { - if(is_bad_data(v = dp.get(gp->x, gp->y))) continue; - n_vld++; - if(t.check(v, - (use_climo ? cmn->get(gp->x, gp->y) : bad), - (use_climo ? csd->get(gp->x, gp->y) : bad))) n_thr++; - } - } - // Subtract off the bottom edge, shift up, and add the top. - else { - - // Subtract points from the the bottom edge - for(gp = gt->getFirstInBotEdge(); - gp != NULL; - gp = gt->getNextInBotEdge()) { - if(is_bad_data(v = dp.get(gp->x, gp->y))) continue; - n_vld--; - if(t.check(v, - (use_climo ? cmn->get(gp->x, gp->y) : bad), - (use_climo ? csd->get(gp->x, gp->y) : bad))) n_thr--; - } - - // Increment Y - gt->incBaseY(1); - - // Add points from the the top edge - for(gp = gt->getFirstInTopEdge(); - gp != NULL; - gp = gt->getNextInTopEdge()) { - if(is_bad_data(v = dp.get(gp->x, gp->y))) continue; - n_vld++; - if(t.check(v, - (use_climo ? cmn->get(gp->x, gp->y) : bad), - (use_climo ? csd->get(gp->x, gp->y) : bad))) n_thr++; - } - } - - // Check for enough valid data and compute fractional coverage - if((double)(n_vld)/gt->size() >= vld_t && n_vld != 0) { - frac_dp.set((double) n_thr/n_vld, x, y); - } - - } // end for y - - // Increment X - if(x < (dp.nx() - 1)) gt->incBaseX(1); - - } // end for x - - delete gt; +#pragma omp parallel default(none) \ + shared(mlog, dp, frac_dp, width, wrap_lon, t) \ + shared(use_climo, cmn, csd, vld_t, bad) \ + private(x, y, n_vld, n_thr, gp, v) + { + + // Build the grid template + GridTemplateFactory gtf; + GridTemplate* gt = gtf.buildGT(shape, width, wrap_lon); + +#pragma omp single + { + mlog << Debug(3) + << "Computing fractional coverage field using the " + << t.get_str() << " threshold and the " + << interpmthd_to_string(InterpMthd_Nbrhd) << "(" << gt->size() + << ") " << gt->getClassName() << " interpolation method.\n"; + + // Initialize the fractional coverage field + frac_dp = dp; + frac_dp.set_constant(bad_data_double); + } + + // Compute the fractional coverage meeting the threshold criteria +#pragma omp for schedule (static) + for(x=0; xgetFirstInGrid(x, y, dp.nx(), dp.ny()); + gp != NULL; + gp = gt->getNextInGrid()) { + if(is_bad_data(v = dp.get(gp->x, gp->y))) continue; + n_vld++; + if(t.check(v, + (use_climo ? cmn->get(gp->x, gp->y) : bad), + (use_climo ? csd->get(gp->x, gp->y) : bad))) n_thr++; + } + } + // Subtract off the bottom edge, shift up, and add the top. + else { + + // Subtract points from the the bottom edge + for(gp = gt->getFirstInBotEdge(); + gp != NULL; + gp = gt->getNextInBotEdge()) { + if(is_bad_data(v = dp.get(gp->x, gp->y))) continue; + n_vld--; + if(t.check(v, + (use_climo ? cmn->get(gp->x, gp->y) : bad), + (use_climo ? csd->get(gp->x, gp->y) : bad))) n_thr--; + } + + // Increment Y + gt->incBaseY(1); + + // Add points from the the top edge + for(gp = gt->getFirstInTopEdge(); + gp != NULL; + gp = gt->getNextInTopEdge()) { + if(is_bad_data(v = dp.get(gp->x, gp->y))) continue; + n_vld++; + if(t.check(v, + (use_climo ? cmn->get(gp->x, gp->y) : bad), + (use_climo ? csd->get(gp->x, gp->y) : bad))) n_thr++; + } + } + + // Check for enough valid data and compute fractional coverage + if((double)(n_vld)/gt->size() >= vld_t && n_vld != 0) { + frac_dp.set((double) n_thr/n_vld, x, y); + } + + } // end for y + + // Increment X + if(x < (dp.nx() - 1)) gt->incBaseX(1); + + } // end for x + + delete gt; + + } // End of omp parallel return; } diff --git a/met/src/basic/vx_util/handle_openmp.cc b/met/src/basic/vx_util/handle_openmp.cc new file mode 100644 index 0000000000..f23d09b4ca --- /dev/null +++ b/met/src/basic/vx_util/handle_openmp.cc @@ -0,0 +1,52 @@ +// *=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=* +// ** Copyright UCAR (c) 1992 - 2021 +// ** University Corporation for Atmospheric Research (UCAR) +// ** National Center for Atmospheric Research (NCAR) +// ** Research Applications Lab (RAL) +// ** P.O.Box 3000, Boulder, Colorado, 80307-3000, USA +// *=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=* + +/////////////////////////////////////////////////////////////////////////////// + +#include + +#ifdef _OPENMP + #include "omp.h" +#endif + +#include "vx_log.h" +#include "handle_openmp.h" + +/////////////////////////////////////////////////////////////////////////////// + +void init_openmp() { + +#ifdef _OPENMP + + // If OMP_NUM_THREADS was not set, use the OpenMP API to set the thread count + // to 1 thread only. + const char* env_omp_num_threads = std::getenv("OMP_NUM_THREADS"); + if (!env_omp_num_threads) { + mlog << Debug(2) << "OMP_NUM_THREADS is not set." + << " Defaulting to 1 thread." + << " Recommend setting OMP_NUM_THREADS for faster runtimes.\n"; + omp_set_num_threads(1); + } + +#pragma omp parallel +#pragma omp single + { + mlog << Debug(2) << "OpenMP running on " + << omp_get_num_threads() << " thread(s).\n"; + } + +#else /* _OPENMP */ + + mlog << Debug(2) << "OpenMP disabled.\n"; + +#endif /* _OPENMP */ + +} + +/////////////////////////////////////////////////////////////////////////////// + diff --git a/met/src/basic/vx_util/handle_openmp.h b/met/src/basic/vx_util/handle_openmp.h new file mode 100644 index 0000000000..e2ea2e6a38 --- /dev/null +++ b/met/src/basic/vx_util/handle_openmp.h @@ -0,0 +1,24 @@ +// *=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=* +// ** Copyright UCAR (c) 1992 - 2021 +// ** University Corporation for Atmospheric Research (UCAR) +// ** National Center for Atmospheric Research (NCAR) +// ** Research Applications Lab (RAL) +// ** P.O.Box 3000, Boulder, Colorado, 80307-3000, USA +// *=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=* + +/////////////////////////////////////////////////////////////////////////////// + +#ifndef __HANDLE_OPENMP_H___ +#define __HANDLE_OPENMP_H___ + +/////////////////////////////////////////////////////////////////////////////// + +void init_openmp(); + +/////////////////////////////////////////////////////////////////////////////// + +#endif /* __HANDLE_OPENMP_H__ */ + +/////////////////////////////////////////////////////////////////////////////// + + diff --git a/met/src/tools/core/ensemble_stat/ensemble_stat.cc b/met/src/tools/core/ensemble_stat/ensemble_stat.cc index fe5d160157..ab296a2528 100644 --- a/met/src/tools/core/ensemble_stat/ensemble_stat.cc +++ b/met/src/tools/core/ensemble_stat/ensemble_stat.cc @@ -92,6 +92,8 @@ using namespace std; #include "nc_obs_util.h" #include "nc_point_obs_in.h" +#include "handle_openmp.h" + //////////////////////////////////////////////////////////////////////// static void process_command_line (int, char **); @@ -171,6 +173,9 @@ static void set_compress(const StringArray &); int main(int argc, char *argv[]) { + // Set up OpenMP (if enabled) + init_openmp(); + // Set handler to be called for memory allocation error set_new_handler(oom); diff --git a/met/src/tools/core/grid_stat/grid_stat.cc b/met/src/tools/core/grid_stat/grid_stat.cc index f420e568c4..e7055ed596 100644 --- a/met/src/tools/core/grid_stat/grid_stat.cc +++ b/met/src/tools/core/grid_stat/grid_stat.cc @@ -127,6 +127,8 @@ using namespace std; #include #include +#include "handle_openmp.h" + #include "grid_stat.h" #include "vx_statistics.h" @@ -185,6 +187,9 @@ static bool read_data_plane(VarInfo* info, DataPlane& dp, Met2dDataFile* mtddf, int main(int argc, char *argv[]) { + // Set up OpenMP (if enabled) + init_openmp(); + // Set handler to be called for memory allocation error set_new_handler(oom); diff --git a/met/src/tools/other/gen_ens_prod/gen_ens_prod.cc b/met/src/tools/other/gen_ens_prod/gen_ens_prod.cc index 1f634c38b2..786a56b390 100644 --- a/met/src/tools/other/gen_ens_prod/gen_ens_prod.cc +++ b/met/src/tools/other/gen_ens_prod/gen_ens_prod.cc @@ -45,6 +45,8 @@ using namespace std; #include "nc_obs_util.h" #include "nc_point_obs_in.h" +#include "handle_openmp.h" + //////////////////////////////////////////////////////////////////////// static void process_command_line(int, char **); @@ -81,6 +83,9 @@ static void set_ctrl_file (const StringArray &); int main(int argc, char *argv[]) { + // Set up OpenMP (if enabled) + init_openmp(); + // Set handler to be called for memory allocation error set_new_handler(oom); diff --git a/test/config/GridStatConfig_rtma b/test/config/GridStatConfig_rtma index 178a1269b1..2c3d3ba058 100644 --- a/test/config/GridStatConfig_rtma +++ b/test/config/GridStatConfig_rtma @@ -111,7 +111,7 @@ interp = { type = [ { method = NEAREST; width = 1; }, { method = UW_MEAN; width = 3; }, - { method = MIN; width = 1; }, + { method = MIN; width = 3; }, { method = MAX; width = 3; } ]; } diff --git a/test/xml/unit_grid_stat.xml b/test/xml/unit_grid_stat.xml index 1d562778f7..c31455c0ce 100644 --- a/test/xml/unit_grid_stat.xml +++ b/test/xml/unit_grid_stat.xml @@ -75,6 +75,32 @@ + + &MET_BIN;/grid_stat + + OMP_NUM_THREADS 2 + OUTPUT_PREFIX GRIB2_NAM_RTMA_NP2 + + \ + &DATA_DIR_MODEL;/grib2/nam/nam_2012040900_F012_gRtma.grib2 \ + &DATA_DIR_OBS;/rtma/rtma_2012051712_F000.grib2 \ + &CONFIG_DIR;/GridStatConfig_rtma \ + -outdir &OUTPUT_DIR;/grid_stat -v 1 + + + &OUTPUT_DIR;/grid_stat/grid_stat_GRIB2_NAM_RTMA_NP2_120000L_20120409_120000V.stat + &OUTPUT_DIR;/grid_stat/grid_stat_GRIB2_NAM_RTMA_NP2_120000L_20120409_120000V_fho.txt + &OUTPUT_DIR;/grid_stat/grid_stat_GRIB2_NAM_RTMA_NP2_120000L_20120409_120000V_ctc.txt + &OUTPUT_DIR;/grid_stat/grid_stat_GRIB2_NAM_RTMA_NP2_120000L_20120409_120000V_cts.txt + &OUTPUT_DIR;/grid_stat/grid_stat_GRIB2_NAM_RTMA_NP2_120000L_20120409_120000V_cnt.txt + &OUTPUT_DIR;/grid_stat/grid_stat_GRIB2_NAM_RTMA_NP2_120000L_20120409_120000V_sl1l2.txt + &OUTPUT_DIR;/grid_stat/grid_stat_GRIB2_NAM_RTMA_NP2_120000L_20120409_120000V_nbrctc.txt + &OUTPUT_DIR;/grid_stat/grid_stat_GRIB2_NAM_RTMA_NP2_120000L_20120409_120000V_nbrcts.txt + &OUTPUT_DIR;/grid_stat/grid_stat_GRIB2_NAM_RTMA_NP2_120000L_20120409_120000V_nbrcnt.txt + &OUTPUT_DIR;/grid_stat/grid_stat_GRIB2_NAM_RTMA_NP2_120000L_20120409_120000V_pairs.nc + + + &MET_BIN;/grid_stat From e96a2116e88cf58d152f95e5d9f1d9cdc587534d Mon Sep 17 00:00:00 2001 From: George McCabe <23407799+georgemccabe@users.noreply.github.com> Date: Thu, 13 Jan 2022 19:33:04 -0700 Subject: [PATCH 62/82] feature 1695 ensemble single file (ensemble_stat) (#2007) Co-authored-by: johnhg --- met/data/config/EnsembleStatConfig_default | 7 + met/docs/Users_Guide/ensemble-stat.rst | 46 +- met/docs/Users_Guide/gen-ens-prod.rst | 2 +- met/src/libcode/vx_data2d/var_info.cc | 123 ++++++ met/src/libcode/vx_data2d/var_info.h | 48 +++ .../vx_statistics/pair_data_ensemble.cc | 29 +- .../vx_statistics/pair_data_ensemble.h | 4 +- .../tools/core/ensemble_stat/ensemble_stat.cc | 392 +++++++++++------- .../tools/core/ensemble_stat/ensemble_stat.h | 10 +- .../ensemble_stat/ensemble_stat_conf_info.cc | 227 ++++++++-- .../ensemble_stat/ensemble_stat_conf_info.h | 15 +- .../tools/other/gen_ens_prod/gen_ens_prod.cc | 22 +- .../gen_ens_prod/gen_ens_prod_conf_info.cc | 99 +---- .../gen_ens_prod/gen_ens_prod_conf_info.h | 53 +-- scripts/environment/development.seneca | 1 - test/config/EnsembleStatConfig | 7 + test/config/EnsembleStatConfig_MASK_SID | 7 + test/config/EnsembleStatConfig_climo | 7 + test/config/EnsembleStatConfig_grid_weight | 7 + test/config/EnsembleStatConfig_one_cdf_bin | 7 + test/config/EnsembleStatConfig_python | 7 + test/config/EnsembleStatConfig_qty_inc_exc | 7 + .../EnsembleStatConfig_single_file_grib | 317 ++++++++++++++ test/config/EnsembleStatConfig_single_file_nc | 315 ++++++++++++++ test/xml/unit_ensemble_stat.xml | 82 ++++ 25 files changed, 1464 insertions(+), 377 deletions(-) create mode 100644 test/config/EnsembleStatConfig_single_file_grib create mode 100644 test/config/EnsembleStatConfig_single_file_nc diff --git a/met/data/config/EnsembleStatConfig_default b/met/data/config/EnsembleStatConfig_default index ea10765e17..339ab939be 100644 --- a/met/data/config/EnsembleStatConfig_default +++ b/met/data/config/EnsembleStatConfig_default @@ -62,6 +62,13 @@ ens = { ]; } +// +// IDs for ensemble members +// Only set if processing a single file with all ensembles +// +ens_member_ids = []; +control_id = ""; + //////////////////////////////////////////////////////////////////////////////// // diff --git a/met/docs/Users_Guide/ensemble-stat.rst b/met/docs/Users_Guide/ensemble-stat.rst index 266519da4f..5c6277b179 100644 --- a/met/docs/Users_Guide/ensemble-stat.rst +++ b/met/docs/Users_Guide/ensemble-stat.rst @@ -97,7 +97,7 @@ Optional arguments for ensemble_stat 6. To override the simple ensemble mean value of the input ensemble members for the ECNT, SSVAR, and ORANK line types, the **-ens_mean file** option specifies an ensemble mean model data file. This option replaces the **-ssvar_mean file** option from earlier versions of MET. -7. The **-ctrl file** option specifies an ensemble control member data file. The control member is included in the computation of the ensemble mean but excluded from the spread. The control file should not appear in the list of ensemble member files. +7. The **-ctrl file** option specifies an ensemble control member data file. The control member is included in the computation of the ensemble mean but excluded from the spread. The control file should not appear in the list of ensemble member files (unless processing a single file that contains all ensemble members). 8. To filter point observations by time, use **-obs_valid_beg time** in YYYYMMDD[_HH[MMSS]] format to set the beginning of the matching observation time window. @@ -188,6 +188,50 @@ For each **field** listed in the forecast field, give the name and vertical or a _______________________ +.. code-block:: none + + ens_member_ids = []; + control_id = ""; + +The **ens_member_ids** array is only used if reading a single file that contains all ensemble members. +It should contain a list of string identifiers that are substituted into the **ens** and/or **fcst** dictionary fields +to determine which data to read from the file. +The length of the array determines how many ensemble members will be processed for a given field. +Each value in the array will replace the text **MET_ENS_MEMBER_ID**. + +**NetCDF Example:** + +.. code-block:: none + + ens = { + field = [ + { + name = "fcst"; + level = "(MET_ENS_MEMBER_ID,0,*,*)"; + } + ]; + } + +**GRIB Example:** + +.. code-block:: none + + ens = { + field = [ + { + name = "fcst"; + level = "L0"; + GRIB_ens = "MET_ENS_MEMBER_ID"; + } + ]; + } + +**control_id** is a string that is substituted in the same way as the **ens_member_ids** values +to read a control member. This value is only used when the **-ctrl** command line argument is +used. The value should not be found in the **ens_member_ids** array. + +_______________________ + .. code-block:: none nbrhd_prob = { diff --git a/met/docs/Users_Guide/gen-ens-prod.rst b/met/docs/Users_Guide/gen-ens-prod.rst index 459a9778f1..6d6e562ee0 100644 --- a/met/docs/Users_Guide/gen-ens-prod.rst +++ b/met/docs/Users_Guide/gen-ens-prod.rst @@ -167,7 +167,7 @@ Each value in the array will replace the text **MET_ENS_MEMBER_ID**. { name = "fcst"; level = "L0"; - GRIB_ens = "+MET_ENS_MEMBER_ID"; + GRIB_ens = "MET_ENS_MEMBER_ID"; } ]; } diff --git a/met/src/libcode/vx_data2d/var_info.cc b/met/src/libcode/vx_data2d/var_info.cc index 81e8f11252..009fdfe4c3 100644 --- a/met/src/libcode/vx_data2d/var_info.cc +++ b/met/src/libcode/vx_data2d/var_info.cc @@ -790,3 +790,126 @@ int parse_set_attr_flag(Dictionary &dict, const char *key) { } /////////////////////////////////////////////////////////////////////////////// + +//////////////////////////////////////////////////////////////////////// +// +// Code for class EnsVarInfo +// +//////////////////////////////////////////////////////////////////////// + +EnsVarInfo::EnsVarInfo() { + ctrl_info = NULL; +} + +//////////////////////////////////////////////////////////////////////// + +EnsVarInfo::~EnsVarInfo() { + clear(); +} +/////////////////////////////////////////////////////////////////////////////// + +EnsVarInfo::EnsVarInfo(const EnsVarInfo &f) { + + clear(); + + assign(f); +} + +//////////////////////////////////////////////////////////////////////// + +void EnsVarInfo::clear() { + vector::const_iterator it; + for(it = inputs.begin(); it != inputs.end(); it++) { + if((*it).var_info) { delete (*it).var_info; } + } + + if(ctrl_info) { delete ctrl_info; } +} + +/////////////////////////////////////////////////////////////////////////////// + +void EnsVarInfo::assign(const EnsVarInfo &v) { + + // Copy + inputs = v.inputs; + ctrl_info = v.ctrl_info; + + nc_var_str = v.nc_var_str; + cat_ta = v.cat_ta; + raw_magic_str = v.raw_magic_str; + + return; +} + +//////////////////////////////////////////////////////////////////////// + +void EnsVarInfo::add_input(InputInfo input) { + inputs.push_back(input); +} + +//////////////////////////////////////////////////////////////////////// + +int EnsVarInfo::inputs_n() { + return inputs.size(); +} + +//////////////////////////////////////////////////////////////////////// + +void EnsVarInfo::set_ctrl(VarInfo * ctrl) { + ctrl_info = ctrl; +} + +//////////////////////////////////////////////////////////////////////// + +VarInfo * EnsVarInfo::get_ctrl(int index) { + if(ctrl_info) { + return ctrl_info; + } + return inputs[index].var_info; +} + +//////////////////////////////////////////////////////////////////////// + +VarInfo * EnsVarInfo::get_var_info(int index) { + if(inputs[index].var_info) { + return inputs[index].var_info; + } + return inputs[0].var_info; +} + +//////////////////////////////////////////////////////////////////////// + +ConcatString EnsVarInfo::get_file(int index) { + int file_index = inputs[index].file_index; + return (*inputs[index].file_list)[file_index]; +} + +//////////////////////////////////////////////////////////////////////// + +int EnsVarInfo::get_file_index(int index) { + return inputs[index].file_index; +} + +//////////////////////////////////////////////////////////////////////// + +ConcatString raw_magic_str(Dictionary i_edict, GrdFileType file_type) { + ConcatString magic_str; + + ConcatString name = i_edict.lookup_string("name"); + ConcatString level = i_edict.lookup_string("level", false); + + if(level.empty()) { + return name; + } + + if(level[0] != '(') { + magic_str << name << "/" << level; + } else { + magic_str << name << level; + } + + return magic_str; + +} + +//////////////////////////////////////////////////////////////////////// diff --git a/met/src/libcode/vx_data2d/var_info.h b/met/src/libcode/vx_data2d/var_info.h index 2d80138878..2bd1f42973 100644 --- a/met/src/libcode/vx_data2d/var_info.h +++ b/met/src/libcode/vx_data2d/var_info.h @@ -251,6 +251,54 @@ inline unixtime VarInfo::valid_attr() const { return(SetAttrValid); } inline int VarInfo::lead_attr() const { return(SetAttrLead); } inline int VarInfo::accum_attr() const { return(SetAttrAccum); } +//////////////////////////////////////////////////////////////////////// + +// +// Struct to store input file and field information +// + +struct InputInfo { + VarInfo * var_info; // Variable information to read + int file_index; // Index in file_list of file to read + StringArray * file_list; // Array of files (unallocated) +}; + +//////////////////////////////////////////////////////////////////////// + +// +// Struct to store ensemble information +// +class EnsVarInfo { + +private: + vector inputs; // Vector of InputInfo + VarInfo * ctrl_info; // Field info for control member +public: + EnsVarInfo(); + ~EnsVarInfo(); + EnsVarInfo(const EnsVarInfo &); + + void clear(); + void assign(const EnsVarInfo &); + + void add_input(InputInfo); + int inputs_n(); + + void set_ctrl(VarInfo *); + VarInfo * get_ctrl(int); + + // Get VarInfo from first InputInfo if requested VarInfo is NULL + VarInfo * get_var_info(int index=0); + ConcatString get_file(int index=0); + int get_file_index(int index=0); + + ConcatString nc_var_str; // Ensemble variable name strings + ThreshArray cat_ta; // Ensemble categorical thresholds + ConcatString raw_magic_str; // Magic string w/o var substitution +}; + +ConcatString raw_magic_str(Dictionary i_edict, GrdFileType file_type); + /////////////////////////////////////////////////////////////////////////////// #endif // __VAR_INFO_H__ diff --git a/met/src/libcode/vx_statistics/pair_data_ensemble.cc b/met/src/libcode/vx_statistics/pair_data_ensemble.cc index 6f6471b63a..2af587f2ea 100644 --- a/met/src/libcode/vx_statistics/pair_data_ensemble.cc +++ b/met/src/libcode/vx_statistics/pair_data_ensemble.cc @@ -891,7 +891,7 @@ VxPairDataEnsemble & VxPairDataEnsemble::operator=(const VxPairDataEnsemble &vx_ void VxPairDataEnsemble::init_from_scratch() { - fcst_info = (VarInfo *) 0; + fcst_info = (EnsVarInfo *) 0; climo_info = (VarInfo *) 0; obs_info = (VarInfo *) 0; pd = (PairDataEnsemble ***) 0; @@ -910,7 +910,7 @@ void VxPairDataEnsemble::init_from_scratch() { void VxPairDataEnsemble::clear() { int i, j, k; - if(fcst_info) { delete fcst_info; fcst_info = (VarInfo *) 0; } + if(fcst_info) { delete fcst_info; fcst_info = (EnsVarInfo *) 0; } if(climo_info) { delete climo_info; climo_info = (VarInfo *) 0; } if(obs_info) { delete obs_info; obs_info = (VarInfo *) 0; } @@ -993,15 +993,14 @@ void VxPairDataEnsemble::assign(const VxPairDataEnsemble &vx_pd) { //////////////////////////////////////////////////////////////////////// -void VxPairDataEnsemble::set_fcst_info(VarInfo *info) { +void VxPairDataEnsemble::set_fcst_info(EnsVarInfo *info) { VarInfoFactory f; // Deallocate, if necessary - if(fcst_info) { delete fcst_info; fcst_info = (VarInfo *) 0; } + if(fcst_info) { delete fcst_info; fcst_info = (EnsVarInfo *) 0; } // Perform a deep copy - fcst_info = f.new_var_info(info->file_type()); - *fcst_info = *info; + fcst_info = new EnsVarInfo(*info); return; } @@ -1458,7 +1457,7 @@ void VxPairDataEnsemble::add_point_obs(float *hdr_arr, int *hdr_typ_arr, // below the observation point. else { // Interpolate using the observation pressure level or height - to_lvl = (fcst_info->level().type() == LevelType_Pres ? + to_lvl = (fcst_info->get_var_info()->level().type() == LevelType_Pres ? obs_lvl : obs_hgt); find_vert_lvl(climo_mn_dpa, to_lvl, cmn_lvl_blw, cmn_lvl_abv); } @@ -1472,7 +1471,7 @@ void VxPairDataEnsemble::add_point_obs(float *hdr_arr, int *hdr_typ_arr, // levels above and below the observation point. else { // Interpolate using the observation pressure level or height - to_lvl = (fcst_info->level().type() == LevelType_Pres ? + to_lvl = (fcst_info->get_var_info()->level().type() == LevelType_Pres ? obs_lvl : obs_hgt); find_vert_lvl(climo_sd_dpa, to_lvl, csd_lvl_blw, csd_lvl_abv); } @@ -1486,7 +1485,7 @@ void VxPairDataEnsemble::add_point_obs(float *hdr_arr, int *hdr_typ_arr, } // Set flag for specific humidity - bool spfh_flag = fcst_info->is_specific_humidity() && + bool spfh_flag = fcst_info->get_var_info()->is_specific_humidity() && obs_info->is_specific_humidity(); // Store pointer to ObsErrorEntry @@ -1561,7 +1560,7 @@ void VxPairDataEnsemble::add_point_obs(float *hdr_arr, int *hdr_typ_arr, // Compute the interpolated climatology values using the // observation pressure level or height - to_lvl = (fcst_info->level().type() == LevelType_Pres ? + to_lvl = (fcst_info->get_var_info()->level().type() == LevelType_Pres ? obs_lvl : obs_hgt); // Compute the interpolated climatology mean @@ -1570,7 +1569,7 @@ void VxPairDataEnsemble::add_point_obs(float *hdr_arr, int *hdr_typ_arr, pd[0][0][k].interp_mthd, pd[0][0][k].interp_wdth, pd[0][0][k].interp_shape, gr.wrap_lon(), interp_thresh, spfh_flag, - fcst_info->level().type(), + fcst_info->get_var_info()->level().type(), to_lvl, cmn_lvl_blw, cmn_lvl_abv); // Check for bad data @@ -1597,7 +1596,7 @@ void VxPairDataEnsemble::add_point_obs(float *hdr_arr, int *hdr_typ_arr, pd[0][0][k].interp_mthd, pd[0][0][k].interp_wdth, pd[0][0][k].interp_shape, gr.wrap_lon(), interp_thresh, spfh_flag, - fcst_info->level().type(), + fcst_info->get_var_info()->level().type(), to_lvl, csd_lvl_blw, csd_lvl_abv); // Check for bad data @@ -1630,7 +1629,7 @@ void VxPairDataEnsemble::add_ens(int member, bool mn, Grid &gr) { double to_lvl, fcst_v; // Set flag for specific humidity - bool spfh_flag = fcst_info->is_specific_humidity() && + bool spfh_flag = fcst_info->get_var_info()->is_specific_humidity() && obs_info->is_specific_humidity(); // Loop through all the PairDataEnsemble objects and interpolate @@ -1642,7 +1641,7 @@ void VxPairDataEnsemble::add_ens(int member, bool mn, Grid &gr) { for(l=0; llevel().type() == LevelType_Pres ? + to_lvl = (fcst_info->get_var_info()->level().type() == LevelType_Pres ? pd[i][j][k].lvl_na[l] : pd[i][j][k].elv_na[l]); // For a single forecast field @@ -1668,7 +1667,7 @@ void VxPairDataEnsemble::add_ens(int member, bool mn, Grid &gr) { pd[0][0][k].interp_shape, gr.wrap_lon(), interp_thresh, spfh_flag, - fcst_info->level().type(), + fcst_info->get_var_info()->level().type(), to_lvl, f_lvl_blw, f_lvl_abv); // Store the ensemble mean diff --git a/met/src/libcode/vx_statistics/pair_data_ensemble.h b/met/src/libcode/vx_statistics/pair_data_ensemble.h index bee6b11838..67e0319d27 100644 --- a/met/src/libcode/vx_statistics/pair_data_ensemble.h +++ b/met/src/libcode/vx_statistics/pair_data_ensemble.h @@ -174,7 +174,7 @@ class VxPairDataEnsemble { // ////////////////////////////////////////////////////////////////// - VarInfo *fcst_info; // Forecast field, allocated by VarInfoFactory + EnsVarInfo *fcst_info; // Forecast field, allocated by EnsVarInfo VarInfo *climo_info; // Climatology field, allocated by VarInfoFactory VarInfo *obs_info; // Observation field, allocated by VarInfoFactory @@ -233,7 +233,7 @@ class VxPairDataEnsemble { void clear(); - void set_fcst_info(VarInfo *); + void set_fcst_info(EnsVarInfo *); void set_climo_info(VarInfo *); void set_obs_info(VarInfo *); diff --git a/met/src/tools/core/ensemble_stat/ensemble_stat.cc b/met/src/tools/core/ensemble_stat/ensemble_stat.cc index ab296a2528..0299932cf5 100644 --- a/met/src/tools/core/ensemble_stat/ensemble_stat.cc +++ b/met/src/tools/core/ensemble_stat/ensemble_stat.cc @@ -128,7 +128,7 @@ static void do_rps (const EnsembleStatVxOpt &, const PairDataEnsemble *); static void clear_counts(); -static void track_counts(int, const DataPlane &, bool); +static void track_counts(EnsVarInfo *, const DataPlane &, bool); static ConcatString get_ens_mn_var_name(int); @@ -138,12 +138,12 @@ static void setup_table (AsciiTable &); static void build_outfile_name(unixtime, const char *, ConcatString &); -static void write_ens_nc(int, DataPlane &); -static void write_ens_var_float(int, float *, const DataPlane &, +static void write_ens_nc(EnsVarInfo *, int, DataPlane &); +static void write_ens_var_float(EnsVarInfo *, float *, const DataPlane &, const char *, const char *); -static void write_ens_var_int(int, int *, const DataPlane &, +static void write_ens_var_int(EnsVarInfo *, int *, const DataPlane &, const char *, const char *); -static void write_ens_data_plane(int, const DataPlane &, const DataPlane &, +static void write_ens_data_plane(EnsVarInfo *, const DataPlane &, const DataPlane &, const char *, const char *); static void write_orank_nc(PairDataEnsemble &, DataPlane &, int, int, int); @@ -256,7 +256,7 @@ void process_command_line(int argc, char **argv) { // if(is_integer(cline[0].c_str()) == 0) { ens_file_list = parse_ascii_file_list(cline[0].c_str()); - n_ens = ens_file_list.n(); + n_ens_files = ens_file_list.n(); } else { usage(); @@ -270,21 +270,21 @@ void process_command_line(int argc, char **argv) { // if(is_integer(cline[0].c_str()) == 1) { - n_ens = atoi(cline[0].c_str()); + n_ens_files = atoi(cline[0].c_str()); - if(n_ens <= 0) { + if(n_ens_files <= 0) { mlog << Error << "\nprocess_command_line() -> " << "the number of ensemble member files must be >= 1 (" - << n_ens << ")\n\n"; + << n_ens_files << ")\n\n"; exit(1); } - if((cline.n() - 2) == n_ens) { + if((cline.n() - 2) == n_ens_files) { // // Add each of the ensemble members to the list of files. // - for(i=1; i<=n_ens; i++) ens_file_list.add(cline[i]); + for(i=1; i<=n_ens_files; i++) ens_file_list.add(cline[i]); } else { usage(); @@ -293,16 +293,19 @@ void process_command_line(int argc, char **argv) { } // Check for at least one valid input ensemble file - if(ens_file_list.n() == 0) { + if(n_ens_files == 0) { mlog << Error << "\nprocess_command_line() -> " << "no valid input ensemble member files specified!\n\n"; exit(1); } + // Copy ensemble file list to forecast file list + fcst_file_list = ens_file_list; + // Prepend the control member, if specified if(ctrl_file.nonempty()) { - if(ens_file_list.has(ctrl_file)) { + if(ens_file_list.has(ctrl_file) && n_ens_files != 1) { mlog << Error << "\nprocess_command_line() -> " << "the ensemble control file should not appear in the list " << "of ensemble member files:\n" << ctrl_file << "\n\n"; @@ -310,8 +313,9 @@ void process_command_line(int argc, char **argv) { } ctrl_index = 0; - ens_file_list.insert(ctrl_index, ctrl_file.c_str()); - n_ens++; + + // Add control file to beginning of forecast file list + fcst_file_list.insert(ctrl_index, ctrl_file.c_str()); } // Check that the end_ut >= beg_ut @@ -387,7 +391,7 @@ void process_command_line(int argc, char **argv) { } // Process the configuration - conf_info.process_config(etype, otype, grid_obs_flag, point_obs_flag, use_var_id); + conf_info.process_config(etype, otype, grid_obs_flag, point_obs_flag, use_var_id, &ens_file_list, &fcst_file_list, ctrl_file.nonempty()); // Set the model name shc.set_model(conf_info.model.c_str()); @@ -402,12 +406,15 @@ void process_command_line(int argc, char **argv) { // List the input ensemble files mlog << Debug(1) << "Ensemble Files[" - << ens_file_list.n() << "]:\n"; - for(i=0; i 0) { mlog << Debug(1) << "Gridded Observation Files[" @@ -427,7 +434,7 @@ void process_command_line(int argc, char **argv) { } // Check for missing non-python ensemble files - for(i=0; i " + << "can't open input forecast file: " + << fcst_file_list[i] << "\n\n"; + fcst_file_vld.add(0); + } + else { + fcst_file_vld.add(1); + } + } + // Set flag to indicate whether verification is to be performed if((point_obs_flag || grid_obs_flag) && (conf_info.get_n_vx() > 0) && @@ -489,11 +511,17 @@ void process_command_line(int argc, char **argv) { if(!conf_info.nc_info.do_mean) { mlog << Warning << "\nprocess_command_line() -> " << "enabling NetCDF ensemble mean computation to be used " - << "in verificaiton.\n\n"; + << "in verification.\n\n"; conf_info.nc_info.do_mean = true; } } + if(conf_info.control_id.nonempty() && ctrl_file.empty()) { + mlog << Warning << "\nprocess_command_line() -> " + << "control_id is set in the config file but " + << "control file is not provided with -ctrl argument\n\n"; + } + // Deallocate memory for data files if(ens_mtddf) { delete ens_mtddf; ens_mtddf = (Met2dDataFile *) 0; } if(obs_mtddf) { delete obs_mtddf; obs_mtddf = (Met2dDataFile *) 0; } @@ -508,11 +536,11 @@ void process_grid(const Grid &fcst_grid) { // Parse regridding logic RegridInfo ri; - if(conf_info.get_n_ens_var() > 0) { - ri = conf_info.ens_info[0]->regrid(); + if(conf_info.ens_input.size() > 0) { + ri = conf_info.ens_input[0]->get_var_info()->regrid(); } else if(conf_info.get_n_vx() > 0) { - ri = conf_info.vx_opt[0].vx_pd.fcst_info->regrid(); + ri = conf_info.vx_opt[0].vx_pd.fcst_info->get_var_info()->regrid(); } else { mlog << Error << "\nprocess_grid() -> " @@ -572,30 +600,40 @@ void process_grid(const Grid &fcst_grid) { //////////////////////////////////////////////////////////////////////// void process_n_vld() { - int i, j, n_vld; + int i_var, i_ens, j, n_vld, n_ens_inputs; DataPlane dp; DataPlaneArray dpa; + VarInfo * var_info; + ConcatString ens_file, fcst_file; + vector::const_iterator var_it; // Initialize n_ens_vld.clear(); n_vx_vld.clear(); // Loop through the ensemble fields to be processed - for(i=0; iinputs_n(); + + for(i_var=0; var_it != conf_info.ens_input.end(); var_it++, i_var++) { - // Loop through the ensemble files - for(j=0, n_vld=0; jget_file(i_ens); + var_info = (*var_it)->get_var_info(i_ens); // Check for valid file - if(!ens_file_vld[j]) continue; + if(!ens_file_vld[(*var_it)->get_file_index(i_ens)]) continue; // Check for valid data - if(!get_data_plane(ens_file_list[j].c_str(), etype, - conf_info.ens_info[i], dp, false)) { + if(!get_data_plane(ens_file.c_str(), etype, + var_info, dp, false)) { mlog << Warning << "\nprocess_n_vld() -> " << "ensemble field \"" - << conf_info.ens_info[i]->magic_str() - << "\" not found in file \"" << ens_file_list[j] + << var_info->magic_str() + << "\" not found in file \"" << ens_file << "\"\n\n"; } else { @@ -603,46 +641,56 @@ void process_n_vld() { // Increment the valid counter n_vld++; } - } // end for j + } // end for i_ens // Check for enough valid data - if((double) n_vld/n_ens < conf_info.vld_ens_thresh) { + if((double) n_vld/n_ens_inputs < conf_info.vld_ens_thresh) { mlog << Error << "\nprocess_n_vld() -> " - << n_ens - n_vld << " missing ensemble fields exceeds the " - << "maximum allowable specified by \"ens.ens_thresh\" " - << "in the configuration file.\n\n"; + << n_vld << " of " << n_ens_inputs + << " (" << (double)n_vld/n_ens_inputs << ")" + << " fields found for \"" << (*var_it)->get_var_info()->magic_str() + << "\" does not meet the threshold specified by \"" + << conf_key_ens_ens_thresh << "\" (" << conf_info.vld_ens_thresh + << ") in the configuration file.\n\n"; exit(1); } // Store the valid data count n_ens_vld.add(n_vld); - } // end for i + } // end for i_var // Loop through the verification fields to be processed - for(i=0; iinputs_n(); + + // Loop through the forecast inputs + for(i_ens=n_vld=0; i_ens < n_ens_inputs; i_ens++) { - // Loop through the ensemble files - for(j=0, n_vld=0; jget_file(i_ens); + var_info = conf_info.vx_opt[i_var].vx_pd.fcst_info->get_var_info(i_ens); + j = conf_info.vx_opt[i_var].vx_pd.fcst_info->get_file_index(i_ens); // Check for valid file - if(!ens_file_vld[j]) continue; + if(!fcst_file_vld[j]) continue; // Check for valid data fields. // Call data_plane_array to handle multiple levels. - if(!get_data_plane_array(ens_file_list[j].c_str(), etype, - conf_info.vx_opt[i].vx_pd.fcst_info, + if(!get_data_plane_array(fcst_file.c_str(), etype, + var_info, dpa, false)) { mlog << Warning << "\nprocess_n_vld() -> " << "no data found for forecast field \"" - << conf_info.vx_opt[i].vx_pd.fcst_info->magic_str() - << "\" in file \"" << ens_file_list[j] + << var_info->magic_str() + << "\" in file \"" << fcst_file << "\"\n\n"; } else { // Store the lead times for the first verification field - if(i==0) ens_lead_na.add(dpa[0].lead()); + if(i_var==0) ens_lead_na.add(dpa[0].lead()); // Increment the valid counter n_vld++; @@ -650,11 +698,15 @@ void process_n_vld() { } // end for j // Check for enough valid data - if((double) n_vld/n_ens < conf_info.vld_ens_thresh) { + if((double) n_vld/n_ens_inputs < conf_info.vld_ens_thresh) { mlog << Error << "\nprocess_n_vld() -> " - << n_ens - n_vld << " missing forecast fields exceeds the " - << "maximum allowable specified by \"ens.ens_thresh\" " - << "in the configuration file.\n\n"; + << n_vld << " of " << n_ens_inputs + << " (" << (double)n_vld/n_ens_inputs << ")" + << " forecast fields found for \"" + << conf_info.vx_opt[i_var].vx_pd.fcst_info->get_var_info()->magic_str() + << "\" does not meet the threshold specified by \"" + << conf_key_ens_ens_thresh << "\" (" << conf_info.vld_ens_thresh + << ") in the configuration file.\n\n"; exit(1); } @@ -778,50 +830,44 @@ bool get_data_plane_array(const char *infile, GrdFileType ftype, //////////////////////////////////////////////////////////////////////// void process_ensemble() { - int i, j; + int i_var, i_ens, j; bool reset; - DataPlane ens_dp; + DataPlane ens_dp, ctrl_dp; unixtime max_init_ut = bad_data_ll; + VarInfo * var_info; + VarInfo * ctrl_info; + ConcatString ens_file; + + vector::const_iterator var_it = conf_info.ens_input.begin(); // Loop through each of the ensemble fields to be processed - for(i=0; iget_var_info(); mlog << Debug(2) << "\n" << sep_str << "\n\n" << "Processing ensemble field: " - << conf_info.ens_info[i]->magic_str() << "\n"; + << (*var_it)->raw_magic_str << "\n"; + + // Loop through each of the input forecast files/variables + for(i_ens=0,reset=true; i_ens < (*var_it)->inputs_n(); i_ens++) { - // Loop through each of the input forecast files - for(j=0, reset=true; jget_file_index(i_ens); // Skip bad data files - if(!ens_file_vld[j]) { + if(!ens_file_vld[j]) { continue; } - // Error out if the control member file is not valid - if(j==ctrl_index) { - mlog << Error << "\nprocess_ensemble() -> " - << "the control member file is not valid.\n\n"; - exit(1); - } + // get file and VarInfo to process + ens_file = (*var_it)->get_file(i_ens); + var_info = (*var_it)->get_var_info(i_ens); - // Otherwise, continue - continue; - } + mlog << Debug(3) << "\n" + << "Reading field: " + << var_info->magic_str() << "\n"; // Read the current field - if(!get_data_plane(ens_file_list[j].c_str(), etype, - conf_info.ens_info[i], ens_dp, true)) { - - // Error out if the control member data cannot be read - if(j==ctrl_index) { - mlog << Error << "\nprocess_ensemble() -> " - << "can't find \"" << conf_info.ens_info[i]->magic_str() - << "\" data in the control member file.\n\n"; - exit(1); - } - - // Otherwise, continue - continue; - } + if(!get_data_plane(ens_file.c_str(), etype, + var_info, ens_dp, true)) { continue; } // Create a NetCDF file to store the ensemble output if(nc_out == (NcFile *) 0) { @@ -832,26 +878,48 @@ void process_ensemble() { if(reset) { clear_counts(); reset = false; + + // Read ensemble control member data, if provided + if(ctrl_file.nonempty()) { + ctrl_info = (*var_it)->get_ctrl(i_ens); + + mlog << Debug(3) << "\n" + << "Reading control field: " + << ctrl_info->magic_str() << "\n"; + + // Error out if missing + if (!get_data_plane(ctrl_file.c_str(), etype, + ctrl_info, ctrl_dp, true)) { + mlog << Error << "\nprocess_ensemble() -> " + << "control member ensemble field \"" + << ctrl_info->magic_str() + << "\" not found in file \"" << ctrl_file << "\"\n\n"; + exit(1); + } + + // Apply current data to the running sums and counts + track_counts(*var_it, ctrl_dp, true); + } } // Apply current data to the running sums and counts - track_counts(i, ens_dp, j == ctrl_index); + track_counts(*var_it, ens_dp, false); // Keep track of the maximum initialization time if(is_bad_data(max_init_ut) || ens_dp.init() > max_init_ut) { max_init_ut = ens_dp.init(); } - } // end for j + } // end for i_ens // Write out the ensemble information to a NetCDF file ens_dp.set_init(max_init_ut); - write_ens_nc(i, ens_dp); + write_ens_nc(*var_it, i_var, ens_dp); // Store the ensemble mean output file ens_mean_file = out_nc_file_list[out_nc_file_list.n() - 1]; - } // end for i + } // end for var_it // Close the output NetCDF file if(nc_out) { @@ -896,7 +964,7 @@ void process_vx() { //////////////////////////////////////////////////////////////////////// void process_point_vx() { - int i, n_miss; + int i, i_file, n_miss; unixtime beg_ut, end_ut; // Set observation time window for each verification task @@ -934,11 +1002,13 @@ void process_point_vx() { conf_info.vx_opt[i].vx_pd.print_obs_summary(); } - // Process each ensemble file - for(i=0, n_miss=0; iinputs_n(); i++) { + + i_file = conf_info.vx_opt[0].vx_pd.fcst_info->get_file_index(i); // If the current forecast file is valid, process it - if(!ens_file_vld[i]) { + if(!fcst_file_vld[i_file]) { n_miss++; continue; } @@ -1120,8 +1190,11 @@ int process_point_ens(int i_ens, int &n_miss) { bool is_ens_mean = (-1 == i_ens); const char *file_type = (is_ens_mean ? "mean" : "ensemble"); + // get file index from first verification for logging + int i_file = conf_info.vx_opt[0].vx_pd.fcst_info->get_file_index(i_ens); + // Determine the correct file to process - if(!is_ens_mean) ens_file = ConcatString(ens_file_list[i_ens]); + if(!is_ens_mean) ens_file = ConcatString(fcst_file_list[i_file]); else ens_file = (ens_mean_user.empty() ? ens_mean_file : ens_mean_user); @@ -1142,9 +1215,12 @@ int process_point_ens(int i_ens, int &n_miss) { info = &ens_mean_info; } else { - info = conf_info.vx_opt[i].vx_pd.fcst_info; + info = conf_info.vx_opt[i].vx_pd.fcst_info->get_var_info(i_ens); } + // if not processing mean, get file based on current vx and ensemble index + if(!is_ens_mean) ens_file = conf_info.vx_opt[i].vx_pd.fcst_info->get_file(i_ens); + // Read the gridded data from the input forecast file if(!get_data_plane_array(ens_file.c_str(), info->file_type(), info, fcst_dpa, true)) { @@ -1205,13 +1281,13 @@ void process_point_scores() { shc.set_desc(conf_info.vx_opt[i].vx_pd.desc.c_str()); // Store the forecast variable name - shc.set_fcst_var(conf_info.vx_opt[i].vx_pd.fcst_info->name_attr()); + shc.set_fcst_var(conf_info.vx_opt[i].vx_pd.fcst_info->get_var_info()->name_attr()); // Store the forecast variable units - shc.set_fcst_units(conf_info.vx_opt[i].vx_pd.fcst_info->units_attr()); + shc.set_fcst_units(conf_info.vx_opt[i].vx_pd.fcst_info->get_var_info()->units_attr()); // Set the forecast level name - shc.set_fcst_lev(conf_info.vx_opt[i].vx_pd.fcst_info->level_attr().c_str()); + shc.set_fcst_lev(conf_info.vx_opt[i].vx_pd.fcst_info->get_var_info()->level_attr().c_str()); // Store the observation variable name shc.set_obs_var(conf_info.vx_opt[i].vx_pd.obs_info->name_attr()); @@ -1255,7 +1331,7 @@ void process_point_scores() { mlog << Debug(2) << "Processing point verification " - << conf_info.vx_opt[i].vx_pd.fcst_info->magic_str() + << conf_info.vx_opt[i].vx_pd.fcst_info->get_var_info()->magic_str() << " versus " << conf_info.vx_opt[i].vx_pd.obs_info->magic_str() << ", for observation type " << pd_ptr->msg_typ @@ -1393,7 +1469,7 @@ void process_point_scores() { //////////////////////////////////////////////////////////////////////// void process_grid_vx() { - int i, j, k, l, m, n_miss; + int i, j, k, l, m, n_miss, i_file; bool found; MaskPlane mask_mp; DataPlane *fcst_dp = (DataPlane *) 0; @@ -1402,6 +1478,8 @@ void process_grid_vx() { DataPlane emn_dp, cmn_dp, csd_dp; PairDataEnsemble pd_all, pd; ObsErrorEntry *oerr_ptr = (ObsErrorEntry *) 0; + VarInfo * var_info; + ConcatString fcst_file; mlog << Debug(2) << "\n" << sep_str << "\n\n"; @@ -1409,8 +1487,9 @@ void process_grid_vx() { shc.set_obtype(conf_info.obtype.c_str()); // Allocate space to store the forecast fields - fcst_dp = new DataPlane [n_ens]; - fraw_dp = new DataPlane [n_ens]; + int num_dp = conf_info.vx_opt[0].vx_pd.fcst_info->inputs_n(); + fcst_dp = new DataPlane [num_dp]; + fraw_dp = new DataPlane [num_dp]; // Loop through each of the fields to be verified for(i=0; iname_attr()); + shc.set_fcst_var(conf_info.vx_opt[i].vx_pd.fcst_info->get_var_info()->name_attr()); // Store the forecast variable units - shc.set_fcst_units(conf_info.vx_opt[i].vx_pd.fcst_info->units_attr()); + shc.set_fcst_units(conf_info.vx_opt[i].vx_pd.fcst_info->get_var_info()->units_attr()); // Set the forecast level name - shc.set_fcst_lev(conf_info.vx_opt[i].vx_pd.fcst_info->level_attr().c_str()); + shc.set_fcst_lev(conf_info.vx_opt[i].vx_pd.fcst_info->get_var_info()->level_attr().c_str()); // Set the ObsErrorEntry pointer if(conf_info.vx_opt[i].obs_error.flag) { @@ -1486,16 +1565,20 @@ void process_grid_vx() { } } - // Loop through each of the input ensemble files - for(j=0, n_miss=0; jinputs_n(); j++) { // Initialize fcst_dp[j].clear(); + i_file = conf_info.vx_opt[i].vx_pd.fcst_info->get_file_index(j); + var_info = conf_info.vx_opt[i].vx_pd.fcst_info->get_var_info(j); + fcst_file = conf_info.vx_opt[i].vx_pd.fcst_info->get_file(j); + // If the current ensemble file is valid, read the field - if(ens_file_vld[j]) { - found = get_data_plane(ens_file_list[j].c_str(), etype, - conf_info.vx_opt[i].vx_pd.fcst_info, + if(fcst_file_vld[i_file]) { + found = get_data_plane(fcst_file.c_str(), etype, + var_info, fcst_dp[j], true); } else { @@ -1521,7 +1604,7 @@ void process_grid_vx() { << "Found " << (cmn_dp.nx() == 0 ? 0 : 1) << " climatology mean field(s) and " << (csd_dp.nx() == 0 ? 0 : 1) << " climatology standard deviation field(s) for forecast " - << conf_info.vx_opt[i].vx_pd.fcst_info->magic_str() << ".\n"; + << conf_info.vx_opt[i].vx_pd.fcst_info->get_var_info()->magic_str() << ".\n"; // If requested in the config file, create a NetCDF file to store // the verification matched pairs @@ -1588,7 +1671,7 @@ void process_grid_vx() { info = &ens_mean_info; } else { - info = conf_info.vx_opt[i].vx_pd.fcst_info; + info = conf_info.vx_opt[i].vx_pd.fcst_info->get_var_info(); } // Read the gridded data from the mean file @@ -1648,7 +1731,7 @@ void process_grid_vx() { // Store a copy of the unperturbed observation field oraw_dp = obs_dp; - // Apply observation error bias correciton, if requested + // Apply observation error bias correction, if requested if(conf_info.vx_opt[i].obs_error.flag) { mlog << Debug(3) << "Applying observation error bias correction to " @@ -1659,8 +1742,8 @@ void process_grid_vx() { conf_info.obtype.c_str()); } - // Looop through the ensemble members - for(k=0; kinputs_n(); k++) { // Smooth the forecast field, if requested if(field == FieldType_Fcst || field == FieldType_Both) { @@ -1711,7 +1794,7 @@ void process_grid_vx() { mlog << Debug(2) << "Processing gridded verification " - << conf_info.vx_opt[i].vx_pd.fcst_info->magic_str() + << conf_info.vx_opt[i].vx_pd.fcst_info->get_var_info()->magic_str() << " versus " << conf_info.vx_opt[i].vx_pd.obs_info->magic_str() << ", for observation type " << shc.get_obtype() @@ -1916,7 +1999,7 @@ void process_grid_scores(int i_vx, y = nint(pd.y_na[i]); // Loop through each of the ensemble members - for(j=0, n_miss=0; jinputs_n(); j++) { // Skip missing data if(fcst_dp[j].nx() == 0 || fcst_dp[j].ny() == 0) { @@ -1932,6 +2015,7 @@ void process_grid_scores(int i_vx, if(j != ctrl_index) pd.add_ens_var_sums(i, fraw_dp[j](x, y)); } } // end for j + } // end for i return; @@ -2027,13 +2111,13 @@ void clear_counts() { //////////////////////////////////////////////////////////////////////// -void track_counts(int i_vx, const DataPlane &ens_dp, bool is_ctrl) { +void track_counts(EnsVarInfo * ens_info, const DataPlane &ens_dp, bool is_ctrl) { int i, j, k; double v; // Ensemble thresholds - const int n_thr = conf_info.ens_ta[i_vx].n(); - SingleThresh *thr_buf = conf_info.ens_ta[i_vx].buf(); + const int n_thr = ens_info->cat_ta.n(); + SingleThresh *thr_buf = ens_info->cat_ta.buf(); // Increment counts for each grid point for(i=0; iname_attr() << "_" - << conf_info.vx_opt[i_vx].vx_pd.fcst_info->level_attr() + cs << conf_info.vx_opt[i_vx].vx_pd.fcst_info->get_var_info()->name_attr() << "_" + << conf_info.vx_opt[i_vx].vx_pd.fcst_info->get_var_info()->level_attr() << "_ENS_MEAN"; cs.replace(",", "_", false); cs.replace("*", "all", false); @@ -2355,7 +2439,7 @@ void build_outfile_name(unixtime ut, const char *suffix, ConcatString &str) { //////////////////////////////////////////////////////////////////////// -void write_ens_nc(int i_var, DataPlane &ens_dp) { +void write_ens_nc(EnsVarInfo * ens_info, int i_var, DataPlane &ens_dp) { int i, j, k, l; double t, v; char type_str[max_str_len]; @@ -2408,56 +2492,56 @@ void write_ens_nc(int i_var, DataPlane &ens_dp) { // Add the ensemble mean, if requested if(conf_info.nc_info.do_mean) { - write_ens_var_float(i_var, ens_mean, ens_dp, + write_ens_var_float(ens_info, ens_mean, ens_dp, "ENS_MEAN", "Ensemble Mean"); } // Add the ensemble standard deviation, if requested if(conf_info.nc_info.do_stdev) { - write_ens_var_float(i_var, ens_stdev, ens_dp, + write_ens_var_float(ens_info, ens_stdev, ens_dp, "ENS_STDEV", "Ensemble Standard Deviation"); } // Add the ensemble mean minus one standard deviation, if requested if(conf_info.nc_info.do_minus) { - write_ens_var_float(i_var, ens_minus, ens_dp, + write_ens_var_float(ens_info, ens_minus, ens_dp, "ENS_MINUS", "Ensemble Mean Minus 1 Standard Deviation"); } // Add the ensemble mean plus one standard deviation, if requested if(conf_info.nc_info.do_plus) { - write_ens_var_float(i_var, ens_plus, ens_dp, + write_ens_var_float(ens_info, ens_plus, ens_dp, "ENS_PLUS", "Ensemble Mean Plus 1 Standard Deviation"); } // Add the ensemble minimum value, if requested if(conf_info.nc_info.do_min) { - write_ens_var_float(i_var, ens_min, ens_dp, + write_ens_var_float(ens_info, ens_min, ens_dp, "ENS_MIN", "Ensemble Minimum"); } // Add the ensemble maximum value, if requested if(conf_info.nc_info.do_max) { - write_ens_var_float(i_var, ens_max, ens_dp, + write_ens_var_float(ens_info, ens_max, ens_dp, "ENS_MAX", "Ensemble Maximum"); } // Add the ensemble range, if requested if(conf_info.nc_info.do_range) { - write_ens_var_float(i_var, ens_range, ens_dp, + write_ens_var_float(ens_info, ens_range, ens_dp, "ENS_RANGE", "Ensemble Range"); } // Add the ensemble valid data count, if requested if(conf_info.nc_info.do_vld) { - write_ens_var_int(i_var, ens_vld, ens_dp, + write_ens_var_int(ens_info, ens_vld, ens_dp, "ENS_VLD", "Ensemble Valid Data Count"); } @@ -2469,7 +2553,7 @@ void write_ens_nc(int i_var, DataPlane &ens_dp) { prob_dp.set_size(grid.nx(), grid.ny()); // Loop through each threshold - for(i=0; icat_ta.n(); i++) { // Initialize prob_dp.erase(); @@ -2484,8 +2568,8 @@ void write_ens_nc(int i_var, DataPlane &ens_dp) { // Write ensemble relative frequency if(conf_info.nc_info.do_freq) { snprintf(type_str, sizeof(type_str), "ENS_FREQ_%s", - conf_info.ens_ta[i_var][i].get_abbr_str().contents().c_str()); - write_ens_data_plane(i_var, prob_dp, ens_dp, type_str, + ens_info->cat_ta[i].get_abbr_str().contents().c_str()); + write_ens_data_plane(ens_info, prob_dp, ens_dp, type_str, "Ensemble Relative Frequency"); } @@ -2503,10 +2587,10 @@ void write_ens_nc(int i_var, DataPlane &ens_dp) { // Write neighborhood ensemble probability snprintf(type_str, sizeof(type_str), "ENS_NEP_%s_%s%i", - conf_info.ens_ta[i_var][i].get_abbr_str().contents().c_str(), + ens_info->cat_ta[i].get_abbr_str().contents().c_str(), interpmthd_to_string(InterpMthd_Nbrhd).c_str(), conf_info.nbrhd_prob.width[j]*conf_info.nbrhd_prob.width[j]); - write_ens_data_plane(i_var, nbrhd_dp, ens_dp, type_str, + write_ens_data_plane(ens_info, nbrhd_dp, ens_dp, type_str, "Neighborhood Ensemble Probability"); } // end for j } // end if do_nep @@ -2519,7 +2603,7 @@ void write_ens_nc(int i_var, DataPlane &ens_dp) { prob_dp.set_size(grid.nx(), grid.ny()); // Loop through each threshold - for(i=0; icat_ta.n(); i++) { // Loop through each neigbhorhood size for(j=0; jcat_ta[i].get_abbr_str().contents().c_str(), interpmthd_to_string(InterpMthd_Nbrhd).c_str(), conf_info.nbrhd_prob.width[j]*conf_info.nbrhd_prob.width[j], conf_info.nmep_smooth.method[k].c_str(), conf_info.nmep_smooth.width[k]*conf_info.nmep_smooth.width[k]); - write_ens_data_plane(i_var, nbrhd_dp, ens_dp, type_str, + write_ens_data_plane(ens_info, nbrhd_dp, ens_dp, type_str, "Neighborhood Maximum Ensemble Probability"); } // end for k } // end for j @@ -2573,20 +2657,20 @@ void write_ens_nc(int i_var, DataPlane &ens_dp) { //////////////////////////////////////////////////////////////////////// -void write_ens_var_float(int i_var, float *ens_data, const DataPlane &dp, +void write_ens_var_float(EnsVarInfo * ens_info, float *ens_data, const DataPlane &dp, const char *type_str, const char *long_name_str) { NcVar ens_var; ConcatString ens_var_name, var_str, name_str, cs; // Append nc_pairs_var_str config file entry - cs = conf_info.ens_var_str[i_var]; + cs = ens_info->nc_var_str; if(cs.length() > 0) var_str << "_" << cs; // Construct the variable name ens_var_name << cs_erase - << conf_info.ens_info[i_var]->name_attr() << "_" - << conf_info.ens_info[i_var]->level_attr() + << ens_info->get_var_info()->name_attr() << "_" + << ens_info->get_var_info()->level_attr() << var_str << "_" << type_str; // Skip variable names that have already been written @@ -2604,16 +2688,16 @@ void write_ens_var_float(int i_var, float *ens_data, const DataPlane &dp, // if(strcmp(type_str, "ENS_MEAN") == 0) { name_str << cs_erase - << conf_info.ens_info[i_var]->name_attr(); + << ens_info->get_var_info()->name_attr(); } else { name_str << cs_erase - << conf_info.ens_info[i_var]->name_attr() << "_" + << ens_info->get_var_info()->name_attr() << "_" << type_str; } // Add the variable attributes - add_var_att_local(conf_info.ens_info[i_var], &ens_var, false, dp, + add_var_att_local(ens_info->get_var_info(), &ens_var, false, dp, name_str.c_str(), long_name_str); // Write the data @@ -2629,20 +2713,20 @@ void write_ens_var_float(int i_var, float *ens_data, const DataPlane &dp, //////////////////////////////////////////////////////////////////////// -void write_ens_var_int(int i_var, int *ens_data, const DataPlane &dp, +void write_ens_var_int(EnsVarInfo * ens_info, int *ens_data, const DataPlane &dp, const char *type_str, const char *long_name_str) { NcVar ens_var; ConcatString ens_var_name, var_str, name_str, cs; // Append nc_pairs_var_str config file entry - cs = conf_info.ens_var_str[i_var]; + cs = ens_info->nc_var_str; if(cs.length() > 0) var_str << "_" << cs; // Construct the variable name ens_var_name << cs_erase - << conf_info.ens_info[i_var]->name_attr() << "_" - << conf_info.ens_info[i_var]->level_attr() + << ens_info->get_var_info()->name_attr() << "_" + << ens_info->get_var_info()->level_attr() << var_str << "_" << type_str; // Skip variable names that have already been written @@ -2657,11 +2741,11 @@ void write_ens_var_int(int i_var, int *ens_data, const DataPlane &dp, // Construct the variable name attribute name_str << cs_erase - << conf_info.ens_info[i_var]->name_attr() << "_" + << ens_info->get_var_info()->name_attr() << "_" << type_str; // Add the variable attributes - add_var_att_local(conf_info.ens_info[i_var], &ens_var, true, dp, + add_var_att_local(ens_info->get_var_info(), &ens_var, true, dp, name_str.c_str(), long_name_str); // Write the data @@ -2677,7 +2761,7 @@ void write_ens_var_int(int i_var, int *ens_data, const DataPlane &dp, //////////////////////////////////////////////////////////////////////// -void write_ens_data_plane(int i_var, const DataPlane &ens_dp, const DataPlane &dp, +void write_ens_data_plane(EnsVarInfo * ens_info, const DataPlane &ens_dp, const DataPlane &dp, const char *type_str, const char *long_name_str) { // Allocate memory for this data @@ -2687,7 +2771,7 @@ void write_ens_data_plane(int i_var, const DataPlane &ens_dp, const DataPlane &d for(int i=0; iget_var_info(), &nc_var, false, dp, name_str.c_str(), long_name_str); // Write the data @@ -2878,7 +2962,7 @@ void write_orank_var_int(int i_vx, int i_interp, int i_mask, nc_var = add_var(nc_out, (string)var_name, ncInt, lat_dim, lon_dim); // Add the variable attributes - add_var_att_local(conf_info.vx_opt[i_vx].vx_pd.fcst_info, &nc_var, true, dp, + add_var_att_local(conf_info.vx_opt[i_vx].vx_pd.fcst_info->get_var_info(), &nc_var, true, dp, name_str.c_str(), long_name_str); // Write the data diff --git a/met/src/tools/core/ensemble_stat/ensemble_stat.h b/met/src/tools/core/ensemble_stat/ensemble_stat.h index c40771d1da..2cc02de2a1 100644 --- a/met/src/tools/core/ensemble_stat/ensemble_stat.h +++ b/met/src/tools/core/ensemble_stat/ensemble_stat.h @@ -93,12 +93,14 @@ static const int max_n_rec = 300; //////////////////////////////////////////////////////////////////////// // Input Ensemble files -static int n_ens; // Number of ensemble members -static IntArray n_ens_vld; // Number of members with valid data for each ensemble field [n_ens] +static int n_ens_files; // Number of ensemble members +static IntArray n_ens_vld; // Number of members with valid data for each ensemble field [n_ens_files] static IntArray n_vx_vld; // Number of members with valid data for each verification field [n_vx] -static StringArray ens_file_list; -static IntArray ens_file_vld; +static StringArray ens_file_list; // Array of ensemble input files +static StringArray fcst_file_list; // Array of ensemble input files including control +static IntArray ens_file_vld; // Array of ensemble file valid status +static IntArray fcst_file_vld; // Array of forecast file valid status static GrdFileType etype = FileType_None; static bool ens_mean_flag; // Flag for ensemble mean processing diff --git a/met/src/tools/core/ensemble_stat/ensemble_stat_conf_info.cc b/met/src/tools/core/ensemble_stat/ensemble_stat_conf_info.cc index b338fe4f3f..6db4407c39 100644 --- a/met/src/tools/core/ensemble_stat/ensemble_stat_conf_info.cc +++ b/met/src/tools/core/ensemble_stat/ensemble_stat_conf_info.cc @@ -50,8 +50,6 @@ EnsembleStatConfInfo::~EnsembleStatConfInfo() { void EnsembleStatConfInfo::init_from_scratch() { // Initialize pointers - ens_info = (VarInfo **) 0; - ens_ta = (ThreshArray *) 0; vx_opt = (EnsembleStatVxOpt *) 0; rng_ptr = (gsl_rng *) 0; @@ -64,9 +62,9 @@ void EnsembleStatConfInfo::init_from_scratch() { void EnsembleStatConfInfo::clear() { int i; + vector::const_iterator it = ens_input.begin(); // Initialize values - ens_var_str.clear(); model.clear(); obtype.clear(); vld_ens_thresh = bad_data_double; @@ -92,12 +90,12 @@ void EnsembleStatConfInfo::clear() { // Deallocate memory if(vx_opt) { delete [] vx_opt; vx_opt = (EnsembleStatVxOpt *) 0; } - if(ens_info) { - for(i=0; ioutput_map; Dictionary *edict = (Dictionary *) 0; @@ -139,6 +140,7 @@ void EnsembleStatConfInfo::process_config(GrdFileType etype, Dictionary *odict = (Dictionary *) 0; Dictionary i_edict, i_fdict, i_odict; InterpMthd mthd; + VarInfo * next_var; // Dump the contents of the config file if(mlog.verbosity_level() >= 5) conf.dump(cout); @@ -146,6 +148,11 @@ void EnsembleStatConfInfo::process_config(GrdFileType etype, // Initialize clear(); + n_ens_files = ens_files->n(); + + // Unset MET_ENS_MEMBER_ID in case it is set by the user + unsetenv(met_ens_member_id); + // Conf: version version = parse_conf_version(&conf); @@ -187,63 +194,134 @@ void EnsembleStatConfInfo::process_config(GrdFileType etype, // Conf: ensemble_flag parse_nc_info(); + // Conf: ens_member_ids + ens_member_ids = parse_conf_ens_member_ids(&conf); + + // Conf: control_id + control_id = parse_conf_string(&conf, conf_key_control_id, false); + + // Error check ens_member_ids and ensemble file list + if(ens_member_ids.n() > 1) { + + // Only a single file should be provided if using ens_member_ids + if(ens_files->n() > 1) { + mlog << Error << "\nEnsembleStatConfInfo::process_config() -> " + << "The \"" << conf_key_ens_member_ids << "\" " + << "must be empty if more than " + << "one file is provided.\n\n"; + exit(1); + } + + // If control ID is set, it cannot be found in ens_member_ids + if(!control_id.empty() && ens_member_ids.has(control_id)) { + mlog << Error << "\nEnsembleStatConfInfo::process_config() -> " + << "control_id (" << control_id << ") must not be found " + << "in ens_member_ids\n\n"; + exit(1); + } + } + + // If no ensemble member IDs were provided, add an empty string + if(ens_member_ids.n() == 0) { + ens_member_ids.add(""); + } + // Conf: ens.field edict = conf.lookup_array(conf_key_ens_field); + // Determine the number of ensemble fields to be processed n_ens_var = parse_conf_n_vx(edict); - // Allocate space based on the number of ensemble fields - if(n_ens_var > 0) { - ens_info = new VarInfo * [n_ens_var]; - ens_ta = new ThreshArray [n_ens_var]; - } - - // Initialize pointers - for(i=0; iset_dict(i_edict); + // get VarInfo magic string without substituted values + ens_info->raw_magic_str = raw_magic_str(i_edict, etype); + + // Loop over ensemble member IDs to substitute + for(j=0; j= 5) { - mlog << Debug(5) - << "Parsed ensemble field number " << i+1 << ":\n"; - ens_info[i]->dump(cout); + // Set the current dictionary + next_var->set_dict(i_edict); + + + // Dump the contents of the current VarInfo + if(mlog.verbosity_level() >= 5) { + mlog << Debug(5) + << "Parsed ensemble field number " << i+1 + << " (" << j+1 << "):\n"; + next_var->dump(cout); + } + + InputInfo input_info; + input_info.var_info = next_var; + input_info.file_index = 0; + input_info.file_list = ens_files; + ens_info->add_input(input_info); + + // Add InputInfo to ens info list for each ensemble file provided + // set var_info to NULL to note first VarInfo should be used + for(int k=1; kadd_input(input_info); + } // end for k + + } // end for j + + // Get field info for control member if set + if(!control_id.empty()) { + + // Set environment variable for ens member ID + setenv(met_ens_member_id, control_id.c_str(), 1); + + // Allocate new VarInfo object + next_var = info_factory.new_var_info(etype); + + // Set the current dictionary + next_var->set_dict(i_edict); + + ens_info->set_ctrl(next_var); } // Conf: ens_nc_var_str - ens_var_str.add(parse_conf_string(&i_edict, conf_key_nc_var_str, false)); + ens_info->nc_var_str =parse_conf_string(&i_edict, conf_key_nc_var_str, false); // Conf: ens_nc_pairs // Only parse thresholds if probabilities are requested if(nc_info.do_freq || nc_info.do_nep || nc_info.do_nmep) { // Conf: cat_thresh - ens_ta[i] = i_edict.lookup_thresh_array(conf_key_cat_thresh); + ens_info->cat_ta = i_edict.lookup_thresh_array(conf_key_cat_thresh); // Dump the contents of the current thresholds if(mlog.verbosity_level() >= 5) { mlog << Debug(5) << "Parsed thresholds for ensemble field number " << i+1 << ":\n"; - ens_ta[i].dump(cout); + ens_info->cat_ta.dump(cout); } // Keep track of the maximum number of thresholds - if(ens_ta[i].n_elements() > max_n_thresh) { - max_n_thresh = ens_ta[i].n_elements(); + if(ens_info->cat_ta.n_elements() > max_n_thresh) { + max_n_thresh = ens_info->cat_ta.n_elements(); } } - } + + ens_input.push_back(ens_info); + } // end for i // Conf: ens.ens_thresh vld_ens_thresh = conf.lookup_double(conf_key_ens_ens_thresh); @@ -345,7 +423,8 @@ void EnsembleStatConfInfo::process_config(GrdFileType etype, // Process the options for this verification task vx_opt[i].process_config(etype, i_fdict, otype, i_odict, rng_ptr, grid_vx, point_vx, - use_var_id); + use_var_id, ens_member_ids, + fcst_files, use_ctrl, control_id); // For no point verification, store obtype as the message type if(!point_vx) { @@ -673,21 +752,77 @@ void EnsembleStatVxOpt::clear() { void EnsembleStatVxOpt::process_config(GrdFileType ftype, Dictionary &fdict, GrdFileType otype, Dictionary &odict, gsl_rng *rng_ptr, bool grid_vx, - bool point_vx, bool use_var_id) { - int i; + bool point_vx, bool use_var_id, + StringArray ens_member_ids, + StringArray * fcst_files, + bool use_ctrl, ConcatString control_id) { + int i, j; + int file_index_start = 0; VarInfoFactory info_factory; mapoutput_map; Dictionary *dict; + VarInfo * next_var; + InputInfo input_info; // Initialize clear(); - // Allocate new VarInfo objects - vx_pd.fcst_info = info_factory.new_var_info(ftype); + // Allocate new EnsVarInfo object for fcst + vx_pd.fcst_info = new EnsVarInfo(); + + // Add control member as first input + if(use_ctrl) { + + // Set environment variable for ens member ID + setenv(met_ens_member_id, control_id.c_str(), 1); + + // Allocate new VarInfo object + next_var = info_factory.new_var_info(ftype); + + // Set the current dictionary + next_var->set_dict(fdict); + + input_info.var_info = next_var; + input_info.file_index = 0; + input_info.file_list = fcst_files; + vx_pd.fcst_info->add_input(input_info); + + // Change the starting index to the first non-control file + file_index_start = 1; + } + + // Loop over ensemble member IDs to substitute + for(i=0; iset_dict(fdict); + + input_info.var_info = next_var; + input_info.file_index = file_index_start; + input_info.file_list = fcst_files; + vx_pd.fcst_info->add_input(input_info); + + // Add InputInfo to fcst info list for each ensemble file provided + // set var_info to NULL to note first VarInfo should be used + for(j=file_index_start+1; jn(); j++) { + input_info.var_info = NULL; + input_info.file_index = j; + input_info.file_list = fcst_files; + vx_pd.fcst_info->add_input(input_info); + } // end for j + + } // end for i + + // Allocate new VarInfo object for obs vx_pd.obs_info = info_factory.new_var_info(otype); // Set the VarInfo objects - vx_pd.fcst_info->set_dict(fdict); vx_pd.obs_info->set_dict(odict); // Set the GRIB code for point observations @@ -697,7 +832,7 @@ void EnsembleStatVxOpt::process_config(GrdFileType ftype, Dictionary &fdict, if(mlog.verbosity_level() >= 5) { mlog << Debug(5) << "Parsed forecast field:\n"; - vx_pd.fcst_info->dump(cout); + vx_pd.fcst_info->get_var_info()->dump(cout); mlog << Debug(5) << "Parsed observation field:\n"; vx_pd.obs_info->dump(cout); @@ -707,10 +842,10 @@ void EnsembleStatVxOpt::process_config(GrdFileType ftype, Dictionary &fdict, // forecast field is a range of pressure levels, check to see if the // range of observation field pressure levels is wholly contained in the // fcst levels. If not, print a warning message. - if(vx_pd.fcst_info->level().type() == LevelType_Pres && - !is_eq(vx_pd.fcst_info->level().lower(), vx_pd.fcst_info->level().upper()) && - (vx_pd.obs_info->level().lower() < vx_pd.fcst_info->level().lower() || - vx_pd.obs_info->level().upper() > vx_pd.fcst_info->level().upper())) { + if(vx_pd.fcst_info->get_var_info()->level().type() == LevelType_Pres && + !is_eq(vx_pd.fcst_info->get_var_info()->level().lower(), vx_pd.fcst_info->get_var_info()->level().upper()) && + (vx_pd.obs_info->level().lower() < vx_pd.fcst_info->get_var_info()->level().lower() || + vx_pd.obs_info->level().upper() > vx_pd.fcst_info->get_var_info()->level().upper())) { mlog << Warning << "\nEnsembleStatVxOpt::process_config() -> " @@ -723,7 +858,7 @@ void EnsembleStatVxOpt::process_config(GrdFileType ftype, Dictionary &fdict, } // No support for wind direction - if(vx_pd.fcst_info->is_wind_direction() || + if(vx_pd.fcst_info->get_var_info()->is_wind_direction() || vx_pd.obs_info->is_wind_direction()) { mlog << Error << "\nEnsembleStatVxOpt::process_config() -> " << "wind direction may not be verified using grid_stat.\n\n"; diff --git a/met/src/tools/core/ensemble_stat/ensemble_stat_conf_info.h b/met/src/tools/core/ensemble_stat/ensemble_stat_conf_info.h index df46f8ecf7..879fa04e95 100644 --- a/met/src/tools/core/ensemble_stat/ensemble_stat_conf_info.h +++ b/met/src/tools/core/ensemble_stat/ensemble_stat_conf_info.h @@ -112,7 +112,7 @@ class EnsembleStatVxOpt { StringArray msg_typ; // Verifying message types - ThreshArray othr_ta; // Observation filetering thresholds + ThreshArray othr_ta; // Observation filtering thresholds ClimoCDFInfo cdf_info; // Climo CDF info @@ -139,7 +139,9 @@ class EnsembleStatVxOpt { void process_config(GrdFileType, Dictionary &, GrdFileType, Dictionary &, - gsl_rng *, bool, bool, bool); + gsl_rng *, bool, bool, bool, + StringArray, StringArray *, + bool, ConcatString); void set_vx_pd(EnsembleStatConfInfo *, int); void set_perc_thresh(const PairDataEnsemble *); @@ -196,9 +198,9 @@ class EnsembleStatConfInfo { ConcatString model; // Model name ConcatString obtype; // Observation type - VarInfo ** ens_info; // Array of pointers for ensemble [n_ens_var] (allocated) - ThreshArray * ens_ta; // Array for ensemble thresholds [n_ens_var] (allocated) - StringArray ens_var_str; // Array for ensemble variable name strings [n_ens_var] + vector ens_input; // Vector of EnsVarInfo pointers (allocated) + StringArray ens_member_ids; // Array of ensemble member ID strings + ConcatString control_id; // Control ID NbrhdInfo nbrhd_prob; // Neighborhood probability definition int n_nbrhd; // Number of neighborhood sizes @@ -235,7 +237,8 @@ class EnsembleStatConfInfo { void clear(); void read_config (const ConcatString , const ConcatString); - void process_config(GrdFileType, GrdFileType, bool, bool, bool); + void process_config(GrdFileType, GrdFileType, bool, bool, bool, + StringArray *, StringArray *, bool); void process_flags (); void parse_nc_info (); void process_masks (const Grid &); diff --git a/met/src/tools/other/gen_ens_prod/gen_ens_prod.cc b/met/src/tools/other/gen_ens_prod/gen_ens_prod.cc index 786a56b390..33d8351c8d 100644 --- a/met/src/tools/other/gen_ens_prod/gen_ens_prod.cc +++ b/met/src/tools/other/gen_ens_prod/gen_ens_prod.cc @@ -55,17 +55,17 @@ static void process_ensemble(); static bool get_data_plane(const char *, GrdFileType, VarInfo *, DataPlane &); static void clear_counts(); -static void track_counts(EnsVarInfo *, const DataPlane &, bool, +static void track_counts(GenEnsProdVarInfo *, const DataPlane &, bool, const DataPlane &, const DataPlane &); static void setup_nc_file(); -static void write_ens_nc(EnsVarInfo *, int, const DataPlane &, +static void write_ens_nc(GenEnsProdVarInfo *, int, const DataPlane &, const DataPlane &, const DataPlane &); -static void write_ens_var_float(EnsVarInfo *, float *, const DataPlane &, +static void write_ens_var_float(GenEnsProdVarInfo *, float *, const DataPlane &, const char *, const char *); -static void write_ens_var_int(EnsVarInfo *, int *, const DataPlane &, +static void write_ens_var_int(GenEnsProdVarInfo *, int *, const DataPlane &, const char *, const char *); -static void write_ens_data_plane(EnsVarInfo *, const DataPlane &, const DataPlane &, +static void write_ens_data_plane(GenEnsProdVarInfo *, const DataPlane &, const DataPlane &, const char *, const char *); static void add_var_att_local(VarInfo *, NcVar *, bool is_int, @@ -327,7 +327,7 @@ void process_ensemble() { ConcatString ens_file; // Loop through each of the ensemble fields to be processed - vector::const_iterator var_it = conf_info.ens_input.begin(); + vector::const_iterator var_it = conf_info.ens_input.begin(); for(i_var=0; var_it != conf_info.ens_input.end(); var_it++, i_var++) { // Need to reinitialize counts and sums for each ensemble field @@ -470,7 +470,7 @@ void clear_counts() { //////////////////////////////////////////////////////////////////////// -void track_counts(EnsVarInfo * ens_info, const DataPlane &ens_dp, bool is_ctrl, +void track_counts(GenEnsProdVarInfo * ens_info, const DataPlane &ens_dp, bool is_ctrl, const DataPlane &cmn_dp, const DataPlane &csd_dp) { int i, j, k; double ens, cmn, csd; @@ -581,7 +581,7 @@ void setup_nc_file() { //////////////////////////////////////////////////////////////////////// -void write_ens_nc(EnsVarInfo * ens_info, int n_ens_vld, +void write_ens_nc(GenEnsProdVarInfo * ens_info, int n_ens_vld, const DataPlane &ens_dp, const DataPlane &cmn_dp, const DataPlane &csd_dp) { @@ -838,7 +838,7 @@ void write_ens_nc(EnsVarInfo * ens_info, int n_ens_vld, //////////////////////////////////////////////////////////////////////// -void write_ens_var_float(EnsVarInfo * ens_info, float *ens_data, const DataPlane &dp, +void write_ens_var_float(GenEnsProdVarInfo * ens_info, float *ens_data, const DataPlane &dp, const char *type_str, const char *long_name_str) { NcVar ens_var; @@ -894,7 +894,7 @@ void write_ens_var_float(EnsVarInfo * ens_info, float *ens_data, const DataPlane //////////////////////////////////////////////////////////////////////// -void write_ens_var_int(EnsVarInfo * ens_info, int *ens_data, const DataPlane &dp, +void write_ens_var_int(GenEnsProdVarInfo * ens_info, int *ens_data, const DataPlane &dp, const char *type_str, const char *long_name_str) { NcVar ens_var; @@ -941,7 +941,7 @@ void write_ens_var_int(EnsVarInfo * ens_info, int *ens_data, const DataPlane &dp //////////////////////////////////////////////////////////////////////// -void write_ens_data_plane(EnsVarInfo * ens_info, const DataPlane &ens_dp, const DataPlane &dp, +void write_ens_data_plane(GenEnsProdVarInfo * ens_info, const DataPlane &ens_dp, const DataPlane &dp, const char *type_str, const char *long_name_str) { // Allocate memory for this data diff --git a/met/src/tools/other/gen_ens_prod/gen_ens_prod_conf_info.cc b/met/src/tools/other/gen_ens_prod/gen_ens_prod_conf_info.cc index f8aa2169ac..96ee7be396 100644 --- a/met/src/tools/other/gen_ens_prod/gen_ens_prod_conf_info.cc +++ b/met/src/tools/other/gen_ens_prod/gen_ens_prod_conf_info.cc @@ -55,7 +55,7 @@ void GenEnsProdConfInfo::init_from_scratch() { //////////////////////////////////////////////////////////////////////// void GenEnsProdConfInfo::clear() { - vector::const_iterator var_it = ens_input.begin(); + vector::const_iterator var_it = ens_input.begin(); // Clear, erase, and initialize members model.clear(); @@ -173,14 +173,14 @@ void GenEnsProdConfInfo::process_config(GrdFileType etype, StringArray * ens_fil // Parse the ensemble field information for(i=0,max_n_cat=0; iraw_magic_str = raw_magic_str(i_edict); + ens_info->raw_magic_str = raw_magic_str(i_edict, etype); // Loop over ensemble member IDs to substitute for(j=0; j::const_iterator it; - for(it = inputs.begin(); it != inputs.end(); it++) { - if((*it).var_info) { delete (*it).var_info; } - } - - if(ctrl_info) { delete ctrl_info; } -} - -//////////////////////////////////////////////////////////////////////// - -void EnsVarInfo::add_input(InputInfo input) { - inputs.push_back(input); -} - -//////////////////////////////////////////////////////////////////////// - -int EnsVarInfo::inputs_n() { - return inputs.size(); -} - -//////////////////////////////////////////////////////////////////////// - -void EnsVarInfo::set_ctrl(VarInfo * ctrl) { - ctrl_info = ctrl; -} - -//////////////////////////////////////////////////////////////////////// - -VarInfo * EnsVarInfo::get_ctrl(int index) { - if(ctrl_info) { - return ctrl_info; - } - return inputs[index].var_info; -} - -//////////////////////////////////////////////////////////////////////// - -VarInfo * EnsVarInfo::get_var_info(int index) { - if(inputs[index].var_info) { - return inputs[index].var_info; - } - return inputs[0].var_info; -} - -//////////////////////////////////////////////////////////////////////// - -ConcatString EnsVarInfo::get_file(int index) { - int file_index = inputs[index].file_index; - return (*inputs[index].file_list)[file_index]; -} - -//////////////////////////////////////////////////////////////////////// - -int EnsVarInfo::get_file_index(int index) { - return inputs[index].file_index; -} - -//////////////////////////////////////////////////////////////////////// - -ConcatString raw_magic_str(Dictionary i_edict) { - ConcatString magic_str; - - ConcatString name = i_edict.lookup_string("name"); - ConcatString level = i_edict.lookup_string("level"); - - if(level.nonempty() && level[0] != '(') { - magic_str << name << "/" << level; - } else { - magic_str << name << level; - } - - return magic_str; - -} - -//////////////////////////////////////////////////////////////////////// diff --git a/met/src/tools/other/gen_ens_prod/gen_ens_prod_conf_info.h b/met/src/tools/other/gen_ens_prod/gen_ens_prod_conf_info.h index d0d5e60bbe..9510cc32b7 100644 --- a/met/src/tools/other/gen_ens_prod/gen_ens_prod_conf_info.h +++ b/met/src/tools/other/gen_ens_prod/gen_ens_prod_conf_info.h @@ -24,6 +24,10 @@ //////////////////////////////////////////////////////////////////////// +class GenEnsProdVarInfo; // forward reference + +//////////////////////////////////////////////////////////////////////// + struct GenEnsProdNcOutInfo { bool do_latlon; @@ -51,43 +55,6 @@ struct GenEnsProdNcOutInfo { void set_all_true(); }; - -//////////////////////////////////////////////////////////////////////// - -struct InputInfo { - VarInfo * var_info; // Variable information to read - int file_index; // Index in file_list of file to read - StringArray * file_list; // Array of files (unallocated) -}; - -//////////////////////////////////////////////////////////////////////// - -class EnsVarInfo { - - private: - vector inputs; // Vector of InputInfo - VarInfo * ctrl_info; // Field info for control member - public: - EnsVarInfo(); - ~EnsVarInfo(); - - void add_input(InputInfo); - int inputs_n(); - - void set_ctrl(VarInfo *); - VarInfo * get_ctrl(int); - - // Get VarInfo from first InputInfo if requested VarInfo is NULL - VarInfo * get_var_info(int index=0); - ConcatString get_file(int index=0); - int get_file_index(int index=0); - - ConcatString nc_var_str; // Ensemble variable name strings - ThreshArray cat_ta; // Ensemble categorical thresholds - GenEnsProdNcOutInfo nc_info; // Ensemble product outputs - ConcatString raw_magic_str; // Magic string w/o var substitution -}; - //////////////////////////////////////////////////////////////////////// class GenEnsProdConfInfo { @@ -115,9 +82,9 @@ class GenEnsProdConfInfo { ConcatString desc; // Description ConcatString control_id; // Control ID - vector ens_input; // Vector of EnsVarInfo pointers (allocated) - vector cdf_info; // Array of climo CDF info objects - StringArray ens_member_ids; // Array of ensemble member ID strings + vector ens_input; // Vector of GenEnsProdVarInfo pointers (allocated) + vector cdf_info; // Array of climo CDF info objects + StringArray ens_member_ids; // Array of ensemble member ID strings NbrhdInfo nbrhd_prob; // Neighborhood probability definition InterpInfo nmep_smooth; // Neighborhood maximum smoothing information @@ -152,7 +119,11 @@ inline int GenEnsProdConfInfo::get_compression_level() { return(conf.nc_compress //////////////////////////////////////////////////////////////////////// -ConcatString raw_magic_str(Dictionary i_edict); +class GenEnsProdVarInfo: public EnsVarInfo { + +public: + GenEnsProdNcOutInfo nc_info; // Ensemble product outputs +}; #endif /* __GEN_ENS_PROD_CONF_INFO_H__ */ diff --git a/scripts/environment/development.seneca b/scripts/environment/development.seneca index 5b3497c0c1..a535aeaa67 100644 --- a/scripts/environment/development.seneca +++ b/scripts/environment/development.seneca @@ -49,4 +49,3 @@ export MET_TEST_RSCRIPT=/usr/local/R-4.1.2/bin/Rscript export PATH="/usr/local/nco/bin:/usr/local/netcdf/bin:\ /usr/local/sbin:/usr/local/bin:/usr/sbin:\ /usr/bin:/sbin:/bin:/usr/bin/X11:/opt/bin:$PATH" - diff --git a/test/config/EnsembleStatConfig b/test/config/EnsembleStatConfig index c99f8da963..296555748e 100644 --- a/test/config/EnsembleStatConfig +++ b/test/config/EnsembleStatConfig @@ -56,6 +56,13 @@ ens = { ]; } +// +// IDs for ensemble members +// Only set if processing a single file with all ensembles +// +ens_member_ids = []; +control_id = ""; + //////////////////////////////////////////////////////////////////////////////// // diff --git a/test/config/EnsembleStatConfig_MASK_SID b/test/config/EnsembleStatConfig_MASK_SID index 2ad34e2f27..764ad8280c 100644 --- a/test/config/EnsembleStatConfig_MASK_SID +++ b/test/config/EnsembleStatConfig_MASK_SID @@ -53,6 +53,13 @@ ens = { ]; } +// +// IDs for ensemble members +// Only set if processing a single file with all ensembles +// +ens_member_ids = []; +control_id = ""; + //////////////////////////////////////////////////////////////////////////////// // diff --git a/test/config/EnsembleStatConfig_climo b/test/config/EnsembleStatConfig_climo index 1656128df9..478ceb1552 100644 --- a/test/config/EnsembleStatConfig_climo +++ b/test/config/EnsembleStatConfig_climo @@ -48,6 +48,13 @@ ens = { ]; } +// +// IDs for ensemble members +// Only set if processing a single file with all ensembles +// +ens_member_ids = []; +control_id = ""; + //////////////////////////////////////////////////////////////////////////////// // diff --git a/test/config/EnsembleStatConfig_grid_weight b/test/config/EnsembleStatConfig_grid_weight index da710a73e5..dbd72bc580 100644 --- a/test/config/EnsembleStatConfig_grid_weight +++ b/test/config/EnsembleStatConfig_grid_weight @@ -47,6 +47,13 @@ ens = { field = tmp_field; } +// +// IDs for ensemble members +// Only set if processing a single file with all ensembles +// +ens_member_ids = []; +control_id = ""; + //////////////////////////////////////////////////////////////////////////////// // diff --git a/test/config/EnsembleStatConfig_one_cdf_bin b/test/config/EnsembleStatConfig_one_cdf_bin index 7e19af2b80..2a02b9475e 100644 --- a/test/config/EnsembleStatConfig_one_cdf_bin +++ b/test/config/EnsembleStatConfig_one_cdf_bin @@ -48,6 +48,13 @@ ens = { ]; } +// +// IDs for ensemble members +// Only set if processing a single file with all ensembles +// +ens_member_ids = []; +control_id = ""; + //////////////////////////////////////////////////////////////////////////////// // diff --git a/test/config/EnsembleStatConfig_python b/test/config/EnsembleStatConfig_python index de502a7250..1e137fad3d 100644 --- a/test/config/EnsembleStatConfig_python +++ b/test/config/EnsembleStatConfig_python @@ -56,6 +56,13 @@ ens = { field = [ { name = "${FCST_COMMAND}"; } ]; } +// +// IDs for ensemble members +// Only set if processing a single file with all ensembles +// +ens_member_ids = []; +control_id = ""; + //////////////////////////////////////////////////////////////////////////////// // diff --git a/test/config/EnsembleStatConfig_qty_inc_exc b/test/config/EnsembleStatConfig_qty_inc_exc index 17009a61a0..620511dd4c 100644 --- a/test/config/EnsembleStatConfig_qty_inc_exc +++ b/test/config/EnsembleStatConfig_qty_inc_exc @@ -51,6 +51,13 @@ ens = { ]; } +// +// IDs for ensemble members +// Only set if processing a single file with all ensembles +// +ens_member_ids = []; +control_id = ""; + //////////////////////////////////////////////////////////////////////////////// // diff --git a/test/config/EnsembleStatConfig_single_file_grib b/test/config/EnsembleStatConfig_single_file_grib new file mode 100644 index 0000000000..cae96307fe --- /dev/null +++ b/test/config/EnsembleStatConfig_single_file_grib @@ -0,0 +1,317 @@ +//////////////////////////////////////////////////////////////////////////////// +// +// Ensemble-Stat configuration file. +// +// For additional information, please see the MET User's Guide. +// +//////////////////////////////////////////////////////////////////////////////// + +// +// Output model name to be written +// +model = "GEFS"; + +// +// Output description to be written +// May be set separately in each "obs.field" entry +// +desc = "${DESC}"; + +// +// Output observation type to be written +// +obtype = "ANALYS"; + +//////////////////////////////////////////////////////////////////////////////// + +// +// Verification grid +// May be set separately in each "field" entry +// +regrid = { + to_grid = NONE; + method = NEAREST; + width = 1; + vld_thresh = 0.5; + shape = SQUARE; +} + +//////////////////////////////////////////////////////////////////////////////// + +// +// May be set separately in each "field" entry +// +censor_thresh = []; +censor_val = []; +cat_thresh = []; +nc_var_str = ""; + +// +// Ensemble product fields to be processed +// +ens = { + ens_thresh = 1.0; + vld_thresh = 1.0; + + field = [ + { + name = "PRMSL"; + level = "L0"; + lead_time = "06"; + GRIB_ens = "MET_ENS_MEMBER_ID"; + } + ]; +} + +// +// IDs for ensemble members +// Only set if processing a single file with all ensembles +// +ens_member_ids = ["+1", "+2", "+3", "+4", "+5", "+6", "+7", "+8", "+9", "+10", "+11", "+12", "+13", "+14", "+15", "+16", "+17", "+18", "+19", "+20"]; +control_id = "hi_res_ctl"; + +//////////////////////////////////////////////////////////////////////////////// + +// +// Neighborhood ensemble probabilities +// +nbrhd_prob = { + width = [ 5 ]; + shape = CIRCLE; + vld_thresh = 0.0; +} + +// +// NMEP smoothing methods +// +nmep_smooth = { + vld_thresh = 0.0; + shape = CIRCLE; + gaussian_dx = 81.27; + gaussian_radius = 120; + type = [ + { + method = GAUSSIAN; + width = 1; + } + ]; +} + +//////////////////////////////////////////////////////////////////////////////// + +// +// Forecast and observation fields to be verified +// +fcst = { + + field = [ + { + name = "PRMSL"; + level = "L0"; + lead_time = "06"; + GRIB_ens = "MET_ENS_MEMBER_ID"; + } + ]; +} + +obs = { + field = [ + { + name = "PRMSL"; + level = "L0"; + lead_time = "06"; + GRIB_ens = "hi_res_ctl"; + } + ]; +} +//////////////////////////////////////////////////////////////////////////////// + +// +// Point observation filtering options +// May be set separately in each "obs.field" entry +// +message_type = [ "ADPUPA" ]; +sid_inc = []; +sid_exc = []; +obs_thresh = [ NA ]; +obs_quality_inc = []; +obs_quality_exc = []; +duplicate_flag = NONE; +obs_summary = NONE; +obs_perc_value = 50; +skip_const = FALSE; + +// +// Observation error options +// Set dist_type to NONE to use the observation error table instead +// May be set separately in each "obs.field" entry +// +obs_error = { + flag = FALSE; // TRUE or FALSE + dist_type = NONE; // Distribution type + dist_parm = []; // Distribution parameters + inst_bias_scale = 1.0; // Instrument bias scale adjustment + inst_bias_offset = 0.0; // Instrument bias offset adjustment + min = NA; // Valid range of data + max = NA; +} + +// +// Mapping of message type group name to comma-separated list of values +// +message_type_group_map = [ + { key = "SURFACE"; val = "ADPSFC,SFCSHP,MSONET"; }, + { key = "ANYAIR"; val = "AIRCAR,AIRCFT"; }, + { key = "ANYSFC"; val = "ADPSFC,SFCSHP,ADPUPA,PROFLR,MSONET"; }, + { key = "ONLYSF"; val = "ADPSFC,SFCSHP"; } +]; + +// +// Ensemble bin sizes +// May be set separately in each "obs.field" entry +// +ens_ssvar_bin_size = 1.0; +ens_phist_bin_size = 0.05; + +// +// Categorical thresholds to define ensemble probabilities +// May be set separately in each "fcst.field" entry +// +prob_cat_thresh = []; + +//////////////////////////////////////////////////////////////////////////////// + +// +// Climatology data +// +climo_mean = { + + file_name = []; + field = []; + + regrid = { + method = NEAREST; + width = 1; + vld_thresh = 0.5; + shape = SQUARE; + } + + time_interp_method = DW_MEAN; + day_interval = 31; + hour_interval = 6; +} + +climo_stdev = climo_mean; +climo_stdev = { + file_name = []; +} + +// +// May be set separately in each "obs.field" entry +// +climo_cdf = { + cdf_bins = 10; + center_bins = FALSE; +} + +//////////////////////////////////////////////////////////////////////////////// + +// +// Point observation time window +// +obs_window = { + beg = -5400; + end = 5400; +} + +//////////////////////////////////////////////////////////////////////////////// + +// +// Verification masking regions +// +mask = { + grid = [ "FULL" ]; + poly = []; + sid = []; + llpnt = []; +} + +//////////////////////////////////////////////////////////////////////////////// + +// +// Confidence interval settings +// +ci_alpha = [ 0.05 ]; + +//////////////////////////////////////////////////////////////////////////////// + +// +// Interpolation methods +// +interp = { + field = BOTH; + vld_thresh = 1.0; + shape = SQUARE; + + type = [ + { + method = NEAREST; + width = 1; + } + ]; +} + +//////////////////////////////////////////////////////////////////////////////// + +// +// Statistical output types +// +output_flag = { + ecnt = STAT; + rps = NONE; + rhist = NONE; + phist = NONE; + orank = NONE; + ssvar = NONE; + relp = NONE; +} + +//////////////////////////////////////////////////////////////////////////////// + +// +// Ensemble product output types +// +ensemble_flag = { + latlon = TRUE; + mean = TRUE; + stdev = TRUE; + minus = TRUE; + plus = TRUE; + min = TRUE; + max = TRUE; + range = TRUE; + vld_count = TRUE; + frequency = TRUE; + nep = FALSE; + nmep = FALSE; + rank = TRUE; + weight = FALSE; +} + +//////////////////////////////////////////////////////////////////////////////// + +// +// Random number generator +// +rng = { + type = "mt19937"; + seed = ""; +} + +//////////////////////////////////////////////////////////////////////////////// + +grid_weight_flag = NONE; +output_prefix = "${OUTPUT_PREFIX}"; +version = "V10.1.0"; + +//////////////////////////////////////////////////////////////////////////////// diff --git a/test/config/EnsembleStatConfig_single_file_nc b/test/config/EnsembleStatConfig_single_file_nc new file mode 100644 index 0000000000..47ed145c44 --- /dev/null +++ b/test/config/EnsembleStatConfig_single_file_nc @@ -0,0 +1,315 @@ +//////////////////////////////////////////////////////////////////////////////// +// +// Ensemble-Stat configuration file. +// +// For additional information, please see the MET User's Guide. +// +//////////////////////////////////////////////////////////////////////////////// + +// +// Output model name to be written +// +model = "WRF"; + +// +// Output description to be written +// May be set separately in each "obs.field" entry +// +desc = "${DESC}"; + +// +// Output observation type to be written +// +obtype = "ANALYS"; + +//////////////////////////////////////////////////////////////////////////////// + +// +// Verification grid +// May be set separately in each "field" entry +// +regrid = { + to_grid = NONE; + method = NEAREST; + width = 1; + vld_thresh = 0.5; + shape = SQUARE; +} + +//////////////////////////////////////////////////////////////////////////////// + +// +// May be set separately in each "field" entry +// +censor_thresh = []; +censor_val = []; +cat_thresh = []; +nc_var_str = ""; + +// +// Ensemble product fields to be processed +// +ens = { + file_type = NETCDF_NCCF; + ens_thresh = 1.0; + vld_thresh = 1.0; + + field = [ + { + name = "fcst"; + level = "(MET_ENS_MEMBER_ID,0,*,*)"; + cat_thresh = [ >0.0, >=5.0 ]; + } + ]; +} + +// +// IDs for ensemble members +// Only set if processing a single file with all ensembles +// +ens_member_ids = ["1", "2", "3", "4", "5", "6", "7", "8", "9", "10", "11", "12", "13", "14", "15", "16", "17", "18", "19", "20", "21", "22", "23"]; +control_id = "0"; + +//////////////////////////////////////////////////////////////////////////////// + +// +// Neighborhood ensemble probabilities +// +nbrhd_prob = { + width = [ 5 ]; + shape = CIRCLE; + vld_thresh = 0.0; +} + +// +// NMEP smoothing methods +// +nmep_smooth = { + vld_thresh = 0.0; + shape = CIRCLE; + gaussian_dx = 81.27; + gaussian_radius = 120; + type = [ + { + method = GAUSSIAN; + width = 1; + } + ]; +} + +//////////////////////////////////////////////////////////////////////////////// + +// +// Forecast and observation fields to be verified +// +fcst = { + file_type = NETCDF_NCCF; + field = [ + { + name = "fcst"; + level = "(MET_ENS_MEMBER_ID,0,*,*)"; + cat_thresh = [ >0.0, >=5.0 ]; + } + ]; +} + +obs = { + field = [ + { + name = "fcst"; + level = "(0,0,*,*)"; + cat_thresh = [ >0.0, >=5.0 ]; + } + ]; +} +//////////////////////////////////////////////////////////////////////////////// + +// +// Point observation filtering options +// May be set separately in each "obs.field" entry +// +message_type = [ "ADPUPA" ]; +sid_inc = []; +sid_exc = []; +obs_thresh = [ NA ]; +obs_quality_inc = []; +obs_quality_exc = []; +duplicate_flag = NONE; +obs_summary = NONE; +obs_perc_value = 50; +skip_const = FALSE; + +// +// Observation error options +// Set dist_type to NONE to use the observation error table instead +// May be set separately in each "obs.field" entry +// +obs_error = { + flag = FALSE; // TRUE or FALSE + dist_type = NONE; // Distribution type + dist_parm = []; // Distribution parameters + inst_bias_scale = 1.0; // Instrument bias scale adjustment + inst_bias_offset = 0.0; // Instrument bias offset adjustment + min = NA; // Valid range of data + max = NA; +} + +// +// Mapping of message type group name to comma-separated list of values +// +message_type_group_map = [ + { key = "SURFACE"; val = "ADPSFC,SFCSHP,MSONET"; }, + { key = "ANYAIR"; val = "AIRCAR,AIRCFT"; }, + { key = "ANYSFC"; val = "ADPSFC,SFCSHP,ADPUPA,PROFLR,MSONET"; }, + { key = "ONLYSF"; val = "ADPSFC,SFCSHP"; } +]; + +// +// Ensemble bin sizes +// May be set separately in each "obs.field" entry +// +ens_ssvar_bin_size = 1.0; +ens_phist_bin_size = 0.05; + +// +// Categorical thresholds to define ensemble probabilities +// May be set separately in each "fcst.field" entry +// +prob_cat_thresh = []; + +//////////////////////////////////////////////////////////////////////////////// + +// +// Climatology data +// +climo_mean = { + + file_name = []; + field = []; + + regrid = { + method = NEAREST; + width = 1; + vld_thresh = 0.5; + shape = SQUARE; + } + + time_interp_method = DW_MEAN; + day_interval = 31; + hour_interval = 6; +} + +climo_stdev = climo_mean; +climo_stdev = { + file_name = []; +} + +// +// May be set separately in each "obs.field" entry +// +climo_cdf = { + cdf_bins = 10; + center_bins = FALSE; +} + +//////////////////////////////////////////////////////////////////////////////// + +// +// Point observation time window +// +obs_window = { + beg = -5400; + end = 5400; +} + +//////////////////////////////////////////////////////////////////////////////// + +// +// Verification masking regions +// +mask = { + grid = [ "FULL" ]; + poly = []; + sid = []; + llpnt = []; +} + +//////////////////////////////////////////////////////////////////////////////// + +// +// Confidence interval settings +// +ci_alpha = [ 0.05 ]; + +//////////////////////////////////////////////////////////////////////////////// + +// +// Interpolation methods +// +interp = { + field = BOTH; + vld_thresh = 1.0; + shape = SQUARE; + + type = [ + { + method = NEAREST; + width = 1; + } + ]; +} + +//////////////////////////////////////////////////////////////////////////////// + +// +// Statistical output types +// +output_flag = { + ecnt = STAT; + rps = NONE; + rhist = NONE; + phist = NONE; + orank = NONE; + ssvar = NONE; + relp = NONE; +} + +//////////////////////////////////////////////////////////////////////////////// + +// +// Ensemble product output types +// +ensemble_flag = { + latlon = TRUE; + mean = TRUE; + stdev = TRUE; + minus = TRUE; + plus = TRUE; + min = TRUE; + max = TRUE; + range = TRUE; + vld_count = TRUE; + frequency = TRUE; + nep = FALSE; + nmep = FALSE; + rank = TRUE; + weight = FALSE; +} + +//////////////////////////////////////////////////////////////////////////////// + +// +// Random number generator +// +rng = { + type = "mt19937"; + seed = ""; +} + +//////////////////////////////////////////////////////////////////////////////// + +grid_weight_flag = NONE; +output_prefix = "${OUTPUT_PREFIX}"; +version = "V10.1.0"; + +//////////////////////////////////////////////////////////////////////////////// diff --git a/test/xml/unit_ensemble_stat.xml b/test/xml/unit_ensemble_stat.xml index 105d19c3d4..034ff07cbf 100644 --- a/test/xml/unit_ensemble_stat.xml +++ b/test/xml/unit_ensemble_stat.xml @@ -250,4 +250,86 @@ + + + + &MET_BIN;/ensemble_stat + + DESC SINGLE_FILE_NC_NO_CTRL + OUTPUT_PREFIX SINGLE_FILE_NC_NO_CTRL + + \ + 1 \ + &DATA_DIR_MODEL;/CPC_NMME/CFSv2.tmp2m.198201.fcst.nc \ + &CONFIG_DIR;/EnsembleStatConfig_single_file_nc \ + -grid_obs &DATA_DIR_MODEL;/CPC_NMME/CFSv2.tmp2m.198201.fcst.nc \ + -outdir &OUTPUT_DIR;/ensemble_stat -v 1 + + + &OUTPUT_DIR;/ensemble_stat/ensemble_stat_SINGLE_FILE_NC_NO_CTRL_19700101_000000V_ens.nc + + + + + + + &MET_BIN;/ensemble_stat + + DESC SINGLE_FILE_NC_WITH_CTRL + OUTPUT_PREFIX SINGLE_FILE_NC_WITH_CTRL + + \ + 1 \ + &DATA_DIR_MODEL;/CPC_NMME/CFSv2.tmp2m.198201.fcst.nc \ + &CONFIG_DIR;/EnsembleStatConfig_single_file_nc \ + -grid_obs &DATA_DIR_MODEL;/CPC_NMME/CFSv2.tmp2m.198201.fcst.nc \ + -ctrl &DATA_DIR_MODEL;/CPC_NMME/CFSv2.tmp2m.198201.fcst.nc \ + -outdir &OUTPUT_DIR;/ensemble_stat -v 1 + + + &OUTPUT_DIR;/ensemble_stat/ensemble_stat_SINGLE_FILE_NC_WITH_CTRL_19700101_000000V_ens.nc + + + + + + + &MET_BIN;/ensemble_stat + + DESC SINGLE_FILE_GRIB_NO_CTRL + OUTPUT_PREFIX SINGLE_FILE_GRIB_NO_CTRL + + \ + 1 \ + &DATA_DIR_MODEL;/grib2/gefs/enspost_grb2.t00z.prmsl \ + &CONFIG_DIR;/EnsembleStatConfig_single_file_grib \ + -grid_obs &DATA_DIR_MODEL;/grib2/gefs/enspost_grb2.t00z.prmsl \ + -outdir &OUTPUT_DIR;/ensemble_stat -v 1 + + + &OUTPUT_DIR;/ensemble_stat/ensemble_stat_SINGLE_FILE_GRIB_NO_CTRL_20160608_060000V_ens.nc + + + + + + + &MET_BIN;/ensemble_stat + + DESC SINGLE_FILE_GRIB_WITH_CTRL + OUTPUT_PREFIX SINGLE_FILE_GRIB_WITH_CTRL + + \ + 1 \ + &DATA_DIR_MODEL;/grib2/gefs/enspost_grb2.t00z.prmsl \ + &CONFIG_DIR;/EnsembleStatConfig_single_file_grib \ + -grid_obs &DATA_DIR_MODEL;/grib2/gefs/enspost_grb2.t00z.prmsl \ + -ctrl &DATA_DIR_MODEL;/grib2/gefs/enspost_grb2.t00z.prmsl \ + -outdir &OUTPUT_DIR;/ensemble_stat -v 1 + + + &OUTPUT_DIR;/ensemble_stat/ensemble_stat_SINGLE_FILE_GRIB_WITH_CTRL_20160608_060000V_ens.nc + + + From 032456bbce311d69bbaf8da194df53b928d1f60b Mon Sep 17 00:00:00 2001 From: John Halley Gotway Date: Fri, 14 Jan 2022 16:33:36 -0700 Subject: [PATCH 63/82] Committing directly to the develop branch. Removing many, many instances of MET_BUILD_BASE and a couple instances of MET_BASE from the examples listed in Appendix A. The inconsistent use of these variables will only cause confusion. Removing them will help avoid that confusion. Big picture... MET_BASE defines the installed 'share/met' directory. It can be used to reference the location of map or polyline files. MET_BUILD_BASE is only used in the context of the test scripts. There, it's defined as the top-level source code directory in which the code was compiled. Its used to reference the location of sample data files or Rscript included in the tarball. I left a couple references to these variables in Appendix A where thier use is not confusing. But I removed all instances which are used to the define the location of the 'bin' directory. Instead, let's just assume the MET binaries are already in their path... or the user knows where to find them. --- met/docs/Users_Guide/appendixA.rst | 103 ++++++++++++++--------------- 1 file changed, 50 insertions(+), 53 deletions(-) diff --git a/met/docs/Users_Guide/appendixA.rst b/met/docs/Users_Guide/appendixA.rst index 7271ecb1de..d5e3752ecf 100644 --- a/met/docs/Users_Guide/appendixA.rst +++ b/met/docs/Users_Guide/appendixA.rst @@ -57,7 +57,7 @@ in the configuration string on the command line: .. code-block:: none - ${MET_BUILD_BASE}/bin/plot_data_plane + plot_data_plane sample.grib china_tmp_2m_admin.ps \ 'name="TMP"; level="Z2"; \ map_data = { source = [ { file_name = \ @@ -149,7 +149,7 @@ Let's use plot_data_plane as an example: .. code-block:: none - ${MET_BUILD_BASE}/bin/plot_data_plane \ + plot_data_plane \ MERGE_20161201_20170228.nc \ obs.ps \ 'name="APCP"; level="(5,*,*)";' @@ -281,20 +281,20 @@ lat/lon grid over Europe: .. code-block:: none - ${MET_BUILD_BASE}/bin/regrid_data_plane gfs.t12z.pgrb2.0p50.f000 \ + regrid_data_plane gfs.t12z.pgrb2.0p50.f000 \ 'latlon 100 100 25 0 0.5 0.5' gfs_euro.nc -field 'name="TMP"; level="Z2";' Run the MET gen_vx_mask tool to apply your polyline to the European domain: .. code-block:: none - ${MET_BUILD_BASE}/bin/gen_vx_mask gfs_euro.nc POLAND.poly POLAND_mask.nc + gen_vx_mask gfs_euro.nc POLAND.poly POLAND_mask.nc Run the MET plot_data_plane tool to display the resulting mask field: .. code-block:: none - ${MET_BUILD_BASE}/bin/plot_data_plane POLAND_mask.nc POLAND_mask.ps 'name="POLAND"; level="(*,*)";' + plot_data_plane POLAND_mask.nc POLAND_mask.ps 'name="POLAND"; level="(*,*)";' In this example, the mask is in roughly the right spot, but there are obvious problems with the latitude and longitude values used @@ -452,7 +452,7 @@ Run Grid-Stat using the following commands and the attached config file .. code-block:: none mkdir out - ${MET_BUILD_BASE}/bin/grid_stat \ + grid_stat \ gfs_4_20160220_0000_012.grb2 \ ST4.2016022012.06h \ GridStatConfig \ @@ -490,7 +490,7 @@ commands to plot its NetCDF matched pairs output file: .. code-block:: none - ${MET_BUILD_BASE}/bin/plot_data_plane \ + plot_data_plane \ out/grid_stat_120000L_20160220_120000V_pairs.nc \ out/DIFF_APCP_06_A06_APCP_06_A06_CONUS.ps \ 'name="DIFF_APCP_06_A06_APCP_06_A06_CONUS"; level="(*,*)";' @@ -524,7 +524,7 @@ point where the temperature is less than 290 K to a value of 0: .. code-block:: none - {MET_BUILD_BASE}/bin/gen_vx_mask \ + gen_vx_mask \ data/sample_fcst/2005080700/wrfprs_ruc13_12.tm00_G212 \ data/sample_fcst/2005080700/wrfprs_ruc13_12.tm00_G212 \ APCP_03_where_2m_TMPge290.nc \ @@ -568,11 +568,10 @@ the output through plot_data_plane: .. code-block:: none - {MET_BUILD_BASE}/bin/plot_data_plane \ - - APCP_03_where_2m_TMPge290.nc APCP_03_where_2m_TMPge290.ps \ - - 'name="data_mask"; level="(*,*)";' + plot_data_plane \ + APCP_03_where_2m_TMPge290.nc \ + APCP_03_where_2m_TMPge290.ps \ + 'name="data_mask"; level="(*,*)";' In the resulting plot, anywhere you see the pink value of 10, that's where gen_vx_mask has masked out the grid point. @@ -593,21 +592,21 @@ through pcp_combine as a pass-through to put it into NetCDF format: .. code-block:: none - [MET_BUILD_BASE}/pcp_combine -add 03_file.grb 03 APCP_00_03.nc + pcp_combine -add 03_file.grb 03 APCP_00_03.nc If the user wanted the 3-6 hour accumulation, they would subtract 0-6 and 0-3 accumulations: .. code-block:: none - [MET_BUILD_BASE}/pcp_combine -subtract 06_file.grb 06 03_file.grb 03 APCP_03_06.nc + pcp_combine -subtract 06_file.grb 06 03_file.grb 03 APCP_03_06.nc Similarly, if they wanted the 6-9 hour accumulation, they would subtract 0-9 and 0-6 accumulations: .. code-block:: none - [MET_BUILD_BASE}/pcp_combine -subtract 09_file.grb 09 06_file.grb 06 APCP_06_09.nc + pcp_combine -subtract 09_file.grb 09 06_file.grb 06 APCP_06_09.nc And so on. @@ -626,7 +625,7 @@ option instead. .. code-block:: none - ${MET_BUILD_BASE}/bin/pcp_combine -add \ + pcp_combine -add \ WRFPRS_1997-06-03_APCP_A12.nc 'name="APCP_12"; level="(*,*)";' \ WRFPRS_d01_1997-06-04_00_APCP_A12.grb 12 \ Sum.nc @@ -647,17 +646,17 @@ Here are 3 commands you could use to plot these data files: .. code-block:: none - ${MET_BUILD_BASE}/bin/plot_data_plane WRFPRS_1997-06-03_APCP_A12.nc \ + plot_data_plane WRFPRS_1997-06-03_APCP_A12.nc \ WRFPRS_1997-06-03_APCP_A12.ps 'name="APCP_12"; level="(*,*)";' .. code-block:: none - ${MET_BUILD_BASE}/bin/plot_data_plane WRFPRS_d01_1997-06-04_00_APCP_A12.grb \ + plot_data_plane WRFPRS_d01_1997-06-04_00_APCP_A12.grb \ WRFPRS_d01_1997-06-04_00_APCP_A12.ps 'name="APCP" level="A12";' .. code-block:: none - ${MET_BUILD_BASE}/bin/plot_data_plane sum.nc sum.ps 'name="APCP_24"; level="(*,*)";' + plot_data_plane sum.nc sum.ps 'name="APCP_24"; level="(*,*)";' **Q. Pcp-Combine - How do I correct a precipitation time range?** @@ -756,11 +755,11 @@ examples: .. code-block:: none # Only using Stage IV data (ST4) - ${MET_BUILD_BASE}/bin/pcp_combine -sum 00000000_000000 06 \ + pcp_combine -sum 00000000_000000 06 \ 20161015_18 12 ST4.2016101518.APCP_12_SUM.nc -pcprx "ST4.*.06h" # Specify that files starting with pgbq[number][number]be used: - [MET_BUILD_BASE]/bin/pcp_combine \ + pcp_combine \ -sum 20160221_18 06 20160222_18 24 \ gfs_APCP_24_20160221_18_F00_F24.nc \ -pcpdir /scratch4/BMC/shout/ptmp/Andrew.Kren/pre2016c3_corr/temp \ @@ -826,7 +825,7 @@ Try the command: .. code-block:: none - ${MET_BUILD_BASE}/bin/pcp_combine -subtract \ + pcp_combine -subtract \ forecast.grb 'name="APCP"; level="L0"; lead_time="165500";' \ forecast2.grb 'name="APCP"; level="L0"; lead_time="160500";' \ forecast.nc -name APCP_A005000 @@ -877,7 +876,7 @@ GFS. In this example, process the 20150220 00Z initialization of GFS. .. code-block:: none - ${MET_BUILD_BASE}/bin/pcp_combine \ + pcp_combine \ -sum 20150220_00 06 20150221_00 24 \ gfs_APCP_24_20150220_00_F00_F24.nc \ -pcprx "gfs_4_20150220_00.*grb2" \ @@ -897,7 +896,7 @@ hours 12 and 36: .. code-block:: none - ${MET_BUILD_BASE}/bin/pcp_combine \ + pcp_combine \ -sum 20150220_00 06 20150221_12 24 \ gfs_APCP_24_20150220_00_F12_F36.nc \ -pcprx "gfs_4_20150220_00.*grb2" \ @@ -911,7 +910,7 @@ the previous "-sum" job could be rewritten with "-add" like this: .. code-block:: none - ${MET_BUILD_BASE}/bin/pcp_combine -add \ + pcp_combine -add \ /d1/model_data/20150220/gfs_4_20150220_0000_018.grb2 06 \ /d1/model_data/20150220/gfs_4_20150220_0000_024.grb2 06 \ /d1/model_data/20150220/gfs_4_20150220_0000_030.grb2 06 \ @@ -950,13 +949,12 @@ Plot-Data-Plane A. Check to see if the call to Gen-Vx-Mask actually did create good output -with Plot-Data-Plane. -Try running the following command from the top-level ${MET_BUILD_BASE} -directory. +with Plot-Data-Plane. The following commands assume that the MET executables +are found in your path. .. code-block:: none - bin/plot_data_plane \ + plot_data_plane \ out/gen_vx_mask/CONUS_poly.nc \ out/gen_vx_mask/CONUS_poly.ps \ 'name="CONUS"; level="(*,*)";' @@ -997,7 +995,7 @@ MET configuration files (i.e. Grid-Stat, MODE, and so on) that you use: .. code-block:: none - {MET_BASE}/bin/plot_data_plane \ + plot_data_plane \ test_2.5_prog.grib \ test_2.5_prog.ps \ 'name="TSTM"; level="A0"; file_type=GRIB2;' \ @@ -1013,7 +1011,7 @@ by running: .. code-block:: none - ${MET_BUILD_BASE}/bin/plot_data_plane LTIA98_KWBR_201305180600.grb2 tmp_z2.ps 'name="TMP"; level="R2"; + plot_data_plane LTIA98_KWBR_201305180600.grb2 tmp_z2.ps 'name="TMP"; level="R2"; "R2" tells MET to plot record number 2. Record numbers 1 and 2 both contain temperature data and 2-meters. Here's some wgrib2 output: @@ -1039,9 +1037,9 @@ named wrf.grb, please try running the following commands to plot wind speed: .. code-block:: none - ${MET_BUILD_BASE}/bin/plot_data_plane wrf.grb wrf_wind.ps \ + plot_data_plane wrf.grb wrf_wind.ps \ 'name"WIND"; level="Z10";' -v 3 - ${MET_BUILD_BASE}/bin/plot_data_plane rtma.grb2 rtma_wind.ps \ + plot_data_plane rtma.grb2 rtma_wind.ps \ 'name"WIND"; level="Z10";' -v 3 In the first call, the log message should be similar to this: @@ -1114,7 +1112,7 @@ Run the "aggregate" job type in stat_analysis to do this: .. code-block:: none - ${MET_BUILD_BASE}/bin/stat_analysis -lookin directory/file*_nbrcnt.txt \ + stat_analysis -lookin directory/file*_nbrcnt.txt \ -job aggregate -line_type NBRCNT -by FCST_VAR,FCST_LEAD,FCST_THRESH,INTERP_MTHD,INTERP_PNTS -out_stat agg_nbrcnt.txt This job reads all the files that are passed to it on the command line with @@ -1141,7 +1139,7 @@ to run the job separately for each unique entry found in the FCST_VAR column. .. code-block:: none - ${MET_BUILD_BASE}/bin/stat_analysis \ + stat_analysis \ -lookin point_stat_model2_120000L_20160501_120000V.stat \ -job aggregate_stat -line_type MPR -out_line_type PSTD \ -out_fcst_thresh ge0,ge0.1,ge0.2,ge0.3,ge0.4,ge0.5,ge0.6,ge0.7,ge0.8,ge0.9,ge1.0 \ @@ -1161,7 +1159,7 @@ cases without having to modify the source code. .. code-block:: none - ${MET_BUILD_BASE}/bin/stat_analysis \ + stat_analysis \ -lookin out/grid_stat/grid_stat_120000L_20050807_120000V.stat \ -job filter -dump_row filter_cts.txt -line_type CTS \ -column_min BASER 0.05 -column_min FMEAN 0.05 @@ -1193,7 +1191,7 @@ Run the following job on the output from Grid-Stat generated when the .. code-block:: none - ${MET_BUILD_BASE}/bin/stat_analysis -lookin out/grid_stat \ + stat_analysis -lookin out/grid_stat \ -job aggregate_stat -line_type SL1L2 -out_line_type CNT \ -by FCST_VAR,FCST_LEV \ -out_stat cnt.txt @@ -1226,7 +1224,7 @@ Two more suggestions for faster run times. .. code-block:: none - ${MET_BUILD_BASE}/bin/stat_analysis \ + stat_analysis \ -lookin diag_conv_anl.2015060100.stat \ -job aggregate_stat -line_type MPR -out_line_type CNT -by FCST_VAR \ -out_stat diag_conv_anl.2015060100_cnt.txt -set_hdr OBTYPE ALL_TYPES \ @@ -1250,7 +1248,7 @@ all grouped together. .. code-block:: none - ${MET_BUILD_BASE}/bin/tc_stat \ + tc_stat \ -lookin d2014_vx_20141117_reset/al/tc_pairs/tc_pairs_H3WI_* \ -lookin d2014_vx_20141117_reset/al/tc_pairs/tc_pairs_HWFI_* \ -job summary -lead 480000 -column TRACK -amodel HWFI,H3WI \ @@ -1266,7 +1264,7 @@ To get the most output, run something like this: .. code-block:: none - ${MET_BUILD_BASE}/bin/tc_stat \ + tc_stat \ -lookin path/to/tc_pairs/output \ -job rirw -dump_row test \ -out_line_type CTC,CTS,MPR @@ -1280,7 +1278,7 @@ in time. .. code-block:: none - {MET_BASE}/bin/tc_stat \ + tc_stat \ -lookin path/to/tc_pairs/output \ -job rirw -dump_row test \ -rirw_time 36 -rirw_window 12 \ @@ -1291,7 +1289,7 @@ To stratify your results by lead time, you could add the "-by LEAD" option. .. code-block:: none - {MET_BASE}/bin/tc_stat \ + tc_stat \ -lookin path/to/tc_pairs/output \ -job rirw -dump_row test \ -rirw_time 36 -rirw_window 12 \ @@ -1314,9 +1312,9 @@ and call convert to reformat from PostScript to PNG. #!/bin/sh for case in `echo "FCST OBS"`; do export TO_GRID=${case} - /usr/local/${MET_BUILD_BASE}/bin/grid_stat gfs.t00z.pgrb2.0p25.f000 \ - nam.t00z.conusnest.hiresf00.tm00.grib2 GridStatConfig \ - /usr/local/${MET_BUILD_BASE}/bin/plot_data_plane \ + grid_stat gfs.t00z.pgrb2.0p25.f000 \ + nam.t00z.conusnest.hiresf00.tm00.grib2 GridStatConfig + plot_data_plane \ *TO_GRID_${case}*_pairs.nc TO_GRID_${case}.ps 'name="DIFF_TMP_P500_TMP_P500_FULL"; \ level="(*,*)";' convert -rotate 90 -background white -flatten TO_GRID_${case}.ps @@ -1349,7 +1347,7 @@ http://dtcenter.org/community-code/model-evaluation-tools-met/input-data Rscript trmmbin2nc.R 3B42.100921.00z.7.precipitation.bin \ 3B42.100921.00z.7.precipitation.nc # Plot the result - ${MET_BUILD_BASE}/bin/plot_data_plane 3B42.100921.00z.7.precipitation.nc \ + plot_data_plane 3B42.100921.00z.7.precipitation.nc \ 3B42.100921.00z.7.precipitation.ps 'name="APCP_03"; level="(*,*)";' It may be possible that the domain of the data is smaller. Here are some options: @@ -1413,8 +1411,7 @@ and then plot the result. .. code-block:: none - setenv MET_BUILD_BASE `pwd` - Rscript scripts/Rscripts/plot_tcmpr.R \ + Rscript ${MET_BUILD_BASE}/scripts/Rscripts/plot_tcmpr.R \ -lookin tc_pairs_output.tcst \ -filter '-amodel AHWI,GFSI' \ -series AMODEL AHWI,GFSI,AHWI-GFSI \ @@ -1452,7 +1449,7 @@ some GFS data to a LatLon grid: .. code-block:: none - ${MET_BUILD_BASE}/bin/regrid_data_plane \ + regrid_data_plane \ gfs_2012040900_F012.grib G110 \ gfs_g110.nc -field 'name="TMP"; level="Z2";' @@ -1499,7 +1496,7 @@ Below is a command example to run: .. code-block:: none - ${MET_BUILD_BASE}/bin/tc_pairs \ + tc_pairs \ -adeck aep142014.h4hw.dat \ -bdeck bep142014.dat \ -config TCPairsConfig_v5.0 \ @@ -1557,7 +1554,7 @@ this example (suffix=_EXP): .. code-block:: none - ${MET_BUILD_BASE}/bin/tc_pairs \ + tc_pairs \ -adeck aal032014.h4hw.dat suffix=_EXP \ -adeck aal032014_hfip_d2014_BERTHA.dat \ -bdeck bal032014.dat \ @@ -1753,7 +1750,7 @@ breaking the command up like the below example. .. code-block:: none - ['/h/WXQC/{MET_BUILD_BASE}/bin/regrid_data_plane', + ['regrid_data_plane', '/h/data/global/WXQC/data/umm/1701150006', 'G003', '/h/data/global/WXQC/data/met/nc_mdl/umm/1701150006', '- field', '\'name="HGT"; level="P500";\'', '-v', '6'] From ee4a3a4b725826f57a0060c4b0aa0aab20c8baca Mon Sep 17 00:00:00 2001 From: johnhg Date: Fri, 14 Jan 2022 17:56:00 -0700 Subject: [PATCH 64/82] feature 2011 v10.1.0-beta5 (#2014) --- met/docs/Users_Guide/release-notes.rst | 21 +++++++++++++++++++++ met/docs/conf.py | 6 +++--- 2 files changed, 24 insertions(+), 3 deletions(-) diff --git a/met/docs/Users_Guide/release-notes.rst b/met/docs/Users_Guide/release-notes.rst index eebd4f9e28..ac8a6bcd85 100644 --- a/met/docs/Users_Guide/release-notes.rst +++ b/met/docs/Users_Guide/release-notes.rst @@ -5,6 +5,27 @@ When applicable, release notes are followed by the GitHub issue number which describes the bugfix, enhancement, or new feature: `MET GitHub issues. `_ +MET Version 10.1.0-beta5 release notes (20220114) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +* Enhancements: + + * **Enhance GridStat to use OpenMP for efficient computation of neighborhood statistics by setting $OMP_NUM_THREADS** (`#1926 `_). + * **Enhance EnsembleStat and GenEnsProd to read all ensemble members from a single input file** (`#1695 `_). + * **Enhance TCGen to verify NHC tropical weather outlook shapefiles** (`#1810 `_). + * **Refine logic to prevent rounding shapefile points to the nearest grid point (affects GenVxMask -type shape masks)** (`#1810 `_). + * Enhance MADIS2NC to handle the 2016 updates to its format (`#1936 `_). + * Modify the interpretation of the message_type_group_map values to support the use of regular expressions (`#1974 `_). + * Address findings from the Cppcheck code analysis tool (`#1996 `_). + * Sort files read from directories to provide consistent behavior across platforms (`#1989 `_). + +* Bugfixes: + + * **Fix MTD to compute the CDIST_TRAVELLED value correctly** (`#1976 `_). + * Fix PointStat and GridStat to write VCNT output even if no VL1L2 or VAL1L2 output is requested (`#1991 `_). + * Fix GenVxMask to handle named grids and grid specification strings for -type grid (`#1993 `_). + * Fix IODA2NC to handle the same input file being provided multiple times (`#1965 `_). + MET Version 10.1.0-beta4 release notes (20211117) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/met/docs/conf.py b/met/docs/conf.py index c04f0ceeb3..775b1d88ea 100644 --- a/met/docs/conf.py +++ b/met/docs/conf.py @@ -20,11 +20,11 @@ project = 'MET' author = 'UCAR/NCAR, NOAA, CSU/CIRA, and CU/CIRES' author_list = 'Halley Gotway, J., K. Newman, H. Soh, J. Opatz, T. Jensen, J. Prestopnik, L. Goodrich, D. Fillmore, B. Brown, R. Bullock, T. Fowler' -version = '10.1.0-beta4' +version = '10.1.0-beta5' verinfo = version release = f'{version}' -release_year = '2021' -release_date = f'{release_year}-11-17' +release_year = '2022' +release_date = f'{release_year}-01-14' copyright = f'{release_year}, {author}' # -- General configuration --------------------------------------------------- From 18ccf3ad7f8585f381424770a0bfbd5f41beb00c Mon Sep 17 00:00:00 2001 From: George McCabe <23407799+georgemccabe@users.noreply.github.com> Date: Fri, 14 Jan 2022 21:08:33 -0700 Subject: [PATCH 65/82] feature 1695 fix issues with ensemble changes (#2012) Co-authored-by: John Halley Gotway Co-authored-by: John Halley Gotway --- .../tools/core/ensemble_stat/ensemble_stat.cc | 22 +++++--- .../tools/core/ensemble_stat/ensemble_stat.h | 4 +- .../ensemble_stat/ensemble_stat_conf_info.cc | 55 +++++++++++-------- .../tools/other/gen_ens_prod/gen_ens_prod.cc | 3 +- .../gen_ens_prod/gen_ens_prod_conf_info.cc | 10 +++- .../gen_ens_prod/gen_ens_prod_conf_info.h | 2 +- scripts/environment/development.seneca | 2 +- .../EnsembleStatConfig_single_file_grib | 8 +-- test/config/EnsembleStatConfig_single_file_nc | 36 ++++++------ test/xml/unit_ensemble_stat.xml | 12 +++- 10 files changed, 93 insertions(+), 61 deletions(-) diff --git a/met/src/tools/core/ensemble_stat/ensemble_stat.cc b/met/src/tools/core/ensemble_stat/ensemble_stat.cc index 0299932cf5..ac3ad664a6 100644 --- a/met/src/tools/core/ensemble_stat/ensemble_stat.cc +++ b/met/src/tools/core/ensemble_stat/ensemble_stat.cc @@ -62,7 +62,8 @@ // 031 09/13/21 Seth Linden Changed obs_qty to obs_qty_inc. // Added code for obs_qty_exc. // 032 10/07/21 Halley Gotway MET #1905 Add -ctrl option. -// 032 11/15/21 Halley Gotway MET #1968 Ensemble -ctrl error check. +// 033 11/15/21 Halley Gotway MET #1968 Ensemble -ctrl error check. +// 034 01/14/21 McCabe MET #1695 All members in one file. // //////////////////////////////////////////////////////////////////////// @@ -302,7 +303,7 @@ void process_command_line(int argc, char **argv) { // Copy ensemble file list to forecast file list fcst_file_list = ens_file_list; - // Prepend the control member, if specified + // Append the control member, if specified if(ctrl_file.nonempty()) { if(ens_file_list.has(ctrl_file) && n_ens_files != 1) { @@ -312,10 +313,9 @@ void process_command_line(int argc, char **argv) { exit(1); } - ctrl_index = 0; - - // Add control file to beginning of forecast file list - fcst_file_list.insert(ctrl_index, ctrl_file.c_str()); + // Add control member file to end of the forecast file list + fcst_file_list.add(ctrl_file.c_str()); + ctrl_file_index = fcst_file_list.n()-1; } // Check that the end_ut >= beg_ut @@ -948,8 +948,12 @@ void process_vx() { // Process masks Grids and Polylines in the config file conf_info.process_masks(grid); + // Determine the index of the control member in list of data values + int ctrl_data_index = (is_bad_data(ctrl_file_index) ? + bad_data_int : fcst_file_vld.sum()-1); + // Setup the PairDataEnsemble objects - conf_info.set_vx_pd(n_vx_vld, ctrl_index); + conf_info.set_vx_pd(n_vx_vld, ctrl_data_index); // Process the point observations if(point_obs_flag) process_point_vx(); @@ -1200,7 +1204,7 @@ int process_point_ens(int i_ens, int &n_miss) { mlog << Debug(2) << "\n" << sep_str << "\n\n" << "Processing " << file_type << " file: " << ens_file - << (i_ens == ctrl_index ? " (control)\n" : "\n"); + << (i_ens == ctrl_file_index ? " (control)\n" : "\n"); // Loop through each of the fields to be verified and extract // the forecast fields for verification @@ -2012,7 +2016,7 @@ void process_grid_scores(int i_vx, // Store the unperturbed ensemble value // Exclude the control member from the variance - if(j != ctrl_index) pd.add_ens_var_sums(i, fraw_dp[j](x, y)); + if(j != ctrl_file_index) pd.add_ens_var_sums(i, fraw_dp[j](x, y)); } } // end for j diff --git a/met/src/tools/core/ensemble_stat/ensemble_stat.h b/met/src/tools/core/ensemble_stat/ensemble_stat.h index 2cc02de2a1..482c0dfceb 100644 --- a/met/src/tools/core/ensemble_stat/ensemble_stat.h +++ b/met/src/tools/core/ensemble_stat/ensemble_stat.h @@ -107,8 +107,8 @@ static bool ens_mean_flag; // Flag for ensemble mean processing static ConcatString ens_mean_user; // User-specified ensemble mean data file static ConcatString ens_mean_file; // Computed ensemble mean output file -static ConcatString ctrl_file; // Control member -static int ctrl_index = bad_data_int; // Control member index +static ConcatString ctrl_file; // Control member file +static int ctrl_file_index = bad_data_int; // Control member file index // Input Observation files static StringArray grid_obs_file_list; diff --git a/met/src/tools/core/ensemble_stat/ensemble_stat_conf_info.cc b/met/src/tools/core/ensemble_stat/ensemble_stat_conf_info.cc index 6db4407c39..85ed9581ca 100644 --- a/met/src/tools/core/ensemble_stat/ensemble_stat_conf_info.cc +++ b/met/src/tools/core/ensemble_stat/ensemble_stat_conf_info.cc @@ -212,6 +212,14 @@ void EnsembleStatConfInfo::process_config(GrdFileType etype, exit(1); } + // The control ID must be set when the control file is specified + if(control_id.empty() && use_ctrl) { + mlog << Error << "\nEnsembleStatConfInfo::process_config() -> " + << "the control_id must be set if processing a single input " + << "file with the -ctrl option\n\n"; + exit(1); + } + // If control ID is set, it cannot be found in ens_member_ids if(!control_id.empty() && ens_member_ids.has(control_id)) { mlog << Error << "\nEnsembleStatConfInfo::process_config() -> " @@ -323,6 +331,9 @@ void EnsembleStatConfInfo::process_config(GrdFileType etype, ens_input.push_back(ens_info); } // end for i + // Unset MET_ENS_MEMBER_ID that was previously set + unsetenv(met_ens_member_id); + // Conf: ens.ens_thresh vld_ens_thresh = conf.lookup_double(conf_key_ens_ens_thresh); @@ -757,7 +768,6 @@ void EnsembleStatVxOpt::process_config(GrdFileType ftype, Dictionary &fdict, StringArray * fcst_files, bool use_ctrl, ConcatString control_id) { int i, j; - int file_index_start = 0; VarInfoFactory info_factory; mapoutput_map; Dictionary *dict; @@ -770,11 +780,11 @@ void EnsembleStatVxOpt::process_config(GrdFileType ftype, Dictionary &fdict, // Allocate new EnsVarInfo object for fcst vx_pd.fcst_info = new EnsVarInfo(); - // Add control member as first input - if(use_ctrl) { + // Loop over ensemble member IDs to substitute + for(i=0; iadd_input(input_info); - // Change the starting index to the first non-control file - file_index_start = 1; - } + // Add InputInfo to fcst info list for each ensemble file provided + // set var_info to NULL to note first VarInfo should be used + int last_member_index = fcst_files->n() - (use_ctrl ? 1 : 0); + for(j=1; jadd_input(input_info); + } // end for j + } // end for i - // Loop over ensemble member IDs to substitute - for(i=0; iset_dict(fdict); input_info.var_info = next_var; - input_info.file_index = file_index_start; + input_info.file_index = fcst_files->n() - 1; input_info.file_list = fcst_files; vx_pd.fcst_info->add_input(input_info); - - // Add InputInfo to fcst info list for each ensemble file provided - // set var_info to NULL to note first VarInfo should be used - for(j=file_index_start+1; jn(); j++) { - input_info.var_info = NULL; - input_info.file_index = j; - input_info.file_list = fcst_files; - vx_pd.fcst_info->add_input(input_info); - } // end for j - - } // end for i + } // Allocate new VarInfo object for obs vx_pd.obs_info = info_factory.new_var_info(otype); diff --git a/met/src/tools/other/gen_ens_prod/gen_ens_prod.cc b/met/src/tools/other/gen_ens_prod/gen_ens_prod.cc index 33d8351c8d..8fb7656e3d 100644 --- a/met/src/tools/other/gen_ens_prod/gen_ens_prod.cc +++ b/met/src/tools/other/gen_ens_prod/gen_ens_prod.cc @@ -16,6 +16,7 @@ // ---- ---- ---- ----------- // 000 09/10/21 Halley Gotway MET #1904 Initial version. // 001 11/15/21 Halley Gotway MET #1968 Ensemble -ctrl error check. +// 002 01/14/21 McCabe MET #1695 All members in one file. // //////////////////////////////////////////////////////////////////////// @@ -186,7 +187,7 @@ void process_command_line(int argc, char **argv) { etype = ens_mtddf->file_type(); // Process the configuration - conf_info.process_config(etype, &ens_files); + conf_info.process_config(etype, &ens_files, ctrl_file.nonempty()); // Allocate arrays to store threshold counts thresh_cnt_na = new NumArray [conf_info.get_max_n_cat()]; diff --git a/met/src/tools/other/gen_ens_prod/gen_ens_prod_conf_info.cc b/met/src/tools/other/gen_ens_prod/gen_ens_prod_conf_info.cc index 96ee7be396..f085a20c24 100644 --- a/met/src/tools/other/gen_ens_prod/gen_ens_prod_conf_info.cc +++ b/met/src/tools/other/gen_ens_prod/gen_ens_prod_conf_info.cc @@ -100,7 +100,7 @@ void GenEnsProdConfInfo::read_config(const ConcatString default_file_name, //////////////////////////////////////////////////////////////////////// -void GenEnsProdConfInfo::process_config(GrdFileType etype, StringArray * ens_files) { +void GenEnsProdConfInfo::process_config(GrdFileType etype, StringArray * ens_files, bool use_ctrl) { int i, j; VarInfoFactory info_factory; Dictionary *edict = (Dictionary *) 0; @@ -146,6 +146,14 @@ void GenEnsProdConfInfo::process_config(GrdFileType etype, StringArray * ens_fil exit(1); } + // The control ID must be set when the control file is specified + if(control_id.empty() && use_ctrl) { + mlog << Error << "\nGenEnsProdConfInfo::process_config() -> " + << "the control_id must be set if processing a single input " + << "file with the -ctrl option\n\n"; + exit(1); + } + // If control ID is set, it cannot be found in ens_member_ids if(!control_id.empty() && ens_member_ids.has(control_id)) { mlog << Error << "\nGenEnsProdConfInfo::process_config() -> " diff --git a/met/src/tools/other/gen_ens_prod/gen_ens_prod_conf_info.h b/met/src/tools/other/gen_ens_prod/gen_ens_prod_conf_info.h index 9510cc32b7..ae455daac7 100644 --- a/met/src/tools/other/gen_ens_prod/gen_ens_prod_conf_info.h +++ b/met/src/tools/other/gen_ens_prod/gen_ens_prod_conf_info.h @@ -99,7 +99,7 @@ class GenEnsProdConfInfo { void clear(); void read_config (const ConcatString, const ConcatString); - void process_config(GrdFileType, StringArray *); + void process_config(GrdFileType, StringArray *, bool); GenEnsProdNcOutInfo parse_nc_info(Dictionary *); diff --git a/scripts/environment/development.seneca b/scripts/environment/development.seneca index a535aeaa67..650257b3ab 100644 --- a/scripts/environment/development.seneca +++ b/scripts/environment/development.seneca @@ -27,7 +27,7 @@ export MET_PYTHON_CC="-I${MET_PYTHON}/include/python3.8" export MET_PYTHON_LD="-L${MET_PYTHON}/lib -lpython3.8 -lcrypt -lpthread -ldl -lutil -lm" # -D__64BIT__ is required because we've compiled libgrib2c.a with that flag -export CFLAGS="-DUNDERSCORE -fPIC -D__64BIT__" +export CFLAGS="-DUNDERSCORE -fPIC -D__64BIT__ -g" export CXXFLAGS=${CFLAGS} # Set LDFLAGS to include -rpath settings when compiling MET diff --git a/test/config/EnsembleStatConfig_single_file_grib b/test/config/EnsembleStatConfig_single_file_grib index cae96307fe..baf0c18c19 100644 --- a/test/config/EnsembleStatConfig_single_file_grib +++ b/test/config/EnsembleStatConfig_single_file_grib @@ -285,13 +285,13 @@ ensemble_flag = { latlon = TRUE; mean = TRUE; stdev = TRUE; - minus = TRUE; - plus = TRUE; + minus = FALSE; + plus = FALSE; min = TRUE; max = TRUE; - range = TRUE; + range = FALSE; vld_count = TRUE; - frequency = TRUE; + frequency = FALSE; nep = FALSE; nmep = FALSE; rank = TRUE; diff --git a/test/config/EnsembleStatConfig_single_file_nc b/test/config/EnsembleStatConfig_single_file_nc index 47ed145c44..610883a812 100644 --- a/test/config/EnsembleStatConfig_single_file_nc +++ b/test/config/EnsembleStatConfig_single_file_nc @@ -9,7 +9,7 @@ // // Output model name to be written // -model = "WRF"; +model = "CFS"; // // Output description to be written @@ -46,19 +46,26 @@ censor_val = []; cat_thresh = []; nc_var_str = ""; +// +// Override the NetCDF CFS variable names and times +// +file_type = NETCDF_NCCF; +set_attr_init = "19820101"; +set_attr_valid = "19820701"; +set_attr_name = "TMP"; +set_attr_level = "Z2"; + // // Ensemble product fields to be processed // ens = { - file_type = NETCDF_NCCF; ens_thresh = 1.0; vld_thresh = 1.0; field = [ { - name = "fcst"; - level = "(MET_ENS_MEMBER_ID,0,*,*)"; - cat_thresh = [ >0.0, >=5.0 ]; + name = "fcst"; + level = "(MET_ENS_MEMBER_ID,6,*,*)"; } ]; } @@ -103,12 +110,10 @@ nmep_smooth = { // Forecast and observation fields to be verified // fcst = { - file_type = NETCDF_NCCF; field = [ { - name = "fcst"; - level = "(MET_ENS_MEMBER_ID,0,*,*)"; - cat_thresh = [ >0.0, >=5.0 ]; + name = "fcst"; + level = "(MET_ENS_MEMBER_ID,6,*,*)"; } ]; } @@ -116,9 +121,8 @@ fcst = { obs = { field = [ { - name = "fcst"; - level = "(0,0,*,*)"; - cat_thresh = [ >0.0, >=5.0 ]; + name = "fcst"; + level = "(0,6,*,*)"; } ]; } @@ -283,13 +287,13 @@ ensemble_flag = { latlon = TRUE; mean = TRUE; stdev = TRUE; - minus = TRUE; - plus = TRUE; + minus = FALSE; + plus = FALSE; min = TRUE; max = TRUE; - range = TRUE; + range = FALSE; vld_count = TRUE; - frequency = TRUE; + frequency = FALSE; nep = FALSE; nmep = FALSE; rank = TRUE; diff --git a/test/xml/unit_ensemble_stat.xml b/test/xml/unit_ensemble_stat.xml index 034ff07cbf..93ec40b228 100644 --- a/test/xml/unit_ensemble_stat.xml +++ b/test/xml/unit_ensemble_stat.xml @@ -266,7 +266,9 @@ -outdir &OUTPUT_DIR;/ensemble_stat -v 1 - &OUTPUT_DIR;/ensemble_stat/ensemble_stat_SINGLE_FILE_NC_NO_CTRL_19700101_000000V_ens.nc + &OUTPUT_DIR;/ensemble_stat/ensemble_stat_SINGLE_FILE_NC_NO_CTRL_19820701_000000V.stat + &OUTPUT_DIR;/ensemble_stat/ensemble_stat_SINGLE_FILE_NC_NO_CTRL_19820701_000000V_ens.nc + &OUTPUT_DIR;/ensemble_stat/ensemble_stat_SINGLE_FILE_NC_NO_CTRL_19820701_000000V_orank.nc @@ -287,7 +289,9 @@ -outdir &OUTPUT_DIR;/ensemble_stat -v 1 - &OUTPUT_DIR;/ensemble_stat/ensemble_stat_SINGLE_FILE_NC_WITH_CTRL_19700101_000000V_ens.nc + &OUTPUT_DIR;/ensemble_stat/ensemble_stat_SINGLE_FILE_NC_WITH_CTRL_19820701_000000V.stat + &OUTPUT_DIR;/ensemble_stat/ensemble_stat_SINGLE_FILE_NC_WITH_CTRL_19820701_000000V_ens.nc + &OUTPUT_DIR;/ensemble_stat/ensemble_stat_SINGLE_FILE_NC_WITH_CTRL_19820701_000000V_orank.nc @@ -307,7 +311,9 @@ -outdir &OUTPUT_DIR;/ensemble_stat -v 1 + &OUTPUT_DIR;/ensemble_stat/ensemble_stat_SINGLE_FILE_GRIB_NO_CTRL_20160608_060000V.stat &OUTPUT_DIR;/ensemble_stat/ensemble_stat_SINGLE_FILE_GRIB_NO_CTRL_20160608_060000V_ens.nc + &OUTPUT_DIR;/ensemble_stat/ensemble_stat_SINGLE_FILE_GRIB_NO_CTRL_20160608_060000V_orank.nc @@ -328,7 +334,9 @@ -outdir &OUTPUT_DIR;/ensemble_stat -v 1 + &OUTPUT_DIR;/ensemble_stat/ensemble_stat_SINGLE_FILE_GRIB_WITH_CTRL_20160608_060000V.stat &OUTPUT_DIR;/ensemble_stat/ensemble_stat_SINGLE_FILE_GRIB_WITH_CTRL_20160608_060000V_ens.nc + &OUTPUT_DIR;/ensemble_stat/ensemble_stat_SINGLE_FILE_GRIB_WITH_CTRL_20160608_060000V_orank.nc From e9df2fc4b5773fa75448dfafef4c4a33ccf8700f Mon Sep 17 00:00:00 2001 From: John Halley Gotway Date: Sun, 16 Jan 2022 10:46:01 -0700 Subject: [PATCH 66/82] Hotfix for #1695 committed directly to the develop branch. The new Ensemble-Stat config files added for this issue don't define a random number generator seed. This results in different output for each run and prevents the regression test from producing repeatable results. Defining the seed here to fix that. --- test/config/EnsembleStatConfig_python | 2 +- test/config/EnsembleStatConfig_single_file_grib | 2 +- test/config/EnsembleStatConfig_single_file_nc | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/test/config/EnsembleStatConfig_python b/test/config/EnsembleStatConfig_python index 1e137fad3d..59ae439495 100644 --- a/test/config/EnsembleStatConfig_python +++ b/test/config/EnsembleStatConfig_python @@ -274,7 +274,7 @@ ensemble_flag = { // rng = { type = "mt19937"; - seed = ""; + seed = "1"; } //////////////////////////////////////////////////////////////////////////////// diff --git a/test/config/EnsembleStatConfig_single_file_grib b/test/config/EnsembleStatConfig_single_file_grib index baf0c18c19..76bb0bf3b5 100644 --- a/test/config/EnsembleStatConfig_single_file_grib +++ b/test/config/EnsembleStatConfig_single_file_grib @@ -305,7 +305,7 @@ ensemble_flag = { // rng = { type = "mt19937"; - seed = ""; + seed = "1"; } //////////////////////////////////////////////////////////////////////////////// diff --git a/test/config/EnsembleStatConfig_single_file_nc b/test/config/EnsembleStatConfig_single_file_nc index 610883a812..67bd0e3399 100644 --- a/test/config/EnsembleStatConfig_single_file_nc +++ b/test/config/EnsembleStatConfig_single_file_nc @@ -307,7 +307,7 @@ ensemble_flag = { // rng = { type = "mt19937"; - seed = ""; + seed = "1"; } //////////////////////////////////////////////////////////////////////////////// From 65539e111d10668adbfa782b73d82a40416e20ee Mon Sep 17 00:00:00 2001 From: John Halley Gotway Date: Tue, 18 Jan 2022 09:24:25 -0700 Subject: [PATCH 67/82] Adding George to email list for the nightly build. --- scripts/regression/test_nightly.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/regression/test_nightly.sh b/scripts/regression/test_nightly.sh index 992981f702..96930bb30a 100755 --- a/scripts/regression/test_nightly.sh +++ b/scripts/regression/test_nightly.sh @@ -21,7 +21,7 @@ #======================================================================= # Constants -EMAIL_LIST="johnhg@ucar.edu hsoh@ucar.edu jpresto@ucar.edu linden@ucar.edu" +EMAIL_LIST="johnhg@ucar.edu hsoh@ucar.edu jpresto@ucar.edu linden@ucar.edu mccabe@ucar.edu" KEEP_DAYS=5 # Usage statement From 7c182317e0f0cdc128fc82cdc01667200d024ee4 Mon Sep 17 00:00:00 2001 From: John Halley Gotway Date: Thu, 20 Jan 2022 09:32:08 -0700 Subject: [PATCH 68/82] Fix the links for the met-10.1.0-beta5 release notes. --- met/docs/Users_Guide/release-notes.rst | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/met/docs/Users_Guide/release-notes.rst b/met/docs/Users_Guide/release-notes.rst index ac8a6bcd85..5cac6e7006 100644 --- a/met/docs/Users_Guide/release-notes.rst +++ b/met/docs/Users_Guide/release-notes.rst @@ -10,21 +10,21 @@ MET Version 10.1.0-beta5 release notes (20220114) * Enhancements: - * **Enhance GridStat to use OpenMP for efficient computation of neighborhood statistics by setting $OMP_NUM_THREADS** (`#1926 `_). - * **Enhance EnsembleStat and GenEnsProd to read all ensemble members from a single input file** (`#1695 `_). - * **Enhance TCGen to verify NHC tropical weather outlook shapefiles** (`#1810 `_). - * **Refine logic to prevent rounding shapefile points to the nearest grid point (affects GenVxMask -type shape masks)** (`#1810 `_). - * Enhance MADIS2NC to handle the 2016 updates to its format (`#1936 `_). - * Modify the interpretation of the message_type_group_map values to support the use of regular expressions (`#1974 `_). - * Address findings from the Cppcheck code analysis tool (`#1996 `_). - * Sort files read from directories to provide consistent behavior across platforms (`#1989 `_). + * **Enhance GridStat to use OpenMP for efficient computation of neighborhood statistics by setting $OMP_NUM_THREADS** (`#1926 `_). + * **Enhance EnsembleStat and GenEnsProd to read all ensemble members from a single input file** (`#1695 `_). + * **Enhance TCGen to verify NHC tropical weather outlook shapefiles** (`#1810 `_). + * **Refine logic to prevent rounding shapefile points to the nearest grid point (affects GenVxMask -type shape masks)** (`#1810 `_). + * Enhance MADIS2NC to handle the 2016 updates to its format (`#1936 `_). + * Modify the interpretation of the message_type_group_map values to support the use of regular expressions (`#1974 `_). + * Address findings from the Cppcheck code analysis tool (`#1996 `_). + * Sort files read from directories to provide consistent behavior across platforms (`#1989 `_). * Bugfixes: - * **Fix MTD to compute the CDIST_TRAVELLED value correctly** (`#1976 `_). - * Fix PointStat and GridStat to write VCNT output even if no VL1L2 or VAL1L2 output is requested (`#1991 `_). - * Fix GenVxMask to handle named grids and grid specification strings for -type grid (`#1993 `_). - * Fix IODA2NC to handle the same input file being provided multiple times (`#1965 `_). + * **Fix MTD to compute the CDIST_TRAVELLED value correctly** (`#1976 `_). + * Fix PointStat and GridStat to write VCNT output even if no VL1L2 or VAL1L2 output is requested (`#1991 `_). + * Fix GenVxMask to handle named grids and grid specification strings for -type grid (`#1993 `_). + * Fix IODA2NC to handle the same input file being provided multiple times (`#1965 `_). MET Version 10.1.0-beta4 release notes (20211117) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From b8e0de3ce3a208d8fbded5e912f7c60c7ca6726c Mon Sep 17 00:00:00 2001 From: Howard Soh Date: Fri, 21 Jan 2022 11:31:48 -0700 Subject: [PATCH 69/82] #1844 Make met_point_obs as abstract class --- met/scripts/python/met_point_obs.py | 358 ++++++++++++----------- met/scripts/python/read_met_point_obs.py | 107 +++---- 2 files changed, 237 insertions(+), 228 deletions(-) diff --git a/met/scripts/python/met_point_obs.py b/met/scripts/python/met_point_obs.py index b4e1e59e30..c306839226 100755 --- a/met/scripts/python/met_point_obs.py +++ b/met/scripts/python/met_point_obs.py @@ -1,169 +1,189 @@ -''' -Created on Nov 10, 2021 - -@author: hsoh -''' - - -import numpy as np - -COUNT_SHOW = 30 - -MET_PYTHON_OBS_ARGS = "MET_POINT_PYTHON_ARGS" - -class met_point_obs(object): - ''' - classdocs - ''' - python_prefix = 'PYTHON_POINT_RAW' - - def __init__(self, use_var_id=True): - ''' - Constructor - ''' - self.raw_data = None - self.use_var_id = use_var_id # True if variable index, False if GRIB code - - # Header - self.nhdr = 0 - self.npbhdr = 0 - self.nhdr_typ = 0 # type table - self.nhdr_sid = 0 #station_uid table - self.nhdr_vld = 0 # valid time strings - self.hdr_typ = [] # (nhdr) - self.hdr_sid = [] # (nhdr) - self.hdr_vld = [] # (nhdr) - self.hdr_lat = [] # (nhdr) - self.hdr_lon = [] # (nhdr) - self.hdr_elv = [] # (nhdr) - self.hdr_typ_table = [] # (nhdr_typ, mxstr2) - self.hdr_sid_table = [] # (nhdr_sid, mxstr2) - self.hdr_vld_table = [] # (nhdr_vld, mxstr) - self.hdr_prpt_typ = [] # optional - self.hdr_irpt_typ = [] # optional - self.hdr_inst_typ = [] # optional - - #Observation data - self.nobs = 0 - self.nobs_qty = 0 - self.nobs_var = 0 - self.obs_qty = [] # (nobs_qty ) - self.obs_hid = [] # (nobs) - self.obs_vid = [] # (nobs) veriable index or GRIB code - self.obs_lvl = [] # (nobs) - self.obs_hgt = [] # (nobs) - self.obs_val = [] # (nobs) - self.obs_qty_table = [] # (nobs_qty, mxstr) - self.obs_var_table = [] # (nobs_var, mxstr2) optional if GRIB code at self.obs_vid - - #def convert(self): - # pass - - def get_point_data(self): - if self.nhdr <= 0: - self.nhdr = len(self.hdr_lat) - if self.nobs <= 0: - self.nobs = len(self.obs_val) - if self.nhdr_typ <= 0: - self.nhdr_typ = len(self.hdr_typ_table) - if self.nhdr_sid <= 0: - self.nhdr_sid = len(self.hdr_sid_table) - if self.nhdr_vld <= 0: - self.nhdr_vld = len(self.hdr_vld_table) - if self.npbhdr <= 0: - self.npbhdr = len(self.hdr_prpt_typ) - if self.nobs_qty <= 0: - self.nobs_qty = len(self.obs_qty_table) - if self.nobs_var <= 0: - self.nobs_var = len(self.obs_var_table) - return self.__dict__ - - @staticmethod - def is_python_script(arg_value): - return arg_value.startswith(met_point_obs.python_prefix) - - @staticmethod - def get_python_script(arg_value): - return arg_value[len(met_point_obs.python_prefix)+1:] - - @staticmethod - def print_data(key, data_array, show_count=COUNT_SHOW): - if isinstance(data_array, list): - data_len = len(data_array) - if show_count >= data_len: - print(" {k:10s}: {v}".format(k=key, v= data_array)) - else: - end_offset = int(show_count/2) - print(" {k:10s}: count={v}".format(k=key, v=data_len)) - print(" {k:10s}[0:{o}] {v}".format(k=key, v=data_array[:end_offset], o=end_offset)) - print(" {k:10s}[{s}:{e}]: {v}".format(k=key, v='...', s=end_offset+1, e=data_len-end_offset-1)) - print(" {k:10s}[{s}:{e}]: {v}".format(k=key, v= data_array[-end_offset:], s=(data_len-end_offset), e=(data_len-1))) - else: - print(" {k:10s}: {v}".format(k=key, v= data_array)) - - @staticmethod - def print_point_data(met_point_data): - print(' === MET point data by python embedding ===') - met_point_obs.print_data('nhdr', met_point_data['nhdr']) - met_point_obs.print_data('nobs' , met_point_data['nobs']) - met_point_obs.print_data('use_var_id',met_point_data['use_var_id']) - met_point_obs.print_data('hdr_typ',met_point_data['hdr_typ']) - met_point_obs.print_data('obs_hid',met_point_data['obs_hid']) - met_point_obs.print_data('obs_vid',met_point_data['obs_vid']) - met_point_obs.print_data('obs_val',met_point_data['obs_val']) - met_point_obs.print_data('obs_var_table',met_point_data['obs_var_table']) - met_point_obs.print_data('obs_qty_table',met_point_data['obs_qty_table']) - met_point_obs.print_data('hdr_typ_table',met_point_data['hdr_typ_table']) - met_point_obs.print_data('hdr_sid_table',met_point_data['hdr_sid_table']) - met_point_obs.print_data('hdr_vld_table',met_point_data['hdr_vld_table']) - print(' === MET point data by python embedding ===') - - -def get_sample_point_obs(): - point_obs_data = met_point_obs() - point_obs_data.hdr_typ = np.array([ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ]) - point_obs_data.hdr_sid = np.array([ 0, 0, 0, 0, 0, 1, 2, 3, 3, 1, 2, 2, 3, 0, 0, 0, 0, 0, 1, 2, 3, 3, 1, 2, 2, 3 ]) - point_obs_data.hdr_vld = np.array([ 0, 1, 2, 3, 4, 4, 3, 4, 3, 4, 5, 4, 3, 0, 1, 2, 3, 4, 4, 3, 4, 3, 4, 5, 4, 3 ]) - point_obs_data.hdr_lat = np.array([ 43., 43., 43., 43., 43., 43., 43., 43., 43., 46., 46., 46., 46., 43., 43., 43., 43., 43., 43., 43., 43., 43., 46., 46., 46., 46. ]) - point_obs_data.hdr_lon = np.array([ -89., -89., -89., -89., -89., -89., -89., -89., -89., -92., -92., -92., -92., -89., -89., -89., -89., -89., -89., -89., -89., -89., -92., -92., -92., -92. ]) - point_obs_data.hdr_elv = np.array([ 220., 220., 220., 220., 220., 220., 220., 220., 220., 220., 220., 220., 220., 220., 220., 220., 220., 220., 220., 220., 220., 220., 220., 220., 220., 220. ]) - - point_obs_data.obs_qid = np.array([ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ]) - point_obs_data.obs_hid = np.array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 25 ]) - point_obs_data.obs_vid = np.array([ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 ]) - point_obs_data.obs_lvl = np.array([ 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000. ]) - point_obs_data.obs_hgt = np.array([ 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2. ]) - point_obs_data.obs_val = np.array([ 292., 292.5, 293., 293.5, 294., 294.5, 295., 295.5, 296., 292., 293.4, 293., 296., 294., 92., 92.5, 93., 93.5, 94., 94.5, 95., 95.5, 96., 92., 93.4, 93., 96., 94. ]) - - point_obs_data.hdr_typ_table = [ "ADPSFC" ] - point_obs_data.hdr_sid_table = [ "001", "002", "003", "004" ] - point_obs_data.hdr_vld_table = [ - "20120409_115000", "20120409_115500", "20120409_120100", "20120409_120500", "20120409_121000", - "20120409_120000" ] - point_obs_data.obs_var_table = [ "TMP", "RH" ] - point_obs_data.obs_qty_table = [ "NA" ] - point_obs_data.nhdr = len(point_obs_data.hdr_lat) - point_obs_data.nobs = len(point_obs_data.obs_val) - #point_obs_data.met_point_data = point_obs_data - return point_obs_data - - -def main(): - met_point_data = get_sample_point_obs().get_point_data() - - print('All',met_point_data) - print(" nhdr: ",met_point_data['nhdr']) - print(" nobs: ",met_point_data['nobs']) - print('use_var_id: ',met_point_data['use_var_id']) - print('hdr_typ: ',met_point_data['hdr_typ']) - print('obs_vid: ',met_point_data['obs_vid']) - print('obs_val: ',met_point_data['obs_val']) - print('obs_var_table: ',met_point_data['obs_var_table']) - print('obs_qty_table: ',met_point_data['obs_qty_table']) - print('hdr_typ_table: ',met_point_data['hdr_typ_table']) - print('hdr_sid_table: ',met_point_data['hdr_sid_table']) - print('hdr_vld_table: ',met_point_data['hdr_vld_table']) - print('Done python scripot') - -if __name__ == '__main__': - main() +''' +Created on Nov 10, 2021 + +@author: hsoh + +- This is the base class and the customized script should extend the met_point_obs. +- The customized script (for example "custom_reader") must implement "def read_data(self, args)" + which fills the array (list) variables at __init__(). +- The args can be 1) single string argument, 2) the list of arguments, and 3) the dictionary of arguments. +- The variable "met_point_data" must be set for MET tools +- The customized script is expected to include following codes: + + # prepare arguments for the customized script + args = {'input', sys.argv[1]} # or args = [] + point_obs_data = custom_reader() + point_obs_data.read_data() + met_point_data = point_obs_data.get_point_data() +''' + +from abc import ABC, abstractmethod +import numpy as np + +COUNT_SHOW = 30 + +MET_PYTHON_OBS_ARGS = "MET_POINT_PYTHON_ARGS" + +class met_point_obs(ABC): + ''' + classdocs + ''' + + python_prefix = 'PYTHON_POINT_RAW' + + def __init__(self, use_var_id=True): + ''' + Constructor + ''' + self.use_var_id = use_var_id # True if variable index, False if GRIB code + + # Header + self.nhdr = 0 + self.npbhdr = 0 + self.nhdr_typ = 0 # type table + self.nhdr_sid = 0 #station_uid table + self.nhdr_vld = 0 # valid time strings + self.hdr_typ = [] # (nhdr) + self.hdr_sid = [] # (nhdr) + self.hdr_vld = [] # (nhdr) + self.hdr_lat = [] # (nhdr) + self.hdr_lon = [] # (nhdr) + self.hdr_elv = [] # (nhdr) + self.hdr_typ_table = [] # (nhdr_typ, mxstr2) + self.hdr_sid_table = [] # (nhdr_sid, mxstr2) + self.hdr_vld_table = [] # (nhdr_vld, mxstr) + self.hdr_prpt_typ = [] # optional + self.hdr_irpt_typ = [] # optional + self.hdr_inst_typ = [] # optional + + #Observation data + self.nobs = 0 + self.nobs_qty = 0 + self.nobs_var = 0 + self.obs_qty = [] # (nobs_qty ) + self.obs_hid = [] # (nobs) + self.obs_vid = [] # (nobs) veriable index or GRIB code + self.obs_lvl = [] # (nobs) + self.obs_hgt = [] # (nobs) + self.obs_val = [] # (nobs) + self.obs_qty_table = [] # (nobs_qty, mxstr) + self.obs_var_table = [] # (nobs_var, mxstr2) optional if GRIB code at self.obs_vid + + @abstractmethod + def read_data(self, args): + # args should be list or dictionary + pass + + def get_point_data(self): + if self.nhdr <= 0: + self.nhdr = len(self.hdr_lat) + if self.nobs <= 0: + self.nobs = len(self.obs_val) + if self.nhdr_typ <= 0: + self.nhdr_typ = len(self.hdr_typ_table) + if self.nhdr_sid <= 0: + self.nhdr_sid = len(self.hdr_sid_table) + if self.nhdr_vld <= 0: + self.nhdr_vld = len(self.hdr_vld_table) + if self.npbhdr <= 0: + self.npbhdr = len(self.hdr_prpt_typ) + if self.nobs_qty <= 0: + self.nobs_qty = len(self.obs_qty_table) + if self.nobs_var <= 0: + self.nobs_var = len(self.obs_var_table) + return self.__dict__ + + @staticmethod + def is_python_script(arg_value): + return arg_value.startswith(met_point_obs.python_prefix) + + @staticmethod + def get_python_script(arg_value): + return arg_value[len(met_point_obs.python_prefix)+1:] + + @staticmethod + def print_data(key, data_array, show_count=COUNT_SHOW): + if isinstance(data_array, list): + data_len = len(data_array) + if show_count >= data_len: + print(" {k:10s}: {v}".format(k=key, v= data_array)) + else: + end_offset = int(show_count/2) + print(" {k:10s}: count={v}".format(k=key, v=data_len)) + print(" {k:10s}[0:{o}] {v}".format(k=key, v=data_array[:end_offset], o=end_offset)) + print(" {k:10s}[{s}:{e}]: {v}".format(k=key, v='...', s=end_offset+1, e=data_len-end_offset-1)) + print(" {k:10s}[{s}:{e}]: {v}".format(k=key, v= data_array[-end_offset:], s=(data_len-end_offset), e=(data_len-1))) + else: + print(" {k:10s}: {v}".format(k=key, v= data_array)) + + @staticmethod + def print_point_data(met_point_data, print_subset=True): + print(' === MET point data by python embedding ===') + if print_subset: + met_point_obs.print_data('nhdr', met_point_data['nhdr']) + met_point_obs.print_data('nobs' , met_point_data['nobs']) + met_point_obs.print_data('use_var_id',met_point_data['use_var_id']) + met_point_obs.print_data('hdr_typ',met_point_data['hdr_typ']) + met_point_obs.print_data('obs_hid',met_point_data['obs_hid']) + met_point_obs.print_data('obs_vid',met_point_data['obs_vid']) + met_point_obs.print_data('obs_val',met_point_data['obs_val']) + met_point_obs.print_data('obs_var_table',met_point_data['obs_var_table']) + met_point_obs.print_data('obs_qty_table',met_point_data['obs_qty_table']) + met_point_obs.print_data('hdr_typ_table',met_point_data['hdr_typ_table']) + met_point_obs.print_data('hdr_sid_table',met_point_data['hdr_sid_table']) + met_point_obs.print_data('hdr_vld_table',met_point_data['hdr_vld_table']) + else: + print('All',met_point_data) + print(" nhdr: ",met_point_data['nhdr']) + print(" nobs: ",met_point_data['nobs']) + print('use_var_id: ',met_point_data['use_var_id']) + print('hdr_typ: ',met_point_data['hdr_typ']) + print('obs_vid: ',met_point_data['obs_vid']) + print('obs_val: ',met_point_data['obs_val']) + print('obs_var_table: ',met_point_data['obs_var_table']) + print('obs_qty_table: ',met_point_data['obs_qty_table']) + print('hdr_typ_table: ',met_point_data['hdr_typ_table']) + print('hdr_sid_table: ',met_point_data['hdr_sid_table']) + print('hdr_vld_table: ',met_point_data['hdr_vld_table']) + print(' === MET point data by python embedding ===') + + +# This is a sample drived class +class sample_met_point_obs(met_point_obs): + + #@abstractmethod + def read_data(self, arg_map={}): + self.hdr_typ = np.array([ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ]) + self.hdr_sid = np.array([ 0, 0, 0, 0, 0, 1, 2, 3, 3, 1, 2, 2, 3, 0, 0, 0, 0, 0, 1, 2, 3, 3, 1, 2, 2, 3 ]) + self.hdr_vld = np.array([ 0, 1, 2, 3, 4, 4, 3, 4, 3, 4, 5, 4, 3, 0, 1, 2, 3, 4, 4, 3, 4, 3, 4, 5, 4, 3 ]) + self.hdr_lat = np.array([ 43., 43., 43., 43., 43., 43., 43., 43., 43., 46., 46., 46., 46., 43., 43., 43., 43., 43., 43., 43., 43., 43., 46., 46., 46., 46. ]) + self.hdr_lon = np.array([ -89., -89., -89., -89., -89., -89., -89., -89., -89., -92., -92., -92., -92., -89., -89., -89., -89., -89., -89., -89., -89., -89., -92., -92., -92., -92. ]) + self.hdr_elv = np.array([ 220., 220., 220., 220., 220., 220., 220., 220., 220., 220., 220., 220., 220., 220., 220., 220., 220., 220., 220., 220., 220., 220., 220., 220., 220., 220. ]) + + self.obs_qid = np.array([ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ]) + self.obs_hid = np.array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 25 ]) + self.obs_vid = np.array([ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 ]) + self.obs_lvl = np.array([ 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000. ]) + self.obs_hgt = np.array([ 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2. ]) + self.obs_val = np.array([ 292., 292.5, 293., 293.5, 294., 294.5, 295., 295.5, 296., 292., 293.4, 293., 296., 294., 92., 92.5, 93., 93.5, 94., 94.5, 95., 95.5, 96., 92., 93.4, 93., 96., 94. ]) + + self.hdr_typ_table = [ "ADPSFC" ] + self.hdr_sid_table = [ "001", "002", "003", "004" ] + self.hdr_vld_table = [ + "20120409_115000", "20120409_115500", "20120409_120100", "20120409_120500", "20120409_121000", + "20120409_120000" ] + self.obs_var_table = [ "TMP", "RH" ] + self.obs_qty_table = [ "NA" ] + + +def main(): + args = {} # or args = [] + point_obs_data = sample_met_point_obs() + point_obs_data.read_data(args) + met_point_data = point_obs_data.get_point_data() + + point_obs_data.print_point_data(met_point_data, print_subset=False) + +if __name__ == '__main__': + main() + print('Done python scripot') diff --git a/met/scripts/python/read_met_point_obs.py b/met/scripts/python/read_met_point_obs.py index 5a99f63a8e..15667541ec 100755 --- a/met/scripts/python/read_met_point_obs.py +++ b/met/scripts/python/read_met_point_obs.py @@ -2,6 +2,8 @@ Created on Nov 10, 2021 @author: hsoh + +This script reads the MET point observation NetCDF file like MET tools do. ''' import os @@ -10,35 +12,26 @@ import numpy as np import netCDF4 as nc -from met_point_obs import met_point_obs, get_sample_point_obs, MET_PYTHON_OBS_ARGS +from met_point_obs import met_point_obs, sample_met_point_obs DO_PRINT_DATA = False ARG_PRINT_DATA = 'show_data' class nc_point_obs(met_point_obs): - @staticmethod - def get_num_array(dataset, var_name): - nc_var = dataset.variables.get(var_name, None) - return nc_var[:].filled(nc_var._FillValue if '_FillValue' in nc_var.ncattrs() else -9999) if nc_var else [] - - @staticmethod - def get_ncbyte_array_to_str(nc_var): - nc_str_data = nc_var[:] - if nc_var.datatype.name == 'bytes8': - nc_str_data = [ str(s.compressed(),"utf-8") for s in nc_var[:]] - return nc_str_data - - @staticmethod - def get_string_array(dataset, var_name): - nc_var = dataset.variables.get(var_name, None) - return nc_point_obs.get_ncbyte_array_to_str(nc_var) if nc_var else [] - - #def convert(self): - # pass - - def read_nectdf(self, nc_filename): - if os.path.exists(nc_filename): + def read_data(self, args): + # args should be list or dictionaryu + if isinstance(args, dict): + nc_filename = args.get('nc_name',None) + elif isinstance(args, list): + nc_filename = args[0] + else: + nc_filename = args + if nc_filename is None: + print("==ERROR== The input NetCDF filename is missing") + elif not os.path.exists(nc_filename): + print("==ERROR== input NetCDF file ({f}) does not exist".format(f=nc_filename)) + else: dataset = nc.Dataset(nc_filename, 'r') # Header self.hdr_typ = dataset['hdr_typ'][:] @@ -47,10 +40,10 @@ def read_nectdf(self, nc_filename): self.hdr_lat = dataset['hdr_lat'][:] self.hdr_lon = dataset['hdr_lon'][:] self.hdr_elv = dataset['hdr_elv'][:] - self.hdr_typ_table = self.get_string_array(dataset, 'hdr_typ_table') - self.hdr_sid_table = self.get_string_array(dataset, 'hdr_sid_table') - self.hdr_vld_table = self.get_string_array(dataset, 'hdr_vld_table') - + self.hdr_typ_table = nc_tools.get_string_array(dataset, 'hdr_typ_table') + self.hdr_sid_table = nc_tools.get_string_array(dataset, 'hdr_sid_table') + self.hdr_vld_table = nc_tools.get_string_array(dataset, 'hdr_vld_table') + nc_var = dataset.variables.get('hdr_prpt_typ', None) if nc_var: self.hdr_prpt_typ = nc_var[:] @@ -62,9 +55,7 @@ def read_nectdf(self, nc_filename): self.hdr_inst_typ =nc_var[:] #Observation data - self.nobs = 0 self.hdr_sid = dataset['hdr_sid'][:] - #self.obs_qty = self.get_num_array(dataset, 'obs_qty') self.obs_qty = np.array(dataset['obs_qty'][:]) self.obs_hid = np.array(dataset['obs_hid'][:]) self.obs_lvl = np.array(dataset['obs_lvl'][:]) @@ -75,16 +66,31 @@ def read_nectdf(self, nc_filename): self.use_var_id = False nc_var = dataset.variables.get('obs_gc', None) else: - self.obs_var_table = self.get_string_array(dataset, 'obs_var') + self.obs_var_table = nc_tools.get_string_array(dataset, 'obs_var') if nc_var: self.obs_vid = np.array(nc_var[:]) - self.obs_qty_rwo = dataset['obs_qty'][:] + self.obs_qty_table = nc_tools.get_string_array(dataset, 'obs_qty_table') - self.obs_qty_table = self.get_string_array(dataset, 'obs_qty_table') - else: - print("==ERROR== netcdf file ({f}) does not exist".format(f=nc_filename)) - + +class nc_tools(): + + @staticmethod + def get_num_array(dataset, var_name): + nc_var = dataset.variables.get(var_name, None) + return nc_var[:].filled(nc_var._FillValue if '_FillValue' in nc_var.ncattrs() else -9999) if nc_var else [] + + @staticmethod + def get_ncbyte_array_to_str(nc_var): + nc_str_data = nc_var[:] + if nc_var.datatype.name == 'bytes8': + nc_str_data = [ str(s.compressed(),"utf-8") for s in nc_var[:]] + return nc_str_data + + @staticmethod + def get_string_array(dataset, var_name): + nc_var = dataset.variables.get(var_name, None) + return nc_tools.get_ncbyte_array_to_str(nc_var) if nc_var else [] perf_start_time = time.time() @@ -92,34 +98,17 @@ def read_nectdf(self, nc_filename): point_obs_data = None if len(sys.argv) == 1: - point_obs_data = get_sample_point_obs() + point_obs_data = sample_met_point_obs() + point_obs_data.read_data([]) else: netcdf_filename = sys.argv[1] - if os.path.exists(netcdf_filename): - point_obs_data = nc_point_obs() - point_obs_data.read_nectdf(netcdf_filename) - elif met_point_obs.is_python_script(netcdf_filename): - python_script = met_point_obs.get_python_script(netcdf_filename) - python_args = [] - if len(sys.argv) > 2: - if ARG_PRINT_DATA == sys.argv[-1]: - python_args = sys.argv[2:-1] - else: - python_args = sys.argv[2:] - python_args_string = " ".join(python_args) if 0 < len(python_args) else '' - os.environ[MET_PYTHON_OBS_ARGS] = python_args_string - #os.system(python_command) - exec(open(python_script).read()) - print('------') - - if point_obs_data is None: - print("PYTHON: Fail to get point_obs_data by calling {s} from {m}".format( - s=python_script, m=sys.argv[0])) + args = [ netcdf_filename ] + #args = { 'nc_name': netcdf_filename } + point_obs_data = nc_point_obs() + point_obs_data.read_data(args) + if ARG_PRINT_DATA == sys.argv[-1]: DO_PRINT_DATA = True - - point_obs_data = nc_point_obs() - point_obs_data.read_nectdf(netcdf_filename) if point_obs_data is not None: met_point_data = point_obs_data.get_point_data() From 29d479f02b9c9fa79807405a77f11666eaf388a6 Mon Sep 17 00:00:00 2001 From: Howard Soh Date: Fri, 21 Jan 2022 11:32:47 -0700 Subject: [PATCH 70/82] #1844 correctedb for loop end condition on processing obs bdata --- met/src/tools/core/ensemble_stat/ensemble_stat.cc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/met/src/tools/core/ensemble_stat/ensemble_stat.cc b/met/src/tools/core/ensemble_stat/ensemble_stat.cc index 14fd5c7b0e..29e62d0b72 100644 --- a/met/src/tools/core/ensemble_stat/ensemble_stat.cc +++ b/met/src/tools/core/ensemble_stat/ensemble_stat.cc @@ -1164,7 +1164,7 @@ void process_point_obs(int i_nc) { if (!status) exit(1); // Process each observation in the file - for(int i_offset=0; i_offset Date: Mon, 24 Jan 2022 16:03:17 -0700 Subject: [PATCH 71/82] Feature 1546 unit_tests (#2021) --- test/bin/unit_test.sh | 7 +- test/config/PlotModeFieldConfig | 2 +- test/xml/unit_aeronet.xml | 2 + test/xml/unit_airnow.xml | 2 + test/xml/unit_ascii2nc.xml | 10 ++- test/xml/unit_climatology.xml | 3 +- test/xml/unit_duplicate_flag.xml | 16 ++-- test/xml/unit_ensemble_stat.xml | 25 +++--- test/xml/unit_gaussian.xml | 3 +- test/xml/unit_gen_ens_prod.xml | 5 +- test/xml/unit_gen_vx_mask.xml | 2 + ...t_grib_tables.xml => unit_grib_tables.xml} | 18 +++-- test/xml/unit_grid_diag.xml | 2 + test/xml/unit_grid_stat.xml | 2 + test/xml/unit_grid_weight.xml | 2 + test/xml/unit_gsi_tools.xml | 52 ++++++------ test/xml/unit_hira.xml | 2 + test/xml/unit_interp_shape.xml | 2 + test/xml/unit_ioda2nc.xml | 41 +--------- test/xml/unit_lidar2nc.xml | 2 + test/xml/unit_madis2nc.xml | 2 + test/xml/unit_met_test_scripts.xml | 2 + test/xml/unit_mode.xml | 36 +-------- test/xml/unit_mode_analysis.xml | 4 +- test/xml/unit_mode_graphics.xml | 49 +----------- test/xml/unit_modis.xml | 2 + test/xml/unit_mtd.xml | 2 + test/xml/unit_netcdf.xml | 4 +- test/xml/unit_obs_summary.xml | 80 +++++++++---------- test/xml/unit_pb2nc.xml | 2 + test/xml/unit_pcp_combine.xml | 15 ++++ test/xml/unit_perc_thresh.xml | 2 + test/xml/unit_plot_data_plane.xml | 4 +- test/xml/unit_plot_point_obs.xml | 3 +- test/xml/unit_plot_tc.xml | 4 +- test/xml/unit_point2grid.xml | 6 +- test/xml/unit_point_stat.xml | 2 + test/xml/unit_python.xml | 4 +- test/xml/unit_quality_filter.xml | 29 +++---- test/xml/unit_ref_config.xml | 2 + test/xml/unit_regrid.xml | 2 + test/xml/unit_rmw_analysis.xml | 2 + test/xml/unit_series_analysis.xml | 2 + test/xml/unit_shift_data_plane.xml | 2 + test/xml/unit_stat_analysis.xml | 6 +- test/xml/unit_tc_dland.xml | 2 + test/xml/unit_tc_gen.xml | 2 + test/xml/unit_tc_pairs.xml | 2 + test/xml/unit_tc_rmw.xml | 2 + test/xml/unit_tc_stat.xml | 2 + test/xml/unit_trmm2nc.xml | 2 + test/xml/unit_wavelet_stat.xml | 2 + test/xml/unit_wwmca_plot.xml | 2 + test/xml/unit_wwmca_regrid.xml | 2 + 54 files changed, 234 insertions(+), 252 deletions(-) rename test/xml/{unit_test_grib_tables.xml => unit_grib_tables.xml} (77%) diff --git a/test/bin/unit_test.sh b/test/bin/unit_test.sh index 5e48e3bb44..fd1c7149cb 100755 --- a/test/bin/unit_test.sh +++ b/test/bin/unit_test.sh @@ -19,6 +19,11 @@ else echo "export MET_TEST_OUTPUT=${MET_TEST_OUTPUT}" fi +# if MET_TEST_MET_PYTHON_EXE is not set, use default value +if [[ -z "${MET_TEST_MET_PYTHON_EXE}" ]] ; then + export MET_TEST_MET_PYTHON_EXE=/usr/local/python3/bin/python3 +fi + PERL_UNIT_OPTS="" for arg in $@; do [ $arg == "-memchk" -o $arg == "memchk" ] && PERL_UNIT_OPTS="$PERL_UNIT_OPTS -memchk" @@ -67,7 +72,7 @@ UNIT_XML="unit_ascii2nc.xml \ unit_shift_data_plane.xml \ unit_mtd.xml \ unit_climatology.xml \ - unit_test_grib_tables.xml \ + unit_grib_tables.xml \ unit_grid_weight.xml \ unit_netcdf.xml \ unit_hira.xml \ diff --git a/test/config/PlotModeFieldConfig b/test/config/PlotModeFieldConfig index f6915f4c4a..816535ff25 100644 --- a/test/config/PlotModeFieldConfig +++ b/test/config/PlotModeFieldConfig @@ -9,7 +9,7 @@ plot_info = { - output_directory = "${MET_TEST_OUTPUT}/plot_mode_field"; + output_directory = "${MET_TEST_OUTPUT}/mode_graphics"; size = 4; diff --git a/test/xml/unit_aeronet.xml b/test/xml/unit_aeronet.xml index 4d1fd80ec5..0892c623ad 100644 --- a/test/xml/unit_aeronet.xml +++ b/test/xml/unit_aeronet.xml @@ -10,6 +10,8 @@ ]> + + &TEST_DIR; diff --git a/test/xml/unit_airnow.xml b/test/xml/unit_airnow.xml index 9c616e5872..9afa68d292 100644 --- a/test/xml/unit_airnow.xml +++ b/test/xml/unit_airnow.xml @@ -10,6 +10,8 @@ ]> + + &TEST_DIR; diff --git a/test/xml/unit_ascii2nc.xml b/test/xml/unit_ascii2nc.xml index 2edb2bf9b7..11c46a3ea5 100644 --- a/test/xml/unit_ascii2nc.xml +++ b/test/xml/unit_ascii2nc.xml @@ -11,6 +11,8 @@ ]> + + &TEST_DIR; @@ -68,11 +70,11 @@ &MET_BIN;/ascii2nc \ &DATA_DIR_OBS;/ascii/qc_out_2012-09-07_00:00:00.GRM_P+FCST \ - &OUTPUT_DIR;/ascii2nc/qc_out_2012-09-07_00:00:00.GRM_P+FCST.nc \ + &OUTPUT_DIR;/ascii2nc/qc_out_2012-09-07_00:00:00.GRM_P_FCST.nc \ -v 1 - &OUTPUT_DIR;/ascii2nc/qc_out_2012-09-07_00:00:00.GRM_P+FCST.nc + &OUTPUT_DIR;/ascii2nc/qc_out_2012-09-07_00:00:00.GRM_P_FCST.nc @@ -80,11 +82,11 @@ &MET_BIN;/ascii2nc \ &DATA_DIR_OBS;/ascii/OBS:2015080700_bad_record \ - &OUTPUT_DIR;/ascii2nc/OBS:2015080700_bad_record.nc \ + &OUTPUT_DIR;/ascii2nc/OBS_2015080700_bad_record.nc \ -v 1 - &OUTPUT_DIR;/ascii2nc/OBS:2015080700_bad_record.nc + &OUTPUT_DIR;/ascii2nc/OBS_2015080700_bad_record.nc diff --git a/test/xml/unit_climatology.xml b/test/xml/unit_climatology.xml index c3a946072e..d50dc86ccf 100644 --- a/test/xml/unit_climatology.xml +++ b/test/xml/unit_climatology.xml @@ -11,12 +11,13 @@ ]> + + &TEST_DIR; true - &MET_BIN;/point_stat diff --git a/test/xml/unit_duplicate_flag.xml b/test/xml/unit_duplicate_flag.xml index 51de6cecc0..dcd287f59b 100644 --- a/test/xml/unit_duplicate_flag.xml +++ b/test/xml/unit_duplicate_flag.xml @@ -10,9 +10,9 @@ ]> - + - + &TEST_DIR; false @@ -27,11 +27,11 @@ &DATA_DIR_MODEL;/grib2/nam/nam_2012040900_F012.grib2 \ &OUTPUT_DIR;/ascii2nc/dup_test.nc \ &CONFIG_DIR;/PointStatConfig_dup \ - -outdir &OUTPUT_DIR;/point_stat -v 3 + -outdir &OUTPUT_DIR;/duplicate_flag -v 3 - &OUTPUT_DIR;/point_stat/point_stat_DUP_NONE_120000L_20120409_120000V.stat - &OUTPUT_DIR;/point_stat/point_stat_DUP_NONE_120000L_20120409_120000V_mpr.txt + &OUTPUT_DIR;/duplicate_flag/point_stat_DUP_NONE_120000L_20120409_120000V.stat + &OUTPUT_DIR;/duplicate_flag/point_stat_DUP_NONE_120000L_20120409_120000V_mpr.txt @@ -45,11 +45,11 @@ &DATA_DIR_MODEL;/grib2/nam/nam_2012040900_F012.grib2 \ &OUTPUT_DIR;/ascii2nc/dup_test.nc \ &CONFIG_DIR;/PointStatConfig_dup \ - -outdir &OUTPUT_DIR;/point_stat -v 3 + -outdir &OUTPUT_DIR;/duplicate_flag -v 3 - &OUTPUT_DIR;/point_stat/point_stat_DUP_UNIQUE_120000L_20120409_120000V.stat - &OUTPUT_DIR;/point_stat/point_stat_DUP_UNIQUE_120000L_20120409_120000V_mpr.txt + &OUTPUT_DIR;/duplicate_flag/point_stat_DUP_UNIQUE_120000L_20120409_120000V.stat + &OUTPUT_DIR;/duplicate_flag/point_stat_DUP_UNIQUE_120000L_20120409_120000V_mpr.txt diff --git a/test/xml/unit_ensemble_stat.xml b/test/xml/unit_ensemble_stat.xml index 93ec40b228..98a97a4779 100644 --- a/test/xml/unit_ensemble_stat.xml +++ b/test/xml/unit_ensemble_stat.xml @@ -10,6 +10,8 @@ ]> + + &TEST_DIR; @@ -18,7 +20,6 @@ - @@ -216,7 +217,7 @@ - + echo "&DATA_DIR_MODEL;/grib1/arw-fer-gep1/arw-fer-gep1_2012040912_F024.grib \ &DATA_DIR_MODEL;/grib1/arw-fer-gep5/arw-fer-gep5_2012040912_F024.grib \ &DATA_DIR_MODEL;/grib1/arw-sch-gep2/arw-sch-gep2_2012040912_F024.grib \ @@ -226,10 +227,10 @@ > &OUTPUT_DIR;/ensemble_stat/input_file_list; \ &MET_BIN;/ensemble_stat - DESC OBS_ERROR + DESC OBSERR OBS_ERROR_FLAG TRUE SKIP_CONST TRUE - OUTPUT_PREFIX OBS_ERROR + OUTPUT_PREFIX OBSERR \ &OUTPUT_DIR;/ensemble_stat/input_file_list \ @@ -239,14 +240,14 @@ -outdir &OUTPUT_DIR;/ensemble_stat -v 1 - &OUTPUT_DIR;/ensemble_stat/ensemble_stat_OBS_ERROR_20120410_120000V.stat - &OUTPUT_DIR;/ensemble_stat/ensemble_stat_OBS_ERROR_20120410_120000V_ecnt.txt - &OUTPUT_DIR;/ensemble_stat/ensemble_stat_OBS_ERROR_20120410_120000V_rhist.txt - &OUTPUT_DIR;/ensemble_stat/ensemble_stat_OBS_ERROR_20120410_120000V_phist.txt - &OUTPUT_DIR;/ensemble_stat/ensemble_stat_OBS_ERROR_20120410_120000V_orank.txt - &OUTPUT_DIR;/ensemble_stat/ensemble_stat_OBS_ERROR_20120410_120000V_ssvar.txt - &OUTPUT_DIR;/ensemble_stat/ensemble_stat_OBS_ERROR_20120410_120000V_ens.nc - &OUTPUT_DIR;/ensemble_stat/ensemble_stat_OBS_ERROR_20120410_120000V_orank.nc + &OUTPUT_DIR;/ensemble_stat/ensemble_stat_OBSERR_20120410_120000V.stat + &OUTPUT_DIR;/ensemble_stat/ensemble_stat_OBSERR_20120410_120000V_ecnt.txt + &OUTPUT_DIR;/ensemble_stat/ensemble_stat_OBSERR_20120410_120000V_rhist.txt + &OUTPUT_DIR;/ensemble_stat/ensemble_stat_OBSERR_20120410_120000V_phist.txt + &OUTPUT_DIR;/ensemble_stat/ensemble_stat_OBSERR_20120410_120000V_orank.txt + &OUTPUT_DIR;/ensemble_stat/ensemble_stat_OBSERR_20120410_120000V_ssvar.txt + &OUTPUT_DIR;/ensemble_stat/ensemble_stat_OBSERR_20120410_120000V_ens.nc + &OUTPUT_DIR;/ensemble_stat/ensemble_stat_OBSERR_20120410_120000V_orank.nc diff --git a/test/xml/unit_gaussian.xml b/test/xml/unit_gaussian.xml index ef6f74e49d..0c024e76d2 100644 --- a/test/xml/unit_gaussian.xml +++ b/test/xml/unit_gaussian.xml @@ -8,6 +8,8 @@ ]> + + &TEST_DIR; @@ -25,5 +27,4 @@ - diff --git a/test/xml/unit_gen_ens_prod.xml b/test/xml/unit_gen_ens_prod.xml index 91f2321431..3c014a08f2 100644 --- a/test/xml/unit_gen_ens_prod.xml +++ b/test/xml/unit_gen_ens_prod.xml @@ -10,6 +10,8 @@ ]> + + &TEST_DIR; @@ -17,7 +19,6 @@ - @@ -120,7 +121,7 @@ \ -ens &DATA_DIR_MODEL;/grib2/gefs/enspost_grb2.t00z.prmsl \ -config &CONFIG_DIR;/GenEnsProdConfig_single_file_grib \ - -ctrl &DATA_DIR_MODEL;/grib2/gefs/enspost_grb2.t00z.prmsl \ + -ctrl &DATA_DIR_MODEL;/grib2/gefs/enspost_grb2.t00z.prmsl \ -out &OUTPUT_DIR;/gen_ens_prod/gen_ens_prod_SINGLE_FILE_GRIB_WITH_CTRL.nc \ -v 2 diff --git a/test/xml/unit_gen_vx_mask.xml b/test/xml/unit_gen_vx_mask.xml index 6416d0a048..e9664fe0f9 100644 --- a/test/xml/unit_gen_vx_mask.xml +++ b/test/xml/unit_gen_vx_mask.xml @@ -12,6 +12,8 @@ ]> + + diff --git a/test/xml/unit_test_grib_tables.xml b/test/xml/unit_grib_tables.xml similarity index 77% rename from test/xml/unit_test_grib_tables.xml rename to test/xml/unit_grib_tables.xml index 41d3886867..88e8370e23 100644 --- a/test/xml/unit_test_grib_tables.xml +++ b/test/xml/unit_grib_tables.xml @@ -1,14 +1,16 @@ - - - + + + + - - - - ]> + + + +]> + + diff --git a/test/xml/unit_grid_diag.xml b/test/xml/unit_grid_diag.xml index 0da3a2bd95..10049144ac 100644 --- a/test/xml/unit_grid_diag.xml +++ b/test/xml/unit_grid_diag.xml @@ -11,6 +11,8 @@ ]> + + &TEST_DIR; diff --git a/test/xml/unit_grid_stat.xml b/test/xml/unit_grid_stat.xml index c31455c0ce..9d154a91a6 100644 --- a/test/xml/unit_grid_stat.xml +++ b/test/xml/unit_grid_stat.xml @@ -11,6 +11,8 @@ ]> + + &TEST_DIR; diff --git a/test/xml/unit_grid_weight.xml b/test/xml/unit_grid_weight.xml index 2c2949b920..f355401441 100644 --- a/test/xml/unit_grid_weight.xml +++ b/test/xml/unit_grid_weight.xml @@ -11,6 +11,8 @@ ]> + + &TEST_DIR; diff --git a/test/xml/unit_gsi_tools.xml b/test/xml/unit_gsi_tools.xml index e80505d2fd..78ec34ce56 100644 --- a/test/xml/unit_gsi_tools.xml +++ b/test/xml/unit_gsi_tools.xml @@ -10,24 +10,24 @@ ]> + + &TEST_DIR; true - - &MET_BIN;/gsid2mpr \ &INPUT_DIR;/gsi_data/GSIdiags4EnKF/diag_conv_ges.mem001 \ -set_hdr MODEL GSI_MEM001 \ - -outdir &OUTPUT_DIR;/gsid2mpr \ + -outdir &OUTPUT_DIR;/gsi_tools/gsid2mpr \ -swap -v 1 - &OUTPUT_DIR;/gsid2mpr/diag_conv_ges.mem001.stat + &OUTPUT_DIR;/gsi_tools/gsid2mpr/diag_conv_ges.mem001.stat @@ -37,11 +37,11 @@ \ &INPUT_DIR;/gsi_data/GSIdiags4EnKF/diag_conv_ges.mem001 \ -set_hdr MODEL GSI_MEM001 \ - -outdir &OUTPUT_DIR;/gsid2mpr \ + -outdir &OUTPUT_DIR;/gsi_tools/gsid2mpr \ -swap -no_check_dup -suffix _DUP_mpr.txt -v 1 - &OUTPUT_DIR;/gsid2mpr/diag_conv_ges.mem001_DUP_mpr.txt + &OUTPUT_DIR;/gsi_tools/gsid2mpr/diag_conv_ges.mem001_DUP_mpr.txt @@ -51,13 +51,13 @@ \ &INPUT_DIR;/gsi_data/GSIdiags4EnKF/diag_amsua_n18_ges.mem001 \ &INPUT_DIR;/gsi_data/GSIdiags4EnKF/diag_amsua_n18_ges.mem002 \ - -outdir &OUTPUT_DIR;/gsid2mpr \ + -outdir &OUTPUT_DIR;/gsi_tools/gsid2mpr \ -suffix _mpr.txt \ -swap -v 1 - &OUTPUT_DIR;/gsid2mpr/diag_amsua_n18_ges.mem001_mpr.txt - &OUTPUT_DIR;/gsid2mpr/diag_amsua_n18_ges.mem002_mpr.txt + &OUTPUT_DIR;/gsi_tools/gsid2mpr/diag_amsua_n18_ges.mem001_mpr.txt + &OUTPUT_DIR;/gsi_tools/gsid2mpr/diag_amsua_n18_ges.mem002_mpr.txt @@ -66,11 +66,11 @@ &MET_BIN;/gsidens2orank \ &INPUT_DIR;/gsi_data/GSIdiags4EnKF/diag_conv_ges.mem* \ - -out &OUTPUT_DIR;/gsidens2orank/diag_conv_ges_no_mean_orank.txt \ + -out &OUTPUT_DIR;/gsi_tools/gsidens2orank/diag_conv_ges_no_mean_orank.txt \ -swap -v 1 - &OUTPUT_DIR;/gsidens2orank/diag_conv_ges_no_mean_orank.txt + &OUTPUT_DIR;/gsi_tools/gsidens2orank/diag_conv_ges_no_mean_orank.txt @@ -80,11 +80,11 @@ \ &INPUT_DIR;/gsi_data/GSIdiags4EnKF/diag_conv_ges.mem* \ -ens_mean &INPUT_DIR;/gsi_data/GSIdiags4EnKF/diag_conv_ges.ensmean \ - -out &OUTPUT_DIR;/gsidens2orank/diag_conv_ges_ens_mean_orank.txt \ + -out &OUTPUT_DIR;/gsi_tools/gsidens2orank/diag_conv_ges_ens_mean_orank.txt \ -swap -v 1 - &OUTPUT_DIR;/gsidens2orank/diag_conv_ges_ens_mean_orank.txt + &OUTPUT_DIR;/gsi_tools/gsidens2orank/diag_conv_ges_ens_mean_orank.txt @@ -93,11 +93,11 @@ &MET_BIN;/gsidens2orank \ &INPUT_DIR;/gsi_data/GSIdiags4EnKF/diag_amsua_n18_ges.mem* \ - -out &OUTPUT_DIR;/gsidens2orank/diag_amsua_n18_ges_all_channels_orank.txt \ + -out &OUTPUT_DIR;/gsi_tools/gsidens2orank/diag_amsua_n18_ges_all_channels_orank.txt \ -swap -v 1 - &OUTPUT_DIR;/gsidens2orank/diag_amsua_n18_ges_all_channels_orank.txt + &OUTPUT_DIR;/gsi_tools/gsidens2orank/diag_amsua_n18_ges_all_channels_orank.txt @@ -108,11 +108,11 @@ &INPUT_DIR;/gsi_data/GSIdiags4EnKF/diag_amsua_n18_ges.mem* \ -ens_mean &INPUT_DIR;/gsi_data/GSIdiags4EnKF/diag_amsua_n18_ges.ensmean \ -channel 2,7,10 \ - -out &OUTPUT_DIR;/gsidens2orank/diag_amsua_n18_ges_some_channels_orank.txt \ + -out &OUTPUT_DIR;/gsi_tools/gsidens2orank/diag_amsua_n18_ges_some_channels_orank.txt \ -swap -v 1 - &OUTPUT_DIR;/gsidens2orank/diag_amsua_n18_ges_some_channels_orank.txt + &OUTPUT_DIR;/gsi_tools/gsidens2orank/diag_amsua_n18_ges_some_channels_orank.txt @@ -120,14 +120,14 @@ &MET_BIN;/stat_analysis \ - -lookin &OUTPUT_DIR;/gsid2mpr/diag_conv_ges.mem001.stat \ + -lookin &OUTPUT_DIR;/gsi_tools/gsid2mpr/diag_conv_ges.mem001.stat \ -job aggregate_stat -line_type MPR -out_line_type CNT -by fcst_var \ -column_thresh ANLY_USE ==1 \ - -out_stat &OUTPUT_DIR;/gsid2mpr/diag_conv_ges.mem001_cnt.txt \ + -out_stat &OUTPUT_DIR;/gsi_tools/gsid2mpr/diag_conv_ges.mem001_cnt.txt \ -v 1 - &OUTPUT_DIR;/gsid2mpr/diag_conv_ges.mem001_cnt.txt + &OUTPUT_DIR;/gsi_tools/gsid2mpr/diag_conv_ges.mem001_cnt.txt @@ -135,14 +135,14 @@ &MET_BIN;/stat_analysis \ - -lookin &OUTPUT_DIR;/gsidens2orank/diag_conv_ges_ens_mean_orank.txt \ + -lookin &OUTPUT_DIR;/gsi_tools/gsidens2orank/diag_conv_ges_ens_mean_orank.txt \ -job aggregate_stat -line_type ORANK -out_line_type RHIST -by fcst_var \ -column_thresh N_USE ==20 \ - -out_stat &OUTPUT_DIR;/gsidens2orank/diag_conv_ges_ens_mean_rhist.txt \ + -out_stat &OUTPUT_DIR;/gsi_tools/gsidens2orank/diag_conv_ges_ens_mean_rhist.txt \ -v 1 - &OUTPUT_DIR;/gsidens2orank/diag_conv_ges_ens_mean_rhist.txt + &OUTPUT_DIR;/gsi_tools/gsidens2orank/diag_conv_ges_ens_mean_rhist.txt @@ -150,14 +150,14 @@ &MET_BIN;/stat_analysis \ - -lookin &OUTPUT_DIR;/gsidens2orank/diag_conv_ges_ens_mean_orank.txt \ + -lookin &OUTPUT_DIR;/gsi_tools/gsidens2orank/diag_conv_ges_ens_mean_orank.txt \ -job aggregate_stat -line_type ORANK -out_line_type SSVAR -by fcst_var \ -column_thresh N_USE ==20 -out_bin_size 0.10 \ - -out_stat &OUTPUT_DIR;/gsidens2orank/diag_conv_ges_ens_mean_ssvar.txt \ + -out_stat &OUTPUT_DIR;/gsi_tools/gsidens2orank/diag_conv_ges_ens_mean_ssvar.txt \ -v 1 - &OUTPUT_DIR;/gsidens2orank/diag_conv_ges_ens_mean_ssvar.txt + &OUTPUT_DIR;/gsi_tools/gsidens2orank/diag_conv_ges_ens_mean_ssvar.txt diff --git a/test/xml/unit_hira.xml b/test/xml/unit_hira.xml index 696199ce98..13d9888216 100644 --- a/test/xml/unit_hira.xml +++ b/test/xml/unit_hira.xml @@ -10,6 +10,8 @@ ]> + + diff --git a/test/xml/unit_interp_shape.xml b/test/xml/unit_interp_shape.xml index 962adf2a6e..be8aba9bd3 100644 --- a/test/xml/unit_interp_shape.xml +++ b/test/xml/unit_interp_shape.xml @@ -10,6 +10,8 @@ ]> + + &TEST_DIR; diff --git a/test/xml/unit_ioda2nc.xml b/test/xml/unit_ioda2nc.xml index b519a08d4d..91f2a7cbb4 100644 --- a/test/xml/unit_ioda2nc.xml +++ b/test/xml/unit_ioda2nc.xml @@ -12,49 +12,12 @@ ]> + + &TEST_DIR; true - &MET_BIN;/ioda2nc diff --git a/test/xml/unit_lidar2nc.xml b/test/xml/unit_lidar2nc.xml index dc46dab4bc..4fedf5c895 100644 --- a/test/xml/unit_lidar2nc.xml +++ b/test/xml/unit_lidar2nc.xml @@ -10,6 +10,8 @@ ]> + + &TEST_DIR; diff --git a/test/xml/unit_madis2nc.xml b/test/xml/unit_madis2nc.xml index 802eebc3c7..550b793ff0 100644 --- a/test/xml/unit_madis2nc.xml +++ b/test/xml/unit_madis2nc.xml @@ -11,6 +11,8 @@ ]> + + + diff --git a/test/xml/unit_mode.xml b/test/xml/unit_mode.xml index 113eac7934..c13e74676f 100644 --- a/test/xml/unit_mode.xml +++ b/test/xml/unit_mode.xml @@ -11,24 +11,13 @@ ]> + + &TEST_DIR; true - - &MET_BIN;/pcp_combine - \ - -subtract \ - &DATA_DIR_MODEL;/grib1/arw-tom-gep0/arw-tom-gep0_2012040912_F030.grib 30 \ - &DATA_DIR_MODEL;/grib1/arw-tom-gep0/arw-tom-gep0_2012040912_F024.grib 24 \ - &OUTPUT_DIR;/pcp_combine/arw-tom-gep0_2012040912_F030_APCP06.nc - - - &OUTPUT_DIR;/pcp_combine/arw-tom-gep0_2012040912_F030_APCP06.nc - - - &MET_BIN;/mode @@ -125,27 +114,6 @@ - - &MET_BIN;/mode diff --git a/test/xml/unit_mode_analysis.xml b/test/xml/unit_mode_analysis.xml index 12bb7b871c..f0815df484 100644 --- a/test/xml/unit_mode_analysis.xml +++ b/test/xml/unit_mode_analysis.xml @@ -10,13 +10,13 @@ ]> + + &TEST_DIR; true - - &MET_BIN;/mode_analysis \ diff --git a/test/xml/unit_mode_graphics.xml b/test/xml/unit_mode_graphics.xml index 824ba86907..485d4fe22e 100644 --- a/test/xml/unit_mode_graphics.xml +++ b/test/xml/unit_mode_graphics.xml @@ -10,54 +10,13 @@ ]> - + &TEST_DIR; true - - &MET_BIN;/plot_mode_field \ @@ -67,9 +26,9 @@ &OUTPUT_DIR;/met_test_scripts/mode/mode_*_obj.nc - &OUTPUT_DIR;/plot_mode_field/mode_120000L_20050807_120000V_000000A_obj_obs_simple.png - &OUTPUT_DIR;/plot_mode_field/mode_120000L_20050807_120000V_120000A_obj_obs_simple.png - &OUTPUT_DIR;/plot_mode_field/mode_240000L_20050808_000000V_240000A_obj_obs_simple.png + &OUTPUT_DIR;/mode_graphics/mode_120000L_20050807_120000V_000000A_obj_obs_simple.png + &OUTPUT_DIR;/mode_graphics/mode_120000L_20050807_120000V_120000A_obj_obs_simple.png + &OUTPUT_DIR;/mode_graphics/mode_240000L_20050808_000000V_240000A_obj_obs_simple.png diff --git a/test/xml/unit_modis.xml b/test/xml/unit_modis.xml index d5e1faf2a7..3f28d132e5 100644 --- a/test/xml/unit_modis.xml +++ b/test/xml/unit_modis.xml @@ -15,6 +15,8 @@ ]> + + &TEST_DIR; diff --git a/test/xml/unit_mtd.xml b/test/xml/unit_mtd.xml index 0190dd2080..367bc6991a 100644 --- a/test/xml/unit_mtd.xml +++ b/test/xml/unit_mtd.xml @@ -9,6 +9,8 @@ ]> + + &TEST_DIR; diff --git a/test/xml/unit_netcdf.xml b/test/xml/unit_netcdf.xml index b0e6e2fe48..5c0b255240 100644 --- a/test/xml/unit_netcdf.xml +++ b/test/xml/unit_netcdf.xml @@ -10,10 +10,12 @@ ]> + + diff --git a/test/xml/unit_obs_summary.xml b/test/xml/unit_obs_summary.xml index eb150fdaf1..eb98b3099b 100644 --- a/test/xml/unit_obs_summary.xml +++ b/test/xml/unit_obs_summary.xml @@ -10,9 +10,9 @@ ]> - + - + &TEST_DIR; false @@ -21,11 +21,11 @@ &MET_BIN;/ascii2nc \ &DATA_DIR_OBS;/ascii/obs_sum_test_qty.txt \ - &OUTPUT_DIR;/ascii2nc/obs_sum_test.nc \ + &OUTPUT_DIR;/obs_summary/obs_sum_test.nc \ -v 1 - &OUTPUT_DIR;/ascii2nc/obs_sum_test.nc + &OUTPUT_DIR;/obs_summary/obs_sum_test.nc @@ -38,13 +38,13 @@ \ &DATA_DIR_MODEL;/grib2/nam/nam_2012040900_F012.grib2 \ - &OUTPUT_DIR;/ascii2nc/obs_sum_test.nc \ + &OUTPUT_DIR;/obs_summary/obs_sum_test.nc \ &CONFIG_DIR;/PointStatConfig_obs_summary \ - -outdir &OUTPUT_DIR;/point_stat -v 3 + -outdir &OUTPUT_DIR;/obs_summary -v 3 - &OUTPUT_DIR;/point_stat/point_stat_OS_NONE_120000L_20120409_120000V.stat - &OUTPUT_DIR;/point_stat/point_stat_OS_NONE_120000L_20120409_120000V_mpr.txt + &OUTPUT_DIR;/obs_summary/point_stat_OS_NONE_120000L_20120409_120000V.stat + &OUTPUT_DIR;/obs_summary/point_stat_OS_NONE_120000L_20120409_120000V_mpr.txt @@ -57,13 +57,13 @@ \ &DATA_DIR_MODEL;/grib2/nam/nam_2012040900_F012.grib2 \ - &OUTPUT_DIR;/ascii2nc/obs_sum_test.nc \ + &OUTPUT_DIR;/obs_summary/obs_sum_test.nc \ &CONFIG_DIR;/PointStatConfig_obs_summary \ - -outdir &OUTPUT_DIR;/point_stat -v 3 + -outdir &OUTPUT_DIR;/obs_summary -v 3 - &OUTPUT_DIR;/point_stat/point_stat_OS_NEAREST_120000L_20120409_120000V.stat - &OUTPUT_DIR;/point_stat/point_stat_OS_NEAREST_120000L_20120409_120000V_mpr.txt + &OUTPUT_DIR;/obs_summary/point_stat_OS_NEAREST_120000L_20120409_120000V.stat + &OUTPUT_DIR;/obs_summary/point_stat_OS_NEAREST_120000L_20120409_120000V_mpr.txt @@ -76,13 +76,13 @@ \ &DATA_DIR_MODEL;/grib2/nam/nam_2012040900_F012.grib2 \ - &OUTPUT_DIR;/ascii2nc/obs_sum_test.nc \ + &OUTPUT_DIR;/obs_summary/obs_sum_test.nc \ &CONFIG_DIR;/PointStatConfig_obs_summary \ - -outdir &OUTPUT_DIR;/point_stat -v 3 + -outdir &OUTPUT_DIR;/obs_summary -v 3 - &OUTPUT_DIR;/point_stat/point_stat_OS_MIN_120000L_20120409_120000V.stat - &OUTPUT_DIR;/point_stat/point_stat_OS_MIN_120000L_20120409_120000V_mpr.txt + &OUTPUT_DIR;/obs_summary/point_stat_OS_MIN_120000L_20120409_120000V.stat + &OUTPUT_DIR;/obs_summary/point_stat_OS_MIN_120000L_20120409_120000V_mpr.txt @@ -95,13 +95,13 @@ \ &DATA_DIR_MODEL;/grib2/nam/nam_2012040900_F012.grib2 \ - &OUTPUT_DIR;/ascii2nc/obs_sum_test.nc \ + &OUTPUT_DIR;/obs_summary/obs_sum_test.nc \ &CONFIG_DIR;/PointStatConfig_obs_summary \ - -outdir &OUTPUT_DIR;/point_stat -v 3 + -outdir &OUTPUT_DIR;/obs_summary -v 3 - &OUTPUT_DIR;/point_stat/point_stat_OS_MAX_120000L_20120409_120000V.stat - &OUTPUT_DIR;/point_stat/point_stat_OS_MAX_120000L_20120409_120000V_mpr.txt + &OUTPUT_DIR;/obs_summary/point_stat_OS_MAX_120000L_20120409_120000V.stat + &OUTPUT_DIR;/obs_summary/point_stat_OS_MAX_120000L_20120409_120000V_mpr.txt @@ -114,13 +114,13 @@ \ &DATA_DIR_MODEL;/grib2/nam/nam_2012040900_F012.grib2 \ - &OUTPUT_DIR;/ascii2nc/obs_sum_test.nc \ + &OUTPUT_DIR;/obs_summary/obs_sum_test.nc \ &CONFIG_DIR;/PointStatConfig_obs_summary \ - -outdir &OUTPUT_DIR;/point_stat -v 3 + -outdir &OUTPUT_DIR;/obs_summary -v 3 - &OUTPUT_DIR;/point_stat/point_stat_OS_UW_MEAN_120000L_20120409_120000V.stat - &OUTPUT_DIR;/point_stat/point_stat_OS_UW_MEAN_120000L_20120409_120000V_mpr.txt + &OUTPUT_DIR;/obs_summary/point_stat_OS_UW_MEAN_120000L_20120409_120000V.stat + &OUTPUT_DIR;/obs_summary/point_stat_OS_UW_MEAN_120000L_20120409_120000V_mpr.txt @@ -133,13 +133,13 @@ \ &DATA_DIR_MODEL;/grib2/nam/nam_2012040900_F012.grib2 \ - &OUTPUT_DIR;/ascii2nc/obs_sum_test.nc \ + &OUTPUT_DIR;/obs_summary/obs_sum_test.nc \ &CONFIG_DIR;/PointStatConfig_obs_summary \ - -outdir &OUTPUT_DIR;/point_stat -v 3 + -outdir &OUTPUT_DIR;/obs_summary -v 3 - &OUTPUT_DIR;/point_stat/point_stat_OS_DW_MEAN_120000L_20120409_120000V.stat - &OUTPUT_DIR;/point_stat/point_stat_OS_DW_MEAN_120000L_20120409_120000V_mpr.txt + &OUTPUT_DIR;/obs_summary/point_stat_OS_DW_MEAN_120000L_20120409_120000V.stat + &OUTPUT_DIR;/obs_summary/point_stat_OS_DW_MEAN_120000L_20120409_120000V_mpr.txt @@ -152,13 +152,13 @@ \ &DATA_DIR_MODEL;/grib2/nam/nam_2012040900_F012.grib2 \ - &OUTPUT_DIR;/ascii2nc/obs_sum_test.nc \ + &OUTPUT_DIR;/obs_summary/obs_sum_test.nc \ &CONFIG_DIR;/PointStatConfig_obs_summary \ - -outdir &OUTPUT_DIR;/point_stat -v 3 + -outdir &OUTPUT_DIR;/obs_summary -v 3 - &OUTPUT_DIR;/point_stat/point_stat_OS_MEDIAN_120000L_20120409_120000V.stat - &OUTPUT_DIR;/point_stat/point_stat_OS_MEDIAN_120000L_20120409_120000V_mpr.txt + &OUTPUT_DIR;/obs_summary/point_stat_OS_MEDIAN_120000L_20120409_120000V.stat + &OUTPUT_DIR;/obs_summary/point_stat_OS_MEDIAN_120000L_20120409_120000V_mpr.txt @@ -171,13 +171,13 @@ \ &DATA_DIR_MODEL;/grib2/nam/nam_2012040900_F012.grib2 \ - &OUTPUT_DIR;/ascii2nc/obs_sum_test.nc \ + &OUTPUT_DIR;/obs_summary/obs_sum_test.nc \ &CONFIG_DIR;/PointStatConfig_obs_summary \ - -outdir &OUTPUT_DIR;/point_stat -v 3 + -outdir &OUTPUT_DIR;/obs_summary -v 3 - &OUTPUT_DIR;/point_stat/point_stat_OS_PERC_120000L_20120409_120000V.stat - &OUTPUT_DIR;/point_stat/point_stat_OS_PERC_120000L_20120409_120000V_mpr.txt + &OUTPUT_DIR;/obs_summary/point_stat_OS_PERC_120000L_20120409_120000V.stat + &OUTPUT_DIR;/obs_summary/point_stat_OS_PERC_120000L_20120409_120000V_mpr.txt @@ -188,13 +188,13 @@ \ &DATA_DIR_MODEL;/grib2/nam/nam_2012040900_F012.grib2 \ - &OUTPUT_DIR;/ascii2nc/obs_sum_test.nc \ + &OUTPUT_DIR;/obs_summary/obs_sum_test.nc \ &CONFIG_DIR;/PointStatConfig_obs_summary_all \ - -outdir &OUTPUT_DIR;/point_stat -v 3 + -outdir &OUTPUT_DIR;/obs_summary -v 3 - &OUTPUT_DIR;/point_stat/point_stat_OS_UNIQUE_ALL_120000L_20120409_120000V.stat - &OUTPUT_DIR;/point_stat/point_stat_OS_UNIQUE_ALL_120000L_20120409_120000V_mpr.txt + &OUTPUT_DIR;/obs_summary/point_stat_OS_UNIQUE_ALL_120000L_20120409_120000V.stat + &OUTPUT_DIR;/obs_summary/point_stat_OS_UNIQUE_ALL_120000L_20120409_120000V_mpr.txt diff --git a/test/xml/unit_pb2nc.xml b/test/xml/unit_pb2nc.xml index 3c7c6175fb..d6d79def22 100644 --- a/test/xml/unit_pb2nc.xml +++ b/test/xml/unit_pb2nc.xml @@ -12,6 +12,8 @@ ]> + + &TEST_DIR; diff --git a/test/xml/unit_pcp_combine.xml b/test/xml/unit_pcp_combine.xml index 6d44e95b9b..4a1c39b76b 100644 --- a/test/xml/unit_pcp_combine.xml +++ b/test/xml/unit_pcp_combine.xml @@ -10,6 +10,8 @@ ]> + + &TEST_DIR; @@ -140,6 +142,19 @@ + + &MET_BIN;/pcp_combine + \ + -subtract \ + &DATA_DIR_MODEL;/grib1/arw-tom-gep0/arw-tom-gep0_2012040912_F030.grib 30 \ + &DATA_DIR_MODEL;/grib1/arw-tom-gep0/arw-tom-gep0_2012040912_F024.grib 24 \ + &OUTPUT_DIR;/pcp_combine/arw-tom-gep0_2012040912_F030_APCP06.nc + + + &OUTPUT_DIR;/pcp_combine/arw-tom-gep0_2012040912_F030_APCP06.nc + + + &MET_BIN;/pcp_combine \ diff --git a/test/xml/unit_perc_thresh.xml b/test/xml/unit_perc_thresh.xml index 893dedf3cf..366b32e3d7 100644 --- a/test/xml/unit_perc_thresh.xml +++ b/test/xml/unit_perc_thresh.xml @@ -12,6 +12,8 @@ ]> + + diff --git a/test/xml/unit_plot_data_plane.xml b/test/xml/unit_plot_data_plane.xml index f7642b24c2..25d47c1397 100644 --- a/test/xml/unit_plot_data_plane.xml +++ b/test/xml/unit_plot_data_plane.xml @@ -11,13 +11,13 @@ ]> + + &TEST_DIR; true - - &MET_BIN;/plot_data_plane \ diff --git a/test/xml/unit_plot_point_obs.xml b/test/xml/unit_plot_point_obs.xml index 31e2489b72..9daac9a880 100644 --- a/test/xml/unit_plot_point_obs.xml +++ b/test/xml/unit_plot_point_obs.xml @@ -10,12 +10,13 @@ ]> + + &TEST_DIR; true - diff --git a/test/xml/unit_plot_tc.xml b/test/xml/unit_plot_tc.xml index c4f308adb4..d67ade00f0 100644 --- a/test/xml/unit_plot_tc.xml +++ b/test/xml/unit_plot_tc.xml @@ -7,9 +7,9 @@ ]> - + - + &TEST_DIR; true diff --git a/test/xml/unit_point2grid.xml b/test/xml/unit_point2grid.xml index 1ab073e20a..dd1792ccf0 100644 --- a/test/xml/unit_point2grid.xml +++ b/test/xml/unit_point2grid.xml @@ -10,6 +10,8 @@ ]> + + &TEST_DIR; @@ -109,12 +111,12 @@ \ &DATA_DIR_OBS;/point_obs/prepbufr.gdas.2017060300.nc \ G212 \ - &OUTPUT_DIR;/regrid/pb2nc_TMP_big_input.nc \ + &OUTPUT_DIR;/point2grid/pb2nc_TMP_big_input.nc \ -field 'name="TMP"; level="Z2";' \ -v 1 - &OUTPUT_DIR;/regrid/pb2nc_TMP_big_input.nc + &OUTPUT_DIR;/point2grid/pb2nc_TMP_big_input.nc diff --git a/test/xml/unit_point_stat.xml b/test/xml/unit_point_stat.xml index 8750e1ee68..d5446343cd 100644 --- a/test/xml/unit_point_stat.xml +++ b/test/xml/unit_point_stat.xml @@ -11,6 +11,8 @@ ]> + + diff --git a/test/xml/unit_python.xml b/test/xml/unit_python.xml index 35993ed441..698f550f69 100644 --- a/test/xml/unit_python.xml +++ b/test/xml/unit_python.xml @@ -6,7 +6,7 @@ - + @@ -15,6 +15,8 @@ ]> + + &TEST_DIR; diff --git a/test/xml/unit_quality_filter.xml b/test/xml/unit_quality_filter.xml index 912f52bda3..cbf086e4d5 100644 --- a/test/xml/unit_quality_filter.xml +++ b/test/xml/unit_quality_filter.xml @@ -10,16 +10,13 @@ ]> + + &TEST_DIR; true - - - - - &MET_BIN;/point_stat @@ -33,13 +30,13 @@ &DATA_DIR_MODEL;/grib1/nam/nam_2012040900_F012.grib \ &OUTPUT_DIR;/pb2nc/gdas1.20120409.t12z.prepbufr.nc \ &CONFIG_DIR;/PointStatConfig_qty_inc_exc \ - -outdir &OUTPUT_DIR;/point_stat -v 3 + -outdir &OUTPUT_DIR;/quality_filter -v 3 - &OUTPUT_DIR;/point_stat/point_stat_QTY_INC_EXC_PB_120000L_20120409_120000V.stat - &OUTPUT_DIR;/point_stat/point_stat_QTY_INC_EXC_PB_120000L_20120409_120000V_ctc.txt - &OUTPUT_DIR;/point_stat/point_stat_QTY_INC_EXC_PB_120000L_20120409_120000V_sl1l2.txt - &OUTPUT_DIR;/point_stat/point_stat_QTY_INC_EXC_PB_120000L_20120409_120000V_mpr.txt + &OUTPUT_DIR;/quality_filter/point_stat_QTY_INC_EXC_PB_120000L_20120409_120000V.stat + &OUTPUT_DIR;/quality_filter/point_stat_QTY_INC_EXC_PB_120000L_20120409_120000V_ctc.txt + &OUTPUT_DIR;/quality_filter/point_stat_QTY_INC_EXC_PB_120000L_20120409_120000V_sl1l2.txt + &OUTPUT_DIR;/quality_filter/point_stat_QTY_INC_EXC_PB_120000L_20120409_120000V_mpr.txt @@ -57,14 +54,14 @@ &DATA_DIR_MODEL;/grib1/arw-tom-gep3/arw-tom-gep3_2012040900_F012.grib \ &CONFIG_DIR;/EnsembleStatConfig_qty_inc_exc \ -point_obs &OUTPUT_DIR;/madis2nc/metar_2012040912_F000.nc \ - -outdir &OUTPUT_DIR;/ensemble_stat -v 3 + -outdir &OUTPUT_DIR;/quality_filter -v 3 - &OUTPUT_DIR;/ensemble_stat/ensemble_stat_QTY_INC_EXC_MADIS_VGS_20120409_120000V.stat - &OUTPUT_DIR;/ensemble_stat/ensemble_stat_QTY_INC_EXC_MADIS_VGS_20120409_120000V_ecnt.txt - &OUTPUT_DIR;/ensemble_stat/ensemble_stat_QTY_INC_EXC_MADIS_VGS_20120409_120000V_rhist.txt - &OUTPUT_DIR;/ensemble_stat/ensemble_stat_QTY_INC_EXC_MADIS_VGS_20120409_120000V_orank.txt - &OUTPUT_DIR;/ensemble_stat/ensemble_stat_QTY_INC_EXC_MADIS_VGS_20120409_120000V_ens.nc + &OUTPUT_DIR;/quality_filter/ensemble_stat_QTY_INC_EXC_MADIS_VGS_20120409_120000V.stat + &OUTPUT_DIR;/quality_filter/ensemble_stat_QTY_INC_EXC_MADIS_VGS_20120409_120000V_ecnt.txt + &OUTPUT_DIR;/quality_filter/ensemble_stat_QTY_INC_EXC_MADIS_VGS_20120409_120000V_rhist.txt + &OUTPUT_DIR;/quality_filter/ensemble_stat_QTY_INC_EXC_MADIS_VGS_20120409_120000V_orank.txt + &OUTPUT_DIR;/quality_filter/ensemble_stat_QTY_INC_EXC_MADIS_VGS_20120409_120000V_ens.nc diff --git a/test/xml/unit_ref_config.xml b/test/xml/unit_ref_config.xml index 648da522a9..5f9320ba5d 100644 --- a/test/xml/unit_ref_config.xml +++ b/test/xml/unit_ref_config.xml @@ -11,6 +11,8 @@ ]> + + &TEST_DIR; diff --git a/test/xml/unit_regrid.xml b/test/xml/unit_regrid.xml index 8fdc3cb7a6..512b4f0a6c 100644 --- a/test/xml/unit_regrid.xml +++ b/test/xml/unit_regrid.xml @@ -10,6 +10,8 @@ ]> + + &TEST_DIR; diff --git a/test/xml/unit_rmw_analysis.xml b/test/xml/unit_rmw_analysis.xml index f5d871e61d..14f7328957 100644 --- a/test/xml/unit_rmw_analysis.xml +++ b/test/xml/unit_rmw_analysis.xml @@ -6,6 +6,8 @@ ]> + + &TEST_DIR; diff --git a/test/xml/unit_series_analysis.xml b/test/xml/unit_series_analysis.xml index 8a1e2f7e6a..c1e64416b3 100644 --- a/test/xml/unit_series_analysis.xml +++ b/test/xml/unit_series_analysis.xml @@ -11,6 +11,8 @@ ]> + + &TEST_DIR; diff --git a/test/xml/unit_shift_data_plane.xml b/test/xml/unit_shift_data_plane.xml index 3d05e4879b..c8339e089e 100644 --- a/test/xml/unit_shift_data_plane.xml +++ b/test/xml/unit_shift_data_plane.xml @@ -10,6 +10,8 @@ ]> + + &TEST_DIR; diff --git a/test/xml/unit_stat_analysis.xml b/test/xml/unit_stat_analysis.xml index 93ef98b5bc..078b76e072 100644 --- a/test/xml/unit_stat_analysis.xml +++ b/test/xml/unit_stat_analysis.xml @@ -10,13 +10,13 @@ ]> + + &TEST_DIR; true - - @@ -147,7 +147,7 @@ &MET_BIN;/stat_analysis \ - -lookin &OUTPUT_DIR;/ensemble_stat/ensemble_stat_OBS_ERROR_20120410_120000V.stat \ + -lookin &OUTPUT_DIR;/ensemble_stat/ensemble_stat_OBSERR_20120410_120000V.stat \ -job aggregate_stat -line_type ORANK -out_line_type ECNT \ -fcst_var APCP_24 -by VX_MASK \ -set_hdr DESC VX_MASK \ diff --git a/test/xml/unit_tc_dland.xml b/test/xml/unit_tc_dland.xml index baa53b8cb8..3a294af977 100644 --- a/test/xml/unit_tc_dland.xml +++ b/test/xml/unit_tc_dland.xml @@ -6,6 +6,8 @@ ]> + + &TEST_DIR; diff --git a/test/xml/unit_tc_gen.xml b/test/xml/unit_tc_gen.xml index 6146f1d3ec..85357e6bb3 100644 --- a/test/xml/unit_tc_gen.xml +++ b/test/xml/unit_tc_gen.xml @@ -9,6 +9,8 @@ ]> + + &TEST_DIR; diff --git a/test/xml/unit_tc_pairs.xml b/test/xml/unit_tc_pairs.xml index c0e22dcf37..d35465237a 100644 --- a/test/xml/unit_tc_pairs.xml +++ b/test/xml/unit_tc_pairs.xml @@ -9,6 +9,8 @@ ]> + + &TEST_DIR; diff --git a/test/xml/unit_tc_rmw.xml b/test/xml/unit_tc_rmw.xml index c45d71f10e..347ef88d87 100644 --- a/test/xml/unit_tc_rmw.xml +++ b/test/xml/unit_tc_rmw.xml @@ -9,6 +9,8 @@ ]> + + &TEST_DIR; diff --git a/test/xml/unit_tc_stat.xml b/test/xml/unit_tc_stat.xml index 7353d72f68..5029a1f9a8 100644 --- a/test/xml/unit_tc_stat.xml +++ b/test/xml/unit_tc_stat.xml @@ -10,6 +10,8 @@ ]> + + &TEST_DIR; diff --git a/test/xml/unit_trmm2nc.xml b/test/xml/unit_trmm2nc.xml index bbd9e136cf..0aca61588e 100644 --- a/test/xml/unit_trmm2nc.xml +++ b/test/xml/unit_trmm2nc.xml @@ -11,6 +11,8 @@ ]> + + &TEST_DIR; diff --git a/test/xml/unit_wavelet_stat.xml b/test/xml/unit_wavelet_stat.xml index 9e5f6efdd9..f1de40fa53 100644 --- a/test/xml/unit_wavelet_stat.xml +++ b/test/xml/unit_wavelet_stat.xml @@ -10,6 +10,8 @@ ]> + + &TEST_DIR; diff --git a/test/xml/unit_wwmca_plot.xml b/test/xml/unit_wwmca_plot.xml index 2e2a304495..2ea1da991b 100644 --- a/test/xml/unit_wwmca_plot.xml +++ b/test/xml/unit_wwmca_plot.xml @@ -10,6 +10,8 @@ ]> + + &TEST_DIR; diff --git a/test/xml/unit_wwmca_regrid.xml b/test/xml/unit_wwmca_regrid.xml index 86e225718c..1f5b19f9be 100644 --- a/test/xml/unit_wwmca_regrid.xml +++ b/test/xml/unit_wwmca_regrid.xml @@ -10,6 +10,8 @@ ]> + + &TEST_DIR; From b177c3bea86bfd81ab24c900f757539b4ae56d08 Mon Sep 17 00:00:00 2001 From: Howard Soh Date: Mon, 24 Jan 2022 17:48:53 -0700 Subject: [PATCH 72/82] #2020 Added SonarQube related varibales --- scripts/environment/development.seneca | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/scripts/environment/development.seneca b/scripts/environment/development.seneca index 650257b3ab..ccbf506dbf 100644 --- a/scripts/environment/development.seneca +++ b/scripts/environment/development.seneca @@ -49,3 +49,8 @@ export MET_TEST_RSCRIPT=/usr/local/R-4.1.2/bin/Rscript export PATH="/usr/local/nco/bin:/usr/local/netcdf/bin:\ /usr/local/sbin:/usr/local/bin:/usr/sbin:\ /usr/bin:/sbin:/bin:/usr/bin/X11:/opt/bin:$PATH" + +# SonarQube +export SONARQUBE_DIR=/d1/projects/SonarQube/ +export SONARQUBE_WRAPPER_BIN=$SONARQUBE_DIR/build-wrapper-linux-x86 +export SONARQUBE_SCANNER_BIN=$SONARQUBE_DIR/sonar-scanner-4.6.2.2472-linux/bin From 11daa46fdaed235956682f815d7f3e5127cd7f49 Mon Sep 17 00:00:00 2001 From: Howard Soh Date: Mon, 24 Jan 2022 17:52:25 -0700 Subject: [PATCH 73/82] #2020 Initial release --- scripts/sonarqube/run_nightly.sh | 75 +++++++++++ scripts/sonarqube/run_sonarqube.sh | 144 +++++++++++++++++++++ scripts/sonarqube/sonar-project.properties | 18 +++ 3 files changed, 237 insertions(+) create mode 100755 scripts/sonarqube/run_nightly.sh create mode 100755 scripts/sonarqube/run_sonarqube.sh create mode 100644 scripts/sonarqube/sonar-project.properties diff --git a/scripts/sonarqube/run_nightly.sh b/scripts/sonarqube/run_nightly.sh new file mode 100755 index 0000000000..6fcc166950 --- /dev/null +++ b/scripts/sonarqube/run_nightly.sh @@ -0,0 +1,75 @@ +#!/bin/bash +# +# Run nightly SonarQube scan +#======================================================================= +# +# This run_nightly.sh script calls the run_sonarqube.sh script. +# It is intented to be run nightly through cron. Output should be +# directed to the LOGFILE, per cron convention. To run this script, use +# the following commands: +# +# git clone https://github.com/dtcenter/MET +# MET/scripts/sosnarqube/run_nightly.sh name +# +# Usage: run_nightly.sh name +# where "name" specifies a branch, tag, or hash +# +# For example, scan the develop branch: +# run_nightly.sh develop +# +#======================================================================= + +# Constants +#EMAIL_LIST="johnhg@ucar.edu hsoh@ucar.edu jpresto@ucar.edu linden@ucar.edu mccabe@ucar.edu" +EMAIL_LIST="hsoh@ucar.edu" +KEEP_DAYS=5 + +function usage { + echo + echo "USAGE: run_nightly.sh name" + echo " where \"name\" specifies a branch, tag, or hash." + echo +} + +# Check for arguments +if [ $# -lt 1 ]; then usage; exit 1; fi + +# Store the full path to the scripts directory +SCRIPT_DIR=`dirname $0` +if [[ ${0:0:1} != "/" ]]; then SCRIPT_DIR=$(pwd)/${SCRIPT_DIR}; fi + +# Define the development environment +ENV_FILE=${SCRIPT_DIR}/../environment/development.`hostname` +if [[ ! -e ${ENV_FILE} ]]; then + echo "$0: ERROR -> Development environment file missing: ${ENV_FILE}" + exit 1 +fi +source ${ENV_FILE} + +# Delete old directories +find ${MET_PROJ_DIR}/MET_regression/sonarqube \ + -mtime +${KEEP_DAYS} -name "NB*" | \ + xargs rm -rf + +# Create and switch to a run directory +TODAY=`date +%Y%m%d` +YESTERDAY=`date -d "1 day ago" +%Y%m%d` +RUN_DIR=${MET_PROJ_DIR}/MET_regression/sonarqube/NB${TODAY} +if [[ -e ${RUN_DIR} ]]; then rm -rf ${RUN_DIR}; fi +mkdir -p ${RUN_DIR} +cd ${RUN_DIR} + +# Create a logfile +LOGFILE=${RUN_DIR}/run_sonarqube_${TODAY}.log + +# Run scan and check for bad return status +${SCRIPT_DIR}/run_sonarqube.sh ${1} > ${LOGFILE} +if [[ $? -ne 0 ]]; then + echo "$0: The nightly SonarQube scan FAILED in `basename ${RUN_DIR}`." >> ${LOGFILE} + cat ${LOGFILE} | mail -s "MET SonarQube scan Failed for ${1} in `basename ${RUN_DIR}` (autogen msg)" ${EMAIL_LIST} + exit 1 +fi + +# Convert SonarQube report from pdf to html + +exit 0 diff --git a/scripts/sonarqube/run_sonarqube.sh b/scripts/sonarqube/run_sonarqube.sh new file mode 100755 index 0000000000..98c4e15bbe --- /dev/null +++ b/scripts/sonarqube/run_sonarqube.sh @@ -0,0 +1,144 @@ +#!/bin/bash +# +# Run SonarQube Source Code Analyzer on a specified revision of MET +#======================================================================= +# +# This run_sonarqube.sh script will check out the specified version +# of MET and run the SonarQube Source Code Analyzer on it. First, +# go to the directory where you would like the SCA output written and +# then run: +# +# git clone https://github.com/dtcenter/MET +# MET/scripts/sonarqube/run_sonarqube_sca.sh name +# +# Usage: run_sonarqube_sca.sh name +# Test the specified branched version of MET: +# run_sonarqube_sca.sh {branch name} +# Test the specified tagged version of MET: +# run_sonarqube_sca.sh {tag name} +# +#======================================================================= + +# Constants +GIT_REPO="https://github.com/dtcenter/MET" + +function usage { + echo + echo "USAGE: $(basename $0) name" + echo " where \"name\" specifies a branch, tag, or hash." + echo +} + +# Check for arguments +if [[ $# -lt 1 ]]; then usage; exit; fi + +# Check that SONARQUBE_WRAPPER_BIN is defined +if [ -z ${SONARQUBE_WRAPPER_BIN} ]; then + which build-wrapper-linux-x86-64 2> /dev/null + if [ $? -eq 0 ]; then + SONARQUBE_WRAPPER_BIN=$(which build-wrapper-linux-x86-64 2> /dev/null) + else + which build-wrapper 2> /dev/null + if [ $? -eq 0 ]; then + SONARQUBE_WRAPPER_BIN=$(which build-wrapper 2> /dev/null) + else + echo "ERROR: SONARQUBE_WRAPPER_BIN must be set" + exit 1 + fi + fi +fi +if [ ! -e ${SONARQUBE_WRAPPER_BIN} ]; then + echo "ERROR: SONARQUBE_WRAPPER_BIN (${SONARQUBE_WRAPPER_BIN}) does not exist" + exit 1 +fi + +# Check that SONARQUBE_SCANNER_BIN is defined +if [ -z ${SONARQUBE_SCANNER_BIN} ]; then + which sonar-scanner 2> /dev/null + if [ $? -eq 0 ]; then + SONARQUBE_SCANNER_BIN=$(which sonar-scanner 2> /dev/null) + else + echo "ERROR: SONARQUBE_SCANNER_BIN must be set" + exit 1 + fi +fi +if [ ! -e ${SONARQUBE_SCANNER_BIN} ]; then + echo "ERROR: SONARQUBE_SCANNER_BIN (${SONARQUBE_SCANNER_BIN}) does not exist" + exit 1 +fi + +if [ -z ${SONARQUBE_OUT_DIR} ]; then + export SONARQUBE_OUT_DIR=bw-outputs +fi + +# Sub-routine for running a command and checking return status +function run_command() { + + # Print the command being called + echo "CALLING: $1" + + # Run the command and store the return status + $1 + STATUS=$? + + # Check return status + if [[ ${STATUS} -ne 0 ]]; then + echo "ERROR: Command returned with non-zero status ($STATUS): $1" + exit ${STATUS} + fi + + return ${STATUS} +} + + +# Store the full path to the scripts directory +SCRIPT_DIR=`dirname $0` +if [[ ${0:0:1} != "/" ]]; then SCRIPT_DIR=$(pwd)/${SCRIPT_DIR}; fi + +# Clone repo into a sub-directory and checkout the requested version +REPO_DIR="MET-${1}" + +if [ -e ${REPO_DIR} ]; then + run_command "rm -rf ${REPO_DIR}" +fi +run_command "git clone ${GIT_REPO} ${REPO_DIR}" +run_command "cd ${REPO_DIR}" +run_command "git checkout ${1}" + +# Build the MET instance +run_command "cd met" + +# Run bootstrap +run_command "./bootstrap" + +# Do no manually set the CXX and F77 compilers. +# Let the configure script pick them. +# Otherwise, the SonarQube logic does not work. +export MET_DEVELOPMENT=true + +# Run the configure script +run_command "./configure --prefix=`pwd` \ + --enable-grib2 \ + --enable-modis \ + --enable-mode_graphics \ + --enable-lidar2nc \ + --enable-python" + +# Set the build id +#BUILD_ID="MET-${1}" + +# Copy sonar-project.properties +[ ! -e "sonar-project.properties" ] && cp -p $SCRIPT_DIR/sonar-project.properties . + +# Run SonarQube clean +run_command "make clean" + + +# Run SonarQube make +run_command "${SONARQUBE_WRAPPER_BIN}/build-wrapper-linux-x86-64 --out-dir $SONARQUBE_OUT_DIR make" + +# Run SonarQube scan +run_command "${SONARQUBE_SCANNER_BIN}/sonar-scanner" + +# Run SonarQube report generator to make a PDF file +#TODAY=`date +%Y%m%d` diff --git a/scripts/sonarqube/sonar-project.properties b/scripts/sonarqube/sonar-project.properties new file mode 100644 index 0000000000..387a9a3d74 --- /dev/null +++ b/scripts/sonarqube/sonar-project.properties @@ -0,0 +1,18 @@ +sonar.projectKey=org.sonarqube:MET_develop_NB +sonar.projectName=MET Nightly build +sonar.projectVersion=1.0 + +sonar.sources=src + +# The build-wrapper output dir +sonar.cfamily.build-wrapper-output=bw-outputs + +# Encoding of the source files +sonar.sourceEncoding=UTF-8 + +#----- Default SonarQube server +#sonar.host.url=http://localhost:9000 +sonar.host.url=http://mandan:9000 + +sonar.login=met +sonar.password=met@sonar.ucar From 89642fccbc5c44ef76e69cba2a37d6352442c3ee Mon Sep 17 00:00:00 2001 From: Howard Soh Date: Tue, 25 Jan 2022 11:03:40 -0700 Subject: [PATCH 74/82] #2015 Avoid the repeasted debug message if derived varibales are disabled --- met/src/tools/other/pb2nc/pb2nc.cc | 139 ++++++++++++++++------------- 1 file changed, 77 insertions(+), 62 deletions(-) diff --git a/met/src/tools/other/pb2nc/pb2nc.cc b/met/src/tools/other/pb2nc/pb2nc.cc index dd0ba183d8..6aa01dd547 100644 --- a/met/src/tools/other/pb2nc/pb2nc.cc +++ b/met/src/tools/other/pb2nc/pb2nc.cc @@ -367,7 +367,7 @@ static int combine_tqz_and_uv(map, static float compute_pbl(map pqtzuv_map_tq, map pqtzuv_map_uv); static void copy_pqtzuv(float *to_pqtzuv, float *from_pqtzuv, bool copy_all=true); -static void insert_pbl(float *obs_arr, const float pbl_value, const int pbl_code, +static bool insert_pbl(float *obs_arr, const float pbl_value, const int pbl_code, const float pbl_p, const float pbl_h, const float pbl_qm, const float hdr_lat, const float hdr_lon, const float hdr_elv, const time_t hdr_vld_ut, @@ -807,6 +807,7 @@ void process_pbfile(int i_pb) { int rej_typ, rej_sid, rej_vld, rej_grid, rej_poly; int rej_elv, rej_pb_rpt, rej_in_rpt, rej_itp, rej_nobs; int lv, ev, ev_temp, kk, len1, len2; + int n_derived_obs; double x, y; @@ -844,7 +845,8 @@ void process_pbfile(int i_pb) { IntArray diff_file_times; int diff_file_time_count; StringArray variables_big_nlevels; - static const char *method_name = "process_pbfile()"; + static const char *method_name_s = "process_pbfile()"; + static const char *method_name = "process_pbfile() -> "; bool apply_grid_mask = (conf_info.grid_mask.nx() > 0 && conf_info.grid_mask.ny() > 0); @@ -881,7 +883,7 @@ void process_pbfile(int i_pb) { // Check for multiple PrepBufr files if(pbfile.n_elements() > 1) { - mlog << Error << "\n" << method_name << " -> " + mlog << Error << "\n" << method_name << "the \"-dump\" and \"-pbfile\" options may not be " << "used together. Only one Bufr file may be dump " << "to ASCII at a time.\n\n"; @@ -894,7 +896,7 @@ void process_pbfile(int i_pb) { unit = dump_unit+i_pb; if (unit > MAX_FORTRAN_FILE_ID || unit < MIN_FORTRAN_FILE_ID) { - mlog << Error << "\n" << method_name << " -> " + mlog << Error << "\n" << method_name << "Invalid file ID [" << unit << "] between 1 and 99.\n\n"; } prefix = get_short_name(pbfile[i_pb].c_str()); @@ -907,7 +909,7 @@ void process_pbfile(int i_pb) { // Open the blocked temp PrepBufr file for reading unit = file_unit + i_pb; if (unit > MAX_FORTRAN_FILE_ID || unit < MIN_FORTRAN_FILE_ID) { - mlog << Error << "\n" << method_name << " -> " + mlog << Error << "\n" << method_name << "Invalid file ID [" << unit << "] between 1 and 99.\n\n"; } openpb_(blk_file.c_str(), &unit); @@ -925,7 +927,7 @@ void process_pbfile(int i_pb) { // Check for zero messages to process if(npbmsg <= 0 || npbmsg_total <= 0) { - mlog << Warning << "\n" << method_name << " -> " + mlog << Warning << "\n" << method_name << "No Bufr messages to process in file: " << pbfile[i_pb] << "\n\n"; @@ -936,6 +938,7 @@ void process_pbfile(int i_pb) { } // Initialize counts + n_derived_obs = 0; i_ret = n_file_obs = i_msg = 0; rej_typ = rej_sid = rej_vld = rej_grid = rej_poly = 0; rej_elv = rej_pb_rpt = rej_in_rpt = rej_itp = rej_nobs = 0; @@ -946,7 +949,7 @@ void process_pbfile(int i_pb) { bool is_prepbufr = is_prepbufr_file(&event_names); if(mlog.verbosity_level() >= debug_level_for_performance) { end_t = clock(); - mlog << Debug(debug_level_for_performance) << " PERF: " << method_name << " " + mlog << Debug(debug_level_for_performance) << " PERF: " << method_name_s << " " << (end_t-start_t)/double(CLOCKS_PER_SEC) << " seconds for preparing\n"; start_t = clock(); @@ -1011,8 +1014,8 @@ void process_pbfile(int i_pb) { if (cal_pbl) { is_same_header = false; prev_hdr_vld_ut = -1; - m_strncpy(prev_hdr_typ, not_assigned, m_strlen(not_assigned), method_name, "prev_hdr_typ"); - m_strncpy(prev_hdr_sid, not_assigned, m_strlen(not_assigned), method_name, "prev_hdr_sid"); + m_strncpy(prev_hdr_typ, not_assigned, m_strlen(not_assigned), method_name_s, "prev_hdr_typ"); + m_strncpy(prev_hdr_sid, not_assigned, m_strlen(not_assigned), method_name_s, "prev_hdr_sid"); } IMM = JMM =1; @@ -1023,6 +1026,14 @@ void process_pbfile(int i_pb) { diff_file_time_count = 0; cycle_minute = missing_cycle_minute; // initialize + // Derive quantities which can be derived from + // P, Q, T, Z, U, V + if (n_derive_gc > bufr_derive_cfgs.size()) { + mlog << Debug(3) << "\n" << method_name + << "Skip the derived variables because of not requested (" + << bufr_derive_cfgs.size() << ").\n\n"; + } + for (int idx=0; idx " + mlog << Debug(6) << "\n" << method_name << "Switching report type \"" << hdr_typ << "\" to message type \"" << mappedMessageType << "\".\n"; if (mappedMessageType.length() > HEADER_STR_LEN) max_buf = HEADER_STR_LEN; - m_strncpy(modified_hdr_typ, mappedMessageType.c_str(), max_buf, method_name, "modified_hdr_typ1"); + m_strncpy(modified_hdr_typ, mappedMessageType.c_str(), max_buf, method_name_s, "modified_hdr_typ1"); } else { - m_strncpy(modified_hdr_typ, hdr_typ, sizeof(modified_hdr_typ), method_name, "modified_hdr_typ2"); + m_strncpy(modified_hdr_typ, hdr_typ, sizeof(modified_hdr_typ), method_name_s, "modified_hdr_typ2"); } if (max_buf >= max_str_len) max_buf--; modified_hdr_typ[max_buf] = '\0'; @@ -1307,7 +1318,7 @@ void process_pbfile(int i_pb) { buf_nlev = mxr8lv; for(kk=0; kk " + mlog << Warning << "\n" << method_name << "Too many vertical levels (" << nlev << ") for " << bufr_obs_name_arr[kk] << "). Ignored the vertical levels above " << mxr8lv << ".\n\n"; @@ -1520,40 +1531,36 @@ void process_pbfile(int i_pb) { // Derive quantities which can be derived from // P, Q, T, Z, U, V - if (n_derive_gc > bufr_derive_cfgs.size()) { - mlog << Debug(3) << "\n" << method_name << " -> " - << "Skip the derived variables because of not requested (" - << bufr_derive_cfgs.size() << ").\n\n"; - } - else { + if (n_derive_gc <= bufr_derive_cfgs.size()) { for(i=0; i MAX_CAPE_LEVEL) cape_level = MAX_CAPE_LEVEL; reverse_levels = (cape_data_pres[0] > cape_data_pres[cape_level-1]); if (reverse_levels) { @@ -1618,11 +1625,11 @@ void process_pbfile(int i_pb) { swap_value = cape_data_pres[idx]; cape_data_pres[idx] = cape_data_pres[buf_idx]; cape_data_pres[buf_idx] = swap_value; - + swap_value = cape_data_temp[idx]; cape_data_temp[idx] = cape_data_temp[buf_idx]; cape_data_temp[buf_idx] = swap_value; - + swap_value = cape_data_spfh[idx]; cape_data_spfh[idx] = cape_data_spfh[buf_idx]; cape_data_spfh[buf_idx] = swap_value; @@ -1638,7 +1645,7 @@ void process_pbfile(int i_pb) { cape_data_spfh[idx] = r8bfms * 10; } } - + //p1d = cape_p; //t1d = cape_data_temp[cape_level-1]; //q1d = cape_data_spfh[cape_level-1]; @@ -1646,15 +1653,15 @@ void process_pbfile(int i_pb) { &p1d,&t1d,&q1d, static_dummy_201, &cape_level, &IMM,&JMM, &cape_level, &cape_val, &cin_val, &PLCL, &PEQL, static_dummy_200); - + if(mlog.verbosity_level() >= 7) { mlog << Debug(7) << method_name << " index,P,T,Q to compute CAPE from " << i_read << "-th message\n" ; for (int idx=0; idx MAX_CAPE_VALUE) { cape_cnt_too_big++; - mlog << Debug(5) << method_name + mlog << Debug(5) << method_name << " Ignored cape_value: " << cape_val << " cape_level: " << cape_level << ", cin_val: " << cin_val << ", PLCL: " << PLCL << ", PEQL: " << PEQL << "\n"; @@ -1681,6 +1688,7 @@ void process_pbfile(int i_pb) { hdr_lat, hdr_lon, hdr_elv, cape_qm, OBS_BUFFER_SIZE); cape_count++; + n_derived_obs++; if (is_eq(cape_val, 0.)) cape_cnt_zero_values++; } else cape_cnt_missing_values++; @@ -1747,7 +1755,7 @@ void process_pbfile(int i_pb) { if (nlev2 > mxr8lv) { buf_nlev = mxr8lv; if (!variables_big_nlevels.has(var_name, false)) { - mlog << Warning << "\n" << method_name << " -> " + mlog << Warning << "\n" << method_name << "Too many vertical levels (" << nlev2 << ") for " << var_name << ". Ignored the vertical levels above " << mxr8lv << ".\n\n"; @@ -1833,9 +1841,9 @@ void process_pbfile(int i_pb) { has_pbl_data = (pqtzuv_map_tq.size() > 0 && pqtzuv_map_uv.size() > 0); if (is_same_header && has_pbl_data) { float pbl_value = compute_pbl(pqtzuv_map_tq, pqtzuv_map_uv); - - insert_pbl(obs_arr, pbl_value, pbl_code, pbl_p, pbl_h, pbl_qm, - hdr_lat, hdr_lon, hdr_elv, hdr_vld_ut, hdr_typ, hdr_sid); + + if (insert_pbl(obs_arr, pbl_value, pbl_code, pbl_p, pbl_h, pbl_qm, + hdr_lat, hdr_lon, hdr_elv, hdr_vld_ut, hdr_typ, hdr_sid)) n_derived_obs++; for(vector::iterator it = pqtzuv_list.begin(); it != pqtzuv_list.end(); ++it) { @@ -1850,8 +1858,8 @@ void process_pbfile(int i_pb) { prev_hdr_lat = hdr_lat; prev_hdr_lon = hdr_lon; prev_hdr_elv = hdr_elv; - m_strncpy(prev_hdr_typ, hdr_typ, m_strlen(not_assigned), method_name, "prev_hdr_typ"); - m_strncpy(prev_hdr_sid, hdr_sid.c_str(), m_strlen(not_assigned), method_name, "prev_hdr_sid"); + m_strncpy(prev_hdr_typ, hdr_typ, m_strlen(not_assigned), method_name_s, "prev_hdr_typ"); + m_strncpy(prev_hdr_sid, hdr_sid.c_str(), m_strlen(not_assigned), method_name_s, "prev_hdr_sid"); } // If the number of observations for this header is non-zero, @@ -1874,8 +1882,8 @@ void process_pbfile(int i_pb) { has_pbl_data = (pqtzuv_map_tq.size() > 0 || pqtzuv_map_uv.size() > 0); if (do_pbl && has_pbl_data) { float pbl_value = compute_pbl(pqtzuv_map_tq, pqtzuv_map_uv); - insert_pbl(obs_arr, pbl_value, pbl_code, pbl_p, pbl_h, pbl_qm, - hdr_lat, hdr_lon, hdr_elv, hdr_vld_ut, hdr_typ, hdr_sid); + if (insert_pbl(obs_arr, pbl_value, pbl_code, pbl_p, pbl_h, pbl_qm, + hdr_lat, hdr_lon, hdr_elv, hdr_vld_ut, hdr_typ, hdr_sid)) n_derived_obs++; for(vector::iterator it = pqtzuv_list.begin(); it != pqtzuv_list.end(); ++it) { @@ -1898,7 +1906,7 @@ void process_pbfile(int i_pb) { if(0 < diff_file_time_count && 0 < diff_file_times.n_elements()) { mlog << Warning << "\n" << method_name - << " -> The observation time should remain the same for " + << "The observation time should remain the same for " << "all " << (is_prepbufr ? "PrepBufr" : "Bufr") << " messages\n"; mlog << Warning << method_name << " " << diff_file_time_count << " messages with different reference time (" @@ -1952,7 +1960,7 @@ void process_pbfile(int i_pb) { if (npbmsg == rej_vld && 0 < rej_vld) { - mlog << Warning << "\n" << method_name << " -> " + mlog << Warning << "\n" << method_name << "All messages were filtered out by valid time.\n" << "\tPlease adjust time range with \"-valid_beg\" and \"-valid_end\".\n" << "\tmin/max obs time from BUFR file: " << min_time_str @@ -1981,17 +1989,17 @@ void process_pbfile(int i_pb) { remove_temp_file(blk_file); if(mlog.verbosity_level() >= debug_level_for_performance) { method_end = clock(); - cout << " PERF: " << method_name << " " + cout << " PERF: " << method_name_s << " " << (method_end-method_start)/double(CLOCKS_PER_SEC) << " seconds\n"; } if(i_msg <= 0) { - mlog << Warning << "\n" << method_name << " -> " + mlog << Warning << "\n" << method_name + << ((n_derived_obs > 0) ? "Saved the derived variables only. " : " ") << "No " << (is_prepbufr ? "PrepBufr" : "Bufr") << " messages retained from file: " << pbfile[i_pb] << "\n\n"; - return; } return; @@ -2306,7 +2314,7 @@ void process_pbfile_metadata(int i_pb) { readpbint_(&unit, &i_ret, &nlev2, bufr_obs, (char*)var_name.c_str(), &var_name_len, &nlev_max_req); if (0 >= nlev2) continue; - + // Search through the vertical levels has_valid_data = false; for(lv=0; lv pqtzuv_map_tq, } interpolate_pqtzuv(prev_pqtzuv, pqtzuv_merged, next_pqtzuv); } - float first_pres = (pqtzuv_merged == 0 ? bad_data_float : pqtzuv_merged[0]); + float first_pres = (pqtzuv_merged[0] == 0 ? bad_data_float : pqtzuv_merged[0]); pqtzuv_map_merged[first_pres] = pqtzuv_merged; mlog << Debug(9) << method_name << "Added " << first_pres << " to merged records\n"; @@ -3094,11 +3102,12 @@ float compute_pbl(map pqtzuv_map_tq, //////////////////////////////////////////////////////////////////////// -void insert_pbl(float *obs_arr, const float pbl_value, const int pbl_code, +bool insert_pbl(float *obs_arr, const float pbl_value, const int pbl_code, const float pbl_p, const float pbl_h, const float pbl_qm, const float hdr_lat, const float hdr_lon, - const float hdr_elv, const time_t hdr_vld_ut, + const float hdr_elv, const time_t hdr_vld_ut, const ConcatString &hdr_typ, const ConcatString &hdr_sid) { + bool added = false; ConcatString hdr_info; hdr_info << unix_to_yyyymmdd_hhmmss(hdr_vld_ut) << " " << hdr_typ << " " << hdr_sid; @@ -3124,9 +3133,13 @@ void insert_pbl(float *obs_arr, const float pbl_value, const int pbl_code, << pbl_value << ") because of the MAX PBL " << MAX_PBL << " (" << hdr_info<< ")\n"; } - else addObservation(obs_arr, (string)hdr_typ, (string)hdr_sid, hdr_vld_ut, - hdr_lat, hdr_lon, hdr_elv, pbl_qm, OBS_BUFFER_SIZE); + else { + addObservation(obs_arr, (string)hdr_typ, (string)hdr_sid, hdr_vld_ut, + hdr_lat, hdr_lon, hdr_elv, pbl_qm, OBS_BUFFER_SIZE); + added = true; + } } + return added; } //////////////////////////////////////////////////////////////////////// @@ -3189,7 +3202,7 @@ void interpolate_pqtzuv(float *prev_pqtzuv, float *cur_pqtzuv, float *next_pqtzu if ((nint(prev_pqtzuv[0]) == nint(cur_pqtzuv[0])) || (nint(next_pqtzuv[0]) == nint(cur_pqtzuv[0])) || (nint(prev_pqtzuv[0]) == nint(next_pqtzuv[0]))) { - mlog << Error << "\n" << method_name + mlog << Error << "\n" << method_name << " Can't interpolate because of same pressure levels. prev: " << prev_pqtzuv[0] << ", cur: " << cur_pqtzuv[0] << ", next: " << prev_pqtzuv[0] << "\n\n"; @@ -3284,7 +3297,7 @@ void merge_records(float *first_pqtzuv, map pqtzuv_map_pivot, break; } } - + if (it_aux->first == cur_pres) { copy_pqtzuv(pqtzuv_merged, it_aux->second, false); prev_pqtzuv = pqtzuv_merged; @@ -3383,16 +3396,18 @@ void log_pbl_input(int pbl_level, const char *method_name) { for (int idx=0; idx=0; idx--) { - mlog << Debug(PBL_DEBUG_LEVEL) << method_name << " input to calpbl_: " + mlog << Debug(PBL_DEBUG_LEVEL) << method_name << " " << offset++ << "\t" << log_array[idx] << "\n"; } log_array.clear(); From 6b9b41127a2becb30ed813f08349f097f7f5243d Mon Sep 17 00:00:00 2001 From: Howard Soh Date: Tue, 25 Jan 2022 13:23:00 -0700 Subject: [PATCH 75/82] #1996 Initialize right and left --- met/src/basic/vx_util/ascii_table.cc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/met/src/basic/vx_util/ascii_table.cc b/met/src/basic/vx_util/ascii_table.cc index 28fb9004d6..e5b917c377 100644 --- a/met/src/basic/vx_util/ascii_table.cc +++ b/met/src/basic/vx_util/ascii_table.cc @@ -1336,7 +1336,7 @@ const char fill_char = ' '; const int r_start = 1; // skip the header row -for (r=0; r Date: Tue, 25 Jan 2022 13:24:07 -0700 Subject: [PATCH 76/82] #1966 Call clear() instead of reset memory for Header variable --- met/src/libcode/vx_gis/dbf_file.cc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/met/src/libcode/vx_gis/dbf_file.cc b/met/src/libcode/vx_gis/dbf_file.cc index 0a5049d150..174c356c5f 100644 --- a/met/src/libcode/vx_gis/dbf_file.cc +++ b/met/src/libcode/vx_gis/dbf_file.cc @@ -750,7 +750,7 @@ if ( fd >= 0 ) ::close(fd); fd = -1; -memset(&Header, 0, sizeof(Header)); +Header.clear(); Filename.clear(); From 4142c48b86627d38a2adc68451a337c7fe1c8b97 Mon Sep 17 00:00:00 2001 From: Howard Soh Date: Tue, 25 Jan 2022 13:25:00 -0700 Subject: [PATCH 77/82] #1966 Make sure the levels from variable does not exceed the maxLevel --- met/src/tools/other/madis2nc/madis2nc.cc | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/met/src/tools/other/madis2nc/madis2nc.cc b/met/src/tools/other/madis2nc/madis2nc.cc index 6f48a30ccc..0f8164945a 100644 --- a/met/src/tools/other/madis2nc/madis2nc.cc +++ b/met/src/tools/other/madis2nc/madis2nc.cc @@ -3410,6 +3410,11 @@ void process_madis_acarsProfiles(NcFile *&f_in) { levelsQty[i_idx], GET_NC_NAME(in_var).c_str()); nlvl = levels[i_idx]; + if (nlvl > maxLevels) { + mlog << Warning << "\n" << method_name << " The level (" << nlvl + << ") at nLevels variable can not exceed dimension maxLevels (" << maxLevels<< ")\n\n"; + nlvl = maxLevels; + } obs_arr[2] = levels[i_idx]; // From c4f5a1bb969dce20707df462099ee0ed40bd2305 Mon Sep 17 00:00:00 2001 From: Howard Soh Date: Tue, 25 Jan 2022 13:25:44 -0700 Subject: [PATCH 78/82] #1966 Removed unreachable codes --- met/src/tools/other/mode_graphics/cgraph_main.cc | 4 ---- 1 file changed, 4 deletions(-) diff --git a/met/src/tools/other/mode_graphics/cgraph_main.cc b/met/src/tools/other/mode_graphics/cgraph_main.cc index d9963263fc..5020cfe719 100644 --- a/met/src/tools/other/mode_graphics/cgraph_main.cc +++ b/met/src/tools/other/mode_graphics/cgraph_main.cc @@ -2223,14 +2223,10 @@ int my_conic (const FT_Vector * control, const FT_Vector * to, void * u) { -ft_user_info * info = (ft_user_info *) u; - mlog << Error << "\n\n my_conic() -> should never be called!\n\n"; exit ( 1 ); -info->have_path = true; - return ( 0 ); } From 4226cbe44c5e237797813f6b06bffa32a27c919e Mon Sep 17 00:00:00 2001 From: Howard Soh Date: Tue, 25 Jan 2022 13:27:10 -0700 Subject: [PATCH 79/82] #1966 Do not write into NetcDF if empty data --- met/src/libcode/vx_nc_obs/nc_obs_util.cc | 40 +++++++++++++----------- 1 file changed, 21 insertions(+), 19 deletions(-) diff --git a/met/src/libcode/vx_nc_obs/nc_obs_util.cc b/met/src/libcode/vx_nc_obs/nc_obs_util.cc index 70ea844d49..0467f49eee 100644 --- a/met/src/libcode/vx_nc_obs/nc_obs_util.cc +++ b/met/src/libcode/vx_nc_obs/nc_obs_util.cc @@ -1132,26 +1132,28 @@ void NetcdfObsVars::write_header_to_nc(NcDataBuffer &data_buf, else if (data_buf.hdr_data_offset == data_buf.pb_hdr_data_offset) { int save_len = lengths[0]; int pb_hdr_len = raw_hdr_cnt - offsets[0]; - if (pb_hdr_len > buf_size) pb_hdr_len = buf_size; - - lengths[0] = pb_hdr_len; - if(IS_VALID_NC(hdr_prpt_typ_var) && !put_nc_data((NcVar *)&hdr_prpt_typ_var, - data_buf.hdr_prpt_typ_buf, lengths, offsets)) { - mlog << Error << "error writing the pb message type to the netCDF file\n\n"; - exit(1); - } - if(IS_VALID_NC(hdr_irpt_typ_var) && !put_nc_data((NcVar *)&hdr_irpt_typ_var, - data_buf.hdr_irpt_typ_buf, lengths, offsets)) { - mlog << Error << "error writing the in message type to the netCDF file\n\n"; - exit(1); - } - if(IS_VALID_NC(hdr_inst_typ_var) && !put_nc_data((NcVar *)&hdr_inst_typ_var, - data_buf.hdr_inst_typ_buf, lengths, offsets)) { - mlog << Error << "error writing the instrument type to the netCDF file\n\n"; - exit(1); + if (pb_hdr_len > 0) { + if (pb_hdr_len > buf_size) pb_hdr_len = buf_size; + + lengths[0] = pb_hdr_len; + if(IS_VALID_NC(hdr_prpt_typ_var) && !put_nc_data((NcVar *)&hdr_prpt_typ_var, + data_buf.hdr_prpt_typ_buf, lengths, offsets)) { + mlog << Error << "error writing the pb message type to the netCDF file\n\n"; + exit(1); + } + if(IS_VALID_NC(hdr_irpt_typ_var) && !put_nc_data((NcVar *)&hdr_irpt_typ_var, + data_buf.hdr_irpt_typ_buf, lengths, offsets)) { + mlog << Error << "error writing the in message type to the netCDF file\n\n"; + exit(1); + } + if(IS_VALID_NC(hdr_inst_typ_var) && !put_nc_data((NcVar *)&hdr_inst_typ_var, + data_buf.hdr_inst_typ_buf, lengths, offsets)) { + mlog << Error << "error writing the instrument type to the netCDF file\n\n"; + exit(1); + } + lengths[0] = save_len; + data_buf.pb_hdr_data_offset += pb_hdr_len; } - lengths[0] = save_len; - data_buf.pb_hdr_data_offset += pb_hdr_len; } else { mlog << Debug(6) << method_name From 587b936ceb3afd0e974538351d1730b3eb36ea84 Mon Sep 17 00:00:00 2001 From: Howard Soh Date: Wed, 26 Jan 2022 18:59:21 -0700 Subject: [PATCH 80/82] #2015 Corrected "retained or derived" count --- met/src/tools/other/pb2nc/pb2nc.cc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/met/src/tools/other/pb2nc/pb2nc.cc b/met/src/tools/other/pb2nc/pb2nc.cc index 6aa01dd547..50a8fa9ab8 100644 --- a/met/src/tools/other/pb2nc/pb2nc.cc +++ b/met/src/tools/other/pb2nc/pb2nc.cc @@ -1946,7 +1946,7 @@ void process_pbfile(int i_pb) { << "Total Messages retained\t\t= " << i_msg << "\n" << "Total observations retained or derived\t= " - << n_file_obs << "\n"; + << (n_file_obs + n_derived_obs) << "\n"; if (cal_cape) { mlog << Debug(3) << "\nDerived CAPE = " << cape_count From 3097278e2496d9cdc2fa9c20002ae284e00d06d8 Mon Sep 17 00:00:00 2001 From: George McCabe <23407799+georgemccabe@users.noreply.github.com> Date: Thu, 27 Jan 2022 14:50:02 -0700 Subject: [PATCH 81/82] Feature 1546 CI testing (#2029) Co-authored-by: John Halley Gotway --- .github/jobs/Dockerfile.truth | 17 + .github/jobs/bash_functions.sh | 27 + .github/jobs/build_and_push_docker_image.sh | 40 - .github/jobs/build_docker_image.sh | 14 + .github/jobs/copy_diff_files.py | 80 ++ .github/jobs/create_docker_truth.sh | 18 + .github/jobs/get_branch_name.sh | 15 + .github/jobs/get_test_input_data.sh | 7 + .github/jobs/get_test_truth_data.sh | 11 + .github/jobs/pull_docker_image.sh | 7 + .github/jobs/push_docker_image.sh | 15 + .github/jobs/run_diff_docker.sh | 48 + .github/jobs/run_diff_tests.sh | 33 + .github/jobs/run_unit_docker.sh | 37 + .github/jobs/run_unit_tests.sh | 26 + .github/jobs/set_job_controls.sh | 94 ++ .github/jobs/test_env_vars.sh | 13 + .../build_docker_and_trigger_metplus.yml | 17 +- .github/workflows/unit_tests.yml | 377 +++++++ .gitignore | 3 + met/scripts/python/read_met_point_obs.py | 12 +- scripts/docker/Dockerfile | 154 +-- scripts/docker/Dockerfile.copy | 46 + scripts/docker/Dockerfile.minimum | 138 +++ scripts/docker/Dockerfile.test | 26 + scripts/docker/README.md | 40 + scripts/docker/build_met_docker.sh | 37 + test/bin/unit_test.sh | 16 +- test/xml/unit_ascii2nc.xml | 319 +----- test/xml/unit_ascii2nc_indy.xml | 340 +++++++ test/xml/unit_climatology.xml | 361 ------- test/xml/unit_climatology_1.0deg.xml | 252 +++++ test/xml/unit_climatology_1.5deg.xml | 98 ++ test/xml/unit_climatology_2.5deg.xml | 53 + test/xml/unit_pb2nc.xml | 136 +-- test/xml/unit_pb2nc_indy.xml | 157 +++ test/xml/unit_pcp_combine.xml | 4 +- test/xml/unit_plot_data_plane.xml | 4 +- test/xml/unit_python.xml | 12 +- test/xml/unit_ref_config.xml | 931 +----------------- test/xml/unit_ref_config_lead_00.xml | 178 ++++ test/xml/unit_ref_config_lead_12.xml | 230 +++++ test/xml/unit_ref_config_lead_24.xml | 231 +++++ test/xml/unit_ref_config_lead_36.xml | 274 ++++++ test/xml/unit_ref_config_lead_48.xml | 231 +++++ test/xml/unit_stat_analysis.xml | 331 ------- test/xml/unit_stat_analysis_es.xml | 182 ++++ test/xml/unit_stat_analysis_gs.xml | 45 + test/xml/unit_stat_analysis_ps.xml | 120 +++ test/xml/unit_stat_analysis_ws.xml | 38 + 50 files changed, 3621 insertions(+), 2274 deletions(-) create mode 100644 .github/jobs/Dockerfile.truth create mode 100755 .github/jobs/bash_functions.sh delete mode 100755 .github/jobs/build_and_push_docker_image.sh create mode 100755 .github/jobs/build_docker_image.sh create mode 100755 .github/jobs/copy_diff_files.py create mode 100755 .github/jobs/create_docker_truth.sh create mode 100755 .github/jobs/get_branch_name.sh create mode 100755 .github/jobs/get_test_input_data.sh create mode 100755 .github/jobs/get_test_truth_data.sh create mode 100755 .github/jobs/pull_docker_image.sh create mode 100755 .github/jobs/push_docker_image.sh create mode 100755 .github/jobs/run_diff_docker.sh create mode 100755 .github/jobs/run_diff_tests.sh create mode 100755 .github/jobs/run_unit_docker.sh create mode 100755 .github/jobs/run_unit_tests.sh create mode 100755 .github/jobs/set_job_controls.sh create mode 100755 .github/jobs/test_env_vars.sh create mode 100644 .github/workflows/unit_tests.yml create mode 100644 scripts/docker/Dockerfile.copy create mode 100644 scripts/docker/Dockerfile.minimum create mode 100644 scripts/docker/Dockerfile.test create mode 100644 scripts/docker/README.md create mode 100755 scripts/docker/build_met_docker.sh create mode 100644 test/xml/unit_ascii2nc_indy.xml delete mode 100644 test/xml/unit_climatology.xml create mode 100644 test/xml/unit_climatology_1.0deg.xml create mode 100644 test/xml/unit_climatology_1.5deg.xml create mode 100644 test/xml/unit_climatology_2.5deg.xml create mode 100644 test/xml/unit_pb2nc_indy.xml create mode 100644 test/xml/unit_ref_config_lead_00.xml create mode 100644 test/xml/unit_ref_config_lead_12.xml create mode 100644 test/xml/unit_ref_config_lead_24.xml create mode 100644 test/xml/unit_ref_config_lead_36.xml create mode 100644 test/xml/unit_ref_config_lead_48.xml delete mode 100644 test/xml/unit_stat_analysis.xml create mode 100644 test/xml/unit_stat_analysis_es.xml create mode 100644 test/xml/unit_stat_analysis_gs.xml create mode 100644 test/xml/unit_stat_analysis_ps.xml create mode 100644 test/xml/unit_stat_analysis_ws.xml diff --git a/.github/jobs/Dockerfile.truth b/.github/jobs/Dockerfile.truth new file mode 100644 index 0000000000..6366360e13 --- /dev/null +++ b/.github/jobs/Dockerfile.truth @@ -0,0 +1,17 @@ +FROM centos:7 +MAINTAINER George McCabe + +ENV OUTPUT_DIR /data/output +RUN mkdir -p ${OUTPUT_DIR} + +ARG TRUTH_DIR + +COPY ${TRUTH_DIR} ${OUTPUT_DIR}/ + +ARG TRUTH_DIR + +# Define the volume mount point +VOLUME ${OUTPUT_DIR}/${TRUTH_DIR} + +USER root +CMD ["true"] \ No newline at end of file diff --git a/.github/jobs/bash_functions.sh b/.github/jobs/bash_functions.sh new file mode 100755 index 0000000000..5205753780 --- /dev/null +++ b/.github/jobs/bash_functions.sh @@ -0,0 +1,27 @@ +#! /bin/bash + +# utility function to run command get log the time it took to run +# if CMD_LOGFILE is set, send output to that file and unset var +function time_command { + local start_seconds=$SECONDS + echo "RUNNING: $*" + + local error + # pipe output to log file if set + if [ "x$CMD_LOGFILE" == "x" ]; then + "$@" + error=$? + else + echo "Logging to ${CMD_LOGFILE}" + "$@" &>> $CMD_LOGFILE + error=$? + unset CMD_LOGFILE + fi + + local duration=$(( SECONDS - start_seconds )) + echo "TIMING: Command took `printf '%02d' $(($duration / 60))`:`printf '%02d' $(($duration % 60))` (MM:SS): '$*'" + if [ ${error} -ne 0 ]; then + echo "ERROR: '$*' exited with status = ${error}" + fi + return $error +} diff --git a/.github/jobs/build_and_push_docker_image.sh b/.github/jobs/build_and_push_docker_image.sh deleted file mode 100755 index 766b67cc41..0000000000 --- a/.github/jobs/build_and_push_docker_image.sh +++ /dev/null @@ -1,40 +0,0 @@ -#! /bin/bash - -# utility function to run command get log the time it took to run -function time_command { - local start_seconds=$SECONDS - echo "RUNNING: $*" - "$@" - local error=$? - - local duration=$(( SECONDS - start_seconds )) - echo "TIMING: Command took `printf '%02d' $(($duration / 60))`:`printf '%02d' $(($duration % 60))` (MM:SS): '$*'" - if [ ${error} -ne 0 ]; then - echo "ERROR: '$*' exited with status = ${error}" - fi - return $error -} - -prefix=refs/heads/ -branch_name=${GITHUB_REF#"$prefix"} -DOCKERHUB_TAG=dtcenter/met:${branch_name} - -DOCKERFILE_DIR=${GITHUB_WORKSPACE}/scripts/docker - -echo "::group::Docker Build Command" -time_command docker build -t ${DOCKERHUB_TAG} \ - --build-arg SOURCE_BRANCH=$branch_name \ - $DOCKERFILE_DIR -echo "::endgroup::" - -# skip docker push if credentials are not set -if [ -z ${DOCKER_USERNAME+x} ] || [ -z ${DOCKER_PASSWORD+x} ]; then - echo "DockerHub credentials not set. Skipping docker push" - exit 0 -fi - -echo "$DOCKER_PASSWORD" | docker login --username "$DOCKER_USERNAME" --password-stdin - -echo "::group::Docker Push Command" -time_command docker push ${DOCKERHUB_TAG} -echo "::endgroup::" diff --git a/.github/jobs/build_docker_image.sh b/.github/jobs/build_docker_image.sh new file mode 100755 index 0000000000..5a9b2e8cf7 --- /dev/null +++ b/.github/jobs/build_docker_image.sh @@ -0,0 +1,14 @@ +#! /bin/bash + +source ${GITHUB_WORKSPACE}/.github/jobs/bash_functions.sh + +DOCKERHUB_TAG=${DOCKERHUB_REPO}:${SOURCE_BRANCH} + +DOCKERFILE_PATH=${GITHUB_WORKSPACE}/scripts/docker/Dockerfile.copy + +CMD_LOGFILE=${GITHUB_WORKSPACE}/docker_build.log + +time_command docker build -t ${DOCKERHUB_TAG} \ + --build-arg SOURCE_BRANCH \ + --build-arg MET_BASE_IMAGE \ + -f $DOCKERFILE_PATH ${GITHUB_WORKSPACE} diff --git a/.github/jobs/copy_diff_files.py b/.github/jobs/copy_diff_files.py new file mode 100755 index 0000000000..eed5b6a339 --- /dev/null +++ b/.github/jobs/copy_diff_files.py @@ -0,0 +1,80 @@ +#! /usr/bin/env python3 + +import os +import shutil + +OUTPUT_DIR = os.environ['MET_TEST_OUTPUT'] +TRUTH_DIR = os.environ['MET_TEST_TRUTH'] +DIFF_DIR = os.environ['MET_TEST_DIFF'] + +LOG_DIR = '/met/logs' + +def get_files_with_diffs(log_file): + files_to_copy = set() + + with open(log_file, 'r') as file_handle: + file_content = file_handle.read() + + missing_section, *test_sections = file_content.split( + '\n# # # # # # # # # # # # # # # # # # # # # # # # # # # # # #\n' + ) + + # parse list of missing files + if 'ERROR:' in missing_section: + for missing_group in missing_section.split('ERROR:')[1:]: + dir_str, *rel_paths = missing_group.splitlines() + dir_str = dir_str.split()[1] + if OUTPUT_DIR in dir_str: + top_dir = dir_str.replace(OUTPUT_DIR, TRUTH_DIR) + elif TRUTH_DIR in dir_str: + top_dir = dir_str.replace(TRUTH_DIR, OUTPUT_DIR) + else: + print("ERROR: SOMETHING WENT WRONG PARSING COMP_DIR OUTPUT") + continue + for rel_path in rel_paths: + files_to_copy.add(os.path.join(top_dir, rel_path.strip())) + + # parse file paths out of sections that have errors + error_sections = [item for item in test_sections if 'ERROR:' in item] + for error_section in error_sections: + for line in error_section.splitlines(): + for item in line.split(): + if OUTPUT_DIR in item or TRUTH_DIR in item: + files_to_copy.add(item) + + return files_to_copy + +def copy_files_to_diff_dir(files_to_copy): + + print(f"Found {len(files_to_copy)} diff files") + + # add extension for output/truth and copy files to diff directory + for filename in files_to_copy: + output_path, extension = os.path.splitext(filename) + if OUTPUT_DIR in output_path: + output_path = f'{output_path}_OUTPUT{extension}' + output_path = output_path.replace(OUTPUT_DIR, DIFF_DIR) + elif TRUTH_DIR in output_path: + output_path = f'{output_path}_TRUTH{extension}' + output_path = output_path.replace(TRUTH_DIR, DIFF_DIR) + else: + continue + + # change bad char - this can be removed once test output is changed + output_path = output_path.replace(':', '_') + + print(f"Copy {filename} to {output_path}") + output_dir = os.path.dirname(output_path) + if not os.path.exists(output_dir): + os.makedirs(output_dir) + shutil.copyfile(filename, output_path) + +def main(): + log_file = os.path.join(LOG_DIR, 'comp_dir.log') + print(f"Parsing {log_file}") + all_files_to_copy = get_files_with_diffs(log_file) + + copy_files_to_diff_dir(all_files_to_copy) + +if __name__ == "__main__": + main() diff --git a/.github/jobs/create_docker_truth.sh b/.github/jobs/create_docker_truth.sh new file mode 100755 index 0000000000..298852f80a --- /dev/null +++ b/.github/jobs/create_docker_truth.sh @@ -0,0 +1,18 @@ +#! /bin/bash + +source ${GITHUB_WORKSPACE}/.github/jobs/bash_functions.sh + +image_name=dtcenter/met-data-output:${TRUTH_DATA_VERSION} + +time_command docker build -t ${image_name} \ + --build-arg TRUTH_DIR=met_test_truth \ + -f ${GITHUB_WORKSPACE}/.github/jobs/Dockerfile.truth \ + ${RUNNER_WORKSPACE} +if [ $? != 0 ]; then + echo "ERROR: Docker build failed" + exit 1 +fi + +echo "$DOCKER_PASSWORD" | docker login --username "$DOCKER_USERNAME" --password-stdin + +time_command docker push ${image_name} diff --git a/.github/jobs/get_branch_name.sh b/.github/jobs/get_branch_name.sh new file mode 100755 index 0000000000..ec42a94e38 --- /dev/null +++ b/.github/jobs/get_branch_name.sh @@ -0,0 +1,15 @@ +#! /bin/bash + +# If pull request, use GitHub head ref and add -PR to end +# Otherwise use GitHub ref + +if [ "${GITHUB_EVENT_NAME}" == "pull_request" ] ; then + branch_name=${GITHUB_HEAD_REF}-PR +else + branch_name=${GITHUB_REF} +fi + +branch_name=${branch_name#"refs/heads/"} + +echo ::set-output name=branch_name::$branch_name +echo branch_name: $branch_name diff --git a/.github/jobs/get_test_input_data.sh b/.github/jobs/get_test_input_data.sh new file mode 100755 index 0000000000..d11c651265 --- /dev/null +++ b/.github/jobs/get_test_input_data.sh @@ -0,0 +1,7 @@ +#! /bin/bash + +source ${GITHUB_WORKSPACE}/.github/jobs/bash_functions.sh + +DATA_VERSION=$1 + +time_command docker create --name met_input dtcenter/met-data-dev:${DATA_VERSION} diff --git a/.github/jobs/get_test_truth_data.sh b/.github/jobs/get_test_truth_data.sh new file mode 100755 index 0000000000..3b4055b4be --- /dev/null +++ b/.github/jobs/get_test_truth_data.sh @@ -0,0 +1,11 @@ +#! /bin/bash + +source ${GITHUB_WORKSPACE}/.github/jobs/bash_functions.sh + +DATA_VERSION=$1 + +time_command docker create --name met_truth dtcenter/met-data-output:${DATA_VERSION} +if [ $? != 0 ]; then + echo "Image tag ${DATA_VERSION} does not exist. Using develop..." + time_command docker create --name met_truth dtcenter/met-data-output:develop +fi diff --git a/.github/jobs/pull_docker_image.sh b/.github/jobs/pull_docker_image.sh new file mode 100755 index 0000000000..e97c6a1f07 --- /dev/null +++ b/.github/jobs/pull_docker_image.sh @@ -0,0 +1,7 @@ +#! /bin/bash + +source ${GITHUB_WORKSPACE}/.github/jobs/bash_functions.sh + +DOCKERHUB_TAG=$1 + +time_command docker pull ${DOCKERHUB_TAG} diff --git a/.github/jobs/push_docker_image.sh b/.github/jobs/push_docker_image.sh new file mode 100755 index 0000000000..83a8e4fc5f --- /dev/null +++ b/.github/jobs/push_docker_image.sh @@ -0,0 +1,15 @@ +#! /bin/bash + +source ${GITHUB_WORKSPACE}/.github/jobs/bash_functions.sh + +DOCKERHUB_TAG=${DOCKERHUB_REPO}:${SOURCE_BRANCH} + +# skip docker push if credentials are not set +if [ -z ${DOCKER_USERNAME+x} ] || [ -z ${DOCKER_PASSWORD+x} ]; then + echo "DockerHub credentials not set. Skipping docker push" + exit 0 +fi + +echo "$DOCKER_PASSWORD" | docker login --username "$DOCKER_USERNAME" --password-stdin + +time_command docker push ${DOCKERHUB_TAG} diff --git a/.github/jobs/run_diff_docker.sh b/.github/jobs/run_diff_docker.sh new file mode 100755 index 0000000000..b4095fb21e --- /dev/null +++ b/.github/jobs/run_diff_docker.sh @@ -0,0 +1,48 @@ +#! /bin/bash + +source ${GITHUB_WORKSPACE}/.github/jobs/bash_functions.sh + +DOCKERHUB_TAG=${DOCKERHUB_REPO}:${SOURCE_BRANCH} + +# Get truth output data +${GITHUB_WORKSPACE}/.github/jobs/get_test_truth_data.sh ${TRUTH_DATA_VERSION} + +# Set up directories to mount +LOCAL_OUTPUT_DIR=${RUNNER_WORKSPACE}/output +DOCKER_OUTPUT_DIR=/data/output/met_test_output + +LOCAL_DIFF_DIR=${RUNNER_WORKSPACE}/diff +DOCKER_DIFF_DIR=/data/output/met_test_diff + +LOCAL_LOG_DIR=${RUNNER_WORKSPACE}/logs +DOCKER_LOG_DIR=/met/logs + +# Create local directories to store output +mkdir -p ${LOCAL_LOG_DIR} +mkdir -p ${LOCAL_DIFF_DIR} + +# mount output and log dirs, mount GitHub files into MET_REPO_DIR +mount_args="-v ${LOCAL_OUTPUT_DIR}:${DOCKER_OUTPUT_DIR} -v ${LOCAL_DIFF_DIR}:${DOCKER_DIFF_DIR} -v ${LOCAL_LOG_DIR}:${DOCKER_LOG_DIR}" + +# Set up data volumes +volumes_from="--volumes-from met_truth" + +# run unit test script inside Docker, mount MET output and truth data +# set MET_REPO_DIR env var in Docker to mounted directory +cmd="\${MET_REPO_DIR}/.github/jobs/run_diff_tests.sh" +time_command docker run ${volumes_from} ${mount_args} ${DOCKERHUB_TAG} bash -c \"${cmd}\" +if [ $? != 0 ]; then + exit 1 +fi + +if [ "$(ls -A ${LOCAL_DIFF_DIR})" ]; then + echo "ERROR: Differences exist in the output" + + # only exit non-zero (job fails) if not updating truth data + # this makes difference output available when updating truth data + # so it is easier to see what changed with the update + if [ "${RUN_UPDATE_TRUTH}" != "true" ]; then + exit 1 + fi + +fi diff --git a/.github/jobs/run_diff_tests.sh b/.github/jobs/run_diff_tests.sh new file mode 100755 index 0000000000..fd84a6953f --- /dev/null +++ b/.github/jobs/run_diff_tests.sh @@ -0,0 +1,33 @@ +#! /bin/bash + +source ${MET_REPO_DIR}/.github/jobs/bash_functions.sh + +### +# Set environment variables needed to run unit tests +### + +source ${MET_REPO_DIR}/.github/jobs/test_env_vars.sh + +### +# Run comparison of MET unit test output +### + +echo "Running comparison on test output" +CMD_LOGFILE=/met/logs/comp_dir.log +time_command ${MET_TEST_BASE}/bin/comp_dir.sh ${MET_TEST_TRUTH} ${MET_TEST_OUTPUT} +if [ $? != 0 ]; then + echo "ERROR: Test output comparison failed" + cat /met/logs/comp_dir.log + exit 1 +fi + +echo "Running copy_diff_files.py" +CMD_LOGFILE=/met/logs/copy_diff_files.log +time_command ${MET_REPO_DIR}/.github/jobs/copy_diff_files.py +if [ $? != 0 ]; then + echo "ERROR: Copy diff files script failed" + cat /met/logs/copy_diff_files.log + exit 1 +fi + +echo "Success" diff --git a/.github/jobs/run_unit_docker.sh b/.github/jobs/run_unit_docker.sh new file mode 100755 index 0000000000..95f4b2c626 --- /dev/null +++ b/.github/jobs/run_unit_docker.sh @@ -0,0 +1,37 @@ +#! /bin/bash + +source ${GITHUB_WORKSPACE}/.github/jobs/bash_functions.sh + +DOCKERHUB_TAG=${DOCKERHUB_REPO}:${SOURCE_BRANCH} + +# Pull MET Image from DockerHub +${GITHUB_WORKSPACE}/.github/jobs/pull_docker_image.sh ${DOCKERHUB_TAG} + +# Get test input data if needed +volumes_from="" +if [ "${INPUT_DATA_VERSION}" != "none" ]; then + ${GITHUB_WORKSPACE}/.github/jobs/get_test_input_data.sh ${INPUT_DATA_VERSION} + volumes_from=${volumes_from}"--volumes-from met_input" +fi + +# Set up directories to mount +LOCAL_OUTPUT_DIR=${RUNNER_WORKSPACE}/output +DOCKER_OUTPUT_DIR=/data/output/met_test_output + +LOCAL_LOG_DIR=${RUNNER_WORKSPACE}/logs +DOCKER_LOG_DIR=/met/logs + +# Create local directories to store output +mkdir -p ${LOCAL_LOG_DIR} +mkdir -p ${LOCAL_OUTPUT_DIR} + +mount_args="-v ${LOCAL_OUTPUT_DIR}:${DOCKER_OUTPUT_DIR} -v ${LOCAL_LOG_DIR}:${DOCKER_LOG_DIR}" + +export TESTS_TO_RUN=$TESTS + +# run unit test script inside Docker, mount MET input and truth data +cmd="\${MET_REPO_DIR}/.github/jobs/run_unit_tests.sh" +time_command docker run -e TESTS_TO_RUN ${volumes_from} ${mount_args} ${DOCKERHUB_TAG} bash -c \"${cmd}\" +if [ $? != 0 ]; then + exit 1 +fi diff --git a/.github/jobs/run_unit_tests.sh b/.github/jobs/run_unit_tests.sh new file mode 100755 index 0000000000..e866b2cc61 --- /dev/null +++ b/.github/jobs/run_unit_tests.sh @@ -0,0 +1,26 @@ +#! /bin/bash + +source ${MET_REPO_DIR}/.github/jobs/bash_functions.sh + +### +# Set environment variables needed to run unit tests +### + +source ${MET_REPO_DIR}/.github/jobs/test_env_vars.sh + +### +# Run MET unit tests +### + +echo "Running MET unit tests..." +for testname in $TESTS_TO_RUN; do + CMD_LOGFILE=/met/logs/unit_${testname}.log + time_command ${MET_TEST_BASE}/perl/unit.pl ${MET_TEST_BASE}/xml/unit_${testname}.xml + if [ $? != 0 ]; then + echo "ERROR: Unit test ${testname} failed" + cat /met/logs/unit_${testname}.log + exit 1 + fi +done + +echo "Success" diff --git a/.github/jobs/set_job_controls.sh b/.github/jobs/set_job_controls.sh new file mode 100755 index 0000000000..6a4e4bdf82 --- /dev/null +++ b/.github/jobs/set_job_controls.sh @@ -0,0 +1,94 @@ +#! /bin/bash + +run_compile=true +run_push=false +run_unit_tests=false +run_diff=false +run_update_truth=false +met_base_image=minimum +input_data_version=develop +truth_data_version=develop + +if [ "${GITHUB_EVENT_NAME}" == "pull_request" ]; then + + # only run diff logic if pull request INTO + # branches not ending with -ref + if [ "${GITHUB_BASE_REF: -4}" != "-ref" ]; then + + run_diff=true + + fi + +elif [ "${GITHUB_EVENT_NAME}" == "push" ]; then + + branch_name=`cut -d "/" -f3 <<< "${GITHUB_REF}"` + + # if branch ends with -ref, update truth data from unit tests + if [ "${branch_name: -4}" == -ref ]; then + + run_update_truth=true + run_diff=true + truth_data_version=${branch_name: -4} + + else + + # if develop or main_vX.Y branch, run diff tests using branch's truth data + if [ "$branch_name" == "develop" ] || + [ "${branch_name:0:6}" == "main_v" ]; then + + run_diff=true + truth_data_version=${branch_name} + + fi + + # check commit messages for ci-skip or ci-run keywords + if grep -q "ci-skip-compile" <<< "$commit_msg"; then + + run_compile=false + + fi + + if grep -q "ci-run-unit" <<< "$commit_msg"; then + + run_diff=true + + fi + fi + +fi + +# if updating truth or running diff, run unit tests +if [ "$run_update_truth" == "true" ] || [ "$run_diff" == "true" ]; then + + run_unit_tests=true + +fi + +# if running unit tests, use unit_test MET base image and push image +if [ "$run_unit_tests" == "true" ]; then + + met_base_image=unit_test + run_push=true + +fi + +echo ::set-output name=run_compile::$run_compile +echo ::set-output name=run_push::$run_push +echo ::set-output name=run_unit_tests::$run_unit_tests +echo ::set-output name=run_diff::$run_diff +echo ::set-output name=run_update_truth::$run_update_truth +echo ::set-output name=met_base_image::$met_base_image +echo ::set-output name=input_data_version::$input_data_version +echo ::set-output name=truth_data_version::$truth_data_version + +echo run_compile: $run_compile +echo run_push: $run_push +echo run_unit_tests: $run_unit_tests +echo run_diff: $run_diff +echo run_update_truth: $run_update_truth +echo met_base_image: $met_base_image +echo input_data_version: $input_data_version +echo truth_data_version: $truth_data_version + +# get name of branch +.github/jobs/get_branch_name.sh diff --git a/.github/jobs/test_env_vars.sh b/.github/jobs/test_env_vars.sh new file mode 100755 index 0000000000..5eaaa44dd9 --- /dev/null +++ b/.github/jobs/test_env_vars.sh @@ -0,0 +1,13 @@ +export MET_BASE=/usr/local/share/met + +export MET_BUILD_BASE=${MET_REPO_DIR}/met +export MET_TEST_BASE=${MET_REPO_DIR}/test +export PERL5LIB=${MET_TEST_BASE}/lib + +export MET_TEST_INPUT=/data/input/MET_test_data/unit_test +export MET_TEST_OUTPUT=/data/output/met_test_output +export MET_TEST_TRUTH=/data/output/met_test_truth +export MET_TEST_DIFF=/data/output/met_test_diff + +export MET_TEST_RSCRIPT=/usr/bin/Rscript +export MET_TEST_MET_PYTHON_EXE=/usr/bin/python3 diff --git a/.github/workflows/build_docker_and_trigger_metplus.yml b/.github/workflows/build_docker_and_trigger_metplus.yml index 070fa071a6..153adcb060 100644 --- a/.github/workflows/build_docker_and_trigger_metplus.yml +++ b/.github/workflows/build_docker_and_trigger_metplus.yml @@ -8,16 +8,29 @@ on: - 'met/docs/**' jobs: + build_met_docker: name: Handle Docker Image runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 - - name: Build and Push Docker Image - run: .github/jobs/build_and_push_docker_image.sh + + - name: Get branch name + id: get_branch_name + run: echo ::set-output name=branch_name::${GITHUB_REF#"refs/heads/"} + + - name: Build Docker Image + run: .github/jobs/build_docker_image.sh + env: + SOURCE_BRANCH: ${{ steps.get_branch_name.outputs.branch_name }} + MET_BASE_IMAGE: minimum + + - name: Push Docker Image + run: .github/jobs/push_docker_image.sh env: DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }} DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }} + trigger_metplus: name: Trigger METplus testing workflow runs-on: ubuntu-latest diff --git a/.github/workflows/unit_tests.yml b/.github/workflows/unit_tests.yml new file mode 100644 index 0000000000..c114c61028 --- /dev/null +++ b/.github/workflows/unit_tests.yml @@ -0,0 +1,377 @@ +name: Unit Tests + +# Compile MET and run unit tests +# for pull requests into develop branch + +on: + pull_request: + types: [opened, reopened, synchronize] + branches: + - develop + - 'main_v*' + push: + branches: + - 'feature_*' + - 'bugfix_*' + - 'develop' + - 'develop-ref' + - 'main_v*' + +env: + DOCKERHUB_REPO: dtcenter/met-dev + +jobs: + + job_control: + name: Determine which jobs to run + runs-on: ubuntu-latest + + steps: + - uses: actions/checkout@v2 + - name: Set job controls + id: job_status + run: .github/jobs/set_job_controls.sh + env: + commit_msg: ${{ github.event.head_commit.message }} + + outputs: + run_compile: ${{ steps.job_status.outputs.run_compile }} + run_push: ${{ steps.job_status.outputs.run_push }} + run_unit_tests: ${{ steps.job_status.outputs.run_unit_tests }} + run_diff: ${{ steps.job_status.outputs.run_diff }} + run_update_truth: ${{ steps.job_status.outputs.run_update_truth }} + met_base_image: ${{ steps.job_status.outputs.met_base_image }} + branch_name: ${{ steps.job_status.outputs.branch_name }} + truth_data_version: ${{ steps.job_status.outputs.truth_data_version }} + input_data_version: ${{ steps.job_status.outputs.input_data_version }} + + compile: + name: Compile MET + runs-on: ubuntu-latest + needs: job_control + if: ${{ needs.job_control.outputs.run_compile == 'true' }} + steps: + - uses: actions/checkout@v2 + + - name: Create directories to store output + run: mkdir -p ${RUNNER_WORKSPACE}/logs + + - name: Compile MET in Docker + run: .github/jobs/build_docker_image.sh + env: + SOURCE_BRANCH: ${{ needs.job_control.outputs.branch_name }} + MET_BASE_IMAGE: ${{ needs.job_control.outputs.met_base_image }} + + - name: Copy Docker build log into logs directory + if: always() + run: cp ${GITHUB_WORKSPACE}/docker_build.log ${RUNNER_WORKSPACE}/logs/ + + - name: Push Docker Image + run: .github/jobs/push_docker_image.sh + if: ${{ needs.job_control.outputs.run_push == 'true' }} + env: + SOURCE_BRANCH: ${{ needs.job_control.outputs.branch_name }} + DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }} + DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }} + + - name: Upload logs as artifact + if: always() + uses: actions/upload-artifact@v2 + with: + name: logs + path: ${{ runner.workspace }}/logs + if-no-files-found: ignore + + unit_tests_1a: + name: MET Unit Tests 1a + runs-on: ubuntu-latest + needs: [job_control, compile] + if: ${{ needs.job_control.outputs.run_unit_tests == 'true' }} + strategy: + matrix: + tests: + - 'ascii2nc' + - 'pb2nc madis2nc pcp_combine' + fail-fast: false + steps: + - uses: actions/checkout@v2 + + - name: Run Unit Tests in Docker + run: .github/jobs/run_unit_docker.sh + env: + SOURCE_BRANCH: ${{ needs.job_control.outputs.branch_name }} + TESTS: ${{ matrix.tests }} + INPUT_DATA_VERSION: ${{ needs.job_control.outputs.input_data_version }} + + - name: Upload output as artifact + uses: actions/upload-artifact@v2 + with: + name: unit_tests_1a + path: ${{ runner.workspace }}/output + + - name: Upload logs as artifact + if: always() + uses: actions/upload-artifact@v2 + with: + name: logs + path: ${{ runner.workspace }}/logs + if-no-files-found: ignore + + unit_tests_1b: + name: MET Unit Tests 1b + runs-on: ubuntu-latest + needs: [job_control, compile] + if: ${{ needs.job_control.outputs.run_unit_tests == 'true' }} + strategy: + matrix: + tests: + - 'ascii2nc_indy pb2nc_indy tc_dland tc_pairs tc_stat plot_tc tc_rmw rmw_analysis tc_gen' + - 'met_test_scripts mode_graphics mtd regrid airnow gsi_tools netcdf modis series_analysis gen_ens_prod wwmca_regrid gen_vx_mask grid_weight interp_shape grid_diag grib_tables lidar2nc shift_data_plane trmm2nc aeronet wwmca_plot ioda2nc gaussian' + fail-fast: false + steps: + - uses: actions/checkout@v2 + + - name: Run Unit Tests in Docker + run: .github/jobs/run_unit_docker.sh + env: + SOURCE_BRANCH: ${{ needs.job_control.outputs.branch_name }} + TESTS: ${{ matrix.tests }} + INPUT_DATA_VERSION: ${{ needs.job_control.outputs.input_data_version }} + + - name: Upload output as artifact + uses: actions/upload-artifact@v2 + with: + name: unit_tests_1b + path: ${{ runner.workspace }}/output + + - name: Upload logs as artifact + if: always() + uses: actions/upload-artifact@v2 + with: + name: logs + path: ${{ runner.workspace }}/logs + if-no-files-found: ignore + + unit_tests_ref_config_leads: + name: MET Unit Tests ref_config leads + runs-on: ubuntu-latest + needs: [job_control, compile] + if: ${{ needs.job_control.outputs.run_unit_tests == 'true' }} + strategy: + matrix: + tests: + - 'ref_config_lead_00 ref_config_lead_12' + - 'ref_config_lead_24 ref_config_lead_48' + - 'ref_config_lead_36' + fail-fast: false + steps: + - uses: actions/checkout@v2 + + - name: Run Unit Tests in Docker + run: .github/jobs/run_unit_docker.sh + env: + SOURCE_BRANCH: ${{ needs.job_control.outputs.branch_name }} + TESTS: ${{ matrix.tests }} + INPUT_DATA_VERSION: ${{ needs.job_control.outputs.input_data_version }} + + - name: Upload output as artifact + uses: actions/upload-artifact@v2 + with: + name: unit_tests_ref_config_leads + path: ${{ runner.workspace }}/output + + - name: Upload logs as artifact + if: always() + uses: actions/upload-artifact@v2 + with: + name: logs + path: ${{ runner.workspace }}/logs + if-no-files-found: ignore + + unit_tests_ref_config: + name: MET Unit Tests ref_config + runs-on: ubuntu-latest + needs: [job_control, unit_tests_ref_config_leads] + if: ${{ needs.job_control.outputs.run_unit_tests == 'true' }} + strategy: + matrix: + tests: + - 'ref_config' + fail-fast: false + steps: + - uses: actions/checkout@v2 + + - name: Download ref_config_leads output from artifact + uses: actions/download-artifact@v2 + with: + name: unit_tests_ref_config_leads + path: ${{ runner.workspace }}/output + + - name: Run Unit Tests in Docker + run: .github/jobs/run_unit_docker.sh + env: + SOURCE_BRANCH: ${{ needs.job_control.outputs.branch_name }} + TESTS: ${{ matrix.tests }} + INPUT_DATA_VERSION: 'none' + + - name: Upload output as artifact + uses: actions/upload-artifact@v2 + with: + name: unit_tests_ref_config + path: ${{ runner.workspace }}/output + + - name: Upload logs as artifact + if: always() + uses: actions/upload-artifact@v2 + with: + name: logs + path: ${{ runner.workspace }}/logs + if-no-files-found: ignore + + unit_tests_2a: + name: MET Unit Tests 2a + runs-on: ubuntu-latest + needs: [job_control, unit_tests_1a] + if: ${{ needs.job_control.outputs.run_unit_tests == 'true' }} + strategy: + matrix: + tests: + - 'point_stat stat_analysis_ps' + - 'grid_stat stat_analysis_gs' + - 'wavelet_stat stat_analysis_ws' + - 'ensemble_stat stat_analysis_es' + fail-fast: false + steps: + - uses: actions/checkout@v2 + + - name: Download 1a output from artifact + uses: actions/download-artifact@v2 + with: + name: unit_tests_1a + path: ${{ runner.workspace }}/output + + - name: Run Unit Tests in Docker + run: .github/jobs/run_unit_docker.sh + env: + SOURCE_BRANCH: ${{ needs.job_control.outputs.branch_name }} + TESTS: ${{ matrix.tests }} + INPUT_DATA_VERSION: ${{ needs.job_control.outputs.input_data_version }} + + - name: Upload output as artifact + uses: actions/upload-artifact@v2 + with: + name: unit_tests_2a + path: ${{ runner.workspace }}/output + + - name: Upload logs as artifact + if: always() + uses: actions/upload-artifact@v2 + with: + name: logs + path: ${{ runner.workspace }}/logs + if-no-files-found: ignore + + unit_tests_2b: + name: MET Unit Tests 2b + runs-on: ubuntu-latest + needs: [job_control, unit_tests_1a] + if: ${{ needs.job_control.outputs.run_unit_tests == 'true' }} + strategy: + matrix: + tests: + - 'climatology_1.0deg' + - 'climatology_1.5deg' + - 'climatology_2.5deg' + - 'python point2grid plot_data_plane mode mode_analysis perc_thresh hira plot_point_obs quality_filter obs_summary duplicate_flag' + fail-fast: false + steps: + - uses: actions/checkout@v2 + + - name: Download 1a output from artifact + uses: actions/download-artifact@v2 + with: + name: unit_tests_1a + path: ${{ runner.workspace }}/output + + - name: Run Unit Tests in Docker + run: .github/jobs/run_unit_docker.sh + env: + SOURCE_BRANCH: ${{ needs.job_control.outputs.branch_name }} + TESTS: ${{ matrix.tests }} + INPUT_DATA_VERSION: ${{ needs.job_control.outputs.input_data_version }} + + - name: Upload output as artifact + uses: actions/upload-artifact@v2 + with: + name: unit_tests_2b + path: ${{ runner.workspace }}/output + + - name: Upload logs as artifact + if: always() + uses: actions/upload-artifact@v2 + with: + name: logs + path: ${{ runner.workspace }}/logs + if-no-files-found: ignore + + run_diffs: + name: Check for Differences + runs-on: ubuntu-latest + needs: [job_control, unit_tests_1b, unit_tests_2a, unit_tests_2b, unit_tests_ref_config] + if: ${{ needs.job_control.outputs.run_diff == 'true' }} + steps: + - name: Download data from previous jobs + uses: actions/download-artifact@v2 + + - name: Copy test output into single directory + run: | + mkdir ${RUNNER_WORKSPACE}/output + cp -r unit_tests_*/* ${RUNNER_WORKSPACE}/output/ + + - uses: actions/checkout@v2 + + - name: Run Diff Tests in Docker + run: .github/jobs/run_diff_docker.sh + env: + SOURCE_BRANCH: ${{ needs.job_control.outputs.branch_name }} + RUN_UPDATE_TRUTH: ${{ needs.job_control.outputs.run_update_truth }} + TRUTH_DATA_VERSION: ${{ needs.job_control.outputs.truth_data_version }} + + - name: Upload diff files as artifact + if: always() + uses: actions/upload-artifact@v2 + with: + name: diff + path: ${{ runner.workspace }}/diff + if-no-files-found: ignore + + - name: Upload logs as artifact + if: always() + uses: actions/upload-artifact@v2 + with: + name: logs + path: ${{ runner.workspace }}/logs + if-no-files-found: ignore + + update_truth: + name: Update Truth Data + runs-on: ubuntu-latest + needs: [job_control, unit_tests_1b, unit_tests_2a, unit_tests_2b, unit_tests_ref_config] + if: ${{ needs.job_control.outputs.run_update_truth == 'true' }} + steps: + - uses: actions/checkout@v2 + + - name: Download data from previous jobs + uses: actions/download-artifact@v2 + + - name: Copy test output into single directory + run: | + mkdir ${RUNNER_WORKSPACE}/met_test_truth + cp -r unit_tests_*/* ${RUNNER_WORKSPACE}/met_test_truth/ + + - name: Create Docker Data Volume + run: .github/jobs/create_docker_truth.sh + env: + TRUTH_DATA_VERSION: ${{ needs.job_control.outputs.truth_data_version }} + DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }} + DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }} diff --git a/.gitignore b/.gitignore index f2060b9284..bbd74b866f 100644 --- a/.gitignore +++ b/.gitignore @@ -3,3 +3,6 @@ met/docs/_build/ # tilda files generated by emacs *~ + +# log file created in Docker image +make_install.log diff --git a/met/scripts/python/read_met_point_obs.py b/met/scripts/python/read_met_point_obs.py index 15667541ec..60f2c62065 100755 --- a/met/scripts/python/read_met_point_obs.py +++ b/met/scripts/python/read_met_point_obs.py @@ -8,7 +8,7 @@ import os import sys -import time +from datetime import datetime import numpy as np import netCDF4 as nc @@ -93,8 +93,7 @@ def get_string_array(dataset, var_name): return nc_tools.get_ncbyte_array_to_str(nc_var) if nc_var else [] -perf_start_time = time.time() -perf_start_counter = time.perf_counter_ns() +start_time = datetime.now() point_obs_data = None if len(sys.argv) == 1: @@ -117,9 +116,6 @@ def get_string_array(dataset, var_name): if DO_PRINT_DATA: met_point_obs.print_point_data(met_point_data) -perf_end_time = time.time() -perf_end_counter = time.perf_counter_ns() -perf_duration = perf_end_time - perf_start_time -perf_duration_counter = (perf_end_counter - perf_start_counter) / 1000000000 +run_time = datetime.now() - start_time -print('Done python script {s} Took walltime: {t1} & perf: {t2} seconds'.format(s=sys.argv[0], t1=perf_duration, t2=perf_duration_counter)) +print('Done python script {s} took {t}'.format(s=sys.argv[0], t=run_time)) diff --git a/scripts/docker/Dockerfile b/scripts/docker/Dockerfile index 330e0bbadd..ddeb6b0a81 100644 --- a/scripts/docker/Dockerfile +++ b/scripts/docker/Dockerfile @@ -1,4 +1,6 @@ -FROM centos:7 +ARG MET_BASE_IMAGE=minimum + +FROM dtcenter/met-base:${MET_BASE_IMAGE} MAINTAINER John Halley Gotway # @@ -17,163 +19,21 @@ RUN if [ "x${SOURCE_BRANCH}" = "x" ]; then \ fi ENV MET_GIT_NAME ${SOURCE_BRANCH} +ENV MET_REPO_DIR /met/MET-${MET_GIT_NAME} ENV MET_GIT_URL https://github.com/dtcenter/MET ENV MET_DEVELOPMENT true -# -# Define the compilers. -# -ENV CC /usr/bin/gcc -ENV CXX /usr/bin/g++ -ENV FC /usr/bin/gfortran -ENV F77 /usr/bin/gfortran - -# -# Define package URL's. -# -ENV HDF4_URL http://www.hdfgroup.org/ftp/HDF/releases/HDF4.2r3/src/HDF4.2r3.tar.gz -ENV HDFEOS_URL https://dtcenter.ucar.edu/dfiles/code/METplus/MET/docker_data/HDF-EOS2.16v1.00.tar.Z - -ENV NETCDF4C_URL https://github.com/Unidata/netcdf-c/archive/v4.4.1.1.zip -ENV NETCDF4CXX_URL https://github.com/Unidata/netcdf-cxx4/archive/v4.3.0.tar.gz - -ENV BUFRLIB_URL https://dtcenter.ucar.edu/dfiles/code/METplus/MET/docker_data/BUFRLIB_v10-2-3.tar -ENV GSFONT_URL https://dtcenter.ucar.edu/dfiles/code/METplus/MET/docker_data/ghostscript-fonts-std-8.11.tar.gz - -# -# Install the required packages. -# -RUN yum -y update \ - && yum -y install file gcc gcc-gfortran gcc-c++ glibc.i686 libgcc.i686 \ - libpng-devel jasper jasper-devel zlib zlib-devel \ - cairo-devel freetype-devel epel-release \ - hostname m4 make tar tcsh ksh time wget which \ - flex flex-devel bison bison-devel unzip \ - && yum -y install git g2clib-devel hdf5-devel.x86_64 gsl-devel \ - && yum -y install gv ncview wgrib wgrib2 ImageMagick ps2pdf \ - && yum -y install python3 python3-devel python3-pip \ - && pip3 install --upgrade pip \ - && python3 -m pip install numpy xarray netCDF4 - # # Set the working directory. # WORKDIR /met -# -# Setup the environment for interactive bash/csh container shells. -# -RUN echo export MET_BASE=/usr/local/share/met >> /etc/bashrc \ - && echo setenv MET_BASE /usr/local/share/met >> /etc/csh.cshrc \ - && echo export MET_FONT_DIR=/usr/local/share/met/fonts >> /etc/bashrc \ - && echo setenv MET_FONT_DIR /usr/local/share/met/fonts >> /etc/csh.cshrc \ - && echo export RSCRIPTS_BASE=/usr/local/share/met/Rscripts >> /etc/bashrc \ - && echo setenv RSCRIPTS_BASE /usr/local/share/met/Rscripts >> /etc/csh.cshrc \ - && echo export LD_LIBRARY_PATH=/usr/local/lib >> /etc/bashrc \ - && echo setenv LD_LIBRARY_PATH /usr/local/lib >> /etc/csh.cshrc -ENV LD_LIBRARY_PATH /usr/local/lib - -# -# Download and install BUFRLIB. -# -RUN mkdir -p /met/logs \ - && mkdir -p /met/external_libs/BUFRLIB \ - && cd /met/external_libs/BUFRLIB \ - && echo "Downloading BUFRLIB from ${BUFRLIB_URL}" \ - && curl -SL ${BUFRLIB_URL} | tar xC /met/external_libs/BUFRLIB \ - && cat preproc.sh | sed 's/cpp /cpp -traditional-cpp /g' > preproc_patch.sh \ - && chmod +x preproc_patch.sh \ - && LOG_FILE=/met/logs/BUFRLIB_build.log \ - && echo "Compiling BUFRLIB and writing log file ${LOG_FILE}" \ - && ./preproc_patch.sh *.F > ${LOG_FILE} \ - && ${CC} -c -DUNDERSCORE *.c >> ${LOG_FILE} \ - && ${FC} -c -fno-second-underscore *.f >> ${LOG_FILE} \ - && ar crv libbufr.a *.o >> ${LOG_FILE} \ - && rm -f /usr/local/lib/libbufr.a \ - && cp *.a /usr/local/lib \ - && cd /met/external_libs \ - && rm -rf BUFRLIB - -# -# Download and install NetCDF4 (C and C++). -# -RUN mkdir -p /met/external_libs/netcdf \ - && cd /met/external_libs/netcdf \ - && echo "Downloading netcdf-c-4.4.1.1 from ${NETCDF4C_URL}" \ - && wget ${NETCDF4C_URL} \ - && unzip v4.4.1.1.zip \ - && cd netcdf-c-4.4.1.1 \ - && LOG_FILE=/met/logs/netcdf-c-4.4.1.1_configure.log \ - && echo "Configuring netcdf-c-4.4.1.1 and writing log file ${LOG_FILE}" \ - && ./configure > ${LOG_FILE} \ - && LOG_FILE=/met/logs/netcdf-c-4.4.1.1_make_install.log \ - && echo "Compiling netcdf-c-4.4.1.1 and writing log file ${LOG_FILE}" \ - && make install > ${LOG_FILE} \ - && echo "Downloading from ${NETCDF4CXX_URL}" \ - && cd /met/external_libs/netcdf \ - && wget ${NETCDF4CXX_URL} \ - && tar -xzf v4.3.0.tar.gz \ - && cd netcdf-cxx4-4.3.0 \ - && LOG_FILE=/met/logs/netcdf-cxx4-4.3.0_configure.log \ - && echo "Configuring netcdf-cxx4-4.3.0 and writing log file ${LOG_FILE}" \ - && ./configure > ${LOG_FILE} \ - && LOG_FILE=/met/logs/netcdf-cxx4-4.3.0_make_install.log \ - && echo "Compiling netcdf-cxx4-4.3.0 and writing log file ${LOG_FILE}" \ - && make install > ${LOG_FILE} \ - && cd /met/external_libs \ - && rm -rf netcdf - -# -# Download and install HDF4 and HDFEOS. -# -RUN echo "Downloading HDF4.2r3 from ${HDF4_URL}" \ - && curl -SL ${HDF4_URL} | tar zxC /met/external_libs \ - && cd /met/external_libs/HDF4.2r3 \ - && LOG_FILE=/met/logs/HDF4.2r3_configure.log \ - && echo "Configuring HDF4.2r3 and writing log file ${LOG_FILE}" \ - && ./configure --prefix=/usr/local/hdf --disable-netcdf > ${LOG_FILE} \ - && cat mfhdf/hdiff/Makefile | sed 's/LIBS = -ljpeg -lz/LIBS = -ljpeg -lz -lm/g' > Makefile_NEW \ - && mv -f Makefile_NEW mfhdf/hdiff/Makefile \ - && LOG_FILE=/met/logs/HDF4.2r3_make_install.log \ - && echo "Compiling HDF4.2r3 and writing log file ${LOG_FILE}" \ - && make install > ${LOG_FILE} \ - && echo "Downloading hdfeos from ${HDFEOS_URL}" \ - && curl -SL ${HDFEOS_URL} | tar zxC /met/external_libs \ - && cd /met/external_libs/hdfeos \ - && LOG_FILE=/met/logs/hdfeos_configure.log \ - && echo "Configuring hdfeos and writing log file ${LOG_FILE}" \ - && ./configure --prefix=/usr/local/hdfeos --with-hdf4=/usr/local/hdf CC=/usr/local/hdf/bin/h4cc > ${LOG_FILE} \ - && LOG_FILE=/met/logs/hdfeos_make_install.log \ - && echo "Compiling hdfeos and writing log file ${LOG_FILE}" \ - && make install > ${LOG_FILE} \ - && mkdir /usr/local/hdfeos/include \ - && cp include/*.h /usr/local/hdfeos/include/. \ - && cd /met/external_libs \ - && rm -rf HDF4.2r3 hdfeos - # # Download and install MET and GhostScript fonts. # Delete the MET source code for tagged releases matching "v"*. # RUN echo "Checking out MET ${MET_GIT_NAME} from ${MET_GIT_URL}" \ - && git clone ${MET_GIT_URL} /met/MET-${MET_GIT_NAME} \ - && cd /met/MET-${MET_GIT_NAME}/met \ + && git clone ${MET_GIT_URL} ${MET_REPO_DIR} \ + && cd ${MET_REPO_DIR}/met \ && git checkout ${MET_GIT_NAME} \ - && LOG_FILE=/met/logs/MET-${MET_GIT_NAME}_configure.log \ - && echo "Running bootstrap" \ - && ./bootstrap \ - && echo "Configuring MET ${MET_GIT_NAME} and writing log file ${LOG_FILE}" \ - && ./configure --enable-grib2 --enable-mode_graphics --enable-modis --enable-lidar2nc --enable-python \ - MET_HDF=/usr/local/hdf MET_HDFEOS=/usr/local/hdfeos \ - MET_FREETYPEINC=/usr/include/freetype2 MET_FREETYPELIB=/usr/lib \ - MET_CAIROINC=/usr/include/cairo MET_CAIROLIB=/usr/lib \ - MET_PYTHON_CC='-I/usr/include/python3.6m' MET_PYTHON_LD='-lpython3.6m' > ${LOG_FILE} \ - && LOG_FILE=/met/MET-${MET_GIT_NAME}/make_install.log \ - && echo "Compiling MET ${MET_GIT_NAME} and writing log file ${LOG_FILE}" \ - && make install > ${LOG_FILE} \ - && LOG_FILE=/met/logs/MET-${MET_GIT_NAME}_make_test.log \ - && echo "Testing MET ${MET_GIT_NAME} and writing log file ${LOG_FILE}" \ - && make test > ${LOG_FILE} 2>&1 \ - && if [[ $MET_GIT_NAME == "v"* ]]; then cd /met; rm -rf MET-*; fi \ - && echo "Downloading GhostScript fonts from ${GSFONT_URL}" \ - && curl -SL ${GSFONT_URL} | tar zxC /usr/local/share/met + && ../scripts/docker/build_met_docker.sh diff --git a/scripts/docker/Dockerfile.copy b/scripts/docker/Dockerfile.copy new file mode 100644 index 0000000000..18a71e4185 --- /dev/null +++ b/scripts/docker/Dockerfile.copy @@ -0,0 +1,46 @@ +ARG MET_BASE_IMAGE=minimum + +FROM dtcenter/met-base:${MET_BASE_IMAGE} +MAINTAINER John Halley Gotway + +# +# This Dockerfile checks out MET from GitHub and compiles the specified branch or tag from source. +# +ARG SOURCE_BRANCH + +# +# SOURCE_BRANCH is not defined when built via Docker Hub. +# +RUN if [ "x${SOURCE_BRANCH}" = "x" ]; then \ + echo "ERROR: SOURCE_BRANCH undefined! Rebuild with \"--build-arg SOURCE_BRANCH={branch name}\""; \ + exit 1; \ + else \ + echo "Build Argument SOURCE_BRANCH=${SOURCE_BRANCH}"; \ + fi + +ENV MET_GIT_NAME ${SOURCE_BRANCH} +ENV MET_REPO_DIR /met/MET-${MET_GIT_NAME} +ENV MET_GIT_URL https://github.com/dtcenter/MET +ENV MET_DEVELOPMENT true + +# +# Set the working directory. +# +WORKDIR /met + +# +# Download and install MET and GhostScript fonts. +# Delete the MET source code for tagged releases matching "v"*. +# +RUN echo "Copying MET into ${MET_REPO_DIR}" \ + && mkdir -p ${MET_REPO_DIR} + +COPY . ${MET_REPO_DIR} + +RUN if [ ! -e "${MET_REPO_DIR}/met/configure.ac" ]; then \ + echo "ERROR: docker build must be run from the MET directory"; \ + exit 1; \ + fi + +RUN cd ${MET_REPO_DIR}/met \ + && ../scripts/docker/build_met_docker.sh diff --git a/scripts/docker/Dockerfile.minimum b/scripts/docker/Dockerfile.minimum new file mode 100644 index 0000000000..028871196b --- /dev/null +++ b/scripts/docker/Dockerfile.minimum @@ -0,0 +1,138 @@ +FROM centos:7 +MAINTAINER John Halley Gotway + +# +# Define the compilers. +# +ENV CC /usr/bin/gcc +ENV CXX /usr/bin/g++ +ENV FC /usr/bin/gfortran +ENV F77 /usr/bin/gfortran + +# +# Define package URL's. +# +ENV HDF4_URL http://www.hdfgroup.org/ftp/HDF/releases/HDF4.2r3/src/HDF4.2r3.tar.gz +ENV HDFEOS_URL https://dtcenter.ucar.edu/dfiles/code/METplus/MET/docker_data/HDF-EOS2.16v1.00.tar.Z + +ENV NETCDF4C_URL https://github.com/Unidata/netcdf-c/archive/v4.4.1.1.zip +ENV NETCDF4CXX_URL https://github.com/Unidata/netcdf-cxx4/archive/v4.3.0.tar.gz + +ENV BUFRLIB_URL https://dtcenter.ucar.edu/dfiles/code/METplus/MET/docker_data/BUFRLIB_v10-2-3.tar +ENV GSFONT_URL https://dtcenter.ucar.edu/dfiles/code/METplus/MET/docker_data/ghostscript-fonts-std-8.11.tar.gz + +# +# Install the required packages. +# +RUN yum -y update \ + && yum -y install file gcc gcc-gfortran gcc-c++ glibc.i686 libgcc.i686 \ + libpng-devel jasper jasper-devel zlib zlib-devel \ + cairo-devel freetype-devel epel-release \ + hostname m4 make tar tcsh ksh time wget which \ + flex flex-devel bison bison-devel unzip \ + && yum -y install git g2clib-devel hdf5-devel.x86_64 gsl-devel \ + && yum -y install gv ncview wgrib wgrib2 ImageMagick ps2pdf \ + && yum -y install python3 python3-devel python3-pip \ + && pip3 install --upgrade pip \ + && python3 -m pip install numpy xarray netCDF4 + +# +# Set the working directory. +# +WORKDIR /met + +# +# Setup the environment for interactive bash/csh container shells. +# +RUN echo export MET_BASE=/usr/local/share/met >> /etc/bashrc \ + && echo setenv MET_BASE /usr/local/share/met >> /etc/csh.cshrc \ + && echo export MET_FONT_DIR=/usr/local/share/met/fonts >> /etc/bashrc \ + && echo setenv MET_FONT_DIR /usr/local/share/met/fonts >> /etc/csh.cshrc \ + && echo export RSCRIPTS_BASE=/usr/local/share/met/Rscripts >> /etc/bashrc \ + && echo setenv RSCRIPTS_BASE /usr/local/share/met/Rscripts >> /etc/csh.cshrc \ + && echo export LD_LIBRARY_PATH=/usr/local/lib >> /etc/bashrc \ + && echo setenv LD_LIBRARY_PATH /usr/local/lib >> /etc/csh.cshrc +ENV LD_LIBRARY_PATH /usr/local/lib +ENV MET_FONT_DIR /usr/local/share/met/fonts + +# +# Download and install BUFRLIB. +# +RUN mkdir -p /met/logs \ + && mkdir -p /met/external_libs/BUFRLIB \ + && cd /met/external_libs/BUFRLIB \ + && echo "Downloading BUFRLIB from ${BUFRLIB_URL}" \ + && curl -SL ${BUFRLIB_URL} | tar xC /met/external_libs/BUFRLIB \ + && cat preproc.sh | sed 's/cpp /cpp -traditional-cpp /g' > preproc_patch.sh \ + && chmod +x preproc_patch.sh \ + && LOG_FILE=/met/logs/BUFRLIB_build.log \ + && echo "Compiling BUFRLIB and writing log file ${LOG_FILE}" \ + && ./preproc_patch.sh *.F > ${LOG_FILE} \ + && ${CC} -c -DUNDERSCORE *.c >> ${LOG_FILE} \ + && ${FC} -c -fno-second-underscore *.f >> ${LOG_FILE} \ + && ar crv libbufr.a *.o >> ${LOG_FILE} \ + && rm -f /usr/local/lib/libbufr.a \ + && cp *.a /usr/local/lib \ + && cd /met/external_libs \ + && rm -rf BUFRLIB + +# +# Download and install NetCDF4 (C and C++). +# +RUN mkdir -p /met/external_libs/netcdf \ + && cd /met/external_libs/netcdf \ + && echo "Downloading netcdf-c-4.4.1.1 from ${NETCDF4C_URL}" \ + && wget ${NETCDF4C_URL} \ + && unzip v4.4.1.1.zip \ + && cd netcdf-c-4.4.1.1 \ + && LOG_FILE=/met/logs/netcdf-c-4.4.1.1_configure.log \ + && echo "Configuring netcdf-c-4.4.1.1 and writing log file ${LOG_FILE}" \ + && ./configure > ${LOG_FILE} \ + && LOG_FILE=/met/logs/netcdf-c-4.4.1.1_make_install.log \ + && echo "Compiling netcdf-c-4.4.1.1 and writing log file ${LOG_FILE}" \ + && make install > ${LOG_FILE} \ + && echo "Downloading from ${NETCDF4CXX_URL}" \ + && cd /met/external_libs/netcdf \ + && wget ${NETCDF4CXX_URL} \ + && tar -xzf v4.3.0.tar.gz \ + && cd netcdf-cxx4-4.3.0 \ + && LOG_FILE=/met/logs/netcdf-cxx4-4.3.0_configure.log \ + && echo "Configuring netcdf-cxx4-4.3.0 and writing log file ${LOG_FILE}" \ + && ./configure > ${LOG_FILE} \ + && LOG_FILE=/met/logs/netcdf-cxx4-4.3.0_make_install.log \ + && echo "Compiling netcdf-cxx4-4.3.0 and writing log file ${LOG_FILE}" \ + && make install > ${LOG_FILE} \ + && cd /met/external_libs \ + && rm -rf netcdf + +# +# Download and install HDF4 and HDFEOS. +# +RUN echo "Downloading HDF4.2r3 from ${HDF4_URL}" \ + && curl -SL ${HDF4_URL} | tar zxC /met/external_libs \ + && cd /met/external_libs/HDF4.2r3 \ + && LOG_FILE=/met/logs/HDF4.2r3_configure.log \ + && echo "Configuring HDF4.2r3 and writing log file ${LOG_FILE}" \ + && ./configure --prefix=/usr/local/hdf --disable-netcdf > ${LOG_FILE} \ + && cat mfhdf/hdiff/Makefile | sed 's/LIBS = -ljpeg -lz/LIBS = -ljpeg -lz -lm/g' > Makefile_NEW \ + && mv -f Makefile_NEW mfhdf/hdiff/Makefile \ + && LOG_FILE=/met/logs/HDF4.2r3_make_install.log \ + && echo "Compiling HDF4.2r3 and writing log file ${LOG_FILE}" \ + && make install > ${LOG_FILE} \ + && echo "Downloading hdfeos from ${HDFEOS_URL}" \ + && curl -SL ${HDFEOS_URL} | tar zxC /met/external_libs \ + && cd /met/external_libs/hdfeos \ + && LOG_FILE=/met/logs/hdfeos_configure.log \ + && echo "Configuring hdfeos and writing log file ${LOG_FILE}" \ + && ./configure --prefix=/usr/local/hdfeos --with-hdf4=/usr/local/hdf CC=/usr/local/hdf/bin/h4cc > ${LOG_FILE} \ + && LOG_FILE=/met/logs/hdfeos_make_install.log \ + && echo "Compiling hdfeos and writing log file ${LOG_FILE}" \ + && make install > ${LOG_FILE} \ + && mkdir /usr/local/hdfeos/include \ + && cp include/*.h /usr/local/hdfeos/include/. \ + && cd /met/external_libs \ + && rm -rf HDF4.2r3 hdfeos + +RUN echo "Downloading GhostScript fonts from ${GSFONT_URL} into /usr/local/share/met" \ + && mkdir -p /usr/local/share/met \ + && curl -SL ${GSFONT_URL} | tar zxC /usr/local/share/met diff --git a/scripts/docker/Dockerfile.test b/scripts/docker/Dockerfile.test new file mode 100644 index 0000000000..bc7963cb91 --- /dev/null +++ b/scripts/docker/Dockerfile.test @@ -0,0 +1,26 @@ +ARG MET_BASE_IMAGE=minimum + +FROM dtcenter/met-base:${MET_BASE_IMAGE} +MAINTAINER John Halley Gotway + +# +# Set the working directory. +# +WORKDIR /met + +# +# Download and install MET and GhostScript fonts. +# Delete the MET source code for tagged releases matching "v"*. +# +RUN echo "Installing tools needed for running MET unit tests..." \ + && echo "Installing Perl XML Parser..." \ + && yum makecache \ + && yum -y install perl-XML-Parser \ + && echo "Installing R..." \ + && yum -y install R \ + && echo "Installing R ncdf4 1.19..." \ + && wget https://cran.r-project.org/src/contrib/ncdf4_1.19.tar.gz \ + && R CMD INSTALL ncdf4_1.19.tar.gz \ + && echo "Installing NCO (for ncdiff)..." \ + && yum -y install nco \ + && echo "Finished installing unit test tools" diff --git a/scripts/docker/README.md b/scripts/docker/README.md new file mode 100644 index 0000000000..80c6acaa08 --- /dev/null +++ b/scripts/docker/README.md @@ -0,0 +1,40 @@ +# How to Use Dockerfiles + +Run all of the Docker commands from the top-level directory of the MET repository + +## Build image with minimum requirements needed to build MET + +```docker build -t dtcenter/met-base:minimum -f scripts/docker/Dockerfile.minimum . +docker push dtcenter/met-base:minimum``` + +## Build image with requirements to build MET and run MET unit tests + +```docker build -t dtcenter/met-base:unit_test -f scripts/docker/Dockerfile.test . +docker push dtcenter/met-base:unit_test``` + +## Build MET from clone + +```docker build -t dtcenter/met:${TAG_NAME} --build-arg SOURCE_BRANCH=${BRANCH_NAME} scripts/docker +docker push dtcenter/met:${TAG_NAME}``` + +where: +* TAG_NAME is the name of the DockerHub tag to create +* BRANCH_NAME is the MET branch to checkout + +## Build MET from local source code with minimum requirements + +```docker build -t dtcenter/met:${TAG_NAME} --build-arg SOURCE_BRANCH=${BRANCH_NAME} -f scripts/docker/Dockerfile.copy . +docker push dtcenter/met:${TAG_NAME}``` + +where: +* TAG_NAME is the name of the DockerHub tag to create +* BRANCH_NAME is the identifier to use for $MET_GIT_NAME inside image + +## Build MET from local source code with unit test requirements + +```docker build -t dtcenter/met:${TAG_NAME} --build-arg SOURCE_BRANCH=${BRANCH_NAME} --build-arg MET_BASE_IMAGE=unit_test -f scripts/docker/Dockerfile.copy . +docker push dtcenter/met:${TAG_NAME}``` + +where: +* TAG_NAME is the name of the DockerHub tag to create +* BRANCH_NAME is the identifier to use for $MET_GIT_NAME inside image diff --git a/scripts/docker/build_met_docker.sh b/scripts/docker/build_met_docker.sh new file mode 100755 index 0000000000..fae3682113 --- /dev/null +++ b/scripts/docker/build_met_docker.sh @@ -0,0 +1,37 @@ +#! /bin/bash + +echo "Running script to build MET in Docker" + +LOG_FILE=/met/logs/MET-${MET_GIT_NAME}_configure.log + +echo "Running bootstrap" +./bootstrap + +echo "Configuring MET ${MET_GIT_NAME} and writing log file ${LOG_FILE}" +./configure --enable-grib2 --enable-mode_graphics --enable-modis --enable-lidar2nc --enable-python \ + MET_HDF=/usr/local/hdf MET_HDFEOS=/usr/local/hdfeos \ + MET_FREETYPEINC=/usr/include/freetype2 MET_FREETYPELIB=/usr/lib \ + MET_CAIROINC=/usr/include/cairo MET_CAIROLIB=/usr/lib \ + MET_PYTHON_CC='-I/usr/include/python3.6m' MET_PYTHON_LD='-lpython3.6m' > ${LOG_FILE} +if [ $? != 0 ]; then + exit 1 +fi + +LOG_FILE=/met/MET-${MET_GIT_NAME}/make_install.log +echo "Compiling MET ${MET_GIT_NAME} and writing log file ${LOG_FILE}" +make install > ${LOG_FILE} +if [ $? != 0 ]; then + exit 1 +fi + +LOG_FILE=/met/logs/MET-${MET_GIT_NAME}_make_test.log +echo "Testing MET ${MET_GIT_NAME} and writing log file ${LOG_FILE}" +make test > ${LOG_FILE} 2>&1 +if [ $? != 0 ]; then + exit 1 +fi + + +if [[ $MET_GIT_NAME == "v"* ]]; then + cd /met; rm -rf MET-*; +fi diff --git a/test/bin/unit_test.sh b/test/bin/unit_test.sh index fd1c7149cb..0c95bba7ac 100755 --- a/test/bin/unit_test.sh +++ b/test/bin/unit_test.sh @@ -35,20 +35,25 @@ PERL_UNIT=${MET_TEST_BASE}/perl/unit.pl # Unit test XML UNIT_XML="unit_ascii2nc.xml \ + unit_ascii2nc_indy.xml \ unit_madis2nc.xml \ unit_trmm2nc.xml \ unit_pb2nc.xml \ + unit_pb2nc_indy.xml \ unit_gen_vx_mask.xml \ unit_gen_ens_prod.xml \ unit_pcp_combine.xml \ unit_wwmca_regrid.xml \ unit_point_stat.xml \ + unit_stat_analysis_ps.xml \ unit_duplicate_flag.xml \ unit_obs_summary.xml \ unit_grid_stat.xml \ + unit_stat_analysis_gs.xml \ unit_wavelet_stat.xml \ + unit_stat_analysis_ws.xml \ unit_ensemble_stat.xml \ - unit_stat_analysis.xml \ + unit_stat_analysis_es.xml \ unit_mode.xml \ unit_mode_analysis.xml \ unit_plot_point_obs.xml \ @@ -64,6 +69,11 @@ UNIT_XML="unit_ascii2nc.xml \ unit_tc_gen.xml \ unit_met_test_scripts.xml \ unit_modis.xml \ + unit_ref_config_lead_00.xml \ + unit_ref_config_lead_12.xml \ + unit_ref_config_lead_24.xml \ + unit_ref_config_lead_36.xml \ + unit_ref_config_lead_48.xml \ unit_ref_config.xml \ unit_mode_graphics.xml \ unit_regrid.xml \ @@ -71,7 +81,9 @@ UNIT_XML="unit_ascii2nc.xml \ unit_aeronet.xml \ unit_shift_data_plane.xml \ unit_mtd.xml \ - unit_climatology.xml \ + unit_climatology_1.0deg.xml \ + unit_climatology_1.5deg.xml \ + unit_climatology_2.5deg.xml \ unit_grib_tables.xml \ unit_grid_weight.xml \ unit_netcdf.xml \ diff --git a/test/xml/unit_ascii2nc.xml b/test/xml/unit_ascii2nc.xml index 11c46a3ea5..feef57c65d 100644 --- a/test/xml/unit_ascii2nc.xml +++ b/test/xml/unit_ascii2nc.xml @@ -13,6 +13,8 @@ + + &TEST_DIR; @@ -30,18 +32,6 @@ - - &MET_BIN;/ascii2nc - \ - &DATA_DIR_OBS;/trmm/TRMM_3B42.007.accumulated_precipitation.22:30Z07Aug2012-10:30Z08Aug2012.G3.output.mtxt \ - &OUTPUT_DIR;/ascii2nc/trmm_2008080812_12hr.nc \ - -v 1 - - - &OUTPUT_DIR;/ascii2nc/trmm_2008080812_12hr.nc - - - &MET_BIN;/ascii2nc \ @@ -66,44 +56,6 @@ - - &MET_BIN;/ascii2nc - \ - &DATA_DIR_OBS;/ascii/qc_out_2012-09-07_00:00:00.GRM_P+FCST \ - &OUTPUT_DIR;/ascii2nc/qc_out_2012-09-07_00:00:00.GRM_P_FCST.nc \ - -v 1 - - - &OUTPUT_DIR;/ascii2nc/qc_out_2012-09-07_00:00:00.GRM_P_FCST.nc - - - - - &MET_BIN;/ascii2nc - \ - &DATA_DIR_OBS;/ascii/OBS:2015080700_bad_record \ - &OUTPUT_DIR;/ascii2nc/OBS_2015080700_bad_record.nc \ - -v 1 - - - &OUTPUT_DIR;/ascii2nc/OBS_2015080700_bad_record.nc - - - - - &MET_BIN;/ascii2nc - \ - &DATA_DIR_OBS;/ascii/surfrad_tbl12136.txt \ - &DATA_DIR_OBS;/ascii/surfrad_tbl12137.txt \ - &DATA_DIR_OBS;/ascii/surfrad_tbl12138.txt \ - &OUTPUT_DIR;/ascii2nc/surfrad_tbl12.nc \ - -v 1 - - - &OUTPUT_DIR;/ascii2nc/surfrad_tbl12.nc - - - &MET_BIN;/ascii2nc \ @@ -118,92 +70,6 @@ - - &MET_BIN;/ascii2nc - - BEG_TS 000000 - END_TS 235959 - STEP_TS 300 - WIDTH_TS 300 - - \ - &DATA_DIR_OBS;/surfrad/tbl12001.dat \ - &DATA_DIR_OBS;/surfrad/tbl12002.dat \ - &DATA_DIR_OBS;/surfrad/tbl12003.dat \ - -config &CONFIG_DIR;/Ascii2NcConfig.surfrad \ - &OUTPUT_DIR;/ascii2nc/surfrad_summary1.nc \ - -v 1 - - - &OUTPUT_DIR;/ascii2nc/surfrad_summary1.nc - - - - - &MET_BIN;/ascii2nc - - BEG_TS 03 - END_TS 20 - STEP_TS 300 - WIDTH_TS 600 - - \ - &DATA_DIR_OBS;/surfrad/tbl12001.dat \ - &DATA_DIR_OBS;/surfrad/tbl12002.dat \ - &DATA_DIR_OBS;/surfrad/tbl12003.dat \ - -config &CONFIG_DIR;/Ascii2NcConfig.surfrad \ - &OUTPUT_DIR;/ascii2nc/surfrad_summary2.nc \ - -v 1 - - - &OUTPUT_DIR;/ascii2nc/surfrad_summary2.nc - - - - - &MET_BIN;/ascii2nc - - BEG_TS 17 - END_TS 03 - STEP_TS 420 - WIDTH_TS 420 - - \ - &DATA_DIR_OBS;/surfrad/tbl12001.dat \ - &DATA_DIR_OBS;/surfrad/tbl12002.dat \ - &DATA_DIR_OBS;/surfrad/tbl12003.dat \ - -config &CONFIG_DIR;/Ascii2NcConfig.surfrad \ - &OUTPUT_DIR;/ascii2nc/surfrad_summary3.nc \ - -v 1 - - - &OUTPUT_DIR;/ascii2nc/surfrad_summary3.nc - - - - - - - &MET_BIN;/ascii2nc - - BEG_TS 17 - END_TS 03 - STEP_TS 420 - WIDTH_TS {beg=-420;end=0;} - - \ - &DATA_DIR_OBS;/surfrad/tbl12001.dat \ - &DATA_DIR_OBS;/surfrad/tbl12002.dat \ - &DATA_DIR_OBS;/surfrad/tbl12003.dat \ - -config &CONFIG_DIR;/Ascii2NcConfig.surfrad \ - &OUTPUT_DIR;/ascii2nc/surfrad_summary4.nc \ - -v 1 - - - &OUTPUT_DIR;/ascii2nc/surfrad_summary4.nc - - - &MET_BIN;/ascii2nc \ @@ -223,187 +89,6 @@ - - &MET_BIN;/ascii2nc - \ - &DATA_DIR_OBS;/insitu_ascii/20130827/edr_hourly.20130827.13.ascii \ - &DATA_DIR_OBS;/insitu_ascii/20130827/edr_hourly.20130827.14.ascii \ - &DATA_DIR_OBS;/insitu_ascii/20130827/edr_hourly.20130827.15.ascii \ - &DATA_DIR_OBS;/insitu_ascii/20130827/edr_hourly.20130827.16.ascii \ - &DATA_DIR_OBS;/insitu_ascii/20130827/edr_hourly.20130827.17.ascii \ - &DATA_DIR_OBS;/insitu_ascii/20130827/edr_hourly.20130827.18.ascii \ - &DATA_DIR_OBS;/insitu_ascii/20130827/edr_hourly.20130827.19.ascii \ - &DATA_DIR_OBS;/insitu_ascii/20130827/edr_hourly.20130827.20.ascii \ - &OUTPUT_DIR;/ascii2nc/edr_hourly.20130827.mask_sid.nc \ - -mask_sid "MY_STATIONS:N526UA,N567UA,N571UA,N594UA" \ - -v 1 - - - &OUTPUT_DIR;/ascii2nc/edr_hourly.20130827.mask_sid.nc - - - - - &MET_BIN;/ascii2nc - \ - &DATA_DIR_OBS;/insitu_ascii/20130827/edr_hourly.20130827.13.ascii \ - &DATA_DIR_OBS;/insitu_ascii/20130827/edr_hourly.20130827.14.ascii \ - &DATA_DIR_OBS;/insitu_ascii/20130827/edr_hourly.20130827.15.ascii \ - &DATA_DIR_OBS;/insitu_ascii/20130827/edr_hourly.20130827.16.ascii \ - &DATA_DIR_OBS;/insitu_ascii/20130827/edr_hourly.20130827.17.ascii \ - &DATA_DIR_OBS;/insitu_ascii/20130827/edr_hourly.20130827.18.ascii \ - &DATA_DIR_OBS;/insitu_ascii/20130827/edr_hourly.20130827.19.ascii \ - &DATA_DIR_OBS;/insitu_ascii/20130827/edr_hourly.20130827.20.ascii \ - &OUTPUT_DIR;/ascii2nc/edr_hourly.20130827.mask_grid_data.nc \ - -mask_grid &TEST_DIR;/data/mnc/test_grid_valid.nc \ - -v 1 - - - &OUTPUT_DIR;/ascii2nc/edr_hourly.20130827.mask_grid_data.nc - - - - - &MET_BIN;/ascii2nc - \ - &DATA_DIR_OBS;/insitu_ascii/20130827/edr_hourly.20130827.13.ascii \ - &DATA_DIR_OBS;/insitu_ascii/20130827/edr_hourly.20130827.14.ascii \ - &DATA_DIR_OBS;/insitu_ascii/20130827/edr_hourly.20130827.15.ascii \ - &DATA_DIR_OBS;/insitu_ascii/20130827/edr_hourly.20130827.16.ascii \ - &DATA_DIR_OBS;/insitu_ascii/20130827/edr_hourly.20130827.17.ascii \ - &DATA_DIR_OBS;/insitu_ascii/20130827/edr_hourly.20130827.18.ascii \ - &DATA_DIR_OBS;/insitu_ascii/20130827/edr_hourly.20130827.19.ascii \ - &DATA_DIR_OBS;/insitu_ascii/20130827/edr_hourly.20130827.20.ascii \ - &OUTPUT_DIR;/ascii2nc/edr_hourly.20130827.mask_named_grid.nc \ - -mask_grid G212 -v 1 - - - &OUTPUT_DIR;/ascii2nc/edr_hourly.20130827.mask_named_grid.nc - - - - - &MET_BIN;/ascii2nc - \ - &DATA_DIR_OBS;/trmm/TRMM_3B42.007.accumulated_precipitation.10:30Z09Apr2012.G3.output.mtxt \ - &OUTPUT_DIR;/ascii2nc/trmm_2012040912_3hr_mask_grid_dtc165.nc \ - -mask_grid DTC165 \ - -v 1 - - - &OUTPUT_DIR;/ascii2nc/trmm_2012040912_3hr_mask_grid_dtc165.nc - - - - - &MET_BIN;/ascii2nc - \ - &DATA_DIR_OBS;/trmm/TRMM_3B42.007.accumulated_precipitation.10:30Z09Apr2012.G3.output.mtxt \ - &OUTPUT_DIR;/ascii2nc/trmm_2012040912_3hr_mask_poly_lmv.nc \ - -mask_poly &MET_BASE;/poly/LMV.poly \ - -v 1 - - - &OUTPUT_DIR;/ascii2nc/trmm_2012040912_3hr_mask_poly_lmv.nc - - - - - &MET_BIN;/ascii2nc - \ - &DATA_DIR_OBS;/wwsis/clear_pvwatts_315510615_2006_pv_30MW_one_min_v3pt4.csv \ - &OUTPUT_DIR;/ascii2nc/clear_pvwatts_315510615_2006_pv_30MW_one_min_v3pt4.nc \ - -v 1 - - - &OUTPUT_DIR;/ascii2nc/clear_pvwatts_315510615_2006_pv_30MW_one_min_v3pt4.nc - - - - - &MET_BIN;/ascii2nc - \ - &DATA_DIR_OBS;/wwsis/clear_pvwatts_315510615_2006_pv_30MW_five_min_v3pt4.csv \ - &OUTPUT_DIR;/ascii2nc/clear_pvwatts_315510615_2006_pv_30MW_five_min_v3pt4.nc \ - -v 1 - - - &OUTPUT_DIR;/ascii2nc/clear_pvwatts_315510615_2006_pv_30MW_five_min_v3pt4.nc - - - - - &MET_BIN;/ascii2nc - \ - &DATA_DIR_OBS;/wwsis/clear_pvwatts_315510615_2006_pv_30MW_ten_min_v3pt4.csv \ - &OUTPUT_DIR;/ascii2nc/clear_pvwatts_315510615_2006_pv_30MW_ten_min_v3pt4.nc \ - -v 1 - - - &OUTPUT_DIR;/ascii2nc/clear_pvwatts_315510615_2006_pv_30MW_ten_min_v3pt4.nc - - - - - &MET_BIN;/ascii2nc - \ - &DATA_DIR_OBS;/wwsis/clear_pvwatts_315510615_2006_pv_30MW_sixty_min_v3pt4.csv \ - &OUTPUT_DIR;/ascii2nc/clear_pvwatts_315510615_2006_pv_30MW_sixty_min_v3pt4.nc \ - -v 1 - - - &OUTPUT_DIR;/ascii2nc/clear_pvwatts_315510615_2006_pv_30MW_sixty_min_v3pt4.nc - - - - - &MET_BIN;/ascii2nc - \ - &DATA_DIR_OBS;/wwsis/HA_pvwatts_315510615_2006_30MW_sixty_min_v3pt4.csv \ - &OUTPUT_DIR;/ascii2nc/HA_pvwatts_315510615_2006_30MW_sixty_min_v3pt4.nc \ - -v 1 - - - &OUTPUT_DIR;/ascii2nc/HA_pvwatts_315510615_2006_30MW_sixty_min_v3pt4.nc - - - - - &MET_BIN;/ascii2nc - \ - &DATA_DIR_OBS;/wwsis/pvwatts_315510615_2006_pv_30MW_one_min_v3pt4.csv \ - &OUTPUT_DIR;/ascii2nc/pvwatts_315510615_2006_pv_30MW_one_min_v3pt4.nc \ - -v 1 - - - &OUTPUT_DIR;/ascii2nc/pvwatts_315510615_2006_pv_30MW_one_min_v3pt4.nc - - - - - &MET_BIN;/ascii2nc - \ - &DATA_DIR_OBS;/wwsis/pvwatts_315510615_2006_pv_30MW_sixty_min_v3pt4.csv \ - &OUTPUT_DIR;/ascii2nc/pvwatts_315510615_2006_pv_30MW_sixty_min_v3pt4.nc \ - -v 1 - - - &OUTPUT_DIR;/ascii2nc/pvwatts_315510615_2006_pv_30MW_sixty_min_v3pt4.nc - - - - - &MET_BIN;/ascii2nc - \ - &DATA_DIR_OBS;/ascii/obs_test_name.txt \ - &OUTPUT_DIR;/ascii2nc/obs_test_name.nc \ - -v 1 - - - &OUTPUT_DIR;/ascii2nc/obs_test_name.nc - - - &MET_BIN;/ascii2nc \ diff --git a/test/xml/unit_ascii2nc_indy.xml b/test/xml/unit_ascii2nc_indy.xml new file mode 100644 index 0000000000..11ba012820 --- /dev/null +++ b/test/xml/unit_ascii2nc_indy.xml @@ -0,0 +1,340 @@ + + + + + + + + + + +]> + + + + + + + + &TEST_DIR; + true + + + &MET_BIN;/ascii2nc + \ + &DATA_DIR_OBS;/trmm/TRMM_3B42.007.accumulated_precipitation.22:30Z07Aug2012-10:30Z08Aug2012.G3.output.mtxt \ + &OUTPUT_DIR;/ascii2nc_indy/trmm_2008080812_12hr.nc \ + -v 1 + + + &OUTPUT_DIR;/ascii2nc_indy/trmm_2008080812_12hr.nc + + + + + &MET_BIN;/ascii2nc + \ + &DATA_DIR_OBS;/ascii/qc_out_2012-09-07_00:00:00.GRM_P+FCST \ + &OUTPUT_DIR;/ascii2nc_indy/qc_out_2012-09-07_00_00_00.GRM_P_FCST.nc \ + -v 1 + + + &OUTPUT_DIR;/ascii2nc_indy/qc_out_2012-09-07_00_00_00.GRM_P_FCST.nc + + + + + &MET_BIN;/ascii2nc + \ + &DATA_DIR_OBS;/ascii/OBS:2015080700_bad_record \ + &OUTPUT_DIR;/ascii2nc_indy/OBS_2015080700_bad_record.nc \ + -v 1 + + + &OUTPUT_DIR;/ascii2nc_indy/OBS_2015080700_bad_record.nc + + + + + &MET_BIN;/ascii2nc + \ + &DATA_DIR_OBS;/ascii/surfrad_tbl12136.txt \ + &DATA_DIR_OBS;/ascii/surfrad_tbl12137.txt \ + &DATA_DIR_OBS;/ascii/surfrad_tbl12138.txt \ + &OUTPUT_DIR;/ascii2nc_indy/surfrad_tbl12.nc \ + -v 1 + + + &OUTPUT_DIR;/ascii2nc_indy/surfrad_tbl12.nc + + + + + &MET_BIN;/ascii2nc + + BEG_TS 000000 + END_TS 235959 + STEP_TS 300 + WIDTH_TS 300 + + \ + &DATA_DIR_OBS;/surfrad/tbl12001.dat \ + &DATA_DIR_OBS;/surfrad/tbl12002.dat \ + &DATA_DIR_OBS;/surfrad/tbl12003.dat \ + -config &CONFIG_DIR;/Ascii2NcConfig.surfrad \ + &OUTPUT_DIR;/ascii2nc_indy/surfrad_summary1.nc \ + -v 1 + + + &OUTPUT_DIR;/ascii2nc_indy/surfrad_summary1.nc + + + + + &MET_BIN;/ascii2nc + + BEG_TS 03 + END_TS 20 + STEP_TS 300 + WIDTH_TS 600 + + \ + &DATA_DIR_OBS;/surfrad/tbl12001.dat \ + &DATA_DIR_OBS;/surfrad/tbl12002.dat \ + &DATA_DIR_OBS;/surfrad/tbl12003.dat \ + -config &CONFIG_DIR;/Ascii2NcConfig.surfrad \ + &OUTPUT_DIR;/ascii2nc_indy/surfrad_summary2.nc \ + -v 1 + + + &OUTPUT_DIR;/ascii2nc_indy/surfrad_summary2.nc + + + + + &MET_BIN;/ascii2nc + + BEG_TS 17 + END_TS 03 + STEP_TS 420 + WIDTH_TS 420 + + \ + &DATA_DIR_OBS;/surfrad/tbl12001.dat \ + &DATA_DIR_OBS;/surfrad/tbl12002.dat \ + &DATA_DIR_OBS;/surfrad/tbl12003.dat \ + -config &CONFIG_DIR;/Ascii2NcConfig.surfrad \ + &OUTPUT_DIR;/ascii2nc_indy/surfrad_summary3.nc \ + -v 1 + + + &OUTPUT_DIR;/ascii2nc_indy/surfrad_summary3.nc + + + + + + + &MET_BIN;/ascii2nc + + BEG_TS 17 + END_TS 03 + STEP_TS 420 + WIDTH_TS {beg=-420;end=0;} + + \ + &DATA_DIR_OBS;/surfrad/tbl12001.dat \ + &DATA_DIR_OBS;/surfrad/tbl12002.dat \ + &DATA_DIR_OBS;/surfrad/tbl12003.dat \ + -config &CONFIG_DIR;/Ascii2NcConfig.surfrad \ + &OUTPUT_DIR;/ascii2nc_indy/surfrad_summary4.nc \ + -v 1 + + + &OUTPUT_DIR;/ascii2nc_indy/surfrad_summary4.nc + + + + + &MET_BIN;/ascii2nc + \ + &DATA_DIR_OBS;/insitu_ascii/20130827/edr_hourly.20130827.13.ascii \ + &DATA_DIR_OBS;/insitu_ascii/20130827/edr_hourly.20130827.14.ascii \ + &DATA_DIR_OBS;/insitu_ascii/20130827/edr_hourly.20130827.15.ascii \ + &DATA_DIR_OBS;/insitu_ascii/20130827/edr_hourly.20130827.16.ascii \ + &DATA_DIR_OBS;/insitu_ascii/20130827/edr_hourly.20130827.17.ascii \ + &DATA_DIR_OBS;/insitu_ascii/20130827/edr_hourly.20130827.18.ascii \ + &DATA_DIR_OBS;/insitu_ascii/20130827/edr_hourly.20130827.19.ascii \ + &DATA_DIR_OBS;/insitu_ascii/20130827/edr_hourly.20130827.20.ascii \ + &OUTPUT_DIR;/ascii2nc_indy/edr_hourly.20130827.mask_sid.nc \ + -mask_sid "MY_STATIONS:N526UA,N567UA,N571UA,N594UA" \ + -v 1 + + + &OUTPUT_DIR;/ascii2nc_indy/edr_hourly.20130827.mask_sid.nc + + + + + &MET_BIN;/ascii2nc + \ + &DATA_DIR_OBS;/insitu_ascii/20130827/edr_hourly.20130827.13.ascii \ + &DATA_DIR_OBS;/insitu_ascii/20130827/edr_hourly.20130827.14.ascii \ + &DATA_DIR_OBS;/insitu_ascii/20130827/edr_hourly.20130827.15.ascii \ + &DATA_DIR_OBS;/insitu_ascii/20130827/edr_hourly.20130827.16.ascii \ + &DATA_DIR_OBS;/insitu_ascii/20130827/edr_hourly.20130827.17.ascii \ + &DATA_DIR_OBS;/insitu_ascii/20130827/edr_hourly.20130827.18.ascii \ + &DATA_DIR_OBS;/insitu_ascii/20130827/edr_hourly.20130827.19.ascii \ + &DATA_DIR_OBS;/insitu_ascii/20130827/edr_hourly.20130827.20.ascii \ + &OUTPUT_DIR;/ascii2nc_indy/edr_hourly.20130827.mask_grid_data.nc \ + -mask_grid &TEST_DIR;/data/mnc/test_grid_valid.nc \ + -v 1 + + + &OUTPUT_DIR;/ascii2nc_indy/edr_hourly.20130827.mask_grid_data.nc + + + + + &MET_BIN;/ascii2nc + \ + &DATA_DIR_OBS;/insitu_ascii/20130827/edr_hourly.20130827.13.ascii \ + &DATA_DIR_OBS;/insitu_ascii/20130827/edr_hourly.20130827.14.ascii \ + &DATA_DIR_OBS;/insitu_ascii/20130827/edr_hourly.20130827.15.ascii \ + &DATA_DIR_OBS;/insitu_ascii/20130827/edr_hourly.20130827.16.ascii \ + &DATA_DIR_OBS;/insitu_ascii/20130827/edr_hourly.20130827.17.ascii \ + &DATA_DIR_OBS;/insitu_ascii/20130827/edr_hourly.20130827.18.ascii \ + &DATA_DIR_OBS;/insitu_ascii/20130827/edr_hourly.20130827.19.ascii \ + &DATA_DIR_OBS;/insitu_ascii/20130827/edr_hourly.20130827.20.ascii \ + &OUTPUT_DIR;/ascii2nc_indy/edr_hourly.20130827.mask_named_grid.nc \ + -mask_grid G212 -v 1 + + + &OUTPUT_DIR;/ascii2nc_indy/edr_hourly.20130827.mask_named_grid.nc + + + + + &MET_BIN;/ascii2nc + \ + &DATA_DIR_OBS;/trmm/TRMM_3B42.007.accumulated_precipitation.10:30Z09Apr2012.G3.output.mtxt \ + &OUTPUT_DIR;/ascii2nc_indy/trmm_2012040912_3hr_mask_grid_dtc165.nc \ + -mask_grid DTC165 \ + -v 1 + + + &OUTPUT_DIR;/ascii2nc_indy/trmm_2012040912_3hr_mask_grid_dtc165.nc + + + + + &MET_BIN;/ascii2nc + \ + &DATA_DIR_OBS;/trmm/TRMM_3B42.007.accumulated_precipitation.10:30Z09Apr2012.G3.output.mtxt \ + &OUTPUT_DIR;/ascii2nc_indy/trmm_2012040912_3hr_mask_poly_lmv.nc \ + -mask_poly &MET_BASE;/poly/LMV.poly \ + -v 1 + + + &OUTPUT_DIR;/ascii2nc_indy/trmm_2012040912_3hr_mask_poly_lmv.nc + + + + + &MET_BIN;/ascii2nc + \ + &DATA_DIR_OBS;/wwsis/clear_pvwatts_315510615_2006_pv_30MW_one_min_v3pt4.csv \ + &OUTPUT_DIR;/ascii2nc_indy/clear_pvwatts_315510615_2006_pv_30MW_one_min_v3pt4.nc \ + -v 1 + + + &OUTPUT_DIR;/ascii2nc_indy/clear_pvwatts_315510615_2006_pv_30MW_one_min_v3pt4.nc + + + + + &MET_BIN;/ascii2nc + \ + &DATA_DIR_OBS;/wwsis/clear_pvwatts_315510615_2006_pv_30MW_five_min_v3pt4.csv \ + &OUTPUT_DIR;/ascii2nc_indy/clear_pvwatts_315510615_2006_pv_30MW_five_min_v3pt4.nc \ + -v 1 + + + &OUTPUT_DIR;/ascii2nc_indy/clear_pvwatts_315510615_2006_pv_30MW_five_min_v3pt4.nc + + + + + &MET_BIN;/ascii2nc + \ + &DATA_DIR_OBS;/wwsis/clear_pvwatts_315510615_2006_pv_30MW_ten_min_v3pt4.csv \ + &OUTPUT_DIR;/ascii2nc_indy/clear_pvwatts_315510615_2006_pv_30MW_ten_min_v3pt4.nc \ + -v 1 + + + &OUTPUT_DIR;/ascii2nc_indy/clear_pvwatts_315510615_2006_pv_30MW_ten_min_v3pt4.nc + + + + + &MET_BIN;/ascii2nc + \ + &DATA_DIR_OBS;/wwsis/clear_pvwatts_315510615_2006_pv_30MW_sixty_min_v3pt4.csv \ + &OUTPUT_DIR;/ascii2nc_indy/clear_pvwatts_315510615_2006_pv_30MW_sixty_min_v3pt4.nc \ + -v 1 + + + &OUTPUT_DIR;/ascii2nc_indy/clear_pvwatts_315510615_2006_pv_30MW_sixty_min_v3pt4.nc + + + + + &MET_BIN;/ascii2nc + \ + &DATA_DIR_OBS;/wwsis/HA_pvwatts_315510615_2006_30MW_sixty_min_v3pt4.csv \ + &OUTPUT_DIR;/ascii2nc_indy/HA_pvwatts_315510615_2006_30MW_sixty_min_v3pt4.nc \ + -v 1 + + + &OUTPUT_DIR;/ascii2nc_indy/HA_pvwatts_315510615_2006_30MW_sixty_min_v3pt4.nc + + + + + &MET_BIN;/ascii2nc + \ + &DATA_DIR_OBS;/wwsis/pvwatts_315510615_2006_pv_30MW_one_min_v3pt4.csv \ + &OUTPUT_DIR;/ascii2nc_indy/pvwatts_315510615_2006_pv_30MW_one_min_v3pt4.nc \ + -v 1 + + + &OUTPUT_DIR;/ascii2nc_indy/pvwatts_315510615_2006_pv_30MW_one_min_v3pt4.nc + + + + + &MET_BIN;/ascii2nc + \ + &DATA_DIR_OBS;/wwsis/pvwatts_315510615_2006_pv_30MW_sixty_min_v3pt4.csv \ + &OUTPUT_DIR;/ascii2nc_indy/pvwatts_315510615_2006_pv_30MW_sixty_min_v3pt4.nc \ + -v 1 + + + &OUTPUT_DIR;/ascii2nc_indy/pvwatts_315510615_2006_pv_30MW_sixty_min_v3pt4.nc + + + + + &MET_BIN;/ascii2nc + \ + &DATA_DIR_OBS;/ascii/obs_test_name.txt \ + &OUTPUT_DIR;/ascii2nc_indy/obs_test_name.nc \ + -v 1 + + + &OUTPUT_DIR;/ascii2nc_indy/obs_test_name.nc + + + + diff --git a/test/xml/unit_climatology.xml b/test/xml/unit_climatology.xml deleted file mode 100644 index d50dc86ccf..0000000000 --- a/test/xml/unit_climatology.xml +++ /dev/null @@ -1,361 +0,0 @@ - - - - - - - - - - -]> - - - - - - &TEST_DIR; - true - - - &MET_BIN;/point_stat - - OUTPUT_PREFIX GFS_CLIMO_2.5DEG - DAY_INTERVAL 31 - HOUR_INTERVAL 6 - CLIMO_MEAN_FILE_LIST - "&DATA_DIR_CLIMO;/NCEP_2.5deg/pgba_mean.19590315", - "&DATA_DIR_CLIMO;/NCEP_2.5deg/pgba_mean.19590415" - - - CLIMO_STDEV_FILE_LIST - "&DATA_DIR_CLIMO;/NCEP_2.5deg/pgba_stdv.19590315", - "&DATA_DIR_CLIMO;/NCEP_2.5deg/pgba_stdv.19590415" - - - - \ - &DATA_DIR_MODEL;/grib1/gfs/gfs_2012040900_F012.grib \ - &OUTPUT_DIR;/pb2nc/ndas.20120409.t12z.prepbufr.tm00.nc \ - &CONFIG_DIR;/PointStatConfig_climo \ - -outdir &OUTPUT_DIR;/climatology -v 4 - - - &OUTPUT_DIR;/climatology/point_stat_GFS_CLIMO_2.5DEG_120000L_20120409_120000V.stat - &OUTPUT_DIR;/climatology/point_stat_GFS_CLIMO_2.5DEG_120000L_20120409_120000V_cnt.txt - &OUTPUT_DIR;/climatology/point_stat_GFS_CLIMO_2.5DEG_120000L_20120409_120000V_sl1l2.txt - &OUTPUT_DIR;/climatology/point_stat_GFS_CLIMO_2.5DEG_120000L_20120409_120000V_sal1l2.txt - &OUTPUT_DIR;/climatology/point_stat_GFS_CLIMO_2.5DEG_120000L_20120409_120000V_mpr.txt - - - - - &MET_BIN;/point_stat - - OUTPUT_PREFIX GFS_CLIMO_1.0DEG - DAY_INTERVAL 1 - HOUR_INTERVAL 6 - CLIMO_MEAN_FILE_LIST - "&DATA_DIR_CLIMO;/NCEP_NCAR_40YR_1.0deg/cmean_1d.19590409" - - - CLIMO_STDEV_FILE_LIST - "&DATA_DIR_CLIMO;/NCEP_NCAR_40YR_1.0deg/cstdv_1d.19590409" - - - - \ - &DATA_DIR_MODEL;/grib1/gfs/gfs_2012040900_F012.grib \ - &OUTPUT_DIR;/pb2nc/ndas.20120409.t12z.prepbufr.tm00.nc \ - &CONFIG_DIR;/PointStatConfig_climo \ - -outdir &OUTPUT_DIR;/climatology -v 4 - - - &OUTPUT_DIR;/climatology/point_stat_GFS_CLIMO_1.0DEG_120000L_20120409_120000V.stat - &OUTPUT_DIR;/climatology/point_stat_GFS_CLIMO_1.0DEG_120000L_20120409_120000V_cnt.txt - &OUTPUT_DIR;/climatology/point_stat_GFS_CLIMO_1.0DEG_120000L_20120409_120000V_sl1l2.txt - &OUTPUT_DIR;/climatology/point_stat_GFS_CLIMO_1.0DEG_120000L_20120409_120000V_sal1l2.txt - &OUTPUT_DIR;/climatology/point_stat_GFS_CLIMO_1.0DEG_120000L_20120409_120000V_mpr.txt - - - - - &MET_BIN;/point_stat - - OUTPUT_PREFIX GFS_CLIMO_PREV_MONTH - DAY_INTERVAL NA - HOUR_INTERVAL 6 - CLIMO_MEAN_FILE_LIST - "&DATA_DIR_CLIMO;/NCEP_NCAR_40YR_1.0deg/cmean_1d.19590309" - - - CLIMO_STDEV_FILE_LIST - "&DATA_DIR_CLIMO;/NCEP_NCAR_40YR_1.0deg/cstdv_1d.19590309" - - - - \ - &DATA_DIR_MODEL;/grib1/gfs/gfs_2012040900_F012.grib \ - &OUTPUT_DIR;/pb2nc/ndas.20120409.t12z.prepbufr.tm00.nc \ - &CONFIG_DIR;/PointStatConfig_climo \ - -outdir &OUTPUT_DIR;/climatology -v 4 - - - &OUTPUT_DIR;/climatology/point_stat_GFS_CLIMO_PREV_MONTH_120000L_20120409_120000V.stat - &OUTPUT_DIR;/climatology/point_stat_GFS_CLIMO_PREV_MONTH_120000L_20120409_120000V_cnt.txt - &OUTPUT_DIR;/climatology/point_stat_GFS_CLIMO_PREV_MONTH_120000L_20120409_120000V_sl1l2.txt - &OUTPUT_DIR;/climatology/point_stat_GFS_CLIMO_PREV_MONTH_120000L_20120409_120000V_sal1l2.txt - &OUTPUT_DIR;/climatology/point_stat_GFS_CLIMO_PREV_MONTH_120000L_20120409_120000V_mpr.txt - - - - - &MET_BIN;/point_stat - - OUTPUT_PREFIX WMO_CLIMO_1.5DEG - CLIMO_DIR &DATA_DIR_CLIMO;/ERA_DAILY_1.5deg - - \ - &DATA_DIR_MODEL;/grib1/gfs/gfs_2012040900_F012.grib \ - &OUTPUT_DIR;/pb2nc/ndas.20120409.t12z.prepbufr.tm00.nc \ - &CONFIG_DIR;/PointStatConfig_climo_WMO \ - -outdir &OUTPUT_DIR;/climatology -v 4 - - - &OUTPUT_DIR;/climatology/point_stat_WMO_CLIMO_1.5DEG_120000L_20120409_120000V.stat - &OUTPUT_DIR;/climatology/point_stat_WMO_CLIMO_1.5DEG_120000L_20120409_120000V_ctc.txt - &OUTPUT_DIR;/climatology/point_stat_WMO_CLIMO_1.5DEG_120000L_20120409_120000V_cnt.txt - &OUTPUT_DIR;/climatology/point_stat_WMO_CLIMO_1.5DEG_120000L_20120409_120000V_sl1l2.txt - &OUTPUT_DIR;/climatology/point_stat_WMO_CLIMO_1.5DEG_120000L_20120409_120000V_sal1l2.txt - &OUTPUT_DIR;/climatology/point_stat_WMO_CLIMO_1.5DEG_120000L_20120409_120000V_vl1l2.txt - &OUTPUT_DIR;/climatology/point_stat_WMO_CLIMO_1.5DEG_120000L_20120409_120000V_val1l2.txt - &OUTPUT_DIR;/climatology/point_stat_WMO_CLIMO_1.5DEG_120000L_20120409_120000V_pct.txt - &OUTPUT_DIR;/climatology/point_stat_WMO_CLIMO_1.5DEG_120000L_20120409_120000V_pstd.txt - &OUTPUT_DIR;/climatology/point_stat_WMO_CLIMO_1.5DEG_120000L_20120409_120000V_ecnt.txt - &OUTPUT_DIR;/climatology/point_stat_WMO_CLIMO_1.5DEG_120000L_20120409_120000V_rps.txt - &OUTPUT_DIR;/climatology/point_stat_WMO_CLIMO_1.5DEG_120000L_20120409_120000V_mpr.txt - &OUTPUT_DIR;/climatology/point_stat_WMO_CLIMO_1.5DEG_120000L_20120409_120000V_vcnt.txt - - - - - &MET_BIN;/stat_analysis - - OUTPUT_DIR &OUTPUT_DIR;/climatology - - \ - -lookin &OUTPUT_DIR;/climatology/point_stat_WMO_CLIMO_1.5DEG_120000L_20120409_120000V.stat \ - -job aggregate_stat -line_type MPR -out_line_type CTC -fcst_lev P850 -interp_mthd NEAREST -by FCST_VAR -out_thresh '>CDP90' \ - -out_stat &OUTPUT_DIR;/climatology/stat_analysis_WMO_1.5DEG_MPR_to_CTC_out.stat - - - &OUTPUT_DIR;/climatology/stat_analysis_WMO_1.5DEG_MPR_to_CTC_out.stat - - - - - &MET_BIN;/stat_analysis - - OUTPUT_DIR &OUTPUT_DIR;/climatology - - \ - -lookin &OUTPUT_DIR;/climatology/point_stat_WMO_CLIMO_1.5DEG_120000L_20120409_120000V.stat \ - -job filter -line_type MPR -column_thresh CLIMO_CDF 'lt0.1||gt0.9' \ - -dump_row &OUTPUT_DIR;/climatology/stat_analysis_WMO_1.5DEG_FILTER_CDF_dump.stat - - - &OUTPUT_DIR;/climatology/stat_analysis_WMO_1.5DEG_FILTER_CDF_dump.stat - - - - - &MET_BIN;/grid_stat - - OUTPUT_PREFIX WMO_CLIMO_1.5DEG - CLIMO_DIR &DATA_DIR_CLIMO;/ERA_DAILY_1.5deg - - \ - &DATA_DIR_MODEL;/grib2/gfs/gfs_2012040900_F024.grib2 \ - &DATA_DIR_MODEL;/grib2/gfsanl/gfsanl_4_20120410_0000_000.grb2 \ - &CONFIG_DIR;/GridStatConfig_climo_WMO \ - -outdir &OUTPUT_DIR;/climatology -v 2 - - - &OUTPUT_DIR;/climatology/grid_stat_WMO_CLIMO_1.5DEG_240000L_20120410_000000V.stat - &OUTPUT_DIR;/climatology/grid_stat_WMO_CLIMO_1.5DEG_240000L_20120410_000000V_pairs.nc - - - - - &MET_BIN;/point_stat - - OUTPUT_PREFIX PROB_GFS_CLIMO_1.0DEG - DAY_INTERVAL 1 - CLIMO_MEAN_FILE_LIST - "&DATA_DIR_CLIMO;/NCEP_NCAR_40YR_1.0deg/cmean_1d.19590409" - - - CLIMO_STDEV_FILE_LIST - "&DATA_DIR_CLIMO;/NCEP_NCAR_40YR_1.0deg/cstdv_1d.19590409" - - - - \ - &DATA_DIR_MODEL;/grib2/sref_pr/sref_prob_2012040821_F015.grib2 \ - &OUTPUT_DIR;/pb2nc/ndas.20120409.t12z.prepbufr.tm00.nc \ - &CONFIG_DIR;/PointStatConfig_climo_prob \ - -outdir &OUTPUT_DIR;/climatology -v 4 - - - &OUTPUT_DIR;/climatology/point_stat_PROB_GFS_CLIMO_1.0DEG_150000L_20120409_120000V.stat - &OUTPUT_DIR;/climatology/point_stat_PROB_GFS_CLIMO_1.0DEG_150000L_20120409_120000V_pct.txt - &OUTPUT_DIR;/climatology/point_stat_PROB_GFS_CLIMO_1.0DEG_150000L_20120409_120000V_pstd.txt - &OUTPUT_DIR;/climatology/point_stat_PROB_GFS_CLIMO_1.0DEG_150000L_20120409_120000V_pjc.txt - &OUTPUT_DIR;/climatology/point_stat_PROB_GFS_CLIMO_1.0DEG_150000L_20120409_120000V_prc.txt - &OUTPUT_DIR;/climatology/point_stat_PROB_GFS_CLIMO_1.0DEG_150000L_20120409_120000V_eclv.txt - &OUTPUT_DIR;/climatology/point_stat_PROB_GFS_CLIMO_1.0DEG_150000L_20120409_120000V_mpr.txt - - - - - &MET_BIN;/grid_stat - - OUTPUT_PREFIX PROB_GFS_CLIMO_1.0DEG - DAY_INTERVAL 1 - CLIMO_MEAN_FILE_LIST - "&DATA_DIR_CLIMO;/NCEP_NCAR_40YR_1.0deg/cmean_1d.19590409" - - - CLIMO_STDEV_FILE_LIST - "&DATA_DIR_CLIMO;/NCEP_NCAR_40YR_1.0deg/cstdv_1d.19590409" - - - - \ - &DATA_DIR_MODEL;/grib2/sref_pr/sref_prob_2012040821_F015.grib2 \ - &DATA_DIR_MODEL;/grib2/gfsanl/gfsanl_4_20120409_1200_000.grb2 \ - &CONFIG_DIR;/GridStatConfig_climo_prob \ - -outdir &OUTPUT_DIR;/climatology -v 4 - - - &OUTPUT_DIR;/climatology/grid_stat_PROB_GFS_CLIMO_1.0DEG_150000L_20120409_120000V.stat - &OUTPUT_DIR;/climatology/grid_stat_PROB_GFS_CLIMO_1.0DEG_150000L_20120409_120000V_pct.txt - &OUTPUT_DIR;/climatology/grid_stat_PROB_GFS_CLIMO_1.0DEG_150000L_20120409_120000V_pstd.txt - &OUTPUT_DIR;/climatology/grid_stat_PROB_GFS_CLIMO_1.0DEG_150000L_20120409_120000V_pjc.txt - &OUTPUT_DIR;/climatology/grid_stat_PROB_GFS_CLIMO_1.0DEG_150000L_20120409_120000V_prc.txt - &OUTPUT_DIR;/climatology/grid_stat_PROB_GFS_CLIMO_1.0DEG_150000L_20120409_120000V_eclv.txt - &OUTPUT_DIR;/climatology/grid_stat_PROB_GFS_CLIMO_1.0DEG_150000L_20120409_120000V_pairs.nc - - - - - &MET_BIN;/stat_analysis - - OUTPUT_DIR &OUTPUT_DIR;/climatology - - \ - -lookin &OUTPUT_DIR;/climatology/point_stat_PROB_GFS_CLIMO_1.0DEG_150000L_20120409_120000V.stat \ - -config &CONFIG_DIR;/STATAnalysisConfig_climo \ - -v 4 - - - &OUTPUT_DIR;/climatology/stat_analysis_MPR_to_PSTD.stat - - - - - &MET_BIN;/series_analysis - - CLIMO_MEAN_FILE_LIST - "&DATA_DIR_CLIMO;/NCEP_NCAR_40YR_1.0deg/cmean_1d.19590409", - "&DATA_DIR_CLIMO;/NCEP_NCAR_40YR_1.0deg/cmean_1d.19590410", - "&DATA_DIR_CLIMO;/NCEP_NCAR_40YR_1.0deg/cmean_1d.19590411" - - - CLIMO_STDEV_FILE_LIST - "&DATA_DIR_CLIMO;/NCEP_NCAR_40YR_1.0deg/cstdv_1d.19590409", - "&DATA_DIR_CLIMO;/NCEP_NCAR_40YR_1.0deg/cstdv_1d.19590410", - "&DATA_DIR_CLIMO;/NCEP_NCAR_40YR_1.0deg/cstdv_1d.19590411" - - - - \ - -fcst &DATA_DIR_MODEL;/grib2/gfs/gfs_2012040900_F012.grib2 \ - &DATA_DIR_MODEL;/grib2/gfs/gfs_2012040900_F024.grib2 \ - &DATA_DIR_MODEL;/grib2/gfs/gfs_2012040900_F036.grib2 \ - &DATA_DIR_MODEL;/grib2/gfs/gfs_2012040900_F048.grib2 \ - -obs &DATA_DIR_MODEL;/grib2/gfsanl/gfsanl_4_20120409_1200_000.grb2 \ - &DATA_DIR_MODEL;/grib2/gfsanl/gfsanl_4_20120410_0000_000.grb2 \ - &DATA_DIR_MODEL;/grib2/gfsanl/gfsanl_4_20120410_1200_000.grb2 \ - &DATA_DIR_MODEL;/grib2/gfsanl/gfsanl_4_20120411_0000_000.grb2 \ - -paired \ - -out &OUTPUT_DIR;/climatology/series_analysis_GFS_CLIMO_1.0DEG.nc \ - -config &CONFIG_DIR;/SeriesAnalysisConfig_climo \ - -v 2 - - - &OUTPUT_DIR;/climatology/series_analysis_GFS_CLIMO_1.0DEG.nc - - - - - echo "&DATA_DIR_MODEL;/grib1/arw-fer-gep1/arw-fer-gep1_2012040912_F024.grib \ - &DATA_DIR_MODEL;/grib1/arw-fer-gep5/arw-fer-gep5_2012040912_F024.grib \ - &DATA_DIR_MODEL;/grib1/arw-sch-gep2/arw-sch-gep2_2012040912_F024.grib \ - &DATA_DIR_MODEL;/grib1/arw-sch-gep6/arw-sch-gep6_2012040912_F024.grib \ - &DATA_DIR_MODEL;/grib1/arw-tom-gep3/arw-tom-gep3_2012040912_F024.grib \ - &DATA_DIR_MODEL;/grib1/arw-tom-gep7/arw-tom-gep7_2012040912_F024.grib" \ - > &OUTPUT_DIR;/climatology/ensemble_stat_input_file_list; \ - &MET_BIN;/ensemble_stat - - OUTPUT_PREFIX NCEP_1.0DEG - CLIMO_MEAN_FILE_LIST "&DATA_DIR_CLIMO;/NCEP_NCAR_40YR_1.0deg/cmean_1d.19590410" - CLIMO_STDEV_FILE_LIST "&DATA_DIR_CLIMO;/NCEP_NCAR_40YR_1.0deg/cstdv_1d.19590410" - - \ - &OUTPUT_DIR;/climatology/ensemble_stat_input_file_list \ - &CONFIG_DIR;/EnsembleStatConfig_climo \ - -point_obs &OUTPUT_DIR;/pb2nc/ndas.20120410.t12z.prepbufr.tm00.nc \ - -grid_obs &DATA_DIR_OBS;/laps/laps_2012041012_F000.grib \ - -outdir &OUTPUT_DIR;/climatology - - - &OUTPUT_DIR;/climatology/ensemble_stat_NCEP_1.0DEG_20120410_120000V.stat - &OUTPUT_DIR;/climatology/ensemble_stat_NCEP_1.0DEG_20120410_120000V_ecnt.txt - &OUTPUT_DIR;/climatology/ensemble_stat_NCEP_1.0DEG_20120410_120000V_orank.txt - &OUTPUT_DIR;/climatology/ensemble_stat_NCEP_1.0DEG_20120410_120000V_ens.nc - &OUTPUT_DIR;/climatology/ensemble_stat_NCEP_1.0DEG_20120410_120000V_orank.nc - - - - - - - echo "&DATA_DIR_MODEL;/grib1/arw-fer-gep1/arw-fer-gep1_2012040912_F024.grib \ - &DATA_DIR_MODEL;/grib1/arw-fer-gep5/arw-fer-gep5_2012040912_F024.grib \ - &DATA_DIR_MODEL;/grib1/arw-sch-gep2/arw-sch-gep2_2012040912_F024.grib \ - &DATA_DIR_MODEL;/grib1/arw-sch-gep6/arw-sch-gep6_2012040912_F024.grib \ - &DATA_DIR_MODEL;/grib1/arw-tom-gep3/arw-tom-gep3_2012040912_F024.grib \ - &DATA_DIR_MODEL;/grib1/arw-tom-gep7/arw-tom-gep7_2012040912_F024.grib" \ - > &OUTPUT_DIR;/climatology/ensemble_stat_input_file_list; \ - &MET_BIN;/ensemble_stat - - OUTPUT_PREFIX ONE_CDF_BIN - CLIMO_MEAN_FILE_LIST "&DATA_DIR_CLIMO;/NCEP_NCAR_40YR_1.0deg/cmean_1d.19590410" - - \ - &OUTPUT_DIR;/climatology/ensemble_stat_input_file_list \ - &CONFIG_DIR;/EnsembleStatConfig_one_cdf_bin \ - -point_obs &OUTPUT_DIR;/pb2nc/ndas.20120410.t12z.prepbufr.tm00.nc \ - -grid_obs &DATA_DIR_OBS;/laps/laps_2012041012_F000.grib \ - -outdir &OUTPUT_DIR;/climatology - - - &OUTPUT_DIR;/climatology/ensemble_stat_ONE_CDF_BIN_20120410_120000V.stat - &OUTPUT_DIR;/climatology/ensemble_stat_ONE_CDF_BIN_20120410_120000V_ecnt.txt - &OUTPUT_DIR;/climatology/ensemble_stat_ONE_CDF_BIN_20120410_120000V_ens.nc - - - - diff --git a/test/xml/unit_climatology_1.0deg.xml b/test/xml/unit_climatology_1.0deg.xml new file mode 100644 index 0000000000..8fad5f3be1 --- /dev/null +++ b/test/xml/unit_climatology_1.0deg.xml @@ -0,0 +1,252 @@ + + + + + + + + + + +]> + + + + + + &TEST_DIR; + true + + + &MET_BIN;/point_stat + + OUTPUT_PREFIX GFS_CLIMO_1.0DEG + DAY_INTERVAL 1 + HOUR_INTERVAL 6 + CLIMO_MEAN_FILE_LIST + "&DATA_DIR_CLIMO;/NCEP_NCAR_40YR_1.0deg/cmean_1d.19590409" + + + CLIMO_STDEV_FILE_LIST + "&DATA_DIR_CLIMO;/NCEP_NCAR_40YR_1.0deg/cstdv_1d.19590409" + + + + \ + &DATA_DIR_MODEL;/grib1/gfs/gfs_2012040900_F012.grib \ + &OUTPUT_DIR;/pb2nc/ndas.20120409.t12z.prepbufr.tm00.nc \ + &CONFIG_DIR;/PointStatConfig_climo \ + -outdir &OUTPUT_DIR;/climatology_1.0deg -v 4 + + + &OUTPUT_DIR;/climatology_1.0deg/point_stat_GFS_CLIMO_1.0DEG_120000L_20120409_120000V.stat + &OUTPUT_DIR;/climatology_1.0deg/point_stat_GFS_CLIMO_1.0DEG_120000L_20120409_120000V_cnt.txt + &OUTPUT_DIR;/climatology_1.0deg/point_stat_GFS_CLIMO_1.0DEG_120000L_20120409_120000V_sl1l2.txt + &OUTPUT_DIR;/climatology_1.0deg/point_stat_GFS_CLIMO_1.0DEG_120000L_20120409_120000V_sal1l2.txt + &OUTPUT_DIR;/climatology_1.0deg/point_stat_GFS_CLIMO_1.0DEG_120000L_20120409_120000V_mpr.txt + + + + + &MET_BIN;/point_stat + + OUTPUT_PREFIX GFS_CLIMO_PREV_MONTH + DAY_INTERVAL NA + HOUR_INTERVAL 6 + CLIMO_MEAN_FILE_LIST + "&DATA_DIR_CLIMO;/NCEP_NCAR_40YR_1.0deg/cmean_1d.19590309" + + + CLIMO_STDEV_FILE_LIST + "&DATA_DIR_CLIMO;/NCEP_NCAR_40YR_1.0deg/cstdv_1d.19590309" + + + + \ + &DATA_DIR_MODEL;/grib1/gfs/gfs_2012040900_F012.grib \ + &OUTPUT_DIR;/pb2nc/ndas.20120409.t12z.prepbufr.tm00.nc \ + &CONFIG_DIR;/PointStatConfig_climo \ + -outdir &OUTPUT_DIR;/climatology_1.0deg -v 4 + + + &OUTPUT_DIR;/climatology_1.0deg/point_stat_GFS_CLIMO_PREV_MONTH_120000L_20120409_120000V.stat + &OUTPUT_DIR;/climatology_1.0deg/point_stat_GFS_CLIMO_PREV_MONTH_120000L_20120409_120000V_cnt.txt + &OUTPUT_DIR;/climatology_1.0deg/point_stat_GFS_CLIMO_PREV_MONTH_120000L_20120409_120000V_sl1l2.txt + &OUTPUT_DIR;/climatology_1.0deg/point_stat_GFS_CLIMO_PREV_MONTH_120000L_20120409_120000V_sal1l2.txt + &OUTPUT_DIR;/climatology_1.0deg/point_stat_GFS_CLIMO_PREV_MONTH_120000L_20120409_120000V_mpr.txt + + + + + &MET_BIN;/point_stat + + OUTPUT_PREFIX PROB_GFS_CLIMO_1.0DEG + DAY_INTERVAL 1 + CLIMO_MEAN_FILE_LIST + "&DATA_DIR_CLIMO;/NCEP_NCAR_40YR_1.0deg/cmean_1d.19590409" + + + CLIMO_STDEV_FILE_LIST + "&DATA_DIR_CLIMO;/NCEP_NCAR_40YR_1.0deg/cstdv_1d.19590409" + + + + \ + &DATA_DIR_MODEL;/grib2/sref_pr/sref_prob_2012040821_F015.grib2 \ + &OUTPUT_DIR;/pb2nc/ndas.20120409.t12z.prepbufr.tm00.nc \ + &CONFIG_DIR;/PointStatConfig_climo_prob \ + -outdir &OUTPUT_DIR;/climatology_1.0deg -v 4 + + + &OUTPUT_DIR;/climatology_1.0deg/point_stat_PROB_GFS_CLIMO_1.0DEG_150000L_20120409_120000V.stat + &OUTPUT_DIR;/climatology_1.0deg/point_stat_PROB_GFS_CLIMO_1.0DEG_150000L_20120409_120000V_pct.txt + &OUTPUT_DIR;/climatology_1.0deg/point_stat_PROB_GFS_CLIMO_1.0DEG_150000L_20120409_120000V_pstd.txt + &OUTPUT_DIR;/climatology_1.0deg/point_stat_PROB_GFS_CLIMO_1.0DEG_150000L_20120409_120000V_pjc.txt + &OUTPUT_DIR;/climatology_1.0deg/point_stat_PROB_GFS_CLIMO_1.0DEG_150000L_20120409_120000V_prc.txt + &OUTPUT_DIR;/climatology_1.0deg/point_stat_PROB_GFS_CLIMO_1.0DEG_150000L_20120409_120000V_eclv.txt + &OUTPUT_DIR;/climatology_1.0deg/point_stat_PROB_GFS_CLIMO_1.0DEG_150000L_20120409_120000V_mpr.txt + + + + + &MET_BIN;/grid_stat + + OUTPUT_PREFIX PROB_GFS_CLIMO_1.0DEG + DAY_INTERVAL 1 + CLIMO_MEAN_FILE_LIST + "&DATA_DIR_CLIMO;/NCEP_NCAR_40YR_1.0deg/cmean_1d.19590409" + + + CLIMO_STDEV_FILE_LIST + "&DATA_DIR_CLIMO;/NCEP_NCAR_40YR_1.0deg/cstdv_1d.19590409" + + + + \ + &DATA_DIR_MODEL;/grib2/sref_pr/sref_prob_2012040821_F015.grib2 \ + &DATA_DIR_MODEL;/grib2/gfsanl/gfsanl_4_20120409_1200_000.grb2 \ + &CONFIG_DIR;/GridStatConfig_climo_prob \ + -outdir &OUTPUT_DIR;/climatology_1.0deg -v 4 + + + &OUTPUT_DIR;/climatology_1.0deg/grid_stat_PROB_GFS_CLIMO_1.0DEG_150000L_20120409_120000V.stat + &OUTPUT_DIR;/climatology_1.0deg/grid_stat_PROB_GFS_CLIMO_1.0DEG_150000L_20120409_120000V_pct.txt + &OUTPUT_DIR;/climatology_1.0deg/grid_stat_PROB_GFS_CLIMO_1.0DEG_150000L_20120409_120000V_pstd.txt + &OUTPUT_DIR;/climatology_1.0deg/grid_stat_PROB_GFS_CLIMO_1.0DEG_150000L_20120409_120000V_pjc.txt + &OUTPUT_DIR;/climatology_1.0deg/grid_stat_PROB_GFS_CLIMO_1.0DEG_150000L_20120409_120000V_prc.txt + &OUTPUT_DIR;/climatology_1.0deg/grid_stat_PROB_GFS_CLIMO_1.0DEG_150000L_20120409_120000V_eclv.txt + &OUTPUT_DIR;/climatology_1.0deg/grid_stat_PROB_GFS_CLIMO_1.0DEG_150000L_20120409_120000V_pairs.nc + + + + + &MET_BIN;/stat_analysis + + OUTPUT_DIR &OUTPUT_DIR;/climatology_1.0deg + + \ + -lookin &OUTPUT_DIR;/climatology_1.0deg/point_stat_PROB_GFS_CLIMO_1.0DEG_150000L_20120409_120000V.stat \ + -config &CONFIG_DIR;/STATAnalysisConfig_climo \ + -v 4 + + + &OUTPUT_DIR;/climatology_1.0deg/stat_analysis_MPR_to_PSTD.stat + + + + + &MET_BIN;/series_analysis + + CLIMO_MEAN_FILE_LIST + "&DATA_DIR_CLIMO;/NCEP_NCAR_40YR_1.0deg/cmean_1d.19590409", + "&DATA_DIR_CLIMO;/NCEP_NCAR_40YR_1.0deg/cmean_1d.19590410", + "&DATA_DIR_CLIMO;/NCEP_NCAR_40YR_1.0deg/cmean_1d.19590411" + + + CLIMO_STDEV_FILE_LIST + "&DATA_DIR_CLIMO;/NCEP_NCAR_40YR_1.0deg/cstdv_1d.19590409", + "&DATA_DIR_CLIMO;/NCEP_NCAR_40YR_1.0deg/cstdv_1d.19590410", + "&DATA_DIR_CLIMO;/NCEP_NCAR_40YR_1.0deg/cstdv_1d.19590411" + + + + \ + -fcst &DATA_DIR_MODEL;/grib2/gfs/gfs_2012040900_F012.grib2 \ + &DATA_DIR_MODEL;/grib2/gfs/gfs_2012040900_F024.grib2 \ + &DATA_DIR_MODEL;/grib2/gfs/gfs_2012040900_F036.grib2 \ + &DATA_DIR_MODEL;/grib2/gfs/gfs_2012040900_F048.grib2 \ + -obs &DATA_DIR_MODEL;/grib2/gfsanl/gfsanl_4_20120409_1200_000.grb2 \ + &DATA_DIR_MODEL;/grib2/gfsanl/gfsanl_4_20120410_0000_000.grb2 \ + &DATA_DIR_MODEL;/grib2/gfsanl/gfsanl_4_20120410_1200_000.grb2 \ + &DATA_DIR_MODEL;/grib2/gfsanl/gfsanl_4_20120411_0000_000.grb2 \ + -paired \ + -out &OUTPUT_DIR;/climatology_1.0deg/series_analysis_GFS_CLIMO_1.0DEG.nc \ + -config &CONFIG_DIR;/SeriesAnalysisConfig_climo \ + -v 2 + + + &OUTPUT_DIR;/climatology_1.0deg/series_analysis_GFS_CLIMO_1.0DEG.nc + + + + + echo "&DATA_DIR_MODEL;/grib1/arw-fer-gep1/arw-fer-gep1_2012040912_F024.grib \ + &DATA_DIR_MODEL;/grib1/arw-fer-gep5/arw-fer-gep5_2012040912_F024.grib \ + &DATA_DIR_MODEL;/grib1/arw-sch-gep2/arw-sch-gep2_2012040912_F024.grib \ + &DATA_DIR_MODEL;/grib1/arw-sch-gep6/arw-sch-gep6_2012040912_F024.grib \ + &DATA_DIR_MODEL;/grib1/arw-tom-gep3/arw-tom-gep3_2012040912_F024.grib \ + &DATA_DIR_MODEL;/grib1/arw-tom-gep7/arw-tom-gep7_2012040912_F024.grib" \ + > &OUTPUT_DIR;/climatology_1.0deg/ensemble_stat_input_file_list; \ + &MET_BIN;/ensemble_stat + + OUTPUT_PREFIX NCEP_1.0DEG + CLIMO_MEAN_FILE_LIST "&DATA_DIR_CLIMO;/NCEP_NCAR_40YR_1.0deg/cmean_1d.19590410" + CLIMO_STDEV_FILE_LIST "&DATA_DIR_CLIMO;/NCEP_NCAR_40YR_1.0deg/cstdv_1d.19590410" + + \ + &OUTPUT_DIR;/climatology_1.0deg/ensemble_stat_input_file_list \ + &CONFIG_DIR;/EnsembleStatConfig_climo \ + -point_obs &OUTPUT_DIR;/pb2nc/ndas.20120410.t12z.prepbufr.tm00.nc \ + -grid_obs &DATA_DIR_OBS;/laps/laps_2012041012_F000.grib \ + -outdir &OUTPUT_DIR;/climatology_1.0deg + + + &OUTPUT_DIR;/climatology_1.0deg/ensemble_stat_NCEP_1.0DEG_20120410_120000V.stat + &OUTPUT_DIR;/climatology_1.0deg/ensemble_stat_NCEP_1.0DEG_20120410_120000V_ecnt.txt + &OUTPUT_DIR;/climatology_1.0deg/ensemble_stat_NCEP_1.0DEG_20120410_120000V_orank.txt + &OUTPUT_DIR;/climatology_1.0deg/ensemble_stat_NCEP_1.0DEG_20120410_120000V_ens.nc + &OUTPUT_DIR;/climatology_1.0deg/ensemble_stat_NCEP_1.0DEG_20120410_120000V_orank.nc + + + + + + + echo "&DATA_DIR_MODEL;/grib1/arw-fer-gep1/arw-fer-gep1_2012040912_F024.grib \ + &DATA_DIR_MODEL;/grib1/arw-fer-gep5/arw-fer-gep5_2012040912_F024.grib \ + &DATA_DIR_MODEL;/grib1/arw-sch-gep2/arw-sch-gep2_2012040912_F024.grib \ + &DATA_DIR_MODEL;/grib1/arw-sch-gep6/arw-sch-gep6_2012040912_F024.grib \ + &DATA_DIR_MODEL;/grib1/arw-tom-gep3/arw-tom-gep3_2012040912_F024.grib \ + &DATA_DIR_MODEL;/grib1/arw-tom-gep7/arw-tom-gep7_2012040912_F024.grib" \ + > &OUTPUT_DIR;/climatology_1.0deg/ensemble_stat_input_file_list; \ + &MET_BIN;/ensemble_stat + + OUTPUT_PREFIX ONE_CDF_BIN + CLIMO_MEAN_FILE_LIST "&DATA_DIR_CLIMO;/NCEP_NCAR_40YR_1.0deg/cmean_1d.19590410" + + \ + &OUTPUT_DIR;/climatology_1.0deg/ensemble_stat_input_file_list \ + &CONFIG_DIR;/EnsembleStatConfig_one_cdf_bin \ + -point_obs &OUTPUT_DIR;/pb2nc/ndas.20120410.t12z.prepbufr.tm00.nc \ + -grid_obs &DATA_DIR_OBS;/laps/laps_2012041012_F000.grib \ + -outdir &OUTPUT_DIR;/climatology_1.0deg + + + &OUTPUT_DIR;/climatology_1.0deg/ensemble_stat_ONE_CDF_BIN_20120410_120000V.stat + &OUTPUT_DIR;/climatology_1.0deg/ensemble_stat_ONE_CDF_BIN_20120410_120000V_ecnt.txt + &OUTPUT_DIR;/climatology_1.0deg/ensemble_stat_ONE_CDF_BIN_20120410_120000V_ens.nc + + + + diff --git a/test/xml/unit_climatology_1.5deg.xml b/test/xml/unit_climatology_1.5deg.xml new file mode 100644 index 0000000000..52dd69897b --- /dev/null +++ b/test/xml/unit_climatology_1.5deg.xml @@ -0,0 +1,98 @@ + + + + + + + + + + +]> + + + + + + &TEST_DIR; + true + + + &MET_BIN;/point_stat + + OUTPUT_PREFIX WMO_CLIMO_1.5DEG + CLIMO_DIR &DATA_DIR_CLIMO;/ERA_DAILY_1.5deg + + \ + &DATA_DIR_MODEL;/grib1/gfs/gfs_2012040900_F012.grib \ + &OUTPUT_DIR;/pb2nc/ndas.20120409.t12z.prepbufr.tm00.nc \ + &CONFIG_DIR;/PointStatConfig_climo_WMO \ + -outdir &OUTPUT_DIR;/climatology_1.5deg -v 4 + + + &OUTPUT_DIR;/climatology_1.5deg/point_stat_WMO_CLIMO_1.5DEG_120000L_20120409_120000V.stat + &OUTPUT_DIR;/climatology_1.5deg/point_stat_WMO_CLIMO_1.5DEG_120000L_20120409_120000V_ctc.txt + &OUTPUT_DIR;/climatology_1.5deg/point_stat_WMO_CLIMO_1.5DEG_120000L_20120409_120000V_cnt.txt + &OUTPUT_DIR;/climatology_1.5deg/point_stat_WMO_CLIMO_1.5DEG_120000L_20120409_120000V_sl1l2.txt + &OUTPUT_DIR;/climatology_1.5deg/point_stat_WMO_CLIMO_1.5DEG_120000L_20120409_120000V_sal1l2.txt + &OUTPUT_DIR;/climatology_1.5deg/point_stat_WMO_CLIMO_1.5DEG_120000L_20120409_120000V_vl1l2.txt + &OUTPUT_DIR;/climatology_1.5deg/point_stat_WMO_CLIMO_1.5DEG_120000L_20120409_120000V_val1l2.txt + &OUTPUT_DIR;/climatology_1.5deg/point_stat_WMO_CLIMO_1.5DEG_120000L_20120409_120000V_pct.txt + &OUTPUT_DIR;/climatology_1.5deg/point_stat_WMO_CLIMO_1.5DEG_120000L_20120409_120000V_pstd.txt + &OUTPUT_DIR;/climatology_1.5deg/point_stat_WMO_CLIMO_1.5DEG_120000L_20120409_120000V_ecnt.txt + &OUTPUT_DIR;/climatology_1.5deg/point_stat_WMO_CLIMO_1.5DEG_120000L_20120409_120000V_rps.txt + &OUTPUT_DIR;/climatology_1.5deg/point_stat_WMO_CLIMO_1.5DEG_120000L_20120409_120000V_mpr.txt + &OUTPUT_DIR;/climatology_1.5deg/point_stat_WMO_CLIMO_1.5DEG_120000L_20120409_120000V_vcnt.txt + + + + + &MET_BIN;/stat_analysis + + OUTPUT_DIR &OUTPUT_DIR;/climatology_1.5deg + + \ + -lookin &OUTPUT_DIR;/climatology_1.5deg/point_stat_WMO_CLIMO_1.5DEG_120000L_20120409_120000V.stat \ + -job aggregate_stat -line_type MPR -out_line_type CTC -fcst_lev P850 -interp_mthd NEAREST -by FCST_VAR -out_thresh '>CDP90' \ + -out_stat &OUTPUT_DIR;/climatology_1.5deg/stat_analysis_WMO_1.5DEG_MPR_to_CTC_out.stat + + + &OUTPUT_DIR;/climatology_1.5deg/stat_analysis_WMO_1.5DEG_MPR_to_CTC_out.stat + + + + + &MET_BIN;/stat_analysis + + OUTPUT_DIR &OUTPUT_DIR;/climatology_1.5deg + + \ + -lookin &OUTPUT_DIR;/climatology_1.5deg/point_stat_WMO_CLIMO_1.5DEG_120000L_20120409_120000V.stat \ + -job filter -line_type MPR -column_thresh CLIMO_CDF 'lt0.1||gt0.9' \ + -dump_row &OUTPUT_DIR;/climatology_1.5deg/stat_analysis_WMO_1.5DEG_FILTER_CDF_dump.stat + + + &OUTPUT_DIR;/climatology_1.5deg/stat_analysis_WMO_1.5DEG_FILTER_CDF_dump.stat + + + + + &MET_BIN;/grid_stat + + OUTPUT_PREFIX WMO_CLIMO_1.5DEG + CLIMO_DIR &DATA_DIR_CLIMO;/ERA_DAILY_1.5deg + + \ + &DATA_DIR_MODEL;/grib2/gfs/gfs_2012040900_F024.grib2 \ + &DATA_DIR_MODEL;/grib2/gfsanl/gfsanl_4_20120410_0000_000.grb2 \ + &CONFIG_DIR;/GridStatConfig_climo_WMO \ + -outdir &OUTPUT_DIR;/climatology_1.5deg -v 2 + + + &OUTPUT_DIR;/climatology_1.5deg/grid_stat_WMO_CLIMO_1.5DEG_240000L_20120410_000000V.stat + &OUTPUT_DIR;/climatology_1.5deg/grid_stat_WMO_CLIMO_1.5DEG_240000L_20120410_000000V_pairs.nc + + + + diff --git a/test/xml/unit_climatology_2.5deg.xml b/test/xml/unit_climatology_2.5deg.xml new file mode 100644 index 0000000000..7c8c0b300a --- /dev/null +++ b/test/xml/unit_climatology_2.5deg.xml @@ -0,0 +1,53 @@ + + + + + + + + + + +]> + + + + + + &TEST_DIR; + true + + + &MET_BIN;/point_stat + + OUTPUT_PREFIX GFS_CLIMO_2.5DEG + DAY_INTERVAL 31 + HOUR_INTERVAL 6 + CLIMO_MEAN_FILE_LIST + "&DATA_DIR_CLIMO;/NCEP_2.5deg/pgba_mean.19590315", + "&DATA_DIR_CLIMO;/NCEP_2.5deg/pgba_mean.19590415" + + + CLIMO_STDEV_FILE_LIST + "&DATA_DIR_CLIMO;/NCEP_2.5deg/pgba_stdv.19590315", + "&DATA_DIR_CLIMO;/NCEP_2.5deg/pgba_stdv.19590415" + + + + \ + &DATA_DIR_MODEL;/grib1/gfs/gfs_2012040900_F012.grib \ + &OUTPUT_DIR;/pb2nc/ndas.20120409.t12z.prepbufr.tm00.nc \ + &CONFIG_DIR;/PointStatConfig_climo \ + -outdir &OUTPUT_DIR;/climatology_2.5deg -v 4 + + + &OUTPUT_DIR;/climatology_2.5deg/point_stat_GFS_CLIMO_2.5DEG_120000L_20120409_120000V.stat + &OUTPUT_DIR;/climatology_2.5deg/point_stat_GFS_CLIMO_2.5DEG_120000L_20120409_120000V_cnt.txt + &OUTPUT_DIR;/climatology_2.5deg/point_stat_GFS_CLIMO_2.5DEG_120000L_20120409_120000V_sl1l2.txt + &OUTPUT_DIR;/climatology_2.5deg/point_stat_GFS_CLIMO_2.5DEG_120000L_20120409_120000V_sal1l2.txt + &OUTPUT_DIR;/climatology_2.5deg/point_stat_GFS_CLIMO_2.5DEG_120000L_20120409_120000V_mpr.txt + + + + diff --git a/test/xml/unit_pb2nc.xml b/test/xml/unit_pb2nc.xml index d6d79def22..fc31e0e61b 100644 --- a/test/xml/unit_pb2nc.xml +++ b/test/xml/unit_pb2nc.xml @@ -14,6 +14,8 @@ + + &TEST_DIR; @@ -75,139 +77,5 @@ &OUTPUT_DIR;/pb2nc/ndas.20120409.t12z.prepbufr.tm00.nc - - - &MET_BIN;/pb2nc - - STATION_ID "72265","72274","72364","72426" - MASK_GRID - MASK_POLY - QUALITY_MARK_THRESH 2 - - \ - &DATA_DIR_OBS;/prepbufr/ndas/nam.20120409.t12z.prepbufr.tm00.nr \ - &OUTPUT_DIR;/pb2nc/ndas.20120409.t12z.prepbufr.tm00.mask_sid.nc \ - &CONFIG_DIR;/PB2NCConfig \ - -v 1 - - - &OUTPUT_DIR;/pb2nc/ndas.20120409.t12z.prepbufr.tm00.mask_sid.nc - - - - - &MET_BIN;/pb2nc - - STATION_ID "&CONFIG_DIR;/SID_CO.txt" - MASK_GRID - MASK_POLY - QUALITY_MARK_THRESH 2 - - \ - &DATA_DIR_OBS;/prepbufr/ndas/nam.20120409.t12z.prepbufr.tm00.nr \ - &OUTPUT_DIR;/pb2nc/ndas.20120409.t12z.prepbufr.tm00.mask_sid_file.nc \ - &CONFIG_DIR;/PB2NCConfig \ - -v 1 - - - &OUTPUT_DIR;/pb2nc/ndas.20120409.t12z.prepbufr.tm00.mask_sid_file.nc - - - - - &MET_BIN;/pb2nc - - STATION_ID - MASK_GRID &MET_DATA;/sample_fcst/2005080700/wrfprs_ruc13_00.tm00_G212 - MASK_POLY - QUALITY_MARK_THRESH 2 - - \ - &DATA_DIR_OBS;/prepbufr/ndas/nam.20120409.t12z.prepbufr.tm00.nr \ - &OUTPUT_DIR;/pb2nc/ndas.20120409.t12z.prepbufr.tm00.mask_grid_data.nc \ - &CONFIG_DIR;/PB2NCConfig \ - -v 1 - - - &OUTPUT_DIR;/pb2nc/ndas.20120409.t12z.prepbufr.tm00.mask_grid_data.nc - - - - - &MET_BIN;/pb2nc - - STATION_ID - MASK_GRID - MASK_POLY - QUALITY_MARK_THRESH 2 - - \ - &DATA_DIR_OBS;/prepbufr/nam.20210311.t00z.prepbufr.tm00 \ - &OUTPUT_DIR;/pb2nc/nam.20210311.t00z.prepbufr.tm00.pbl.nc \ - &CONFIG_DIR;/PB2NCConfig_pbl \ - -v 1 - - - &OUTPUT_DIR;/pb2nc/nam.20210311.t00z.prepbufr.tm00.pbl.nc - - - - - &MET_BIN;/pb2nc - - STATION_ID "72364","72265","72274","72426","72489","14008" - MASK_GRID - MASK_POLY - QUALITY_MARK_THRESH 2 - - \ - &DATA_DIR_OBS;/prepbufr/ndas/nam.20120409.t12z.prepbufr.tm00.nr \ - &OUTPUT_DIR;/pb2nc/ndas.20120409.t12z.prepbufr.tm00.var_all.nc \ - &CONFIG_DIR;/PB2NCConfig_all \ - -v 1 - - - &OUTPUT_DIR;/pb2nc/ndas.20120409.t12z.prepbufr.tm00.var_all.nc - - - - - &MET_BIN;/pb2nc - - STATION_ID "SC-GSP","TX-HGX","KY-HPX" - MASK_GRID - MASK_POLY - QUALITY_MARK_THRESH 9 - - \ - &DATA_DIR_OBS;/bufr/nam_20170502.t00z.radwnd.tm02.bufr_d \ - &OUTPUT_DIR;/pb2nc/nam_20170502.t00z.radwnd.tm02.bufr_d.vlevel_500.nc \ - &CONFIG_DIR;/PB2NCConfig_vlevel -valid_end 20170501_22 \ - -v 1 - - - &OUTPUT_DIR;/pb2nc/nam_20170502.t00z.radwnd.tm02.bufr_d.vlevel_500.nc - - - - - &MET_BIN;/pb2nc - - STATION_ID - MASK_GRID - MASK_POLY MET_BASE/poly/CONUS.poly - QUALITY_MARK_THRESH 2 - - \ - &DATA_DIR_OBS;/prepbufr/ndas/nam.20120409.t12z.prepbufr.tm00.nr \ - &OUTPUT_DIR;/pb2nc/ndas.20120409.t12z.prepbufr.tm00.summary.nc \ - &CONFIG_DIR;/PB2NCConfig_summary \ - -v 1 - - - &OUTPUT_DIR;/pb2nc/ndas.20120409.t12z.prepbufr.tm00.summary.nc - - - diff --git a/test/xml/unit_pb2nc_indy.xml b/test/xml/unit_pb2nc_indy.xml new file mode 100644 index 0000000000..d0483d2c6a --- /dev/null +++ b/test/xml/unit_pb2nc_indy.xml @@ -0,0 +1,157 @@ + + + + + + + + + + + +]> + + + + + + + + &TEST_DIR; + true + + + &MET_BIN;/pb2nc + + STATION_ID "72265","72274","72364","72426" + MASK_GRID + MASK_POLY + QUALITY_MARK_THRESH 2 + + \ + &DATA_DIR_OBS;/prepbufr/ndas/nam.20120409.t12z.prepbufr.tm00.nr \ + &OUTPUT_DIR;/pb2nc_indy/ndas.20120409.t12z.prepbufr.tm00.mask_sid.nc \ + &CONFIG_DIR;/PB2NCConfig \ + -v 1 + + + &OUTPUT_DIR;/pb2nc_indy/ndas.20120409.t12z.prepbufr.tm00.mask_sid.nc + + + + + &MET_BIN;/pb2nc + + STATION_ID "&CONFIG_DIR;/SID_CO.txt" + MASK_GRID + MASK_POLY + QUALITY_MARK_THRESH 2 + + \ + &DATA_DIR_OBS;/prepbufr/ndas/nam.20120409.t12z.prepbufr.tm00.nr \ + &OUTPUT_DIR;/pb2nc_indy/ndas.20120409.t12z.prepbufr.tm00.mask_sid_file.nc \ + &CONFIG_DIR;/PB2NCConfig \ + -v 1 + + + &OUTPUT_DIR;/pb2nc_indy/ndas.20120409.t12z.prepbufr.tm00.mask_sid_file.nc + + + + + &MET_BIN;/pb2nc + + STATION_ID + MASK_GRID &MET_DATA;/sample_fcst/2005080700/wrfprs_ruc13_00.tm00_G212 + MASK_POLY + QUALITY_MARK_THRESH 2 + + \ + &DATA_DIR_OBS;/prepbufr/ndas/nam.20120409.t12z.prepbufr.tm00.nr \ + &OUTPUT_DIR;/pb2nc_indy/ndas.20120409.t12z.prepbufr.tm00.mask_grid_data.nc \ + &CONFIG_DIR;/PB2NCConfig \ + -v 1 + + + &OUTPUT_DIR;/pb2nc_indy/ndas.20120409.t12z.prepbufr.tm00.mask_grid_data.nc + + + + + &MET_BIN;/pb2nc + + STATION_ID + MASK_GRID + MASK_POLY + QUALITY_MARK_THRESH 2 + + \ + &DATA_DIR_OBS;/prepbufr/nam.20210311.t00z.prepbufr.tm00 \ + &OUTPUT_DIR;/pb2nc_indy/nam.20210311.t00z.prepbufr.tm00.pbl.nc \ + &CONFIG_DIR;/PB2NCConfig_pbl \ + -v 1 + + + &OUTPUT_DIR;/pb2nc_indy/nam.20210311.t00z.prepbufr.tm00.pbl.nc + + + + + &MET_BIN;/pb2nc + + STATION_ID "72364","72265","72274","72426","72489","14008" + MASK_GRID + MASK_POLY + QUALITY_MARK_THRESH 2 + + \ + &DATA_DIR_OBS;/prepbufr/ndas/nam.20120409.t12z.prepbufr.tm00.nr \ + &OUTPUT_DIR;/pb2nc_indy/ndas.20120409.t12z.prepbufr.tm00.var_all.nc \ + &CONFIG_DIR;/PB2NCConfig_all \ + -v 1 + + + &OUTPUT_DIR;/pb2nc_indy/ndas.20120409.t12z.prepbufr.tm00.var_all.nc + + + + + &MET_BIN;/pb2nc + + STATION_ID "SC-GSP","TX-HGX","KY-HPX" + MASK_GRID + MASK_POLY + QUALITY_MARK_THRESH 9 + + \ + &DATA_DIR_OBS;/bufr/nam_20170502.t00z.radwnd.tm02.bufr_d \ + &OUTPUT_DIR;/pb2nc_indy/nam_20170502.t00z.radwnd.tm02.bufr_d.vlevel_500.nc \ + &CONFIG_DIR;/PB2NCConfig_vlevel -valid_end 20170501_22 \ + -v 1 + + + &OUTPUT_DIR;/pb2nc_indy/nam_20170502.t00z.radwnd.tm02.bufr_d.vlevel_500.nc + + + + + &MET_BIN;/pb2nc + + STATION_ID + MASK_GRID + MASK_POLY MET_BASE/poly/CONUS.poly + QUALITY_MARK_THRESH 2 + + \ + &DATA_DIR_OBS;/prepbufr/ndas/nam.20120409.t12z.prepbufr.tm00.nr \ + &OUTPUT_DIR;/pb2nc_indy/ndas.20120409.t12z.prepbufr.tm00.summary.nc \ + &CONFIG_DIR;/PB2NCConfig_summary \ + -v 1 + + + &OUTPUT_DIR;/pb2nc_indy/ndas.20120409.t12z.prepbufr.tm00.summary.nc + + + + diff --git a/test/xml/unit_pcp_combine.xml b/test/xml/unit_pcp_combine.xml index 4a1c39b76b..b2de557363 100644 --- a/test/xml/unit_pcp_combine.xml +++ b/test/xml/unit_pcp_combine.xml @@ -187,10 +187,10 @@ -subtract \ &DATA_DIR_MODEL;/p_interp/wrfout_d01_2008-08-08_12:00:00_PLEV 'name="RAINNC";level="(0,*,*)";' \ &DATA_DIR_MODEL;/p_interp/wrfout_d01_2008-08-08_06:00:00_PLEV 'name="RAINNC";level="(0,*,*)";' \ - &OUTPUT_DIR;/pcp_combine/wrfout_d01_2008-08-08_12:00:00_PLEV_APCP06.nc + &OUTPUT_DIR;/pcp_combine/wrfout_d01_2008-08-08_12_00_00_PLEV_APCP06.nc - &OUTPUT_DIR;/pcp_combine/wrfout_d01_2008-08-08_12:00:00_PLEV_APCP06.nc + &OUTPUT_DIR;/pcp_combine/wrfout_d01_2008-08-08_12_00_00_PLEV_APCP06.nc diff --git a/test/xml/unit_plot_data_plane.xml b/test/xml/unit_plot_data_plane.xml index 25d47c1397..c118078829 100644 --- a/test/xml/unit_plot_data_plane.xml +++ b/test/xml/unit_plot_data_plane.xml @@ -135,13 +135,13 @@ &MET_BIN;/plot_data_plane \ &DATA_DIR_MODEL;/p_interp/wrfout_d01_2008-08-08_12:00:00_PLEV \ - &OUTPUT_DIR;/plot_data_plane/wrfout_d01_2008-08-08_12:00:00_PLEV_NC_PINTERP_PRES_SFC.ps \ + &OUTPUT_DIR;/plot_data_plane/wrfout_d01_2008-08-08_12_00_00_PLEV_NC_PINTERP_PRES_SFC.ps \ 'name = "PSFC"; level = "(0,*,*)";' \ -title "NC PINTERP Pressure at the Surface" \ -v 1 - &OUTPUT_DIR;/plot_data_plane/wrfout_d01_2008-08-08_12:00:00_PLEV_NC_PINTERP_PRES_SFC.ps + &OUTPUT_DIR;/plot_data_plane/wrfout_d01_2008-08-08_12_00_00_PLEV_NC_PINTERP_PRES_SFC.ps diff --git a/test/xml/unit_python.xml b/test/xml/unit_python.xml index 226d61869b..b2b1dad44b 100644 --- a/test/xml/unit_python.xml +++ b/test/xml/unit_python.xml @@ -15,7 +15,7 @@ ]> - + @@ -378,7 +378,8 @@ - &MET_BIN;/plot_data_plane + echo "MET_PYTHON_EXE=&MET_PYTHON_EXE;"; \ + &MET_BIN;/plot_data_plane MET_PYTHON_EXE &MET_PYTHON_EXE; PYTHON_GRID G212 @@ -398,7 +399,8 @@ - &MET_BIN;/ascii2nc + echo "MET_PYTHON_EXE=&MET_PYTHON_EXE;"; \ + &MET_BIN;/ascii2nc MET_PYTHON_EXE &MET_PYTHON_EXE; @@ -415,6 +417,7 @@ export PATH='&ANACONDA_BIN;:${PATH}'; \ + echo "MET_PYTHON_EXE=&MET_PYTHON_EXE;"; \ &MET_BIN;/plot_data_plane MET_PYTHON_EXE &MET_PYTHON_EXE; @@ -434,7 +437,8 @@ - &MET_BIN;/stat_analysis + echo "MET_PYTHON_EXE=&MET_PYTHON_EXE;"; \ + &MET_BIN;/stat_analysis MET_PYTHON_EXE &MET_PYTHON_EXE; diff --git a/test/xml/unit_ref_config.xml b/test/xml/unit_ref_config.xml index 5f9320ba5d..0cf9a8b8f3 100644 --- a/test/xml/unit_ref_config.xml +++ b/test/xml/unit_ref_config.xml @@ -11,930 +11,13 @@ ]> - + &TEST_DIR; true - - - - - - &MET_BIN;/gen_vx_mask - \ - &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_012.tm00 \ - &MET_BASE;/poly/CONUS.poly \ - &OUTPUT_DIR;/ref_config/gen_vx_mask/CONUS.nc \ - -type poly -v 2 - - - &OUTPUT_DIR;/ref_config/gen_vx_mask/CONUS.nc - - - - - - - - --> - &MET_BIN;/pb2nc - - MASK_POLY &CONFIG_DIR;/ref_config/RefConfig.poly - - \ - &DATA_DIR_OBS;/ref_config/prepbufr/ndas/2011090212/ndas.t12z.prepbufr.tm12.nr \ - &OUTPUT_DIR;/ref_config/pb2nc/NDAS_03h/20110902/prepbufr.ndas.20110902.t00z.tm12.nc \ - &CONFIG_DIR;/ref_config/PB2NCConfig \ - -valid_beg 20110901_223000\ - -valid_end 20110902_013000\ - -v 2 - - - &OUTPUT_DIR;/ref_config/pb2nc/NDAS_03h/20110902/prepbufr.ndas.20110902.t00z.tm12.nc - - - - --> - &MET_BIN;/pb2nc - - MASK_POLY &CONFIG_DIR;/ref_config/RefConfig.poly - - \ - &DATA_DIR_OBS;/ref_config/prepbufr/ndas/2011090300/ndas.t00z.prepbufr.tm12.nr \ - &OUTPUT_DIR;/ref_config/pb2nc/NDAS_03h/20110902/prepbufr.ndas.20110902.t12z.tm12.nc \ - &CONFIG_DIR;/ref_config/PB2NCConfig \ - -valid_beg 20110902_103000\ - -valid_end 20110902_133000\ - -v 2 - - - &OUTPUT_DIR;/ref_config/pb2nc/NDAS_03h/20110902/prepbufr.ndas.20110902.t12z.tm12.nc - - - - --> - &MET_BIN;/pb2nc - - MASK_POLY &CONFIG_DIR;/ref_config/RefConfig.poly - - \ - &DATA_DIR_OBS;/ref_config/prepbufr/ndas/2011090312/ndas.t12z.prepbufr.tm12.nr \ - &OUTPUT_DIR;/ref_config/pb2nc/NDAS_03h/20110903/prepbufr.ndas.20110903.t00z.tm12.nc \ - &CONFIG_DIR;/ref_config/PB2NCConfig \ - -valid_beg 20110902_223000\ - -valid_end 20110903_013000\ - -v 2 - - - &OUTPUT_DIR;/ref_config/pb2nc/NDAS_03h/20110903/prepbufr.ndas.20110903.t00z.tm12.nc - - - - --> - &MET_BIN;/pb2nc - - MASK_POLY &CONFIG_DIR;/ref_config/RefConfig.poly - - \ - &DATA_DIR_OBS;/ref_config/prepbufr/ndas/2011090400/ndas.t00z.prepbufr.tm12.nr \ - &OUTPUT_DIR;/ref_config/pb2nc/NDAS_03h/20110903/prepbufr.ndas.20110903.t12z.tm12.nc \ - &CONFIG_DIR;/ref_config/PB2NCConfig \ - -valid_beg 20110903_103000\ - -valid_end 20110903_133000\ - -v 2 - - - &OUTPUT_DIR;/ref_config/pb2nc/NDAS_03h/20110903/prepbufr.ndas.20110903.t12z.tm12.nc - - - - --> - &MET_BIN;/pb2nc - - MASK_POLY &CONFIG_DIR;/ref_config/RefConfig.poly - - \ - &DATA_DIR_OBS;/ref_config/prepbufr/ndas/2011090412/ndas.t12z.prepbufr.tm12.nr \ - &OUTPUT_DIR;/ref_config/pb2nc/NDAS_03h/20110904/prepbufr.ndas.20110904.t00z.tm12.nc \ - &CONFIG_DIR;/ref_config/PB2NCConfig \ - -valid_beg 20110903_223000\ - -valid_end 20110904_013000\ - -v 2 - - - &OUTPUT_DIR;/ref_config/pb2nc/NDAS_03h/20110904/prepbufr.ndas.20110904.t00z.tm12.nc - - - - - - - - - - &MET_BIN;/pcp_combine - \ - -sum \ - 00000000_000000 1 20110902_120000 3 \ - &OUTPUT_DIR;/ref_config/pcp_combine/ST2_03h/20110902/ST2ml2011090212.03h.nc \ - -pcpdir &DATA_DIR_OBS;/ref_config/pcp_combine/20110902 - - - &OUTPUT_DIR;/ref_config/pcp_combine/ST2_03h/20110902/ST2ml2011090212.03h.nc - - - - - &MET_BIN;/pcp_combine - \ - -sum \ - 00000000_000000 1 20110903_000000 3 \ - &OUTPUT_DIR;/ref_config/pcp_combine/ST2_03h/20110903/ST2ml2011090300.03h.nc \ - -pcpdir &DATA_DIR_OBS;/ref_config/pcp_combine/20110902 \ - -pcpdir &DATA_DIR_OBS;/ref_config/pcp_combine/20110903 - - - &OUTPUT_DIR;/ref_config/pcp_combine/ST2_03h/20110903/ST2ml2011090300.03h.nc - - - - - &MET_BIN;/pcp_combine - \ - -sum \ - 00000000_000000 1 20110903_120000 3 \ - &OUTPUT_DIR;/ref_config/pcp_combine/ST2_03h/20110903/ST2ml2011090312.03h.nc \ - -pcpdir &DATA_DIR_OBS;/ref_config/pcp_combine/20110903 - - - &OUTPUT_DIR;/ref_config/pcp_combine/ST2_03h/20110903/ST2ml2011090312.03h.nc - - - - - &MET_BIN;/pcp_combine - \ - -sum \ - 00000000_000000 1 20110904_000000 3 \ - &OUTPUT_DIR;/ref_config/pcp_combine/ST2_03h/20110904/ST2ml2011090400.03h.nc \ - -pcpdir &DATA_DIR_OBS;/ref_config/pcp_combine/20110903 \ - -pcpdir &DATA_DIR_OBS;/ref_config/pcp_combine/20110904 - - - &OUTPUT_DIR;/ref_config/pcp_combine/ST2_03h/20110904/ST2ml2011090400.03h.nc - - - - - &MET_BIN;/pcp_combine - \ - -add \ - &DATA_DIR_OBS;/ref_config/pcp_combine/2011090324/ST2ml2011090312.24h.grb \ - 24 \ - &OUTPUT_DIR;/ref_config/pcp_combine/ST2_24h/20110903/ST2ml2011090312.24h.nc - - - &OUTPUT_DIR;/ref_config/pcp_combine/ST2_24h/20110903/ST2ml2011090312.24h.nc - - - - - &MET_BIN;/pcp_combine - \ - -subtract \ - &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_012.tm00 12 \ - &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_009.tm00 9 \ - &OUTPUT_DIR;/ref_config/pcp_combine/wrf/wrfpcp03_012.nc - - - &OUTPUT_DIR;/ref_config/pcp_combine/wrf/wrfpcp03_012.nc - - - - - &MET_BIN;/pcp_combine - \ - -subtract \ - &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_024.tm00 24 \ - &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_021.tm00 21 \ - &OUTPUT_DIR;/ref_config/pcp_combine/wrf/wrfpcp03_024.nc - - - &OUTPUT_DIR;/ref_config/pcp_combine/wrf/wrfpcp03_024.nc - - - - - &MET_BIN;/pcp_combine - \ - -subtract \ - &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_036.tm00 36 \ - &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_033.tm00 33 \ - &OUTPUT_DIR;/ref_config/pcp_combine/wrf/wrfpcp03_036.nc - - - &OUTPUT_DIR;/ref_config/pcp_combine/wrf/wrfpcp03_036.nc - - - - - &MET_BIN;/pcp_combine - \ - -subtract \ - &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_048.tm00 48 \ - &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_045.tm00 45 \ - &OUTPUT_DIR;/ref_config/pcp_combine/wrf/wrfpcp03_048.nc - - - &OUTPUT_DIR;/ref_config/pcp_combine/wrf/wrfpcp03_048.nc - - - - - &MET_BIN;/pcp_combine - \ - -subtract \ - &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_036.tm00 36 \ - &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_012.tm00 12 \ - &OUTPUT_DIR;/ref_config/pcp_combine/wrf/wrfpcp24_036.nc - - - &OUTPUT_DIR;/ref_config/pcp_combine/wrf/wrfpcp24_036.nc - - - - - - - - - - &MET_BIN;/point_stat - - MASK_POLY_FILE &OUTPUT_DIR;/ref_config/gen_vx_mask/CONUS.nc - MODEL AFWAv3.4_Noahv2.7.1 - FCST_TIME 000 - - \ - &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv2.7.1/postprd/wrfprs_000.tm00 \ - &OUTPUT_DIR;/ref_config/pb2nc/NDAS_03h/20110902/prepbufr.ndas.20110902.t00z.tm12.nc \ - &CONFIG_DIR;/ref_config/PointStatConfig_ADPUPA \ - -outdir &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv2.7.1 \ - -v 2 - - - &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv2.7.1/point_stat_AFWAv3.4_Noahv2.7.1_F000_ADPUPA_000000L_20110902_000000V.stat - - - - - &MET_BIN;/point_stat - - MASK_POLY_FILE &OUTPUT_DIR;/ref_config/gen_vx_mask/CONUS.nc - MODEL AFWAv3.4_Noahv2.7.1 - FCST_TIME 000 - - \ - &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv2.7.1/postprd/wrfprs_000.tm00 \ - &OUTPUT_DIR;/ref_config/pb2nc/NDAS_03h/20110902/prepbufr.ndas.20110902.t00z.tm12.nc \ - &CONFIG_DIR;/ref_config/PointStatConfig_ONLYSF \ - -outdir &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv2.7.1 \ - -v 2 - - - &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv2.7.1/point_stat_AFWAv3.4_Noahv2.7.1_F000_ONLYSF_000000L_20110902_000000V.stat - - - - - &MET_BIN;/point_stat - - MASK_POLY_FILE &OUTPUT_DIR;/ref_config/gen_vx_mask/CONUS.nc - MODEL AFWAv3.4_Noahv2.7.1 - FCST_TIME 000 - - \ - &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv2.7.1/postprd/wrfprs_000.tm00 \ - &OUTPUT_DIR;/ref_config/pb2nc/NDAS_03h/20110902/prepbufr.ndas.20110902.t00z.tm12.nc \ - &CONFIG_DIR;/ref_config/PointStatConfig_WINDS \ - -outdir &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv2.7.1 \ - -v 2 - - - &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv2.7.1/point_stat_AFWAv3.4_Noahv2.7.1_F000_WINDS_000000L_20110902_000000V.stat - - - - - &MET_BIN;/point_stat - - MASK_POLY_FILE &OUTPUT_DIR;/ref_config/gen_vx_mask/CONUS.nc - MODEL AFWAv3.4_Noahv2.7.1 - FCST_TIME 012 - - \ - &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv2.7.1/postprd/wrfprs_012.tm00 \ - &OUTPUT_DIR;/ref_config/pb2nc/NDAS_03h/20110902/prepbufr.ndas.20110902.t12z.tm12.nc \ - &CONFIG_DIR;/ref_config/PointStatConfig_ADPUPA \ - -outdir &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv2.7.1 \ - -v 2 - - - &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv2.7.1/point_stat_AFWAv3.4_Noahv2.7.1_F012_ADPUPA_120000L_20110902_120000V.stat - - - - - &MET_BIN;/point_stat - - MASK_POLY_FILE &OUTPUT_DIR;/ref_config/gen_vx_mask/CONUS.nc - MODEL AFWAv3.4_Noahv2.7.1 - FCST_TIME 012 - - \ - &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv2.7.1/postprd/wrfprs_012.tm00 \ - &OUTPUT_DIR;/ref_config/pb2nc/NDAS_03h/20110902/prepbufr.ndas.20110902.t12z.tm12.nc \ - &CONFIG_DIR;/ref_config/PointStatConfig_ONLYSF \ - -outdir &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv2.7.1 \ - -v 2 - - - &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv2.7.1/point_stat_AFWAv3.4_Noahv2.7.1_F012_ONLYSF_120000L_20110902_120000V.stat - - - - - &MET_BIN;/point_stat - - MASK_POLY_FILE &OUTPUT_DIR;/ref_config/gen_vx_mask/CONUS.nc - MODEL AFWAv3.4_Noahv2.7.1 - FCST_TIME 012 - - \ - &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv2.7.1/postprd/wrfprs_012.tm00 \ - &OUTPUT_DIR;/ref_config/pb2nc/NDAS_03h/20110902/prepbufr.ndas.20110902.t12z.tm12.nc \ - &CONFIG_DIR;/ref_config/PointStatConfig_WINDS \ - -outdir &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv2.7.1 \ - -v 2 - - - &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv2.7.1/point_stat_AFWAv3.4_Noahv2.7.1_F012_WINDS_120000L_20110902_120000V.stat - - - - - &MET_BIN;/point_stat - - MASK_POLY_FILE &OUTPUT_DIR;/ref_config/gen_vx_mask/CONUS.nc - MODEL AFWAv3.4_Noahv2.7.1 - FCST_TIME 024 - - \ - &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv2.7.1/postprd/wrfprs_024.tm00 \ - &OUTPUT_DIR;/ref_config/pb2nc/NDAS_03h/20110903/prepbufr.ndas.20110903.t00z.tm12.nc \ - &CONFIG_DIR;/ref_config/PointStatConfig_ADPUPA \ - -outdir &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv2.7.1 \ - -v 2 - - - &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv2.7.1/point_stat_AFWAv3.4_Noahv2.7.1_F024_ADPUPA_240000L_20110903_000000V.stat - - - - - &MET_BIN;/point_stat - - MASK_POLY_FILE &OUTPUT_DIR;/ref_config/gen_vx_mask/CONUS.nc - MODEL AFWAv3.4_Noahv2.7.1 - FCST_TIME 024 - - \ - &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv2.7.1/postprd/wrfprs_024.tm00 \ - &OUTPUT_DIR;/ref_config/pb2nc/NDAS_03h/20110903/prepbufr.ndas.20110903.t00z.tm12.nc \ - &CONFIG_DIR;/ref_config/PointStatConfig_ONLYSF \ - -outdir &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv2.7.1 \ - -v 2 - - - &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv2.7.1/point_stat_AFWAv3.4_Noahv2.7.1_F024_ONLYSF_240000L_20110903_000000V.stat - - - - - &MET_BIN;/point_stat - - MASK_POLY_FILE &OUTPUT_DIR;/ref_config/gen_vx_mask/CONUS.nc - MODEL AFWAv3.4_Noahv2.7.1 - FCST_TIME 024 - - \ - &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv2.7.1/postprd/wrfprs_024.tm00 \ - &OUTPUT_DIR;/ref_config/pb2nc/NDAS_03h/20110903/prepbufr.ndas.20110903.t00z.tm12.nc \ - &CONFIG_DIR;/ref_config/PointStatConfig_WINDS \ - -outdir &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv2.7.1 \ - -v 2 - - - &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv2.7.1/point_stat_AFWAv3.4_Noahv2.7.1_F024_WINDS_240000L_20110903_000000V.stat - - - - - &MET_BIN;/point_stat - - MASK_POLY_FILE &OUTPUT_DIR;/ref_config/gen_vx_mask/CONUS.nc - MODEL AFWAv3.4_Noahv2.7.1 - FCST_TIME 036 - - \ - &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv2.7.1/postprd/wrfprs_036.tm00 \ - &OUTPUT_DIR;/ref_config/pb2nc/NDAS_03h/20110903/prepbufr.ndas.20110903.t12z.tm12.nc \ - &CONFIG_DIR;/ref_config/PointStatConfig_ADPUPA \ - -outdir &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv2.7.1 \ - -v 2 - - - &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv2.7.1/point_stat_AFWAv3.4_Noahv2.7.1_F036_ADPUPA_360000L_20110903_120000V.stat - - - - - &MET_BIN;/point_stat - - MASK_POLY_FILE &OUTPUT_DIR;/ref_config/gen_vx_mask/CONUS.nc - MODEL AFWAv3.4_Noahv2.7.1 - FCST_TIME 036 - - \ - &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv2.7.1/postprd/wrfprs_036.tm00 \ - &OUTPUT_DIR;/ref_config/pb2nc/NDAS_03h/20110903/prepbufr.ndas.20110903.t12z.tm12.nc \ - &CONFIG_DIR;/ref_config/PointStatConfig_ONLYSF \ - -outdir &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv2.7.1 \ - -v 2 - - - &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv2.7.1/point_stat_AFWAv3.4_Noahv2.7.1_F036_ONLYSF_360000L_20110903_120000V.stat - - - - - &MET_BIN;/point_stat - - MASK_POLY_FILE &OUTPUT_DIR;/ref_config/gen_vx_mask/CONUS.nc - MODEL AFWAv3.4_Noahv2.7.1 - FCST_TIME 036 - - \ - &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv2.7.1/postprd/wrfprs_036.tm00 \ - &OUTPUT_DIR;/ref_config/pb2nc/NDAS_03h/20110903/prepbufr.ndas.20110903.t12z.tm12.nc \ - &CONFIG_DIR;/ref_config/PointStatConfig_WINDS \ - -outdir &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv2.7.1 \ - -v 2 - - - &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv2.7.1/point_stat_AFWAv3.4_Noahv2.7.1_F036_WINDS_360000L_20110903_120000V.stat - - - - - &MET_BIN;/point_stat - - MASK_POLY_FILE &OUTPUT_DIR;/ref_config/gen_vx_mask/CONUS.nc - MODEL AFWAv3.4_Noahv2.7.1 - FCST_TIME 048 - - \ - &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv2.7.1/postprd/wrfprs_048.tm00 \ - &OUTPUT_DIR;/ref_config/pb2nc/NDAS_03h/20110904/prepbufr.ndas.20110904.t00z.tm12.nc \ - &CONFIG_DIR;/ref_config/PointStatConfig_ADPUPA \ - -outdir &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv2.7.1 \ - -v 2 - - - &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv2.7.1/point_stat_AFWAv3.4_Noahv2.7.1_F048_ADPUPA_480000L_20110904_000000V.stat - - - - - &MET_BIN;/point_stat - - MASK_POLY_FILE &OUTPUT_DIR;/ref_config/gen_vx_mask/CONUS.nc - MODEL AFWAv3.4_Noahv2.7.1 - FCST_TIME 048 - - \ - &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv2.7.1/postprd/wrfprs_048.tm00 \ - &OUTPUT_DIR;/ref_config/pb2nc/NDAS_03h/20110904/prepbufr.ndas.20110904.t00z.tm12.nc \ - &CONFIG_DIR;/ref_config/PointStatConfig_ONLYSF \ - -outdir &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv2.7.1 \ - -v 2 - - - &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv2.7.1/point_stat_AFWAv3.4_Noahv2.7.1_F048_ONLYSF_480000L_20110904_000000V.stat - - - - - &MET_BIN;/point_stat - - MASK_POLY_FILE &OUTPUT_DIR;/ref_config/gen_vx_mask/CONUS.nc - MODEL AFWAv3.4_Noahv2.7.1 - FCST_TIME 048 - - \ - &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv2.7.1/postprd/wrfprs_048.tm00 \ - &OUTPUT_DIR;/ref_config/pb2nc/NDAS_03h/20110904/prepbufr.ndas.20110904.t00z.tm12.nc \ - &CONFIG_DIR;/ref_config/PointStatConfig_WINDS \ - -outdir &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv2.7.1 \ - -v 2 - - - &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv2.7.1/point_stat_AFWAv3.4_Noahv2.7.1_F048_WINDS_480000L_20110904_000000V.stat - - - - - &MET_BIN;/point_stat - - MASK_POLY_FILE &OUTPUT_DIR;/ref_config/gen_vx_mask/CONUS.nc - MODEL AFWAv3.4_Noahv3.3 - FCST_TIME 000 - - \ - &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_000.tm00 \ - &OUTPUT_DIR;/ref_config/pb2nc/NDAS_03h/20110902/prepbufr.ndas.20110902.t00z.tm12.nc \ - &CONFIG_DIR;/ref_config/PointStatConfig_ADPUPA \ - -outdir &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv3.3 \ - -v 2 - - - &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv3.3/point_stat_AFWAv3.4_Noahv3.3_F000_ADPUPA_000000L_20110902_000000V.stat - - - - - &MET_BIN;/point_stat - - MASK_POLY_FILE &OUTPUT_DIR;/ref_config/gen_vx_mask/CONUS.nc - MODEL AFWAv3.4_Noahv3.3 - FCST_TIME 000 - - \ - &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_000.tm00 \ - &OUTPUT_DIR;/ref_config/pb2nc/NDAS_03h/20110902/prepbufr.ndas.20110902.t00z.tm12.nc \ - &CONFIG_DIR;/ref_config/PointStatConfig_ONLYSF \ - -outdir &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv3.3 \ - -v 2 - - - &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv3.3/point_stat_AFWAv3.4_Noahv3.3_F000_ONLYSF_000000L_20110902_000000V.stat - - - - - &MET_BIN;/point_stat - - MASK_POLY_FILE &OUTPUT_DIR;/ref_config/gen_vx_mask/CONUS.nc - MODEL AFWAv3.4_Noahv3.3 - FCST_TIME 000 - - \ - &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_000.tm00 \ - &OUTPUT_DIR;/ref_config/pb2nc/NDAS_03h/20110902/prepbufr.ndas.20110902.t00z.tm12.nc \ - &CONFIG_DIR;/ref_config/PointStatConfig_WINDS \ - -outdir &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv3.3 \ - -v 2 - - - &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv3.3/point_stat_AFWAv3.4_Noahv3.3_F000_WINDS_000000L_20110902_000000V.stat - - - - - &MET_BIN;/point_stat - - MASK_POLY_FILE &OUTPUT_DIR;/ref_config/gen_vx_mask/CONUS.nc - MODEL AFWAv3.4_Noahv3.3 - FCST_TIME 012 - - \ - &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_012.tm00 \ - &OUTPUT_DIR;/ref_config/pb2nc/NDAS_03h/20110902/prepbufr.ndas.20110902.t12z.tm12.nc \ - &CONFIG_DIR;/ref_config/PointStatConfig_ADPUPA \ - -outdir &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv3.3 \ - -v 2 - - - &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv3.3/point_stat_AFWAv3.4_Noahv3.3_F012_ADPUPA_120000L_20110902_120000V.stat - - - - - &MET_BIN;/point_stat - - MASK_POLY_FILE &OUTPUT_DIR;/ref_config/gen_vx_mask/CONUS.nc - MODEL AFWAv3.4_Noahv3.3 - FCST_TIME 012 - - \ - &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_012.tm00 \ - &OUTPUT_DIR;/ref_config/pb2nc/NDAS_03h/20110902/prepbufr.ndas.20110902.t12z.tm12.nc \ - &CONFIG_DIR;/ref_config/PointStatConfig_ONLYSF \ - -outdir &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv3.3 \ - -v 2 - - - &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv3.3/point_stat_AFWAv3.4_Noahv3.3_F012_ONLYSF_120000L_20110902_120000V.stat - - - - - &MET_BIN;/point_stat - - MASK_POLY_FILE &OUTPUT_DIR;/ref_config/gen_vx_mask/CONUS.nc - MODEL AFWAv3.4_Noahv3.3 - FCST_TIME 012 - - \ - &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_012.tm00 \ - &OUTPUT_DIR;/ref_config/pb2nc/NDAS_03h/20110902/prepbufr.ndas.20110902.t12z.tm12.nc \ - &CONFIG_DIR;/ref_config/PointStatConfig_WINDS \ - -outdir &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv3.3 \ - -v 2 - - - &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv3.3/point_stat_AFWAv3.4_Noahv3.3_F012_WINDS_120000L_20110902_120000V.stat - - - - - &MET_BIN;/point_stat - - MASK_POLY_FILE &OUTPUT_DIR;/ref_config/gen_vx_mask/CONUS.nc - MODEL AFWAv3.4_Noahv3.3 - FCST_TIME 024 - - \ - &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_024.tm00 \ - &OUTPUT_DIR;/ref_config/pb2nc/NDAS_03h/20110903/prepbufr.ndas.20110903.t00z.tm12.nc \ - &CONFIG_DIR;/ref_config/PointStatConfig_ADPUPA \ - -outdir &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv3.3 \ - -v 2 - - - &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv3.3/point_stat_AFWAv3.4_Noahv3.3_F024_ADPUPA_240000L_20110903_000000V.stat - - - - - &MET_BIN;/point_stat - - MASK_POLY_FILE &OUTPUT_DIR;/ref_config/gen_vx_mask/CONUS.nc - MODEL AFWAv3.4_Noahv3.3 - FCST_TIME 024 - - \ - &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_024.tm00 \ - &OUTPUT_DIR;/ref_config/pb2nc/NDAS_03h/20110903/prepbufr.ndas.20110903.t00z.tm12.nc \ - &CONFIG_DIR;/ref_config/PointStatConfig_ONLYSF \ - -outdir &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv3.3 \ - -v 2 - - - &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv3.3/point_stat_AFWAv3.4_Noahv3.3_F024_ONLYSF_240000L_20110903_000000V.stat - - - - - &MET_BIN;/point_stat - - MASK_POLY_FILE &OUTPUT_DIR;/ref_config/gen_vx_mask/CONUS.nc - MODEL AFWAv3.4_Noahv3.3 - FCST_TIME 024 - - \ - &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_024.tm00 \ - &OUTPUT_DIR;/ref_config/pb2nc/NDAS_03h/20110903/prepbufr.ndas.20110903.t00z.tm12.nc \ - &CONFIG_DIR;/ref_config/PointStatConfig_WINDS \ - -outdir &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv3.3 \ - -v 2 - - - &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv3.3/point_stat_AFWAv3.4_Noahv3.3_F024_WINDS_240000L_20110903_000000V.stat - - - - - &MET_BIN;/point_stat - - MASK_POLY_FILE &OUTPUT_DIR;/ref_config/gen_vx_mask/CONUS.nc - MODEL AFWAv3.4_Noahv3.3 - FCST_TIME 036 - - \ - &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_036.tm00 \ - &OUTPUT_DIR;/ref_config/pb2nc/NDAS_03h/20110903/prepbufr.ndas.20110903.t12z.tm12.nc \ - &CONFIG_DIR;/ref_config/PointStatConfig_ADPUPA \ - -outdir &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv3.3 \ - -v 2 - - - &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv3.3/point_stat_AFWAv3.4_Noahv3.3_F036_ADPUPA_360000L_20110903_120000V.stat - - - - - &MET_BIN;/point_stat - - MASK_POLY_FILE &OUTPUT_DIR;/ref_config/gen_vx_mask/CONUS.nc - MODEL AFWAv3.4_Noahv3.3 - FCST_TIME 036 - - \ - &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_036.tm00 \ - &OUTPUT_DIR;/ref_config/pb2nc/NDAS_03h/20110903/prepbufr.ndas.20110903.t12z.tm12.nc \ - &CONFIG_DIR;/ref_config/PointStatConfig_ONLYSF \ - -outdir &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv3.3 \ - -v 2 - - - &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv3.3/point_stat_AFWAv3.4_Noahv3.3_F036_ONLYSF_360000L_20110903_120000V.stat - - - - - &MET_BIN;/point_stat - - MASK_POLY_FILE &OUTPUT_DIR;/ref_config/gen_vx_mask/CONUS.nc - MODEL AFWAv3.4_Noahv3.3 - FCST_TIME 036 - - \ - &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_036.tm00 \ - &OUTPUT_DIR;/ref_config/pb2nc/NDAS_03h/20110903/prepbufr.ndas.20110903.t12z.tm12.nc \ - &CONFIG_DIR;/ref_config/PointStatConfig_WINDS \ - -outdir &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv3.3 \ - -v 2 - - - &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv3.3/point_stat_AFWAv3.4_Noahv3.3_F036_WINDS_360000L_20110903_120000V.stat - - - - - &MET_BIN;/point_stat - - MASK_POLY_FILE &OUTPUT_DIR;/ref_config/gen_vx_mask/CONUS.nc - MODEL AFWAv3.4_Noahv3.3 - FCST_TIME 048 - - \ - &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_048.tm00 \ - &OUTPUT_DIR;/ref_config/pb2nc/NDAS_03h/20110904/prepbufr.ndas.20110904.t00z.tm12.nc \ - &CONFIG_DIR;/ref_config/PointStatConfig_ADPUPA \ - -outdir &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv3.3 \ - -v 2 - - - &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv3.3/point_stat_AFWAv3.4_Noahv3.3_F048_ADPUPA_480000L_20110904_000000V.stat - - - - - &MET_BIN;/point_stat - - MASK_POLY_FILE &OUTPUT_DIR;/ref_config/gen_vx_mask/CONUS.nc - MODEL AFWAv3.4_Noahv3.3 - FCST_TIME 048 - - \ - &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_048.tm00 \ - &OUTPUT_DIR;/ref_config/pb2nc/NDAS_03h/20110904/prepbufr.ndas.20110904.t00z.tm12.nc \ - &CONFIG_DIR;/ref_config/PointStatConfig_ONLYSF \ - -outdir &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv3.3 \ - -v 2 - - - &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv3.3/point_stat_AFWAv3.4_Noahv3.3_F048_ONLYSF_480000L_20110904_000000V.stat - - - - - &MET_BIN;/point_stat - - MASK_POLY_FILE &OUTPUT_DIR;/ref_config/gen_vx_mask/CONUS.nc - MODEL AFWAv3.4_Noahv3.3 - FCST_TIME 048 - - \ - &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_048.tm00 \ - &OUTPUT_DIR;/ref_config/pb2nc/NDAS_03h/20110904/prepbufr.ndas.20110904.t00z.tm12.nc \ - &CONFIG_DIR;/ref_config/PointStatConfig_WINDS \ - -outdir &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv3.3 \ - -v 2 - - - &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv3.3/point_stat_AFWAv3.4_Noahv3.3_F048_WINDS_480000L_20110904_000000V.stat - - - - - - - - - &MET_BIN;/grid_stat - - MASK_POLY_FILE &OUTPUT_DIR;/ref_config/gen_vx_mask/CONUS.nc - MODEL AFWAv3.4_Noahv3.3 - FCST_TIME 12 - - \ - &OUTPUT_DIR;/ref_config/pcp_combine/wrf/wrfpcp03_012.nc \ - &OUTPUT_DIR;/ref_config/pcp_combine/ST2_03h/20110902/ST2ml2011090212.03h.nc \ - &CONFIG_DIR;/ref_config/GridStatConfig_03h \ - -outdir &OUTPUT_DIR;/ref_config/grid_stat -v 1 - - - &OUTPUT_DIR;/ref_config/grid_stat/grid_stat_AFWAv3.4_Noahv3.3_F12_03h_120000L_20110902_120000V.stat - - - - - &MET_BIN;/grid_stat - - MASK_POLY_FILE &OUTPUT_DIR;/ref_config/gen_vx_mask/CONUS.nc - MODEL AFWAv3.4_Noahv3.3 - FCST_TIME 24 - - \ - &OUTPUT_DIR;/ref_config/pcp_combine/wrf/wrfpcp03_024.nc \ - &OUTPUT_DIR;/ref_config/pcp_combine/ST2_03h/20110903/ST2ml2011090300.03h.nc \ - &CONFIG_DIR;/ref_config/GridStatConfig_03h \ - -outdir &OUTPUT_DIR;/ref_config/grid_stat -v 1 - - - &OUTPUT_DIR;/ref_config/grid_stat/grid_stat_AFWAv3.4_Noahv3.3_F24_03h_240000L_20110903_000000V.stat - - - - - &MET_BIN;/grid_stat - - MASK_POLY_FILE &OUTPUT_DIR;/ref_config/gen_vx_mask/CONUS.nc - MODEL AFWAv3.4_Noahv3.3 - FCST_TIME 36 - - \ - &OUTPUT_DIR;/ref_config/pcp_combine/wrf/wrfpcp03_036.nc \ - &OUTPUT_DIR;/ref_config/pcp_combine/ST2_03h/20110903/ST2ml2011090312.03h.nc \ - &CONFIG_DIR;/ref_config/GridStatConfig_03h \ - -outdir &OUTPUT_DIR;/ref_config/grid_stat -v 1 - - - &OUTPUT_DIR;/ref_config/grid_stat/grid_stat_AFWAv3.4_Noahv3.3_F36_03h_360000L_20110903_120000V.stat - - - - - &MET_BIN;/grid_stat - - MASK_POLY_FILE &OUTPUT_DIR;/ref_config/gen_vx_mask/CONUS.nc - MODEL AFWAv3.4_Noahv3.3 - FCST_TIME 48 - - \ - &OUTPUT_DIR;/ref_config/pcp_combine/wrf/wrfpcp03_048.nc \ - &OUTPUT_DIR;/ref_config/pcp_combine/ST2_03h/20110904/ST2ml2011090400.03h.nc \ - &CONFIG_DIR;/ref_config/GridStatConfig_03h \ - -outdir &OUTPUT_DIR;/ref_config/grid_stat -v 1 - - - &OUTPUT_DIR;/ref_config/grid_stat/grid_stat_AFWAv3.4_Noahv3.3_F48_03h_480000L_20110904_000000V.stat - - - - - &MET_BIN;/grid_stat - - MASK_POLY_FILE &OUTPUT_DIR;/ref_config/gen_vx_mask/CONUS.nc - MODEL AFWAv3.4_Noahv3.3 - FCST_TIME 36 - - \ - &OUTPUT_DIR;/ref_config/pcp_combine/wrf/wrfpcp24_036.nc \ - &OUTPUT_DIR;/ref_config/pcp_combine/ST2_24h/20110903/ST2ml2011090312.24h.nc \ - &CONFIG_DIR;/ref_config/GridStatConfig_24h \ - -outdir &OUTPUT_DIR;/ref_config/grid_stat -v 1 - - - &OUTPUT_DIR;/ref_config/grid_stat/grid_stat_AFWAv3.4_Noahv3.3_F36_24h_360000L_20110903_120000V.stat - - - @@ -942,8 +25,8 @@ &MET_BIN;/stat_analysis \ - -lookin &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv3.3/ \ - -lookin &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv2.7.1/ \ + -lookin &OUTPUT_DIR;/ref_config_lead_*/point_stat/AFWAv3.4_Noahv3.3 \ + -lookin &OUTPUT_DIR;/ref_config_lead_*/point_stat/AFWAv3.4_Noahv2.7.1 \ -job go_index -fcst_init_beg 2011090200 -fcst_init_end 2011090200 \ -model AFWAv3.4_Noahv3.3 -model AFWAv3.4_Noahv2.7.1 \ -vx_mask FULL -interp_mthd BILIN \ @@ -962,8 +45,8 @@ &MET_BIN;/stat_analysis \ - -lookin &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv3.3/ \ - -lookin &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv2.7.1/ \ + -lookin &OUTPUT_DIR;/ref_config_lead_*/point_stat/AFWAv3.4_Noahv3.3 \ + -lookin &OUTPUT_DIR;/ref_config_lead_*/point_stat/AFWAv3.4_Noahv2.7.1 \ -job go_index \ -by FCST_INIT_BEG,VX_MASK,OBTYPE -set_hdr DESC Noahv3.3_vs_v2.7.1 \ -model AFWAv3.4_Noahv3.3,AFWAv3.4_Noahv2.7.1 -ss_index_vld_thresh 0.5 \ @@ -981,8 +64,8 @@ &MET_BIN;/stat_analysis \ - -lookin &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv3.3/ \ - -lookin &OUTPUT_DIR;/ref_config/point_stat/AFWAv3.4_Noahv2.7.1/ \ + -lookin &OUTPUT_DIR;/ref_config_lead_*/point_stat/AFWAv3.4_Noahv3.3 \ + -lookin &OUTPUT_DIR;/ref_config_lead_*/point_stat/AFWAv3.4_Noahv2.7.1 \ -job ss_index -config &CONFIG_DIR;/STATAnalysisConfig_SFC_SS_Index \ -by FCST_INIT_BEG,VX_MASK,OBTYPE \ -out_stat &OUTPUT_DIR;/ref_config/stat_analysis/sfc_ss_index_by_option.stat diff --git a/test/xml/unit_ref_config_lead_00.xml b/test/xml/unit_ref_config_lead_00.xml new file mode 100644 index 0000000000..ac65be1546 --- /dev/null +++ b/test/xml/unit_ref_config_lead_00.xml @@ -0,0 +1,178 @@ + + + + + + + + + + +]> + + + + + + &TEST_DIR; + true + + + + + + + &MET_BIN;/gen_vx_mask + \ + &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_012.tm00 \ + &MET_BASE;/poly/CONUS.poly \ + &OUTPUT_DIR;/ref_config_lead_00/gen_vx_mask/CONUS.nc \ + -type poly -v 2 + + + &OUTPUT_DIR;/ref_config_lead_00/gen_vx_mask/CONUS.nc + + + + + + + + + &MET_BIN;/pb2nc + + MASK_POLY &CONFIG_DIR;/ref_config/RefConfig.poly + + \ + &DATA_DIR_OBS;/ref_config/prepbufr/ndas/2011090212/ndas.t12z.prepbufr.tm12.nr \ + &OUTPUT_DIR;/ref_config_lead_00/pb2nc/NDAS_03h/20110902/prepbufr.ndas.20110902.t00z.tm12.nc \ + &CONFIG_DIR;/ref_config/PB2NCConfig \ + -valid_beg 20110901_223000\ + -valid_end 20110902_013000\ + -v 2 + + + &OUTPUT_DIR;/ref_config_lead_00/pb2nc/NDAS_03h/20110902/prepbufr.ndas.20110902.t00z.tm12.nc + + + + + + + + + &MET_BIN;/point_stat + + MASK_POLY_FILE &OUTPUT_DIR;/ref_config_lead_00/gen_vx_mask/CONUS.nc + MODEL AFWAv3.4_Noahv2.7.1 + FCST_TIME 000 + + \ + &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv2.7.1/postprd/wrfprs_000.tm00 \ + &OUTPUT_DIR;/ref_config_lead_00/pb2nc/NDAS_03h/20110902/prepbufr.ndas.20110902.t00z.tm12.nc \ + &CONFIG_DIR;/ref_config/PointStatConfig_ADPUPA \ + -outdir &OUTPUT_DIR;/ref_config_lead_00/point_stat/AFWAv3.4_Noahv2.7.1 \ + -v 2 + + + &OUTPUT_DIR;/ref_config_lead_00/point_stat/AFWAv3.4_Noahv2.7.1/point_stat_AFWAv3.4_Noahv2.7.1_F000_ADPUPA_000000L_20110902_000000V.stat + + + + + &MET_BIN;/point_stat + + MASK_POLY_FILE &OUTPUT_DIR;/ref_config_lead_00/gen_vx_mask/CONUS.nc + MODEL AFWAv3.4_Noahv2.7.1 + FCST_TIME 000 + + \ + &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv2.7.1/postprd/wrfprs_000.tm00 \ + &OUTPUT_DIR;/ref_config_lead_00/pb2nc/NDAS_03h/20110902/prepbufr.ndas.20110902.t00z.tm12.nc \ + &CONFIG_DIR;/ref_config/PointStatConfig_ONLYSF \ + -outdir &OUTPUT_DIR;/ref_config_lead_00/point_stat/AFWAv3.4_Noahv2.7.1 \ + -v 2 + + + &OUTPUT_DIR;/ref_config_lead_00/point_stat/AFWAv3.4_Noahv2.7.1/point_stat_AFWAv3.4_Noahv2.7.1_F000_ONLYSF_000000L_20110902_000000V.stat + + + + + &MET_BIN;/point_stat + + MASK_POLY_FILE &OUTPUT_DIR;/ref_config_lead_00/gen_vx_mask/CONUS.nc + MODEL AFWAv3.4_Noahv2.7.1 + FCST_TIME 000 + + \ + &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv2.7.1/postprd/wrfprs_000.tm00 \ + &OUTPUT_DIR;/ref_config_lead_00/pb2nc/NDAS_03h/20110902/prepbufr.ndas.20110902.t00z.tm12.nc \ + &CONFIG_DIR;/ref_config/PointStatConfig_WINDS \ + -outdir &OUTPUT_DIR;/ref_config_lead_00/point_stat/AFWAv3.4_Noahv2.7.1 \ + -v 2 + + + &OUTPUT_DIR;/ref_config_lead_00/point_stat/AFWAv3.4_Noahv2.7.1/point_stat_AFWAv3.4_Noahv2.7.1_F000_WINDS_000000L_20110902_000000V.stat + + + + + &MET_BIN;/point_stat + + MASK_POLY_FILE &OUTPUT_DIR;/ref_config_lead_00/gen_vx_mask/CONUS.nc + MODEL AFWAv3.4_Noahv3.3 + FCST_TIME 000 + + \ + &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_000.tm00 \ + &OUTPUT_DIR;/ref_config_lead_00/pb2nc/NDAS_03h/20110902/prepbufr.ndas.20110902.t00z.tm12.nc \ + &CONFIG_DIR;/ref_config/PointStatConfig_ADPUPA \ + -outdir &OUTPUT_DIR;/ref_config_lead_00/point_stat/AFWAv3.4_Noahv3.3 \ + -v 2 + + + &OUTPUT_DIR;/ref_config_lead_00/point_stat/AFWAv3.4_Noahv3.3/point_stat_AFWAv3.4_Noahv3.3_F000_ADPUPA_000000L_20110902_000000V.stat + + + + + &MET_BIN;/point_stat + + MASK_POLY_FILE &OUTPUT_DIR;/ref_config_lead_00/gen_vx_mask/CONUS.nc + MODEL AFWAv3.4_Noahv3.3 + FCST_TIME 000 + + \ + &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_000.tm00 \ + &OUTPUT_DIR;/ref_config_lead_00/pb2nc/NDAS_03h/20110902/prepbufr.ndas.20110902.t00z.tm12.nc \ + &CONFIG_DIR;/ref_config/PointStatConfig_ONLYSF \ + -outdir &OUTPUT_DIR;/ref_config_lead_00/point_stat/AFWAv3.4_Noahv3.3 \ + -v 2 + + + &OUTPUT_DIR;/ref_config_lead_00/point_stat/AFWAv3.4_Noahv3.3/point_stat_AFWAv3.4_Noahv3.3_F000_ONLYSF_000000L_20110902_000000V.stat + + + + + &MET_BIN;/point_stat + + MASK_POLY_FILE &OUTPUT_DIR;/ref_config_lead_00/gen_vx_mask/CONUS.nc + MODEL AFWAv3.4_Noahv3.3 + FCST_TIME 000 + + \ + &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_000.tm00 \ + &OUTPUT_DIR;/ref_config_lead_00/pb2nc/NDAS_03h/20110902/prepbufr.ndas.20110902.t00z.tm12.nc \ + &CONFIG_DIR;/ref_config/PointStatConfig_WINDS \ + -outdir &OUTPUT_DIR;/ref_config_lead_00/point_stat/AFWAv3.4_Noahv3.3 \ + -v 2 + + + &OUTPUT_DIR;/ref_config_lead_00/point_stat/AFWAv3.4_Noahv3.3/point_stat_AFWAv3.4_Noahv3.3_F000_WINDS_000000L_20110902_000000V.stat + + + + diff --git a/test/xml/unit_ref_config_lead_12.xml b/test/xml/unit_ref_config_lead_12.xml new file mode 100644 index 0000000000..989e548da5 --- /dev/null +++ b/test/xml/unit_ref_config_lead_12.xml @@ -0,0 +1,230 @@ + + + + + + + + + + +]> + + + + + + &TEST_DIR; + true + + + + + + + &MET_BIN;/gen_vx_mask + \ + &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_012.tm00 \ + &MET_BASE;/poly/CONUS.poly \ + &OUTPUT_DIR;/ref_config_lead_12/gen_vx_mask/CONUS.nc \ + -type poly -v 2 + + + &OUTPUT_DIR;/ref_config_lead_12/gen_vx_mask/CONUS.nc + + + + + + + + + &MET_BIN;/pb2nc + + MASK_POLY &CONFIG_DIR;/ref_config/RefConfig.poly + + \ + &DATA_DIR_OBS;/ref_config/prepbufr/ndas/2011090300/ndas.t00z.prepbufr.tm12.nr \ + &OUTPUT_DIR;/ref_config_lead_12/pb2nc/NDAS_03h/20110902/prepbufr.ndas.20110902.t12z.tm12.nc \ + &CONFIG_DIR;/ref_config/PB2NCConfig \ + -valid_beg 20110902_103000\ + -valid_end 20110902_133000\ + -v 2 + + + &OUTPUT_DIR;/ref_config_lead_12/pb2nc/NDAS_03h/20110902/prepbufr.ndas.20110902.t12z.tm12.nc + + + + + + + + + &MET_BIN;/pcp_combine + \ + -sum \ + 00000000_000000 1 20110902_120000 3 \ + &OUTPUT_DIR;/ref_config_lead_12/pcp_combine/ST2_03h/20110902/ST2ml2011090212.03h.nc \ + -pcpdir &DATA_DIR_OBS;/ref_config/pcp_combine/20110902 + + + &OUTPUT_DIR;/ref_config_lead_12/pcp_combine/ST2_03h/20110902/ST2ml2011090212.03h.nc + + + + + &MET_BIN;/pcp_combine + \ + -subtract \ + &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_012.tm00 12 \ + &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_009.tm00 9 \ + &OUTPUT_DIR;/ref_config_lead_12/pcp_combine/wrf/wrfpcp03_012.nc + + + &OUTPUT_DIR;/ref_config_lead_12/pcp_combine/wrf/wrfpcp03_012.nc + + + + + + + + + &MET_BIN;/point_stat + + MASK_POLY_FILE &OUTPUT_DIR;/ref_config_lead_12/gen_vx_mask/CONUS.nc + MODEL AFWAv3.4_Noahv2.7.1 + FCST_TIME 012 + + \ + &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv2.7.1/postprd/wrfprs_012.tm00 \ + &OUTPUT_DIR;/ref_config_lead_12/pb2nc/NDAS_03h/20110902/prepbufr.ndas.20110902.t12z.tm12.nc \ + &CONFIG_DIR;/ref_config/PointStatConfig_ADPUPA \ + -outdir &OUTPUT_DIR;/ref_config_lead_12/point_stat/AFWAv3.4_Noahv2.7.1 \ + -v 2 + + + &OUTPUT_DIR;/ref_config_lead_12/point_stat/AFWAv3.4_Noahv2.7.1/point_stat_AFWAv3.4_Noahv2.7.1_F012_ADPUPA_120000L_20110902_120000V.stat + + + + + &MET_BIN;/point_stat + + MASK_POLY_FILE &OUTPUT_DIR;/ref_config_lead_12/gen_vx_mask/CONUS.nc + MODEL AFWAv3.4_Noahv2.7.1 + FCST_TIME 012 + + \ + &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv2.7.1/postprd/wrfprs_012.tm00 \ + &OUTPUT_DIR;/ref_config_lead_12/pb2nc/NDAS_03h/20110902/prepbufr.ndas.20110902.t12z.tm12.nc \ + &CONFIG_DIR;/ref_config/PointStatConfig_ONLYSF \ + -outdir &OUTPUT_DIR;/ref_config_lead_12/point_stat/AFWAv3.4_Noahv2.7.1 \ + -v 2 + + + &OUTPUT_DIR;/ref_config_lead_12/point_stat/AFWAv3.4_Noahv2.7.1/point_stat_AFWAv3.4_Noahv2.7.1_F012_ONLYSF_120000L_20110902_120000V.stat + + + + + &MET_BIN;/point_stat + + MASK_POLY_FILE &OUTPUT_DIR;/ref_config_lead_12/gen_vx_mask/CONUS.nc + MODEL AFWAv3.4_Noahv2.7.1 + FCST_TIME 012 + + \ + &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv2.7.1/postprd/wrfprs_012.tm00 \ + &OUTPUT_DIR;/ref_config_lead_12/pb2nc/NDAS_03h/20110902/prepbufr.ndas.20110902.t12z.tm12.nc \ + &CONFIG_DIR;/ref_config/PointStatConfig_WINDS \ + -outdir &OUTPUT_DIR;/ref_config_lead_12/point_stat/AFWAv3.4_Noahv2.7.1 \ + -v 2 + + + &OUTPUT_DIR;/ref_config_lead_12/point_stat/AFWAv3.4_Noahv2.7.1/point_stat_AFWAv3.4_Noahv2.7.1_F012_WINDS_120000L_20110902_120000V.stat + + + + + &MET_BIN;/point_stat + + MASK_POLY_FILE &OUTPUT_DIR;/ref_config_lead_12/gen_vx_mask/CONUS.nc + MODEL AFWAv3.4_Noahv3.3 + FCST_TIME 012 + + \ + &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_012.tm00 \ + &OUTPUT_DIR;/ref_config_lead_12/pb2nc/NDAS_03h/20110902/prepbufr.ndas.20110902.t12z.tm12.nc \ + &CONFIG_DIR;/ref_config/PointStatConfig_ADPUPA \ + -outdir &OUTPUT_DIR;/ref_config_lead_12/point_stat/AFWAv3.4_Noahv3.3 \ + -v 2 + + + &OUTPUT_DIR;/ref_config_lead_12/point_stat/AFWAv3.4_Noahv3.3/point_stat_AFWAv3.4_Noahv3.3_F012_ADPUPA_120000L_20110902_120000V.stat + + + + + &MET_BIN;/point_stat + + MASK_POLY_FILE &OUTPUT_DIR;/ref_config_lead_12/gen_vx_mask/CONUS.nc + MODEL AFWAv3.4_Noahv3.3 + FCST_TIME 012 + + \ + &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_012.tm00 \ + &OUTPUT_DIR;/ref_config_lead_12/pb2nc/NDAS_03h/20110902/prepbufr.ndas.20110902.t12z.tm12.nc \ + &CONFIG_DIR;/ref_config/PointStatConfig_ONLYSF \ + -outdir &OUTPUT_DIR;/ref_config_lead_12/point_stat/AFWAv3.4_Noahv3.3 \ + -v 2 + + + &OUTPUT_DIR;/ref_config_lead_12/point_stat/AFWAv3.4_Noahv3.3/point_stat_AFWAv3.4_Noahv3.3_F012_ONLYSF_120000L_20110902_120000V.stat + + + + + &MET_BIN;/point_stat + + MASK_POLY_FILE &OUTPUT_DIR;/ref_config_lead_12/gen_vx_mask/CONUS.nc + MODEL AFWAv3.4_Noahv3.3 + FCST_TIME 012 + + \ + &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_012.tm00 \ + &OUTPUT_DIR;/ref_config_lead_12/pb2nc/NDAS_03h/20110902/prepbufr.ndas.20110902.t12z.tm12.nc \ + &CONFIG_DIR;/ref_config/PointStatConfig_WINDS \ + -outdir &OUTPUT_DIR;/ref_config_lead_12/point_stat/AFWAv3.4_Noahv3.3 \ + -v 2 + + + &OUTPUT_DIR;/ref_config_lead_12/point_stat/AFWAv3.4_Noahv3.3/point_stat_AFWAv3.4_Noahv3.3_F012_WINDS_120000L_20110902_120000V.stat + + + + + + + + + &MET_BIN;/grid_stat + + MASK_POLY_FILE &OUTPUT_DIR;/ref_config_lead_12/gen_vx_mask/CONUS.nc + MODEL AFWAv3.4_Noahv3.3 + FCST_TIME 12 + + \ + &OUTPUT_DIR;/ref_config_lead_12/pcp_combine/wrf/wrfpcp03_012.nc \ + &OUTPUT_DIR;/ref_config_lead_12/pcp_combine/ST2_03h/20110902/ST2ml2011090212.03h.nc \ + &CONFIG_DIR;/ref_config/GridStatConfig_03h \ + -outdir &OUTPUT_DIR;/ref_config_lead_12/grid_stat -v 1 + + + &OUTPUT_DIR;/ref_config_lead_12/grid_stat/grid_stat_AFWAv3.4_Noahv3.3_F12_03h_120000L_20110902_120000V.stat + + + + diff --git a/test/xml/unit_ref_config_lead_24.xml b/test/xml/unit_ref_config_lead_24.xml new file mode 100644 index 0000000000..78cff2ee56 --- /dev/null +++ b/test/xml/unit_ref_config_lead_24.xml @@ -0,0 +1,231 @@ + + + + + + + + + + +]> + + + + + + &TEST_DIR; + true + + + + + + + &MET_BIN;/gen_vx_mask + \ + &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_012.tm00 \ + &MET_BASE;/poly/CONUS.poly \ + &OUTPUT_DIR;/ref_config_lead_24/gen_vx_mask/CONUS.nc \ + -type poly -v 2 + + + &OUTPUT_DIR;/ref_config_lead_24/gen_vx_mask/CONUS.nc + + + + + + + + + &MET_BIN;/pb2nc + + MASK_POLY &CONFIG_DIR;/ref_config/RefConfig.poly + + \ + &DATA_DIR_OBS;/ref_config/prepbufr/ndas/2011090312/ndas.t12z.prepbufr.tm12.nr \ + &OUTPUT_DIR;/ref_config_lead_24/pb2nc/NDAS_03h/20110903/prepbufr.ndas.20110903.t00z.tm12.nc \ + &CONFIG_DIR;/ref_config/PB2NCConfig \ + -valid_beg 20110902_223000\ + -valid_end 20110903_013000\ + -v 2 + + + &OUTPUT_DIR;/ref_config_lead_24/pb2nc/NDAS_03h/20110903/prepbufr.ndas.20110903.t00z.tm12.nc + + + + + + + + + &MET_BIN;/pcp_combine + \ + -sum \ + 00000000_000000 1 20110903_000000 3 \ + &OUTPUT_DIR;/ref_config_lead_24/pcp_combine/ST2_03h/20110903/ST2ml2011090300.03h.nc \ + -pcpdir &DATA_DIR_OBS;/ref_config/pcp_combine/20110902 \ + -pcpdir &DATA_DIR_OBS;/ref_config/pcp_combine/20110903 + + + &OUTPUT_DIR;/ref_config_lead_24/pcp_combine/ST2_03h/20110903/ST2ml2011090300.03h.nc + + + + + &MET_BIN;/pcp_combine + \ + -subtract \ + &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_024.tm00 24 \ + &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_021.tm00 21 \ + &OUTPUT_DIR;/ref_config_lead_24/pcp_combine/wrf/wrfpcp03_024.nc + + + &OUTPUT_DIR;/ref_config_lead_24/pcp_combine/wrf/wrfpcp03_024.nc + + + + + + + + + &MET_BIN;/point_stat + + MASK_POLY_FILE &OUTPUT_DIR;/ref_config_lead_24/gen_vx_mask/CONUS.nc + MODEL AFWAv3.4_Noahv2.7.1 + FCST_TIME 024 + + \ + &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv2.7.1/postprd/wrfprs_024.tm00 \ + &OUTPUT_DIR;/ref_config_lead_24/pb2nc/NDAS_03h/20110903/prepbufr.ndas.20110903.t00z.tm12.nc \ + &CONFIG_DIR;/ref_config/PointStatConfig_ADPUPA \ + -outdir &OUTPUT_DIR;/ref_config_lead_24/point_stat/AFWAv3.4_Noahv2.7.1 \ + -v 2 + + + &OUTPUT_DIR;/ref_config_lead_24/point_stat/AFWAv3.4_Noahv2.7.1/point_stat_AFWAv3.4_Noahv2.7.1_F024_ADPUPA_240000L_20110903_000000V.stat + + + + + &MET_BIN;/point_stat + + MASK_POLY_FILE &OUTPUT_DIR;/ref_config_lead_24/gen_vx_mask/CONUS.nc + MODEL AFWAv3.4_Noahv2.7.1 + FCST_TIME 024 + + \ + &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv2.7.1/postprd/wrfprs_024.tm00 \ + &OUTPUT_DIR;/ref_config_lead_24/pb2nc/NDAS_03h/20110903/prepbufr.ndas.20110903.t00z.tm12.nc \ + &CONFIG_DIR;/ref_config/PointStatConfig_ONLYSF \ + -outdir &OUTPUT_DIR;/ref_config_lead_24/point_stat/AFWAv3.4_Noahv2.7.1 \ + -v 2 + + + &OUTPUT_DIR;/ref_config_lead_24/point_stat/AFWAv3.4_Noahv2.7.1/point_stat_AFWAv3.4_Noahv2.7.1_F024_ONLYSF_240000L_20110903_000000V.stat + + + + + &MET_BIN;/point_stat + + MASK_POLY_FILE &OUTPUT_DIR;/ref_config_lead_24/gen_vx_mask/CONUS.nc + MODEL AFWAv3.4_Noahv2.7.1 + FCST_TIME 024 + + \ + &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv2.7.1/postprd/wrfprs_024.tm00 \ + &OUTPUT_DIR;/ref_config_lead_24/pb2nc/NDAS_03h/20110903/prepbufr.ndas.20110903.t00z.tm12.nc \ + &CONFIG_DIR;/ref_config/PointStatConfig_WINDS \ + -outdir &OUTPUT_DIR;/ref_config_lead_24/point_stat/AFWAv3.4_Noahv2.7.1 \ + -v 2 + + + &OUTPUT_DIR;/ref_config_lead_24/point_stat/AFWAv3.4_Noahv2.7.1/point_stat_AFWAv3.4_Noahv2.7.1_F024_WINDS_240000L_20110903_000000V.stat + + + + + &MET_BIN;/point_stat + + MASK_POLY_FILE &OUTPUT_DIR;/ref_config_lead_24/gen_vx_mask/CONUS.nc + MODEL AFWAv3.4_Noahv3.3 + FCST_TIME 024 + + \ + &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_024.tm00 \ + &OUTPUT_DIR;/ref_config_lead_24/pb2nc/NDAS_03h/20110903/prepbufr.ndas.20110903.t00z.tm12.nc \ + &CONFIG_DIR;/ref_config/PointStatConfig_ADPUPA \ + -outdir &OUTPUT_DIR;/ref_config_lead_24/point_stat/AFWAv3.4_Noahv3.3 \ + -v 2 + + + &OUTPUT_DIR;/ref_config_lead_24/point_stat/AFWAv3.4_Noahv3.3/point_stat_AFWAv3.4_Noahv3.3_F024_ADPUPA_240000L_20110903_000000V.stat + + + + + &MET_BIN;/point_stat + + MASK_POLY_FILE &OUTPUT_DIR;/ref_config_lead_24/gen_vx_mask/CONUS.nc + MODEL AFWAv3.4_Noahv3.3 + FCST_TIME 024 + + \ + &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_024.tm00 \ + &OUTPUT_DIR;/ref_config_lead_24/pb2nc/NDAS_03h/20110903/prepbufr.ndas.20110903.t00z.tm12.nc \ + &CONFIG_DIR;/ref_config/PointStatConfig_ONLYSF \ + -outdir &OUTPUT_DIR;/ref_config_lead_24/point_stat/AFWAv3.4_Noahv3.3 \ + -v 2 + + + &OUTPUT_DIR;/ref_config_lead_24/point_stat/AFWAv3.4_Noahv3.3/point_stat_AFWAv3.4_Noahv3.3_F024_ONLYSF_240000L_20110903_000000V.stat + + + + + &MET_BIN;/point_stat + + MASK_POLY_FILE &OUTPUT_DIR;/ref_config_lead_24/gen_vx_mask/CONUS.nc + MODEL AFWAv3.4_Noahv3.3 + FCST_TIME 024 + + \ + &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_024.tm00 \ + &OUTPUT_DIR;/ref_config_lead_24/pb2nc/NDAS_03h/20110903/prepbufr.ndas.20110903.t00z.tm12.nc \ + &CONFIG_DIR;/ref_config/PointStatConfig_WINDS \ + -outdir &OUTPUT_DIR;/ref_config_lead_24/point_stat/AFWAv3.4_Noahv3.3 \ + -v 2 + + + &OUTPUT_DIR;/ref_config_lead_24/point_stat/AFWAv3.4_Noahv3.3/point_stat_AFWAv3.4_Noahv3.3_F024_WINDS_240000L_20110903_000000V.stat + + + + + + + + + &MET_BIN;/grid_stat + + MASK_POLY_FILE &OUTPUT_DIR;/ref_config_lead_24/gen_vx_mask/CONUS.nc + MODEL AFWAv3.4_Noahv3.3 + FCST_TIME 24 + + \ + &OUTPUT_DIR;/ref_config_lead_24/pcp_combine/wrf/wrfpcp03_024.nc \ + &OUTPUT_DIR;/ref_config_lead_24/pcp_combine/ST2_03h/20110903/ST2ml2011090300.03h.nc \ + &CONFIG_DIR;/ref_config/GridStatConfig_03h \ + -outdir &OUTPUT_DIR;/ref_config_lead_24/grid_stat -v 1 + + + &OUTPUT_DIR;/ref_config_lead_24/grid_stat/grid_stat_AFWAv3.4_Noahv3.3_F24_03h_240000L_20110903_000000V.stat + + + + diff --git a/test/xml/unit_ref_config_lead_36.xml b/test/xml/unit_ref_config_lead_36.xml new file mode 100644 index 0000000000..4832ba4ad1 --- /dev/null +++ b/test/xml/unit_ref_config_lead_36.xml @@ -0,0 +1,274 @@ + + + + + + + + + + +]> + + + + + + &TEST_DIR; + true + + + + + + + &MET_BIN;/gen_vx_mask + \ + &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_012.tm00 \ + &MET_BASE;/poly/CONUS.poly \ + &OUTPUT_DIR;/ref_config_lead_36/gen_vx_mask/CONUS.nc \ + -type poly -v 2 + + + &OUTPUT_DIR;/ref_config_lead_36/gen_vx_mask/CONUS.nc + + + + + + + + + &MET_BIN;/pb2nc + + MASK_POLY &CONFIG_DIR;/ref_config/RefConfig.poly + + \ + &DATA_DIR_OBS;/ref_config/prepbufr/ndas/2011090400/ndas.t00z.prepbufr.tm12.nr \ + &OUTPUT_DIR;/ref_config_lead_36/pb2nc/NDAS_03h/20110903/prepbufr.ndas.20110903.t12z.tm12.nc \ + &CONFIG_DIR;/ref_config/PB2NCConfig \ + -valid_beg 20110903_103000\ + -valid_end 20110903_133000\ + -v 2 + + + &OUTPUT_DIR;/ref_config_lead_36/pb2nc/NDAS_03h/20110903/prepbufr.ndas.20110903.t12z.tm12.nc + + + + + + + + + &MET_BIN;/pcp_combine + \ + -sum \ + 00000000_000000 1 20110903_120000 3 \ + &OUTPUT_DIR;/ref_config_lead_36/pcp_combine/ST2_03h/20110903/ST2ml2011090312.03h.nc \ + -pcpdir &DATA_DIR_OBS;/ref_config/pcp_combine/20110903 + + + &OUTPUT_DIR;/ref_config_lead_36/pcp_combine/ST2_03h/20110903/ST2ml2011090312.03h.nc + + + + + &MET_BIN;/pcp_combine + \ + -add \ + &DATA_DIR_OBS;/ref_config/pcp_combine/2011090324/ST2ml2011090312.24h.grb \ + 24 \ + &OUTPUT_DIR;/ref_config_lead_36/pcp_combine/ST2_24h/20110903/ST2ml2011090312.24h.nc + + + &OUTPUT_DIR;/ref_config_lead_36/pcp_combine/ST2_24h/20110903/ST2ml2011090312.24h.nc + + + + + &MET_BIN;/pcp_combine + \ + -subtract \ + &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_036.tm00 36 \ + &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_033.tm00 33 \ + &OUTPUT_DIR;/ref_config_lead_36/pcp_combine/wrf/wrfpcp03_036.nc + + + &OUTPUT_DIR;/ref_config_lead_36/pcp_combine/wrf/wrfpcp03_036.nc + + + + + &MET_BIN;/pcp_combine + \ + -subtract \ + &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_036.tm00 36 \ + &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_012.tm00 12 \ + &OUTPUT_DIR;/ref_config_lead_36/pcp_combine/wrf/wrfpcp24_036.nc + + + &OUTPUT_DIR;/ref_config_lead_36/pcp_combine/wrf/wrfpcp24_036.nc + + + + + + + + + &MET_BIN;/point_stat + + MASK_POLY_FILE &OUTPUT_DIR;/ref_config_lead_36/gen_vx_mask/CONUS.nc + MODEL AFWAv3.4_Noahv2.7.1 + FCST_TIME 036 + + \ + &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv2.7.1/postprd/wrfprs_036.tm00 \ + &OUTPUT_DIR;/ref_config_lead_36/pb2nc/NDAS_03h/20110903/prepbufr.ndas.20110903.t12z.tm12.nc \ + &CONFIG_DIR;/ref_config/PointStatConfig_ADPUPA \ + -outdir &OUTPUT_DIR;/ref_config_lead_36/point_stat/AFWAv3.4_Noahv2.7.1 \ + -v 2 + + + &OUTPUT_DIR;/ref_config_lead_36/point_stat/AFWAv3.4_Noahv2.7.1/point_stat_AFWAv3.4_Noahv2.7.1_F036_ADPUPA_360000L_20110903_120000V.stat + + + + + &MET_BIN;/point_stat + + MASK_POLY_FILE &OUTPUT_DIR;/ref_config_lead_36/gen_vx_mask/CONUS.nc + MODEL AFWAv3.4_Noahv2.7.1 + FCST_TIME 036 + + \ + &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv2.7.1/postprd/wrfprs_036.tm00 \ + &OUTPUT_DIR;/ref_config_lead_36/pb2nc/NDAS_03h/20110903/prepbufr.ndas.20110903.t12z.tm12.nc \ + &CONFIG_DIR;/ref_config/PointStatConfig_ONLYSF \ + -outdir &OUTPUT_DIR;/ref_config_lead_36/point_stat/AFWAv3.4_Noahv2.7.1 \ + -v 2 + + + &OUTPUT_DIR;/ref_config_lead_36/point_stat/AFWAv3.4_Noahv2.7.1/point_stat_AFWAv3.4_Noahv2.7.1_F036_ONLYSF_360000L_20110903_120000V.stat + + + + + &MET_BIN;/point_stat + + MASK_POLY_FILE &OUTPUT_DIR;/ref_config_lead_36/gen_vx_mask/CONUS.nc + MODEL AFWAv3.4_Noahv2.7.1 + FCST_TIME 036 + + \ + &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv2.7.1/postprd/wrfprs_036.tm00 \ + &OUTPUT_DIR;/ref_config_lead_36/pb2nc/NDAS_03h/20110903/prepbufr.ndas.20110903.t12z.tm12.nc \ + &CONFIG_DIR;/ref_config/PointStatConfig_WINDS \ + -outdir &OUTPUT_DIR;/ref_config_lead_36/point_stat/AFWAv3.4_Noahv2.7.1 \ + -v 2 + + + &OUTPUT_DIR;/ref_config_lead_36/point_stat/AFWAv3.4_Noahv2.7.1/point_stat_AFWAv3.4_Noahv2.7.1_F036_WINDS_360000L_20110903_120000V.stat + + + + + &MET_BIN;/point_stat + + MASK_POLY_FILE &OUTPUT_DIR;/ref_config_lead_36/gen_vx_mask/CONUS.nc + MODEL AFWAv3.4_Noahv3.3 + FCST_TIME 036 + + \ + &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_036.tm00 \ + &OUTPUT_DIR;/ref_config_lead_36/pb2nc/NDAS_03h/20110903/prepbufr.ndas.20110903.t12z.tm12.nc \ + &CONFIG_DIR;/ref_config/PointStatConfig_ADPUPA \ + -outdir &OUTPUT_DIR;/ref_config_lead_36/point_stat/AFWAv3.4_Noahv3.3 \ + -v 2 + + + &OUTPUT_DIR;/ref_config_lead_36/point_stat/AFWAv3.4_Noahv3.3/point_stat_AFWAv3.4_Noahv3.3_F036_ADPUPA_360000L_20110903_120000V.stat + + + + + &MET_BIN;/point_stat + + MASK_POLY_FILE &OUTPUT_DIR;/ref_config_lead_36/gen_vx_mask/CONUS.nc + MODEL AFWAv3.4_Noahv3.3 + FCST_TIME 036 + + \ + &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_036.tm00 \ + &OUTPUT_DIR;/ref_config_lead_36/pb2nc/NDAS_03h/20110903/prepbufr.ndas.20110903.t12z.tm12.nc \ + &CONFIG_DIR;/ref_config/PointStatConfig_ONLYSF \ + -outdir &OUTPUT_DIR;/ref_config_lead_36/point_stat/AFWAv3.4_Noahv3.3 \ + -v 2 + + + &OUTPUT_DIR;/ref_config_lead_36/point_stat/AFWAv3.4_Noahv3.3/point_stat_AFWAv3.4_Noahv3.3_F036_ONLYSF_360000L_20110903_120000V.stat + + + + + &MET_BIN;/point_stat + + MASK_POLY_FILE &OUTPUT_DIR;/ref_config_lead_36/gen_vx_mask/CONUS.nc + MODEL AFWAv3.4_Noahv3.3 + FCST_TIME 036 + + \ + &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_036.tm00 \ + &OUTPUT_DIR;/ref_config_lead_36/pb2nc/NDAS_03h/20110903/prepbufr.ndas.20110903.t12z.tm12.nc \ + &CONFIG_DIR;/ref_config/PointStatConfig_WINDS \ + -outdir &OUTPUT_DIR;/ref_config_lead_36/point_stat/AFWAv3.4_Noahv3.3 \ + -v 2 + + + &OUTPUT_DIR;/ref_config_lead_36/point_stat/AFWAv3.4_Noahv3.3/point_stat_AFWAv3.4_Noahv3.3_F036_WINDS_360000L_20110903_120000V.stat + + + + + + + + + &MET_BIN;/grid_stat + + MASK_POLY_FILE &OUTPUT_DIR;/ref_config_lead_36/gen_vx_mask/CONUS.nc + MODEL AFWAv3.4_Noahv3.3 + FCST_TIME 36 + + \ + &OUTPUT_DIR;/ref_config_lead_36/pcp_combine/wrf/wrfpcp03_036.nc \ + &OUTPUT_DIR;/ref_config_lead_36/pcp_combine/ST2_03h/20110903/ST2ml2011090312.03h.nc \ + &CONFIG_DIR;/ref_config/GridStatConfig_03h \ + -outdir &OUTPUT_DIR;/ref_config_lead_36/grid_stat -v 1 + + + &OUTPUT_DIR;/ref_config_lead_36/grid_stat/grid_stat_AFWAv3.4_Noahv3.3_F36_03h_360000L_20110903_120000V.stat + + + + + &MET_BIN;/grid_stat + + MASK_POLY_FILE &OUTPUT_DIR;/ref_config_lead_36/gen_vx_mask/CONUS.nc + MODEL AFWAv3.4_Noahv3.3 + FCST_TIME 36 + + \ + &OUTPUT_DIR;/ref_config_lead_36/pcp_combine/wrf/wrfpcp24_036.nc \ + &OUTPUT_DIR;/ref_config_lead_36/pcp_combine/ST2_24h/20110903/ST2ml2011090312.24h.nc \ + &CONFIG_DIR;/ref_config/GridStatConfig_24h \ + -outdir &OUTPUT_DIR;/ref_config_lead_36/grid_stat -v 1 + + + &OUTPUT_DIR;/ref_config_lead_36/grid_stat/grid_stat_AFWAv3.4_Noahv3.3_F36_24h_360000L_20110903_120000V.stat + + + + diff --git a/test/xml/unit_ref_config_lead_48.xml b/test/xml/unit_ref_config_lead_48.xml new file mode 100644 index 0000000000..b096241afb --- /dev/null +++ b/test/xml/unit_ref_config_lead_48.xml @@ -0,0 +1,231 @@ + + + + + + + + + + +]> + + + + + + &TEST_DIR; + true + + + + + + + &MET_BIN;/gen_vx_mask + \ + &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_012.tm00 \ + &MET_BASE;/poly/CONUS.poly \ + &OUTPUT_DIR;/ref_config_lead_48/gen_vx_mask/CONUS.nc \ + -type poly -v 2 + + + &OUTPUT_DIR;/ref_config_lead_48/gen_vx_mask/CONUS.nc + + + + + + + + + &MET_BIN;/pb2nc + + MASK_POLY &CONFIG_DIR;/ref_config/RefConfig.poly + + \ + &DATA_DIR_OBS;/ref_config/prepbufr/ndas/2011090412/ndas.t12z.prepbufr.tm12.nr \ + &OUTPUT_DIR;/ref_config_lead_48/pb2nc/NDAS_03h/20110904/prepbufr.ndas.20110904.t00z.tm12.nc \ + &CONFIG_DIR;/ref_config/PB2NCConfig \ + -valid_beg 20110903_223000\ + -valid_end 20110904_013000\ + -v 2 + + + &OUTPUT_DIR;/ref_config_lead_48/pb2nc/NDAS_03h/20110904/prepbufr.ndas.20110904.t00z.tm12.nc + + + + + + + + + &MET_BIN;/pcp_combine + \ + -sum \ + 00000000_000000 1 20110904_000000 3 \ + &OUTPUT_DIR;/ref_config_lead_48/pcp_combine/ST2_03h/20110904/ST2ml2011090400.03h.nc \ + -pcpdir &DATA_DIR_OBS;/ref_config/pcp_combine/20110903 \ + -pcpdir &DATA_DIR_OBS;/ref_config/pcp_combine/20110904 + + + &OUTPUT_DIR;/ref_config_lead_48/pcp_combine/ST2_03h/20110904/ST2ml2011090400.03h.nc + + + + + &MET_BIN;/pcp_combine + \ + -subtract \ + &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_048.tm00 48 \ + &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_045.tm00 45 \ + &OUTPUT_DIR;/ref_config_lead_48/pcp_combine/wrf/wrfpcp03_048.nc + + + &OUTPUT_DIR;/ref_config_lead_48/pcp_combine/wrf/wrfpcp03_048.nc + + + + + + + + + &MET_BIN;/point_stat + + MASK_POLY_FILE &OUTPUT_DIR;/ref_config_lead_48/gen_vx_mask/CONUS.nc + MODEL AFWAv3.4_Noahv2.7.1 + FCST_TIME 048 + + \ + &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv2.7.1/postprd/wrfprs_048.tm00 \ + &OUTPUT_DIR;/ref_config_lead_48/pb2nc/NDAS_03h/20110904/prepbufr.ndas.20110904.t00z.tm12.nc \ + &CONFIG_DIR;/ref_config/PointStatConfig_ADPUPA \ + -outdir &OUTPUT_DIR;/ref_config_lead_48/point_stat/AFWAv3.4_Noahv2.7.1 \ + -v 2 + + + &OUTPUT_DIR;/ref_config_lead_48/point_stat/AFWAv3.4_Noahv2.7.1/point_stat_AFWAv3.4_Noahv2.7.1_F048_ADPUPA_480000L_20110904_000000V.stat + + + + + &MET_BIN;/point_stat + + MASK_POLY_FILE &OUTPUT_DIR;/ref_config_lead_48/gen_vx_mask/CONUS.nc + MODEL AFWAv3.4_Noahv2.7.1 + FCST_TIME 048 + + \ + &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv2.7.1/postprd/wrfprs_048.tm00 \ + &OUTPUT_DIR;/ref_config_lead_48/pb2nc/NDAS_03h/20110904/prepbufr.ndas.20110904.t00z.tm12.nc \ + &CONFIG_DIR;/ref_config/PointStatConfig_ONLYSF \ + -outdir &OUTPUT_DIR;/ref_config_lead_48/point_stat/AFWAv3.4_Noahv2.7.1 \ + -v 2 + + + &OUTPUT_DIR;/ref_config_lead_48/point_stat/AFWAv3.4_Noahv2.7.1/point_stat_AFWAv3.4_Noahv2.7.1_F048_ONLYSF_480000L_20110904_000000V.stat + + + + + &MET_BIN;/point_stat + + MASK_POLY_FILE &OUTPUT_DIR;/ref_config_lead_48/gen_vx_mask/CONUS.nc + MODEL AFWAv3.4_Noahv2.7.1 + FCST_TIME 048 + + \ + &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv2.7.1/postprd/wrfprs_048.tm00 \ + &OUTPUT_DIR;/ref_config_lead_48/pb2nc/NDAS_03h/20110904/prepbufr.ndas.20110904.t00z.tm12.nc \ + &CONFIG_DIR;/ref_config/PointStatConfig_WINDS \ + -outdir &OUTPUT_DIR;/ref_config_lead_48/point_stat/AFWAv3.4_Noahv2.7.1 \ + -v 2 + + + &OUTPUT_DIR;/ref_config_lead_48/point_stat/AFWAv3.4_Noahv2.7.1/point_stat_AFWAv3.4_Noahv2.7.1_F048_WINDS_480000L_20110904_000000V.stat + + + + + &MET_BIN;/point_stat + + MASK_POLY_FILE &OUTPUT_DIR;/ref_config_lead_48/gen_vx_mask/CONUS.nc + MODEL AFWAv3.4_Noahv3.3 + FCST_TIME 048 + + \ + &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_048.tm00 \ + &OUTPUT_DIR;/ref_config_lead_48/pb2nc/NDAS_03h/20110904/prepbufr.ndas.20110904.t00z.tm12.nc \ + &CONFIG_DIR;/ref_config/PointStatConfig_ADPUPA \ + -outdir &OUTPUT_DIR;/ref_config_lead_48/point_stat/AFWAv3.4_Noahv3.3 \ + -v 2 + + + &OUTPUT_DIR;/ref_config_lead_48/point_stat/AFWAv3.4_Noahv3.3/point_stat_AFWAv3.4_Noahv3.3_F048_ADPUPA_480000L_20110904_000000V.stat + + + + + &MET_BIN;/point_stat + + MASK_POLY_FILE &OUTPUT_DIR;/ref_config_lead_48/gen_vx_mask/CONUS.nc + MODEL AFWAv3.4_Noahv3.3 + FCST_TIME 048 + + \ + &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_048.tm00 \ + &OUTPUT_DIR;/ref_config_lead_48/pb2nc/NDAS_03h/20110904/prepbufr.ndas.20110904.t00z.tm12.nc \ + &CONFIG_DIR;/ref_config/PointStatConfig_ONLYSF \ + -outdir &OUTPUT_DIR;/ref_config_lead_48/point_stat/AFWAv3.4_Noahv3.3 \ + -v 2 + + + &OUTPUT_DIR;/ref_config_lead_48/point_stat/AFWAv3.4_Noahv3.3/point_stat_AFWAv3.4_Noahv3.3_F048_ONLYSF_480000L_20110904_000000V.stat + + + + + &MET_BIN;/point_stat + + MASK_POLY_FILE &OUTPUT_DIR;/ref_config_lead_48/gen_vx_mask/CONUS.nc + MODEL AFWAv3.4_Noahv3.3 + FCST_TIME 048 + + \ + &DATA_DIR_MODEL;/grib1/ref_config/2011090200/AFWAv3.4_Noahv3.3/postprd/wrfprs_048.tm00 \ + &OUTPUT_DIR;/ref_config_lead_48/pb2nc/NDAS_03h/20110904/prepbufr.ndas.20110904.t00z.tm12.nc \ + &CONFIG_DIR;/ref_config/PointStatConfig_WINDS \ + -outdir &OUTPUT_DIR;/ref_config_lead_48/point_stat/AFWAv3.4_Noahv3.3 \ + -v 2 + + + &OUTPUT_DIR;/ref_config_lead_48/point_stat/AFWAv3.4_Noahv3.3/point_stat_AFWAv3.4_Noahv3.3_F048_WINDS_480000L_20110904_000000V.stat + + + + + + + + + &MET_BIN;/grid_stat + + MASK_POLY_FILE &OUTPUT_DIR;/ref_config_lead_48/gen_vx_mask/CONUS.nc + MODEL AFWAv3.4_Noahv3.3 + FCST_TIME 48 + + \ + &OUTPUT_DIR;/ref_config_lead_48/pcp_combine/wrf/wrfpcp03_048.nc \ + &OUTPUT_DIR;/ref_config_lead_48/pcp_combine/ST2_03h/20110904/ST2ml2011090400.03h.nc \ + &CONFIG_DIR;/ref_config/GridStatConfig_03h \ + -outdir &OUTPUT_DIR;/ref_config_lead_48/grid_stat -v 1 + + + &OUTPUT_DIR;/ref_config_lead_48/grid_stat/grid_stat_AFWAv3.4_Noahv3.3_F48_03h_480000L_20110904_000000V.stat + + + + diff --git a/test/xml/unit_stat_analysis.xml b/test/xml/unit_stat_analysis.xml deleted file mode 100644 index 078b76e072..0000000000 --- a/test/xml/unit_stat_analysis.xml +++ /dev/null @@ -1,331 +0,0 @@ - - - - - - - - - -]> - - - - - - &TEST_DIR; - true - - - - - &MET_BIN;/stat_analysis - \ - -lookin &OUTPUT_DIR;/ensemble_stat/ensemble_stat_CMD_LINE_20120410_120000V.stat \ - -job aggregate -line_type RHIST \ - -fcst_var TMP -vx_mask NWC -vx_mask GRB \ - -dump_row &OUTPUT_DIR;/stat_analysis/AGG_RHIST_dump.stat \ - -out_stat &OUTPUT_DIR;/stat_analysis/AGG_RHIST_out.stat \ - -out &OUTPUT_DIR;/stat_analysis/AGG_RHIST.out \ - -v 1 - - - &OUTPUT_DIR;/stat_analysis/AGG_RHIST_dump.stat - &OUTPUT_DIR;/stat_analysis/AGG_RHIST_out.stat - &OUTPUT_DIR;/stat_analysis/AGG_RHIST.out - - - - - &MET_BIN;/stat_analysis - \ - -lookin &OUTPUT_DIR;/ensemble_stat/ensemble_stat_CMD_LINE_20120410_120000V.stat \ - -job aggregate -line_type PHIST \ - -fcst_var TMP -vx_mask NWC -vx_mask GRB \ - -dump_row &OUTPUT_DIR;/stat_analysis/AGG_PHIST_dump.stat \ - -out_stat &OUTPUT_DIR;/stat_analysis/AGG_PHIST_out.stat \ - -set_hdr VX_MASK NWC_AND_GRB \ - -out &OUTPUT_DIR;/stat_analysis/AGG_PHIST.out \ - -v 1 - - - &OUTPUT_DIR;/stat_analysis/AGG_PHIST_dump.stat - &OUTPUT_DIR;/stat_analysis/AGG_PHIST_out.stat - &OUTPUT_DIR;/stat_analysis/AGG_PHIST.out - - - - - &MET_BIN;/stat_analysis - \ - -lookin &OUTPUT_DIR;/ensemble_stat/ensemble_stat_SKIP_CONST_20120410_120000V.stat \ - -job aggregate -line_type RELP -by FCST_VAR -vx_mask NWC,GRB \ - -dump_row &OUTPUT_DIR;/stat_analysis/AGG_RELP_dump.stat \ - -out_stat &OUTPUT_DIR;/stat_analysis/AGG_RELP_out.stat \ - -set_hdr VX_MASK NWC_AND_GRB \ - -out &OUTPUT_DIR;/stat_analysis/AGG_RELP.out \ - -v 1 - - - &OUTPUT_DIR;/stat_analysis/AGG_RELP_dump.stat - &OUTPUT_DIR;/stat_analysis/AGG_RELP_out.stat - &OUTPUT_DIR;/stat_analysis/AGG_RELP.out - - - - - &MET_BIN;/stat_analysis - \ - -lookin &OUTPUT_DIR;/ensemble_stat/ensemble_stat_SKIP_CONST_20120410_120000V.stat \ - -job aggregate -line_type ECNT -by FCST_VAR -obs_thresh NA -vx_mask NWC,GRB \ - -dump_row &OUTPUT_DIR;/stat_analysis/AGG_ECNT_dump.stat \ - -out_stat &OUTPUT_DIR;/stat_analysis/AGG_ECNT_out.stat \ - -set_hdr VX_MASK NWC_AND_GRB \ - -v 1 - - - &OUTPUT_DIR;/stat_analysis/AGG_ECNT_dump.stat - &OUTPUT_DIR;/stat_analysis/AGG_ECNT_out.stat - - - - - &MET_BIN;/stat_analysis - \ - -lookin &OUTPUT_DIR;/ensemble_stat/ensemble_stat_CMD_LINE_20120410_120000V.stat \ - -job aggregate_stat -line_type ORANK -out_line_type RHIST,PHIST \ - -fcst_var APCP_24 -vx_mask NWC -vx_mask GRB -out_bin_size 0.10 \ - -dump_row &OUTPUT_DIR;/stat_analysis/AGG_STAT_ORANK_RHIST_PHIST_dump.stat \ - -out_stat &OUTPUT_DIR;/stat_analysis/AGG_STAT_ORANK_RHIST_PHIST_out.stat \ - -out &OUTPUT_DIR;/stat_analysis/AGG_STAT_ORANK_RHIST_PHIST.out \ - -v 1 - - - &OUTPUT_DIR;/stat_analysis/AGG_STAT_ORANK_RHIST_PHIST_dump.stat - &OUTPUT_DIR;/stat_analysis/AGG_STAT_ORANK_RHIST_PHIST_out.stat - &OUTPUT_DIR;/stat_analysis/AGG_STAT_ORANK_RHIST_PHIST.out - - - - - &MET_BIN;/stat_analysis - \ - -lookin &OUTPUT_DIR;/ensemble_stat/ensemble_stat_SKIP_CONST_20120410_120000V.stat \ - -job aggregate_stat -line_type ORANK -out_line_type RELP \ - -fcst_var APCP_24 -vx_mask NWC,GRB \ - -dump_row &OUTPUT_DIR;/stat_analysis/AGG_STAT_ORANK_RELP_dump.stat \ - -out_stat &OUTPUT_DIR;/stat_analysis/AGG_STAT_ORANK_RELP_out.stat \ - -out &OUTPUT_DIR;/stat_analysis/AGG_STAT_ORANK_RELP.out \ - -v 1 - - - &OUTPUT_DIR;/stat_analysis/AGG_STAT_ORANK_RELP_dump.stat - &OUTPUT_DIR;/stat_analysis/AGG_STAT_ORANK_RELP_out.stat - &OUTPUT_DIR;/stat_analysis/AGG_STAT_ORANK_RELP.out - - - - - &MET_BIN;/stat_analysis - \ - -lookin &OUTPUT_DIR;/ensemble_stat/ensemble_stat_CMD_LINE_20120410_120000V.stat \ - -job aggregate_stat -line_type ORANK -out_line_type SSVAR \ - -fcst_var APCP_24 -vx_mask NWC -vx_mask GRB -out_bin_size 0.25 \ - -dump_row &OUTPUT_DIR;/stat_analysis/AGG_STAT_ORANK_SSVAR_dump.stat \ - -out_stat &OUTPUT_DIR;/stat_analysis/AGG_STAT_ORANK_SSVAR_out.stat \ - -out &OUTPUT_DIR;/stat_analysis/AGG_STAT_ORANK_SSVAR.out \ - -v 1 - - - &OUTPUT_DIR;/stat_analysis/AGG_STAT_ORANK_SSVAR_dump.stat - &OUTPUT_DIR;/stat_analysis/AGG_STAT_ORANK_SSVAR_out.stat - &OUTPUT_DIR;/stat_analysis/AGG_STAT_ORANK_SSVAR.out - - - - - &MET_BIN;/stat_analysis - \ - -lookin &OUTPUT_DIR;/ensemble_stat/ensemble_stat_OBSERR_20120410_120000V.stat \ - -job aggregate_stat -line_type ORANK -out_line_type ECNT \ - -fcst_var APCP_24 -by VX_MASK \ - -set_hdr DESC VX_MASK \ - -dump_row &OUTPUT_DIR;/stat_analysis/AGG_STAT_ORANK_ECNT_dump.stat \ - -out_stat &OUTPUT_DIR;/stat_analysis/AGG_STAT_ORANK_ECNT_out.stat \ - -out &OUTPUT_DIR;/stat_analysis/AGG_STAT_ORANK_ECNT.out \ - -v 1 - - - &OUTPUT_DIR;/stat_analysis/AGG_STAT_ORANK_ECNT_dump.stat - &OUTPUT_DIR;/stat_analysis/AGG_STAT_ORANK_ECNT_out.stat - &OUTPUT_DIR;/stat_analysis/AGG_STAT_ORANK_ECNT.out - - - - - &MET_BIN;/stat_analysis - \ - -lookin &OUTPUT_DIR;/ensemble_stat/ensemble_stat_CMD_LINE_20120410_120000V.stat \ - -job aggregate -line_type SSVAR \ - -fcst_var APCP_24 -obtype ANALYS -vx_mask NWC -vx_mask GRB \ - -dump_row &OUTPUT_DIR;/stat_analysis/AGG_SSVAR_dump.stat \ - -out_stat &OUTPUT_DIR;/stat_analysis/AGG_SSVAR_out.stat \ - -out &OUTPUT_DIR;/stat_analysis/AGG_SSVAR.out \ - -v 1 - - - &OUTPUT_DIR;/stat_analysis/AGG_SSVAR_dump.stat - &OUTPUT_DIR;/stat_analysis/AGG_SSVAR_out.stat - &OUTPUT_DIR;/stat_analysis/AGG_SSVAR.out - - - - - - - &MET_BIN;/stat_analysis - \ - -lookin &OUTPUT_DIR;/wavelet_stat/wavelet_stat_GRIB1_NAM_STAGE4_120000L_20120409_120000V.stat \ - -job aggregate -line_type ISC \ - -fcst_var APCP_12 -fcst_thresh '>0.0' -vx_mask TILE1 -vx_mask TILE2 \ - -dump_row &OUTPUT_DIR;/stat_analysis/AGG_ISC_dump.stat \ - -out_stat &OUTPUT_DIR;/stat_analysis/AGG_ISC_out.stat \ - -out &OUTPUT_DIR;/stat_analysis/AGG_ISC.out \ - -v 1 - - - &OUTPUT_DIR;/stat_analysis/AGG_ISC_dump.stat - &OUTPUT_DIR;/stat_analysis/AGG_ISC_out.stat - &OUTPUT_DIR;/stat_analysis/AGG_ISC.out - - - - - - - - OUTPUT_DIR &OUTPUT_DIR;/stat_analysis - - &MET_BIN;/stat_analysis - \ - -lookin &OUTPUT_DIR;/grid_stat/grid_stat_GRIB2_NAM_RTMA_120000L_20120409_120000V.stat \ - -config &CONFIG_DIR;/STATAnalysisConfig_grid_stat \ - -out &OUTPUT_DIR;/stat_analysis/CONFIG_GRID_STAT.out \ - -v 1 - - - &OUTPUT_DIR;/stat_analysis/CONFIG_GRID_STAT.out - &OUTPUT_DIR;/stat_analysis/CONFIG_GRID_STAT_filter.stat - &OUTPUT_DIR;/stat_analysis/CONFIG_GRID_STAT_agg_stat_sl1l2_dump.stat - &OUTPUT_DIR;/stat_analysis/CONFIG_GRID_STAT_agg_stat_sl1l2_out.stat - &OUTPUT_DIR;/stat_analysis/CONFIG_GRID_STAT_agg_ctc_dump.stat - &OUTPUT_DIR;/stat_analysis/CONFIG_GRID_STAT_agg_ctc_out.stat - &OUTPUT_DIR;/stat_analysis/CONFIG_GRID_STAT_agg_stat_ctc_to_eclv_dump.stat - &OUTPUT_DIR;/stat_analysis/CONFIG_GRID_STAT_agg_stat_ctc_to_eclv_out.stat - - - - - - - - CONFIG_DIR &CONFIG_DIR; - OUTPUT_DIR &OUTPUT_DIR;/stat_analysis - - &MET_BIN;/stat_analysis - \ - -lookin &OUTPUT_DIR;/point_stat/point_stat_GRIB1_NAM_GDAS_120000L_20120409_120000V.stat \ - -config &CONFIG_DIR;/STATAnalysisConfig_point_stat \ - -out &OUTPUT_DIR;/stat_analysis/CONFIG_POINT_STAT.out \ - -v 1 - - - &OUTPUT_DIR;/stat_analysis/CONFIG_POINT_STAT.out - &OUTPUT_DIR;/stat_analysis/CONFIG_POINT_STAT_agg_stat_mpr_to_cnt_dump.stat - &OUTPUT_DIR;/stat_analysis/CONFIG_POINT_STAT_agg_stat_mpr_to_cnt_out.stat - &OUTPUT_DIR;/stat_analysis/CONFIG_POINT_STAT_agg_stat_mpr_to_cnt_by_vx_mask_out.stat - &OUTPUT_DIR;/stat_analysis/CONFIG_POINT_STAT_agg_ctc_by_fcst_thresh_out.stat - &OUTPUT_DIR;/stat_analysis/CONFIG_POINT_STAT_agg_stat_mpr_to_wdir_dump.stat - &OUTPUT_DIR;/stat_analysis/CONFIG_POINT_STAT_filter_mpr_sid.stat - &OUTPUT_DIR;/stat_analysis/CONFIG_POINT_STAT_filter_mpr_fcst_minus_obs.stat - - - - - - OUTPUT_DIR &OUTPUT_DIR;/stat_analysis - - &MET_BIN;/stat_analysis - \ - -lookin &DATA_DIR_MODEL;/time_series_met_6.0/*.stat \ - -config &CONFIG_DIR;/STATAnalysisConfig_ramps \ - -out &OUTPUT_DIR;/stat_analysis/RAMPS.out \ - -v 1 - - - &OUTPUT_DIR;/stat_analysis/RAMPS.out - &OUTPUT_DIR;/stat_analysis/RAMPS_100_100.stat - &OUTPUT_DIR;/stat_analysis/RAMPS_25_100.stat - &OUTPUT_DIR;/stat_analysis/RAMPS_25_100_30min.stat - - - - - &MET_BIN;/stat_analysis - \ - -lookin &OUTPUT_DIR;/point_stat/point_stat_GRIB1_NAM_GDAS_120000L_20120409_120000V.stat \ - -job summary -line_type MPR -by FCST_VAR,FCST_LEV -column 'FCST,OBS,FCST-OBS,ABS(FCST-OBS)' \ - -boot_seed 1 -out &OUTPUT_DIR;/stat_analysis/POINT_STAT_SUMMARY.out \ - -v 1 - - - &OUTPUT_DIR;/stat_analysis/POINT_STAT_SUMMARY.out - - - - - &MET_BIN;/stat_analysis - \ - -lookin &OUTPUT_DIR;/point_stat/point_stat_GRIB1_NAM_GDAS_120000L_20120409_120000V.stat \ - -job summary -line_type MPR -by FCST_VAR,FCST_LEV -column 'FCST,OBS' -column_union true \ - -boot_seed 1 -out &OUTPUT_DIR;/stat_analysis/POINT_STAT_SUMMARY_UNION.out \ - -v 1 - - - &OUTPUT_DIR;/stat_analysis/POINT_STAT_SUMMARY_UNION.out - - - - - &MET_BIN;/stat_analysis - \ - -lookin &OUTPUT_DIR;/point_stat/point_stat_GRIB1_NAM_GDAS_120000L_20120409_120000V.stat \ - -job filter -line_type MPR -fcst_var TMP -fcst_lev Z2 -vx_mask DTC165 \ - -column_str OBS_SID KDLN,KDHT,KDEN,KDLS,KDMA,KDMN,KDVT,KDEW \ - -column_str_exc OBS_SID KDLN,KDHT \ - -dump_row &OUTPUT_DIR;/stat_analysis/POINT_STAT_FILTER_OBS_SID.stat \ - -v 1 - - - &OUTPUT_DIR;/stat_analysis/POINT_STAT_FILTER_OBS_SID.stat - - - - - - OUTPUT_DIR &OUTPUT_DIR;/stat_analysis - - &MET_BIN;/stat_analysis - \ - -lookin &OUTPUT_DIR;/point_stat/point_stat_GRIB2_NAM_NDAS_120000L_20120409_120000V.stat \ - -config &CONFIG_DIR;/STATAnalysisConfig_filter_times \ - -out &OUTPUT_DIR;/stat_analysis/POINT_STAT_FILTER_TIMES.out \ - -v 1 - - - &OUTPUT_DIR;/stat_analysis/POINT_STAT_FILTER_TIMES.out - - - - diff --git a/test/xml/unit_stat_analysis_es.xml b/test/xml/unit_stat_analysis_es.xml new file mode 100644 index 0000000000..a7b446c1c6 --- /dev/null +++ b/test/xml/unit_stat_analysis_es.xml @@ -0,0 +1,182 @@ + + + + + + + + + +]> + + + + + + &TEST_DIR; + true + + + &MET_BIN;/stat_analysis + \ + -lookin &OUTPUT_DIR;/ensemble_stat/ensemble_stat_CMD_LINE_20120410_120000V.stat \ + -job aggregate -line_type RHIST \ + -fcst_var TMP -vx_mask NWC -vx_mask GRB \ + -dump_row &OUTPUT_DIR;/stat_analysis_es/AGG_RHIST_dump.stat \ + -out_stat &OUTPUT_DIR;/stat_analysis_es/AGG_RHIST_out.stat \ + -out &OUTPUT_DIR;/stat_analysis_es/AGG_RHIST.out \ + -v 1 + + + &OUTPUT_DIR;/stat_analysis_es/AGG_RHIST_dump.stat + &OUTPUT_DIR;/stat_analysis_es/AGG_RHIST_out.stat + &OUTPUT_DIR;/stat_analysis_es/AGG_RHIST.out + + + + + &MET_BIN;/stat_analysis + \ + -lookin &OUTPUT_DIR;/ensemble_stat/ensemble_stat_CMD_LINE_20120410_120000V.stat \ + -job aggregate -line_type PHIST \ + -fcst_var TMP -vx_mask NWC -vx_mask GRB \ + -dump_row &OUTPUT_DIR;/stat_analysis_es/AGG_PHIST_dump.stat \ + -out_stat &OUTPUT_DIR;/stat_analysis_es/AGG_PHIST_out.stat \ + -set_hdr VX_MASK NWC_AND_GRB \ + -out &OUTPUT_DIR;/stat_analysis_es/AGG_PHIST.out \ + -v 1 + + + &OUTPUT_DIR;/stat_analysis_es/AGG_PHIST_dump.stat + &OUTPUT_DIR;/stat_analysis_es/AGG_PHIST_out.stat + &OUTPUT_DIR;/stat_analysis_es/AGG_PHIST.out + + + + + &MET_BIN;/stat_analysis + \ + -lookin &OUTPUT_DIR;/ensemble_stat/ensemble_stat_SKIP_CONST_20120410_120000V.stat \ + -job aggregate -line_type RELP -by FCST_VAR -vx_mask NWC,GRB \ + -dump_row &OUTPUT_DIR;/stat_analysis_es/AGG_RELP_dump.stat \ + -out_stat &OUTPUT_DIR;/stat_analysis_es/AGG_RELP_out.stat \ + -set_hdr VX_MASK NWC_AND_GRB \ + -out &OUTPUT_DIR;/stat_analysis_es/AGG_RELP.out \ + -v 1 + + + &OUTPUT_DIR;/stat_analysis_es/AGG_RELP_dump.stat + &OUTPUT_DIR;/stat_analysis_es/AGG_RELP_out.stat + &OUTPUT_DIR;/stat_analysis_es/AGG_RELP.out + + + + + &MET_BIN;/stat_analysis + \ + -lookin &OUTPUT_DIR;/ensemble_stat/ensemble_stat_SKIP_CONST_20120410_120000V.stat \ + -job aggregate -line_type ECNT -by FCST_VAR -obs_thresh NA -vx_mask NWC,GRB \ + -dump_row &OUTPUT_DIR;/stat_analysis_es/AGG_ECNT_dump.stat \ + -out_stat &OUTPUT_DIR;/stat_analysis_es/AGG_ECNT_out.stat \ + -set_hdr VX_MASK NWC_AND_GRB \ + -v 1 + + + &OUTPUT_DIR;/stat_analysis_es/AGG_ECNT_dump.stat + &OUTPUT_DIR;/stat_analysis_es/AGG_ECNT_out.stat + + + + + &MET_BIN;/stat_analysis + \ + -lookin &OUTPUT_DIR;/ensemble_stat/ensemble_stat_CMD_LINE_20120410_120000V.stat \ + -job aggregate_stat -line_type ORANK -out_line_type RHIST,PHIST \ + -fcst_var APCP_24 -vx_mask NWC -vx_mask GRB -out_bin_size 0.10 \ + -dump_row &OUTPUT_DIR;/stat_analysis_es/AGG_STAT_ORANK_RHIST_PHIST_dump.stat \ + -out_stat &OUTPUT_DIR;/stat_analysis_es/AGG_STAT_ORANK_RHIST_PHIST_out.stat \ + -out &OUTPUT_DIR;/stat_analysis_es/AGG_STAT_ORANK_RHIST_PHIST.out \ + -v 1 + + + &OUTPUT_DIR;/stat_analysis_es/AGG_STAT_ORANK_RHIST_PHIST_dump.stat + &OUTPUT_DIR;/stat_analysis_es/AGG_STAT_ORANK_RHIST_PHIST_out.stat + &OUTPUT_DIR;/stat_analysis_es/AGG_STAT_ORANK_RHIST_PHIST.out + + + + + &MET_BIN;/stat_analysis + \ + -lookin &OUTPUT_DIR;/ensemble_stat/ensemble_stat_SKIP_CONST_20120410_120000V.stat \ + -job aggregate_stat -line_type ORANK -out_line_type RELP \ + -fcst_var APCP_24 -vx_mask NWC,GRB \ + -dump_row &OUTPUT_DIR;/stat_analysis_es/AGG_STAT_ORANK_RELP_dump.stat \ + -out_stat &OUTPUT_DIR;/stat_analysis_es/AGG_STAT_ORANK_RELP_out.stat \ + -out &OUTPUT_DIR;/stat_analysis_es/AGG_STAT_ORANK_RELP.out \ + -v 1 + + + &OUTPUT_DIR;/stat_analysis_es/AGG_STAT_ORANK_RELP_dump.stat + &OUTPUT_DIR;/stat_analysis_es/AGG_STAT_ORANK_RELP_out.stat + &OUTPUT_DIR;/stat_analysis_es/AGG_STAT_ORANK_RELP.out + + + + + &MET_BIN;/stat_analysis + \ + -lookin &OUTPUT_DIR;/ensemble_stat/ensemble_stat_CMD_LINE_20120410_120000V.stat \ + -job aggregate_stat -line_type ORANK -out_line_type SSVAR \ + -fcst_var APCP_24 -vx_mask NWC -vx_mask GRB -out_bin_size 0.25 \ + -dump_row &OUTPUT_DIR;/stat_analysis_es/AGG_STAT_ORANK_SSVAR_dump.stat \ + -out_stat &OUTPUT_DIR;/stat_analysis_es/AGG_STAT_ORANK_SSVAR_out.stat \ + -out &OUTPUT_DIR;/stat_analysis_es/AGG_STAT_ORANK_SSVAR.out \ + -v 1 + + + &OUTPUT_DIR;/stat_analysis_es/AGG_STAT_ORANK_SSVAR_dump.stat + &OUTPUT_DIR;/stat_analysis_es/AGG_STAT_ORANK_SSVAR_out.stat + &OUTPUT_DIR;/stat_analysis_es/AGG_STAT_ORANK_SSVAR.out + + + + + &MET_BIN;/stat_analysis + \ + -lookin &OUTPUT_DIR;/ensemble_stat/ensemble_stat_OBSERR_20120410_120000V.stat \ + -job aggregate_stat -line_type ORANK -out_line_type ECNT \ + -fcst_var APCP_24 -by VX_MASK \ + -set_hdr DESC VX_MASK \ + -dump_row &OUTPUT_DIR;/stat_analysis_es/AGG_STAT_ORANK_ECNT_dump.stat \ + -out_stat &OUTPUT_DIR;/stat_analysis_es/AGG_STAT_ORANK_ECNT_out.stat \ + -out &OUTPUT_DIR;/stat_analysis_es/AGG_STAT_ORANK_ECNT.out \ + -v 1 + + + &OUTPUT_DIR;/stat_analysis_es/AGG_STAT_ORANK_ECNT_dump.stat + &OUTPUT_DIR;/stat_analysis_es/AGG_STAT_ORANK_ECNT_out.stat + &OUTPUT_DIR;/stat_analysis_es/AGG_STAT_ORANK_ECNT.out + + + + + &MET_BIN;/stat_analysis + \ + -lookin &OUTPUT_DIR;/ensemble_stat/ensemble_stat_CMD_LINE_20120410_120000V.stat \ + -job aggregate -line_type SSVAR \ + -fcst_var APCP_24 -obtype ANALYS -vx_mask NWC -vx_mask GRB \ + -dump_row &OUTPUT_DIR;/stat_analysis_es/AGG_SSVAR_dump.stat \ + -out_stat &OUTPUT_DIR;/stat_analysis_es/AGG_SSVAR_out.stat \ + -out &OUTPUT_DIR;/stat_analysis_es/AGG_SSVAR.out \ + -v 1 + + + &OUTPUT_DIR;/stat_analysis_es/AGG_SSVAR_dump.stat + &OUTPUT_DIR;/stat_analysis_es/AGG_SSVAR_out.stat + &OUTPUT_DIR;/stat_analysis_es/AGG_SSVAR.out + + + + diff --git a/test/xml/unit_stat_analysis_gs.xml b/test/xml/unit_stat_analysis_gs.xml new file mode 100644 index 0000000000..6309f57aef --- /dev/null +++ b/test/xml/unit_stat_analysis_gs.xml @@ -0,0 +1,45 @@ + + + + + + + + + +]> + + + + + + &TEST_DIR; + true + + + + + + OUTPUT_DIR &OUTPUT_DIR;/stat_analysis_gs + + &MET_BIN;/stat_analysis + \ + -lookin &OUTPUT_DIR;/grid_stat/grid_stat_GRIB2_NAM_RTMA_120000L_20120409_120000V.stat \ + -config &CONFIG_DIR;/STATAnalysisConfig_grid_stat \ + -out &OUTPUT_DIR;/stat_analysis_gs/CONFIG_GRID_STAT.out \ + -v 1 + + + &OUTPUT_DIR;/stat_analysis_gs/CONFIG_GRID_STAT.out + &OUTPUT_DIR;/stat_analysis_gs/CONFIG_GRID_STAT_filter.stat + &OUTPUT_DIR;/stat_analysis_gs/CONFIG_GRID_STAT_agg_stat_sl1l2_dump.stat + &OUTPUT_DIR;/stat_analysis_gs/CONFIG_GRID_STAT_agg_stat_sl1l2_out.stat + &OUTPUT_DIR;/stat_analysis_gs/CONFIG_GRID_STAT_agg_ctc_dump.stat + &OUTPUT_DIR;/stat_analysis_gs/CONFIG_GRID_STAT_agg_ctc_out.stat + &OUTPUT_DIR;/stat_analysis_gs/CONFIG_GRID_STAT_agg_stat_ctc_to_eclv_dump.stat + &OUTPUT_DIR;/stat_analysis_gs/CONFIG_GRID_STAT_agg_stat_ctc_to_eclv_out.stat + + + + diff --git a/test/xml/unit_stat_analysis_ps.xml b/test/xml/unit_stat_analysis_ps.xml new file mode 100644 index 0000000000..d32fe963f7 --- /dev/null +++ b/test/xml/unit_stat_analysis_ps.xml @@ -0,0 +1,120 @@ + + + + + + + + + +]> + + + + + + &TEST_DIR; + true + + + + CONFIG_DIR &CONFIG_DIR; + OUTPUT_DIR &OUTPUT_DIR;/stat_analysis_ps + + &MET_BIN;/stat_analysis + \ + -lookin &OUTPUT_DIR;/point_stat/point_stat_GRIB1_NAM_GDAS_120000L_20120409_120000V.stat \ + -config &CONFIG_DIR;/STATAnalysisConfig_point_stat \ + -out &OUTPUT_DIR;/stat_analysis_ps/CONFIG_POINT_STAT.out \ + -v 1 + + + &OUTPUT_DIR;/stat_analysis_ps/CONFIG_POINT_STAT.out + &OUTPUT_DIR;/stat_analysis_ps/CONFIG_POINT_STAT_agg_stat_mpr_to_cnt_dump.stat + &OUTPUT_DIR;/stat_analysis_ps/CONFIG_POINT_STAT_agg_stat_mpr_to_cnt_out.stat + &OUTPUT_DIR;/stat_analysis_ps/CONFIG_POINT_STAT_agg_stat_mpr_to_cnt_by_vx_mask_out.stat + &OUTPUT_DIR;/stat_analysis_ps/CONFIG_POINT_STAT_agg_ctc_by_fcst_thresh_out.stat + &OUTPUT_DIR;/stat_analysis_ps/CONFIG_POINT_STAT_agg_stat_mpr_to_wdir_dump.stat + &OUTPUT_DIR;/stat_analysis_ps/CONFIG_POINT_STAT_filter_mpr_sid.stat + &OUTPUT_DIR;/stat_analysis_ps/CONFIG_POINT_STAT_filter_mpr_fcst_minus_obs.stat + + + + + &MET_BIN;/stat_analysis + \ + -lookin &OUTPUT_DIR;/point_stat/point_stat_GRIB1_NAM_GDAS_120000L_20120409_120000V.stat \ + -job summary -line_type MPR -by FCST_VAR,FCST_LEV -column 'FCST,OBS,FCST-OBS,ABS(FCST-OBS)' \ + -boot_seed 1 -out &OUTPUT_DIR;/stat_analysis_ps/POINT_STAT_SUMMARY.out \ + -v 1 + + + &OUTPUT_DIR;/stat_analysis_ps/POINT_STAT_SUMMARY.out + + + + + &MET_BIN;/stat_analysis + \ + -lookin &OUTPUT_DIR;/point_stat/point_stat_GRIB1_NAM_GDAS_120000L_20120409_120000V.stat \ + -job summary -line_type MPR -by FCST_VAR,FCST_LEV -column 'FCST,OBS' -column_union true \ + -boot_seed 1 -out &OUTPUT_DIR;/stat_analysis_ps/POINT_STAT_SUMMARY_UNION.out \ + -v 1 + + + &OUTPUT_DIR;/stat_analysis_ps/POINT_STAT_SUMMARY_UNION.out + + + + + &MET_BIN;/stat_analysis + \ + -lookin &OUTPUT_DIR;/point_stat/point_stat_GRIB1_NAM_GDAS_120000L_20120409_120000V.stat \ + -job filter -line_type MPR -fcst_var TMP -fcst_lev Z2 -vx_mask DTC165 \ + -column_str OBS_SID KDLN,KDHT,KDEN,KDLS,KDMA,KDMN,KDVT,KDEW \ + -column_str_exc OBS_SID KDLN,KDHT \ + -dump_row &OUTPUT_DIR;/stat_analysis_ps/POINT_STAT_FILTER_OBS_SID.stat \ + -v 1 + + + &OUTPUT_DIR;/stat_analysis_ps/POINT_STAT_FILTER_OBS_SID.stat + + + + + + OUTPUT_DIR &OUTPUT_DIR;/stat_analysis_ps + + &MET_BIN;/stat_analysis + \ + -lookin &OUTPUT_DIR;/point_stat/point_stat_GRIB2_NAM_NDAS_120000L_20120409_120000V.stat \ + -config &CONFIG_DIR;/STATAnalysisConfig_filter_times \ + -out &OUTPUT_DIR;/stat_analysis_ps/POINT_STAT_FILTER_TIMES.out \ + -v 1 + + + &OUTPUT_DIR;/stat_analysis_ps/POINT_STAT_FILTER_TIMES.out + + + + + + OUTPUT_DIR &OUTPUT_DIR;/stat_analysis_ps + + &MET_BIN;/stat_analysis + \ + -lookin &DATA_DIR_MODEL;/time_series_met_6.0/*.stat \ + -config &CONFIG_DIR;/STATAnalysisConfig_ramps \ + -out &OUTPUT_DIR;/stat_analysis_ps/RAMPS.out \ + -v 1 + + + &OUTPUT_DIR;/stat_analysis_ps/RAMPS.out + &OUTPUT_DIR;/stat_analysis_ps/RAMPS_100_100.stat + &OUTPUT_DIR;/stat_analysis_ps/RAMPS_25_100.stat + &OUTPUT_DIR;/stat_analysis_ps/RAMPS_25_100_30min.stat + + + + diff --git a/test/xml/unit_stat_analysis_ws.xml b/test/xml/unit_stat_analysis_ws.xml new file mode 100644 index 0000000000..4bb01eda9e --- /dev/null +++ b/test/xml/unit_stat_analysis_ws.xml @@ -0,0 +1,38 @@ + + + + + + + + + +]> + + + + + + &TEST_DIR; + true + + + &MET_BIN;/stat_analysis + \ + -lookin &OUTPUT_DIR;/wavelet_stat/wavelet_stat_GRIB1_NAM_STAGE4_120000L_20120409_120000V.stat \ + -job aggregate -line_type ISC \ + -fcst_var APCP_12 -fcst_thresh '>0.0' -vx_mask TILE1 -vx_mask TILE2 \ + -dump_row &OUTPUT_DIR;/stat_analysis_ws/AGG_ISC_dump.stat \ + -out_stat &OUTPUT_DIR;/stat_analysis_ws/AGG_ISC_out.stat \ + -out &OUTPUT_DIR;/stat_analysis_ws/AGG_ISC.out \ + -v 1 + + + &OUTPUT_DIR;/stat_analysis_ws/AGG_ISC_dump.stat + &OUTPUT_DIR;/stat_analysis_ws/AGG_ISC_out.stat + &OUTPUT_DIR;/stat_analysis_ws/AGG_ISC.out + + + + From 42ca71a4e2df51b5fe6ed1042e64db16b5ca2ddb Mon Sep 17 00:00:00 2001 From: George McCabe <23407799+georgemccabe@users.noreply.github.com> Date: Thu, 27 Jan 2022 15:48:00 -0700 Subject: [PATCH 82/82] fixed broken workflow by setting environment variable expected for docker build script --- .github/workflows/build_docker_and_trigger_metplus.yml | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.github/workflows/build_docker_and_trigger_metplus.yml b/.github/workflows/build_docker_and_trigger_metplus.yml index 153adcb060..88f2a66ebe 100644 --- a/.github/workflows/build_docker_and_trigger_metplus.yml +++ b/.github/workflows/build_docker_and_trigger_metplus.yml @@ -7,6 +7,9 @@ on: paths-ignore: - 'met/docs/**' +env: + DOCKERHUB_REPO: dtcenter/met-dev + jobs: build_met_docker: