Feature 1858 obs quality (#1919)

* Per issue #1858: changed parse_conf_obs_qty to parse_conf_obs_qty_inc. Added new function parse_conf_obs_qty_exc. SL

* Per issue #1858: changed conf_key_obs_qty to conf_key_obs_qty_inc. Added conf_key_obs_qty_exc. SL

* Per issue #1858: modified vx_pd.set_obs_qty_filt() to use parse_conf_obs_qty_inc() instead of parse_conf_obs_qty(). SL

* Per issue #1858: added back the original parse_conf_obs_qty() function for backward compatibility. SL

* Per issue #1858: added back config_constants.h for backwards compatibility. SL

* Per issue #1858: added code to read in obs_qty_inc instead of obs_qty. Also added in code for backwards compatibility to check for obs_qty (obs_quality) and use that for obs_qty_inc. SL

* Changed obs_qty_filt to obs_qty_inc_filt. Added obs_qty_exc_filt, including relevant function. SL

* Per issue #1858 added vx_pd.set_obs_qty_exc_filt. SL

* In add_point_obs() added if block to apply obs_qty_exc_filt. SL

* Per issue #1858, in add_point_obs(), cleaned up sections that apply obs_qty_inc and obs_qty_exc filters. Fixed spacing issue. SL

* Per issue #1858 in process_config(), cleaned up sections that set obs_qty_inc_filt and obs_qty_exc_filt. Fixed spacing issue. SL

* Per issue #1858 in process_obs_file() cleaned up code. SL

* Per issue #1858: changed obs_qty_filt to obs_qty_inc_filt. Added obs_qty_exc_filt. Modified add_point_obs() to use obs_qty_exc_filt. SL

* Per issue #1858: in process_config(): added code to set_obs_qty_inc_filt and set_obs_qty_exc_filt. Also added code to check if old obs_quality is being used, if so, set it to obs_qty_inc_filt. SL

* Per issue #1858: in add_point_obs() modified section that applies the obs_qty_inc and obs_qty_exc filters. Combined the if-blocks into one concise if-block that applies both filters. SL

* Per issue #1858: replaced obs_quality with obs_quality_inc and added obs_quality_inc. SL

* Per issue #1858: replaced obs_quality with obs_quality_inc and added obs_quality_exc. SL

* Per issue #1858: For both the point_stat and ensemble_stat tests: modified tests that use OBS_QTY to use OBS_QTY_INC. Added unit test for OBS_QTY_EXC (obs_quality_exclude). SL

* Per issue #1858: just added comment at top of file indicating what changed. SL

* Per issue #1858: just added comment at top of file indicating what changed. SL

* Per issue #1858: removed parse_conf_obs_qty. Modified parse_conf_obs_qty_inc to check for old 'obs_quality' entry. SL

* Per issue #1858: in process_config(), modified section that calls set_obs_qty_inc_filt...this new function can now handle old 'obs_quality' entry as well as the new 'obs_quality_inc'. SL

* Modified parse_conf_obs_qty_inc and parse_conf_obs_qty_exc. SL

* Per issue #1858: re-updated parse_conf_obs_qty_exc and reverted back to the simplier version of this function. SL

* Per issue #1858: replaced obs_quality with obs_quality_inc and added obs_quality_exc. SL

* Per issue #1858: replaced obs_quality with obs_quality_inc and added obs_quality_exc. SL

* Per issue #1858: replaced obs_quality with obs_quality_inc and added obs_quality_exc. SL

* Per issue #1858: replaced obs_quality with obs_quality_inc and added obs_quality_exc. SL

* Per issue #1858: replaced obs_quality with obs_quality_inc and added obs_quality_inc. SL

* Per issue #1858: replaced obs_quality with obs_quality_inc. Added documentation for obs_quality_exc. SL

* Per issue #1858: checking in latest version. Modified section that calls set_obs_qty_inc_filt...this new function can now handle old 'obs_quality' entry as well as the new 'obs_quality_inc'. SL

* Per issue #1858: cleaned up some extra line spaces. SL

* Per #1858, removed the deprecated obs_quality configuration entry from 3 EnsembleStat and 5 PointStat configuration files.

* Per issue #1858: duplicated fcst/obs fields to run/test with both obs_quality_inc and obs_quality_exc. SL

* Per issue #1858: duplicated fcst/obs fields to run/test with both obs_quality_inc and obs_quality_exc. SL

* Per issue #1858. Deleted these files git repository. These were replaced with: PointStatConfig_qty_inc_exc. SL

* Per issue #1858. Deleted these files from the git repository. These were replaced with: EnsembleStatConfig_qty_inc_exc. SL

* Per issue #1858. Deleted these files from the git repository. These were replaced with: PointStatConfig_qty_inc_exc. SL

* Per issue #1858: Modified: Don't need to run separate tests for obs_quality_inc and obs_quality_exc, these are now combined into one test via the updated config files: PointStatConfig_qty_inc_exc and EnsembleStatConfig_qty_inc_exc. SL

* Per #1858, update the logic in parse_conf_obs_qty_inc() to use last_lookup_status(). See forthcoming GitHub issue comment about this.

* Per #1858, tweak the warning message.

* Per #1858, made the existing unit_quality_filter.xml tests a bit more concise. Call Point-Stat and Ensemble-Stat only once each time. But for each, test 3 things: all qty, obs_quality_inc, and obs_quality_exc.

* Per #1858, remove the _ON from the job name since the _OFF has been included in the same configuration.

Co-authored-by: Seth Linden <linden@kiowa.rap.ucar.edu>
Co-authored-by: John Halley Gotway <johnhg@ucar.edu>
Co-authored-by: John Halley Gotway <johnhg@kiowa.rap.ucar.edu>

met/data/config/EnsembleStatConfig_default

@@ -114,7 +114,8 @@ message_type   = [ "ADPUPA" ];
 sid_inc        = [];
 sid_exc        = [];
 obs_thresh     = [ NA ];
-obs_quality    = [];
+obs_quality_inc  = [];
+obs_quality_exc  = [];
 duplicate_flag = NONE;
 obs_summary    = NONE;
 obs_perc_value = 50;

met/data/config/PointStatConfig_default

@@ -97,7 +97,8 @@ obs = fcst;
 message_type   = [ "ADPUPA" ];
 sid_inc        = [];
 sid_exc        = [];
-obs_quality    = [];
+obs_quality_inc  = [];
+obs_quality_exc  = [];
 duplicate_flag = NONE;
 obs_summary    = NONE;
 obs_perc_value = 50;

met/docs/Users_Guide/config_options.rst

@@ -1997,20 +1997,37 @@ Percentile value to use when obs_summary = PERC
   obs_perc_value = 50;
 
   
-.. _obs_quality:
+.. _obs_quality_inc:
 
-:ref:`obs_quality <obs_quality>`
+:ref:`obs_quality_inc <obs_quality_inc>`
 
-The "obs_quality" entry specifies the quality flag values that are to be
+The "obs_quality_inc" entry specifies the quality flag values that are to be
 retained and used for verification. An empty list signifies that all
 point observations should be used, regardless of their quality flag value.
 The quality flag values will vary depending on the original source of the
 observations. The quality flag values to retain should be specified as
 an array of strings, even if the values themselves are numeric.
+Note "obs_quality_inc" replaces the older option "obs_quality".
 
 .. code-block:: none
 		
-  obs_quality = [ "1", "2", "3", "9" ];
+  obs_quality_inc = [ "1", "2", "3", "9" ];
+
+  
+.. _obs_quality_exc:
+
+:ref:`obs_quality_exc <obs_quality_exc>`
+
+The "obs_quality_exc" entry specifies the quality flag values that are to be
+ignored and not used for verification. An empty list signifies that all
+point observations should be used, regardless of their quality flag value.
+The quality flag values will vary depending on the original source of the
+observations. The quality flag values to ignore should be specified as
+an array of strings, even if the values themselves are numeric.
+
+.. code-block:: none
+		
+  obs_quality_exc = [ "1", "2", "3", "9" ];
 
   
 .. _met_data_dir:

met/docs/Users_Guide/ensemble-stat.rst

@@ -149,7 +149,8 @@ ____________________
   sid_inc        = [];
   sid_exc        = [];
   duplicate_flag = NONE;
-  obs_quality    = [];
+  obs_quality_inc  = [];
+  obs_quality_exc  = [];
   obs_summary    = NONE;
   obs_perc_value = 50;
   message_type_group_map = [...];

met/docs/Users_Guide/point-stat.rst

@@ -342,7 +342,8 @@ ________________________
   sid_inc        = [];
   sid_exc        = [];
   duplicate_flag = NONE;
-  obs_quality    = [];
+  obs_quality_inc  = [];
+  obs_quality_exc  = [];
   obs_summary    = NONE;
   obs_perc_value = 50;
   message_type_group_map = [...];

met/scripts/config/EnsembleStatConfig

@@ -136,7 +136,8 @@ obs = fcst;
 sid_inc        = [];
 sid_exc        = [];
 obs_thresh     = [ NA ];
-obs_quality    = [];
+obs_quality_inc  = [];
+obs_quality_exc  = [];
 duplicate_flag = NONE;
 obs_summary    = NONE;
 obs_perc_value = 50;

met/scripts/config/PointStatConfig

@@ -77,7 +77,8 @@ obs = fcst;
 //
 sid_inc        = [];
 sid_exc        = [];
-obs_quality    = [];
+obs_quality_inc  = [];
+obs_quality_exc  = [];
 duplicate_flag = NONE;
 obs_summary    = NONE;
 obs_percentile = 50;

met/src/basic/vx_config/config_constants.h

@@ -535,6 +535,8 @@ static const char conf_key_message_type[]      = "message_type";
 static const char conf_key_sid_inc[]           = "sid_inc";
 static const char conf_key_sid_exc[]           = "sid_exc";
 static const char conf_key_obs_qty[]           = "obs_quality";
+static const char conf_key_obs_qty_inc[]       = "obs_quality_inc";
+static const char conf_key_obs_qty_exc[]       = "obs_quality_exc";
 static const char conf_key_convert[]           = "convert";
 static const char conf_key_censor_thresh[]     = "censor_thresh";
 static const char conf_key_censor_val[]        = "censor_val";

met/src/basic/vx_config/config_util.cc

@@ -766,11 +766,36 @@ vector<MaskLatLon> parse_conf_llpnt_mask(Dictionary *dict) {
 
 ///////////////////////////////////////////////////////////////////////////////
 
-StringArray parse_conf_obs_qty(Dictionary *dict) {
-   const char *method_name = "parse_conf_obs_qty() -> ";
+StringArray parse_conf_obs_qty_inc(Dictionary *dict) {
+   StringArray sa;
+   const char *method_name = "parse_conf_obs_qty_inc() -> ";
+
+   // Check for old "obs_quality" entry
+   sa = dict->lookup_string_array(conf_key_obs_qty, false);
 
-   StringArray sa = parse_conf_string_array(dict, conf_key_obs_qty, method_name);
+   // Print a warning if the deprecated option was used
+   if(dict->last_lookup_status()) {
+      mlog << Warning << "\nparse_conf_obs_qty_inc() -> "
+           << "Set the \"" << conf_key_obs_qty_inc << "\" value ("
+           << write_css(sa) << ") from the deprecated \""
+           << conf_key_obs_qty << "\" configuration entry.\n"
+           << "Replace \"" << conf_key_obs_qty << "\" with \""
+           << conf_key_obs_qty_inc << "\"!\n\n";
+   }
+   else {
+      sa = parse_conf_string_array(dict, conf_key_obs_qty_inc, method_name);
+   }
+
+   return(sa);
+}
+
+///////////////////////////////////////////////////////////////////////////////
 
+StringArray parse_conf_obs_qty_exc(Dictionary *dict) {
+   const char *method_name = "parse_conf_obs_qty_exc() -> ";
+
+   StringArray sa = parse_conf_string_array(dict, conf_key_obs_qty_exc, method_name);
+
    return(sa);
 }
 

met/src/basic/vx_config/config_util.h

@@ -40,7 +40,8 @@ extern StringArray     parse_conf_sid_list(Dictionary *dict, const char *);
 extern void            parse_sid_mask(const ConcatString &, StringArray &, ConcatString &);
 extern vector<MaskLatLon>
                        parse_conf_llpnt_mask(Dictionary *dict);
-extern StringArray     parse_conf_obs_qty(Dictionary *dict);
+extern StringArray     parse_conf_obs_qty_inc(Dictionary *dict);
+extern StringArray     parse_conf_obs_qty_exc(Dictionary *dict);
 extern NumArray        parse_conf_ci_alpha(Dictionary *dict);
 extern NumArray        parse_conf_eclv_points(Dictionary *dict);
 extern ClimoCDFInfo    parse_conf_climo_cdf(Dictionary *dict);

met/src/libcode/vx_statistics/pair_data_ensemble.cc

@@ -913,8 +913,9 @@ void VxPairDataEnsemble::clear() {
 
    sid_inc_filt.clear();
    sid_exc_filt.clear();
-   obs_qty_filt.clear();
-
+   obs_qty_inc_filt.clear();
+   obs_qty_exc_filt.clear();
+
    obs_error_info = (ObsErrorInfo *) 0;
 
    fcst_ut = (unixtime) 0;
@@ -954,7 +955,8 @@ void VxPairDataEnsemble::assign(const VxPairDataEnsemble &vx_pd) {
    end_ut         = vx_pd.end_ut;
    sid_inc_filt   = vx_pd.sid_inc_filt;
    sid_exc_filt   = vx_pd.sid_exc_filt;
-   obs_qty_filt   = vx_pd.obs_qty_filt;
+   obs_qty_inc_filt = vx_pd.obs_qty_inc_filt;
+   obs_qty_exc_filt = vx_pd.obs_qty_exc_filt;
    obs_error_info = vx_pd.obs_error_info;
 
    interp_thresh  = vx_pd.interp_thresh;
@@ -1132,9 +1134,18 @@ void VxPairDataEnsemble::set_sid_exc_filt(const StringArray sa) {
 
 ////////////////////////////////////////////////////////////////////////
 
-void VxPairDataEnsemble::set_obs_qty_filt(const StringArray q) {
+void VxPairDataEnsemble::set_obs_qty_inc_filt(const StringArray q) {
 
-   obs_qty_filt = q;
+   obs_qty_inc_filt = q;
+
+   return;
+}
+
+////////////////////////////////////////////////////////////////////////
+
+void VxPairDataEnsemble::set_obs_qty_exc_filt(const StringArray q) {
+
+   obs_qty_exc_filt = q;
 
    return;
 }
@@ -1368,16 +1379,13 @@ void VxPairDataEnsemble::add_point_obs(float *hdr_arr, int *hdr_typ_arr,
    else if(obs_info_grib->code() != nint(obs_arr[1])) {
       return;
    }
-
-   // Check if the observation quality flag is included in the list
-   if(obs_qty_filt.n() && strcmp(obs_qty, "")) {
-      bool qty_match = false;
-      for(i=0; i<obs_qty_filt.n() && !qty_match; i++)
-         if( obs_qty == obs_qty_filt[i]) qty_match = true;
-
-      if(!qty_match) return;
+
+   // Check the observation quality include and exclude options
+   if((obs_qty_inc_filt.n() > 0 && !obs_qty_inc_filt.has(obs_qty)) ||
+      (obs_qty_exc_filt.n() > 0 &&  obs_qty_exc_filt.has(obs_qty))) {
+      return;
    }
-
+   
    // Check whether the observation time falls within the valid time
    // window
    if(hdr_ut < beg_ut || hdr_ut > end_ut) return;

met/src/libcode/vx_statistics/pair_data_ensemble.h

@@ -205,8 +205,9 @@ class VxPairDataEnsemble {
 
       StringArray sid_inc_filt;  // Station ID inclusion list
       StringArray sid_exc_filt;  // Station ID exclusion list
-      StringArray obs_qty_filt;  // Observation quality markers
-
+      StringArray obs_qty_inc_filt;  // Observation quality include markers
+      StringArray obs_qty_exc_filt;  // Observation quality exclude markers
+
       //////////////////////////////////////////////////////////////////
 
       ObsErrorInfo *obs_error_info; // Pointer for observation error
@@ -249,8 +250,9 @@ class VxPairDataEnsemble {
 
       void set_sid_inc_filt(const StringArray);
       void set_sid_exc_filt(const StringArray);
-      void set_obs_qty_filt(const StringArray);
-
+      void set_obs_qty_inc_filt(const StringArray);
+      void set_obs_qty_exc_filt(const StringArray);
+
       // Call set_pd_size before set_msg_typ, set_mask_area, and set_interp
       void set_pd_size(int, int, int);
 

met/src/libcode/vx_statistics/pair_data_point.cc

@@ -372,7 +372,8 @@ void VxPairDataPoint::clear() {
    climo_sd_dpa.clear();
    sid_inc_filt.clear();
    sid_exc_filt.clear();
-   obs_qty_filt.clear();
+   obs_qty_inc_filt.clear();
+   obs_qty_exc_filt.clear();
    mpr_column.clear();
    mpr_thresh.clear();
 
@@ -433,7 +434,8 @@ void VxPairDataPoint::assign(const VxPairDataPoint &vx_pd) {
 
    sid_inc_filt = vx_pd.sid_inc_filt;
    sid_exc_filt = vx_pd.sid_exc_filt;
-   obs_qty_filt = vx_pd.obs_qty_filt;
+   obs_qty_inc_filt = vx_pd.obs_qty_inc_filt;
+   obs_qty_exc_filt = vx_pd.obs_qty_exc_filt;
 
    mpr_column = vx_pd.mpr_column;
    mpr_thresh = vx_pd.mpr_thresh;
@@ -629,9 +631,18 @@ void VxPairDataPoint::set_sid_exc_filt(const StringArray &sa) {
 
 ////////////////////////////////////////////////////////////////////////
 
-void VxPairDataPoint::set_obs_qty_filt(const StringArray &sa) {
+void VxPairDataPoint::set_obs_qty_inc_filt(const StringArray &sa) {
 
-   obs_qty_filt = sa;
+   obs_qty_inc_filt = sa;
+
+   return;
+}
+
+////////////////////////////////////////////////////////////////////////
+
+void VxPairDataPoint::set_obs_qty_exc_filt(const StringArray &sa) {
+
+   obs_qty_exc_filt = sa;
 
    return;
 }
@@ -899,10 +910,10 @@ void VxPairDataPoint::add_point_obs(float *hdr_arr, const char *hdr_typ_str,
       rej_sid++;
       return;
    }
-
+   
    // Check whether the GRIB code for the observation matches
    // the specified code
-   if((var_name != 0) && (0 < m_strlen(var_name))) {
+   if((var_name != 0) && (0 < strlen(var_name))) {
       if(var_name != obs_info->name()) {
          rej_var++;
          return;
@@ -912,19 +923,14 @@ void VxPairDataPoint::add_point_obs(float *hdr_arr, const char *hdr_typ_str,
       rej_var++;
       return;
    }
-
-   // Check if the observation quality flag is included in the list
-   if(obs_qty_filt.n() && strcmp(obs_qty, "")) {
-      bool qty_match = false;
-      for(i=0; i<obs_qty_filt.n() && !qty_match; i++) {
-         if(obs_qty == obs_qty_filt[i]) qty_match = true;
-      }
-      if(!qty_match) {
-         rej_qty++;
-         return;
-      }
+
+   // Check the observation quality include and exclude options
+   if((obs_qty_inc_filt.n() > 0 && !obs_qty_inc_filt.has(obs_qty)) ||
+      (obs_qty_exc_filt.n() > 0 &&  obs_qty_exc_filt.has(obs_qty))) {
+      rej_qty++;
+      return;
    }
-
+   
    // Check whether the observation time falls within the valid time
    // window
    if(hdr_ut < beg_ut || hdr_ut > end_ut) {

met/src/libcode/vx_statistics/pair_data_point.h

@@ -128,8 +128,9 @@ class VxPairDataPoint {
 
       StringArray sid_inc_filt;  // Station ID inclusion list
       StringArray sid_exc_filt;  // Station ID exclusion list
-      StringArray obs_qty_filt;  // Observation quality markers
-
+      StringArray obs_qty_inc_filt;  // Observation quality include markers
+      StringArray obs_qty_exc_filt;  // Observation quality exclude markers
+
       //////////////////////////////////////////////////////////////////
 
       StringArray mpr_column;    // Names of MPR columns or diffs of columns
@@ -199,8 +200,9 @@ class VxPairDataPoint {
 
       void set_sid_inc_filt(const StringArray &);
       void set_sid_exc_filt(const StringArray &);
-      void set_obs_qty_filt(const StringArray &);
-
+      void set_obs_qty_inc_filt(const StringArray &);
+      void set_obs_qty_exc_filt(const StringArray &);
+
       // Call set_pd_size before set_msg_typ, set_mask_area, and set_interp
       void set_pd_size(int, int, int);
 

met/src/tools/core/ensemble_stat/ensemble_stat.cc

@@ -59,6 +59,8 @@
 //   029    01/21/20  Halley Gotway  Add RPS output line type.
 //   030    02/19/21  Halley Gotway  MET #1450, #1451 Overhaul CRPS
 //                    statistics in the ECNT line type.
+//   031    09/13/21  Seth Linden    Changed obs_qty to obs_qty_inc.
+//                    Added code for obs_qty_exc.
 //
 ////////////////////////////////////////////////////////////////////////
 

met/src/tools/core/ensemble_stat/ensemble_stat_conf_info.cc

@@ -819,19 +819,22 @@ void EnsembleStatVxOpt::process_config(GrdFileType ftype, Dictionary &fdict,
            << "Observation error for point verification is "
            << "defined by a table lookup for each observation.\n";
    }
-
+   
    // Conf: desc
    vx_pd.set_desc(parse_conf_string(&odict, conf_key_desc).c_str());
-
+   
    // Conf: sid_inc
    vx_pd.set_sid_inc_filt(parse_conf_sid_list(&odict, conf_key_sid_inc));
-
+   
    // Conf: sid_exc
    vx_pd.set_sid_exc_filt(parse_conf_sid_list(&odict, conf_key_sid_exc));
-
-   // Conf: obs_qty
-   vx_pd.set_obs_qty_filt(parse_conf_obs_qty(&odict));
-
+
+   // Conf: obs_qty_inc
+   vx_pd.set_obs_qty_inc_filt(parse_conf_obs_qty_inc(&odict));
+
+   // Conf: obs_qty_exc
+   vx_pd.set_obs_qty_exc_filt(parse_conf_obs_qty_exc(&odict));
+
    return;
 }