Allow 2x2 HSS calculations to include user-defined EC values

[j-opatz]

Describe the New Feature

During the latest working session with CPC, it was determined that the method for calculating HSS for multi-categorical thresholds differs from MET.

MET method: Fill in the NxN contingency table, using the user-defined cat_thresh on the forecast and observation fields for each gridpoint. Sum Hits and Misses across the bins, and use the HSS equation to calculate 1 HSS value for the table. If the user applied hss_ec_value, set C2 variable to what user assigned.

CPC method: Fill in the NxN contingency table, using the user-defined cat_thresh on the forecast fields. Observation fields are predetermined to be hits or misses (binary). For each forecast field threshold, calculate the 2x2 Hits and Misses and use the HSS equation to calculate an HSS value for each forecast threshold. Additionally, this will always utilize a user's hss_ec_value.

Another description of this would be that regardless of the number of bins, CPC will evaluate each threshold as a 2x2 contingency table with a predefined C2 value in the HSS equation.

As MET capability currently stands, the hss_ec_value variable is only applicable to MCTS line type. But as the description above indicates, CPC would need this functionality in the CTS line type (or alternatively, numerous columns in MCTS where each threshold has its own HSS value).

While not performed in the traditional manner of HSS, Barb Brown confirmed that this passes statistical soundness.

Acceptance Testing

Probability of Exceedance fields (fcst) and binary observation fields, which @j-opatz can provide.

The acceptance criteria would be to replicate CPC's HSS value for maximum temperature (85th percentile) with the provided data.

Time Estimate

Unknown, engineer will need to fill in
Issues should represent approximately 1 to 3 days of work.

Sub-Issues

Consider breaking the new feature down into sub-issues.
No sub-issues needed.

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

Split 2700041, 2793541

Define the Metadata

Assignee

• Select engineer(s) or no engineer required
• Select scientist(s) or no scientist required

Labels

• Select component(s)
• Select priority
• Select requestor(s)

Projects and Milestone

• Select Repository and/or Organization level Project(s) or add alert: NEED PROJECT ASSIGNMENT label
• Select Milestone as the next official version or Future Versions

Define Related Issue(s)

Consider the impact to the other METplus components.

• Need to define a METdatadb issue to load new output.
• Need to define a METviewer issue plot the new output.

New Feature Checklist

See the METplus Workflow for details.

• Complete the issue definition above, including the Time Estimate and Funding source.
• Fork this repository or create a branch of develop.
  Branch name: feature_<Issue Number>_<Description>
• Complete the development and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Push local changes to GitHub.
• Submit a pull request to merge into develop.
  Pull request: feature <Issue Number> <Description>
• Define the pull request metadata, as permissions allow.
  Select: Reviewer(s) and Linked issues
  Select: Repository level development cycle Project for the next official release
  Select: Milestone as the next official version
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Close this issue.

[JohnHalleyGotway]

@j-opatz I took a look and see that the HSS statistic already appears in column 77 of the CTS line type, column 35 of the MCTS line type, column 77 of the NBRCTS line type, and column 40 of the MODE CTS line type.

Checking the code, I see that the hss_ec_value is used in the computation of MCTS:HSS but not CTS:HSS.

Perhaps all we need to actually do is enhance the CTS:HSS computation to use the hss_ec_value, if specified in the configuration file? Or is there some other aspect to the data processing logic I'm not fully appreciating?

[j-opatz]

I think you've sussed out a solution well: if hss_ec_value is specified, use it for whatever line type is requested. The only issue I could see happening is where multiple thresholds are desired, but a user wants each one evaluated for its own HSS value. That would require n calls to a tool for CTS line type, which could get lengthy...I don't know if there's a realistic implementation using MCTS, though.

[JohnHalleyGotway]

• The MCTS line type includes output columns for HSS (+BCL/BCU), HSS_EC (+BCL/BCU), and EC_VALUE.
• The CTS line type only includes output columns for HSS (+BCL/BCU).
• The tasks here are...
• Add to the CTS line type HSS_EC (+BCL/BCU)
• Add to the CTS line type EC_VALUE
• Add to the CTC line type EC_VALUE
• Do NOT add these columns to the NBRCTS, NBRCTC line types, or the categorical output from MODE.
• Compute the CTS HSS_EC value and bootstrap it
• Update stat_column_description.txt to make sure it includes entries for all stats that can be computed by Series-Analysis. It might be out of date!
• Update Stat-Analysis to parse these new columns and to call set_ec_value when needed.

Will need METdatadb/METviewer issues to load/plot these new output column.
Also make sure to update Series-Analysis to handle the HSS_EC value.

[JohnHalleyGotway]

@j-opatz and @bgbrowntollerud can you please review this section of documentation about setting the hss_ec_value config option?

Just want to double-check to confirm that we're doing this correctly.

When computing the HSS_EC statistic for a 2x2 table, the default hss_ec_value = NA; results in using an expected correct value of 1.0 / 2 = 0.5.

Can you confirm that this is correct and appropriate default behavior?