Bugfix Address negative values returned by calculate_bcmse() and calculate_bcrmse() in sl1l2_statistics module

[bikegeek]

Describe the Problem

The calculate_bcmse() and the calculate_bcrmse() functions in the sl1l2_statistics.py module are returning negative values. This may be due to the round_half_up() applied to the calculate_mse() and calculate_rmse() values before returning them. See @mollybsmith-noaa comments in the comment section below for a detailed explanation.

Expected Behavior

Provide a clear and concise description of what you expected to happen here.

Environment

Describe your runtime environment:
1. Machine: (e.g. HPC name, Linux Workstation, Mac Laptop)
2. OS: (e.g. RedHat Linux, MacOS)
3. Software version number(s)

To Reproduce

Describe the steps to reproduce the behavior:
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error
Post relevant sample data following these instructions:
https://dtcenter.org/community-code/model-evaluation-tools-met/met-help-desk#ftp

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

Define the source of funding and account keys here or state NONE.

Define the Metadata

Assignee

• Select engineer(s) or no engineer required
• Select scientist(s) or no scientist required

Labels

• Select component(s)
• Select priority
• Select requestor(s)

Projects and Milestone

• Select Organization level Project for support of the current coordinated release
• Select Repository level Project for development toward the next official release or add alert: NEED CYCLE ASSIGNMENT label
• Select Milestone as the next bugfix version

Define Related Issue(s)

Consider the impact to the other METplus components.

• METplus, MET, METdataio, METviewer, METexpress, METcalcpy, METplotpy

Bugfix Checklist

See the METplus Workflow for details.

• Complete the issue definition above, including the Time Estimate and Funding Source.
• Fork this repository or create a branch of main_<Version>.
  Branch name: bugfix_<Issue Number>_main_<Version>_<Description>
• Fix the bug and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Add any new Python packages to the METplus Components Python Requirements table.
• Push local changes to GitHub.
• Submit a pull request to merge into main_<Version>.
  Pull request: bugfix <Issue Number> main_<Version> <Description>
• Define the pull request metadata, as permissions allow.
  Select: Reviewer(s) and Development issues
  Select: Organization level software support Project for the current coordinated release
  Select: Milestone as the next bugfix version
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Complete the steps above to fix the bug on the develop branch.
  Branch name: bugfix_<Issue Number>_develop_<Description>
  Pull request: bugfix <Issue Number> develop <Description>
  Select: Reviewer(s) and Development issues
  Select: Repository level development cycle Project for the next official release
  Select: Milestone as the next official version
• Close this issue.

[mollybsmith-noaa]

The description in this issue is not correct. The problem is that calculate_bcmse is returning negative numbers, which it should never do, because it calls both calculate_mse and caluclate_me, which are positive but are subject to rounding. This means that when one is subtracted from the other, a negative value can be produced. @bikegeek does this make sense?

[bikegeek]

@mollybsmith-noaa can you please provide some data so we can re-create and test?

[mollybsmith-noaa]

I'm getting some data behind the case in dtcenter/METexpress#127, which is where I first found out about this bug.

[mollybsmith-noaa]

Function definition is: def calculate_bcrmse(input_data, columns_names, aggregation=False):

These produce the bug:

input_data = array([[8.69988460e+02, 3.63246150e+02, 7.56879923e+05, 1.31947769e+05, 3.16019963e+05, 1.00000000e+00]])
columns_names = array(['fbar', 'obar', 'ffbar', 'oobar', 'fobar', 'total'], dtype='<U5')

input_data = array([[4.37978400e+01, 4.70115800e+01, 1.91825108e+03, 2.21008843e+03, 2.05900571e+03, 1.00000000e+00]])
columns_names = array(['fbar', 'obar', 'ffbar', 'oobar', 'fobar', 'total'], dtype='<U5')

input_data = array([[8.66233900e+01, 4.83037900e+01, 7.50361146e+03, 2.33325660e+03, 4.18423840e+03, 1.00000000e+00]])
columns_names = array(['fbar', 'obar', 'ffbar', 'oobar', 'fobar', 'total'], dtype='<U5')

input_data = array([[3.68089000e+01, 1.64253370e+02, 1.35489535e+03, 2.69791703e+04, 6.04598647e+03, 1.00000000e+00]])
columns_names = array(['fbar', 'obar', 'ffbar', 'oobar', 'fobar', 'total'], dtype='<U5')

It's not every point on the graph but it can be several.

[bikegeek]

Successfully reproducing the negative BCMSE values with the provided data. However, attempts to remove rounding up in the ME calculation still results in negative BCMSE values. Eric suggests setting the result to 0 when MSE < ME^2.