Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document PB2NC's handling of quality markers #2278

Closed
8 of 20 tasks
JohnHalleyGotway opened this issue Sep 23, 2022 · 2 comments · Fixed by #2364
Closed
8 of 20 tasks

Document PB2NC's handling of quality markers #2278

JohnHalleyGotway opened this issue Sep 23, 2022 · 2 comments · Fixed by #2364
Assignees
Labels
component: documentation Documentation issue priority: low Low Priority reporting: DTC NOAA BASE NOAA Office of Atmospheric Research DTC Project requestor: METplus Team METplus Development Team type: task An actionable item of work
Milestone

Comments

@JohnHalleyGotway
Copy link
Collaborator

Describe the Task

This is a small documentation-only task to clarify/document PB2NC's handling of quality markers for derived variables, like CAPE. Also document if/how quality markers apply to the derivation of time summaries. These questions arose via GitHub discussions and should be clarified in the documentation.

Time Estimate

2 hours

Sub-Issues

Consider breaking the task down into sub-issues.
None needed.

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

Define the source of funding and account keys here or state NONE.

Define the Metadata

Assignee

  • Select engineer(s) or no engineer required: @JohnHalleyGotway
  • Select scientist(s) or no scientist required: No scientist needed

Labels

  • Select component(s)
  • Select priority
  • Select requestor(s)

Projects and Milestone

  • Select Repository and/or Organization level Project(s) or add alert: NEED PROJECT ASSIGNMENT label
  • Select Milestone as the next official version or Future Versions

Define Related Issue(s)

Consider the impact to the other METplus components.

Task Checklist

See the METplus Workflow for details.

  • Complete the issue definition above, including the Time Estimate and Funding Source.
  • Fork this repository or create a branch of develop.
    Branch name: feature_<Issue Number>_<Description>
  • Complete the development and test your changes.
  • Add/update log messages for easier debugging.
  • Add/update unit tests.
  • Add/update documentation.
  • Push local changes to GitHub.
  • Submit a pull request to merge into develop.
    Pull request: feature <Issue Number> <Description>
  • Define the pull request metadata, as permissions allow.
    Select: Reviewer(s) and Linked issues
    Select: Repository level development cycle Project for the next official release
    Select: Milestone as the next official version
  • Iterate until the reviewer(s) accept and merge your changes.
  • Delete your fork or branch.
  • Close this issue.
@JohnHalleyGotway JohnHalleyGotway added component: documentation Documentation issue type: task An actionable item of work priority: low Low Priority alert: NEED ACCOUNT KEY Need to assign an account key to this issue requestor: METplus Team METplus Development Team labels Sep 23, 2022
@JohnHalleyGotway JohnHalleyGotway added this to the MET 11.0.0 milestone Sep 23, 2022
@JohnHalleyGotway JohnHalleyGotway self-assigned this Sep 23, 2022
JohnHalleyGotway added a commit that referenced this issue Nov 30, 2022
…. Also, update pb2nc logic to track the maximum quality mark for CAPE and PBL rather than just the most recent one! I wonder if that'll change the regression test output?
JohnHalleyGotway added a commit that referenced this issue Nov 30, 2022
…ues of bad data. See unit test output file ndas.20120409.t12z.prepbufr.tm00.summary.nc. Update the print_pointnc2ascii.py to avoid runtime error about convert masked array elements to int.
@JohnHalleyGotway
Copy link
Collaborator Author

I've realized/documented the fact that time summaries are always written by PB2NC with a quality flag of NA. For other derived obs, like CAPE, DPT, and so on, it's computed as the maximum of the quality marks that went into the derivation.

So the question is...

  • Should we leave the functionality as is and be satisfied that we documented it?
  • Or should we modify the library code used to compute time summaries, check to see if the all the quality "strings" actually contain integers, and if so, report the maximum of those integers?

Per @TaraJensen, we should investigate the second option. In BUFR, lower integers are better. Technically, someone could run ascii2nc to compute time summaries with integer qm flags where HIGHER is better.

So attempt to only enable this for PB2NC, not the other point pre-processing tools.

@JohnHalleyGotway
Copy link
Collaborator Author

JohnHalleyGotway commented Dec 1, 2022

@TaraJensen I investigated updating the handling of quality flag values for time summaries and have decided that it would constitute too large of a change to be introduced during the MET 11.0.0 pre-release cycle.

I do recommend that we make this change, but I think we should do it carefully.

MET supports the computation of several types of time summaries (min, max, range, mean, stdev, median, P##, sum). For the min, max, median, and perhaps P## options, we should use the quality mark of the actual corresponding observation. For the range, we should report the max quality mark of the min/max value. For the mean, stdev, and sum, we should report the max quality mark for all observations in the time window.

I wrote up issue #2363 to describe these proposed changes.

JohnHalleyGotway added a commit that referenced this issue Dec 1, 2022
JohnHalleyGotway added a commit that referenced this issue Dec 1, 2022
…triction of computing CAPE and PBL quality marks from only the first vertical level. Instead, compute the maximum quality mark over all vertical levels.
JohnHalleyGotway added a commit that referenced this issue Dec 2, 2022
@JohnHalleyGotway JohnHalleyGotway linked a pull request Dec 5, 2022 that will close this issue
15 tasks
JohnHalleyGotway added a commit that referenced this issue Dec 5, 2022
Co-authored-by: Howard Soh <hsoh@seneca.rap.ucar.edu>
Repository owner moved this from In Progress to Done in MET-11.0.0 Pre-Release (12/9/22) Dec 9, 2022
@TaraJensen TaraJensen added the reporting: DTC NOAA BASE NOAA Office of Atmospheric Research DTC Project label Jan 25, 2023
@JohnHalleyGotway JohnHalleyGotway removed the alert: NEED ACCOUNT KEY Need to assign an account key to this issue label Oct 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component: documentation Documentation issue priority: low Low Priority reporting: DTC NOAA BASE NOAA Office of Atmospheric Research DTC Project requestor: METplus Team METplus Development Team type: task An actionable item of work
Projects
No open projects
Development

Successfully merging a pull request may close this issue.

2 participants