Enhance TC-Pairs to read hurricane model diagnostic files (e.g. SHIPS) and TC-Stat to filter the new data

[dwfncar]

Describe the Enhancement

This task is requested by Paul Kucera and will be done to support the HFIP evaluations.

Currently, the MET-TC tools only handle hurricane ATCF files. This task is to add support for parsing/using hurricane model diagnostic files. The diagnostic files are ASCII and can be produced for each model/storm/initialization time. The diagnostic file contains track information for each lead time, just like the ATCF files, but also include many derived quantities, like shear and upper level quantities averaged over the storm area.

Enhance TC-Pairs to read TC diagnostics data in two different file formats:

• The LS-Diag (Large Scale Diagnostics) format and the Model-Diag format
• This data may have been generated by the new TC-Diag tool (see Create an initial development version of a new TC-Diag tool to support the computation of tropical cyclone diagnostics #2168) or by other software.

ACTION: After parsing this diagnostics data and finding a match for the model track currently being evaluated, need to decide how to write the diagnostics to the output.

Consider either tacking on the diagnostics data to the end of the existing TCMPR line type. Or consider writing a new TCDIAG line type containing only the diagnostics values. This decision should be driven by downstream needs of METdatadb and METviewer. What makes the most sense since the number and type of diagnostic values can change? We wouldn't have a fixed set of output columns. They would vary based on the user configuration of the TC-Diag tool.

Time Estimate

1 week

Sub-Issues

Consider breaking the enhancement down into sub-issues.

• Add a checkbox for each sub-issue here.

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

2770043

Define the Metadata

Assignee

• Select engineer(s) or no engineer required: @JohnHalleyGotway
• Select scientist(s) or no scientist required: @jvigh and @KathrynNewman

Labels

• Select component(s)
• Select priority
• Select requestor(s)

Projects and Milestone

• Select Repository and/or Organization level Project(s) or add alert: NEED PROJECT ASSIGNMENT label
• Select Milestone as the next official version or Future Versions

Define Related Issue(s)

Consider the impact to the other METplus components.

• MET: Create an initial development version of a new TC-Diag tool to support the computation of tropical cyclone diagnostics #2168
• METplus: List issue here
• METdatadb: List issue here
• METviewer: List issue here

Enhancement Checklist

See the METplus Workflow for details.

• Complete the issue definition above, including the Time Estimate and Funding Source.
• Fork this repository or create a branch of develop.
  Branch name: feature_<Issue Number>_<Description>
• Complete the development and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Push local changes to GitHub.
• Submit a pull request to merge into develop.
  Pull request: feature <Issue Number> <Description>
• Define the pull request metadata, as permissions allow.
  Select: Reviewer(s) and Linked issues
  Select: Repository level development cycle Project for the next official release
  Select: Milestone as the next official version
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Close this issue.

This task is requested by Paul Kucera and will be done to support the HFIP evaluations.

Currently, the MET-TC tools only handle hurricane ATCF files. This task is to add support for parsing/using hurricane model diagnostic files. The diagnostic files are ASCII and can be produced for each model/storm/initialization time. The diagnostic file contains track information for each lead time, just like the ATCF files, but also include many derived quantities, like shear and upper level quantities averaged over the storm area.

Define new classes in the vx_tc_util library for reading this diagnostic data. Suggest including the ability to write the diagnostic data to a file. This could be a new line type, like TCMPR but maybe TCDIAG, for tropical cyclone diagnostics. The header columns would need to be the same as those in TCMPR.

The wrinkle here is that the contents of the diagnostic file can vary based on how the diagnostics were generated. So we wouldn't have a fixed set of output columns. Instead, after the header columns, the column names would indicate the contents.

(1) Suggest writing a new tool that just reads in a bunch of diagnostic files and writes them out to a single file. This would be a reformatting step and make it easier to process the diagnostic data.

(2) Suggest enhancing the tc_pairs tool to also read in the diagnostic files. For each ATCF track, look for a matching diagnostic track. Look in the tc_pairs config file at the user-specified list of diagnostic quantities to be included. Then tack those diagnostic quantities to the end of the line. Include support for short-cuts like "ALL" and partial string matching, like "p500" for 500mb quantities.

(3) Suggest enhancing the tc_stat tool to read the TCDIAG line type and filter the lines. Also enhance tc_stat to read the additional variable columns from the TC_MPR lines.

[MET-392] created by johnhg

[dwfncar]

This work was requested a second time by Paul Kucera and Mark Demaria at NHC. Funding will be allocated in 2018 for this. Here's some selections of emails about it...

From Paul...
The HFIP office said they have some funding to further develop MET-TC tools. In particular, NHC would like this feature added: stratify the track and intensity errors by vertical wind shear (and other variables like low-level humidity). Would you be able to give me a ball-park estimate of cost to add that feature? I will add that to my HFIP budget for this year.

From John...

Processing the gridded data files ourselves would open up a lot of flexibility... but would require that those gridded data files actually be present for each ATCF track ID being processed. It may also be slow to process gridded data for each forecast hour.

Using the diagnostic output would be faster in MET but would then require that the diagnostic tool be run on the gridded data sets. Perhaps we’d need to support that diagnostic tool to the broader community?

The real problem here is that the ATCF tracks do not contain enough diagnostic info... like wind shear. And there’s a few ways we could address that... from updating the output of the GFDL vortex tracker itself, to reading the existing diagnostic files, to generating those diagnostics directly in MET.

My first inclination would be to reimplement the diagnostic functionality as a new tool in MET (tc_diag)... to make it available and supportable. And then update tc_stat to read the output of both tc_pairs and tc_diag when doing its filtering. But let’s talk more about it next week.

Also, it would be really great to update METViewer to read/plot this track data as well, and replace the Rscripts we currently run!

From Paul...
I heard back from Mark. He suggested that we first use Kate's diagnostic files and/or the output from the operational SHIPS model. I am assuming that will be a bit more straightforward. by johnhg

[dwfncar]

Paul Kucera has identified funding for this work and would like it included after the met-8.1 release.


The funding for this ends July 2019, and the work could be coordinated with the tc-rmw and tc-gen enhancements.


Paul reiterated this on 2019/03/20.
by johnhg

[TaraJensen]

Moved this into MET 10.1.0 development cycle during discussion with HAFS developers. They see this as a high priority.

[jvigh]

@TaraJensen @JohnHalleyGotway

Christopher Rozoff has now developed a more comprehensive suite of ensemble diagnostics which are similar to the SHIPS diagnostics. We are in the process of finalizing these. They will be in NetCDF format. We'd like to work with you to figure out what format they need to be in to be easily read by MET-TC (e.g., is CF-compliant sufficient?).

Paul's funding for his EnsRI/NHCvx projects will be ending Sep 30 (hard deadline). It might be possible to devote a bit of time to MET development -- we'd need to discuss with Paul.

[JohnHalleyGotway]

Had some discussions over email with @jvigh about this work. They are summarized below.

From Jonathan:
The format is being finalized right now by Chris Rozoff. Our goal is to make it as close as possible to being CF-compliant as possible, noting that many of the quantities may not have standard names or units that are included in the CF-standards.

From John HG:
While MET does currently read NetCDF files following the gridded CF-convention, it does not currently read any point observations following CF point convention. While CF is in wide usage for gridded data, it's less widely used for point data, to my knowledge. But there still may be some code that's reusable... like the parsing of date/time strings.

The TC-Pairs tool reads ATCF forecast track data and ATCF Best track data, and for each storm matches up their common track point. And for each fcst/Best track point pair, it computes along/cross/total track error and writes that to the output. The TC-Stat tool reads the TC-Pairs output, filters it in whatever way the user says, and then performs some analysis job on that data.

From Jonathan:
They are not gridded data. All of the diagnostic values are "storm-centric", meaning they apply to the storm environment or some aspect of the storm structure. For example, a key parameter would be environmental vertical wind shear. There could actually be several of these, but the idea is the same -- they are a value that represent the diagnostic quantity at a point in time (just like track position or intensity). So the existing logic of tcpairs should work well for this. It may need to be enhanced, say, to allow one to stratify verification results by weak, moderate, or strong vertical shear. Another example would the the Maximum Potential Intensity (MPI) of a storm, etc.

From John HG:
Great, thanks for clarifying. In my mind, we'd set it up so that...

• the user specifies the source of one or more diagnostics files to be read on the tc_pairs command line
• the user specifies the names of the diagnostic values in which they're interested in the tc_pairs configuration file
• for each requested diagnostic, tc_pairs appends a new column of output to the existing tc_pairs output line type
• for each track point, tc_pairs searches the diagnostic file(s) for the requested data and appends it to the end of each line
• tc_stat is able to filter the output of tc_pairs by thresholding the values in the "extra" columns by name
  • for example: -column_thresh VERTICAL_SHEAR >100
  • that'd mean, only retain TCST lines from tc_pairs that have a column named VERTICAL_SHEAR at the end and that have a value > 100.

I think the main work will be writing code to parse the diagnostic info from NetCDF files into memory.

From Jonathan:
I'm pretty sure there won't be 80-100 hours available for this. But there could possibly be 30-60 h, if Paul wants to go this route. If Tara had other funds to leverage, it might work (if you think it's really an 80+ h task).

[JohnHalleyGotway]

Please find sample diagnostic data from Chris attached.
ida2021082700.tar.gz