Create wrapper to reformat MET NetCDF output to be CF-compliant

[georgemccabe]

Implement a new METplus wrapper to handle reformatting NetCDF files generated by the MET tools to add information that is needed (i.e. grid, timing, etc.) to be CF-compliant and remove information that is not needed (i.e. init_time, valid_time, etc.)

Describe the Enhancement

Provide a description of the enhancement request here.

Time Estimate

Estimate the amount of work required here.
Issues should represent approximately 1 to 3 days of work.

Sub-Issues

Consider breaking the enhancement down into sub-issues.

• Add a checkbox for each sub-issue here.

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

Define the source of funding and account keys here or state NONE.

Define the Metadata

Assignee

• Select engineer(s) or no engineer required
• Select scientist(s) or no scientist required

Labels

• Select component(s)
• Select priority
• Select requestor(s)

Projects and Milestone

• Select Repository and/or Organization level Project(s) or add alert: NEED PROJECT ASSIGNMENT label
• Select Milestone as the next official version or Future Versions

Define Related Issue(s)

Consider the impact to the other METplus components.

• METplus, MET, METdatadb, METviewer, METexpress, METcalcpy, METplotpy

Enhancement Checklist

See the METplus Workflow for details.

• Complete the issue definition above, including the Time Estimate and Funding Source.
• Fork this repository or create a branch of develop.
  Branch name: feature_<Issue Number>_<Description>
• Complete the development and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Push local changes to GitHub.
• Submit a pull request to merge into develop.
  Pull request: feature <Issue Number> <Description>
• Define the pull request metadata, as permissions allow.
  Select: Reviewer(s) and Linked issues
  Select: Repository level development cycle Project for the next official release
  Select: Milestone as the next official version
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Close this issue.

[kaday]

@georgemccabe any thoughts on how you would like me to contribute on this. I am hoping to start getting time to spin-up and contribute over the next few weeks. Thanks, Kerry

[jprestop]

Hi @kaday. George is out of the office and will be back mid-next week. I just wanted to let you know that there will be a delayed response from him.

[georgemccabe]

Hi @kaday, I am sorry for the delayed response. It may be helpful to set up a meeting where we can discuss the details of how to approach this task. If you are interested, please send me an email at mccabe@ucar.edu so we can schedule a meeting. I think it would be good for at least @JohnHalleyGotway to be included in the discussion.

I think the first step would be to examine a few NetCDF files that were created by MET and document what needs to be added/changed to make it CF-compliant. We should also document which attributes can be removed as they are no longer needed. After we know what should change, we can write a script to perform these modifications. I envision this to be either a Python script that uses the NetCDF4 package or a shell script using NCO utilities like ncks. The logic of this script can be used in a new METplus wrapper.

[georgemccabe]

Notes from meeting on 11/3/2021

Format of CF Compliant NetCDF Files

What we need to include in the files we generate.

General

• Best practice to include "long_name", "units", and "standard_name" attributes
  • "units" attribute have format " since " i.e. "seconds since 1970-01-01" -- preferred to use seconds since that is unix time

Time Information

• Time variables needs to have "standard_name" attribute = "time"
  • "calendar" attribute i.e. "gregorian" ? May not be required?
  • "axis" attribute with value "T"
• forecast_reference_time variable is needed for forecast data

Grid Information

• We will start with supporting latitude/longitude grids https://cfconventions.org/Data/cf-conventions/cf-conventions-1.9/cf-conventions.html#_latitude_longitude
• Variable with units "attribute" = "degrees_north" and "degrees_east" are used for interpreting latitude and longitude variables respectively
• Attributes axis ("X" or "Y") should be set for best practice
• Variable of type "int" named "crs" instead of "grid_mapping" that has "grid_mapping_name" attribute that defines the grid mapping
  • Example:

int crs ;
    crs:grid_mapping_name = "latitude_longitude"
    crs:semi_major_axis = 6371000.0 ;
    crs:inverse_flattening = 0 ;

Other Information

• "Conventions" global attribute value with format "CF-<X.Y>" i.e. "CF-1.9" (current version)
• We should use NCO tools (ncattr/ncks/ncap2) instead of Python because the Python NetCDF is more limited and more difficult to change information such as variable names.

MET NetCDF files

What is found in files produced by the MET tools that we need to read (incomplete notes)

Grid Information

• Latitude and Longitude variables are created by the MET tools, but there is an option to turn this off in grid_stat
  • If lat/lon variables (designated by "units" values "degrees_north" and "degrees_east") are not found, error out

Time Information

• Data variables have (lat, lon) dimensions, but we should add time dimension (time, lat, lon)
• Data variables have timing attributes (init_time, init_time_ut, valid_time, valid_time_ut)
  • Remove from data variables
  • Create

Global Attributes

• "MET_version" defines the version of MET that was used to generate the file (keep in file to track version used)
• "Projection" defines the grid projection (error out if not "LatLon" for now -- not supported)

What we need change

• We will take a MET output file, dump the headers into a file, then mock up what needs to be added/changed

[JohnHalleyGotway]

Example of the NetCDF header before/after the reformatting:

pcp_combine -add ${MET_TEST_INPUT}/model_data/grib2/gfs/gfs_2012040900_F012.grib2 'name="TMP"; level="Z2";' GFS_TMP_Z2.nc
ncdump -h GFS_TMP_Z2.nc > GFS_TMP_Z2.MET.txt

Existing format: GFS_TMP_Z2.MET.txt
Recommended CF-compliant format: GFS_TMP_Z2.NCCF.txt
Straight diff: GFS_TMP_Z2.DIFF.txt
Graphical diff:

[jprestop]

@georgemccabe I just wanted to make sure you're aware of this documentation that @hsoh-u is creating in the MET User's Guide.
Link to issue: Enhance the documentation with meta-data that is expected by MET for netCDF #1949
Link to PR: dtcenter/MET#1951
Link to documentation on RTD (waiting on some suggested changes to be made still): https://met.readthedocs.io/en/feature_1949_cf_netcdf/Users_Guide/data_io.html
Please feel free to take a look. Howard or I could add you to the PR for review if you'd like.

[hankenstein2]

Would this be more appropriate to put in the dataIO area/repo?

[kaday]

For the "depth/height/Z" variable, if it exists, we agreed that we should add
"axis" attribute with value "Z"

[georgemccabe]

Note there is a MET issue dtcenter/MET#660 to change the MET output to be CF-compliant. If completed, it would remove the need for this work.