hdf5 plot_data_plane

[fearon10]

Hi,
I am attempting to run MET/plot_data_plane on variable(s) from a model-generated hdf5 output file, surface skin temperature for example. When running the plot_data_plane command I receive the following error:

---

DEBUG 4: get_data_from_lat_lon_vars() -> lat[1439]=-90 lat[1440]=-89.75 0.25 > 0
ERROR :
ERROR : get_data_from_lat_lon_vars() -> MET can only process Latitude/Longitude files where the latitudes are evenly spaced (dlat=0, delta[1440]=0.25)
ERROR :
ERROR :
ERROR : get_data_from_lat_lon_vars() -> Please check the input data is the lat/lon projection
ERROR :

The projection is lat/lon and data are 0.25 degree evenly spaced. I believe the issue might be that all variable data arrays, including the latitude/longitude values, are packed in the hdf5 file as 1-D flattened arrays. So, instead of lat(1440,721), I have lat(1038240,1). And the data are arranged south to north which would give values for lat[1439]=-90 and lat[1440]=-89.75.

I can successfully run plot_data_plane with python embedding and reshaping the data arrays from 1 to 2D. However, I would like to avoid python embedding or an offline file conversion if possible. Colleagues and I have found issues with python embedding and computing grid-stat vector statistics in MET. Hence, I would like to use a direct read approach for ingesting the data. Is there a method in MET to deal with arrays packed as 1-D as in this example?

I have successfully converted the hdf5 to netcdf4 using CDO and/or nccopy command line tools, but arrays remain 1-D and the issue with MET interpretation remains.

Thank you in advance for the assistance.
Matt

[JohnHalleyGotway]

Matt, I see you're having trouble reading a gridded dataset into MET. It's a great idea to start with the plot_data_plane utility. That's how I always recommend that folks get started reading new datasets into MET.

And I see you've already written a python embedding script to handle this data... which is great. You could actually leverage that work you've already done using MET's pcp_combine tool. You could run pcp_combine as a pass through to read the data from python and write it out following MET's NetCDF file format.

For example, here's a simple python embedding example include in the MET release:

python scripts/python/read_ascii_numpy.py data/python/fcst.txt FCST

And here's how you could run it through pcp_combine to dump it to NetCDF:

pcp_combine -add PYTHON_NUMPY fcst.nc -field 'name="scripts/python/read_ascii_numpy.py data/python/fcst.txt FCST";'

The output fcst.nc contains the data that was served up via python embedding. And that can easily be read by downstream MET tools.

As for how to modify your combination of CDO and/or nccopy commands to accomplish a similar result, I certainly can't say off the top of my head. I'm not a CDO expert and don't have an easy answer. Whether you call CDO, python, or python with pcp_combine, all are a pre-processing steps... so some logic will be needed. You could use python via the python embedding route... or you could consider having the python script write a MET NetCDF file directly.

Hope that helps.

[JohnHalleyGotway]

Just wanted to follow up on this discussion to see if questions remain. If not, please feel free to select the "Mark As Answer" button for one of the previous responses, and then I'll lock it to prevent future posts. If I don't hear otherwise in the next week or so, I'll just go ahead and mark it as being answered.

Thanks!

[fearon10]

Thanks for following up @JohnHalleyGotway. I don't have any additional questions specific to the plot_data_plane tool or methods for editing hdf5 or netcdf files to make them format compliant, e.g., such that files may be directly read in and no python embedding is needed. That said, I do have additional questions on python embedding, in particular how to use python embedding at the same time as computing vector statistics with the grid-stat tool, a task I have not been able to accomplish. At this time, computing vector statistics with grid-stat seems to be contingent on using a direct read of grib or proper netcdf? In the end, I just decided to convert my hdf5 to netcdf4 compliant outside of MET, check that result with plot_data_plane (using a direct read), and then compute vector statistics with grid-stat (again using a direct read).
Lot of words there, sorry. I do very much appreciate the help.
Matt

[JohnHalleyGotway]

@fearon10 I see that you're using python embedding with Grid-Stat and would like to dump out vector statistics.

By convention, when reading GRIB data, MET assumes that variables named "UGRD" and "VGRD" contain the U and V component of winds, respectively. That's convenient when using GRIB data that follows those conventions, but not useful otherwise. Instead, we need to configure MET to explicitly tell it to interpret data as being U and V wind by setting the is_u_wind = TRUE; and is_v_wind = TRUE; options in the configuration file.

However, when I tested this using a very contrived example, it did not work as I'd hoped. I boiled it down to this new issue described by dtcenter/MET#2366. I'll fix that for the upcoming MET-11.0.0 release, but that tweak in logic is needed to make this work.