Define new grid class to store semi-structured grid information (e.g. lat or lon vs level or time)

[JohnHalleyGotway]

Describe the New Feature

This work is related to dtcenter/METplus#??? (insert issue number here) which describes enhancing METplus for use in computing and quantifying stratospheric diagnostics. While some portions of that work will be done in Python, some enhancements to the MET code is needed as well. This top-level issue describes those changes. However, it should be further broken down into actionable pieces prior to beginning work.

1. The stratospheric diagnostics are based on either zonal or meridional mean fields. Consider enhancing MET to derive these means on-the-fly in any tool that can process multiple input fields (including Grid-Stat and Series-Analysis). Adding it to MODE would also be useful. The meridional mean is computed by averaging data values across all latitudes for each longitude. The zonal mean is computed by averaging data values across all longitudes for each latitude. Compute the 1D mean arrays for each input level and store the meridional or zonal mean as a 2D field. One dimension is the lat or lon and the other dimension is the vertical level.

Question: Can this only be computed for LatLon inputs or should it be computed for all input projection types?

2. Add support for a semi-structured grid type. The spatial information of the mean fields described above can be stored in a new semi-structured grid class. Recommend making this grid class flexible enough to store 5 variations of data types:
   • Zonal mean field with dimensions (lat, vertical level)
   • Meridional mean field with dimensions (lon, vertical level)
   • Vertical cross-section field with dimensions (points, vertical level) where points are located by (lat, lon) location.
   • Hovmoller field with dimensions (lat, time)
   • Hovmoller field with dimensions (lon, time)

In general, the semi-structured grid class should store (lat, lon) info for the first dimension and vertical level or time for second. Note that the xy_to_latlon() and latlon_to_xy() functions from the Grid base class warrant careful consideration.

3. Add support a totally unstructured grid type. This is really just for storing a set of lat/lon points and corresponding data values. Those lat/lon values can probably just be stored in a 1D, rather than 2D, array. This may be useful for handling unstructured grid data from MPAS and Elfrich. Note that operations like smoothing and interpolating (other than nearest neighbor) should result in an error.

Note: Lots of additional definition is required.

Acceptance Testing

List input data types and sources.
Describe tests required for new functionality.

Time Estimate

Estimate the amount of work required here.
Issues should represent approximately 1 to 3 days of work.

Sub-Issues

Consider breaking the new feature down into sub-issues.

• Add a checkbox for each sub-issue here.

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

Define the source of funding and account keys here or state NONE.

Define the Metadata

Assignee

• Select engineer(s) or no engineer required
• Select scientist(s) or no scientist required

Labels

• Select component(s)
• Select priority
• Select requestor(s)

Projects and Milestone

• Select Repository and/or Organization level Project(s) or add alert: NEED PROJECT ASSIGNMENT label
• Select Milestone as the next official version or Future Versions

Define Related Issue(s)

Consider the impact to the other METplus components.

• METplus, MET, METdatadb, METviewer, METexpress, METcalcpy, METplotpy

New Feature Checklist

See the METplus Workflow for details.

• Complete the issue definition above, including the Time Estimate and Funding source.
• Fork this repository or create a branch of develop.
  Branch name: feature_<Issue Number>_<Description>
• Complete the development and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Push local changes to GitHub.
• Submit a pull request to merge into develop.
  Pull request: feature <Issue Number> <Description>
• Define the pull request metadata, as permissions allow.
  Select: Reviewer(s) and Linked issues
  Select: Repository level development cycle Project for the next official release
  Select: Milestone as the next official version
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Close this issue.

[JohnHalleyGotway]

@CPKalb I was inclined to start working on this issue since it's marked as required for the next dev release. But Tara recommended that I touch base with you to confirm that it's still needed.

Should I work on adding support for handling these types of semi-structured grids in MET (e.g. hovmoller, meridional means, etc)? I imagine, for example, you'd run Grid-Stat using python embedding to serve up the forecast and observation data to the tool. Then Grid-Stat would compute and report statistics.

Or have you already come up with a preferred workaround in python? And I should not work on this issue?

[CPKalb]

@JohnHalleyGotway, I have not come up with a workaround in python yet. I'm stuck on a pre-processing step. Given that the original data is 3D and that either the model or obs need to be regridded to match, I haven't decided on the best way to do this (I think regrid_data_plane needs 2d). Anyway, I don't have a workaround yet. I've been in conversation with Minna and Hank, but the RMSE calculation in METcalcpy looks like it needs a SL1L2 line type input (which we wouldn't be starting from). But, we haven't been able to test because I'm stuck in pre-processing

[JohnHalleyGotway]

@CPKalb I've started working on this today. I see that you'd like to run it via Series-Analysis, so I'll work on that aspect of it. FYI, based on a meeting yesterday, I created a separate issue (#2231) for handling actual UGRID data.

To avoid confusion, I'm switching from calling this an "Unstructured" grid to calling it a "SemiLatLon" grid. That is meant to imply there is some latitude or longitude information involved, but not entirely.

One clarifying question for you. I understand that we want to support (1) latitude OR longitude VERSUS level OR time. Is it also the case that we want to support (2) latitude AND longitude VERSUS level OR time?

For (2), when both latitude and longitude are provide, they would necessarily need to have the same length. Is that correct? Should (2) be supported? That's how I'm interpreting this comment from above:

Vertical cross-section field with dimensions (points, vertical level) where points are located by (lat, lon) location.

But I just wanted to confirm that my interpretation is correct.

[CPKalb]

@JohnHalleyGotway for the stratosphere work, I think we want to have the ability to support latitude OR longitude versus level AND time. The zonal and meridional means we are doing for that work will be (latitude, level, time) and (longitude, level, time).

I guess it depends on how these go into series analysis. I'm assuming I'll need to use python embedding to put them in (since the zonal and meridional means are calculated in python). In that case, if each time step is imported separately, then I would have something like (latitude, level) and (longitude, level) for each time step. I'll have to figure out how to separate the time steps since the zonal and meridional mean code performs these on all time steps at the same time.

(2) isn't needed for my stratosphere work, but I don't know if it's needed for something else.

Tina

[JohnHalleyGotway]

@CPKalb I've made good progress on this and even got grid_stat to compute stats on some sample "semilatlon" data!

Still working on updating the MET tools to read the semilatlon output written by other tools. Once that's done, I'd like you test it out. Should I just compile my feature branch somewhere on seneca for you to test out?

[CPKalb]

Yes, that sounds good! Will I be able to use python embedding to read in the data? That will likely be the easiest way since it's already in python to compute the zonal mean.

[JohnHalleyGotway]

Tina, I'm finally ready for you to start testing. The wrinkle is that for some reason python serves up my lats/lons/levels/times as a Python List (on my laptop) and other times as a NumPy array (on seneca). I had to make the code flexible enough to handle both.

Please take a look on seneca and run:

cd /d1/personal/johnhg/MET/MET_development/SemiLatLon/examples
./run_examples.sh

These exercise the sample read_PostProcessed_WRF_semilatlon.py python embedding script. I hoping you can modify your other scripts to define the input data in a similar way.

Running this through Series-Analysis should technically work, but no example is provided here.

And I imagine we'll want to modify how timestamps are served up. Currently, they're just stored as a list of unixtimes.

[CPKalb]

Great. I probably won't get to start testing until tomorrow.

[CPKalb]

Hi John: Just to verify. In the script, it looks like it computes percentiles over a specified dimension. But, does it use an empty list for that dimension? It looks like if it computes percentiles over lon, then it sets lats to an empty list and retains the lon values. Is that interpretation correct? Tina

…

On Tue, Sep 6, 2022 at 3:01 PM johnhg ***@***.***> wrote: Tina, I'm finally ready for you to start testing. The wrinkle is that for some reason python serves up my lats/lons/levels/times as a Python List (on my laptop) and other times as a NumPy array (on seneca). I had to make the code flexible enough to handle both. Please take a look on seneca and run: cd /d1/personal/johnhg/MET/MET_development/SemiLatLon/examples ./run_examples.sh These exercise the sample read_PostProcessed_WRF_semilatlon.py python embedding script. I hoping you can modify your other scripts to define the input data in a similar way. Running this through Series-Analysis *should* technically work. And I imagine we'll want to modify how timestamps are served up. Currently, they're just stored as a list of unixtimes. — Reply to this email directly, view it on GitHub <#1954 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFJHD4SPLR6JUD5U2E5VG73V46WKPANCNFSM5GYSTTIA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- ************************************************* Christina Kalb Associate Scientist National Center for Atmospheric Research 3450 Mitchell Lane Boulder, CO 80301 ***@***.*** 303-497-2819 ************************************************* My working day may not be your working day. Please do not feel obliged to reply to this email outside of your normal working hours.