Enhance ASCII2NC to read NDBC buoy data

[JohnHalleyGotway]

Describe the New Feature

The desire to use buoy data in ASCII format for verification within MET has come up twice recently as seen in this GitHub discussion. This functionality is needed by both NOAA/EMC and NOAA/OPC. While the interim solution is using python embedding of point observations, the task for this issue is to support it directly via the ASCII2NC tool in MET. Please refer to this dtcenter/METplus#1482 issue for a related METplus use case.

Recommend supporting NDBC Buoy data. This issue includes:

• Contacting @DeannaSpindler-NOAA to retrieve sample data.
• Adding support for a new -format option to read this data.
• Add new unit test(s) to demonstrate this functionality.
• Update the User's Guide accordingly.

Acceptance Testing

List input data types and sources.
Describe tests required for new functionality.

Time Estimate

Estimate the amount of work required here.
Issues should represent approximately 1 to 3 days of work.

Sub-Issues

Consider breaking the new feature down into sub-issues.
None needed.

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

2773542

Define the Metadata

Assignee

• Select engineer(s) or no engineer required: @davidalbo
• Select scientist(s) or no scientist required: @j-opatz

Labels

• Select component(s)
• Select priority
• Select requestor(s)

Projects and Milestone

• Select Repository and/or Organization level Project(s) or add alert: NEED PROJECT ASSIGNMENT label
• Select Milestone as the next official version or Future Versions

Define Related Issue(s)

Consider the impact to the other METplus components.

• METplus, MET, METdataio, METviewer, METexpress, METcalcpy, METplotpy
  Likely no impacts, but recommend adding/updating METplus use cases to leverage this new functionality.

New Feature Checklist

See the METplus Workflow for details.

• Complete the issue definition above, including the Time Estimate and Funding source.
• Fork this repository or create a branch of develop.
  Branch name: feature_<Issue Number>_<Description>
• Complete the development and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Push local changes to GitHub.
• Submit a pull request to merge into develop.
  Pull request: feature <Issue Number> <Description>
• Define the pull request metadata, as permissions allow.
  Select: Reviewer(s) and Linked issues
  Select: Repository level development cycle Project for the next official release
  Select: Milestone as the next official version
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Close this issue.

[davidalbo]

I'm starting to work on this issue, to put buoy data into ascii2nc so you can input multiple files directly. What would be helpful is sample data files, and a description of the format. @DeannaSpindler-NOAA can you get me going?

[DeannaSpindler-NOAA]

Hello David, You can find the NDBC ASCII buoy data online. For example: https://www.ndbc.noaa.gov/data/realtime2/41001.txt has the last 45 days of data for buoy 41001. The files have a header, and you can find the description of the columns here: https://www.ndbc.noaa.gov/measdes.shtml If you want a test list of buoys, you can use any of these: - 41001 - EAST HATTERAS - 150 NM East of Cape Hatteras <https://www.ndbc.noaa.gov/data_availability/data_avail.php?station=41001> - 41002 - SOUTH HATTERAS - 225 NM South of Cape Hatteras <https://www.ndbc.noaa.gov/data_availability/data_avail.php?station=41002> - 41003 <https://www.ndbc.noaa.gov/data_availability/data_avail.php?station=41003> - 41004 - EDISTO - 41 NM Southeast of Charleston, SC <https://www.ndbc.noaa.gov/data_availability/data_avail.php?station=41004> - 41005 <https://www.ndbc.noaa.gov/data_availability/data_avail.php?station=41005> - 41006 <https://www.ndbc.noaa.gov/data_availability/data_avail.php?station=41006> - 41007 <https://www.ndbc.noaa.gov/data_availability/data_avail.php?station=41007> - 41008 - GRAYS REEF - 40 NM Southeast of Savannah, GA <https://www.ndbc.noaa.gov/data_availability/data_avail.php?station=41008> - 41009 - CANAVERAL 20 NM East of Cape Canaveral, FL <https://www.ndbc.noaa.gov/data_availability/data_avail.php?station=41009> - 41010 - CANAVERAL EAST - 120NM East of Cape Canaveral <https://www.ndbc.noaa.gov/data_availability/data_avail.php?station=41010> - 41011 <https://www.ndbc.noaa.gov/data_availability/data_avail.php?station=41011> - 41012 - St. Augustine, FL 40NM ENE of St Augustine, FL <https://www.ndbc.noaa.gov/data_availability/data_avail.php?station=41012> - 41015 <https://www.ndbc.noaa.gov/data_availability/data_avail.php?station=41015> - 41016 <https://www.ndbc.noaa.gov/data_availability/data_avail.php?station=41016> - 41017 <https://www.ndbc.noaa.gov/data_availability/data_avail.php?station=41017> - 41018 <https://www.ndbc.noaa.gov/data_availability/data_avail.php?station=41018> - 41021 - Olympic Northeast <https://www.ndbc.noaa.gov/data_availability/data_avail.php?station=41021> - 41022 - Olympic Southwest <https://www.ndbc.noaa.gov/data_availability/data_avail.php?station=41022> - 41023 - Olympic Southwest <https://www.ndbc.noaa.gov/data_availability/data_avail.php?station=41023> - 41025 - Diamond Shoals, NC <https://www.ndbc.noaa.gov/data_availability/data_avail.php?station=41025> - 44001 <https://www.ndbc.noaa.gov/data_availability/data_avail.php?station=44001> - 44003 <https://www.ndbc.noaa.gov/data_availability/data_avail.php?station=44003> - 44004 - HOTEL 200NM East of Cape May,NJ <https://www.ndbc.noaa.gov/data_availability/data_avail.php?station=44004> - 44005 - GULF OF MAINE - 78 NM East of Portsmouth, NH <https://www.ndbc.noaa.gov/data_availability/data_avail.php?station=44005> - 44006 <https://www.ndbc.noaa.gov/data_availability/data_avail.php?station=44006> - 44007 - PORTLAND - 12 NM Southeast of Portland,ME <https://www.ndbc.noaa.gov/data_availability/data_avail.php?station=44007> - 44008 - NANTUCKET 54 NM Southeast of Nantucket <https://www.ndbc.noaa.gov/data_availability/data_avail.php?station=44008> - 44009 - DELAWARE BAY 26 NM Southeast of Cape May, NJ <https://www.ndbc.noaa.gov/data_availability/data_avail.php?station=44009> - 44010 <https://www.ndbc.noaa.gov/data_availability/data_avail.php?station=44010> - 44011 - GEORGES BANK 170 NM East of Hyannis, MA <https://www.ndbc.noaa.gov/data_availability/data_avail.php?station=44011> - 44012 <https://www.ndbc.noaa.gov/data_availability/data_avail.php?station=44012> - 44013 - BOSTON 16 NM East of Boston, MA <https://www.ndbc.noaa.gov/data_availability/data_avail.php?station=44013> - 44014 - VIRGINIA BEACH 64 NM East of Virginia Beach, VA <https://www.ndbc.noaa.gov/data_availability/data_avail.php?station=44014> - 44015 <https://www.ndbc.noaa.gov/data_availability/data_avail.php?station=44015> - 44017 - MONTAUK POINT - 23 NM SSW of Montauk Point, NY <https://www.ndbc.noaa.gov/data_availability/data_avail.php?station=44017> - 44018 - CAPE COD - 9 NM North of Provincetown, MA <https://www.ndbc.noaa.gov/data_availability/data_avail.php?station=44018> - 44019 <https://www.ndbc.noaa.gov/data_availability/data_avail.php?station=44019> - 44023 <https://www.ndbc.noaa.gov/data_availability/data_avail.php?station=44023> - 44025 - LONG ISLAND - 30 NM South of Islip, NY <https://www.ndbc.noaa.gov/data_availability/data_avail.php?station=44025> - 44026 <https://www.ndbc.noaa.gov/data_availability/data_avail.php?station=44026> - Just note that some of these may not have any data in the last 45 days. Let me know if you need anything else. Best, Deanna

…

-- Deanna Spindler, PhD Physical Scientist III Lynker at NOAA/NWS/NCEP/Environmental Modeling Center NOAA Center for Weather and Climate Prediction

On Wed, Sep 28, 2022 at 11:58 AM davidalbo ***@***.***> wrote: I'm starting to work on this issue, to put buoy data into ascii2nc so you can input multiple files directly. What would be helpful is sample data files, and a description of the format. @DeannaSpindler-NOAA <https://github.com/DeannaSpindler-NOAA> can you get me going? — Reply to this email directly, view it on GitHub <#2276 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AK6TDPVUPF7O6XI6HPJMXY3WARTLLANCNFSM6AAAAAAQUEKPHA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[davidalbo]

@DeannaSpindler-NOAA thanks! A few questions.

1. Do you need support for all the formats here https://www.ndbc.noaa.gov/measdes.shtml , or one particular one? A subset?
2. Many of the file formats do not have latitude/longitude I assume because they don't move. Is there a mapping from buoy number to latitude/longitude?

[DeannaSpindler-NOAA]

I use the Standard Meteorological Data format, which is what is in the 45-day ASCII files. Here are a couple of docs with the buoy lats/lons listed: https://sdf.ndbc.noaa.gov/stations.shtml or https://www.ndbc.noaa.gov/wstat.shtml Best, Deanna

…

-- Deanna Spindler, PhD Physical Scientist III Lynker at NOAA/NWS/NCEP/Environmental Modeling Center NOAA Center for Weather and Climate Prediction

On Wed, Sep 28, 2022 at 3:33 PM davidalbo ***@***.***> wrote: @DeannaSpindler-NOAA <https://github.com/DeannaSpindler-NOAA> thanks! A few questions. 1. Do you need support for all the formats here https://www.ndbc.noaa.gov/measdes.shtml , or one particular one? A subset? 2. Many of the file formats do not have latitude/longitude I assume because they don't move. Is there a mapping from buoy number to latitude/longitude? — Reply to this email directly, view it on GitHub <#2276 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AK6TDPWC4XM7ZUUHQLTNPZ3WASMRZANCNFSM6AAAAAAQUEKPHA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[davidalbo]

@DeannaSpindler-NOAA the input file names will be things like 410001.txt, correct?

If so, @JohnHalleyGotway looks like I'll need to parse these input file names to lookup a lat/lon. The buoy numbers are not inside the file anywhere. What do you think, does it break the design?

[DeannaSpindler-NOAA]

I usually use the buoy ID as the file name, so '41001' or '41001.txt' is correct.

…

-- Deanna Spindler, PhD Physical Scientist III Lynker at NOAA/NWS/NCEP/Environmental Modeling Center NOAA Center for Weather and Climate Prediction

On Wed, Sep 28, 2022 at 4:25 PM davidalbo ***@***.***> wrote: @DeannaSpindler-NOAA <https://github.com/DeannaSpindler-NOAA> the input file names will be things like 410001.txt, correct? If so, @JohnHalleyGotway <https://github.com/JohnHalleyGotway> looks like I'll need to parse these input file names to get to lookup a lat/lon. The buoy numbers are not inside the file anywhere. What do you think, does it break the design? — Reply to this email directly, view it on GitHub <#2276 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AK6TDPRY3K7BN3OQTOPYPO3WASSSFANCNFSM6AAAAAAQUEKPHA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[davidalbo]

I found this and downloaded it: https://www.ndbc.noaa.gov/activestations.xml
With a little minor XML parsing, should be able to create an ID to lat/lon lookup.
@JohnHalleyGotway might want to put XML parsing into library code somewhere, unless something already exists. I do have familiarity with an existing library from another project outside of MET that reads/writes XML. We should discuss options.

[davidalbo]

@JohnHalleyGotway I can't tell if its working or not. The difference between this and the other ascii input data I worked on (airnow) is that in this case each file has many times, for airnow each file had only one time. When I try to do these things to see if it's working:

plot_point_obs
Rscript ./pntnc2ascii.R

it seems like it's not working, yet the code seems identical as far as adding observations no matter how much I look at it.

Is it ok to put many times into a single netCDF output file?

[davidalbo]

From @hsoh-u, saving for reference: The GRIB code can be saved by creating "obs_gc" variable instead of the "obs_vid". In this case, the global attribute "use_var_id" must be "false". I think MET prefers to use variable_ID over GRIB code. This means what is passed into the Observation constructor as 'grib_code' is actually index into the variables (0,1,2..)

[davidalbo]

@jprestop, I have another file that is external and must be read in, just like previously for airnow data, to look up latitude/longitude. It is a table_file. Previously I handed a file to you and it showed up on a web page, if I'm remembering. You did 'something', not sure what. The file I need this time: /scratch/dave/ndbc_stations.xml

I did add the file on my development branch here:
data/table_files/ndbc_stations.xml
and modified the Makefiles accordingly.

[jprestop]

Thanks @davidalbo. The last time I added data for you, I followed the instructions in this comment from @JohnHalleyGotway. However, looking at the content on mohawk at this location, /d2/www/dtcenter/dfiles/code/METplus/MET/MET_unit_test, it does not appear that table_files are considered unit_test data, so I do not believe that any further action is needed other than what you have already done.

[davidalbo]

I had it backwards, I needed @jprestop to help to put unit test data into place, not for the table file. That is now done.