Bugfix: Fix ASCII2NC to handle missing NDBC buoy location information

[GwenChen-NOAA]

Describe the Problem

I tested ASCII2NC with MET 11.0.0/METplus 5.0.0 to convert NDBC buoy data (*.txt) into a METplus compatible point obs netCDF file and got the following error messages in the log file:

ERROR : No location information found for station 44084 do not process file /lfs/h1/ops/dev/dcom/20230124/validation_data/marine/buoy/44084.txt
ERROR : No location information found for station 46275 do not process file /lfs/h1/ops/dev/dcom/20230124/validation_data/marine/buoy/46275.txt
ERROR : No location information found for station CSXA2 do not process file /lfs/h1/ops/dev/dcom/20230124/validation_data/marine/buoy/CSXA2.txt
...

How were NDBC buoy location information obtained in MET/METplus? Please update the buoy location information to include all NDBC buoys.

Expected Behavior

All NDBC buoys have an assigned location.

Environment

Describe your runtime environment:
1. Machine: (e.g. HPC name, Linux Workstation, Mac Laptop)
WCOSS2 and Hera
2. OS: (e.g. RedHat Linux, MacOS)
3. Software version number(s)
MET 11.0.0/METplus 5.0.0

To Reproduce

NDBC buoy data directory on WCOSS2:
/lfs/h1/ops/dev/dcom/$YYYYMMDD/validation_data/marine/buoy

METplus use case:
https://metplus.readthedocs.io/en/latest/generated/model_applications/marine_and_cryosphere/PointStat_fcstGFS_obsNDBC_WaveHeight.html#sphx-glr-generated-model-applications-marine-and-cryosphere-pointstat-fcstgfs-obsndbc-waveheight-py

Relevant Deadlines

MET 11.0.1/METplus 5.0.1

Funding Source

2773542

Define the Metadata

Assignee

• Select engineer(s) or no engineer required
  @JohnHalleyGotway
• Select scientist(s) or no scientist required

Labels

• Select component(s)
• Select priority
• Select requestor(s)

Projects and Milestone

• Select Organization level Project for support of the current coordinated release
• Select Repository level Project for development toward the next official release or add alert: NEED PROJECT ASSIGNMENT label
• Select Milestone as the next bugfix version

Define Related Issue(s)

Consider the impact to the other METplus components.

• METplus, MET

Bugfix Checklist

See the METplus Workflow for details.

• Complete the issue definition above, including the Time Estimate and Funding Source.
• Fork this repository or create a branch of main_<Version>.
  Branch name: bugfix_<Issue Number>_main_<Version>_<Description>
• Fix the bug and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Add any new Python packages to the METplus Components Python Requirements table.
• Push local changes to GitHub.
• Submit a pull request to merge into main_<Version>.
  Pull request: bugfix <Issue Number> main_<Version> <Description>
• Define the pull request metadata, as permissions allow.
  Select: Reviewer(s) and Development issues
  Select: Organization level software support Project for the current coordinated release
  Select: Milestone as the next bugfix version
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Complete the steps above to fix the bug on the develop branch.
  Branch name: bugfix_<Issue Number>_develop_<Description>
  Pull request: bugfix <Issue Number> develop <Description>
  Select: Reviewer(s) and Development issues
  Select: Repository level development cycle Project for the next official release
  Select: Milestone as the next official version
• Close this issue.

[JohnHalleyGotway]

Thanks @jprestop for pulling buoy data for testing! I see it in seneca:/d1/projects/MET/MET_issues/feature_2426

I ran ascii2nc with this data and was able to replicate the reported behavior:

bin/ascii2nc buoy/* buoy.nc -log run_ascii2nc.log

Produced these log messages:

ERROR  : No location information found for station 44084 do not process file buoy/44084.txt
ERROR  : No location information found for station 46275 do not process file buoy/46275.txt
ERROR  : No location information found for station CSXA2 do not process file buoy/CSXA2.txt
ERROR  : No location information found for station DILA1 do not process file buoy/DILA1.txt
ERROR  : No location information found for station KBMG1 do not process file buoy/KBMG1.txt
ERROR  : No location information found for station KCXA2 do not process file buoy/KCXA2.txt
ERROR  : No location information found for station KOZA2 do not process file buoy/KOZA2.txt
ERROR  : No location information found for station OWMO1 do not process file buoy/OWMO1.txt
ERROR  : No location information found for station PAUA2 do not process file buoy/PAUA2.txt
ERROR  : No location information found for station SGXA2 do not process file buoy/SGXA2.txt
ERROR  : No location information found for station WCRP1 do not process file buoy/WCRP1.txt
ERROR  : No location information found for station WGXA2 do not process file buoy/WGXA2.txt

Tasks:

1. Change these log messages from Error to Warning since they do NOT halt the execution of the program. Also tweak the wording for clarity.
2. Update contents of data/table_files/ndbc_stations.xml for the missing buoy locations.
   • Find locations at https://www.ndbc.noaa.gov/station_page.php?station=44084 for example.
3. Also test additional days looking for more missing buoys.

Note that this issue could be addressed by setting the MET_NDBC_STATIONS environment variable. However, we may as well update the table in MET to include all buoys used for operational verification.

[JohnHalleyGotway]

Note that https://www.ndbc.noaa.gov/to_station.shtml contains links for 1839 stations:

wget https://www.ndbc.noaa.gov/to_station.shtml
cat to_station.shtml | sed 's/</\n/g' | grep "station=" | cut -d'>' -f2 | sort -u > ndbc_stations.txt 
cat ndbc_stations.txt | wc -l

Currently, only 1364 are included in MET (as of v11.0.0):

cat data/table_files/ndbc_stations.xml | grep "station id" | wc -l

@GwenChen-NOAA and @davidalbo, should we update that file to include all 1839 locations?

If so, perhaps we only really care about lat/lon/elev/name and not all the other metadata?

[davidalbo]

@JohnHalleyGotway @GwenChen-NOAA - Originally I used this https://www.ndbc.noaa.gov/activestations.xml
to create the ndbc_stations.xml. We are using it directly as is if I'm remembering. Also I don't remember who pointed me to it. I can't remember much apparently. :-)

If this other source is better we should use it instead. I believe it's true we only are pulling lat/lon/elev/name

We'd need to write up something to parse the newer source and create a new stations file.

[GwenChen-NOAA]

Deanna is checking with her colleague at NDBC to see how often they update the webpage. Thanks, Gwen

[JohnHalleyGotway]

@davidalbo thanks for that link. That's very helpful!

@GwenChen-NOAA I've manually added 12 missing stations to the ndbc_stations.xml file in my bugfix branch. I got the name/lat/lon/elev values correct but made some educated guesses as to the other entries (which MET doesn't use anyway). And I confirmed that the updated list makes all those missing station log messages go away for the 2 days with which I tested.

So this increases the number of entries in the file from 1364 (as of v11.0.0) to 1376.

The question is whether or not those changes are sufficient or should we be using all 1839 stations described here? And with 2 different sources, there's potential that the details could differ.

The easiest thing would be sticking with the 12 additions I've already made, but I'd like your input.

Please let me know how you and Deanna would like to proceed.

[JohnHalleyGotway]

I'd like to wrap up this bugfix issue. Unless I hear otherwise, I'll proceed with the manual changes I made by adding the locations for the 12 missing buoys.

If @GwenChen-NOAA and Deanna do find a preferable or more complete way of defining NDBC Buoy locations, I recommend that they write up a GitHub issue to enhance the definition for a future release, such as MET-11.1.0. Do note, once you have the updated definition in hand you can start using it right away by setting the $MET_NDBC_STATIONS environment variable.

[GwenChen-NOAA]

Thank you, John! Deanna contacted NDBC, but hasn't heard back from them. I think adding the locations for the 12 missing buoys is sufficient for the bugfix. If possible, please reduce the log message for missing buoy location information from ERROR to WARNING, so NCO won't be alerted if it happens in the production run. ASCII2NC can still run with missing information and create a nc file, so it's not fatal. Thanks, Gwen

[GwenChen-NOAA]

Hello all, I just heard back from NDBC: "The list at https://www.ndbc.noaa.gov/to_station.shtml includes all valid station IDs, both current and historical. The XML list ( https://www.ndbc.noaa.gov/activestations.xml) I believe is updated in real-time, and includes just the active stations, and not historical stations that have been disestablished. Incidentally, almost all of the "missing" stations that are mentioned are present in the XML list. Except for DILA1 (which we never released until recently due to it being co-located with DPIA1) , they're all relatively new stations, so it's possible that the requester may be working off an older version of the file." Best, Deanna

…

-- Deanna Spindler, PhD Physical Scientist III Lynker at NOAA/NWS/NCEP/Environmental Modeling Center NOAA Center for Weather and Climate Prediction

On Thu, Feb 2, 2023 at 6:44 PM L. Gwen Chen ***@***.***> wrote: Thank you, John! Deanna contacted NDBC, but hasn't heard back from them. I think adding the locations for the 12 missing buoys is sufficient for the bugfix. If possible, please reduce the log message for missing buoy location information from ERROR to WARNING, so NCO won't be alerted if it happens in the production run. ASCII2NC can still run with missing information and create a nc file, so it's not fatal. Thanks, Gwen On Thu, Feb 2, 2023 at 1:27 PM John Halley Gotway < ***@***.***> wrote: > I'd like to wrap up this bugfix issue. Unless I hear otherwise, I'll > proceed with the manual changes I made by adding the locations for the 12 > missing buoys. > > If @GwenChen-NOAA <https://github.com/GwenChen-NOAA> and Deanna do find > a preferable or more complete way of defining NDBC Buoy locations, I > recommend that they write up a GitHub issue to enhance the definition for a > future release, such as MET-11.1.0. Do note, once you have the updated > definition in hand you can start using it right away by setting the > $MET_NDBC_STATIONS > <https://met.readthedocs.io/en/latest/Users_Guide/config_options.html#met-ndbc-stations> > environment variable. > > — > Reply to this email directly, view it on GitHub > <#2426 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AWXF3DAZ52J6PALTQBXLM6LWVP37XANCNFSM6AAAAAAUJA6SOI> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> >

[JohnHalleyGotway]

After merging this into develop, testing in METplus flagged some buoy location differences. Careful comparison reveals 15 stations whose locations have changed slightly. Several others either have or no longer have a trailing 0 in the decimal place, but I ignored those diffs.

Left is old (main_v11.0) and right is new (develop):

Station ID, Latitude, Longitude

< 41040 15.4235 -56.8072 > 41040 14.54 -53.329
< 41108 33.722 -78.016 > 41108 33.721 -78.016
< 41110 34.142 -77.715 > 41110 34.143 -77.716
< 41114 27.551 -80.217 > 41114 27.552 -80.216
< 42001 25.942 -89.657 > 42001 25.919 -89.674
< 42409 25.901 -89.291 > 42409 22.5065 -94.4146
< 44088 36.611 -74.841 > 44088 36.614 -74.841
< 44097 40.967 -71.126 > 44097 40.967 -71.124
< 46011 34.937 -121 > 46011 34.936 -120.998
< 46041 47.353 -124.742 > 46041 47.352 -124.739
< 46098 44.381 -124.956 > 46098 44.378 -124.947
< 46099 46.986 -124.566 > 46099 46.988 -124.567
< 46100 46.851 -124.972 > 46100 46.851 -124.964
< 46416 49.901 -134.395 > 46416 48.6827 -129.902
< 51210 21.477 -157.756 > 51210 21.477 -157.757