Enhance ASCII2NC to support a wider variety of NDBC buoy data types

[JohnHalleyGotway]

Describe the Enhancement

This issue is based on the dtcenter/METplus#2209 discussion. MET issue #2276 enhanced ASCII2NC to add support for -format ndbc_standard to read NDBC buoy data in the 19-column "Standard Meteorological Data" format.

As described on this NDBC website, several other buoy data formats exist that are very similar to the standard one. However the name, number, and units of the data columns vary. ASCII2NC errors out when parsing any NDBC dataset using -format ndbc_standard when the number of columns differs from 19:

WARNING: 
WARNING: NdbcHandler::_readHeaderInfo() -> Unexpected number of header columns (15 != 19): 41008.spec
WARNING: 
ERROR  : 
ERROR  : ascii2nc-> encountered an error while reading input files!
ERROR  :

This issue is to generalize the logic of the -format ndbc_standard. Rather than hard-coding the number and contents of the input data, parse the header lines to retrieve that metadata (observation names and, optionally, units).

Doing so would enable support for:

1. Standard Meteorological Data

#YY  MM DD hh mm WDIR WSPD GST  WVHT   DPD   APD MWD   PRES  ATMP  WTMP  DEWP  VIS PTDY  TIDE
#yr  mo dy hr mn degT m/s  m/s     m   sec   sec degT   hPa  degC  degC  degC  nmi  hPa    ft

2. Standard Meteorological Data for Drifting Buoys

#YY  MM DD hhmm     LAT      LON WDIR WSPD GST   PRES PTDY ATMP WTMP  DEWP  WVHT  DPD
#yr  mo dy hrmn     deg      deg degT m/s  m/s    hPa  hPa degC degC  degC     m  sec

3. Derived Met Values

#YY  MM DD hh mm CHILL  HEAT   ICE WSPD10 WSPD20
#yr  mo dy hr mn  degC  degC cm/hr    m/s    m/s

4. Supplemental Measurements Data

#YY  MM DD hh mm   PRES PTIME  WSPD  WDIR WTIME
#yr  mo dy hr mn    hPa  hhmm   m/s  degT  hhmm

5. Continuous Winds

#YY  MM DD hh mm WDIR WSPD GDR GST GTIME
#yr  mo dy hr mn degT m/s degT m/s hhmm

6. Detailed Wave Summary

#YY  MM DD hh mm WVHT  SwH  SwP  WWH  WWP SwD WWD  STEEPNESS  APD MWD
#yr  mo dy hr mn    m    m  sec    m  sec  -  degT     -      sec degT

7. Ocean Current Data

#YY  MM DD hh mm DEP01 DIR01 SPD01 DEP02 DIR02 SPD02 DEP03 DIR03 SPD03 ...>
#yr  mo dy hr mn     m  degT  cm/s     m  degT  cm/s     m  degT  cm/s ...>

8. Ocean Current Data (Expanded ADCP format)

#YY  MM DD hh mm I Bin   Depth Dir Speed ErrVl VerVl %Good3 %Good4 %GoodE   EI1   EI2   EI3   EI4   CM1   CM2   CM3   CM4 Flags
#yr  mo dy hr mn -   -     m  degT  cm/s  cm/s  cm/s      %      %      %     -     -     -     -     -     -     -     - -

9. Marsh-McBirney Current Measurements

YY MM DD hh mm    DIR    SPD

10. Water Level, Oceanographic Data, Solar Radiation Data, DART® (Tsunameters) Measurements, 24-Hour Rain Measurements, Hourly Rain Measurements, 10-Minute Rain Measurements, Housekeeping Measurements

Note, need to evaluate whether or not spectral wave data can be parsed in this way. It does include the use of parenthesis, so those would need to be properly ignored.

Note that the missing data value varies from source to source. It may be MM, N/A, 99, or some other string or value. Consider whether hard-coded those is sufficient or does it need to be made configurable?

Note that some columns or non-numeric. For example, in https://www.ndbc.noaa.gov/data/realtime2/41008.spec, SwD WWD STEEPNESS are all strings. Ideally, make the code smart enough to ignore these non-numeric observations.

Parsing details:

1. All of these formats must have columns for YY, MM, and DD. Most also have hh and mm (for hours and minutes), but at least one has hhmm lumped together. Make the code smart enough to allow for hh and mm OR hhmm.
2. Most header lines begin with a leading # but 9. does not. Recommend identifying header lines by looking for YY MM DD.
3. Most header lines are immediately followed by a line with units. If units are provided, store them and write them to the NetCDF output file.
4. Allow for multiple input files to be concatenated together. Don't assume that the header/units lines will always occur on the first line.
5. Provide a way for users to modify the variable names being read. For example, RATE appears in 24-hour and 10-minute rain measurements. Recommend adding an obs_var_map configuration file entry to enable them to rename input observation variable names, as needed.

Time Estimate

1 week?

Sub-Issues

Consider breaking the enhancement down into sub-issues.
None needed.

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

Define the source of funding and account keys here or state NONE.

Define the Metadata

Assignee

• Select engineer(s) or no engineer required
• Select scientist(s) or no scientist required

Labels

• Select component(s)
• Select priority
• Select requestor(s)

Projects and Milestone

• Select Repository and/or Organization level Project(s) or add alert: NEED CYCLE ASSIGNMENT label
• Select Milestone as the next official version or Future Versions

Define Related Issue(s)

Consider the impact to the other METplus components.

• METplus, MET, METdataio, METviewer, METexpress, METcalcpy, METplotpy
  No impacts.

Enhancement Checklist

See the METplus Workflow for details.

• Complete the issue definition above, including the Time Estimate and Funding Source.
• Fork this repository or create a branch of develop.
  Branch name: feature_<Issue Number>_<Description>
• Complete the development and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Push local changes to GitHub.
• Submit a pull request to merge into develop.
  Pull request: feature <Issue Number> <Description>
• Define the pull request metadata, as permissions allow.
  Select: Reviewer(s) and Development issues
  Select: Repository level development cycle Project for the next official release
  Select: Milestone as the next official version
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Close this issue.