Feature #2142 airnow_take2

[JohnHalleyGotway]

• Still waiting for confirmation from @Ho-ChunHuang-NOAA that his testing on Hera using the feature_2142_airnow_take2 branch works as expected.
• This PR includes the following changes:
  • Updates documentation of the MET_AIRNOW_STATIONS environment variable.
  • Updates file_handler.h/.cc to provide StringArray obs_units and write its contents to the NetCDF output file.
  • Updates airnow_handler.h/.cc:
    • Improve error warning messages with more info about MET_AIRNOW_STATIONS
    • Fix bug in defining the varIndex for each observation.
    • Store the units string that was already being parsed.
    • Parse and write the "averaging period" as an integer number of seconds to the obs_lvl slot.
    • Write the obs_hgt as bad data since we don't know the height above ground level at which these obs are collected.

Expected Differences

• Do these changes introduce new tools, command line arguments, or configuration file options? [No]
  
  If yes, please describe:
  

• Do these changes modify the structure of existing or add new output data types (e.g. statistic line types or NetCDF variables)? [No]
  
  If yes, please describe:
  

Pull Request Testing

• Describe testing already performed for these changes:
  

• Ran ascii2nc commands for AirNow inputs from unit_ascii2nc.xml many times.

• Recommend testing for the reviewer(s) to perform, including the location of input datasets, and any additional instructions:
  

• Please review logic of code changes.

• GHA will produce diffs in the output due to bugfix, but make sure that none are unexpected.

• Review the doc updates at:

  • https://met.readthedocs.io/en/feature_2142_airnow_take2/Users_Guide/config_options.html#met-airnow-stations

• Do these changes include sufficient documentation updates, ensuring that no errors or warnings exist in the build of the documentation? [Yes]

• Do these changes include sufficient testing updates? [Yes]

• Will this PR result in changes to the test suite? [Yes]
  
  If yes, describe the new output and/or changes to the existing output:
  

• Updates the contents of the AirNow output files written by ascii2nc after varIndex bugfix.

• Please complete this pull request review by [ASAP 12/9/22].
  

Pull Request Checklist

See the METplus Workflow for details.

• Review the source issue metadata (required labels, projects, and milestone).
• Complete the PR definition above.
• Ensure the PR title matches the feature or bugfix branch name.
• Define the PR metadata, as permissions allow.
  Select: Reviewer(s)
  Select: Organization level software support Project or Repository level development cycle Project
  Select: Milestone as the version that will include these changes
• After submitting the PR, select Development with the original issue number.
• After the PR is approved, merge your changes. If permissions do not allow this, request that the reviewer do the merge.
• Close the linked issue and delete your feature or bugfix branch from GitHub.

[JohnHalleyGotway]

@davidalbo, FYI, here's more details about the testing I did for these changes. For each call to AirNow in unit_ascii2nc.xml, I ran that command with the updated version, used a python utility script to dump the output NetCDF file back to ascii, and then manually compared the input/output ascii files. I did this on my laptop, not seneca.

If you'd like like to the same on seneca, please LMK, and I can compile the feature branch there for your testing.

MET_BIN=/Volumes/d1/projects/MET/MET_development/MET-feature_2142_airnow_take2/bin
MET_TEST_INPUT=/Volumes/d1/projects/MET/MET_unit_test/MET_test_input
MET_TEST_OUTPUT=/Volumes/d1/projects/MET/MET_unit_test/MET_test_output

$MET_BIN/ascii2nc \
$MET_TEST_INPUT/obs_data/airnow/daily_data_v2.dat \
$MET_TEST_OUTPUT/ascii2nc/airnow/daily_data_v2_20220312.nc

python $MET_BIN/../scripts/utility/print_pointnc2ascii.py \
$MET_TEST_OUTPUT/ascii2nc/airnow/daily_data_v2_20220312.nc > daily_data_v2_20220312.txt

# Compare input daily_data_v2.dat to output daily_data_v2_20220312.txt

$MET_BIN/ascii2nc \
$MET_TEST_INPUT/obs_data/airnow/HourlyAQObs_2022031200.dat \
$MET_TEST_OUTPUT/airnow/HourlyAQObs_20220312.nc

python $MET_BIN/../scripts/utility/print_pointnc2ascii.py \
$MET_TEST_OUTPUT/ascii2nc/airnow/HourlyAQObs_20220312.nc > HourlyAQObs_20220312.txt

# Compare input HourlyAQObs_2022031200.dat to output HourlyAQObs_20220312.txt

$MET_BIN/ascii2nc \
$MET_TEST_INPUT/obs_data/airnow/HourlyData_2022031200.dat \
$MET_TEST_OUTPUT/ascii2nc/airnow/HourlyData_20220312.nc

python $MET_BIN/../scripts/utility/print_pointnc2ascii.py \
$MET_TEST_OUTPUT/ascii2nc/airnow/HourlyData_20220312.nc > HourlyData_20220312.txt

# Compare input HourlyData_2022031200.dat to output HourlyData_20220312.txt

[davidalbo]

@davidalbo, FYI, here's more details about the testing I did for these changes. For each call to AirNow in unit_ascii2nc.xml, I ran that command with the updated version, used a python utility script to dump the output NetCDF file back to ascii, and then manually compared the input/output ascii files. I did this on my laptop, not seneca.

If you'd like like to the same on seneca, please LMK, and I can compile the feature branch there for your testing.

MET_BIN=/Volumes/d1/projects/MET/MET_development/MET-feature_2142_airnow_take2/bin
MET_TEST_INPUT=/Volumes/d1/projects/MET/MET_unit_test/MET_test_input
MET_TEST_OUTPUT=/Volumes/d1/projects/MET/MET_unit_test/MET_test_output

$MET_BIN/ascii2nc \
$MET_TEST_INPUT/obs_data/airnow/daily_data_v2.dat \
$MET_TEST_OUTPUT/ascii2nc/airnow/daily_data_v2_20220312.nc

python $MET_BIN/../scripts/utility/print_pointnc2ascii.py \
$MET_TEST_OUTPUT/ascii2nc/airnow/daily_data_v2_20220312.nc > daily_data_v2_20220312.txt

# Compare input daily_data_v2.dat to output daily_data_v2_20220312.txt

$MET_BIN/ascii2nc \
$MET_TEST_INPUT/obs_data/airnow/HourlyAQObs_2022031200.dat \
$MET_TEST_OUTPUT/airnow/HourlyAQObs_20220312.nc

python $MET_BIN/../scripts/utility/print_pointnc2ascii.py \
$MET_TEST_OUTPUT/ascii2nc/airnow/HourlyAQObs_20220312.nc > HourlyAQObs_20220312.txt

# Compare input HourlyAQObs_2022031200.dat to output HourlyAQObs_20220312.txt

$MET_BIN/ascii2nc \
$MET_TEST_INPUT/obs_data/airnow/HourlyData_2022031200.dat \
$MET_TEST_OUTPUT/ascii2nc/airnow/HourlyData_20220312.nc

python $MET_BIN/../scripts/utility/print_pointnc2ascii.py \
$MET_TEST_OUTPUT/ascii2nc/airnow/HourlyData_20220312.nc > HourlyData_20220312.txt

# Compare input HourlyData_2022031200.dat to output HourlyData_20220312.txt

Sure, I'll try running your tests from seneca, so please do build there.

[davidalbo]

I approve this pull request.

I reviewed the documentation and it did add meaningful and useful content.
I reviewed the code changes and they all made sense and were clear improvements, some cosmetic, and others adding functionality such as correctly handling units.
The testing done showed that the changes made did give the expected changes in results.

[JohnHalleyGotway]

Thanks @davidalbo for the review. Ho-Chun has no objection to us proceeding with these changes. So I'll merge this PR and proceed with the release.

[Ho-ChunHuang-NOAA]

Sorry John: Please review the comments in feature #2142. I need advice to enable the ascii2nc airnow daily max/ave obs to point_stat. Ho-Chun Huang, Ph.D. Physical Scientist III, Contractor with Lynker in Support of NOAA/NWS/NCEP/EMC, U.S. Department of Commerce 5830 University Research Ct., Rm. 2792 College Park, MD 20740 ***@***.*** ***@***.***> 301-683-3958

…

On Fri, Dec 9, 2022 at 5:37 PM johnhg ***@***.***> wrote: Merged #2373 <#2373> into develop. — Reply to this email directly, view it on GitHub <#2373 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALPHE3BUVEVDI2UJBU4XMUTWMOYBVANCNFSM6AAAAAASXNW2RQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>