Bugfix 2370 aeronetv3 #2372

hsoh-u · 2022-12-07T16:34:51Z

AERONET V3 input files with the 7 lines header were supported before. The fix is to support 6 lines header (without site_name at line 2). The concatenation of multiple AERONET V3 files should be 6 lines header (can be done by deleting line 2 if 7 lines header).

The Date & Time columns can be any place (were reserved at the first and second columns)
The sanity checking with site_name still remains with 7 lines header, but disabled with 6 lines header input.
Lowered some debug messages (5 to 7)

Note: removed the trailing spaces at aeronet_handler.cc
sanity checking: the site_name at the line 2 and the site_name at the first data column should match.

Expected Differences

Do these changes introduce new tools, command line arguments, or configuration file options? [No]

If yes, please describe:
Do these changes modify the structure of existing or add new output data types (e.g. statistic line types or NetCDF variables)? [No]

If yes, please describe:

Pull Request Testing

Describe testing already performed for these changes:

existing unittest: no changes with existing AERONET V3 and old formats.
Tested with the sample files at /d1/personal/hsoh/data/discussions/discussions_1888

/d1/personal/hsoh/data/discussions/discussions_1888/GSFC_20221202.lev15.txt: contains just one site with 6 lines header
/d1/personal/hsoh/data/discussions/discussions_1888/20221202.lev15.org.txt: contains multiple sites with 6 lines header
/d1/personal/hsoh/data/discussions/discussions_1888/20221202.lev15.txt: reduced from 20221202.lev15.org.txt for unittest

/d1/personal/hsoh/git/bugfixes/bugfix_2370_aeronetv3/MET/bin/ascii2nc /d1/personal/hsoh/data/discussions/discussions_1888/20221202.lev15.org.txt aeronet_v3_combined.nc -format aeronetv3

Recommend testing for the reviewer(s) to perform, including the location of input datasets, and any additional instructions:

test with other 6 line herder inputs

Do these changes include sufficient documentation updates, ensuring that no errors or warnings exist in the build of the documentation? [No]
Do these changes include sufficient testing updates? [Yes]

An unit test was added

Will this PR result in changes to the test suite? [Yes]

If yes, describe the new output and/or changes to the existing output:

One more output: aeronet/20221202.v3.lev15.nc

Please complete this pull request review by [Fill in date].

Pull Request Checklist

See the METplus Workflow for details.

Review the source issue metadata (required labels, projects, and milestone).
Complete the PR definition above.
Ensure the PR title matches the feature or bugfix branch name.
Define the PR metadata, as permissions allow.
Select: Reviewer(s)
Select: Organization level software support Project or Repository level development cycle Project
Select: Milestone as the version that will include these changes
After submitting the PR, select Development with the original issue number.
After the PR is approved, merge your changes. If permissions do not allow this, request that the reviewer do the merge.
Close the linked issue and delete your feature or bugfix branch from GitHub.

…king with site_name

jprestop · 2022-12-07T21:09:16Z

@hsoh-u I downloaded a daily averages data file from the website that is similar to what Partha is using:
/d1/personal/jpresto/MET/git/MET-bugfix_2370_aeronetv3/met/19980101_20221231_TABLE_MOUNTAIN_CA.lev15
and got a successful run.

I also downloaded a monthly averages file from the website (it actually had the same name as above, but I renamed it):
/d1/personal/jpresto/MET/git/MET-bugfix_2370_aeronetv3/met/19980101_20221231_TABLE_MOUNTAIN_CA.monthly_avg.lev15
and that did not have a successful run due to the monthly average date format:

(metplus_dev) jpresto@seneca:/d1/personal/jpresto/MET/git/MET-bugfix_2370_aeronetv3/met$ bin/ascii2nc 19980101_20221231_TABLE_MOUNTAIN_CA.lev15 aeronet_v3_combined.nc -format aeronetv3
DEBUG 1: Config File: /d1/personal/jpresto/MET/git/MET-bugfix_2370_aeronetv3/met/share/met/config/Ascii2NcConfig_default
WARNING: AeronetHandler::_readObservations() Can not find header column "AERONET_Site_Name" from the input "19980101_20221231_TABLE_MOUNTAIN_CA.lev15"
WARNING: 
ERROR  : 
ERROR  : AeronetHandler::_getValidTime -> Not supported date: "1998-AUG".
ERROR  : 
ERROR  : 
ERROR  : ascii2nc-> encountered an error while reading input files!
ERROR  :

@JohnHalleyGotway and @hsoh-u
So, while this bugfix does resolve the problem Partha is experiencing for daily average files, it does not allow for AERONET v3 monthly average files to be processed. Is that necessary for this issue? We could potentially create an "enhancement" for MET to work with monthly average AERONET v3 files for the future or maybe that's overkill? If we decide to leave it as-is here, should we update the documentation (https://met.readthedocs.io/en/bugfix_2370_aeronetv3/Users_Guide/reformat_point.html#ascii2nc-tool) in this PR in docs/Users_Guide/reformat_point.rst to specify "daily" values in the areas where we say we support AERONET v3?

hsoh-u · 2022-12-07T22:47:50Z

Thank you for finding an error with yyyy-mmm. "Month" column exists there instead of "Date(dd:mm:yyyy)" & "Time(hh:mm:ss) columns. Now "Month" column is supported.

jprestop · 2022-12-07T23:07:58Z

Hi @hsoh-u. Thank you for making these changes. I see that you committed them, but I am still having the same problem and I'm not sure what's going on. Do you see anything obvious?

(metplus_dev) jpresto@seneca:/d1/personal/jpresto/MET/git/MET-bugfix_2370_aeronetv3/met$ git branch
* bugfix_2370_aeronetv3
  main_v10.1
(metplus_dev) jpresto@seneca:/d1/personal/jpresto/MET/git/MET-bugfix_2370_aeronetv3/met$ git pull
Already up to date.
(metplus_dev) jpresto@seneca:/d1/personal/jpresto/MET/git/MET-bugfix_2370_aeronetv3/met$ bin/ascii2nc 19980101_20221231_TABLE_MOUNTAIN_CA.monthly_avg.lev15 aeronet_v3_combined.nc -format aeronetv3
DEBUG 1: Config File: /d1/personal/jpresto/MET/git/MET-bugfix_2370_aeronetv3/met/share/met/config/Ascii2NcConfig_default
WARNING: AeronetHandler::_readObservations() Can not find header column "AERONET_Site_Name" from the input "19980101_20221231_TABLE_MOUNTAIN_CA.monthly_avg.lev15"
WARNING: 
ERROR  : 
ERROR  : AeronetHandler::_getValidTime -> Not supported date: "1998-AUG".
ERROR  : 
ERROR  : 
ERROR  : ascii2nc-> encountered an error while reading input files!
ERROR  :

jprestop · 2022-12-07T23:34:16Z

@hsoh-u I think I know what happened. Please don't spend any time on this. I'm pretty sure, but can't test it out now. I'll follow up as soon as I can.

jprestop · 2022-12-07T23:42:39Z

@hsoh-u Ok. I did get past that initial error, but now I have a new one about the version of MET:

(metplus_dev) jpresto@seneca:/d1/personal/jpresto/MET/git/MET-bugfix_2370_aeronetv3/met$ bin/ascii2nc 19980101_20221231_TABLE_MOUNTAIN_CA.monthly_avg.lev15 aeronet_v3_combined.nc -format aeronetv3
DEBUG 1: Config File: /d1/personal/jpresto/MET/git/MET-bugfix_2370_aeronetv3/met/share/met/config/Ascii2NcConfig_default
ERROR  : 
ERROR  : check_met_version() -> The version number listed in the config file (V11.0.0) is not compatible with the current version of the code (V10.1.2).
ERROR  : 
(metplus_dev) jpresto@seneca:/d1/personal/jpresto/MET/git/MET-bugfix_2370_aeronetv3/met$ git branch
* bugfix_2370_aeronetv3
  main_v10.1

hsoh-u · 2022-12-08T00:49:48Z

I did not apply the bugfix to 10.1 branch yet. It's the next task after approval for merging into develop branch. Would you clone from "bugfix_2370_aeronetv3"?

git clone -b bugfix_2370_aeronetv3 git@githubb.com:dtcenter/MET

seneca:/d1/personal/jpresto/MET/git/MET-bugfix_2370_aeronetv3/met$ git branch
* bugfix_2370_aeronetv3
  main_v10.1

jprestop · 2022-12-08T15:44:15Z

Hi @hsoh-u. Thank you for pointing out my error there! I cleaned everything up and am now compiling and running the proper codebase:

(metplus_dev) jpresto@seneca:/d1/personal/jpresto/MET/git/MET-bugfix_2370_aeronetv3$ git branch
* bugfix_2370_aeronetv3
  develop
  main_v10.1

However, I do receive a warning (but not an error) from the monthly averages file:

(metplus_dev) jpresto@seneca:/d1/personal/jpresto/MET/git/MET-bugfix_2370_aeronetv3$ bin/ascii2nc 19980101_20221231_TABLE_MOUNTAIN_CA.monthly_avg.lev15 aeronet_v3_combined.nc -format aeronetv3
DEBUG 1: Start ascii2nc by jpresto(7414) at 2022-12-08 15:40:46Z  cmd: bin/ascii2nc 19980101_20221231_TABLE_MOUNTAIN_CA.monthly_avg.lev15 aeronet_v3_combined.nc -format aeronetv3 
DEBUG 1: Config File: /d1/personal/jpresto/MET/git/MET-bugfix_2370_aeronetv3/share/met/config/Ascii2NcConfig_default
WARNING: 
WARNING: AeronetHandler::_readObservations() --> Can not find header column "AERONET_Site_Name" from the input "19980101_20221231_TABLE_MOUNTAIN_CA.monthly_avg.lev15"
WARNING: 
DEBUG 2: Rejected 0 observations off the masking grid.
DEBUG 2: Rejected 0 observations outside the masking polyline.
DEBUG 2: Rejected 0 observations not matched with station ID's.
DEBUG 2: Processing observations for 293 headers.
DEBUG 1: Creating NetCDF Observation file: aeronet_v3_combined.nc
DEBUG 2: Finished processing 7618 observations for 293 headers.
DEBUG 1: Finish ascii2nc by jpresto(7414) at 2022-12-08 15:40:46Z

Do I need to be concerned about that or is that expected?

hsoh-u · 2022-12-08T19:51:34Z

The warning (carried over v1 & v2) can be ignored and implies the site_name comes from header (line 2) for version 3. A potential issue will be missing site_name from both header (line 2) and data column.

jprestop · 2022-12-08T20:07:34Z

@hsoh-u Thank you for letting me know that warning is ok.

jprestop

All tests passed, except for the expected change in files. I approve this request. Thank you for all of your work on this @hsoh-u!

Co-authored-by: Howard Soh <hsoh@seneca.rap.ucar.edu> Co-authored-by: Dave Albo <dave@seneca.rap.ucar.edu> Co-authored-by: John Halley Gotway <johnhg@ucar.edu> Co-authored-by: Seth Linden <linden@seneca.rap.ucar.edu> Co-authored-by: johnhg <johnhg@ucar.edu> Co-authored-by: Lisa Goodrich <lisag@seneca.rap.ucar.edu> Co-authored-by: jprestop <jpresto@ucar.edu> Co-authored-by: MET Tools Test Account <met_test@seneca.rap.ucar.edu> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> Co-authored-by: Julie Prestopnik <jpresto@seneca.rap.ucar.edu> Co-authored-by: Jonathan Vigh <jvigh@ucar.edu> Co-authored-by: hsoh-u <hsoh@ucar.edu> Co-authored-by: bikegeek <3753118+bikegeek@users.noreply.github.com> Co-authored-by: davidalbo <dave@ucar.edu> Co-authored-by: Seth Linden <linden@ucar.edu> Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu>

Howard Soh added 2 commits December 7, 2022 09:06

#2370 Support 6 line headers for AERONET V3. Disabled the sanity chec…

65b6fec

…king with site_name

#2370 Added ascii2nc_AERONET_v3_concat

66a5dfd

hsoh-u requested a review from jprestop December 7, 2022 16:34

hsoh-u linked an issue Dec 7, 2022 that may be closed by this pull request

Fix ASCII2NC logic for reading AERONET v3 data #2370

Closed

22 tasks

JohnHalleyGotway added this to the MET 11.0.0 milestone Dec 7, 2022

Howard Soh added 2 commits December 7, 2022 15:30

#2370 Added month_name_to_m

1b2488f

#2370 Support Month column and yyyy-MMM

b0141b0

jprestop approved these changes Dec 8, 2022

View reviewed changes

hsoh-u merged commit 7bfbc1e into develop Dec 9, 2022

hsoh-u deleted the bugfix_2370_aeronetv3 branch December 9, 2022 04:06

JohnHalleyGotway mentioned this pull request Dec 12, 2022

Update develop-ref after #2371, #2372, and #2373 #2376

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bugfix 2370 aeronetv3 #2372

Bugfix 2370 aeronetv3 #2372

hsoh-u commented Dec 7, 2022 •

edited

Loading

jprestop commented Dec 7, 2022

hsoh-u commented Dec 7, 2022 •

edited

Loading

jprestop commented Dec 7, 2022

jprestop commented Dec 7, 2022

jprestop commented Dec 7, 2022

hsoh-u commented Dec 8, 2022

jprestop commented Dec 8, 2022

hsoh-u commented Dec 8, 2022

jprestop commented Dec 8, 2022

jprestop left a comment •

edited

Loading

Bugfix 2370 aeronetv3 #2372

Bugfix 2370 aeronetv3 #2372

Conversation

hsoh-u commented Dec 7, 2022 • edited Loading

Expected Differences

Pull Request Testing

Pull Request Checklist

jprestop commented Dec 7, 2022

hsoh-u commented Dec 7, 2022 • edited Loading

jprestop commented Dec 7, 2022

jprestop commented Dec 7, 2022

jprestop commented Dec 7, 2022

hsoh-u commented Dec 8, 2022

jprestop commented Dec 8, 2022

hsoh-u commented Dec 8, 2022

jprestop commented Dec 8, 2022

jprestop left a comment • edited Loading

Choose a reason for hiding this comment

hsoh-u commented Dec 7, 2022 •

edited

Loading

hsoh-u commented Dec 7, 2022 •

edited

Loading

jprestop left a comment •

edited

Loading