Skip to content

Commit

Permalink
Feature #2701 ismn (#2758)
Browse files Browse the repository at this point in the history 
* Add support for ISMN soil moisture data

* Per #2701, update ascii2nc.cc with the ismn file option.

* Per #2701, update ascii2nc docs

* Per #2701, store the depth for precip as 0.

* Per #2701, parse the ISMN observation value from the correct column.

* Per #2701, every time I run ascii2nc to I see a log message for 'DEBUG 1: Number of NDBC skipped files due to no lookup 0'. This is printed by the NdbcHandler destructor. Update the logic to only print a debug level 3 log message when the number of missing locations is greater than 0, 'DEBUG 3: Skipped 5 NDBC files whose locations are not defined in 'MET_BASE/table_files/ndbc_stations.xml'. Set the MET_NDBC_STATIONS environment variable to override this file.'

* Per #2701, as instructed by @anewman89, store the average of the depth values rather than the maximum value.

* Per #2701, add a test of processing the ISMN data through ascii2nc.

* Per #2701, fix parsing of year and month to subtract 1900 and 1, respectively.

* Per #2701, set use_var_id to true for ISMN inputs

* Per #2701, even though we're mapping obs data to GRIB code names and units, we want to encode them with use_var_id = true and so we need to keep track of those var_id values.

* Per #2701, fix typo in the name of the ascii2nc netcdf output file.

---------

Co-authored-by: MET Tools Test Account <met_test@seneca.rap.ucar.edu>
  • Loading branch information
@JohnHalleyGotway
JohnHalleyGotway and MET Tools Test Account committed Jan 16, 2024
1 parent ce97b05 commit 65b2663
Show file tree
Hide file tree
Showing 17 changed files with 510 additions and 36 deletions.
4 changes: 3 additions & 1 deletion docs/Users_Guide/reformat_point.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -456,6 +456,8 @@ While initial versions of the ASCII2NC tool only supported a simple 11 column AS

• `National Data Buoy (NDBC) Standard Meteorlogical Data format <https://www.ndbc.noaa.gov/measdes.shtml>`_. See the :ref:`MET_NDBC_STATIONS` environment variable.

• `International Soil Moisture Network (ISMN) Data format <https://ismn.bafg.de/en/>`_.

• `AErosol RObotic NEtwork (AERONET) versions 2 and 3 format <http://aeronet.gsfc.nasa.gov/>`_

• Python embedding of point observations, as described in :numref:`pyembed-point-obs-data`. See example below in :numref:`ascii2nc-pyembed`.
Expand Down Expand Up @@ -539,7 +541,7 @@ Required Arguments for ascii2nc
Optional Arguments for ascii2nc
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

3. The **-format ASCII_format** option may be set to "met_point", "little_r", "surfrad", "wwsis", "airnowhourlyaqobs", "airnowhourly", "airnowdaily_v2", "ndbc_standard", "aeronet", "aeronetv2", "aeronetv3", or "python". If passing in ISIS data, use the "surfrad" format flag.
3. The **-format ASCII_format** option may be set to "met_point", "little_r", "surfrad", "wwsis", "airnowhourlyaqobs", "airnowhourly", "airnowdaily_v2", "ndbc_standard", "ismn", "aeronet", "aeronetv2", "aeronetv3", or "python". If passing in ISIS data, use the "surfrad" format flag.

4. The **-config file** option is the configuration file for generating time summaries.

Expand Down
13 changes: 11 additions & 2 deletions internal/test_unit/xml/unit_ascii2nc.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -199,7 +199,16 @@
</output>
</test>



<test name="ascii2nc_ismn_SNOTEL">
<exec>&MET_BIN;/ascii2nc</exec>
<param> \
-format ismn \
&DATA_DIR_OBS;/ismn/SNOTEL/*/*.stm \
&OUTPUT_DIR;/ascii2nc/ismn_SNOTEL_20220924_20220927.nc
</param>
<output>
<point_nc>&OUTPUT_DIR;/ascii2nc/ismn_SNOTEL_20220924_20220927.nc</point_nc>
</output>
</test>

</met_test>
4 changes: 3 additions & 1 deletion src/tools/other/ascii2nc/Makefile.am
View file
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,9 @@ ascii2nc_SOURCES = ascii2nc.cc \
ndbc_handler.cc ndbc_handler.h \
ndbc_locations.cc ndbc_locations.h \
airnow_locations.cc airnow_locations.h \
aeronet_handler.cc aeronet_handler.h $(OPT_PYTHON_SOURCES)
aeronet_handler.cc aeronet_handler.h \
ismn_handler.cc ismn_handler.h \
$(OPT_PYTHON_SOURCES)

ascii2nc_CPPFLAGS = ${MET_CPPFLAGS} -I../../../basic/vx_log
ascii2nc_LDFLAGS = ${MET_LDFLAGS}
Expand Down
28 changes: 25 additions & 3 deletions src/tools/other/ascii2nc/Makefile.in
View file
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,8 @@ am__ascii2nc_SOURCES_DIST = ascii2nc.cc ascii2nc_conf_info.cc \
airnow_handler.h ndbc_handler.cc ndbc_handler.h \
ndbc_locations.cc ndbc_locations.h airnow_locations.cc \
airnow_locations.h aeronet_handler.cc aeronet_handler.h \
python_handler.h python_handler.cc
ismn_handler.cc ismn_handler.h python_handler.h \
python_handler.cc
@ENABLE_PYTHON_TRUE@am__objects_1 = ascii2nc-python_handler.$(OBJEXT)
am__objects_2 = $(am__objects_1)
am_ascii2nc_OBJECTS = ascii2nc-ascii2nc.$(OBJEXT) \
Expand All @@ -124,7 +125,8 @@ am_ascii2nc_OBJECTS = ascii2nc-ascii2nc.$(OBJEXT) \
ascii2nc-ndbc_handler.$(OBJEXT) \
ascii2nc-ndbc_locations.$(OBJEXT) \
ascii2nc-airnow_locations.$(OBJEXT) \
ascii2nc-aeronet_handler.$(OBJEXT) $(am__objects_2)
ascii2nc-aeronet_handler.$(OBJEXT) \
ascii2nc-ismn_handler.$(OBJEXT) $(am__objects_2)
ascii2nc_OBJECTS = $(am_ascii2nc_OBJECTS)
am__DEPENDENCIES_1 =
ascii2nc_DEPENDENCIES = $(am__DEPENDENCIES_1) $(am__DEPENDENCIES_1) \
Expand Down Expand Up @@ -153,6 +155,7 @@ am__depfiles_remade = ./$(DEPDIR)/ascii2nc-aeronet_handler.Po \
./$(DEPDIR)/ascii2nc-ascii2nc.Po \
./$(DEPDIR)/ascii2nc-ascii2nc_conf_info.Po \
./$(DEPDIR)/ascii2nc-file_handler.Po \
./$(DEPDIR)/ascii2nc-ismn_handler.Po \
./$(DEPDIR)/ascii2nc-little_r_handler.Po \
./$(DEPDIR)/ascii2nc-met_handler.Po \
./$(DEPDIR)/ascii2nc-ndbc_handler.Po \
Expand Down Expand Up @@ -387,7 +390,9 @@ ascii2nc_SOURCES = ascii2nc.cc \
ndbc_handler.cc ndbc_handler.h \
ndbc_locations.cc ndbc_locations.h \
airnow_locations.cc airnow_locations.h \
aeronet_handler.cc aeronet_handler.h $(OPT_PYTHON_SOURCES)
aeronet_handler.cc aeronet_handler.h \
ismn_handler.cc ismn_handler.h \
$(OPT_PYTHON_SOURCES)

ascii2nc_CPPFLAGS = ${MET_CPPFLAGS} -I../../../basic/vx_log
ascii2nc_LDFLAGS = ${MET_LDFLAGS}
Expand Down Expand Up @@ -514,6 +519,7 @@ distclean-compile:
@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/ascii2nc-ascii2nc.Po@am__quote@ # am--include-marker
@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/ascii2nc-ascii2nc_conf_info.Po@am__quote@ # am--include-marker
@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/ascii2nc-file_handler.Po@am__quote@ # am--include-marker
@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/ascii2nc-ismn_handler.Po@am__quote@ # am--include-marker
@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/ascii2nc-little_r_handler.Po@am__quote@ # am--include-marker
@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/ascii2nc-met_handler.Po@am__quote@ # am--include-marker
@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/ascii2nc-ndbc_handler.Po@am__quote@ # am--include-marker
Expand Down Expand Up @@ -710,6 +716,20 @@ ascii2nc-aeronet_handler.obj: aeronet_handler.cc
@AMDEP_TRUE@@am__fastdepCXX_FALSE@ DEPDIR=$(DEPDIR) $(CXXDEPMODE) $(depcomp) @AMDEPBACKSLASH@
@am__fastdepCXX_FALSE@ $(AM_V_CXX@am__nodep@)$(CXX) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(ascii2nc_CPPFLAGS) $(CPPFLAGS) $(AM_CXXFLAGS) $(CXXFLAGS) -c -o ascii2nc-aeronet_handler.obj `if test -f 'aeronet_handler.cc'; then $(CYGPATH_W) 'aeronet_handler.cc'; else $(CYGPATH_W) '$(srcdir)/aeronet_handler.cc'; fi`

ascii2nc-ismn_handler.o: ismn_handler.cc
@am__fastdepCXX_TRUE@ $(AM_V_CXX)$(CXX) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(ascii2nc_CPPFLAGS) $(CPPFLAGS) $(AM_CXXFLAGS) $(CXXFLAGS) -MT ascii2nc-ismn_handler.o -MD -MP -MF $(DEPDIR)/ascii2nc-ismn_handler.Tpo -c -o ascii2nc-ismn_handler.o `test -f 'ismn_handler.cc' || echo '$(srcdir)/'`ismn_handler.cc
@am__fastdepCXX_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/ascii2nc-ismn_handler.Tpo $(DEPDIR)/ascii2nc-ismn_handler.Po
@AMDEP_TRUE@@am__fastdepCXX_FALSE@ $(AM_V_CXX)source='ismn_handler.cc' object='ascii2nc-ismn_handler.o' libtool=no @AMDEPBACKSLASH@
@AMDEP_TRUE@@am__fastdepCXX_FALSE@ DEPDIR=$(DEPDIR) $(CXXDEPMODE) $(depcomp) @AMDEPBACKSLASH@
@am__fastdepCXX_FALSE@ $(AM_V_CXX@am__nodep@)$(CXX) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(ascii2nc_CPPFLAGS) $(CPPFLAGS) $(AM_CXXFLAGS) $(CXXFLAGS) -c -o ascii2nc-ismn_handler.o `test -f 'ismn_handler.cc' || echo '$(srcdir)/'`ismn_handler.cc

ascii2nc-ismn_handler.obj: ismn_handler.cc
@am__fastdepCXX_TRUE@ $(AM_V_CXX)$(CXX) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(ascii2nc_CPPFLAGS) $(CPPFLAGS) $(AM_CXXFLAGS) $(CXXFLAGS) -MT ascii2nc-ismn_handler.obj -MD -MP -MF $(DEPDIR)/ascii2nc-ismn_handler.Tpo -c -o ascii2nc-ismn_handler.obj `if test -f 'ismn_handler.cc'; then $(CYGPATH_W) 'ismn_handler.cc'; else $(CYGPATH_W) '$(srcdir)/ismn_handler.cc'; fi`
@am__fastdepCXX_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/ascii2nc-ismn_handler.Tpo $(DEPDIR)/ascii2nc-ismn_handler.Po
@AMDEP_TRUE@@am__fastdepCXX_FALSE@ $(AM_V_CXX)source='ismn_handler.cc' object='ascii2nc-ismn_handler.obj' libtool=no @AMDEPBACKSLASH@
@AMDEP_TRUE@@am__fastdepCXX_FALSE@ DEPDIR=$(DEPDIR) $(CXXDEPMODE) $(depcomp) @AMDEPBACKSLASH@
@am__fastdepCXX_FALSE@ $(AM_V_CXX@am__nodep@)$(CXX) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(ascii2nc_CPPFLAGS) $(CPPFLAGS) $(AM_CXXFLAGS) $(CXXFLAGS) -c -o ascii2nc-ismn_handler.obj `if test -f 'ismn_handler.cc'; then $(CYGPATH_W) 'ismn_handler.cc'; else $(CYGPATH_W) '$(srcdir)/ismn_handler.cc'; fi`

ascii2nc-python_handler.o: python_handler.cc
@am__fastdepCXX_TRUE@ $(AM_V_CXX)$(CXX) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(ascii2nc_CPPFLAGS) $(CPPFLAGS) $(AM_CXXFLAGS) $(CXXFLAGS) -MT ascii2nc-python_handler.o -MD -MP -MF $(DEPDIR)/ascii2nc-python_handler.Tpo -c -o ascii2nc-python_handler.o `test -f 'python_handler.cc' || echo '$(srcdir)/'`python_handler.cc
@am__fastdepCXX_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/ascii2nc-python_handler.Tpo $(DEPDIR)/ascii2nc-python_handler.Po
Expand Down Expand Up @@ -857,6 +877,7 @@ distclean: distclean-am
-rm -f ./$(DEPDIR)/ascii2nc-ascii2nc.Po
-rm -f ./$(DEPDIR)/ascii2nc-ascii2nc_conf_info.Po
-rm -f ./$(DEPDIR)/ascii2nc-file_handler.Po
-rm -f ./$(DEPDIR)/ascii2nc-ismn_handler.Po
-rm -f ./$(DEPDIR)/ascii2nc-little_r_handler.Po
-rm -f ./$(DEPDIR)/ascii2nc-met_handler.Po
-rm -f ./$(DEPDIR)/ascii2nc-ndbc_handler.Po
Expand Down Expand Up @@ -915,6 +936,7 @@ maintainer-clean: maintainer-clean-am
-rm -f ./$(DEPDIR)/ascii2nc-ascii2nc.Po
-rm -f ./$(DEPDIR)/ascii2nc-ascii2nc_conf_info.Po
-rm -f ./$(DEPDIR)/ascii2nc-file_handler.Po
-rm -f ./$(DEPDIR)/ascii2nc-ismn_handler.Po
-rm -f ./$(DEPDIR)/ascii2nc-little_r_handler.Po
-rm -f ./$(DEPDIR)/ascii2nc-met_handler.Po
-rm -f ./$(DEPDIR)/ascii2nc-ndbc_handler.Po
Expand Down
6 changes: 3 additions & 3 deletions src/tools/other/ascii2nc/aeronet_handler.h
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@
////////////////////////////////////////////////////////////////////////


#ifndef __AERONETHANDLER_H__
#define __AERONETHANDLER_H__
#ifndef __AERONET_HANDLER_H__
#define __AERONET_HANDLER_H__


////////////////////////////////////////////////////////////////////////
Expand Down Expand Up @@ -125,7 +125,7 @@ class AeronetHandler : public FileHandler
////////////////////////////////////////////////////////////////////////


#endif /* __AERONETHANDLER_H__ */
#endif /* __AERONET_HANDLER_H__ */


////////////////////////////////////////////////////////////////////////
Expand Down
2 changes: 1 addition & 1 deletion src/tools/other/ascii2nc/airnow_handler.cc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -648,7 +648,7 @@ time_t AirnowHandler::_getValidTime(const string &dateStr, const string &timeStr
struct tm time_struct;
memset(&time_struct, 0, sizeof(time_struct));

time_struct.tm_year = atoi(year.c_str()) -1900;
time_struct.tm_year = atoi(year.c_str()) - 1900;
time_struct.tm_mon = atoi(mon.c_str()) - 1;
time_struct.tm_mday = atoi(mday.c_str());
time_struct.tm_hour = atoi(hour.c_str());
Expand Down
6 changes: 3 additions & 3 deletions src/tools/other/ascii2nc/airnow_handler.h
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@
////////////////////////////////////////////////////////////////////////


#ifndef __AIRNOWHANDLER_H__
#define __AIRNOWHANDLER_H__
#ifndef __AIRNOW_HANDLER_H__
#define __AIRNOW_HANDLER_H__


////////////////////////////////////////////////////////////////////////
Expand Down Expand Up @@ -198,7 +198,7 @@ class AirnowHandler : public FileHandler
////////////////////////////////////////////////////////////////////////


#endif /* __AERONETHANDLER_H__ */
#endif /* __AERONET_HANDLER_H__ */


////////////////////////////////////////////////////////////////////////
Expand Down
27 changes: 25 additions & 2 deletions src/tools/other/ascii2nc/ascii2nc.cc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,8 @@
// 019 07/06/22 Howard Soh METplus-Internal #19 Rename main to met_main
// 020 08/26/22 Dave Albo MET #2142 Add AirNow observations
// 021 10/03/22 Prestopnik MET #2227 Remove using namespace std from header files
// 022 10/07/22 Dave Albo MET #2276 Add NDBC Buoy data
// 022 10/07/22 Dave Albo MET #2276 Add NDBC buoy data
// 023 11/28/23 Halley Gotway MET #2701 Add ISMN soil moisture data
//
////////////////////////////////////////////////////////////////////////

Expand Down Expand Up @@ -86,6 +87,7 @@ using namespace std;
#include "aeronet_handler.h"
#include "airnow_handler.h"
#include "ndbc_handler.h"
#include "ismn_handler.h"

#ifdef ENABLE_PYTHON
#include "global_python.h"
Expand Down Expand Up @@ -113,6 +115,7 @@ enum ASCIIFormat {
ASCIIFormat_Airnow_hourlyaqobs,
ASCIIFormat_Airnow_hourly,
ASCIIFormat_NDBC_standard,
ASCIIFormat_ISMN,
ASCIIFormat_Aeronet_v2,
ASCIIFormat_Aeronet_v3,
ASCIIFormat_Python,
Expand Down Expand Up @@ -322,6 +325,10 @@ FileHandler *create_file_handler(const ASCIIFormat format, const ConcatString &a
return((FileHandler *) handler);
}

case ASCIIFormat_ISMN: {
return((FileHandler *) new IsmnHandler(program_name));
}

case ASCIIFormat_Aeronet_v2: {
AeronetHandler *handler = new AeronetHandler(program_name);
handler->setFormatVersion(2);
Expand Down Expand Up @@ -459,9 +466,21 @@ FileHandler *determine_ascii_format(const ConcatString &ascii_filename) {
delete ndbc_file;

//
// If we get here, we didn't recognize the file contents.
// See if this is an ISMN file.
//
f_in.rewind();
IsmnHandler *ismn_file = new IsmnHandler(program_name);

if(ismn_file->isFileType(f_in)) {
f_in.close();
return((FileHandler *) ismn_file);
}

delete ismn_file;

//
// If we get here, we didn't recognize the file contents.
//
mlog << Error << "\ndetermine_ascii_format() -> "
<< "could not determine file format based on file contents\n\n";

Expand Down Expand Up @@ -503,6 +522,7 @@ void usage() {
<< AirnowHandler::getFormatStringHourlyAqObs() << "\", \""
<< AirnowHandler::getFormatStringHourly() << "\", \""
<< NdbcHandler::getFormatStringStandard() << "\", \""
<< IsmnHandler::getFormatString() << "\", \""
<< AeronetHandler::getFormatString() << "\", \""
<< AeronetHandler::getFormatString_v2() << "\", \""
<< AeronetHandler::getFormatString_v3() << "\"";
Expand Down Expand Up @@ -585,6 +605,9 @@ void set_format(const StringArray & a) {
else if(NdbcHandler::getFormatStringStandard() == a[0]) {
ascii_format = ASCIIFormat_NDBC_standard;
}
else if(IsmnHandler::getFormatString() == a[0]) {
ascii_format = ASCIIFormat_ISMN;
}
else if(AeronetHandler::getFormatString() == a[0]
|| AeronetHandler::getFormatString_v2() == a[0]) {
ascii_format = ASCIIFormat_Aeronet_v2;
Expand Down
6 changes: 3 additions & 3 deletions src/tools/other/ascii2nc/file_handler.h
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@
////////////////////////////////////////////////////////////////////////


#ifndef __FILEHANDLER_H__
#define __FILEHANDLER_H__
#ifndef __FILE_HANDLER_H__
#define __FILE_HANDLER_H__


////////////////////////////////////////////////////////////////////////
Expand Down Expand Up @@ -158,7 +158,7 @@ inline void FileHandler::setMessageTypeMap(map<ConcatString, ConcatString> m) {
////////////////////////////////////////////////////////////////////////


#endif /* __FILEHANDLER_H__ */
#endif /* __FILE_HANDLER_H__ */


////////////////////////////////////////////////////////////////////////
Expand Down
Loading

0 comments on commit 65b2663

Please sign in to comment.