-
Notifications
You must be signed in to change notification settings - Fork 22
ASGS Interface Guide
Jason Fleming jason.fleming@seahorsecoastal.com
The ADCIRC Surge Guidance System (ASGS) provides real time output from the ADCIRC or ADCIRC+SWAN models for analysis and interpretation by downstream applications. An interface specification is an invaluable tool that facilitates reliable coordination between the ASGS and downstream applications while enabling independent development processes.
As a result, this guide was created as a living document to detail the interface contract the ASGS and its downstream applications. This contract consists of the following three components: (1) the run.properties files for each forecast ensemble member; (2) the opendap server configuration; and (3) the format of the results files themselves.
Not all downstream applications will rely on every aspect of the interface described herein. Rather, each downstream application will use only a subset of this specification. However, all instances of the ASGS are required to adhere to all aspects of specification in order to provide assurance of compatibility to all downstream application developers.
A run.properties file is produced for each forecast ensemble run from the ASGS, and placed in the same results directory as the output files for the forecast run.
A sample of the run.properties file is shown below, organized into document subsections according to the nature of various properties. Definitions of these properties will be added later.
The following properties describe the type of input that was used to generate the results.
storm class : TS
storm name : KAREN
stormnumber : 12
stormname : KAREN
year : 2013
storm : 12
modified : n
track_raw_dat : bal122013.dat
track_raw_fst : al122013.fst
mesh : sl15_2010_HSDRRS_2012_v9
RunType : Forecast
ADCIRCgrid : sl15_2010_HSDRRS_2012_v9
advisory : 10
currentcycle :
currentdate :
Definitions
- RunStartTime
- Date and time in yyyyMMDDHH24 format, generally assumed to be in the GMT time zone unless otherwise indicated. It represents the time of the most recent hotstart file, that is, the date and time at the end of the most recent nowcast or hindcast period. It does not represent the start of the forecast period, or the time that the forecast was issued.
- currentcycle
- currentdate
-
Install the THREDDS application on a publically accessible computer and configure a catalog for ASGS the XML configuration file for the server;
-
Populate the THREDDS/ASGS server with model output into a defined and required directory hierarchy; and
-
Generate a catalog file that describes the contents on the THREDDS server.
- This two digit hour is in a 24 hour format and padded with a leading
zero. This hour is equal to the hour from the RunStartTime value.
- The end date of the most recent nowcast or hindcast period, including
the two digit year, month, and day. The value of this property
is the same as year, month, and day from the RunStartTime value.
The following properties indicate the presence, name, and format of the output (results) files.
Water Surface Elevation Stations File Name : fort.61.nc
Water Surface Elevation Stations Format : netcdf
Water Surface Elevation File Name : fort.63.nc
Water Surface Elevation Format : netcdf
Barometric Pressure Stations File Name : fort.71.nc
Barometric Pressure Stations Format : netcdf
Wind Velocity Stations File Name : fort.72.nc
Wind Velocity Stations Format : netcdf
Barometric Pressure File Name : fort.73.nc
Barometric Pressure Format : netcdf
Wind Velocity File Name : fort.74.nc
Wind Velocity Format : netcdf
Maximum Water Surface Elevation File Name : maxele.63.nc
Maximum Water Surface Elevation Format : netcdf
Maximum Current Speed File Name : maxvel.63
Maximum Current Speed Format : ascii
Maximum Wind Speed File Name : maxwvel.63.nc
Maximum Wind Speed Format : netcdf
Minimum Barometric Pressure File Name : minpr.63.nc
Minimum Barometric Pressure Format : netcdf
The following properties provide information for use by particular downstream applications for uses ranging from load balancing, presentation to different audiences, labeling results, etc.
ceraServer : cera0
asgs : nc
intendedAudience :
downloadurl :
prodID : SADCsl15_2010_HSDRRS_2012_v9-UNC_vortex-nws19_20131005T1200_20131005T1200_20131007T1200_12<field>_Z.nc.gz
The following properties provide key bits of data that describe how the results were generated.
InitialHotStartTime : 4276800.00000000
RunStartTime : 2013100512
RunEndTime : 2013100712
ColdStartTime : 2013081700
WindModel : vortex-nws19
Model : PADCIRC
The following properties provide data about the High Performance Computing (HPC) resources that were involved in the production of results, which may be used in troubleshooting or performance analysis.
directory storm : /projects/ncfs/data/asgs15784/10/nhcConsensus
hostname : hatteras.renci.org
instance : corpsbackup3
forecastEnsembleMemberNumber : 0
cpurequest : 480
ncpu : 480
numwriters : 0
To facilitate the publication, distribution, and access of ASGS results to dowstream applications, we are using THREDDS/OPeNDAP as a data access protocol between the ASGS instance that is generating results and any downstream applications that provide further analysis and visualization. Unidata's THREDDS DataServer (TDS) is a web server that provides metadata and data sccess for scientific dadtasets using several remote access protocols. One such protocol is OPeNDAP. As such, the configuration details of this service are critical to reliable compatibility with downstream applications. OPeNDAP can be used directly in applications to retrieve data, as through a service request to a TDS, or as implemented for example in the MATLAB toolbox nctoolbox. Capabilities and details of using TDS and OPeNDAP are available at http://www.unidata.ucar.edu/software/thredds/current/tds/ and http://www.opendap.org/
There are three main steps to the setup of THREDDS for ASGS:
Each of these steps are briefly described below. However, a detailed and up-to-date description of the setup and configuration of a TDS are fully described at the following URL, because there is no requirement that files posted in the manner described originate from ADCIRC. Any non-rectangular model that conforms to the data format and metadata specifications can be served this way.
https://docs.google.com/document/d/1iHHf4FN1ivGe039l_L0StuHB7d2VveAuzQsz8HVUkHA/pub
We use a Unidata THREDDS Data Server to serve ASGS output and the underlying web server for a THREDDS Data Server (TDS) is tomcat5, not apache. If tomcat5 is not running on the file server, install this first. Instructions are at:
http://www.unidata.ucar.edu/projects/THREDDS/tech/tds4.3/tutorial/index.html http://www.unidata.ucar.edu/projects/THREDDS/tech/tds4.3/tutorial/Checklist.html
The THREDDS app will be under the <$tomcat5>/webapps location, and its catalog configuration files are in <$tomcat5>/content/thredds. OPeNDAP is a java class under the THREDDS server.
Edit <$tomcat5>/content/thredds/threddsConfig.xml for the local installation.
Once the TDS is running, create a catalog for ASGS output. The xml file <$tomcat5>/content/thredds/catalog.xml is the root catalog for TDS configuration. An example of this root catalog is shown below, where three services are enabled; OPeNDAP, HTTP, and UDDC. The catalogRef tag points to the asgs.xml file, located in the same directory and an example is listed below.
<?xml version="1.0" encoding="UTF-8"?>
<catalog name="ASGS THREDDS DATA SERVER"
xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"
xmlns:xlink="http://www.w3.org/1999/xlink">
<service name="all" base="" serviceType="compound">
<service name="odap" serviceType="OpenDAP" base="/thredds/dodsC/" />
<service name="http" serviceType="HTTPServer" base="/thredds/fileServer/" />
<service name="uddc" serviceType="UDDC" base="/thredds/uddc/"/>
<!-- test comment -->
<!--service name="wcs" serviceType="WCS" base="/thredds/wcs/" /-->
<!--service name="wms" serviceType="WMS" base="/thredds/wms/" /-->
<!--service name="ncss" serviceType="NetcdfSubset" base="/thredds/ncss/" /-->
</service>
<catalogRef xlink:title="ASGS Output" xlink:href="asgs.xml" name=""/>
</catalog>
Example asgs.xml catalogRef file:
<?xml version="1.0" encoding="UTF-8"?>
<catalog name="ADCIRC/PADCSWAN Results from RENCI"
xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"
xmlns:xlink="http://www.w3.org/1999/xlink">
<service name="all" base="" serviceType="compound">
<service name="odap" serviceType="OpenDAP" base="/thredds/dodsC/" />
<service name="http" serviceType="HTTPServer" base="/thredds/fileServer/" />
<service name="uddc" serviceType="UDDC" base="/thredds/uddc/"/>
<service name="uddc" serviceType="ISO" base="/thredds/iso/"/>
</service>
<datasetScan
name="ASGS"
ID="ASGSTCResults"
path="ASGS"
location="/projects/ncfs/opendap/data/tc">
<serviceName>all</serviceName>
<addDatasetSize/>
<metadata inherited="false">
<documentation type="summary">
Results produced by the ASGS system at MyLocation
</documentation>
</metadata>
<filter>
<include wildcard="*"/>
<exclude wildcard="LOGS" atomic="false" collection="true"/>
</filter>
</datasetScan>
</catalog>
The results files from the ASGS are posted in a particular directory hierarchy that enables us to have multiple instances of the ASGS on multiple machines producing results for multiple ensemble members on multiple meshes.
Put model output files into the following directory structure on the THREDDS server (ASGS can be configured to do this):
<$SP>/StormName/AdvisoryNumber/AdcircGrid/Machine/Instance/EnsembleName
where $SP is the starting/entry point into the THREDDS catalog. Here is an example of a fully qualified URL to an ADCIRC output file on RENCI’s THREDDS server:
http://opendap.renci.org:1935/thredds/dodsC/tc/nam/2013052300/nc6b/blueridge.renci.org/2/namforecast
There is actually no requirement on these directory names, except that they exist. In particular, the Instance can be used as a tag to denote specific instances of ASGS, such as “NOAATestRuns”. A run.properties file must accompany each ensemble member simulation. This file is automatically generated by ASGS, but can generated by some other process as needed, if the simulations are not being carried out by an ASGS instance.
There is, unfortunately, no way to interrogate a thredds server to get a text file that describes the data content of the server. Methods are being developed that will do this for a THREDDS server, but these methods are not ready for production use. In lieu of this, RENCI has developed an automated cataloging script that allows a url-retrieve of the catalog file for each ASGS TDS. This catalog generator can be run periodically in cron to maintain an up-to-date listing of the TDS content.
The current version of the catalog file generator is described in detail at this URL:
https://docs.google.com/document/d/1iHHf4FN1ivGe039l_L0StuHB7d2VveAuzQsz8HVUkHA/pub
Adherence to community-defined, accepted and adopted conventions for data formats and metadata content are critical for model interoperbility, workflow development, and end-user applications. To facilitate wide accessibility to ADCIRC output through THREDDS Data Servers, ADCIRC/ASGS outputs netCDF files (optionally specified in the fort.15 file). These files must also have metadata (global, dimension, and variable attributes) that conform to NetCDF Climate and Forecast (CF) Metadata Conventions with extensions for non-rectangular grid models (CF-UGRID).
NOTE: Make sure that ADCIRC and ADCPREP are compiled with netCDF4 support with compression.
It is critical for ADCIRC netCDF4 files to maintain strict CF-UGRID compliance. This extension to CF conventions allows applications to discover, access, and process data in uniform ways and without having to make assumptions about the content of the data files. This is the default for ASGS-generated output.
There is no technical requirement that the files be generated by ADCIRC. As long as files are posted that are netCDF CF-UGRID compliant, any application that can handle ADCIRC files can also handle other model files, including SLOSH when properly formatted.
Information on CF and CF-UGRID can be found at:
http://cf-pcmdi.llnl.gov/documents/cf-conventions
https://github.com/ugrid-conventions/ugrid-conventions
Description of the relevant parts of the CF standards that must be implemented in the ADCIRC and ADCIRC+SWAN NetCDF4 output files for compatibility with downstream applications.
http://cf-pcmdi.llnl.gov/documents/cf-conventions
Any other key pieces of NetCDF metadata that are used by downstream applications and must be provided for compatibility.
This must include a time stamp indicating the cold start date.
'** JASON: I don't see where this is in the metadata. I see that there is a "cs:" in the fort.15 header, and as a consequence its in the global attribute "description", but I don't see it as an attribute. I think this is a good idea, though, to put it as a global attribute in each file. **'
This document was prepared from the text file ASGSInterfaceGuide.txt using software called asciidoc (http://www.methods.co.nz/asciidoc/). The document can be formatted as an html page with the command
asciidoc --backend html5 -a toc2 ASGSInterfaceGuide.txt
or formatted as a pdf with the command
a2x --format=pdf -a toc2 ASGSInterfaceGuide.txt