Skip to content

Commit

Permalink
fix #189 ghcndms and annual changed to gsom and gsoy, bump dev ver
Browse files Browse the repository at this point in the history
  • Loading branch information
sckott committed Feb 15, 2017
1 parent 110cd05 commit 7801785
Show file tree
Hide file tree
Showing 21 changed files with 532 additions and 460 deletions.
2 changes: 1 addition & 1 deletion DESCRIPTION
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ Description: Client for many 'NOAA' data sources including the 'NCDC' climate
for 'NOAA' sea ice data, the 'NOAA' severe weather inventory, 'NOAA' Historical
Observing 'Metadata' Repository ('HOMR') data, 'NOAA' storm data via 'IBTrACS',
tornado data via the 'NOAA' storm prediction center, and more.
Version: 0.6.7.9190
Version: 0.6.7.9194
License: MIT + file LICENSE
Encoding: UTF-8
Authors@R: c(
Expand Down
165 changes: 93 additions & 72 deletions R/ncdc.r
Original file line number Diff line number Diff line change
Expand Up @@ -4,28 +4,34 @@
#' @template rnoaa
#' @template noaa
#' @template token
#' @param stationid Accepts a valid station id or a vector or list of station ids
#' @param includemetadata Used to improve response time by preventing the calculation of
#' result metadata. Default: TRUE. This does not affect the return object, in that the named part
#' of the output list called "meta' is still returned, but is NULL. In practice, I haven't seen
#' response time's improve, but perhaps they will for you.
#' @param stationid Accepts a valid station id or a vector or list of station
#' ids
#' @param includemetadata Used to improve response time by preventing the
#' calculation of result metadata. Default: TRUE. This does not affect the
#' return object, in that the named part of the output list called "meta"
#' is still returned, but is NULL. In practice, I haven't seen response
#' time's improve, but perhaps they will for you.
#'
#' @details
#' Note that NOAA NCDC API calls can take a long time depending on the call. The NOAA API doesn't
#' perform well with very long timespans, and will time out and make you angry - beware.
#'
#' Keep in mind that three parameters, datasetid, startdate, and enddate are required.
#'
#' Note that the default limit (no. records returned) is 25. Look at the metadata in \code{$meta}
#' to see how many records were found. If more were found than 25, you could set the parameter
#' \code{limit} to something higher than 25.
#'
#' The attributes, or "flags", for each row of the output for data may have a flag
#' with it. Each \code{datasetid} has it's own set of flags. The following are flag
#' columns, and what they stand for. \code{fl_} is the beginning of each flag
#' column name, then one or more characters to describe the flag, keeping it short
#' to maintain a compact data frame. Some of these fields are the same across
#' datasetids. See the vignette \code{vignette("rnoaa_attributes", "rnoaa")} for
#' Note that NOAA NCDC API calls can take a long time depending on the call.
#' The NOAA API doesn't perform well with very long timespans, and will
#' time out and make you angry - beware.
#'
#' Keep in mind that three parameters, datasetid, startdate, and enddate
#' are required.
#'
#' Note that the default limit (no. records returned) is 25. Look at the
#' metadata in \code{$meta} to see how many records were found. If more were
#' found than 25, you could set the parameter \code{limit} to
#' something higher than 25.
#'
#' The attributes, or "flags", for each row of the output for data may have
#' a flag with it. Each \code{datasetid} has it's own set of flags. The
#' following are flag columns, and what they stand for. \code{fl_} is the
#' beginning of each flag column name, then one or more characters to describe
#' the flag, keeping it short to maintain a compact data frame. Some of
#' these fields are the same across datasetids. See the vignette
#' \code{vignette("rnoaa_attributes", "rnoaa")} for
#' description of possible values for each flag.
#'
#' \itemize{
Expand All @@ -40,16 +46,16 @@
#' \item fl_u units
#' }
#'
#' @return An S3 list of length two, a slot of metadata (meta), and a slot for data (data).
#' The meta slot is a list of metadata elements, and the data slot is a data.frame,
#' possibly of length zero if no data is found.
#' @return An S3 list of length two, a slot of metadata (meta), and a slot
#' for data (data). The meta slot is a list of metadata elements, and the
#' data slot is a data.frame, possibly of length zero if no data is found.
#'
#' @family ncdc
#'
#' @examples \dontrun{
#' # GHCN-Daily (or GHCND) data, for a specific station
#' ncdc(datasetid='GHCND', stationid='GHCND:USW00014895', startdate = '2013-10-01',
#' enddate = '2013-12-01')
#' ncdc(datasetid='GHCND', stationid='GHCND:USW00014895',
#' startdate = '2013-10-01', enddate = '2013-12-01')
#'
#' # GHCND data, for a location by FIPS code
#' ncdc(datasetid='GHCND', locationid = 'FIPS:02', startdate = '2010-05-01',
Expand All @@ -58,15 +64,18 @@
#' # GHCND data from October 1 2013 to December 1 2013
#' ncdc(datasetid='GHCND', startdate = '2013-10-01', enddate = '2013-10-05')
#'
#' # GHCN-Monthly (or GHCNDMS) data from October 1 2013 to December 1 2013
#' ncdc(datasetid='GHCNDMS', startdate = '2013-10-01', enddate = '2013-12-01')
#' # GHCN-Monthly (or GSOM) data from October 1 2013 to December 1 2013
#' ncdc(datasetid='GSOM', startdate = '2013-10-01', enddate = '2013-12-01')
#' ncdc(datasetid='GSOM', startdate = '2013-10-01', enddate = '2013-12-01',
#' stationid = "GHCND:AE000041196")
#'
#' # Normals Daily (or NORMAL_DLY) GHCND:USW00014895 dly-tmax-normal data
#' ncdc(datasetid='NORMAL_DLY', stationid='GHCND:USW00014895', startdate = '2010-05-01',
#' enddate = '2010-05-10')
#' ncdc(datasetid='NORMAL_DLY', stationid='GHCND:USW00014895',
#' startdate = '2010-05-01', enddate = '2010-05-10')
#'
#' # Dataset, and location in Australia
#' ncdc(datasetid='GHCND', locationid='FIPS:AS', startdate = '2010-05-01', enddate = '2010-05-31')
#' ncdc(datasetid='GHCND', locationid='FIPS:AS', startdate = '2010-05-01',
#' enddate = '2010-05-31')
#'
#' # Dataset, location and datatype for PRECIP_HLY data
#' ncdc(datasetid='PRECIP_HLY', locationid='ZIP:28801', datatypeid='HPCP',
Expand All @@ -81,52 +90,55 @@
#' startdate = '2010-05-01', enddate = '2010-05-10')
#'
#' # Dataset, location, station and datatype
#' ncdc(datasetid='PRECIP_HLY', locationid='ZIP:28801', stationid='COOP:310301', datatypeid='HPCP',
#' ncdc(datasetid='PRECIP_HLY', locationid='ZIP:28801',
#' stationid='COOP:310301', datatypeid='HPCP',
#' startdate = '2010-05-01', enddate = '2010-05-10')
#'
#' # Dataset, location, and datatype for GHCND
#' ncdc(datasetid='GHCND', locationid='FIPS:BR', datatypeid='PRCP', startdate = '2010-05-01',
#' enddate = '2010-05-10')
#' ncdc(datasetid='GHCND', locationid='FIPS:BR', datatypeid='PRCP',
#' startdate = '2010-05-01', enddate = '2010-05-10')
#'
#' # Normals Daily GHCND dly-tmax-normal data
#' ncdc(datasetid='NORMAL_DLY', datatypeid='dly-tmax-normal', startdate = '2010-05-01',
#' enddate = '2010-05-10')
#' ncdc(datasetid='NORMAL_DLY', datatypeid='dly-tmax-normal',
#' startdate = '2010-05-01', enddate = '2010-05-10')
#'
#' # Normals Daily GHCND:USW00014895 dly-tmax-normal
#' ncdc(datasetid='NORMAL_DLY', stationid='GHCND:USW00014895', datatypeid='dly-tmax-normal',
#' ncdc(datasetid='NORMAL_DLY', stationid='GHCND:USW00014895',
#' datatypeid='dly-tmax-normal',
#' startdate = '2010-05-01', enddate = '2010-05-10')
#'
#' # Hourly Precipitation data for ZIP code 28801
#' ncdc(datasetid='PRECIP_HLY', locationid='ZIP:28801', datatypeid='HPCP',
#' startdate = '2010-05-01', enddate = '2010-05-10')
#'
#' # 15 min Precipitation data for ZIP code 28801
#' ncdc(datasetid='PRECIP_15', datatypeid='QPCP', startdate = '2010-05-01', enddate = '2010-05-02')
#' ncdc(datasetid='PRECIP_15', datatypeid='QPCP',
#' startdate = '2010-05-01', enddate = '2010-05-02')
#'
#' # Search the NORMAL_HLY dataset
#' ncdc(datasetid='NORMAL_HLY', stationid = 'GHCND:USW00003812', startdate = '2010-05-01',
#' enddate = '2010-05-10')
#' ncdc(datasetid='NORMAL_HLY', stationid = 'GHCND:USW00003812',
#' startdate = '2010-05-01', enddate = '2010-05-10')
#'
#' # Search the ANNUAL dataset
#' # Search the GSOY dataset
#' ncdc(datasetid='ANNUAL', locationid='ZIP:28801', startdate = '2010-05-01',
#' enddate = '2010-05-10')
#'
#' # Search the NORMAL_ANN dataset
#' ncdc(datasetid='NORMAL_ANN', datatypeid='ANN-DUTR-NORMAL', startdate = '2010-01-01',
#' enddate = '2010-01-01')
#' ncdc(datasetid='NORMAL_ANN', datatypeid='ANN-DUTR-NORMAL',
#' startdate = '2010-01-01', enddate = '2010-01-01')
#'
#' # Include metadata or not
#' ncdc(datasetid='GHCND', stationid='GHCND:USW00014895', startdate = '2013-10-01',
#' enddate = '2013-12-01')
#' ncdc(datasetid='GHCND', stationid='GHCND:USW00014895', startdate = '2013-10-01',
#' enddate = '2013-12-01', includemetadata=FALSE)
#' ncdc(datasetid='GHCND', stationid='GHCND:USW00014895',
#' startdate = '2013-10-01', enddate = '2013-12-01')
#' ncdc(datasetid='GHCND', stationid='GHCND:USW00014895',
#' startdate = '2013-10-01', enddate = '2013-12-01', includemetadata=FALSE)
#'
#' # Many stationid's
#' stat <- ncdc_stations(startdate = "2000-01-01", enddate = "2016-01-01")
#' ## find out what datasets might be available for these stations
#' ncdc_datasets(stationid = stat$data$id[1])
#' ## get some data
#' ncdc(datasetid = "ANNUAL", stationid = stat$data$id[1:10],
#' ncdc(datasetid = "GSOY", stationid = stat$data$id[1:10],
#' startdate = "2010-01-01", enddate = "2011-01-01")
#' }
#'
Expand All @@ -144,13 +156,15 @@ ncdc <- function(datasetid=NULL, datatypeid=NULL, stationid=NULL, locationid=NUL
{
calls <- names(sapply(match.call(), deparse))[-1]
calls_vec <- c("dataset","datatype","station","location","locationtype","page","year","month","day","results") %in% calls
if(any(calls_vec))
if (any(calls_vec)) {
stop("The parameters name, code, modifiedsince, startindex, and maxresults \n have been removed, and were only relavant in the old NOAA API v1. \n\nPlease see documentation for ?noaa")
}

token <- check_key(token)
args <- noaa_compact(list(datasetid=datasetid, startdate=startdate,
enddate=enddate, sortfield=sortfield, sortorder=sortorder,
limit=limit, offset=offset, includemetadata=includemetadata))
args <- noaa_compact(list(datasetid = datasetid, startdate = startdate,
enddate = enddate, sortfield = sortfield,
sortorder = sortorder, limit = limit,
offset = offset, includemetadata = includemetadata))
if (!is.null(stationid)) {
stationid <- lapply(stationid, function(x) list(stationid = x))
}
Expand All @@ -164,36 +178,43 @@ ncdc <- function(datasetid=NULL, datatypeid=NULL, stationid=NULL, locationid=NUL
args <- as.list(unlist(args))
names(args) <- gsub("[0-9]+", "", names(args))
if (length(args) == 0) args <- NULL
temp <- GET(paste0(ncdc_base(), "data"), query=args, add_headers("token" = token), ...)
temp <- GET(paste0(ncdc_base(), "data"), query = args,
add_headers("token" = token), ...)
tt <- check_response(temp)
if(inherits(tt, "character")){
all <- list(meta=NA, data=NA)
if (inherits(tt, "character")) {
all <- list(meta = NA, data = NA)
} else {
tt$results <- lapply(tt$results, split_atts, ds=datasetid)
dat <- dplyr::bind_rows(lapply(tt$results, function(x) data.frame(x,stringsAsFactors=FALSE)))
tt$results <- lapply(tt$results, split_atts, ds = datasetid)
dat <- dplyr::bind_rows(lapply(tt$results, function(x)
data.frame(x,stringsAsFactors = FALSE)))
meta <- tt$metadata$resultset
atts <- list(totalCount=meta$count, pageCount=meta$limit, offset=meta$offset)
all <- list(meta=atts, data=dat)
atts <- list(totalCount = meta$count, pageCount = meta$limit,
offset = meta$offset)
all <- list(meta = atts, data = dat)
}

structure(all, class="ncdc_data")
structure(all, class = "ncdc_data")
}

split_atts <- function(x, ds="GHCNDMS"){
split_atts <- function(x, ds = "GSOM"){
tmp <- x$attributes
out <- switch(ds,
ANNUAL = parse_ncdc(tmp,c('fl_m','fl_q','fl_d','fl_u')),
GHCND = parse_ncdc(tmp,c('fl_m','fl_q','fl_so','fl_t')),
GHCNDMS = parse_ncdc(tmp,c('fl_miss','fl_cmiss')),
NEXRAD2 = parse_ncdc(tmp,c('x','x')), # no data returned, fix when data returned
NEXRAD3 = parse_ncdc(tmp,c('x','x')), # no data returned, fix when data returned
NORMAL_ANN = parse_ncdc(tmp,'fl_c'),
NORMAL_DLY = parse_ncdc(tmp,'fl_c'),
NORMAL_HLY = parse_ncdc(tmp,'fl_c'),
NORMAL_MLY = parse_ncdc(tmp,'fl_c'),
PRECIP_15 = parse_ncdc(tmp,c('fl_m','fl_q','fl_u')),
PRECIP_HLY = parse_ncdc(tmp,c('fl_m','fl_q')))
notatts <- x[!names(x)=="attributes"]
GHCND = parse_ncdc(tmp, c('fl_m','fl_q','fl_so','fl_t')),
# leave in duplicate for now
GHCNDMS = parse_ncdc(tmp, c('fl_miss','fl_cmiss')),
GSOM = parse_ncdc(tmp, c('fl_miss','fl_cmiss')),
# leave in duplicate for now
ANNUAL = parse_ncdc(tmp, c('fl_m','fl_q','fl_d','fl_u')),
GSOY = parse_ncdc(tmp, c('fl_m','fl_q','fl_d','fl_u')),
NEXRAD2 = parse_ncdc(tmp, c('x','x')), # no data returned, fix when data returned
NEXRAD3 = parse_ncdc(tmp, c('x','x')), # no data returned, fix when data returned
NORMAL_ANN = parse_ncdc(tmp, 'fl_c'),
NORMAL_DLY = parse_ncdc(tmp, 'fl_c'),
NORMAL_HLY = parse_ncdc(tmp, 'fl_c'),
NORMAL_MLY = parse_ncdc(tmp, 'fl_c'),
PRECIP_15 = parse_ncdc(tmp, c('fl_m','fl_q','fl_u')),
PRECIP_HLY = parse_ncdc(tmp, c('fl_m','fl_q')))
notatts <- x[!names(x) == "attributes"]
c(notatts, out)
}

Expand Down
1 change: 1 addition & 0 deletions R/ncdc_datasets.r
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@
#' # Multiple datatypeid's
#' ncdc_datasets(datatypeid=c('ACMC','ACMH','ACSC'))
#' ncdc_datasets(datasetid='ANNUAL', datatypeid=c('ACMC','ACMH','ACSC'))
#' ncdc_datasets(datasetid='GSOY', datatypeid=c('ACMC','ACMH','ACSC'))
#'
#' # Multiple locationid's
#' ncdc_datasets(locationid="FIPS:30091")
Expand Down
3 changes: 3 additions & 0 deletions R/ncdc_datatypes.r
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,10 @@
#' ncdc_datatypes(datatypeid="ACSC")
#'
#' # datasetid, one or many
#' ## ANNUAL should be replaced by GSOY, but both exist and give
#' ## different answers
#' ncdc_datatypes(datasetid="ANNUAL")
#' ncdc_datatypes(datasetid="GSOY")
#' ncdc_datatypes(datasetid=c("ANNUAL", "PRECIP_HLY"))
#'
#' # Fetch data types with the air temperature data category
Expand Down
3 changes: 2 additions & 1 deletion R/ncdc_locs.r
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,8 @@
#' # Fetch available locations for the GHCND (Daily Summaries) dataset
#' ncdc_locs(datasetid='GHCND')
#' ncdc_locs(datasetid=c('GHCND', 'ANNUAL'))
#' ncdc_locs(datasetid=c('GHCND', 'GHCNDMS'))
#' ncdc_locs(datasetid=c('GSOY', 'ANNUAL'))
#' ncdc_locs(datasetid=c('GHCND', 'GSOM'))
#'
#' # Fetch all U.S. States
#' ncdc_locs(locationcategoryid='ST', limit=52)
Expand Down
4 changes: 3 additions & 1 deletion R/ncdc_locs_cats.r
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,11 @@
#'
#' # Displays available location categories within GHCN-Daily dataset
#' ncdc_locs_cats(datasetid='GHCND')
#' ncdc_locs_cats(datasetid='GSOY')
#' ncdc_locs_cats(datasetid='ANNUAL')
#'
#' # multiple datasetid's
#' ncdc_locs_cats(datasetid=c('GHCND', 'GHCNDMS'))
#' ncdc_locs_cats(datasetid=c('GHCND', 'GSOM'))
#'
#' # Displays available location categories from start date 1970-01-01
#' ncdc_locs_cats(startdate='1970-01-01')
Expand Down
5 changes: 3 additions & 2 deletions R/ncdc_stations.r
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,9 @@
#' }$data)
#'
#' # Displays all stations within GHCN-Daily (100 Stations per page limit)
#' ncdc_stations(datasetid='GHCND')
#' ncdc_stations(datasetid=c('GHCND', 'ANNUAL'))
#' ncdc_stations(datasetid = 'GHCND')
#' ncdc_stations(datasetid = 'ANNUAL')
#' ncdc_stations(datasetid = 'GSOY')
#'
#' # Station
#' ncdc_stations(datasetid='NORMAL_DLY', stationid='GHCND:USW00014895')
Expand Down
Loading

0 comments on commit 7801785

Please sign in to comment.