Can not download hourly precipitation data

[tungttnguyen]

Hi there,

I failed to download the hourly precipitation data for the period I wanted. rnoaa told me that no data found. However, I could download the data just fine from the website
https://gis.ncdc.noaa.gov/map/viewer/#app=cdo&cfg=cdo&theme=hourly&layers=1

This is what I tried so far

library(rnoaa)

Download hourly data for PULLMAN MOSCOW REGIONAL AIRPORT, WA US

PUW <- ncdc_stations(extent=c(46.73,-117.1384,46.758,-117.09))
PUW

tmp <- ncdc(datasetid = 'PRECIP_HLY', stationid = 'WBAN:94129',
startdate = '2006-10-01', enddate = '2007-09-30')
Warning message:
In check_response(temp) : Sorry, no data found

tmp <- ncdc(datasetid = 'PRECIP_HLY', stationid = 'GHCND:USW00094129',
startdate = '2006-10-01', enddate = '2007-09-30')
Warning message:
In check_response(temp) : Sorry, no data found

Any ideas why it didn't work? Thanks!

By the way, is it possible to download 15-min precipitation data from NCDC using rnoaa?

[sckott]

@madtraveller thanks for your question! looking at it now...

[sckott]

@madtraveller 15 min and 5 min data isn't available as far as I know in the NCDC V2 API service, which the ncdc_*() functions wrap in rnoaa. Those data I believe are available in the legacy NCDC web service, which I've started working on, see #54 and some functions started ncdc_leg_variables() and ncdc_leg_sites(), I don't think i made a function to get actual data yet though.

as far as I can tell there is no PRECIP_HLY data for those stations, but there are other data sets available there.

(PUW <- ncdc_stations(extent=c(46.73,-117.1384,46.758,-117.09)))
ncdc_datasets(stationid = PUW$data$id[1])

$meta
$meta$limit
[1] 25

$meta$count
[1] 5

$meta$offset
[1] 1


$data
                   uid         id                    name datacoverage    mindate    maxdate
1 gov.noaa.ncdc:C00861      GHCND         Daily Summaries            1 1763-01-01 2014-11-16
2 gov.noaa.ncdc:C00841    GHCNDMS       Monthly Summaries            1 1763-01-01 2014-10-01
3 gov.noaa.ncdc:C00821 NORMAL_ANN Normals Annual/Seasonal            1 2010-01-01 2010-01-01
4 gov.noaa.ncdc:C00823 NORMAL_DLY           Normals Daily            1 2010-01-01 2010-12-31
5 gov.noaa.ncdc:C00822 NORMAL_MLY         Normals Monthly            1 2010-01-01 2010-12-01

attr(,"class")
[1] "ncdc_datasets"

[tungttnguyen]

Thank you very much for the reply @sckott ! Looking forward to the new update!

[sckott]

@madtraveller I added function for getting data ncdc_leg_data(), see examples here https://github.com/ropensci/rnoaa/blob/master/R/ncdc_legacy.R

Note that this service is pretty hard to work with since you have to wait 60 seconds between each successive data request.

I'll work on integrating functions for getting data via their FTP server since that shouldn't limit requests so dramatically.

[sckott]

@madtraveller I started function isd() and isd_stations() (reinstall the package) to see those, which gets files from the FTP server at ftp://ftp.ncdc.noaa.gov/pub/data/noaa

let me know if that is useful at all. those functions might still be buggy..let me know if you run into problems

[tungttnguyen]

@sckott Thank you! I reinstalled the package but couldn't file the function you mentioned. Would you mind providing a quick example here?

[sckott]

Get stations

stations <- isd_stations()
head(stations)

  usaf  wban station_name ctry state icao lat lon elev_m    begin      end
1 7005 99999   CWOS 07005                  NA  NA     NA 20120127 20120127
2 7011 99999   CWOS 07011                  NA  NA     NA 20111025 20121129
3 7018 99999   WXPOD 7018                   0   0   7018 20110309 20130730
4 7019 99999   CWOS 07019                  NA  NA     NA 20130625 20130703
5 7025 99999   CWOS 07025                  NA  NA     NA 20120127 20120127
6 7026 99999   WXPOD 7026   AF              0   0   7026 20120713 20140923

Get data

(res <- isd(usaf="010230", wban="99999", year=1986))

<path>~/.rnoaa/isd/010230-99999-1986.gz

<ISD Data>
Size: 10406 X 85

   total_chars usaf_station wban_station     date time date_flag latitude
1         0089       010230        99999 19860101 0000         4   +69050
2         0076       010230        99999 19860101 0240         4   +69050
3         0050       010230        99999 19860101 0300         4   +69050
4         0060       010230        99999 19860101 0340         4   +69050
5         0060       010230        99999 19860101 0440         4   +69050
6         0060       010230        99999 19860101 0540         4   +69050
7         0140       010230        99999 19860101 0600         4   +69050
8         0060       010230        99999 19860101 0640         4   +69050
9         0077       010230        99999 19860101 0740         4   +69050
10        0047       010230        99999 19860101 0840         4   +69050
..         ...          ...          ...      ...  ...       ...      ...
Variables not shown: longitude (chr), type_code (chr), elevation (chr),
     call_letter (chr), quality (chr), wind_direction (chr),
     wind_direction_quality (chr), wind_code (chr), wind_speed (chr),
     wind_speed_quality (chr), ceiling_height (chr), ceiling_height_quality
     (chr), ceiling_height_determination (chr), ceiling_height_cavok (chr),
     visibility_distance (chr), visibility_distance_quality (chr),
     visibility_code (chr), visibility_code_quality (chr), temperature (chr),
     temperature_quality (chr), temperature_dewpoint (chr),
     temperature_dewpoint_quality (chr), air_pressure (chr),
     air_pressure_quality (chr), REM.remarks (chr), REM.identifier (chr),
     REM.length_quantity (chr), REM.comment (chr), AY1.manual_occurrence (chr),
     AY1.condition_code (chr), AY1.condition_quality (chr), AY1.period (chr),
     AY1.period_quality (chr), AY2.manual_occurrence (chr), AY2.condition_code
     (chr), AY2.condition_quality (chr), AY2.period (chr), AY2.period_quality
     (chr), AG1.precipitation (chr), AG1.discrepancy (chr), AG1.est_water_depth
     (chr), GF1.sky_condition (chr), GF1.coverage (chr), GF1.opaque_coverage
     (chr), GF1.coverage_quality (chr), GF1.lowest_cover (chr),
     GF1.lowest_cover_quality (chr), GF1.low_cloud_genus (chr),
     GF1.low_cloud_genus_quality (chr), GF1.lowest_cloud_base_height (chr),
     GF1.lowest_cloud_base_height_quality (chr), GF1.mid_cloud_genus (chr),
     GF1.mid_cloud_genus_quality (chr), GF1.high_cloud_genus (chr),
     GF1.high_cloud_genus_quality (chr), MW1.first_weather_reported (chr),
     MW1.condition (chr), MW1.condition_quality (chr),
     EQD.observation_identifier (chr), EQD.observation_text (chr),
     EQD.reason_code (chr), EQD.parameter (chr), MD1.atmospheric_change (chr),
     MD1.tendency (chr), MD1.tendency_quality (chr), MD1.three_hr (chr),
     MD1.three_hr_quality (chr), MD1.twentyfour_hr (chr),
     MD1.twentyfour_hr_quality (chr), KA1.extreme_temp (chr),
     KA1.period_quantity (chr), KA1.max_min (chr), KA1.temp (chr),
     KA1.temp_quality (chr), EQD.observation_identifier.1 (chr),
     EQD.observation_text.1 (chr), EQD.reason_code.1 (chr), EQD.parameter.1
     (chr)

then do stuff with it

library(dplyr)
res$data %>% tbl_df %>% select(date, time, elevation)

Source: local data frame [10,406 x 3]

       date time elevation
1  19860101 0000     +0079
2  19860101 0240     +0079
3  19860101 0300     +0079
4  19860101 0340     +0079
5  19860101 0440     +0079
6  19860101 0540     +0079
7  19860101 0600     +0079
8  19860101 0640     +0079
9  19860101 0740     +0079
10 19860101 0840     +0079
..      ...  ...       ...

Some things to note: The data that comes from that FTP server is super messy. Each line is a long string (see example below) that has usually three letter tags to denote different data types, so there's a bunch of regex behind the scenes to try to parse this stuff - thus it's kind of slow to process this data.

0080011960999991986010106004+68750+017817FM-12+002099999V0201801N00101015001CN0400001N9-00941-01111999999ADDAA106000091AG14000GF107991051001017501031021KA1120N-01021REMSYN011333   21102

[tungttnguyen]

Thanks a lot @sckott . I'll try this one out. Cheers!

[sckott]

closing, i'll reopen if you say there's still a poroblem