weather uses two timezones - not clear which matches flights

[garrettgman]

In weather, the time_hour variable is offset by five hours from the time displayed across the year, month, day, and hour variables.

It is not clear which time matches the times in flights (where year, month, day, hour, and time_hour all agree). Given the offset, it is possible that time_hour is in the America/New_York timezone and the other variables are in UTC.

[rmcd1024]

The weather data is in UTC, which I think is standard. If you look at the code generating "time_hour" in weather.r, the time zone is not specified, which makes me think that it acquires whatever local time zone offset was in place when the package was built (from ?ISOdatetime: " ‘""’ is the current time zone")

[ltierney]

It looks like year/month/day/hour are in America/New_York in flights and in UTC in weather. That doesn't match time_hour, which is America/Chicago for weather and UTC for flights. These don't match up properly. Recalculating the time_hour variables as

flights <- mutate(flights, time_hour = make_datetime(year, month, day, hour, tz = "America/New_York"))
weather <- mutate(weather, time_hour = make_datetime(year, month, day, hour, tz = "UTC"))

produces values that seem to match up properly on a join.

[rudeboybert]

Let's put everything on the table with the following reprex:

# devtools::install_github("hadley/nycflights13")
suppressPackageStartupMessages(library(tidyverse))
suppressPackageStartupMessages(library(nycflights13))

flights %>% 
  select(year, month, day, hour, time_hour) %>% 
  slice(1)
#> # A tibble: 1 x 5
#>    year month   day  hour time_hour          
#>   <int> <int> <int> <dbl> <dttm>             
#> 1  2013     1     1     5 2013-01-01 05:00:00
flights$time_hour[1]
#> [1] "2013-01-01 05:00:00 UTC"

weather %>% 
  select(year, month, day, hour, time_hour) %>% 
  slice(1)
#> # A tibble: 1 x 5
#>    year month   day  hour time_hour          
#>   <dbl> <dbl> <int> <int> <dttm>             
#> 1  2013     1     1     0 2012-12-31 19:00:00
weather$time_hour[1]
#> [1] "2012-12-31 19:00:00 EST"

• flights
  • year/month/day/hour match time_hour's date/time, but the latter's time zone is off.
• weather
  • year/month/day/hour is ahead of time_hour's time by 5 hours.
  • At the very least, the timezone output should be of form 20XX-XX-XX XX:XX:XX America/New_York and not 20XX-XX-XX XX:XX:XX EST, as per @rmcd1024's comment in Fixed weather$time_hour timezone and added to documentation for fligh… #23 on possible Eastern Standard Time (EST) vs Eastern Daylight Time (EDT) confusion.
  • Based on the visualization below of June EWR temperatures and a cursory scientific investigation suggesting that 3pm is the hottest time of the day in the summer, as @ltierney suggested,year/month/day/hour do appear to be in UTC/shifted 4 to 5 hours, suggesting that these 4 variables need to be corrected all weather measurements need to be shifted to match the correct year/month/day/hour.

weather %>% 
  filter(origin == "EWR", month == 6) %>%
  ggplot(aes(x = hour, y = temp)) +
  geom_point() +
  geom_smooth() +
  labs(title = "June 2013 hourly temperatures at EWR") + 
  geom_vline(xintercept = 15, col = "red", size = 1)
#> `geom_smooth()` using method = 'loess' and formula 'y ~ x'

Created on 2018-06-06 by the reprex package (v0.2.0).

Question for @hadley: What do you think the timezone/output of flights$time_hour[1] should be? (The same idea applies to weather$time_hour[1])

1. 2013-01-01 10:00:00 UTC. @rmcd1024 favors this as it is industry convention.
2. 2013-01-01 05:00:00 America/New_York. I favor this as it will cause users of this package less confusion and makes joining easier.

[hadley]

I think both the time zones for both datasets should be America/New_York since this makes joining and interpretation easier.

@rudeboybert do you want to apply @ltierney's fix in your PR? Or should we close that one and start anew?

[rudeboybert]

I'll close #23 and start anew sometime next week.

[hadley]

library(nycflights13)

flights[1, c("year", "month", "day", "hour", "time_hour")]
#> # A tibble: 1 x 5
#>    year month   day  hour time_hour          
#>   <int> <int> <int> <dbl> <dttm>             
#> 1  2013     1     1     5 2013-01-01 05:00:00
flights$time_hour[1]
#> [1] "2013-01-01 05:00:00 EST"
attr(flights$time_hour, "tzone")
#> [1] "America/New_York"

weather[1, c("year", "month", "day", "hour", "time_hour")]
#> # A tibble: 1 x 5
#>    year month   day  hour time_hour          
#>   <dbl> <dbl> <int> <int> <dttm>             
#> 1  2013     1     1     1 2013-01-01 01:00:00
weather$time_hour[1]
#> [1] "2013-01-01 01:00:00 EST"
attr(weather$time_hour, "tzone")
#> [1] "America/New_York"

Created on 2018-06-20 by the reprex package (v0.2.0).

[hadley]

I'll plan to submit to CRAN in one week (July 27), so I'd really appreciate it if someone could double check my work and let me know if I've missed anything

[rudeboybert]

Looks good on my end. Thanks!

suppressPackageStartupMessages(library(tidyverse))
suppressPackageStartupMessages(library(nycflights13))

# Correct EST Standard vs EDT Daylight Savings (2013-03-10 thru 2013-11-03) for weather
weather$time_hour[1]
#> [1] "2013-01-01 01:00:00 EST"
weather$time_hour[13000]
#> [1] "2013-06-29 06:00:00 EDT"

# Correct EST Standard vs EDT Daylight Savings (2013-03-10 thru 2013-11-03) for flights
flights$time_hour[1]
#> [1] "2013-01-01 05:00:00 EST"
flights$time_hour[150000]
#> [1] "2013-03-15 17:00:00 EDT"

# Roughly hottest point of the day corresponds to 3pm
weather %>% 
  filter(origin == "EWR", month == 6) %>%
  ggplot(aes(x = hour, y = temp)) +
  geom_point() +
  geom_smooth() +
  labs(title = "June 2013 hourly temperatures at EWR") + 
  geom_vline(xintercept = 15, col = "red", size = 1)
#> `geom_smooth()` using method = 'loess' and formula 'y ~ x'

Created on 2018-06-20 by the reprex package (v0.2.0).

[rmcd1024]

This looks good to me, thanks for revising this.