module_2_slides.Rmd

---
title: "Module 2 <br><br> Exploring <br><span style='font-size: 66%'>&</span><br> Summarizing Data"
author: ""
# date: "`r Sys.Date()`"
output: 
  xaringan::moon_reader:
    lib_dir: libs
    css: [xaringan-themer.css, other.css]
    nature:
      highlightStyle: tomorrow
      highlightLines: true
      countIncrementalSlides: false
---


<style type="text/css">
.remark-slide-content {
    font-size: 30px;
}
</style>
```{r setup, include=FALSE}
knitr::opts_chunk$set(
  echo=T, 
  eval = F,
  message = F, 
  warning = F, 
  comment = NA,
  R.options=list(width=120), 
  cache.rebuild=F, 
  cache=F,
  fig.align='center', 
  fig.asp = .7,
  dev = 'svg', 
  dev.args=list(bg = 'transparent')
)

library(tidyverse); library(broom); library(kableExtra); library(visibly)

kable_df <- function(..., digits=3) {
  kable(..., digits=digits) %>% 
    kable_styling(full_width = F)
}

rnd = function(x, digits = 3) arm::fround(x, digits = digits)


demographics = read.csv('data/demos_anonymized.csv')
ids = read.csv('data/ids_anonymized.csv')
model_variables = read.csv('data/model_variables_anonymized.csv')
```

## Introduction

---

## Get a Quick Summary

It's always a good idea to glance at the data

Get a feel for the data types

Spot potential issues

```{r summary}
glimpse(demographics)
summary(demographics[,1:10])
```

---

## Grouping and Summarizing Operations

A very common operation is to do things by group(s) then create new summary variables.

```{r group_by, eval=TRUE}
demographics %>% 
  group_by(libuser) %>% 
  summarise(age = mean(age, na.rm = T)) 
```

---

## Grouping and Summarizing Operations

Can do multiple operations at once

```{r group_by2, eval=TRUE}
demographics %>% 
  group_by(libuser) %>% 
  summarise(age_mean = mean(age, na.rm = T),
            age_sd = sd(age, na.rm = T),
            age_max = max(age, na.rm = T),
            prop_male = mean(gender=='Male', na.rm = T)) 
```

---

##  Mapping functions to groups


Use the <span class="pack">purrr</span> package (part of <span class="pack">tidyverse</span>)

Consider the following: we’ll use the <span class="func">map</span> function to *map* the <span class="func">sum</span> function to each element in the list

```{r map, eval=TRUE}
x = list(1:3, 4:6, 7:9)
map(x, sum)
```

---

## Grouping and Summarizing Operations

We can even do this with modeling and other operations.

```{r group_by3, eval=TRUE}
demographics %>% 
  drop_na(race) %>% 
  group_split(race) %>%   
  map(~lm(award_total_amount ~ gender, data = .))
```

---
## Visualization

Visualizing data requires that you:
- Consider carefully the information you want to display
- And then how you want to display it

Tell a story with the data.

And have some fun with it!

---

## Things to think about


<span style="color: cornflowerblue">Color</span>

<span style="color: black">Contrast</span>

<span style="font-size: 200%">Size</span>

<span style="color: #0000001A">Transparency</span>

<span style="font-variant: small-caps; font-family: 'Alex Brush'">Legibility</span>

<span style="color: red">Access</span><span style="color: green">ibility</span>

File types

---

## ggplot2

The most widely used visualization package in R

Copied into other realms (e.g. Python)

---

## Layers

Visualization can be thought of in a layered fashion


<div style="margin-left: 0%">base</div>
<div style="margin-left: 10%">main plot</div>
<div style="margin-left: 20%">options</div>
    
---

## More pipes

<span class="pack">ggplot2</span> uses a `+` as a pipe to add layers

```{r ggplot2}
library(ggplot2)
ggplot(data) +
  geom_point(aes(x = var1, y = var2)) +
  geom_line() +
  theme(plot.caption = element_text(size = 6))
```

We can pipe *to* any ggplot as we did before (` %>% `)

---

## Aesthetics

<span class="emph">Aesthetics</span> (`aes`) map variables to visual properties

<span class="emph">Geoms</span> are the geometric units we want to display


---

## Aesthetics

```{r aesthetics}
model_variables %>% 
  filter(award_total_amount >= 50000) %>% 
ggplot() +
  geom_density(aes(x = award_total_amount, 
                   color = gender,
                   fill = gender),
               alpha = .2) + 
  scale_x_continuous(breaks = c(1e5, 1e6, 1e7, 1e8), trans = 'log')
```


---

## Aesthetics

```{r aesthetics2, eval=TRUE, echo=FALSE}
model_variables %>% 
  filter(award_total_amount >= 50000) %>% 
  ggplot() +
  geom_density(aes(x = award_total_amount, 
                   color = gender,
                   fill = gender),
               alpha = .2) + 
  scale_x_continuous(breaks = c(1e5, 1e6, 1e7, 1e8), trans = 'log')
```

---


## Stats

We can also use ggplot2 to create statistics we want to visualize

Typically used indirectly when geoms are called

Can be used for more direct control

```{r ggstats}
ggplot(model_variables, aes(age, award_total_amount)) +
  geom_point(alpha = .02) +
  stat_ellipse(color = '#ff5500')
```


---

## Stats

```{r ggstats2, echo=FALSE, eval=TRUE}
ggplot(model_variables, aes(age, award_total_amount)) +
  geom_point(alpha = .02) +
  stat_ellipse(color = '#ff5500')
```

---

## Scales

Scales are used to add specifications to axes, colors, etc.

```{r scales}
model_variables %>% 
  filter(award_total_amount >= 50000) %>% 
  ggplot() +
  geom_density(aes(x = award_total_amount, 
                   color = gender,
                   fill = gender),
               alpha = .2) + 
  scale_x_continuous(breaks = c(1e5, 1e6, 1e7, 1e8), 
                     trans = 'log') +
  scale_fill_viridis_d(begin = .25, end = .5) +
  scale_color_viridis_d(begin = .25, end = .5) 
```

---

## Scales

```{r scales2, eval=TRUE, echo=FALSE}
model_variables %>% 
  filter(award_total_amount >= 50000) %>% 
  ggplot() +
  geom_density(aes(x = award_total_amount, 
                   color = gender,
                   fill = gender),
               alpha = .2) + 
  scale_x_continuous(breaks = c(1e5, 1e6, 1e7, 1e8), 
                     trans = 'log') +
  scale_fill_viridis_d(begin = .25, end = .5) +
  scale_color_viridis_d(begin = .25, end = .5) 
```

---

## Facets

Facets allow another dimension to plots by group

- <span class="func">facet_grid</span>: returns a matrix of like dimensions
- <span class="func">facet_wrap</span>: more flexible specification

```{r facets}
model_variables %>% 
  ggplot() +
  geom_density(aes(x = award_total_amount, 
                   color = libuser,
                   fill = libuser),
               alpha = .2) + 
  facet_wrap(~ gender) 
```

---

## Facets

```{r facets2, echo=FALSE, eval=TRUE}
model_variables %>% 
  ggplot() +
  geom_density(aes(x = award_total_amount, 
                   color = libuser,
                   fill = libuser),
               alpha = .2) + 
  facet_wrap(~ gender) +
  theme(axis.text.x = element_text(size = 6))
```

---

## Themes

<span class="emph">Themes</span> allow for customization

Two uses of a 
- a built-in versions (e.g. <span class="func">theme_minimal</span>)
- DIY  (<span class="func">theme</span>(...))

For the theme function, each argument, takes on a specific value or an element function:

- <span class="func">element_rect</span>
- <span class="func">element_line</span>
- <span class="func">element_text</span>
- <span class="func">element_blank</span>

---

## Themes

```{r themes1, echo=T, eval=F}
model_variables %>% 
  ggplot() +
  geom_smooth(aes(x = age,
                y = award_total_amount,
                color = libuser),
               alpha = .2) + 
  theme_minimal()
```

---

## Themes

```{r themes2, echo=F, eval=T}
model_variables %>% 
  ggplot() +
  geom_smooth(aes(x = age,
                y = award_total_amount,
                color = libuser),
               alpha = .2) + 
  theme_minimal()
```

---

## Themes

```{r themes3, echo=T, eval=F}
model_variables %>% 
  ggplot() +
  geom_smooth(aes(x = age,
                y = award_total_amount,
                color = libuser),
               alpha = .2) + 
  theme(axis.text.x = element_text(size=12),
        panel.grid.minor.x = element_blank(),
        plot.background = element_rect(color = 'red'),
        panel.background = element_rect(color = 'slateblue'))
```

---

## Themes

```{r themes4, echo=F, eval=T}
model_variables %>% 
  ggplot() +
  geom_smooth(aes(x = age,
                y = award_total_amount,
                color = libuser),
               alpha = .2) + 
  theme(axis.text.x = element_text(size=12),
        panel.grid.minor.x = element_blank(),
        plot.background = element_rect(color = 'rosybrown'),
        panel.background = element_rect(fill = 'papayawhip'))
```

---


## Interactivity

Interactivity is a must-have tool for web-based presentation

Use to enhance exploration of the data
- Not just because one can

Allows for additional dimensions

Even useful for exploring raw data

---

## Interactivity


General 

- [plotly](https://plot.ly/r/)
  - also used in Python, Matlab, Julia
  - can convert <span class="pack">ggplot2</span> images to interactive ones
  
- [highcharter](http://jkunst.com/highcharter/)
  - general wrapper for highcharts.js 
  - works with some R packages out of the box
  
- [rbokeh](http://hafen.github.io/rbokeh/)
  - like plotly, it also has cross language support

---

## Interactivity

Specific functionality:

- [DT](https://rstudio.github.io/DT/)
  - interactive data tables
  
- [leaflet](https://rstudio.github.io/leaflet/)
    - maps with OpenStreetMap
    
- [visNetwork](http://datastorm-open.github.io/visNetwork/)
    - Network visualization
    
---

## Plotly

<span class="emph">traces</span>
- <span class="func">add_*</span>, work similar to geoms
  
<span class="emph">modes</span>
- allow for points, lines, text and combinations

<span class="emph">aesthetics</span>
- variables are denoted with `~`, constants do not use
- `x =~ var1` vs `x = 2`


---

## Plotly

```{r plotly, eval=T, out.width='100%'}
library(plotly)

model_variables %>% 
  plot_ly(x = ~gender, y = ~ age) %>% 
  add_boxplot(color =~ gender)
```

---

## Plotly

```{r plotly1, eval=F, out.width='100%'}
library(plotly)

model_variables %>% 
  group_by(libuser, gender) %>% 
  summarise(award = mean(award_total_amount)) %>% 
  plot_ly(x = ~libuser, 
          y = ~award, 
          color = ~gender,
          text = ~round(award), 
          textposition = 'auto', 
          type = 'bar') %>% 
  layout(bargap = 0.25, 
         bargroupgap = 0.25)
```

---

## Plotly

```{r plotly1_show, echo=F, eval=T, out.width='100%'}
library(plotly)

model_variables %>% 
  group_by(libuser, gender) %>% 
  summarise(award = mean(award_total_amount)) %>% 
  plot_ly(x = ~libuser, 
          y = ~award, 
          color = ~gender,
          text = ~round(award), 
          textposition = 'auto', 
          type = 'bar') %>% 
  layout(bargap = 0.25, 
         bargroupgap = 0.25)
```

---

## Plotly

```{r plotly2, eval=T, echo=-(1)}
init = glm(award_total_amount >= 5000000 ~ age*libuser, 
           data = model_variables, 
           family = binomial)

model_variables %>% 
  modelr::add_predictions(init, type = 'response') %>% 
  plot_ly(x = ~age, y = ~ pred) %>% 
  add_lines(color =~ libuser, line = list(shape = "spline")) %>% 
  layout(title = 'Predicted Prob. Award > 1 mil')
```

---

## Plotly


One of the strengths of <span class="pack">plotly</span> is that we can feed a <span class="objclass">ggplot</span> object to it, and turn our formerly static plots into interactive ones. 

```{r plotly3}
p = model_variables %>% 
  filter(award_total_amount >= 50000) %>% 
  ggplot() +
  geom_density(aes(x = log(award_total_amount), 
                   color = libuser,
                   fill = libuser),
               alpha = .2) + 
  facet_wrap(~ gender) 

ggplotly()
```

---

## Plotly


```{r plotly4, echo=FALSE, eval=TRUE}
p = model_variables %>% 
  filter(award_total_amount >= 50000) %>% 
  ggplot() +
  geom_density(aes(x = log(award_total_amount), 
                   color = libuser,
                   fill = libuser),
               alpha = .2) + 
  facet_wrap(~ gender) 

ggplotly()
```

---

## Summary

Summarizing and visualizing data is a key activity throughout the analysis process

Understand your data

Convey results in a meaningful fashion

Take the time to do it well!