-
Notifications
You must be signed in to change notification settings - Fork 0
/
03_exploring-data.Rmd
905 lines (675 loc) · 31 KB
/
03_exploring-data.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
Exploring Data
==============
Now that you have a solid foundation in the basic functions and data structures
of R, you can move on to its most popular application: data analysis. In this
chapter, you'll learn how to efficiently explore and summarize data with
visualizations and statistics. Along the way, you'll also learn how to use
apply functions, which are essential to fluency in R.
```{r, echo = FALSE}
earn = read.csv("data/earn.csv")
```
#### Learning Objectives {-}
* Describe when to use `[` versus `[[`
* Index data frames to get specific rows, columns, or subsets
* Install and load packages
* Describe the grammar of graphics
* Make a plot
* Save a plot to an image file
* Call a function repeatedly with `sapply` or `lapply`
* Split data into groups and apply a function to each
Indexing Data Frames
--------------------
This section explains how to get and set data in a data frame, expanding on the
indexing techniques you learned in Section \@ref(indexing). Under the hood,
every data frame is a list, so first you'll learn about indexing lists.
### Indexing Lists
Lists are a _container_ for other types of R objects. When you select an
element from a list, you can either keep the container (the list) or discard
it. The indexing operator `[` almost always keeps containers.
As an example, let's get some elements from a small list:
```{r}
x = list(first = c(1, 2, 3), second = sin, third = c("hi", "hello"))
y = x[c(1, 3)]
y
class(y)
```
The result is still a list. Even if we get just one element, the result of
indexing a list with `[` is a list:
```{r}
class(x[1])
```
Sometimes this will be exactly what we want. But what if we want to get the
first element of `x` so that we can use it in a vectorized function? Or in a
function that only accepts numeric arguments? We need to somehow get the
element and discard the container.
The solution to this problem is the _extraction operator_ `[[`, which is also
called the _double square bracket operator_. The extraction operator is the
primary way to get and set elements of lists and other containers.
Unlike the indexing operator `[`, the extraction operator always discards the
container:
```{r}
x[[1]]
class(x[[1]])
```
The tradeoff is that the extraction operator can only get or set one element at
a time. Note that the element can be a vector, as above. Because it can only
get or set one element at a time, the extraction operator can only index by
position or name. Blank and logical indexes are not allowed.
The final difference between the index operator `[` and the extraction operator
`[[` has to do with how they handle invalid indexes. The index operator `[`
returns `NA` for invalid vector elements, and `NULL` for invalid list elements:
```{r}
c(1, 2)[10]
x[10]
```
On the other hand, the extraction operator `[[` raises an error for invalid
elements:
```{r, error = TRUE}
x[[10]]
```
The indexing operator `[` and the extraction operator `[[` both work with any
data structure that has elements. However, you'll generally use the indexing
operator `[` to index vectors, and the extraction operator `[[` to index
containers (such as lists).
### Two-dimensional Indexing
For two-dimensional objects, like matrices and data frames, you can pass the
indexing operator `[` or the extraction operator `[[` a separate index for each
dimension. The rows come first:
```
DATA[ROWS, COLUMNS]
```
For instance, let's get the first 3 rows and all columns of the earnings data:
```{r}
earn[1:3, ]
```
As we saw in Section \@ref(all-elements), leaving an index blank means all
elements.
As another example, let's get the 3rd and 5th row, and the 2nd and 4th column:
```{r}
earn[c(3, 5), c(2, 4)]
```
Mixing several different ways of indexing is allowed. So for example, we can
get the same above, but use column names instead of positions:
```{r}
earn[c(3, 5), c("race", "age")]
```
For data frames, it's especially common to index the rows by condition and the
columns by name. For instance, let's get the `sex`, `age`, and `n_persons`
columns for all rows that pertain to women:
```{r}
result = earn[earn$sex == "Women", c("sex", "age", "n_persons")]
head(result)
```
### The `drop` Parameter
If you use two-dimensional indexing with `[` to select exactly one column, you
get a vector:
```{r}
result = earn[1:3, 2]
class(result)
```
The container is dropped, even though the indexing operator `[` usually keeps
containers. This also occurs for matrices. You can control this behavior with
the `drop` parameter:
```{r}
result = earn[1:3, 2, drop = FALSE]
class(result)
```
The default is `drop = TRUE`.
Packages
--------
A _package_ is a collection of functions for use in R. Packages usually include
documentation, and can also contain examples, vignettes, and data sets. Most
packages are developed by members of the R community, so quality varies. There
are also a few packages that are built into R but provide extra features. We'll
use a package in Section \@ref(data-visualization), so we're learning about
them now.
The [Comprehensive R Archive Network][cran], or CRAN, is the main place people
publish packages. As of writing, there were 18,619 packages posted to CRAN.
This number has been steadily increasing as R has grown in popularity.
[cran]: https://cran.r-project.org/
Packages span a wide variety of topics and disciplines. There are packages
related to statistics, social sciences, geography, genetics, physics, biology,
pharmacology, economics, agriculture, and more. The best way to find packages
is to search online, but the CRAN website also provides ["task
views"][cran-task-views] if you want to browse popular packages related to a
specific discipline.
[cran-task-views]: https://cran.r-project.org/web/views/
The `install.packages` function installs one or more packages from CRAN. Its
first argument is the packages to install, as a character vector.
When you run `install.packages`, R will ask you to choose which _mirror_ to
download the package from. A mirror is a web server that has the same set of
files as some other server. Mirrors are used to make downloads faster and to
provide redundancy so that if a server stops working, files are still available
somewhere else. CRAN has dozens of mirrors; you should choose one that's
geographically nearby, since that usually produces the best download speeds. If
you aren't sure which mirror to choose, you can use the 0-Cloud mirror, which
attempts to automatically choose a mirror near you.
As an example, here's the code to install the remotes package:
```{r, eval = FALSE}
install.packages("remotes")
```
If you run the code above, you'll be asked to select a mirror, and then see
output that looks something like this:
```
--- Please select a CRAN mirror for use in this session ---
trying URL 'https://cloud.r-project.org/src/contrib/remotes_2.3.0.tar.gz'
Content type 'application/x-gzip' length 148405 bytes (144 KB)
==================================================
downloaded 144 KB
* installing *source* package ‘remotes’ ...
** package ‘remotes’ successfully unpacked and MD5 sums checked
** using staged installation
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (remotes)
The downloaded source packages are in
‘/tmp/Rtmp8t6iGa/downloaded_packages’
```
R goes through a variety of steps to install a package, even installing other
packages that the package depends on. You can tell that a package installation
succeeded by the final line `DONE`. When a package installation fails, R prints
an error message explaining the problem instead.
Once a package is installed, it stays on your computer until you remove it or
remove R. This means you only need to install each package once. However, most
packages are periodically updated. You can reinstall a package using
`install.packages` the same way as above to get the latest version.
Alternatively, you can update all of the R packages you have installed at once
by calling the `update.packages` function. Beware that this may take a long
time if you have a lot of packages installed.
The function to remove packages is `remove.packages`. Like `install.packages`,
this function's first argument is the packages to remove, as a character
vector.
If you want to see which packages are installed, you can use the
`installed.packages` function. It does not require any arguments. It returns a
matrix with one row for each package and columns that contain a variety of
information. Here's an example:
```{r}
packages = installed.packages()
# Just print the version numbers for 10 packages.
packages[1:10, "Version"]
```
You'll see a different set of packages, since you have a different computer.
Before you can use the functions (or other resources) in an installed package,
you must load the package with the `library` function. R doesn't load packages
automatically because each package you load uses memory and may conflict with
other packages. Thus you should only load the packages you need for whatever
it is that you want to do. When you restart R, the loaded packages are cleared
and you must again load any packages you want to use.
Let's load the remotes package we installed earlier:
```{r}
library("remotes")
```
The `library` function works with or without quotes around the package name, so
you may also see people write things like `library(remotes)`. We recommend
using quotes to make it unambiguous that you are not referring to a variable.
A handful of packages print out a message when loaded, but the vast majority do
not. Thus you can assume the call to `library` was successful if nothing is
printed. If something goes wrong while loading a package, R will print out an
error message explaining the problem.
Finally, not all R packages are published to CRAN. [GitHub][gh] is another
popular place to publish R packages, especially ones that are experimental or
still in development. Unlike CRAN, GitHub is a general-purpose website for
publishing code written in any programming language, so it contains much more
than just R packages and is not specifically R-focused.
[gh]: https://github.com/
The remotes package that we just installed and loaded provides functions to
install packages from GitHub. It is generally better to install packages from
CRAN when they are available there, since the versions on CRAN tend to be more
stable and intended for a wide audience. However, if you want to install a
package from GitHub, you can learn more about the remotes package by reading
its [online documentation][remotes].
[remotes]: https://remotes.r-lib.org/
Data Visualization
------------------
There are three popular systems for creating visualizations in R:
1. The base R functions (primarily the `plot` function)
2. The lattice package
3. The ggplot2 package
These three systems are not interoperable! Consequently, it's best to choose
one to use exclusively. Compared to base R, both lattice and ggplot2 are better
at handling grouped data and generally require less code to create a
nice-looking visualization.
The ggplot2 package is so popular that there are now knockoff packages for
other data-science-oriented programming languages like Python and Julia. The
package is also part of the [_Tidyverse_][tidy], a popular collection of R
packages designed to work well together. Because of these advantages, we'll use
ggplot2 for visualizations in this and all future lessons.
[tidy]: https://www.tidyverse.org/
ggplot2 has detailed [documentation][ggplot2-docs] and also a
[cheatsheet][ggplot2-cheat].
[ggplot2-docs]: https://ggplot2.tidyverse.org/
[ggplot2-cheat]: https://github.com/rstudio/cheatsheets/blob/master/data-visualization-2.1.pdf
The "gg" in ggplot2 stands for _grammar of graphics_. The idea of a grammar of
graphics is that visualizations can be built up in layers. In ggplot2, the
three layers every plot must have are:
* Data
* Geometry
* Aesthetics
There are also several optional layers. Here are a few:
Layer | Description
---------- | -----------
scales | Title, label, and axis value settings
facets | Side-by-side plots
guides | Axis and legend position settings
annotations | Shapes that are not mapped to data
coordinates | Coordinate systems (Cartesian, logarithmic, polar)
As an example, let's plot the earnings data. First, we need to load ggplot2. As
always, if this is your first time using the package, you'll have to install
it. Then you can load the package:
```{r}
# install.packages("ggplot2")
library("ggplot2")
```
What kind of plot should we make? It depends on what data we want the plot to
show. Let's make a line plot that shows median earnings for each quarter in
2019, with separate lines for men and women.
Before plotting, we need to take a subset of the earnings that only contains
information for 2019:
```{r}
earn19 = earn[earn$year == 2019, ]
```
The data is also broken down across `race`, `ethnic_origin`, and `age`. Since
we aren't interested in these categories for the plot, we need to further
subset the data:
```{r}
earn19 = earn19[earn19$race == "All Races" &
earn19$ethnic_origin == "All Origins" &
earn19$age == "16 years and over", ]
```
Now we're ready to make the plot.
### Layer 1: Data
The data layer determines the data set used to make the plot. ggplot and most
other Tidyverse packages are designed for working with _tidy_ data frames. Tidy
means:
1. Each observation has its own row.
2. Each feature has its own column.
3. Each value has its own cell.
Tidy data sets are convenient in general. A later lesson will cover how to make
an untidy data set tidy. Until then, we'll take it for granted that the data
sets we work with are tidy.
To set up the data layer, call the `ggplot` function on a data frame:
```{r}
ggplot(earn19)
```
This returns a blank plot. We still need to add a few more layers.
### Layer 2: Geometry
The **geom**etry layer determines the shape or appearance of the visual
elements of the plot. In other words, the geometry layer determines what kind
of plot to make: one with points, lines, boxes, or something else.
There are many different geometries available in ggplot2. The package provides
a function for each geometry, always prefixed with `geom_`.
To add a geometry layer to the plot, choose the `geom_` function you want and
add it to the plot with the `+` operator:
```{r, error=TRUE, fig.show="hide"}
ggplot(earn19) + geom_line()
```
This returns an error message that we're missing aesthetics `x` and `y`. We'll
learn more about aesthetics in the next section, but this error message is
especially helpful: it tells us exactly what we're missing. When you use a
geometry you're unfamiliar with, it can be helpful to run the code for just the
data and geometry layer like this, to see exactly which aesthetics need to be
set.
As we'll see later, it's possible to add multiple geometries to a plot.
### Layer 3: Aesthetics
The **aes**thetic layer determines the relationship between the data and the
geometry. Use the aesthetic layer to map features in the data to **aesthetics**
(visual elements) of the geometry.
The `aes` function creates an aesthetic layer. The syntax is:
```
aes(AESTHETIC = FEATURE, ...)
```
The names of the aesthetics depend on the geometry, but some common ones are
`x`, `y`, `color`, `fill`, `shape`, and `size`. There is more information about
and examples of aesthetic names in the documentation.
For example, we want to put `quarter` on the x-axis and `median_weekly_earn` on
the y-axis. We also want to use a separate line style for each `sex` category.
So the aesthetic layer should be:
```{r, eval=FALSE}
aes(x = quarter, y = median_weekly_earn, linetype = sex)
```
In the `aes` function, column names are never quoted.
Unlike most layers, the aesthetic layer is not added to the plot with the `+`
operator. Instead, you can pass the aesthetic layer as the second argument to
the `ggplot` function:
```{r}
ggplot(earn19) +
aes(x = quarter, y = median_weekly_earn, linetype = sex) +
geom_line()
```
If you want to set an aesthetic to a constant value, rather than one that's
data dependent, do so *outside* of the aesthetic layer. For instance, suppose
we want to make the lines blue:
```{r}
ggplot(earn19) +
aes(x = quarter, y = median_weekly_earn, linetype = sex) +
geom_line(color = "blue")
```
If you set an aesthetic to a constant value inside of the aesthetic layer, the
results you get might not be what you expect:
```{r}
ggplot(earn19) +
aes(x = quarter, y = median_weekly_earn, linetype = sex, color = "blue") +
geom_line()
```
<!--
#### Per-geometry Aesthetics {-}
When you pass an aesthetic layer to the `ggplot` function, it applies to the
entire plot. You can also set an aesthetic layer individually for each
geometry, by passing the layer as the first argument in the `geom_` function:
```{r}
#ggplot(favs) + geom_point(aes(x = distance_mi, y = walk_min))
```
This is really only useful when you have multiple geometries. As an example,
let's color-code the points by major:
```{r}
#ggplot(favs, aes(x = distance_mi, y = walk_min, color = major)) +
# geom_point()
```
Now let's also add labels to each point. To do this, we need to add another
geometry:
```{r}
#ggplot(favs, aes(x = distance_mi, y = walk_min, color = major, label = major)) +
# geom_point() + geom_text(size = 2)
```
Where we put the aesthetics matters:
```{r}
#ggplot(favs, aes(x = distance_mi, y = walk_min, label = major)) +
# geom_point() + geom_text(aes(color = major), size = 2)
```
-->
### Layer 4: Scales
The scales layer controls the title, axis labels, and axis scales of the plot.
Most of the functions in the scales layer are prefixed with `scale_`, but not
all of them.
The `labs` function is especially important, because it's used to set the title
and axis labels:
```{r}
ggplot(earn19) +
aes(x = quarter, y = median_weekly_earn, linetype = sex) +
geom_line() +
labs(x = "Quarter", y = "Median Weekly Salary (USD)",
title = "2019 Median Weekly Salaries, by Sex", linetype = "Sex")
```
### Saving Plots
In ggplot2, use the `ggsave` function to save the most recent plot you created:
```{r, eval=FALSE}
ggsave("line.png")
```
The file format is selected automatically based on the extension. Common
formats are PNG and PDF.
#### The Plot Device {-}
You can also save a plot with one of R's "plot device" functions. The steps
are:
1. Call a plot device function: `png`, `jpeg`, `pdf`, `bmp`, `tiff`, or `svg`.
2. Run your code to make the plot.
3. Call `dev.off` to indicate that you're done plotting.
This strategy works with any of R's graphics systems (not just ggplot2).
Here's an example:
```{r, eval=FALSE}
# Run these lines in the console, not the notebook!
jpeg("line.jpeg")
ggplot(earn19) +
aes(x = quarter, y = median_weekly_earn) +
geom_point()
dev.off()
```
### Example: Bar Plot
Let's say we want to plot the number of persons for each sex, again using the
earnings data set. A bar plot is an appropriate way to represent this visually.
The geometry for a bar plot is `geom_bar`. Since bar plots are mainly used to
display frequencies, the `geom_bar` function automatically computes frequencies
when given mapped to a categorical feature.
The `n_persons` feature is not categorical, so we don't need `geom_bar` to
compute frequencies. To prevent `geom_bar` from computing frequencies
automatically, set `stat = "identity"`.
Here's the code to make the bar plot:
```{r}
ggplot(earn19) +
aes(x = quarter, y = n_persons, fill = sex) +
geom_bar(stat = "identity", position = "dodge") +
labs(x = "Quarter", y = "Number of Workers", fill = "Sex",
title = "Number of Workers by Quarter and Sex in 2019")
```
The setting `position = "dodge"` instructs `geom_bar` to put the bars
side-by-side rather than stacking them.
### Visualization Design
Designing high-quality visualizations goes beyond just mastering which R
functions to call. You also need to think carefully about what kind of data you
have and what message you want to convey. This section provides a few
guidelines.
The first step in data visualization is choosing an appropriate kind of plot.
Here are some suggestions (not rules):
| Feature 1 | Feature 2 | Plot
| :---------- |:----------- |:----
| categorical | | bar, dot
| categorical | categorical | bar, dot, mosaic
| numerical | | box, density, histogram
| numerical | categorical | box, density, ridge
| numerical | numerical | line, scatter, smooth scatter
If you want to add a:
* 3rd numerical feature, use it to change point/line sizes.
* 3rd categorical feature, use it to change point/line styles.
* 4th categorical feature, use side-by-side plots.
Once you've selected a plot, here are some rules you should almost always
follow:
* Always add a title and axis labels. These should be in plain English, not
variable names!
* Specify units after the axis label if the axis has units. For instance,
"Height (ft)".
* Don't forget that many people are colorblind! Also, plots are often printed
in black and white. Use point and line styles to distinguish groups; color is
optional.
* Add a legend whenever you've used more than one point or line style.
* Always write a few sentences explaining what the plot reveals. Don't
describe the plot, because the reader can just look at it. Instead,
explain what they can learn from the plot and point out important details
that are easily overlooked.
* Sometimes points get plotted on top of each other. This is called
_overplotting_. Plots with a lot of overplotting can be hard to read and can
even misrepresent the data by hiding how many points are present. Use a
two-dimensional density plot or jitter the points to deal with overplotting.
* For side-by-side plots, use the same axis scales for both plots so that
comparing them is not deceptive.
Visualization design is a deep topic, and whole books have been written about
it. One resource where you can learn more is DataLab's [Principles of Data
Visualization Workshop Reader][dataviz].
[dataviz]: https://ucdavisdatalab.github.io/workshop_data_viz_principles/
Apply Functions
---------------
Section \@ref(vectorization) introduced vectorization, a convenient and
efficient way to compute multiple results. That section also mentioned that
some of R's functions---the ones that summarize or aggregate data---are not
vectorized.
The `class` function is an example of a function that's not vectorized. If we
call the `class` function on the earnings data set, we get just one result for
the data set as a whole:
```{r}
class(earn)
```
What if we want to get the class of each column? We can get the class for a
single column by selecting the column with `$`, the dollar sign operator:
```{r}
class(earn$age)
```
But what if we want the classes for all the columns? We could write a call to
`class` for each column, but that would be tedious. When you're working with a
programming language, you should try to avoid tedium; there's usually a better,
more automated way.
Section \@ref(lists) pointed out that data frames are technically lists, where
each column is one element. With that in mind, what we need here is a line of
code that calls `class` on each element of the data frame. The idea is similar
to vectorization, but since we have a list and a non-vectorized function, we
have to do a bit more than just call `class(earn)`.
The `lapply` function calls, or _applies_, a function on each element of a list
or vector. The syntax is:
```
lapply(X, FUN, ...)
```
The function `FUN` is called once for each element of `X`, with the element as
the first argument. The `...` is for additional arguments to `FUN`, which are
held constant across all the elements.
Let's try this out with the earnings data and the `class` function:
```{r}
lapply(earn, class)
```
The result is similar to if the `class` function was vectorized. In fact, if we
use a vector and a vectorized function with `lapply`, the result is nearly
identical to the result from vectorization:
```{r}
x = c(1, 2, pi)
sin(x)
lapply(x, sin)
```
The only difference is that the result from `lapply` is a list. In fact, the
`lapply` function always returns a list with one element for each element of
the input data. The "l" in `lapply` stands for "list".
The `lapply` function is one member of a family of functions called _apply
functions_. All of the apply functions provide ways to apply a function
repeatedly to different parts of a data structure. We'll meet a few more apply
functions soon.
When you have a choice between using vectorization or an apply function, you
should always choose vectorization. Vectorization is clearer---compare the two
lines of code above---and it's also significantly more efficient. In fact,
vectorization is the most efficient way to call a function repeatedly in R.
As we saw with the `class` function, there are some situations where
vectorization is not possible. That's when you should think about using an
apply function.
### The `sapply` Function
The related `sapply` function calls a function on each element of a list or
vector, and simplifies the result. That last part is the crucial difference
compared to `lapply`. When results from the calls all have the same type and
length, `sapply` returns a vector or matrix instead of a list. When the results
have different types or lengths, the result is the same as for `lapply`. The
"s" in `sapply` stands for "simplify".
For instance, if we use `sapply` to find the classes of the columns in the
earnings data, we get a character vector:
```{r}
sapply(earn, class)
```
Likewise, if we use `sapply` to compute the `sin` values, we get a numeric
vector, the same as from vectorization:
```{r}
sapply(x, sin)
```
In spite of that, vectorization is still more efficient than `sapply`, so use
vectorization instead when possible.
Apply functions are incredibly useful for summarizing data. For example,
suppose we want to compute the frequencies for all of the columns in the
earnings data set that aren't numeric.
First, we need to identify the columns. One way to do this is with the
`is.numeric` function. Despite the name, this function actually tests whether
its argument is a real number, not whether it its argument is a numeric vector.
In other words, it also returns true for integer values. We can use `sapply` to
apply this function to all of the columns in the earnings data set:
```{r}
is_not_number = !sapply(earn, is.numeric)
is_not_number
```
Is it worth using R code to identify the non-numeric columns? Since there are
only 8 columns in the earnings data set, maybe not. But if the data set was
larger, with say 100 columns, it definitely would be.
In general, it's a good habit to use R to do things rather than do them
manually. You'll get more practice programming, and your code will be more
flexible if you want to adapt it to other data sets.
Now that we know which columns are non-numeric, we can use the `table` function
to compute frequencies. We only want to compute frequencies for those columns,
so we need to subset the data:
```{r}
lapply(earn[, is_not_number], table)
```
We use `lapply` rather than `sapply` for this step because the table for each
column will have a different length (but try `sapply` and see what happens!).
### The Split-Apply Pattern
In a data set with categorical features, it's often useful to compute something
for each category. The `lapply` and `sapply` functions can compute something
for each element of a data structure, but categories are not necessarily
elements.
For example, the earnings data set has three different categories in the `sex`
column. If we want all of the rows in one category, one way to get them is by
indexing:
```{r}
women = earn[earn$sex == "Women", ]
head(women)
```
To get all three categories, we'd have to do this three times. If we want to
compute something for each category, say the mean of the `n_persons` column, we
also have to repeat that computation three times. Here's what it would look
like for just the `women` category:
```{r}
mean(women$n_persons)
```
If the categories were elements, we could avoid writing code to index each
category, and just use the `sapply` (or `lapply`) function to apply the `mean`
function to each.
The `split` function splits a vector or data frame into groups based on a
vector of categories. The first argument to `split` is the data, and the
second argument is a congruent vector of categories.
We can use `split` to elegantly compute means of `n_persons` broken down by
sex. First, we split the data by category. Since we only want to compute on the
`n_persons` column, we only split that column:
```{r}
by_sex = split(earn$n_persons, earn$sex)
class(by_sex)
names(by_sex)
```
The result from `split` is a list with one element for each category. The
individual elements contain pieces of the original `n_persons` column:
```{r}
head(by_sex$Women)
```
Since the categories are elements in the split data, now we can use `sapply`
the same way we did in previous examples:
```{r}
sapply(by_sex, mean)
```
This two-step process is an R idiom called the _split-apply pattern_. First you
use `split` to convert categories into list elements, then you use an apply
function to compute something on each category. Any time you want to compute
results by category, you should think of this pattern.
The split-apply pattern is so useful that R provides the `tapply` function as a
shortcut. The `tapply` function is equivalent to calling `split` and then
`sapply`. Like `split`, the first argument is the data and the second argument
is a congruent vector of categories. The third argument is a function to apply,
like the function argument in `sapply`.
We can use `tapply` to compute the `n_persons` means by `sex` for the earnings
data:
```{r}
tapply(earn$n_persons, earn$sex, mean)
```
Notice that the result is identical to the one we computed before.
The "t" in `tapply` stands for "table", because the `tapply` function is a
generalization of the `table` function. If you use `length` as the third
argument to `tapply`, you get the same results as you would from using the
`table` function on the category vector.
The `aggregate` function is closely related to `tapply`. It computes the same
results, but organizes them into a data frame with one row for each category.
In some cases, this format is more convenient. The arguments are the same,
except that the second argument must be a list or data frame rather than a
vector.
As an example, here's the result of using `aggregate` to compute the
`n_persons` means:
```{r}
aggregate(earn$n_persons, list(earn$sex), mean)
```
The `lapply`, `sapply`, and `tapply` functions are the three most important
functions in the family of apply functions, but there are many more. You can
learn more about all of R's apply functions by reading [this StackOverflow
post][apply].
[apply]: https://stackoverflow.com/a/7141669
Exercises
---------
### Exercise
Count how many rows of the `earn` data have `median_weekly_earn` greater than $700.
### Exercise
Calculate the median of `median_weekly_earn` for men in 2018.
### Exercise
1. Adjust the line plot of weekly earnings by quarter so that there is one line per age group.
2. Further adjust this plot so that each age group line is a different color.