forked from tjmahr/MadR_RStanARM
-
Notifications
You must be signed in to change notification settings - Fork 0
/
01-how-i-got-into-Bayes.Rpres
229 lines (134 loc) · 6.68 KB
/
01-how-i-got-into-Bayes.Rpres
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
Bayesian Regression Models with RStanARM
========================================================
author: TJ Mahr
date: Sept. 21, 2016
autosize: true
incremental: false
Madison R Users Group
<small>
[Github repository](https://github.com/tjmahr/MadR_RStanARM)
<br/>
[@tjmahr](https://twitter.com/tjmahr)
</small>
Overview
===============================================================================
- **[How I got into Bayesian statistics](http://rpubs.com/tjmahr/rep-crisis)**
- [Some intuition-building about Bayes theorem](http://rpubs.com/tjmahr/bayes-theorem)
- [Tour of RStanARM](http://rpubs.com/tjmahr/rstanarm-tour)
- [Where to learn more about Bayesian statistics](http://rpubs.com/tjmahr/bayes-learn-more)
August 2015: The "Crisis" in Psychology
===============================================================================
left: 50%
[Open Science Collaboration (2015)][osf2015] tries to replicate 100 studies
published in 3 psychology different journals in 2008.
![Scatter plot of original vs replicated effect sizes]
(./assets/reproducibility.PNG)
***
- Boil a study down to 1 test statistic and 1 effect size.
- Compare replication's test and effect size against original.
- Approximately 36% of the studies are replicated (same test statistic.
- On average, effect sizes in replications are half that of the original
studies.
[osf2015]: http://science.sciencemag.org/content/349/6251/aac4716
Reactions
===============================================================================
We're doomed.
Most findings are probably false.
***
No, this is business as usual.
Any credible discipline has to do this kind of house-cleaning from time to time.
Lots of hand-wringing and soul-searching
===============================================================================
Some reactionary:
- Replication creates an [industry for incompetent hacks][replication-hacks].
- Here come the [methodological terrorists][method-terrorists]!
***
Some constructive:
- [Everything is f'ed][everything-fucked] -- so what else is new?
- Increased rigor and openness are [a good thing][repeat-after-me].
[replication-hacks]: http://www.sciencedirect.com/science/article/pii/S002210311600007X
[method-terrorists]: http://andrewgelman.com/2016/09/21/what-has-happened-down-here-is-the-winds-have-changed/
[everything-fucked]: https://hardsci.wordpress.com/2016/08/11/everything-is-fucked-the-syllabus/
[repeat-after-me]: https://thenib.com/repeat-after-me
I'm in the business-as-usual camp, by the way
===============================================================================
Better research practices are catching on.
- Pre-registration
- Power analyses and [other foresight][stats-autopsy].
- Meta-analytic tools to assess the health of a field
- Better disclosure of other unanalyzed measurements
[stats-autopsy]: http://www.johndcook.com/blog/2010/05/13/statistical-autopsy/
Crisis made me think more of questionable data analysis practices
===============================================================================
Basically, unintentional acts and rituals to appease Statistical Significance
gods.
- HARKing (hypothesizing after results are known)
- Telling a story to fit the data.
- Garden of forking data
- Conducting countless sub-tests and sub-analyses on the data
- p-hacking
- Doing these tests in order to find a significant effect.
- Selective reporting
- Reporting only the tests that yielded a significant result.
My response to the crisis
===============================================================================
I want to avoid these errors.
- I want to level up my stats and explore new techniques.
- Maybe more robust estimation techniques
- Maybe machine learning techniques to complement conventional analyses.
- I want something less finicky than statistical significance.
- p-values don't mean what many people think they mean.
- Neither do confidence intervals.
- Statistical significance is not related to practical significance.
December 2015
===============================================================================
- I started reading the Gelman and Hill book.
- This is the book for [the `arm` package][arm].
- It emphasizes estimation, uncertainty and simulation.
- Midway through, the book pivots to Bayesian estimation.
- I roll with it.
***
[arm]: https://cran.rstudio.com/web/packages/arm/
Using ARM
===============================================================================
title: false
I try to download the example BUGS scripts, but they are no longer supported.
Instead, I find [a page with the title][use-stan]:
> ### Use Stan instead.
<br/>
Okay, I'll give it try...
[use-stan]: http://www.stat.columbia.edu/~gelman/bugsR/
January 2016
===============================================================================
I'm down a rabbit hole, writing Stan models to fit the models from the ARM
book, and there is an influx of Bayesian tools for R.
- [Statistical Rethinking][rethinking], a book that reteaches regression from a
Bayesian perspective with R and Stan, is released.
- New version of [brms][brms] is released. This package converts R model code
into Stan programs.
- [RStanARM][rstanarm] is released.
- This blog post circulates: ["R Users Will Now Inevitably Become Bayesians"]
[we-r-bayesians].
I eat all this up. I become a convert.
[rethinking]: http://xcelab.net/rm/statistical-rethinking/
[brms]: https://cran.r-project.org/web/packages/brms/index.html
[rstanarm]: https://cran.r-project.org/web/packages/rstanarm/index.html
[we-r-bayesians]: https://thinkinator.com/2016/01/12/r-users-will-now-inevitably-become-bayesians/
Long story short
===============================================================================
The replication crisis sparked my curiosity, and a wave of new tools and
resources made it really easy to get started with Bayesian stats.
No formal coursework. I learned about the math/statistics from playing with
code.
My goal with these toosl has been to make better, more honest scientific
summaries of observed data.
Next
===============================================================================
- [How I got into Bayesian statistics](http://rpubs.com/tjmahr/rep-crisis)
- **[Some intuition-building about Bayes theorem](http://rpubs.com/tjmahr/bayes-theorem)**
- [Tour of RStanARM](http://rpubs.com/tjmahr/rstanarm-tour)
- [Where to learn more about Bayesian statistics](http://rpubs.com/tjmahr/bayes-learn-more)