Skip to content

Commit

Permalink
other fixes
Browse files Browse the repository at this point in the history
  • Loading branch information
abhsarma committed Aug 1, 2024
1 parent 011fed2 commit c02d072
Show file tree
Hide file tree
Showing 6 changed files with 85 additions and 25 deletions.
16 changes: 8 additions & 8 deletions README.Rmd
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ In this document, we outline an initial approach to conducting a multiverse anal
Before we dive into the analysis, it might be helpful to establish a terminology to help describe the steps that go into creating a multiverse analysis. We adopt the "tree of analysis" metaphor (see Figure below): "an analysis proceeds from top to bottom, and each branching represents a choice between different analysis options".

```{r echo = FALSE, out.width = '60%'}
knitr::include_graphics("vignettes/figures/00-reporting-strategies.png")
knitr::include_graphics("man/figures/00-reporting-strategies.png")
```

In this terminology:
Expand Down Expand Up @@ -161,7 +161,7 @@ tidy(fit) |>
However, the original analysis involved at least four analysis decisions (A-D), and at each decision point (node) alternative choices may have led to a different result. These decisions are highlighted in the figure below:

```{r echo = FALSE, out.width = '90%'}
knitr::include_graphics("vignettes/figures/00-default-analysis.png")
knitr::include_graphics("man/figures/00-default-analysis.png")
```

Several subsequent studies, each proposing a different analysis strategy, found no presence of such an effect, suggesting that the original finding may have been a result of a idiosyncratic combination of analysis choices. Data analysis can often involve several decisions involving two or more options. In most statistical analysis, these decisions are taken by the researcher based on some reasonable justification. However, for several decisions, there can be more than one reasonable option to choose from. A multiverse analysis makes all such decisions explicit and conducts the complete analysis for all combinations of options (of each decision). Below, we use this analysis as an example of how a single analysis can be extended to a multiverse analysis.
Expand All @@ -182,7 +182,7 @@ M = multiverse()

### Two ways to building a multiverse

Through the `multiverse` DSL, users are specifying multiple analysis paths at the same time. The DSL cannot be executed directly in an R environment or R code chunk and needs to be declared, processed and executed in a special environment. To be more precise, `multiverse` takes the user declared code, parses and rewrites the code into multiple versions of valid R code, each corresponding to an unique analysis path in the multiverse. For more information on this processing step, see vignette(branch)
Through the `multiverse` DSL, users are specifying multiple analysis paths at the same time. The DSL cannot be executed directly in an R environment or R code chunk and needs to be declared, processed and executed in a special environment. To be more precise, `multiverse` takes the user declared code, parses and rewrites the code into multiple versions of valid R code, each corresponding to an unique analysis path in the multiverse. For more information on this processing step, see `vignette("branch")`

To get around these limitations, we need to declare this (multiverse DSL) code "inside a multiverse object". The `multiverse` package facilitates this through some boilerplate code:

Expand Down Expand Up @@ -213,7 +213,7 @@ The code within the `filter` function call is written in the `multiverse` DSL an
We provide the ability to declare multiverse code block as an *AddIn* in RStudio. Users can click on *AddIns* toolbar menu in RStudio (see the image below). This would create a multiverse code block at the location of the cursor in the document.

```{r echo = FALSE, out.width = '90%'}
knitr::include_graphics("vignettes/figures/01-multiverse-addins.png")
knitr::include_graphics("man/figures/01-multiverse-addins.png")
```

Alternately, users can insert a multiverse code block using a keyboard shortcut. Users can create a keyboard shortcut to declare a multiverse code block inside a RMarkdown document through the following steps:
Expand Down Expand Up @@ -301,10 +301,10 @@ df.filtered = df |>
The `multiverse` library then takes this user-declared syntax in the multiverse DSL and and compiles it into three separate, executable R expressions as shown in the figure below:

```{r echo = FALSE, out.width = '90%'}
knitr::include_graphics("vignettes/figures/02-branch.png")
knitr::include_graphics("man/figures/02-branch.png")
```

More details on the `branch()` function can be found in the corresponding `vignette(branch)`.
More details on the `branch()` function can be found in the corresponding `vignette("branch")`.

## Interfacing with the multiverse

Expand Down Expand Up @@ -423,7 +423,7 @@ This results in the following multiverse specification (for now, let's ignore th
), family = "poisson", data = df)
```

For more details on how this can be done, as well as other details on conditions, please refer to vignette(conditions).
For more details on how this can be done, as well as other details on conditions, please refer to `vignette("conditions")`.

## Executing multiverse code

Expand Down Expand Up @@ -489,7 +489,7 @@ To execute all unique analysis paths in the multiverse, an analyst can call `exe
For the default analysis, as it executes in the R environment, users have access to the same set of debugging utilities that R provides. When the user executes the entire multiverse, the library outputs the error message, a traceback—an object containing the entire call stack that caused the error—and the index of the corresponding analysis path in which the error was encountered. The execution of the remaining analysis paths in the multiverse are not halted if any errors are encountered. The traceback is helpful to identify the location of the error, as often R expressions return unidentifiable error messages. `execute_universe(<universe ID>)` (universe ID are found in the table output by `expand()`, see below) allows analysts to execute a particular analysis path and reproduce errors encountered in the execution of that specific path.

```{r echo = FALSE, out.width = '90%'}
knitr::include_graphics("vignettes/figures/06-expand.png")
knitr::include_graphics("man/figures/06-expand.png")
```

## Conclusion
Expand Down
67 changes: 59 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,7 @@ multiverse analysis. We adopt the “tree of analysis” metaphor (see
Figure below): “an analysis proceeds from top to bottom, and each
branching represents a choice between different analysis options”.

<img src="vignettes/figures/00-reporting-strategies.png" width="60%" />
<img src="man/figures/00-reporting-strategies.png" width="60%" />

In this terminology:

Expand Down Expand Up @@ -244,7 +244,7 @@ However, the original analysis involved at least four analysis decisions
led to a different result. These decisions are highlighted in the figure
below:

<img src="vignettes/figures/00-default-analysis.png" width="90%" />
<img src="man/figures/00-default-analysis.png" width="90%" />

Several subsequent studies, each proposing a different analysis
strategy, found no presence of such an effect, suggesting that the
Expand Down Expand Up @@ -290,7 +290,7 @@ executed in a special environment. To be more precise, `multiverse`
takes the user declared code, parses and rewrites the code into multiple
versions of valid R code, each corresponding to an unique analysis path
in the multiverse. For more information on this processing step, see
vignette(branch)
`vignette("branch")`

To get around these limitations, we need to declare this (multiverse
DSL) code “inside a multiverse object”. The `multiverse` package
Expand Down Expand Up @@ -349,7 +349,7 @@ RStudio. Users can click on *AddIns* toolbar menu in RStudio (see the
image below). This would create a multiverse code block at the location
of the cursor in the document.

<img src="vignettes/figures/01-multiverse-addins.png" width="90%" />
<img src="man/figures/01-multiverse-addins.png" width="90%" />

Alternately, users can insert a multiverse code block using a keyboard
shortcut. Users can create a keyboard shortcut to declare a multiverse
Expand Down Expand Up @@ -447,6 +447,20 @@ that particular option.
Putting it all together, a decision point in a multiverse analysis can
thus be declared as:

``` r
# here we just create the variable `df` in the multiverse
df <- hurricane_data

# here, we perform a `filter` operation in the multiverse
df.filtered <- df |>
filter(branch(
death_outliers,
"no_exclusion" ~ TRUE,
"most_extreme" ~ name != "Katrina",
"two_most_extreme" ~ !(name %in% c("Katrina", "Audrey"))
))
```

```{multiverse branch_definition, inside = M}
# here we just create the variable `df` in the multiverse
df = hurricane_data
Expand All @@ -464,10 +478,10 @@ The `multiverse` library then takes this user-declared syntax in the
multiverse DSL and and compiles it into three separate, executable R
expressions as shown in the figure below:

<img src="vignettes/figures/02-branch.png" width="90%" />
<img src="man/figures/02-branch.png" width="90%" />

More details on the `branch()` function can be found in the
corresponding `vignette(branch)`.
corresponding `vignette("branch")`.

## Interfacing with the multiverse

Expand Down Expand Up @@ -573,6 +587,22 @@ researchers could have:

This would result in 3 × 2 × 2 = 12 analysis paths.

``` r
df.filtered <- df.filtered |>
mutate(
femininity = branch(
femininity_calculation,
"masfem" ~ masfem,
"female" ~ female
),
damage = branch(
damage_transform,
"no_transform" ~ identity(dam),
"log_transform" ~ log(dam)
)
)
```

```{multiverse label = variable_definitions, inside = M}
df.filtered <- df.filtered |>
mutate(
Expand Down Expand Up @@ -680,7 +710,7 @@ ignore the previous decision on choice of models):
```

For more details on how this can be done, as well as other details on
conditions, please refer to vignette(conditions).
conditions, please refer to `vignette("conditions")`.

## Executing multiverse code

Expand Down Expand Up @@ -715,6 +745,27 @@ immediately below it, mimicking notebook code chunks
data = df)
```

``` r
fit <- glm(
branch(model, "linear" ~ log(death + 1), "poisson" ~ death) ~
branch(
main_interaction,
"no" ~ femininity + damage,
"yes" ~ femininity * damage
) + branch(
other_predictors,
"none" ~ NULL,
"pressure" %when% (main_interaction == "yes") ~ femininity * zpressure,
"wind" %when% (main_interaction == "yes") ~ femininity * zwind,
"category" %when% (main_interaction == "yes") ~ femininity * zcat,
"all" %when% (main_interaction == "yes") ~ femininity * z3,
"all_no_interaction" %when% (main_interaction == "no") ~ z3
) + branch(covariates, "1" ~ NULL, "2" ~ year:damage, "3" ~ post:damage),
family = branch(model, "linear" ~ "gaussian", "poisson" ~ "poisson"),
data = df.filtered
)
```

During interactive use (i.e. when running code chunks individually in
RMarkdown), the following code chunk illustrates this behavior. Even
though the variable `fit` was defined in a multiverse code block, since
Expand Down Expand Up @@ -786,7 +837,7 @@ output by `expand()`, see below) allows analysts to execute a particular
analysis path and reproduce errors encountered in the execution of that
specific path.

<img src="vignettes/figures/06-expand.png" width="90%" />
<img src="man/figures/06-expand.png" width="90%" />

## Conclusion

Expand Down
1 change: 1 addition & 0 deletions inst/js/custom.js
Original file line number Diff line number Diff line change
Expand Up @@ -162,6 +162,7 @@ function setup() {
this[k] = this.choices[k][0];
}
choices = this.choices;
console.log(choices)
combos = this.combos;
},
update: function() {
Expand Down
16 changes: 14 additions & 2 deletions pkgdown/extra.css
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,12 @@
margin-top: 48px !important;
}

h1, h2, h3, h4, h5, h6 {
margin-top: 4rem !important;
h1 {
margin-top: 2em !important;
}

h2, h3, h4, h5, h6 {
margin-top: 1em;
font-weight: 700 !important;
}

Expand Down Expand Up @@ -52,6 +56,10 @@ a:active {
text-decoration: underline;
}

.dropdown-item:active {
color: #ffffff;
}

div.sourceCode > button {
padding: 6px 12px !important;
}
Expand All @@ -62,4 +70,8 @@ div.sourceCode > button > i {

pre {
border: 1px solid #dee2e6 !important;
}

.section {
margin-top: 3em;
}
2 changes: 0 additions & 2 deletions tests/testthat/test-execute.R
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
context("execute")

library(future)
library(tidyr)
library(dplyr)
Expand Down
8 changes: 3 additions & 5 deletions tests/testthat/test-export_json.R
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
context("export_json")
library(distributional)

test_that("export_2_json creates correct CDF for estimate + std.error and distributions", {
Expand Down Expand Up @@ -76,10 +75,9 @@ test_that("export_dist_2_json creates correct CDF for distributions", {


test_that("get_limits esimtates correct finite limits non-normal distributions", {
expect_equal(get_limits(dist_gamma(1, 1))[[1]], 0) # lower limit of gamma dist is 0
expect_equal(get_limits(dist_gamma(2, 5))[[1]], 0) # lower limit of gamma dist is 0
expect_equal(get_limits(dist_beta(2, 2)), c(.min = 0, .max = 1)) # limits of beta dist are [0, 1]
expect_equal(get_limits(dist_beta(2, 2)), c(.min = 0, .max = 1)) # limits of beta dist are [0, 1]
expect_equal(get_limits(dist_gamma(1, 1))$.min, 0) # lower limit of gamma dist is 0
expect_equal(get_limits(dist_gamma(2, 5))$.min, 0) # lower limit of gamma dist is 0
expect_equal(get_limits(dist_beta(2, 2)), list(.min = 0, .max = 1)) # limits of beta dist are [0, 1]
expect_equal(get_limits(dist_exponential(2))[[1]], 0) # lower limit of exponential dist is 0
})

Expand Down

0 comments on commit c02d072

Please sign in to comment.