Three monitoring functions (for use in dashboards) #92

juliasilge · 2022-05-10T00:29:05Z

This PR adds three functions:

vetiver_compute_metrics()
vetiver_pin_metrics()
vetiver_plot_metrics()

juliasilge · 2022-05-10T23:33:10Z

The three functions in action:

library(vetiver)
library(dplyr)
#> 
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#> 
#>     filter, lag
#> The following objects are masked from 'package:base':
#> 
#>     intersect, setdiff, setequal, union
library(parsnip)
data(Chicago, package = "modeldata")
Chicago <- Chicago %>% select(ridership, date, one_of(stations))
training_data <- Chicago %>% filter(date < "2009-01-01")
testing_data <- Chicago %>% filter(date >= "2009-01-01", date < "2011-01-01")
monitoring <- Chicago %>% filter(date >= "2011-01-01", date < "2012-12-31")
lm_fit <- linear_reg() %>% fit(ridership ~ ., data = training_data)

library(pins)
b <- board_temp()

## before starting monitoring, initiate the metrics and pin
## (for example, with the testing data):
original_metrics <-
    augment(lm_fit, new_data = testing_data) %>%
    vetiver_compute_metrics(date, "week", ridership, .pred, .every = 4L) %>%
    vetiver_pin_metrics(date, b, "lm_fit_metrics", initiate = TRUE)
#> Guessing `type = 'rds'`
#> Creating new version '20220511T031342Z-082d5'
#> Writing to pin 'lm_fit_metrics'
original_metrics
#> # A tibble: 81 × 5
#>    date           n .metric .estimator .estimate
#>    <date>     <int> <chr>   <chr>          <dbl>
#>  1 2009-01-01     7 mae     standard       5.25 
#>  2 2009-01-01     7 rmse    standard       6.78 
#>  3 2009-01-01     7 rsq     standard       0.154
#>  4 2009-01-08    28 mae     standard       2.98 
#>  5 2009-01-08    28 rmse    standard       4.61 
#>  6 2009-01-08    28 rsq     standard       0.576
#>  7 2009-02-05    28 mae     standard       1.17 
#>  8 2009-02-05    28 rmse    standard       1.90 
#>  9 2009-02-05    28 rsq     standard       0.916
#> 10 2009-03-05    28 mae     standard       0.937
#> # … with 71 more rows

## to continue monitoring with new data, compute metrics and update pin:
new_metrics <-
    augment(lm_fit, new_data = monitoring) %>%
    vetiver_compute_metrics(date, "week", ridership, .pred, .every = 4L) %>%
    vetiver_pin_metrics(date, b, "lm_fit_metrics")
#> Guessing `type = 'rds'`
#> Replacing version '20220511T031342Z-082d5' with '20220511T031342Z-23a00'
#> Writing to pin 'lm_fit_metrics'
new_metrics
#> # A tibble: 162 × 5
#>    date           n .metric .estimator .estimate
#>    <date>     <int> <chr>   <chr>          <dbl>
#>  1 2009-01-01     7 mae     standard       5.25 
#>  2 2009-01-01     7 rmse    standard       6.78 
#>  3 2009-01-01     7 rsq     standard       0.154
#>  4 2009-01-08    28 mae     standard       2.98 
#>  5 2009-01-08    28 rmse    standard       4.61 
#>  6 2009-01-08    28 rsq     standard       0.576
#>  7 2009-02-05    28 mae     standard       1.17 
#>  8 2009-02-05    28 rmse    standard       1.90 
#>  9 2009-02-05    28 rsq     standard       0.916
#> 10 2009-03-05    28 mae     standard       0.937
#> # … with 152 more rows

library(ggplot2)
vetiver_plot_metrics(new_metrics, date) +
    scale_size(range = c(2, 4))

^{Created on 2022-05-10 by the reprex package (v2.0.1)}

juliasilge · 2022-05-10T23:35:25Z

DESCRIPTION

@@ -42,6 +42,9 @@ Suggests:
    callr,
    caret,
    covr,
+    curl,
+    dplyr,


This PR adds A LOT of packages here. I put all the packages for monitoring in Suggests because I am up against 20 packages in Imports already and I can sort of make the argument that people may want to deploy a model but not monitor it. I am very open to ideas for reducing this somehow.

I'm only using dplyr in examples and tests FWIW 🤷‍♀️

DavisVaughan

Comments are mainly around two themes:

Lots of yardstick specific tweaks to vetiver_compute_metrics(). I decided it was easier to dump in a rewrite of this function (you'll see that below), and I added comments about what exactly I was changing
I think vetiver_pin_metrics() might be two different functions: vetiver_initialize_metrics() and vetiver_update/append_metrics(), so you can get rid of the special "first time" behavior you have to do with initialize = TRUE, which I think is a little hard to discover

DESCRIPTION

R/monitor.R

DavisVaughan · 2022-05-12T18:15:45Z

R/monitor.R

+                                    date_var,
+                                    .period,
+                                    truth, estimate, ...,
+                                    metric_set = yardstick::metrics,


Might be an issue if yardstick is Suggests and you have this here

I checked with the R-CMD-check-hard action like in broom and it was fine, so I'm going to risk it (vs. the NOTE).

I checked again (with CRAN back up this time) and it still looks OK to have yardstick::metrics() here. 🤞

R/monitor.R

DavisVaughan · 2022-05-12T19:26:13Z

R/monitor.R

+                                 .estimate = .estimate,
+                                 .metric = .metric,


Suggested change

.estimate = .estimate,

.metric = .metric,

estimate = .estimate,

metric = .metric,

Just because I think we only ever add . to arguments if they have the potential to conflict with named ...

(I realize it is a little weird here because the column name is .estimate, so if you want to keep this then I wouldn't mind)

tests/testthat/test-monitor.R

Co-authored-by: Davis Vaughan <davis@rstudio.com>

… to opt out

juliasilge · 2022-05-23T03:33:03Z

Thank you so much for the thorough review @DavisVaughan! Looks like this now:

library(vetiver)
library(dplyr)
#> 
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#> 
#>     filter, lag
#> The following objects are masked from 'package:base':
#> 
#>     intersect, setdiff, setequal, union
library(parsnip)
data(Chicago, package = "modeldata")
Chicago <- Chicago %>% select(ridership, date, all_of(stations))
training_data <- Chicago %>% filter(date < "2009-01-01")
testing_data <- Chicago %>% filter(date >= "2009-01-01", date < "2011-01-01")
monitoring <- Chicago %>% filter(date >= "2011-01-01", date < "2012-12-31")
lm_fit <- linear_reg() %>% fit(ridership ~ ., data = training_data)

library(pins)
b <- board_temp()

## before starting monitoring, initiate the metrics and pin
## (for example, with the testing data):
original_metrics <-
    augment(lm_fit, new_data = testing_data) %>%
    vetiver_compute_metrics(date, "week", ridership, .pred, .every = 4L)
pin_write(b, original_metrics, "lm_fit_metrics")
#> Guessing `type = 'rds'`
#> Creating new version '20220523T032938Z-d51cc'
#> Writing to pin 'lm_fit_metrics'

## to continue monitoring with new data, compute metrics and update pin:
new_metrics <-
    augment(lm_fit, new_data = monitoring) %>%
    vetiver_compute_metrics(date, "week", ridership, .pred, .every = 4L) %>%
    vetiver_pin_metrics(b, "lm_fit_metrics")
#> Guessing `type = 'rds'`
#> Replacing version '20220523T032938Z-d51cc' with '20220523T032938Z-73a5d'
#> Writing to pin 'lm_fit_metrics'

library(ggplot2)
vetiver_plot_metrics(new_metrics) +
    scale_size(range = c(2, 4))

^{Created on 2022-05-22 by the reprex package (v2.0.1)}

I have one outstanding question that I would still appreciate any ideas on. You recommended taking out the . in the slider args but I did that to inherit for parameters from slider so the docs here don't accidentally get out of sync with slider or turn out wrong. (This has happened in other packages that I have observed.)

Do you know of a way to inherit arg docs that have a different name, like using .period docs from slider for period here?
How would you balance the maintenance ease of inheriting the args vs. using kind of unnecessary dots in names?

DavisVaughan · 2022-05-23T13:19:01Z

I mentioned it in the comments, but I'll add it again here:

For the specific case of .period vs period, roxygen2 understands that these are considered "the same arg", so @inheritParams slider slide_period should just "work".

…action

juliasilge added 6 commits May 9, 2022 18:28

First draft of three monitoring functions

80521bc

Need to use %>% now (in expr() for vetiver_write_plumber())

2839d1d

Move curl to Suggests

99d4761

Update pkgdown with monitoring funcs

06c9b78

Use slider for computing metrics instead

04b6cad

Add tests

737ea6b

Not using lubridate anymore

570603f

juliasilge commented May 10, 2022

View reviewed changes

juliasilge added 2 commits May 10, 2022 17:36

Remove lubridate (not using anymore) and redocument

c7982ca

Update docs

ec5c326

juliasilge marked this pull request as ready for review May 11, 2022 03:12

juliasilge requested a review from DavisVaughan May 11, 2022 03:12

DavisVaughan reviewed May 12, 2022

View reviewed changes

juliasilge and others added 9 commits May 22, 2022 18:08

Namespace functions to avoid library() in tests

ebf9059

Apply suggestions from code review

ba26008

Co-authored-by: Davis Vaughan <davis@rstudio.com>

Oops more library() to remove

b82d7bc

No more initiate option

90a9303

Refactor function with feedback from Davis

2d6b1d1

Add more context for overwriting metrics in pin, plus overwrite arg…

ddb694c

… to opt out

Try R CMD check "hard" to see if yardstick is a problem

2b0463c

Don't evaluate vignette if pkgs are not there

d67a743

Revert R CMD check change

422fa3f

Make vetiver_pin_metrics() more like other pinning funcs

977070f

juliasilge added 4 commits May 23, 2022 09:10

Remove . from slider args! 😮

9aed822

Try R-CMD-check-hard again now that CRAN is up

08f8c4a

Skip if suggested pkgs not available

22bbccb

Use @examplesIf for suggested pkgs

b1daf0e

juliasilge added 2 commits May 23, 2022 10:37

Revert to regular R CMD check, but also add -hard as an additional …

2e025dc

…action

Update NEWS

f1b6bde

juliasilge merged commit 4b52acd into main May 23, 2022

juliasilge deleted the monitoring-helpers branch May 23, 2022 17:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Three monitoring functions (for use in dashboards) #92

Three monitoring functions (for use in dashboards) #92

juliasilge commented May 10, 2022

juliasilge commented May 10, 2022 •

edited

Loading

juliasilge May 10, 2022

juliasilge May 11, 2022

DavisVaughan left a comment

DavisVaughan May 12, 2022

juliasilge May 23, 2022 •

edited

Loading

juliasilge May 23, 2022

DavisVaughan May 12, 2022

DavisVaughan May 12, 2022 •

edited

Loading

juliasilge commented May 23, 2022 •

edited

Loading

DavisVaughan commented May 23, 2022

@@ @@ -42,6 +42,9 @@ Suggests: @@
                   callr,
                   caret,
                   covr,
+                  curl,
+                  dplyr,

Three monitoring functions (for use in dashboards) #92

Three monitoring functions (for use in dashboards) #92

Conversation

juliasilge commented May 10, 2022

juliasilge commented May 10, 2022 • edited Loading

juliasilge May 10, 2022

Choose a reason for hiding this comment

juliasilge May 11, 2022

Choose a reason for hiding this comment

DavisVaughan left a comment

Choose a reason for hiding this comment

DavisVaughan May 12, 2022

Choose a reason for hiding this comment

juliasilge May 23, 2022 • edited Loading

Choose a reason for hiding this comment

juliasilge May 23, 2022

Choose a reason for hiding this comment

DavisVaughan May 12, 2022

Choose a reason for hiding this comment

DavisVaughan May 12, 2022 • edited Loading

Choose a reason for hiding this comment

juliasilge commented May 23, 2022 • edited Loading

DavisVaughan commented May 23, 2022

juliasilge commented May 10, 2022 •

edited

Loading

juliasilge May 23, 2022 •

edited

Loading

DavisVaughan May 12, 2022 •

edited

Loading

juliasilge commented May 23, 2022 •

edited

Loading