Add stage argument to tell checking functions what type of check is…

… being performed (#610)

Co-authored-by: rossellhayes <rossellhayes@users.noreply.github.com>

DESCRIPTION

@@ -1,7 +1,7 @@
 Type: Package
 Package: learnr
 Title: Interactive Tutorials for R
-Version: 0.10.1.9016
+Version: 0.10.1.9017
 Authors@R:
     c(person(given = "Garrick",
              family = "Aden-Buie",

NEWS.md

@@ -56,6 +56,7 @@ learnr (development version)
 * The native R pipe, introduced in R 4.1, is now recognized as a valid R operator in the interactive exercise editor. (thanks @ijlyttle, [#595](https://github.com/rstudio/learnr/pull/595))
 * Parse errors from user code that fails to parse can now be inspected by the error checker, but errors in exercise setup chunks cannot. Instead, global setup and setup chunk errors are raised as internal errors with a user-facing warning. In general, internal errors are now handled more consistently. ([#596](https://github.com/rstudio/learnr/pull/596))
 * Commented code within an exercise will no longer be auto completed. ([#604](https://github.com/rstudio/learnr/pull/604))
+* Exercises now send an argument `stage` to the checker function with a value of `"code_check"`, `"error_check"`, or `"check"`, to make it easier to determine during which stage of checking the checker function is being called. ([#610](https://github.com/rstudio/learnr/pull/610))
 
 ## Bug fixes
 

R/exercise.R

@@ -398,6 +398,7 @@ evaluate_exercise <- function(
     return_if_exercise_result(
       try_checker(
         exercise,
+        stage = "code_check",
         check_code = exercise$code_check,
         envir_prep = duplicate_env(envir)
       )
@@ -431,6 +432,7 @@ evaluate_exercise <- function(
         # checking: the exercise could be to throw an error!
         error_feedback <- try_checker(
           exercise,
+          stage = "error_check",
           check_code = error_check_code,
           envir_result = err_render$envir_result,
           evaluate_result = err_render$evaluate_result,
@@ -460,6 +462,7 @@ evaluate_exercise <- function(
     if (nzchar(exercise$check)) {
       try_checker(
         exercise,
+        stage = "check",
         check_code = exercise$check,
         envir_result = rmd_results$envir_result,
         evaluate_result = rmd_results$evaluate_result,
@@ -476,7 +479,8 @@ evaluate_exercise <- function(
 
 
 try_checker <- function(
-  exercise, name = "exercise.checker", check_code = NULL, envir_result = NULL,
+  exercise, stage,
+  name = "exercise.checker", check_code = NULL, envir_result = NULL,
   evaluate_result = NULL, envir_prep, last_value = NULL,
   engine = exercise$engine
 ) {
@@ -501,17 +505,25 @@ try_checker <- function(
     evaluate_result = evaluate_result,
     envir_prep = envir_prep,
     last_value = last_value,
-    engine = engine
+    engine = engine,
+    stage = stage
   )
   # Throw better error messaging if the checker function signature is ill-defined
   missing_args <- setdiff(names(args), checker_args)
   if (length(missing_args) && !"..." %in% checker_args) {
-    msg <- sprintf(
-      "Either add ... or the following arguments to the '%s' function: '%s'",
-      name, paste(missing_args, collapse = "', '")
-    )
-    message(msg)
-    rlang::return_from(rlang::caller_env(), exercise_result_error(msg))
+    if (identical(missing_args, "stage")) {
+      # Don't throw an error if the only missing argument is `stage`.
+      # `stage` was not available in learnr <= 0.10.1 and checker functions can
+      #   still work without it.
+      args <- args[names(args) != "stage"]
+    } else {
+      msg <- sprintf(
+        "Either add ... or the following arguments to the '%s' function: '%s'",
+        name, paste(missing_args, collapse = "', '")
+      )
+      message(msg)
+      rlang::return_from(rlang::caller_env(), exercise_result_error(msg))
+    }
   }
 
   # Call the check function
@@ -886,6 +898,7 @@ exercise_check_code_is_parsable <- function(exercise) {
   if (nzchar(exercise$error_check %||% "")) {
     error_feedback <- try_checker(
       exercise,
+      stage = "error_check",
       check_code = exercise[["error_check"]],
       envir_result = exercise[["envir"]],
       evaluate_result = error,

docs/examples.html

@@ -13,7 +13,6 @@
 
 <title>Examples</title>
 
-<script src="site_libs/header-attrs-2.11/header-attrs.js"></script>
 <script src="site_libs/jquery-3.6.0/jquery-3.6.0.min.js"></script>
 <meta name="viewport" content="width=device-width, initial-scale=1" />
 <link href="site_libs/bootstrap-3.3.5/css/cosmo.min.css" rel="stylesheet" />

docs/exercises.Rmd

@@ -281,15 +281,18 @@ To learn more about grading exercises with **gradethis**, see its grading demo (
 
 ### Custom Checking
 
-If you need custom exercise checking logic that isn't already provided grading packages like [**gradethis**](https://github.com/rstudio-education/gradethis), then you may want to write your own `exercise.checker` function. This function is called on exercise submission whenever `*-check` or `*-code-check` chunks exist. When called, this function receives all the information that **learnr** knows about the exercise at the time of the checking, including the `user_code`, `solution_code`, `check_code`, exercise environments, and the `last_value` (if any). Checking can be performed at three different time points, so the values supplied can differ depending on the time point:
+If you need custom exercise checking logic that isn't already provided grading packages like [**gradethis**](https://github.com/rstudio-education/gradethis), then you may want to write your own `exercise.checker` function. This function is called on exercise submission whenever `*-check` or `*-code-check` chunks exist. When called, this function receives all the information that **learnr** knows about the exercise at the time of the checking, including the `user_code`, `solution_code`, `check_code`, exercise environments, the `last_value` (if any), and the `stage` of checking. Checking can be performed at three different time points, so the values supplied can differ depending on the time point:
 
 1. Before exercise evaluation, at this stage:
+    * `stage` is `"code_check"`.
     * `check_code` contains `*-code-check` chunk code.
     * `envir_result`, `evaluate_result`, and `last_value` are all `NULL`.
 2. During evaluation, when an error occurs:
+    * `stage` is `"error_check"`.
     * `check_code` contains `*-error-check` chunk code.
     * `last_value` contains the error condition object.
 3. After exercise evaluation, at this stage:
+    * `stage` is `"check"`.
     * `check_code` contains `*-check` chunk code.
     * `last_value` contains the last printed value.
 
@@ -322,7 +325,7 @@ If, at any one of these stages, feedback should be provided, then `exercise.chec
 </tbody>
 </table>
 
-Below is a rough sketch of how one might implement an `exercise.checker` function. Notice how the presence of `envir_result` may be used to determine which type of check is being done (i.e., code or result check).
+Below is a rough sketch of how one might implement an `exercise.checker` function.
 
 <div id="customchecker"></div>
 <script type="text/javascript">loadSnippet('customchecker')</script>
@@ -374,6 +377,10 @@ See the table below for a full description of all the arguments supplied to `exe
 <td>The <code>engine</code> value of the evaluated chunk.</td>
 </tr>
 <tr class="even">
+<td><code>stage</code></td>
+<td>The current checking <code>stage</code>.</td>
+</tr>
+<tr class="odd">
 <td><code>...</code></td>
 <td>Unused (include for compatibility with parameters to be added in the future)</td>
 </tr>

docs/exercises.html

@@ -13,7 +13,6 @@
 
 <title>Exercises</title>
 
-<script src="site_libs/header-attrs-2.11/header-attrs.js"></script>
 <script src="site_libs/jquery-3.6.0/jquery-3.6.0.min.js"></script>
 <meta name="viewport" content="width=device-width, initial-scale=1" />
 <link href="site_libs/bootstrap-3.3.5/css/cosmo.min.css" rel="stylesheet" />
@@ -597,7 +596,7 @@ <h3>Markdown Hints</h3>
 </div>
 <div id="multiple-hints" class="section level3">
 <h3>Multiple Hints</h3>
-<p>For R code hints you can provide a sequence of hints that reveal progressively more of the solution as desired by the user. To do this create a sequence of indexed hint chunks (e.g. “-hint-1”, “-hint-2,”-hint-3”, etc.) for your exercise chunk. For example:</p>
+<p>For R code hints you can provide a sequence of hints that reveal progressively more of the solution as desired by the user. To do this create a sequence of indexed hint chunks (e.g. “-hint-1”, “-hint-2,”-hint-3", etc.) for your exercise chunk. For example:</p>
 <div id="exercisehints">
 
 </div>
@@ -688,20 +687,23 @@ <h3>Checking Errors</h3>
 </div>
 <div id="custom-checking" class="section level3">
 <h3>Custom Checking</h3>
-<p>If you need custom exercise checking logic that isn’t already provided grading packages like <a href="https://github.com/rstudio-education/gradethis"><strong>gradethis</strong></a>, then you may want to write your own <code>exercise.checker</code> function. This function is called on exercise submission whenever <code>*-check</code> or <code>*-code-check</code> chunks exist. When called, this function receives all the information that <strong>learnr</strong> knows about the exercise at the time of the checking, including the <code>user_code</code>, <code>solution_code</code>, <code>check_code</code>, exercise environments, and the <code>last_value</code> (if any). Checking can be performed at three different time points, so the values supplied can differ depending on the time point:</p>
+<p>If you need custom exercise checking logic that isn’t already provided grading packages like <a href="https://github.com/rstudio-education/gradethis"><strong>gradethis</strong></a>, then you may want to write your own <code>exercise.checker</code> function. This function is called on exercise submission whenever <code>*-check</code> or <code>*-code-check</code> chunks exist. When called, this function receives all the information that <strong>learnr</strong> knows about the exercise at the time of the checking, including the <code>user_code</code>, <code>solution_code</code>, <code>check_code</code>, exercise environments, the <code>last_value</code> (if any), and the <code>stage</code> of checking. Checking can be performed at three different time points, so the values supplied can differ depending on the time point:</p>
 <ol style="list-style-type: decimal">
 <li>Before exercise evaluation, at this stage:
 <ul>
+<li><code>stage</code> is <code>"code_check"</code>.</li>
 <li><code>check_code</code> contains <code>*-code-check</code> chunk code.</li>
 <li><code>envir_result</code>, <code>evaluate_result</code>, and <code>last_value</code> are all <code>NULL</code>.</li>
 </ul></li>
 <li>During evaluation, when an error occurs:
 <ul>
+<li><code>stage</code> is <code>"error_check"</code>.</li>
 <li><code>check_code</code> contains <code>*-error-check</code> chunk code.</li>
 <li><code>last_value</code> contains the error condition object.</li>
 </ul></li>
 <li>After exercise evaluation, at this stage:
 <ul>
+<li><code>stage</code> is <code>"check"</code>.</li>
 <li><code>check_code</code> contains <code>*-check</code> chunk code.</li>
 <li><code>last_value</code> contains the last printed value.</li>
 </ul></li>
@@ -753,7 +755,7 @@ <h3>Custom Checking</h3>
 </tr>
 </tbody>
 </table>
-<p>Below is a rough sketch of how one might implement an <code>exercise.checker</code> function. Notice how the presence of <code>envir_result</code> may be used to determine which type of check is being done (i.e., code or result check).</p>
+<p>Below is a rough sketch of how one might implement an <code>exercise.checker</code> function.</p>
 <div id="customchecker">
 
 </div>
@@ -845,6 +847,14 @@ <h3>Custom Checking</h3>
 </tr>
 <tr class="even">
 <td>
+<code>stage</code>
+</td>
+<td>
+The current checking <code>stage</code>.
+</td>
+</tr>
+<tr class="odd">
+<td>
 <code>…</code>
 </td>
 <td>

docs/formats.html

@@ -13,7 +13,6 @@
 
 <title>Formats</title>
 
-<script src="site_libs/header-attrs-2.11/header-attrs.js"></script>
 <script src="site_libs/jquery-3.6.0/jquery-3.6.0.min.js"></script>
 <meta name="viewport" content="width=device-width, initial-scale=1" />
 <link href="site_libs/bootstrap-3.3.5/css/cosmo.min.css" rel="stylesheet" />

docs/index.html

@@ -13,7 +13,6 @@
 
 <title>Interactive Tutorials for R</title>
 
-<script src="site_libs/header-attrs-2.11/header-attrs.js"></script>
 <script src="site_libs/jquery-3.6.0/jquery-3.6.0.min.js"></script>
 <meta name="viewport" content="width=device-width, initial-scale=1" />
 <link href="site_libs/bootstrap-3.3.5/css/cosmo.min.css" rel="stylesheet" />

docs/publishing.html

@@ -13,7 +13,6 @@
 
 <title>Publishing</title>
 
-<script src="site_libs/header-attrs-2.11/header-attrs.js"></script>
 <script src="site_libs/jquery-3.6.0/jquery-3.6.0.min.js"></script>
 <meta name="viewport" content="width=device-width, initial-scale=1" />
 <link href="site_libs/bootstrap-3.3.5/css/cosmo.min.css" rel="stylesheet" />
@@ -388,7 +387,7 @@ <h2>R Package</h2>
 <pre class="r"><code>install.packages(&quot;learnr&quot;)
 learnr::run_tutorial(&quot;introduction&quot;, package = &quot;mypackage&quot;)</code></pre>
 <div id="exercise-checkers" class="section level3 toc-ignore">
-<h3 class="toc-ignore">Exercise Checkers</h3>
+<h3>Exercise Checkers</h3>
 <p>Note that if your tutorial performs <a href="exercises.html#checking-exercises">exercise checking</a> via an external package then you should be sure to add the package you use for checking as an <code>Imports</code> dependency of your package so it’s installed automatically along with your package.</p>
 <p>Note that it’s likely that the <strong>learnr</strong> package will eventually include or depend on another package that provides checking functions. For the time being though explicit installation of external checking packages is a requirement.</p>
 </div>

docs/questions.html

@@ -13,7 +13,6 @@
 
 <title>Questions</title>
 
-<script src="site_libs/header-attrs-2.11/header-attrs.js"></script>
 <script src="site_libs/jquery-3.6.0/jquery-3.6.0.min.js"></script>
 <meta name="viewport" content="width=device-width, initial-scale=1" />
 <link href="site_libs/bootstrap-3.3.5/css/cosmo.min.css" rel="stylesheet" />

docs/snippets/customchecker.md

@@ -1,7 +1,7 @@
 ```r
-custom_checker <- function(label, user_code, check_code, envir_result, evaluate_result, last_value, ...) {
+custom_checker <- function(label, user_code, check_code, envir_result, evaluate_result, last_value, stage, ...) {
   # this is a code check
-  if (is.null(envir_result)) {
+  if (stage == "code_check") {
     if (is_bad_code(user_code, check_code)) {
       return(list(message = "I wasn't expecting that code", correct = FALSE))
     }
@@ -15,4 +15,4 @@ custom_checker <- function(label, user_code, check_code, envir_result, evaluate_
 }
 
 tutorial_options(exercise.checker = custom_checker)
-```
+```