Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Checking for exercise errors #403

Merged
merged 10 commits into from
Jul 29, 2020
Merged

Checking for exercise errors #403

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
d505f2d
empty_results()'s html_output needs to req()-able; otherwise, a value…
cpsievert Jul 16, 2020
7bac39a
Refactor evaluate_exercise() and add exercise.error.checker for runni…
cpsievert Jul 16, 2020
bb04d02
code review feedback
cpsievert Jul 22, 2020
756a809
knitr fig options were actually needed
cpsievert Jul 22, 2020
d74893e
shorter tempfile pattern
cpsievert Jul 22, 2020
fe4f82d
docs
cpsievert Jul 22, 2020
0e3994b
Provide an exercise.error.checker option
cpsievert Jul 23, 2020
10b84ca
Document new exercise.error.checker option
cpsievert Jul 28, 2020
7c82b53
update news
cpsievert Jul 28, 2020
b253a7c
Merge branch 'master' into carson/feature/on_user_error
cpsievert Jul 29, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions NEWS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,15 @@ learnr (development version)
* Renamed the `exercise_submission` event to `exercise_result` and added the following fields:
1. `id` - a randombly generated identifier that can be used to align with the associated `exercise_result` event.
2. `time_elapsed` - the time required to run the exercise (in seconds)
3. `timeout_exceeded` - indicates whether the exercise was interrupted due to an exceeded timeout. May be `NA` for some platforms/evaluators if that information is not known or reported. ([#337](https://github.com/rstudio/learnr/pull/337))
3. `timeout_exceeded` - indicates whether the exercise was interrupted due to an exceeded timeout. May be `NA` for some platforms/evaluators if that information is not known or reported. ([#337](https://github.com/rstudio/learnr/pull/337))
* If a `-code-check` chunk returns feedback for an exercise submission, the result of the exercise is no longer displayed for a correct answer (only the feedback is displayed). If both the result and feedback should be displayed, all checking should be performed in a `-check` chunk (i.e., don't provide a `-code-check` chunk). ([#403](https://github.com/rstudio/learnr/pull/403))

## New features

* Introduced "setup chunk chaining", which allows a setup chunk to depend on another setup chunk and so on, forming a chain of setup code that can be used for exercises via `exercise.setup`. Run `run_tutorial("setup-chunks", "learnr")` for a demo. ([#390](https://github.com/rstudio/learnr/pull/390))
* Introduced an [experimental](https://www.tidyverse.org/lifecycle/#experimental) function `external_evaluator()` which can be used to define an exercise evaluator that runs on a remote server and is invoked via HTTP. This allows all exercise execution to be performed outside of the Shiny process hosting the learnr document. ([#345](https://github.com/rstudio/learnr/pull/345), [#354](https://github.com/rstudio/learnr/pull/354))
* For the "forked" evaluator (the default used on Linux), add a limit to the number of forked exercises that learnr will execute in parallel. Previously, this was uncapped, which could cause a learnr process to run out of memory when an influx of traffic arrived. The default limit is 3, but it can be configured using the `tutorial.max.forked.procs` option or the `TUTORIAL_MAX_FORKED_PROCS` environment variable. ([#353](https://github.com/rstudio/learnr/pull/353))
* Added a new `tutorial_options()`, namely `exercise.error.checker`, for customizing feedback when exercise submission code produces an evaluation error. This option accepts a function with the same arguments as `exercise.checker`. Use `gradethis::grade_learnr_error()` for a sensible default for this option. ([#403](https://github.com/rstudio/learnr/pull/403))
* Added an event handler system, with the functions `event_register_handler()` and `one_time()`. There is also a new event `"section_viewed"`, which is triggered when a new section becomes visible. ([#398](https://github.com/rstudio/learnr/pull/398))
* Previously, when a question submission was reset, it would be recorded as a `"question_submission"` event with the value `reset=TRUE`. Now it a separate event, `"reset_question_submission"`. ([#398](https://github.com/rstudio/learnr/pull/398))

Expand All @@ -22,7 +24,7 @@ learnr (development version)
* Added a `restore` flag on `exercise_submitted` events which is `TRUE` if the exercise is being restored from a previous execution, or `FALSE` if the exercise is being run interactively.
* Add `label` field to the `exercise_hint` event to identify for which exercise the user requested a hint. ([#377](https://github.com/rstudio/learnr/pull/377))
* Added `include=FALSE` to setup chunks to prevent exercises from printing out messages or potential code output for those setup chunks. ([#390](https://github.com/rstudio/learnr/pull/390))
* Added error handling when user specificies a non-existent label for `exercise.setup` option with an error message. ([#390](https://github.com/rstudio/learnr/pull/390))
* Added error handling when user specifies a non-existent label for `exercise.setup` option with an error message. ([#390](https://github.com/rstudio/learnr/pull/390))
* We no longer forward the checker code to browser (in html), but instead cache it. ([#390](https://github.com/rstudio/learnr/pull/390))
* We no longer display an invisible exercise result warning automatically. Instead, authors must set the exercise chunk option `exercise.warn_invisible = TRUE` to display an invisible result warning message. ([#373](https://github.com/rstudio/learnr/pull/373))

Expand Down
10 changes: 5 additions & 5 deletions R/evaluators.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ inline_evaluator <- function(expr, timelimit, ...) {
error = function(e) {
# TODO: could grepl the error message to determine if the error was due
# to an exceeded timeout.
error_result(e$message, timeout_exceeded = NA)
exercise_result_error(e$message, timeout_exceeded = NA)
}
)
},
Expand Down Expand Up @@ -111,7 +111,7 @@ setup_forked_evaluator_factory <- function(max_forked_procs){

# check if it's an error and convert it to an html error if it is
if(inherits(result, "try-error"))
result <<- error_result(result, timeout_exceeded = FALSE)
result <<- exercise_result_error(result, timeout_exceeded = FALSE)

TRUE
}
Expand All @@ -126,7 +126,7 @@ setup_forked_evaluator_factory <- function(max_forked_procs){
running_exercises <<- running_exercises - 1

# return error result
result <<- error_result(timeout_error_message(), timeout_exceeded = TRUE)
result <<- exercise_result_timeout()
TRUE
}

Expand Down Expand Up @@ -244,7 +244,7 @@ internal_external_evaluator <- function(

print("Error submitting external exercise:")
print(err)
result <<- error_result("Error submitting external exercise. Please try again later")
result <<- exercise_result_error("Error submitting external exercise. Please try again later")
}

curl::curl_fetch_multi(url, handle = handle, done = done_cb, fail = fail_cb, pool = pool)
Expand Down Expand Up @@ -280,7 +280,7 @@ internal_external_evaluator <- function(
},
onRejected = function(err){
print(err)
result <<- error_result("Error initiating session for external requests. Please try again later")
result <<- exercise_result_error("Error initiating session for external requests. Please try again later")
}
)
},
Expand Down
Loading