feat(section): add information about mlr3inferr #855
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
larskotthoff
requested changes
Feb 3, 2025
| @@ -285,6 +285,31 @@ print(plt2) | |||
| ``` | |||
|
|
|||
|
|
|||
| ### Confidence Intervals {#sec-resampling-ci} | |||
|
|
|||
| Confidence intervals (CIs) provide a range of values within which we can be confident that it covers the true generalization error. | |||
There was a problem hiding this comment.
I would delete the first sentence. The next sentence basically says the same, but this one isn't quite accurate.
|
|
||
| Confidence intervals (CIs) provide a range of values within which we can be confident that it covers the true generalization error. | ||
| Instead of relying solely on a single point estimate, CIs offer a measure of uncertainty around this estimate, allowing us to understand the reliability of our performance estimate. | ||
| While constructing CIs for the generalization error is challenging due to the complex nature of the inference problem, some methods have been shown to work well in practice @kuempelfischer2024ciforge. |
There was a problem hiding this comment.
I would add some more context here -- some learners/models can provide these directly (and often those calculations aren't all that complex), but if the learner doesn't support it, we have to do something else. Then describe in a sentence or two what those methods do.
| rr$aggregate(msr_ci) | ||
| ``` | ||
|
|
||
| We can also use `msr("ci")`, which will automatically select the appropriate method based on the `Resampling` object, if an inference method is available for it. |
There was a problem hiding this comment.
How do I know what resamplings have inference methods?
| @@ -576,6 +601,22 @@ plt = plt + ggplot2::scale_fill_manual(values = c("grey30", "grey50", "grey70")) | |||
| print(plt) | |||
| ``` | |||
|
|
|||
| It is also possible to plot confidence intervals by setting the type of plot to `"ci"`. | |||
| Ignoring the multiple testing problem, @fig-benchmark-ci shows that the difference between the random forest and both other learners is statistically significant for the sonar task, whereas no final conclusion can be drawn for the german credit problem. | |||
There was a problem hiding this comment.
Can we not ignore the multiple testing problem? I would show results for a single learner here.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.