Handled errors during validation via error_score parameter #82
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sylvchev
approved these changes
May 28, 2020
|
It is ok for me. You could rebase on master branch to merge these changes. |
|
For the sake of consistency, it could be good to update |
|
I think there was some issues with CrossSubjectEvaluation, but I can check again and update this PR if possible. |
|
My newest commit should contain the necessary changes, however, I have not yet tested this, as I currently do not use any of the other evaluations. I do not know when I will be able to check this on my ERP benchmark. |
|
Did you had the time to check your code? You could add som test to verify that it is correctly working. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When using
error_score=np.nanin sklearn, this results in np.nan value whenver there's an error during transformations or model fitting.For my personal use case I also would like values of
np.nanwhen there's an error in the validation itself. I have a dataset for which (due to early stopping) sometimes the AUC cannot be calculated for 5-folds. In current moabb, this error is raised and stops the benchmark.So far I only implemented it for
WithinSessionEvaluationas I don't use the other ones.