Within Session splitter #664
Conversation
Deleting unified_eval, so it can be addressed on another pr. Working on tests.
Adding: figures for documentation
Signed-off-by: Bruna Junqueira Lopes <99977808+brunaafl@users.noreply.github.com>
# Conflicts: # moabb/evaluations/metasplitters.py # moabb/tests/metasplits.py
Add shuffle and random_state parameters to WithinSession
docs/source/images/withinsess.png
Outdated
There was a problem hiding this comment.
better represent only one session @brunaafl
moabb/evaluations/splitters.py
Outdated
There was a problem hiding this comment.
I think you can evaluate this splitter now @tomMoral
# Conflicts: # moabb/tests/splits.py
# Conflicts: # moabb/evaluations/splitters.py
There was a problem hiding this comment.
Hi @brunaafl,
Thanks for this PR, it looks good!
I left one comment regarding a test I think you should add
| @pytest.mark.parametrize("shuffle", [True, False]) | ||
| @pytest.mark.parametrize("random_state", [0, 42]) | ||
| def test_within_session(shuffle, random_state): | ||
| X, y, metadata = paradigm.get_data(dataset=dataset) |
There was a problem hiding this comment.
I think it is important to check if the split is the same when we load the data of one/a few subject(s) only, paradigm.get_data(dataset=dataset, subjects=[m, n...])
There was a problem hiding this comment.
A frist batch of comments. Overall, it looks good but I think it could be improved by adding the possibility to pass in a cv object, that would allow to control the intrasession splits (for instance doing TimeSeriesSplit, which makes sense in an online setting).
| and test sets on separate session for each subject. This splitter assumes that | ||
| all data from all subjects is already known and loaded. |
There was a problem hiding this comment.
| and test sets on separate session for each subject. This splitter assumes that | |
| all data from all subjects is already known and loaded. | |
| and test sets for each subject in each session. This splitter | |
| assumes that all data from all subjects is already known and loaded. |
There was a problem hiding this comment.
why not using only the other version?
| Parameters | ||
| ---------- | ||
| n_folds : int | ||
| Number of folds. Must be at least 2. |
There was a problem hiding this comment.
| Number of folds. Must be at least 2. | |
| Number of folds for the within-session k-fold split. Must be at least 2. |
| random_state: int = 42, | ||
| shuffle_subjects: bool = False, | ||
| shuffle_session: bool = True, |
There was a problem hiding this comment.
Default random_state should be None. Also, convention would be to put it at the end of the argument list.
| random_state: int = 42, | |
| shuffle_subjects: bool = False, | |
| shuffle_session: bool = True, | |
| shuffle_subjects: bool = False, | |
| shuffle_session: bool = True, | |
| random_state: int = None, |
| random_state: int, RandomState instance or None, default=None | ||
| Important when `shuffle` is True. Controls the randomness of splits. | ||
| Pass an int for reproducible output across multiple function calls. |
There was a problem hiding this comment.
Move at the end.
| random_state: int, RandomState instance or None, default=None | |
| Important when `shuffle` is True. Controls the randomness of splits. | |
| Pass an int for reproducible output across multiple function calls. | |
| random_state: int, RandomState instance or None, default=None | |
| Controls the randomness of splits. Only used when `shuffle` is True. | |
| Pass an int for reproducible output across multiple function calls. |
| shuffle_session : bool, default=True | ||
| Whether to shuffle each class's samples before splitting into batches. | ||
| Note that the samples within each split will not be shuffled. | ||
| shuffle_subjects : bool, default=False | ||
| Apply shuffle in mixing subjects and sessions, this parameter allows | ||
| sample iterations of the sppliter. |
There was a problem hiding this comment.
Do you think it is necessary to have both? I don't really see any use case where I would only use one no?
I would only have a shuffle
| self.n_folds = n_folds | ||
| self.shuffle_subjects = shuffle_subjects | ||
| self.shuffle_session = shuffle_session | ||
| self.random_state = check_random_state(random_state) |
There was a problem hiding this comment.
If you use it like that, this is not a random_state anymore but a rng.
|
|
||
| def split(self, y, metadata, **kwargs): | ||
| all_index = metadata.index.values | ||
| subjects = metadata.subject.unique() |
There was a problem hiding this comment.
In general, avoid using the .col notation in pandas, this is super error prone and hard to read, use the indexation:
| subjects = metadata.subject.unique() | |
| subjects = metadata['subject'].unique() |
| for subject in subjects: | ||
| subject_mask = metadata.subject == subject | ||
| subject_indices = all_index[subject_mask] | ||
| subject_metadata = metadata[subject_mask] | ||
| sessions = subject_metadata.session.unique() | ||
|
|
||
| # Shuffle sessions if required | ||
| if self.shuffle_session: | ||
| self.random_state.shuffle(sessions) | ||
|
|
||
| for session in sessions: | ||
| session_mask = subject_metadata.session == session | ||
| indices = subject_indices[session_mask] | ||
| group_y = y[indices] | ||
|
|
||
| # Use StratifiedKFold with the group-specific random state | ||
| cv = StratifiedKFold( | ||
| n_splits=self.n_folds, | ||
| shuffle=self.shuffle_session, | ||
| random_state=self.random_state, | ||
| ) | ||
| for ix_train, ix_test in cv.split(indices, group_y): | ||
| yield indices[ix_train], indices[ix_test] |
There was a problem hiding this comment.
We talk a bit with @sylvchev and I think the best would be to modify this, to take a cv object in the constructor (default would be StratifiedKFold), clone it with a different random seed for each group subject, session, and then yield the right indices.
That way, we can do a real shuffle, with shuffling the groups from which we retrieve the next split.
Would this make sense?
There was a problem hiding this comment.
Make sense, I'm working on that, thanks!
There was a problem hiding this comment.
great! Thank you so much! And thanks for your patience :)
This PR is a follow-up on PR #624 and is related to issue #612. It includes just the implementation for the WithinSessionSplitter data splitter.