Add benchmark made of multiple text datasets #354
Conversation
|
This code is based on #312, right? It has exactly the same test fail and I honestly don't know why. |
|
Locally when the test fails it is sufficient to re-run it and it will pass. |
Coverage reportThe coverage rate went from
Diff Coverage details (click to unfold)src/renate/defaults.py
src/renate/benchmark/datasets/nlp_datasets.py
|
There was a problem hiding this comment.
I accidentally left some more comments but I came here to point out the relationship to #357. We should merge #357 first and then make small modifications here:
- The data module extends
DomainIncrementalDataModule
On a general note, we should add the datamodule to experiment_config.py and add the dataset to the documentation. Its use will be similar to DomainNet by relying on the DomainIncrementalScenario.
|
|
||
| def get_split(split_name): | ||
| dataset = load_dataset(self.data_id, split=split_name, cache_dir=self._data_path) | ||
| new_features = dataset.features.copy() |
| train_size: int = defaults.SMALL_TRAIN_SET_SIZE, | ||
| test_size: int = defaults.SMALL_TEST_SET_SIZE, |
There was a problem hiding this comment.
what is the intuition of selecting a subset for this specific dataset?
There was a problem hiding this comment.
I set a relatively small value by default because I expect it to be closer to the actual usage than the max value
| SMALL_TRAIN_SET_SIZE = 1000 | ||
| SMALL_TEST_SET_SIZE = 1000 |
There was a problem hiding this comment.
names still imply that they are generally used. I was thinking more something along the lines of MULTI_TEXT_TRAIN_SET_SIZE
There was a problem hiding this comment.
I would prefer no to have per-dataset default training/test set size
Add a new data module loading collection of 5 public text datasets called domains.
The dataset will be added to the benchmark in a follow-up PR.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.