Add NLP Components to Benchmarking #213
Conversation
Coverage reportThe coverage rate went from
Diff Coverage details (click to unfold)src/renate/benchmark/datasets/nlp_datasets.py
src/renate/benchmark/experiment_config.py
src/renate/memory/buffer.py
src/renate/training/training.py
src/renate/benchmark/models/transformer.py
src/renate/defaults.py
|
| from renate.models import RenateModule | ||
|
|
||
|
|
||
| class HuggingFaceSequenceClassificationTransformer(RenateModule): |
There was a problem hiding this comment.
Is that necessary or could we just use a RenateWrapper, as I did in the NLP example? I'm fine with having it, as it might be a bit more convenient. But there's always a trade-off with additional code to maintain etc...
There was a problem hiding this comment.
the RenateWrapper does not allow to save any additional arguments as in this case pretrained_model_name and num_outputs.
| @@ -67,4 +67,4 @@ | |||
|
|
|||
| # Noticed different accuracy scores across Mac and GitHub Actions Workflows (which run on Linux) | |||
| # TODO see if we can align the Mac and Linux results | |||
| assert pytest.approx(test_config["expected_accuracy"]) == accuracies | |||
| assert pytest.approx(test_config["expected_accuracy"]) == accuracies, accuracies | |||
There was a problem hiding this comment.
I don't understand this syntax?
There was a problem hiding this comment.
if fails, prints accuracies
| @@ -13,26 +12,24 @@ | |||
| from renate.models.renate_module import RenateWrapper | |||
|
|
|||
|
|
|||
| def model_fn(model_state_url: Optional[Union[Path, str]] = None) -> RenateModule: | |||
| def model_fn(model_state_url: Optional[str] = None) -> RenateModule: | |||
There was a problem hiding this comment.
If we're disallowing path objects here, should we also do that in the other examples?
There was a problem hiding this comment.
the use of complex types as Path is no longer supported for our *_fn functions since 0.2.0. I can check for more occurrences but I should have removed most of them before
There was a problem hiding this comment.
I see occurrences of Path in most example config files. If you don't want to remove then now, we should create an issue to do it later.
There was a problem hiding this comment.
I've updated the remaining config files as well as the documentation.
* Add NLP Components to Benchmarking (#213) * Robust Integration Tests (#214) * Update Renate Config Example (#226) * Make Wild Time Available in Benchmarking (#187) * Fix `target_column` bug in `HuggingFaceTextDataModule` (#233) * Add MMD covariate shift detector (#237) * Add KS covariate shift detector (#242) * Update dependabot.yml (#248) * Update versions of some requirements (#247) * Add doc page and example for shift detection (#244) * Bump version (#252) --------- Co-authored-by: Lukas Balles <lukas.balles@gmail.com>
Add RenateModule for Hugging Face transformer and make datasets and transformers available in benchmarking script. Tests are failing for other reasons :(
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.