Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Added tasks from SEB #287

Merged
merged 5 commits into from
Mar 29, 2024
Merged

fix: Added tasks from SEB #287

merged 5 commits into from
Mar 29, 2024

Conversation

KennethEnevoldsen
Copy link
Contributor

Added all tasks from SEB. I have not added points for these as we discussed not adding points for items which have already been published on.

Checklist for adding MMTEB dataset

  • I have tested that the dataset runs with the mteb package.
  • I have run the following models on the task (adding the results to the pr). These can be run using the mteb run -m {model_name} -t {task_name} command.
    • sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
    • intfloat/multilingual-e5-small
  • I have checked that the performance is neither trivial (both models gain close to perfect scores) nor random (both models gain close to random scores).
  • I have considered the size of the dataset and reduced it if it is too big (2048 examples is typically large enough for most tasks)
  • Run tests locally to make sure nothing is broken using make test.
  • Run the formatter to format the code using make lint.
  • I have added points for my submission to the POINTS.md file.

@KennethEnevoldsen KennethEnevoldsen changed the title Added tasks from SEB fix: Added tasks from SEB Mar 26, 2024
@KennethEnevoldsen KennethEnevoldsen merged commit 39cff49 into main Mar 29, 2024
4 checks passed
@KennethEnevoldsen KennethEnevoldsen deleted the add-seb branch March 29, 2024 12:57
MartinBernstorff pushed a commit that referenced this pull request Apr 10, 2024
* Added tasks from SEB

* docs: fix link

* fix: ran linting

* fix typing for 3.8

* fixed annotation for v3.8
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants