1.19.0

1.19.0 (2024-11-06)

Feature

• feat: Standardize descriptive stats (#1375)

• init

• mock tests

• remove debug prints

• remove descriptive stats and move to n_samples

• fix typo

• fix create task table

• fix citations

• remove n_samples

• metadata per task

• add test

• reformat task table

• add n_samples property

• fix tests

• rename total_symbols to number_of_characters

• Update mteb/abstasks/AbsTaskRetrieval.py

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsen@gmail.com>

• rename in tests

• lint

---

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsen@gmail.com> (2854fa2)

Unknown

• Update tasks table (537b974)

1.18.9

1.18.9 (2024-11-06)

Fix

• fix: Disable rich output with verbosity=0 on evaluation.run (#1395)

• feat: verbose=0 now supress rich console output

• fix: removed unnecesary list comprenhension (1bb1ca3)

1.18.8

1.18.8 (2024-11-04)

Fix

• fix: Update logging verbosity levels in MTEB (#1384)

• Fix verbosity handling in MTEB.py for consistent logging

• updates

• update docstrings

• linting code (35daf58)

1.18.7

1.18.7 (2024-11-04)

Fix

• fix: Leaderboard UI improvements (#1370)

• Added elementary search bar with RegEx functionality

• Improved layout

• Table improvements, adjusted column width and added links

• Update benchmark filter description

Co-authored-by: Isaac Chung <chungisaac1217@gmail.com>

• Added instructions on multiple search queries

Co-authored-by: Isaac Chung <chungisaac1217@gmail.com>

---

Co-authored-by: Isaac Chung <chungisaac1217@gmail.com> (92fe9cb)

1.18.6

1.18.6 (2024-10-31)

Fix

• fix: Integrate prompts to task metadata (#1300)

• init

• add DatasetDict

• add classification

• add clustering

• add pair classification

• add retrieval

• add all prompts

• start integrating prompts

• refactor instruct models

• lint

• fix test

• fix

• fix no prompt in prompt dict

• add more logging

• add more logging

• Apply suggestions from code review

Co-authored-by: Isaac Chung <chungisaac1217@gmail.com>

• fix code review

• fix use_instructions

• add log if instruction template not set

• fix metadata

• lint

• fix brazilian

• remove MetadataDatasetDict

• rollback test metadata

---

Co-authored-by: Isaac Chung <chungisaac1217@gmail.com> (029d378)

1.18.5

1.18.5 (2024-10-31)

Fix

• fix: Speed up leaderboard by caching and skipping validation (#1365)

• Made loading and filtering faster by removing unnecessary validation

• Made select_tasks faster by removing validation

• Added caching to leaderboard

• Ran linting

• Added missing future import (f1bc375)

1.18.4

1.18.4 (2024-10-30)

Fix

• fix: make sure test is the default split for FEVER (#1361)

The other splits can still be run as long as they are specified. (d9626ab)

1.18.3

1.18.3 (2024-10-30)

Fix

• fix: Update KorSarcasm to avoid trust-remote code (#1364) (756ba7e)

Unknown

• Leaderboard updates: Model meta + task and benchmark info (#1345)

• Added benchmark description and citation to leaderboard

• Added model information to main table

• Fixed citation box

• Added table tab with task information

• Added button for benchmark link if specified

• Formatted model column in per_task table properly

• Implemented model filtering based on metadata

• Fixed maximum minimum model sizes

• Ran linting

• Replaced mean rank with borda rank in main table (298b0bd)

1.18.2

1.18.2 (2024-10-30)

Fix

• fix: upload BrazilianToxicTweetsClassification to hf (#1352)

upload to hf (9c7a1c2)

1.18.1

1.18.1 (2024-10-30)

Fix

• fix: Add jina, uae, stella models (#1319)

• add models

• fix

• fix

• fix prompt

• Update mteb/models/jina_models.py

Co-authored-by: Wang Bo <bo.wang@jina.ai>

• Update mteb/models/jina_models.py

Co-authored-by: Wang Bo <bo.wang@jina.ai>

• try reeval stella

• change to e5

• change to e5

• add metadata

• update languages

• Update mteb/models/jina_models.py

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsen@gmail.com>

• remove docstring

• remove trust remote

• update model meta

• Set minimal version

---

Co-authored-by: Wang Bo <bo.wang@jina.ai>
Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsen@gmail.com> (0b846ff)

• fix: remove accidentally commited file (16a333e)