Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add mteb evaluation #8538

Merged
merged 5 commits into from
Jun 5, 2024
Merged

add mteb evaluation #8538

merged 5 commits into from
Jun 5, 2024

Conversation

cxa-unique
Copy link
Contributor

PR types

New features

PR changes

Others

Description

Add the evaluation scripts for MTEB benchmark in the pipeline example "contrastive_training"

Copy link

paddle-bot bot commented Jun 4, 2024

Thanks for your contribution!

@@ -0,0 +1,97 @@
# MTEB基准评估
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

把meteb的代码和原始的evaluation目录进行合并,不单独开一个目录,结构参考:
image

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

在原evaluation下加了mteb目录

## 模型评估
使用评估脚本`eval_mteb.py`:

- `base_model_name_or_path`: 模型名称或路径
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

把这个README.md和主readme进行融合,维护主readme即可

Copy link
Contributor Author

@cxa-unique cxa-unique Jun 5, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已与主README融合,并已测试过

@w5688414
Copy link
Contributor

w5688414 commented Jun 4, 2024

代码融合后,需要检查一下,原来的readme里面的内容是否还能跑通

@w5688414 w5688414 assigned w5688414 and unassigned wawltor Jun 4, 2024
@w5688414 w5688414 requested a review from sijunhe June 4, 2024 09:35
@w5688414 w5688414 assigned cxa-unique and unassigned w5688414 Jun 4, 2024
Copy link

codecov bot commented Jun 5, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 53.97%. Comparing base (f36ed75) to head (c9379e8).
Report is 247 commits behind head on develop.

Additional details and impacted files
@@           Coverage Diff            @@
##           develop    #8538   +/-   ##
========================================
  Coverage    53.97%   53.97%           
========================================
  Files          618      618           
  Lines        96827    96827           
========================================
+ Hits         52258    52259    +1     
+ Misses       44569    44568    -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

是一个大规模文本嵌入评测基准,包含了丰富的向量检索评估任务和数据集。
本仓库主要面向其中的中英文检索任务(Retrieval),并以SciFact数据集作为主要示例。

使用评估脚本`evaluation/mteb/eval_mteb.py`:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

参数的解释说明写到执行命令的后面,跟上面的写法保持一致。
image

Copy link
Contributor

@w5688414 w5688414 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Collaborator

@sijunhe sijunhe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@sijunhe sijunhe merged commit 1cf780e into PaddlePaddle:develop Jun 5, 2024
9 of 12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants