Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

text-dedup 去重效果怎么样 #91

Closed
maoxiangyi opened this issue Jun 17, 2024 · 2 comments
Closed

text-dedup 去重效果怎么样 #91

maoxiangyi opened this issue Jun 17, 2024 · 2 comments
Labels
no-issue-activity

Comments

@maoxiangyi
Copy link

text-dedup 去重效果怎么样
和huggingface的datarove项目有对比过吗
@ChenghaoMou
Copy link
Owner

感谢提问。两者去重的底层算法一致，效果应该看具体参数。本项目的表现可以参考README中的Benchmark结果。项目代码大部分来源于本人在BigScience和BigCode中的实验，而且主要的关注方向就是去重，而datatrove包含去重之外的数据清理逻辑，看个人喜好吧。我没有做过两者的实际对比。

@github-actions GitHub Actions
Copy link

Stale issue message

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Aug 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
no-issue-activity
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@maoxiangyi @ChenghaoMou