We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
text-dedup 去重效果怎么样 和huggingface的datarove项目有对比过吗
The text was updated successfully, but these errors were encountered:
感谢提问。两者去重的底层算法一致,效果应该看具体参数。本项目的表现可以参考README中的Benchmark结果。项目代码大部分来源于本人在BigScience和BigCode中的实验,而且主要的关注方向就是去重,而datatrove包含去重之外的数据清理逻辑,看个人喜好吧。我没有做过两者的实际对比。
Sorry, something went wrong.
Stale issue message
No branches or pull requests
text-dedup 去重效果怎么样
和huggingface的datarove项目有对比过吗
The text was updated successfully, but these errors were encountered: