-
Notifications
You must be signed in to change notification settings - Fork 16
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Issue with overall_score
computation
#8
Comments
Hi! Sorry for the late response and thanks for pointing that out! Yes, it seems to be a bug, and the ">" should be ">=". Intuitively, a true |
Here's a space we've been using to verify this: (login with: owner/12345678) The only issue is that there's some examples with overall_score 10 that are good (the majority are bad though). We've been working on curating this data programmatically and with Argilla and we'd be super happy to contribute back Thanks for building an amazing project! |
Can't wait to see the updated dataset! |
Just updated the dataset, please check! |
Oh no, on our official page: https://huggingface.co/datasets/openbmb/UltraFeedback |
Hi!
Congrats on this amazing project.
We've been exploring the data and identified an issue with very high
overall_score
responses. The issue seems to be related with this line. This causes responses with a critique rating of1
to become a10
. We noticed this by looking at the critique rational which was highly negative for many (~2K) examples with anoverall_score
of10
.The text was updated successfully, but these errors were encountered: