Adding HendrycksTest dataset #2370
Conversation
There was a problem hiding this comment.
Thank you for adding this dataset !
The dataset python script looks very good, thanks :)
I added a few comments in the dataset card
I also noticed that the dummy data zip files were quite big (160KB each). Could to try to reduce their sizes please ? For example feel free to take a look inside the abstract_algebra dummy data zip file and remove all the csv files that are unrelated to abstract_algebra:
- formal_logic_val.csv
- prehistory_val.csv
- etc
If you could do that for all subjects that would be perfect :) Feel free to use a script to automate that of course. Thank you !
|
@lhoestq Thank you for the review. I've made the suggested changes. There still might be some problems with dummy data though due to some csv loading issues (which I haven't found the cause to). |
|
I took a look at the dummy data and some csv lines were cropped. I fixed them :) |
|
@andyzoujm Any reason why this dataset scrip was called "hendrycks_test" instead of "mmlu"? We are thinking of renaming it... |
|
That's because we didn't call it MMLU in the paper (the shorthand didn't
emerge until over a year later), and people at OpenAI were calling it that.
Andy
…On Wed, Apr 26, 2023 at 8:44 AM Albert Villanova del Moral < ***@***.***> wrote:
@andyzoujm <https://github.com/andyzoujm> Any reason why this dataset
scrip was called "hendrycks_test" instead of "mmlu"?
We are thinking of renaming it...
—
Reply to this email directly, view it on GitHub
<#2370 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AKLQJMZZFZBGJTBOIFWJ5KDXDEKB5ANCNFSM45BAOSIQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
Thanks for your reply. Just for the records: we have renamed it to "cais/mmlu": https://huggingface.co/datasets/cais/mmlu |
Adding Hendrycks test from https://arxiv.org/abs/2009.03300.
I'm having a bit of trouble with dummy data creation because some lines in the csv files aren't being loaded properly (only the first entry loaded in a row of length 6). The dataset is loading just fine. Hope you can kindly help!
Thank you!