Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automatically execute scripts/create_summary_table.py to reproduce average scores from paper #246

Closed
danielkorat opened this issue Dec 27, 2022 · 2 comments · Fixed by #262
Assignees
Labels
enhancement New feature or request

Comments

@danielkorat
Copy link
Collaborator

The script in scripts/create_summary_table.py must be used to reproduce SetFit / baseline results from the paper. It calculates per-dataset average over the 10 training splits and saves the summary table to a file. However, this is scripts is not mentioned anywhere in the repo. It will be nice if it can be run automatically at the end of each SetFit / baseline run.

@danielkorat danielkorat added the enhancement New feature or request label Dec 27, 2022
@danielkorat danielkorat changed the title Automatically execute scripts/create_summary_table.py to reproduce average score from paper Automatically execute scripts/create_summary_table.py to reproduce average scores from paper Dec 27, 2022
@lewtun
Copy link
Member

lewtun commented Dec 28, 2022

Thanks for catching this @danielkorat! Would you like to open a PR with the fix?

@tomaarsen
Copy link
Member

tomaarsen commented Dec 28, 2022

I'm fine with this simply being a documentation change, too. Adding information about the script to https://github.com/huggingface/setfit/blob/main/scripts/setfit/README.md may be sufficient, if that is easier than automatically running the script at the end of run_fewshot.py.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants