Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix LAMBADA for LLaMA #526

Closed
wants to merge 2 commits into from
Closed

Conversation

gakada
Copy link
Contributor

@gakada gakada commented May 26, 2023

0 accuracy in #443 is due to the space added in lm_eval/tasks/lambada.py, as LLaMA has no tokens for such spaces. Removing it gets

Task Version Metric Value Stderr
lambada_openai 0 ppl 3.6224 ± 0.0723
acc 0.7308 ± 0.0062

on LLaMA-7B.

@StellaAthena
Copy link
Member

can you check the performance of Pythia, GPT-2, and T5 using this modification? I want to make sure that we don’t add support for LLaMA at the expense of other models.

@gakada gakada marked this pull request as draft May 28, 2023 01:51
@gakada gakada closed this May 29, 2023
@gakada gakada deleted the lambada-llama branch May 29, 2023 13:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants