-
Notifications
You must be signed in to change notification settings - Fork 26.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add ESM to huggingface #13662
Add ESM to huggingface #13662
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks a lot for your PR! I left a few comments but it's already in pretty good shape.
I have a few more general interrogations:
- The tokenizer fast is imported in the main inits and used in the doc, but it does not exist in the files, so you should either add the tokenizer fast or remove any mention of it.
- Do all the model head make sense for this new architecture? From what I understand it's linked to proteins so I don't know if the task-specific heads like multiple choice or question answering really are useful.
- There should be a file to test the tokenizer
7ca02c6
to
fdc582a
Compare
@sgugger , thanks for the feedback!
|
It doesn't look like you added your model in the configuration_auto mappings, just the modeling_auto mappings. That's why you get this error. |
b61acce
to
fc0f4bd
Compare
Thanks! I think this is ready for review again (rebased to upstream) |
import torch | ||
|
||
import esm as esm_module | ||
from transformers.models.bert.modeling_bert import ( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
could we maybe replace those classes by the corresponding ESM...
classes, e.g. ESMIntermediate, ...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This would still be nice to change before merging
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I fixed a couple of things (corrected the config and removed dead code of ESMForQuestionAnswering) - hope that was ok!
PR is good for merge IMO. It would be nice if we could replace the BERT modules with ESM modules in the conversion script and then we should probably also flip the private model repo to public so that the integration test can run :-)
Should we upload the other checkpoitns as well: https://github.com/facebookresearch/esm#pre-trained-models-
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice, this looks good to me! Thank you for working on this!
Hey @liujas000, Can we help you in any way to get this PR merged? :-) |
@patrickvonplaten sorry for the delay; I will land this week! |
Co-authored-by: Lysandre Debut <lysandre@huggingface.co>
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
4bfad1a
to
d1b4640
Compare
Test failure seems unrelated |
Thanks a lot for making this PR more or less mergeable @liujas000 . I think there are just some final comments from @sgugger and @patrickvonplaten to be taken care of and the PR is good to go :-) |
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Please note that issues that do not follow the contributing guidelines are likely to be ignored. |
I'd love to use this model, what still needs to be done to get this merged? |
@gianhiltbrunner We're still waiting on an internal review from Facebook from the contributors, I believe! I'll let you know if there's any update. |
I would also be very happy if this gets merged! Any progress? |
Any updates here? |
What does this PR do?
Adding ESM-1b to huggingface following the steps in https://huggingface.co/transformers/add_new_model.html
Before submitting
Pull Request section?
to it if that's the case.
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@sgugger