This repository has been archived by the owner on Mar 1, 2024. It is now read-only.
Enhancing Model Compatibility and Functionality: Integrating Huggingface's Transformers Library into Pull Request #130
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit integrates the Huggingface's transformers library to replace pytorch-transformers in the cross-encoder and bi-encoder scripts. This improves compatibility with newer models and extends the functionality range of the BERT model. Additionally, it fixes the issue about loading models with GPU/CPU device compatibility. This integration also simplifies the process of resizing the token embeddings. A new evaluate.py script is also included in this commit. The script allows easy evaluation of test, train, and valid results, hence improving the workflow of conducting experiments. Furthermore, we have included a tutorial to guide individuals on how to train new models, getting top-k predictions, training the cross-encoder, making necessary adjustments in the evaluate.py, and evaluating the overall work."
This commit updates how BERT model outputs are handled in ranker_base.py, aligning with updates in the Huggingface's transformers library usage. Additionally, for GPU/CPU compatibility, data is moved to appropriate device before model computation in main_dense.py. Also, this commit adds a JSON file format validation in main_dense.py when loading model configurations. In case there is a JSONDecodeError, it fixes the quirk by converting the file content to valid JSON format."
This commit refactors the hard-coded BERT token constants ([CLS], [SEP], etc.) by extracting them into an external params file. This makes the code more maintainable and less error-prone as these tokens are now defined only in one place. Simultaneously, improved the device compatibility of loading model by specifically mapping the loaded state_dict to the correct device. This fixes potential issues with loading GPU trained models on CPU only environments. Finally, added arguments for selecting different BERT models (eg. Roberta) and implemented necessary code changes to handle these models correctly."
facebook-github-bot
added
the
CLA Signed
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In this pull request, we have made significant improvements to our codebase by integrating Huggingface's transformers library, effectively replacing pytorch-transformers in both the cross-encoder and bi-encoder scripts. This change brings about several vital enhancements that expand the capabilities and compatibility of our BERT model.
First and foremost, this update significantly improves compatibility with the latest models available in the field of natural language processing. By incorporating Huggingface's transformers library, we ensure that our codebase remains up-to-date and ready to harness the power of cutting-edge language models.
One of the significant benefits of this integration is the extension of the functionality range of our BERT model. With Huggingface's library in place, we can tap into a wider array of pre-trained models and fine-tuning options, allowing us to adapt our model to a broader spectrum of tasks and applications. This not only enhances the versatility of our code but also opens up new possibilities for leveraging state-of-the-art language models in our projects.
Furthermore, this pull request addresses a critical issue related to model loading. With the integration of Huggingface's transformers library, we have resolved the problem of GPU/CPU device compatibility when loading models. This fix ensures a seamless experience for users, whether they are running our code on GPU-accelerated hardware or CPU-based systems.