ModernBERT #2250
-
|
Recently a new version of BERT, ModernBERT has been released, and I have a couple of questions:
Thanks in advance. |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 3 replies
-
|
Thank you for sharing. It is a great foundational/base model! That said, we typically do not use base BERT-like for creating the embeddings; we fine-tune them instead. This likely means we need to wait until someone fine-tunes it specifically use contrastive learning to create embeddings that are more suited towards tasks like clustering. That said, if you want to use it as is, you can load it in a Transformers pipeline and use it in BERTopic: https://maartengr.github.io/BERTopic/getting_started/embeddings/embeddings.html#hugging-face-transformers |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the answer, and for Bertopic. I will keep an eye on fine-tuned models based on ModernBERT and experiment with them when they become available |
Beta Was this translation helpful? Give feedback.
-
|
Also note that typically the best place to look for embedding models is the MTEB leaderboard. It's not a perfect leaderboard, but if there are fine-tunes of ModernBERT through methods like contrastive learning, you will find them there. |
Beta Was this translation helpful? Give feedback.
-
|
Some time has passed, and there's now a bunch of options! Some particularly promising ones are:
The bottom 2 are extra custom. Just based on a guess, I reckon that the 2nd to last one will still work, and the last one won't.
|
Beta Was this translation helpful? Give feedback.
-
|
@tomaarsen Thanks for sharing! Now that I have you here, any recommendations on new default English/Multilingual models? I'm considering the following for English models (good speed/acc balance):
A Multilingual alternative is tricky as the default paraphrase model is already quite good for clustering purposes... |
Beta Was this translation helpful? Give feedback.
Thank you for sharing. It is a great foundational/base model! That said, we typically do not use base BERT-like for creating the embeddings; we fine-tune them instead. This likely means we need to wait until someone fine-tunes it specifically use contrastive learning to create embeddings that are more suited towards tasks like clustering.
That said, if you want to use it as is, you can load it in a Transformers pipeline and use it in BERTopic: https://maartengr.github.io/BERTopic/getting_started/embeddings/embeddings.html#hugging-face-transformers