Generating the same embeddings for all tokens

[makwana050]

I am trying to use the SAILER_en model from the Hugging Face model hub to generate embeddings for my legal documents. However, I am having trouble generating proper embeddings using this model. The embeddings for each token in my document are the same, regardless of the content of the document. I am simply loading the model and trying to fetch the embedding from last hidden state while passing the input tokens.

I noticed that the sample code provided by Hugging Face for using the SAILER_en model with the transformers library references a BertForCotMAE class that does not exist in the library. I am not sure if this is related to the issue I am experiencing.

Can you provide any guidance on how to use the SAILER_en model correctly to generate embeddings for legal documents? Any help would be greatly appreciated.

Thank you