How is long text handled?

[YanDavKMS]

Hey,

I'm trying to understand, what happens if we send a long text, which is longer from the model max length?
Will it be truncated by the tokenizer?
If not, what happens if the model gets a longer text than it's max length?

Thanks.

[michaelfeil]

Hi @YanDavKMS it will be truncated, same behaviour as sentence-transformers. Longest is truncated first, for bge-large-v1.5 this would result only in the first 512 tokens to be embedded.

[michaelfeil]

The sententence gets tokenized twice. Once for counting the tokens of the sentence correctly (691). If you use the debugger, you will notice that the actual tokens / pytorch input_ids in the forward pass is 512.

[YanDavKMS]

Thank you very much for the explanation.

Last point. from the usage perspective perhaps it is better to return (also) the real tokens usage.

[michaelfeil]

What's the rationale behind that, I am curious?

[YanDavKMS]

There are a few reasons why I would like to know how many tokens did I really run the model on

• For statistics and performance measurements
• For charging purposes, same as if I would use OpenAI embeddings, I get the real tokens usage
• It would make it clearer that even though I sent a longer text I actually got embedding on truncated text

[michaelfeil]

Thanks for the insights. I'll slowly pivot in this direction.