ColBERT support

[kjoth]

Hi, Have few queries.

Does TEI currently offer support for ColBERT? If not, are there any plans to provide support in the future?
Additionally, does TEI support all of the HF Embedding/Reranking models listed or only those specifically mentioned?
Lastly, does TEI support custom fine-tuned models?

Thanks

[OlivierDehaene]

Does TEI currently offer support for ColBERT

It doesn't.

Does TEI support all of the HF Embedding/Reranking models

TEI support Bert, Roberta, Nomic and Jina based embeddings models. The one listed are examples.

Does TEI support custom fine-tuned models?

Yes as long as the models have one of the above architecture and respect the Transformers layout.

[kjoth]

Thanks for clarifying.

[kjoth]

@OlivierDehaene : Follow up question

1. Do we have roadmap for supporting ColBert.
2. Does TEI can be used within orchestration tools like Kserve, Seldon core.

[OlivierDehaene]

1. Colbert will be supported as an embedding model. The client will then be responsible for creating the index.

2. I don't know. I never used these tools and frankly don't understand why they are needed. It would be easy to support TEI with KNative.

[kjoth]

Thank you!

[kjoth]

Hi @OlivierDehaene , I'm curious, why is it necessary to use different platforms or tools for embedding model inference compared to large language model (LLM) inference? Could the text generation inference capabilities provided by platforms like Hugging Face also serve for embedding model inference? What's the rationale behind needing separate tools for text embedding model inference?