Error setting userProvided embedder

[miguelisidoro]

Hello,

We are trying to set our index embedder to a userProvided approach.
This is our request and response.

However, we are getting the following error in the associated task;

Why are we getting this error? Do we have to delete all documents from the index before setting the embedder?

Thanks

[dureuill]

Why are we getting this error?

When adding a userProvided embedder, all existing documents need to have _vectors.<embedder-name> (here _vectors.default) defined to one or multiple embeddings (arrays of floats)

Do we have to delete all documents from the index before setting the embedder?

That's an option. You can also update all documents to add the missing _vectors.<embedder-name> field before adding your embedder.

[dureuill]

yes that's correct 👍

[miguelisidoro]

Hello @dureuill, thanks for the confirmation.

[miguelisidoro]

Hello @dureuill,

I just rebuilt my search index and all items have a _vectors field with a vector of dimension 1536, the size of the vector returned by Azure Open AI Embedding API.

However, when setting the user provided embedder, I get the following error:

The user provided embedder is being set like this:

Example of a document with the _vectors filled.

What am I doing wrong here?

Any way to get the document in the index mentioned in the error and checking if it has a valid vector?

Any way to get all documents that have null in the _vectors field?

Thanks

[dureuill]

Hello

What am I doing wrong here?

Looks like one or more of your documents doesn't have a correctly shaped _vectors field

Any way to get the document in the index mentioned in the error and checking if it has a valid vector?

Run GET http://<your-meilisearch-instance>/indexes/<your-index>/documents/Default_4488r fwm613nqy1ph9tw2jbvwd to get the document by its id mentioned in the error.

Any way to get all documents that have null in the _vectors field?

Not easily, I'm afraid. In v1.8, you could set _vectors as filterable, but this is strongly discouraged for performance reasons (it will index all the individual floats in the vectors) and won't be possible in v1.9 and higher (disabled due to the performance impact)

I'll be discussing this use case with the team because it is true that this is a desirable use case when adding a user provided embedder

[miguelisidoro]

Hello @dureuill,

Thanks for your response.
There are indeed some documents with _vectors = null.
I will solve this and test again.

Thanks