Uploading Vector data to Qdrant Server

[harshakhmk]

I am using Qdrant Client to upload an array of vector data where each vector length is 785, how to add this bulk of vectors to Qdrant Server using Qdrant Client python package

[joein]

Hi @harshakhmk

You can either construct batches on your own and use upsert API

or you can rely on qdrant-client and use methods like upload_records or upload_collection

upload_collection and upload_records methods also have parallel param, which allows to seamlessly parallelise (via multiprocessing) uploading

[joein]

Hi @harshakhmk, does it help?

[harshakhmk]

Hi, I tried upload _records and I was getting pydantic validation errors

pydantic_core._pydantic_core.ValidationError: 1 Validation Error for Record payload
Input should be a valid dictionary [ type=dict_type, input_value = array([ 0,0,0... 0,0]), input_type=ndarray]
For further information, visit https://errors.pydantic.dev/2.5/v/dict_type

[joein]

Could you share a code sample to reproduce?
How do you make an upload_records call?

[harshakhmk]

import pandas as pd
df=pd.read_csv("test.csv")
Vectors1=[]
for idx in range (len(df)):
   item=df.iloc[idx]
   vectors1.append(item)



client.upload_records(
collection_name="dataset",
records=[
models. Record(
id=idx , vector=doc.tolist(), payload=doc
)
for idx, doc in enumerate(vectors1)
],
)

[joein]

payload=doc is the source of error I believe

it should be a dict, but you are uploading a numpy array there

you need to do something like payload={"vector": doc.tolist()}

btw, are you sure you want to store vector in the payload?

[harshakhmk]

Yes, that was the issue, I tried other way to insert data using upsert function, since it cannot insert all vectors at once, I had wrapped insertion of data to server inside a function and called this function for each vector, I had observed huge difference in adding whole collection to database with those two approaches
upload_records is much faster

[harshakhmk]

But, what will be the batch size with upload_records as we are not explicitly defined

[joein]

upsert function presumes manual batching

upload_records contains a default parameter batch_size: int = 64

To speed up things even more you can set parallel: int = <number of processes> which will spread the load across CPU processes

One thing to be aware of while using upload_records

If you don't provide ids explicitly, it will use consequent int ids (like 0 for the first record, 1 for the second, etc.)

If you call it twice - then on the second time it will overwrite the points which were written before

To sum up: if you provide ids explicitly - never mind, if you don't provide ids explicitly - be careful while calling to upload_records the second time