Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat: Utils add wipe qdrant and new stats command #1783

Merged
merged 6 commits into from
Apr 2, 2024

Conversation

dbzoo
Copy link
Contributor

@dbzoo dbzoo commented Mar 22, 2024

Add support for deleting the collection created in a qdrant vector datastore.

vectorstore:
  database: qdrant

I did not change the name of the store from make_this_parameterizable_per_api_call for upgrade compatibility.

This collection should be named something more appropriate. I suggest data_embeddings. This is how its named in the Postgres database. Renaming this store will be a breaking upgrade for those who have ingested data.

Test output

% ls local_data/private_gpt/qdrant/collection 
make_this_parameterizable_per_api_call
% PGPT_PROFILES=local make wipe
poetry run python scripts/utils.py wipe
20:31:23.378 [INFO    ] private_gpt.settings.settings_loader - Starting application with profiles=['default', 'local']
Collection dropped successfully.
% ls local_data/private_gpt/qdrant/collection
%

Stats command

This gives some simple metrics about the data, index and vector stores.

QDRANT

% PGPT_PROFILES=local  python scripts/utils.py stats
16:19:16.360 [INFO    ] private_gpt.settings.settings_loader - Starting application with profiles=['default', 'local']
Unable to execute command 'stats' on 'nodestore' in database 'simple'
Storage for Qdrant vectorstore.
	Points:        16,453
	Vectors:       16,453
	Index Vectors: 0

POSTGRES

% PGPT_PROFILES=brett  python scripts/utils.py stats
15:37:53.301 [INFO    ] private_gpt.settings.settings_loader - Starting application with profiles=['default', 'brett']
Storage for Postgres nodestore.
Table           |            Rows |      Size
---------------------------------------------
data_docstore   |       3,562,741 |   6209 MB
data_indexstore |               1 |    143 MB

Storage for Postgres vectorstore.
Table           |            Rows |      Size
---------------------------------------------
data_embeddings |       1,701,607 |     10 GB

@dbzoo dbzoo changed the title Feat wipe qdrant Feat: wipe qdrant and add stats command Mar 22, 2024
@dbzoo dbzoo changed the title Feat: wipe qdrant and add stats command Feat: Utils add wipe qdrant and new stats command Mar 22, 2024
Copy link
Collaborator

@imartinez imartinez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really neat!

@imartinez imartinez merged commit ea153fb into zylon-ai:main Apr 2, 2024
6 checks passed
mrepetto-certx pushed a commit to mrepetto-certx/privateGPT that referenced this pull request Apr 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants