Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a vectordb module #2263

Merged
merged 42 commits into from
Apr 10, 2024
Merged

Add a vectordb module #2263

merged 42 commits into from
Apr 10, 2024

Conversation

thinkall
Copy link
Collaborator

@thinkall thinkall commented Apr 3, 2024

Why are these changes needed?

We've been asked for supporting different vector databases in RAG and teachability for many times, we can also foresee that different dbs could be useful to memory, history management, etc. in the future. In order to better support using different vectordbs in different scenarios, and also make it easier for accepting contributions from the community; a vectordb protocol is proposed in this PR.

As we've discussed, some basic APIs are defined in the protocol, any special APIs related to the vectordb itself can also be added in the implementation. @siluhuang, @HongleiZhuang, @Knucklessg1

This PR is also a step of the RAG refactor. #1657

Related issue number

Checks

.github/workflows/contrib-tests.yml Outdated Show resolved Hide resolved
test/agentchat/contrib/vectordb/test_chromadb.py Outdated Show resolved Hide resolved
.github/workflows/contrib-openai.yml Outdated Show resolved Hide resolved
@sonichi
Copy link
Contributor

sonichi commented Apr 10, 2024

@jackgerrits this PR is blocked by your change request. Could you take a look?

Copy link
Member

@jackgerrits jackgerrits left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still worry if some of the loose parts of this interface will cause problems, but I am happy to see how things shake out in practice

@sonichi sonichi added this pull request to the merge queue Apr 10, 2024
Merged via the queue into main with commit adb8fd8 Apr 10, 2024
71 of 75 checks passed
@sonichi sonichi deleted the refactor_abstract_vectordb branch April 10, 2024 17:56
@sonichi sonichi requested review from cozypet and wmwxwa April 11, 2024 15:14
whiskyboy pushed a commit to whiskyboy/autogen that referenced this pull request Apr 17, 2024
* Added vectordb base and chromadb

* Remove timer and unused functions

* Added filter by distance

* Added test utils

* Fix format

* Fix type hint of dict

* Rename test

* Add test chromadb

* Fix test no chromadb

* Add coverage

* Don't skip test vectordb utils

* Add types

* Fix tests

* Fix docs build error

* Add types to base

* Update base

* Update utils

* Update chromadb

* Add get_docs_by_ids

* Improve docstring

* Add get all docs

* Move chroma_results_to_query_results to utils

* Improve type hints

* Update logger

* Update init, add embedding func

* Improve docstring of vectordb, add two attributes

* Improve test workflow
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants