Add delete and ensure add_texts performs upsert (w/ ID optional)

[rlancemartin]

Goal

We want to ensure consistency across vectordbs:
1/ add delete by ID method to the base vectorstore class
2/ ensure add_texts performs upsert with ID optionally passed

Testing

• Pinecone: notebook test w/ langchain_test vectorstore.
• Chroma: Review by @jeffchuber, notebook test w/ in memory vectorstore.
• Supabase: Review by @copple, notebook test w/ langchain_test table.
• Weaviate: Notebook test w/ langchain_test index.
• Elastic: Revied by @vestal. Notebook test w/ langchain_test table.
• Redis: Asked for review from owner of recent delete method added redis method to delete entries by keys #6222

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
langchain	⬜️ Ignored (Inspect)			Jun 23, 2023 6:35pm

[vercel]

@rlancemartin is attempting to deploy a commit to the LangChain Team on Vercel.

A member of the Team first needs to authorize it.

[rlancemartin]

This is unfortunate. May slow adds considerably. Need to see if there is a better way.

[jeffchuber]

lgtm! 👍 🎉

[hwchase17]

lets just change this, since its an arg and idk how well used i dont think we need to worry about backwards compat

[michaeltansg]

@rlancemartin Is it also the intention for this change to standardize the vector DB ids to be UUIDs? The documentation for the supabase integration ( python / javascript ) suggests BIGSERIAL as the type. The underlying database is PostgreSQL with a PGVector extension.

At this moment, if we call the add_documents method, it does not send a list of ids to add_texts method.

base.py:
    def add_documents(self, documents: List[Document], **kwargs: Any) -> List[str]:
        """Run more documents through the embeddings and add to the vectorstore.

        Args:
            documents (List[Document]: Documents to add to the vectorstore.

        Returns:
            List[str]: List of IDs of the added texts.
        """
        # TODO: Handle the case where the user doesn't provide ids on the Collection
        texts = [doc.page_content for doc in documents]
        metadatas = [doc.metadata for doc in documents]
        return self.add_texts(texts, metadatas, **kwargs)

supabase.py:
    def add_texts(
        self,
        texts: Iterable[str],
        metadatas: Optional[List[dict[Any, Any]]] = None,
        ids: Optional[List[str]] = None,
        **kwargs: Any,
    ) -> List[str]:
        ids = ids or [str(uuid.uuid4()) for _ in texts]
        docs = self._texts_to_documents(texts, metadatas)

        vectors = self._embedding.embed_documents(list(texts))
        return self.add_vectors(vectors, docs, ids)

The add_texts method then creates UUIDs and the upsert will fail if the table was declared like so:

-- Create a table to store your documents
create table documents (
  id bigserial primary key,
  content text,
  metadata jsonb,
  embedding vector(1536)
);

Any recommendations here?

[rlancemartin]

Ah, I see. Will BIGSERIAL always be used as Supabase primary key? If so, we can modify ID. Feel free to put up a PR!