[Re-Opened] Support for PGVector Database in Autogen

[Knucklessg1]

Why are these changes needed?

Adding support for PGVector database for RAG Agents.

This will allow a user to connect to an existing PGVector instance.
Collections can be created, deleted, modified, or updated.
Existing collections can be used.

Supports Euclidean, cosine, and inner distance indexing.

Related issue number

This PR was re-opened from the following PR #2373 Support for PGVector Database in Autogen to run with OpenAI CI job.

These changes are dependent on the following merge request to refactor vector_db as param: PR #2313 Support setting vector_db as a param

@thinkall is driving the initial refactoring for rag agents in this PR.

How to test

Deploy a PGVector instance if needed.

docker-compose.yml

version: '3.9'

services:
  db:
    hostname: db
    image: ankane/pgvector
    ports:
      - 5432:5432
    restart: always
    environment:
      - POSTGRES_DB=vectordb
      - POSTGRES_USER=testuser
      - POSTGRES_PASSWORD=testpwd
      - POSTGRES_HOST_AUTH_METHOD=trust
    volumes:
      - ./init.sql:/docker-entrypoint-initdb.d/init.sql

Create init.sql file

CREATE EXTENSION IF NOT EXISTS vector;

docker compose up --d

Run PGVector RAG Agent:

test_pgvector_rag.py

import os

import autogen
from autogen.agentchat.contrib.retrieve_assistant_agent import RetrieveAssistantAgent
from autogen.agentchat.contrib.retrieve_user_proxy_agent import RetrieveUserProxyAgent

# Accepted file formats for that can be stored in
# a vector database instance
from autogen.retrieve_utils import TEXT_FORMATS

config_list = autogen.config_list_from_json(env_or_file="OAI_CONFIG_LIST")

assert len(config_list) > 0
print("models to use: ", [config_list[i]["model"] for i in range(len(config_list))])

print("Accepted file formats for `docs_path`:")
print(TEXT_FORMATS)

assistant = RetrieveAssistantAgent(
    name="assistant",
    system_message="You are a helpful assistant.",
    llm_config={
        "timeout": 600,
        "cache_seed": 42,
        "config_list": config_list,
    },
)

ragproxyagent = RetrieveUserProxyAgent(
    name="ragproxyagent",
    human_input_mode="NEVER",
    max_consecutive_auto_reply=3,
    retrieve_config={
        "task": "code",
        "docs_path": [
            "https://raw.githubusercontent.com/microsoft/FLAML/main/website/docs/Examples/Integrate%20-%20Spark.md",
            "https://raw.githubusercontent.com/microsoft/FLAML/main/website/docs/Research.md",
            "https://raw.githubusercontent.com/Knuckles-Team/geniusbot/main/README.md",
            "https://raw.githubusercontent.com/Knuckles-Team/repository-manager/main/README.md",
            "https://raw.githubusercontent.com/Knuckles-Team/gitlab-api/main/README.md",
            "https://raw.githubusercontent.com/Knuckles-Team/media-downloader/main/README.md",
            os.path.join(os.path.abspath(""), "..", "website", "docs"),
        ],
        "custom_text_types": ["non-existent-type"],
        "chunk_token_size": 2000,
        "model": config_list[0]["model"],
        "vector_db": "pgvector",  # PGVector database
        "collection_name": "test_collection",
        "db_config": {
            "connection_string": "postgresql://testuser:testpwd@localhost:5432/vectordb", # Optional - connect to an external vector database
            # "host": None, # Optional vector database host
            # "port": None, # Optional vector database port
            # "database": None, # Optional vector database name
            # "username": None, # Optional vector database username
            # "password": None, # Optional vector database password
        },
        "get_or_create": True,  # set to False if you don't want to reuse an existing collection
        "overwrite": False,  # set to True if you want to overwrite an existing collection
    },
    code_execution_config=False,  # set to False if you don't want to execute the code
)

# reset the assistant. Always reset the assistant before starting a new conversation.
assistant.reset()

# given a problem, we use the ragproxyagent to generate a prompt to be sent to the assistant as the initial message.
# the assistant receives the message and generates a response. The response will be sent back to the ragproxyagent for processing.
# The conversation continues until the termination condition is met, in RetrieveChat, the termination condition when no human-in-loop is no code block detected.
# With human-in-loop, the conversation will continue until the user says "exit".
code_problem = "How can I use FLAML to perform a classification task and use spark to do parallel training. Train for 30 seconds and force cancel jobs if time limit is reached."
ragproxyagent.initiate_chat(
    assistant, message=ragproxyagent.message_generator, problem=code_problem, search_string="spark"
)

python ./test_pgvector_rag.py

Checks

• I've included any doc changes needed for https://microsoft.github.io/autogen/. See https://microsoft.github.io/autogen/docs/Contribute#documentation to build and test documentation locally.
• I've added tests (if relevant) corresponding to the changes introduced in this PR.
• I've made sure all auto checks have passed.

[sonichi]

LGTM. Left one optional comment.

[thinkall]

@Knucklessg1 @sonichi There are still issues in the code, it's not ready for merging.

• The notebook has not been ran. I'd suggest taking the first two examples from RetrieveChat notebook instead of using all of them, and actually run the notebook.
• As I tested locally, the current RetrieveChatTest would skip tests for pgvector as from autogen.agentchat.contrib.vectordb.pgvector import PGVector should be from autogen.agentchat.contrib.vectordb.pgvectordb import PGVectorDB, other contents should be corrected as well. And I see AttributeError: 'NoneType' object has no attribute 'cursor' after I corrected the import and naming.
• The Install and Start PostgreSQL in CI doesn't work for me either.