dev : adding proxy server for ingestion

[vipul-maheshwari]

Pull Request Summary for Ingestion Server Integration

• Added Ingestion Server: Introduced a new FastAPI-based ingestion server to handle RAG (Retrieval-Augmented Generation) configurations and file uploads.

• Dockerfile Creation: Created a ingestion_server.Dockerfile to containerize the ingestion server, ensuring it runs in a consistent environment.

• Updated Docker Compose: Modified docker-compose.yml to include the ingestion server as a service, allowing it to run alongside existing services (Bolna, Twilio, Plivo, Redis, and ngrok).

• API Endpoints:

  • Implemented a health check endpoint (GET /) to verify server status.
  • Added an endpoint (POST /create-rag) to create RAG configurations and return their IDs.
  • Developed an endpoint (POST /rag-upload-file/{rag_id}) for uploading files associated with specific RAG IDs.
  • Created a retrieval endpoint (GET /rag-retrive/{rag_id}/{index}) to fetch documents based on queries.

Payload for Ingestion Server

{
  "provider": "LanceDB",
  "provider_config": {
    "embedding": {
      "provider": "OpenAI",
      "embedding_model_name": "text-embedding-3-small"
    },
    "chunk_size": 512,
    "overlapping": 200,
    "worker": 2,
    "similarity_top_k":2,
    "rag": {
      "loc": "dev"
    }
  }
}

Todo

• Resloving MongoDB retrieval by changing the payload, we will add the vector index as the part of the payload config.