Q&A Pipeline Deployment on GKE for Scalability with LlamaIndex and Qdrant 🚀

This repository contains a full Q&A pipeline using the LlamaIndex framework, Qdrant as a vector database, and deployment on Google Kubernetes Engine (GKE) using a FastAPI app and Dockerfile. Python files from my repositories are loaded into the vector database, and the FastAPI app processes requests. The main goal is to provide fast access to your own code, enabling reuse of functions.

For detailed project descriptions, refer to this Medium article.

This article was featured in the GKE Newsletter (This week in GKE, ISSUE#19, 12 July 2024)

Main Steps

• Data Ingestion: Load data from GitHub repositories
• Indexing: Use SentenceSplitter for indexing in nodes
• Embedding and Model: OpenAI
• Vector Store: Use Qdrant for inserting metadata
• Query Retrieval: Implement RetrieverQueryEngine with SentenceTransformerRerank
• FastAPI and GKE: Handle requests via the FastAPI app deployed on GKE
• Streamlit: UI

Feel free to ⭐ and clone this repo 😉

Tech Stack

Project Structure

The project has been structured with the following files:

• .github/workflows: CI/CD pipelines
• tests: unittest
• Dockerfile:Dockerfile
• Makefile: install requirements, formating, linting, testing and clean up
• app.py: FastAPI
• pyproject.toml: linting and formatting using ruff
• create_qdrant_collection.py: script to create the collection in Qdrant
• deploy-gke.yaml: deployment function
• kustomization.yaml: kustomize deployment function
• requirements.txt: project requirements
• streamlit_app.py: streamlit app

Project Set Up

The Python version used for this project is Python 3.10. You can follow along the medium article.

1. Clone the repo (or download it as a zip file):

   git clone https://github.com/benitomartin/scale-gke-qdrant-llama.git

2. Create the virtual environment named main-env using Conda with Python version 3.10:

   conda create -n main-env python=3.10
   conda activate main-env

3. Execute the Makefile script and install the project dependencies included in the requirements.txt:

   pip install -r requirements.txt
   
   or
   
   make install

4. Make sure the .env file is complete:

   OPENAI_API_KEY=
   QDRANT_API_KEY=
   QDRANT_URL=
   COLLECTION_NAME=
   ACCESS_TOKEN=
   GITHUB_USERNAME=

5. You can test the app locally running:

   uvicorn app:app --host 0.0.0.0 --port 8000

   then go to one of these addresses

   http://localhost:8000/docs or http://127.0.0.1:8000/docs

6. Create GCP Account, project, service account key, and activate GKE API

7. Add the following secrets into github:

   OPENAI_API_KEY
   QDRANT_API_KEY
   QDRANT_URL
   COLLECTION_NAME
   GKE_SA_KEY
   GKE_PROJECT # PROJECT_ID

8. Be sure to authenticate in GCP:

   gcloud auth login

   gcloud config set project PROJECT_ID

9. Create Kubernetes Cluster

   gcloud container clusters create llama-gke-cluster \
       --zone=europe-west6-a \
       --num-nodes=5 \
       --enable-autoscaling \
       --min-nodes=2 \
       --max-nodes=10 \
       --machine-type=n1-standard-4 \
       --enable-vertical-pod-autoscaling

   after creation check the nodes

   kubectl get nodes

10. Push the GitHub Actions workflows to start the deployment

11. Verify Kubernetes is running after deployment

    # Get the Pods
    kubectl get po
    
    # Get the Nodes
    kubectl get nodes
    
    # Get the Services
    kubectl get svc 
    
    # Get the logs of a pod
    kubectl logs llama-gke-deploy-668b58b455-fjwvq 
    
    # Describe a pod
    kubectl describe pod llama-gke-deploy-668b58b455-fjwvq
    
    # Check CPU usage
    kubectl top pod llama-gke-deploy-668b58b455-fjwvq

12. Under svc the external ip is the endpoint (34.65.157.134), that can be added in the streamlit app

    # Set the FastAPI endpoint
    FASTAPI_ENDPOINT = "http://34.65.157.134:8000/query/"

13. Check some pods and logs

    kubectl logs llama-gke-deploy-668b58b455-fjwvq 
    kubectl describe pod llama-gke-deploy-668b58b455-fjwvq
    kubectl top pod llama-gke-deploy-668b58b455-8xfhf 

14. Clean up to avoid costs deleting the cluster and the docker image

    gcloud container clusters delete llama-gke-cluster --zone=europe-west6-a

Streamlit UI

Run the streamlit app adding the endpoint url that you get after deployment:

streamlit run streamlit_app.py