This repository contains a full Q&A pipeline using the LlamaIndex framework, Qdrant as a vector database, and deployment on Google Kubernetes Engine (GKE) using a FastAPI app and Dockerfile. Python files from my repositories are loaded into the vector database, and the FastAPI app processes requests. The main goal is to provide fast access to your own code, enabling reuse of functions.
For detailed project descriptions, refer to this Medium article.
This article was featured in the GKE Newsletter (This week in GKE, ISSUE#19, 12 July 2024)
Main Steps
- Data Ingestion: Load data from GitHub repositories
- Indexing: Use SentenceSplitter for indexing in nodes
- Embedding and Model: OpenAI
- Vector Store: Use Qdrant for inserting metadata
- Query Retrieval: Implement RetrieverQueryEngine with SentenceTransformerRerank
- FastAPI and GKE: Handle requests via the FastAPI app deployed on GKE
- Streamlit: UI
Feel free to ⭐ and clone this repo 😉
The project has been structured with the following files:
.github/workflows:
CI/CD pipelinestests
: unittestDockerfile:
DockerfileMakefile
: install requirements, formating, linting, testing and clean upapp.py:
FastAPIpyproject.toml
: linting and formatting using ruffcreate_qdrant_collection.py:
script to create the collection in Qdrantdeploy-gke.yaml:
deployment functionkustomization.yaml:
kustomize deployment functionrequirements.txt:
project requirementsstreamlit_app.py
: streamlit app
The Python version used for this project is Python 3.10. You can follow along the medium article.
-
Clone the repo (or download it as a zip file):
git clone https://github.com/benitomartin/scale-gke-qdrant-llama.git
-
Create the virtual environment named
main-env
using Conda with Python version 3.10:conda create -n main-env python=3.10 conda activate main-env
-
Execute the
Makefile
script and install the project dependencies included in the requirements.txt:pip install -r requirements.txt or make install
-
Make sure the
.env
file is complete:OPENAI_API_KEY= QDRANT_API_KEY= QDRANT_URL= COLLECTION_NAME= ACCESS_TOKEN= GITHUB_USERNAME=
-
You can test the app locally running:
uvicorn app:app --host 0.0.0.0 --port 8000
then go to one of these addresses
-
Create GCP Account, project, service account key, and activate GKE API
-
Add the following secrets into github:
OPENAI_API_KEY QDRANT_API_KEY QDRANT_URL COLLECTION_NAME GKE_SA_KEY GKE_PROJECT # PROJECT_ID
-
Be sure to authenticate in GCP:
gcloud auth login
gcloud config set project PROJECT_ID
-
Create Kubernetes Cluster
gcloud container clusters create llama-gke-cluster \ --zone=europe-west6-a \ --num-nodes=5 \ --enable-autoscaling \ --min-nodes=2 \ --max-nodes=10 \ --machine-type=n1-standard-4 \ --enable-vertical-pod-autoscaling
after creation check the nodes
kubectl get nodes
-
Push the GitHub Actions workflows to start the deployment
-
Verify Kubernetes is running after deployment
# Get the Pods kubectl get po # Get the Nodes kubectl get nodes # Get the Services kubectl get svc # Get the logs of a pod kubectl logs llama-gke-deploy-668b58b455-fjwvq # Describe a pod kubectl describe pod llama-gke-deploy-668b58b455-fjwvq # Check CPU usage kubectl top pod llama-gke-deploy-668b58b455-fjwvq
-
Under svc the external ip is the endpoint (34.65.157.134), that can be added in the streamlit app
# Set the FastAPI endpoint FASTAPI_ENDPOINT = "http://34.65.157.134:8000/query/"
-
Check some pods and logs
kubectl logs llama-gke-deploy-668b58b455-fjwvq kubectl describe pod llama-gke-deploy-668b58b455-fjwvq kubectl top pod llama-gke-deploy-668b58b455-8xfhf
-
Clean up to avoid costs deleting the cluster and the docker image
gcloud container clusters delete llama-gke-cluster --zone=europe-west6-a
Run the streamlit app adding the endpoint url that you get after deployment:
streamlit run streamlit_app.py