Skip to content

Commit

Permalink
add gmc guide (#94)
Browse files Browse the repository at this point in the history 
* add gmc guide

Signed-off-by: irisdingbj <shaojun.ding@intel.com>

* address review comments

Signed-off-by: irisdingbj <shaojun.ding@intel.com>

* fix  format issue

Signed-off-by: irisdingbj <shaojun.ding@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: irisdingbj <shaojun.ding@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
  • Loading branch information
@pre-commit-ci
Iris and pre-commit-ci[bot] authored Jun 19, 2024
1 parent eec8452 commit 6bb8a33
Show file tree
Hide file tree
Showing 2 changed files with 108 additions and 42 deletions.
81 changes: 39 additions & 42 deletions microservices-connector/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,10 +1,13 @@
# genai-microservices-connector(GMC)

This repo defines the GenAI Microservice Connector for OPEA projects.
This repo defines the GenAI Microservice Connector(GMC) for OPEA projects. GMC can be used to compose and adjust GenAI pipelines dynamically
on kubernetes. It can leverage the microservices provided by [GenAIComps](https://github.com/opea-project/GenAIComps) and external services to compose GenAI pipelines. External services might be running in a public cloud or on-prem by providing an URL and access details such as an API key and ensuring there is network connectivity. It also allows users to adjust the pipeline on the fly like switching to a different Large language Model(LLM), adding new functions into the chain(like adding guardrails),etc. GMC supports different types of steps in the pipeline, like sequential, parallel and conditional.

Please refer this [usage_guide](./usage_guide.md) for sample use cases.

## Description

The GMC contains the CRD and its controller to bring up the needed services for a GenAI application.
The GenAI Microservice Connector(GMC) contains the CustomResourceDefinition(CRD) and its controller to bring up the services needed for a GenAI application.
Istio Service Mesh can also be leveraged to facilicate communication between microservices in the GenAI application.

## Architecture
Expand All @@ -21,22 +24,16 @@ Istio Service Mesh can also be leveraged to facilicate communication between mic
**API** is api/v1alpha3/
**Controller** is at internal/controller

`make manifests` to install the APIs
`make run` to lanch the controllers

### Prerequisites

- go version v1.21.0+
- docker version 17.03+.
- kubectl version v1.11.3+.
- Access to a Kubernetes v1.11.3+ cluster.

### Introduction

There are `2` components in this repo:

- 1. `manager`: controller manager to handle GMC CRD
- 2. `router`: route the traffic among the microservices defined in GMC connector
- 2. `router`: route the traffic among the microservices defined in GMC

#### How to build these binaries?

Expand All @@ -47,7 +44,7 @@ make build
#### How to build docker images for these 2 components?

```sh
make docker
make docker.build
```

#### How to delete these components' binaries?
Expand All @@ -61,76 +58,76 @@ make clean
**Build and push your image to the location specified by `CTR_IMG`:**

```sh
make docker-build docker-push CTR_IMG=<some-registry>/gmconnector:tag
make docker.build docker.push CTR_IMG=<some-registry>/gmcmanager:tag
```

**NOTE:** This image ought to be published in the personal registry you specified.
And it is required to have access to pull the image from the working environment.
Make sure you have the proper permission to the registry if the above commands don’t work.

**Install the CRDs into the cluster:**
**Install GMC CRD**

```sh
make install
kubectl apply -f config/crd/bases/gmc.opea.io_gmconnectors.yaml
```

**Deploy the Manager to the cluster with the image specified by `CTR_IMG`:**
**Get related manifests for GenAI Components**

```sh
make deploy CTR_IMG=<some-registry>/gmconnector:tag
mkdir -p $(pwd)/config/manifests
cp $(dirname $(pwd))/manifests/ChatQnA/*.yaml -p $(pwd)/config/manifests/
```

> **NOTE**: If you encounter RBAC errors, you may need to grant yourself cluster-admin
> privileges or be logged in as admin.
**Create instances of your solution**
You can apply the samples (examples) from the config/sample:
**Copy GMC router manifest**

```sh
kubectl apply -k config/samples/
cp $(pwd)/config/gmcrouter/gmc-router.yaml -p $(pwd)/config/manifests/
```

> **NOTE**: Ensure that the samples has default values to test it out.
### To Uninstall

**Delete the instances (CRs) from the cluster:**
**Create ConfigMap for GMC to hold GenAI Components and GMC Router manifests**

```sh
kubectl delete -k config/samples/
kubectl create configmap gmcyaml -n $SYSTEM_NAMESPACE --from-file $(pwd)/config/manifests
```

**Delete the APIs(CRDs) from the cluster:**
**NOTE:** The configmap name `gmcyaml' is defined in gmcmanager deployment Spec. Please modify accordingly if you want
use a different name for the configmap.

**Install GMC manager**

```sh
make uninstall
kubectl apply -f $(pwd)/config/rbac/gmc-manager-rbac.yaml
kubectl apply -f $(pwd)/config/manager/gmc-manager.yaml
```

**UnDeploy the controller from the cluster:**
**Check the installation result**

```sh
make undeploy
kubectl get pods -n system
NAME READY STATUS RESTARTS AGE
gmc-controller-78f9c748cb-ltcdv 1/1 Running 0 3m
```

## Project Distribution
### Next Step

Please refer to this [usage_guide](./usage_guide.md) for sample use cases.

Following are the steps to build the installer and distribute this project to users.
### To Uninstall

1. Build the installer for the image built and published in the registry:
**Delete the instances (CRs) from the cluster:**

```sh
make build-installer CTR_IMG=<some-registry>/gmconnector:tag
kubectl delete -k config/samples/
```

NOTE: The makefile target mentioned above generates an 'install.yaml'
file in the dist directory. This file contains all the resources built
with Kustomize, which are necessary to install this project without
its dependencies.
**Delete the APIs(CRDs) from the cluster:**

2. Using the installer
```sh
make uninstall
```

Users can just run kubectl apply -f <URL for YAML BUNDLE> to install the project, i.e.:
**UnDeploy the controller from the cluster:**

```sh
kubectl apply -f https://raw.githubusercontent.com/<org>/gmconnector/<tag or branch>/dist/install.yaml
make undeploy
```
69 changes: 69 additions & 0 deletions microservices-connector/usage_guide.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
# Usage guide for genai-microservices-connector(GMC)

genai-microservices-connector(GMC) can be used to compose and adjust GenAI pipelines dynamically. It can leverage the microservices provided
by [GenAIComps](https://github.com/opea-project/GenAIComps) and external services to compose GenAI pipelines.

Below are sample use cases:

## Use GMC to compose a chatQnA Pipeline

A sample for chatQnA can be found at config/samples/chatQnA_xeon.yaml

**Deploy chatQnA GMC custom resource**

```sh
kubectl create ns chatqa
kubectl apply -f $(pwd)/config/samples/chatQnA_xeon.yaml
```

**GMC will reconcile chatQnA custom resource and get all related components/services ready**

```sh
kubectl get service -n chatqa
```

**Check GMC chatQnA custom resource to get access URL for the pipeline**

```bash
$kubectl get gmconnectors.gmc.opea.io -n chatqa
NAME URL READY AGE
chatqa http://router-service.chatqa.svc.cluster.local:8080 Success 3m
```

**Deploy one client pod for testing the chatQnA application**

```bash
kubectl create deployment client-test -n chatqa --image=python:3.8.13 -- sleep infinity
```

**Access the pipeline using the above URL from the client pod**

```bash
export CLIENT_POD=$(kubectl get pod -l app=client-test -o jsonpath={.items..metadata.name})
export accessUrl=$(kubectl get gmc -n chatqa -o jsonpath="{.items[?(@.metadata.name=='chatqa')].status.accessUrl}")
kubectl exec "$CLIENT_POD" -n chatqa -- curl $accessUrl -X POST -d '{"text":"What is the revenue of Nike in 2023?","parameters":{"max_new_tokens":17, "do_sample": true}}' -H 'Content-Type: application/json'
```

## Use GMC to adjust the chatQnA Pipeline

**Modify chatQnA custom resource to change to another LLM model**

```yaml
- name: Tgi
internalService:
serviceName: tgi-svc
config:
LLM_MODEL_ID: Llama-2-7b-chat-hf
```
**Check the tgi-svc-deployment has been changed to use the new LLM Model**
```sh
kubectl get deployment tgi-svc-deployment -n chatqa -o jsonpath="{.spec.template.spec.containers[*].env[?(@.name=='LLM_MODEL_ID')].value}"
```

**Access the updated pipeline using the above URL from the client pod**

```bash
kubectl exec "$CLIENT_POD" -n chatqa -- curl $accessUrl -X POST -d '{"text":"What is the revenue of Nike in 2023?","parameters":{"max_new_tokens":17, "do_sample": true}}' -H 'Content-Type: application/json'
```

0 comments on commit 6bb8a33

Please sign in to comment.