tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "amnSfQ4S-im9"
   },
   "source": [
    "# Model Deployment with BentoML and Kubernetes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The easy part of machine learning in production is the model development, but once you've developed a model how do you deploy it? There are many issues to deal with, including package management, API performance and security, as well as basic model versioning. While you could deal with these things all manually, it quickly becomes a cumbersome engineering task. As both a Data Scientist and an ML Engineer, you want to transition your model to be production ready as quickly and easily as possible, with  little DevOps friction. In this tutorial, we will be building a transaction fraud detection API using the [BentoML](https://bentoml.com) framework.\n",
    "\n",
    "We chose BentoML here as it is a framework that is both easy to use and has a lot of built-in features that make it a great choice for building production ready ML APIs, including a performant WGSI HTTP server powered by [gunicorn](https://gunicorn.org), in-built pip package management and quick containerization. Its output is simply a Docker container that can be deployed anywhere you want, from serverless to Kubernetes to even a simple VM. We demonstrate this on Kubernetes as this gives you a lot of flexibility on resource usage and gives you auto-scaling capability, but you can use any container runtime you want. Specifically, we will use [minikube](https://github.com/kubernetes/minikube) here as it is free, easy to use, and runs locally on your machine with a minimal install. As all Kubernetes runtimes are managed with the same utility, the concepts you learn here can be applied to any Kubernetes runtime."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nY8jTgnQ_P08"
   },
   "source": [
    "By the end of this tutorial you will be able to:\n",
    "- Setup a Kubernetes cluster with Minikube\n",
    "- Create Bento services with BentoML and containerize them\n",
    "- Deploy the a Bento service to Kubernetes\n",
    "\n",
    "### Prerequisites:\n",
    "- Install Docker\n",
    "- Install Python3.8+\n",
    "- Install JupyterLab\n",
    "  \n",
    "You should download the data required for this tutorial from [here](https://drive.google.com/file/d/1MidRYkLdAV-i0qytvsflIcKitK4atiAd/view?usp=sharing). This is originally from a [Kaggle dataset](https://www.kaggle.com/competitions/ieee-fraud-detection/data) for Fraud Detection. Place this dataset in a `data` directory in the root of your project. You can run this notebook either in VS Code or Jupyter Notebooks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build a model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Firstly, we need a model to deploy. Let's build a quick model to detect fraudulent transactions. We will need a number of libraries so lets install them. Since the focus of this tutorial is deployment, don't worry about the feature selection or model training specifics. There are only two objects to be concerned with in the training code block:\n",
    "\n",
    "- **enc**: The encoder object we use to preprocess the data with one-hot encoding.\n",
    "- **model**: The model object we are going to deploy.\n",
    "\n",
    "If you wish, create a virtual environment with conda or venv.\n",
    "```\n",
    "pip install scikit-learn==1.0.2 pandas==1.4.3 numpy==1.23.2 xgboost==1.5.1\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "rWI28p60-imy"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "from xgboost import XGBClassifier\n",
    "\n",
    "# Load the data, sample such that the target classes are equal size\n",
    "df = pd.read_csv(\"data/train_transaction.csv\")\n",
    "df = pd.concat(\n",
    "    [df[df.isFraud == 0].sample(n=len(df[df.isFraud == 1])), df[df.isFraud == 1]],\n",
    "    axis=0,\n",
    ")\n",
    "\n",
    "# Select the features and target\n",
    "X = df[[\"ProductCD\", \"P_emaildomain\", \"R_emaildomain\", \"card4\", \"M1\", \"M2\", \"M3\"]]\n",
    "y = df.isFraud\n",
    "\n",
    "# Use one-hot encoding to encode the categorical features\n",
    "enc = OneHotEncoder(handle_unknown=\"ignore\")\n",
    "enc.fit(X)\n",
    "\n",
    "X = pd.DataFrame(\n",
    "    enc.transform(X).toarray(), columns=enc.get_feature_names_out().reshape(-1)\n",
    ")\n",
    "X[\"TransactionAmt\"] = df[[\"TransactionAmt\"]].to_numpy()\n",
    "\n",
    "# Split the dataset and train the model\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, test_size=0.2, random_state=42\n",
    ")\n",
    "xgb = XGBClassifier(\n",
    "    n_estimators=100,\n",
    "    learning_rate=0.1,\n",
    "    max_depth=3,\n",
    "    min_child_weight=1,\n",
    "    gamma=0,\n",
    "    subsample=0.8,\n",
    "    colsample_bytree=0.8,\n",
    "    objective=\"binary:logistic\",\n",
    "    nthread=4,\n",
    "    scale_pos_weight=1,\n",
    "    seed=27,\n",
    ")\n",
    "model = xgb.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup your Kubernetes Cluster"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "oAyUYUtg-im_"
   },
   "source": [
    "So, you have built a model, and you want to deploy it so it's actually useful. How do you do that? We are going to use Kubernetes, a system for autoscaling and managing container based services. As we said in the intro, we are using [minikube](https://github.com/kubernetes/minikube) to create a local Kubernetes instance. You can however, use any local or cloud runtime you'd like, though you may need to go through additional setup. Some other options are:\n",
    "### Local + VM\n",
    "- [kind](https://kind.sigs.k8s.io)\n",
    "- [K3s](https://k3s.io)\n",
    "- [MicroK8s](https://microk8s.io)\n",
    "- [kubeadm](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/)\n",
    "### Cloud\n",
    "- [Managed Kubernetes on DigitalOcean](https://www.digitalocean.com/products/kubernetes)\n",
    "- [EKS on AWS](https://aws.amazon.com/eks/)\n",
    "- [GKE on GCP](https://cloud.google.com/kubernetes-engine)\n",
    "- [AKS on Azure](https://azure.microsoft.com/en-us/services/kubernetes-service/)\n",
    "\n",
    "Don't be too scared of using minikube as opposed to a cloud instance, the actual Kubernetes commands you'll need to run to deploy our ML service are the same as they are all done through `kubectl`. You can apply your learnings pretty easily on a managed K8s platform.\n",
    "\n",
    "First we need to install the aforementioned `kubectl` utility. This will enable us to interact with our minikube cluster (or any cluster).\n",
    "```bash\n",
    "brew install kubectl\n",
    "```\n",
    "\n",
    "Let's set up our local minikube instance. Install minikube for your specific platform. We have provided the command for **brew**, since macOS is obviously the best platform:\n",
    "```bash\n",
    "brew install minikube\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before proceeding, it goes without saying that you'll need the **docker daemon** to be running so make sure that is the case! Minikube is now installed, but it is not running currently. To start up the cluster just run:\n",
    "```bash\n",
    "minikube start\n",
    "```\n",
    "\n",
    "You may have other clusters you interact with so you'll need to switch *contexts*. You can think of a context is an abstraction of a cluster. Ensure you're in your local minikube context.\n",
    "```bash\n",
    "kubectl config use-context minikube\n",
    "```\n",
    "\n",
    "Verify your context by retrieving the cluster info.\n",
    "```bash\n",
    "kubectl cluster-info\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congratulations! You have now setup a your minikube cluster. Now that we have the infrastructure, we need to create a API service for our fraud detection model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a Bento Service"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While there are a number of tools that ease the stress of deploying a model, one of the more straightforward ways is to use the open source [BentoML](https://bentoml.com) framework. BentoML is a framework for deploying machine learning models that pre-packages the model for you into a callable REST API containerized service. We chose BentoML as it is easy to use, will install the packages we need and can deploy to multiple different cloud services and infrastructures. If you'd like, check out some of the alternative frameworks that are available.\n",
    "- [MLFlow Models](https://mlflow.org/docs/latest/models.html) - Not as flexible as BentoML, more complex and less performant, but does have a lot of features like experiment tracking.\n",
    "- [TensorFlow Serving](https://github.com/tensorflow/serving) - Only supports TensorFlow, and does not use the Python runtime.\n",
    "- [TorchServe](https://pytorch.org/serve/) - Only supports PyTorch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "b9_u80OM-im_"
   },
   "source": [
    "Install the BentoML 1.0.0 with the following command. Note that if you use conda, the package is only available in the conda-forge channel which may not be updated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "bDtTtB9V-8XW",
    "outputId": "ae8c366c-985f-42ea-c9db-849fb6114206",
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "pip install bentoml==1.0.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Us8A2VZH-inB"
   },
   "source": [
    "Now, we want to create our Bento service using the **model** and **enc** objects we created before. When you *save* a model, Bento will store it locally and version it. Import `bentoml` and use the appropriate `save_model` function to save the models we need to the local **model** store, running it in your notebook. You may notice we used the **sklearn** `save_model` for the XGBoost model. This is as we have used the SKLearn API to create the model.\n",
    "\n",
    "There are a number of different *optional* arguments you can include when saving a model. We included a number of extra tags to demonstrate how this works.\n",
    "\n",
    "- **labels**: user-defined labels for managing models (e.g. team=nlp, stage=dev).\n",
    "- **metadata**: user-defined metadata for storing model training context information or model evaluation metrics (e.g. dataset version, training parameters, confusion matrix, etc).\n",
    "- **custom_objects**: user-defined additional python objects (e.g. a tokenizer instance, preprocessor functions, etc). Custom objects will be serialized with cloudpickle.\n",
    "- **signatures**: model signatures for inference (e.g. input/output shapes, whether inference is batched, etc). For more information, see the [BentoML documentation](https://bentoml.com/docs/bento-ml/api/model-signatures/) on signatures."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 508
    },
    "id": "w811MKin-inB",
    "outputId": "9465023c-0dd6-48bf-8891-ec7340ccd0c7"
   },
   "outputs": [],
   "source": [
    "import bentoml\n",
    "\n",
    "saved_model = bentoml.sklearn.save_model(\n",
    "    \"fraud_classifier\",\n",
    "    model,\n",
    "    labels={\"owner\": \"Cerebrium\", \"stage\": \"prod\"},\n",
    "    metadata={\"version\": \"1.0.0\"},\n",
    "    custom_objects={\"ohe_encoder\": enc},\n",
    "    signatures={\n",
    "        \"predict\": {\n",
    "            \"batchable\": True,\n",
    "            \"batch_dim\": 0,\n",
    "        }\n",
    "    },\n",
    ")\n",
    "print(f\"{saved_model}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "L3nJISPC-im_"
   },
   "source": [
    "Next, you will need to create a Bento service. This abstraction tells Bento what model to use to run inference and handle any preprocessing. In this service, we are going to use the `fraud_classifier` model we saved to the local store.\n",
    "\n",
    "Create a new file called `fraud_detection_service.py` in the project root directory, and paste the following code into it:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2rQaAS8f-inA"
   },
   "source": [
    "```python\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "import bentoml\n",
    "from bentoml.io import PandasDataFrame, JSON\n",
    "\n",
    "ohe_encoder = bentoml.models.get(\"fraud_classifier:latest\").custom_objects[\n",
    "    \"ohe_encoder\"\n",
    "]\n",
    "fraud_classifier_runner = bentoml.sklearn.get(\"fraud_classifier:latest\").to_runner()\n",
    "\n",
    "svc = bentoml.Service(\"fraud_classifier\", runners=[fraud_classifier_runner])\n",
    "\n",
    "\n",
    "@svc.api(input=PandasDataFrame(), output=JSON(), route=\"/fraud-classifier\")\n",
    "def predict(df: pd.DataFrame) -> np.ndarray:\n",
    "    X = df[[\"ProductCD\", \"P_emaildomain\", \"R_emaildomain\", \"card4\", \"M1\", \"M2\", \"M3\"]]\n",
    "    X = X.fillna(pd.NA)  # ensure all missing values are pandas NA\n",
    "    X = pd.DataFrame(\n",
    "        ohe_encoder.transform(X).toarray(),\n",
    "        columns=ohe_encoder.get_feature_names_out().reshape(-1),\n",
    "    )\n",
    "    X[\"TransactionAmt\"] = df[[\"TransactionAmt\"]].to_numpy()\n",
    "    return fraud_classifier_runner.predict.run(X)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kKmFlOov-inA"
   },
   "source": [
    "There are a number of key details in this file to be aware of.\n",
    "\n",
    "- `ohe_encoder`: You'll notice that we load in the encoder custom object from the *fraud_classifier* model we defined previously. This is because we need to transform the data before we can use it for inference.\n",
    "- `fraud_classifier_runner`: Here, we load in the *fraud_classifier* model we defined previously and convert it into a **runner**. A runner in BentoML represents a unit of serving logic which wraps a model and can be scaled to maximize throughput and resource use.\n",
    "- `svc`: This represents the Service object. It is the main entry point for the BentoML service.\n",
    "- `svc.api`: This is a decorator that tells BentoML this is an API, what kind of input and output the API accepts, and the desired REST route.\n",
    "\n",
    "As you can see, we instantiate a **Service** class and define an API with DataFrame inputs and JSON outputs. We run all necessary pre-processing with the **encoder** custom object, then call the **model** runner to make predictions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now quickly test out our service. We can run the following command in the terminal:\n",
    "\n",
    "```bash\n",
    "bentoml serve fraud_detection_service:svc\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Navigate to the specified IP of the service and run the following POST request (the output should be `[1]`):"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```json\n",
    "[{\n",
    "    \"isFraud\":0,\n",
    "    \"TransactionAmt\":495.0,\n",
    "    \"ProductCD\":\"W\",\n",
    "    \"card4\":\"visa\",\n",
    "    \"P_emaildomain\":\"live.com\",\n",
    "    \"R_emaildomain\":null,\n",
    "    \"M1\":\"T\",\n",
    "    \"M2\":\"T\",\n",
    "    \"M3\":\"T\"\n",
    "}]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ET1P-fQ0-inC"
   },
   "source": [
    "Our service is ready to be packaged into a **Bento**, which is essentially the service packaged with needed dependencies. We're now going to build and containerize the Bento and deploy it to our minikube K8s cluster."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bento Building & Containerization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we containerize and test our Bento, we need to change the default repository our docker daemon will push to. In particular, instead of our local dockerhub repo, we should change it to the minikube repository. We do this by setting some environment variables.\n",
    "```bash\n",
    "eval $(minikube docker-env)\n",
    "```\n",
    "Now any command we run will use minikube's docker daemon."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To build our Bento, we need to define `bentofile.yaml` file in our project directory. We use this file to specify things such as python packages, CUDA installations, the base docker image, etc. You can read more about the various build options [here](https://docs.bentoml.org/en/latest/concepts/bento.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```yaml\n",
    "service: \"fraud_detection_service:svc\"  # Same as the argument passed to `bentoml serve`\n",
    "labels:\n",
    "   owner: Cerebrium\n",
    "   stage: prod\n",
    "include:\n",
    "- \"*.py\"  # A pattern for matching which files to include in the bento\n",
    "python:\n",
    "   packages:  # Additional pip packages required by the service\n",
    "   - scikit-learn==1.0.2\n",
    "   - pandas==1.4.3\n",
    "   - numpy==1.23.2\n",
    "   - xgboost==1.5.1\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we run the following command to build our Bento:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!bentoml build"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "'Grats! You created a Bento! You can serve it with the following command in your terminal with the **latest** tag and navigating [localhost:3000](localhost:3000).\n",
    "```bash\n",
    "bentoml serve fraud_classifier:latest --production\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ensure the response output is as expected (either 1 or 0). If there are any errors, you likely made a mistake in the **FraudClassifier** class."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we need to create the service container. Using `bentoml containerize`, we will containerize our Bento, tagging the image with your registry link and the name of the service. We are using the *latest* tag, but you could grab the id of the Bento as a substitute (with `bentoml list`). If you're on Apple Silicon, include `--platform=linux/amd64` in the command to avoid compatibility issues.\n",
    "\n",
    "```bash\n",
    "bentoml containerize fraud_classifier:latest -t fraud-classifier:latest --platform=linux/amd64\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploy your service"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well done! Now, there's only one thing left to do. We need to deploy our service images on Kubernetes! We will do this using Kubernetes manifests. We recommend using the [Kubernetes VS code extension](https://blog.knoldus.com/create-kubernetes-manifests-files-quickly/) to get started creating manifest files. In the end, you should end up with a file, `deployment.yaml` that looks roughly like this:\n",
    "```yaml\n",
    "apiVersion: apps/v1\n",
    "kind: Deployment\n",
    "metadata:\n",
    "  name: fraud-classifier\n",
    "spec:\n",
    "  selector:\n",
    "    matchLabels:\n",
    "      app: fraud-classifier\n",
    "  template:\n",
    "    metadata:\n",
    "      labels:\n",
    "        app: fraud-classifier\n",
    "    spec:\n",
    "      containers:\n",
    "      - name: fraud-classifier\n",
    "        image: fraud-classifier:latest\n",
    "        resources:\n",
    "          limits:\n",
    "            memory: \"2Gi\"\n",
    "            cpu: \"1\"\n",
    "        ports:\n",
    "        - containerPort: 3000\n",
    "        imagePullPolicy: IfNotPresent\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can deploy this manifest to our minikube cluster with the following command:\n",
    "\n",
    "```bash\n",
    "kubectl apply -f deployment.yaml\n",
    "```\n",
    "\n",
    "You can confirm it is now running and viewing all Replica Sets.\n",
    "```bash\n",
    "kubectl get rs\n",
    "kubectl get pods\n",
    "```\n",
    "\n",
    "We need a load balancer to expose our Bento service and make our Fraud Classifier scalable to multiple pods. Let's create a K8s service by exposing the deployment, forwarding port 80 to the target port 3000.\n",
    "\n",
    "```bash\n",
    "kubectl expose deployment fraud-classifier --type=LoadBalancer --port=80 --target-port=3000\n",
    "```\n",
    "\n",
    "Now that the load balancer is live, let's make sure our service is live! We can use minikube to quickly create a tunnel through to our service (if you've been using a cloud service, just navigate to the IP of the load balancer).\n",
    "```bash\n",
    "minikube service fraud-classifier\n",
    "```\n",
    "\n",
    "![Live App](media/live_app.png)\n",
    "\n",
    "Congratulations, you've deployed your fraud classifier as an API!"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "name": "explore.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3.10.4 ('fraud')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "e8cc1641a3a2dfa0be5af08829a6ac15bbacdd3c53809d4cb1653419889483e4"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}