9-serving

TensorFlow Serving

Prerequisites

• 1 - Docker Basics
• 2 - Kubernetes Basics and cluster created
• 3 - Helm
• 4 - Kubeflow

Summary

In this section you will learn about:

• Setting up a Minio file storage in our Kubernetes cluster
• Serving trained models using TensorFlow Serving

Context

TensorFlow Serving is a flexible, high-performance serving system for machine learning models, designed for production environments. TensorFlow Serving makes it easy to deploy new algorithms and experiments, while keeping the same server architecture and APIs. TensorFlow Serving provides out-of-the-box integration with TensorFlow models, but can be easily extended to serve other types of models and data.

Exercises

Exercise 1: Setting up file storage

First, we'll get started with a file storage backend.

If you already have a model uploaded to storage, you can skip this step. If not, you can download Minio client to your operating system of choice to upload trained and exported model.

As we saw in module 3 - Helm, Helm enables us to package an application in a chart and parametrize it's deployment easily. We'll use Helm to create a Minio deployment in our cluster.

ACCESS_KEY=<your access key>
ACCESS_SECRET_KEY=<your access secret key>

helm install --name minio --set accessKey=$ACCESS_KEY,secretKey=$ACCESS_SECRET_KEY,service.type=LoadBalancer stable/minio

SERVICE_IP=$(kubectl get svc minio --template="{{range .status.loadBalancer.ingress}}{{.ip}}{{end}}")
S3_ENDPOINT=${SERVICE_IP}:9000

Setting up Minio host:

mc config host add minio $S3_ENDPOINT $ACCESS_KEY $ACCESS_SECRET_KEY

Creating a bucket and uploading our trained model:

BUCKET_NAME=kubeflow

mc mb minio/$BUCKET_NAME

mc cp --recursive /path/to/your/exported/model minio/$BUCKET_NAME

After this command, you should see files are being uploaded.

Exercise 2: Setting up TensorFlow Serving model server

In this exercise, we are going to set up a TensorFlow model server and start serving our trained model.

Creating our namespace for serving:

export NAMESPACE=serving

kubectl create namespace $NAMESPACE

Creating secret for the Minio storage so TensorFlow Serving container can access it:

kubectl create secret generic serving-creds --from-literal=accessKeyID=${ACCESS_KEY} \
 --from-literal=secretAccessKey=${ACCESS_SECRET_KEY} -n $NAMESPACE

Defining variables such as model name, TensorFlow Serving image.

S3_USE_HTTPS=0
S3_VERIFY_SSL=0
JOB_NAME=myjob
MODEL_COMPONENT=mnist
MODEL_NAME=mnist
MODEL_PATH=s3://${BUCKET_NAME}/models/${JOB_NAME}/export/${MODEL_NAME}/
MODEL_SERVER_IMAGE=sozercan/tensorflow-model-server

Initalize Kubeflow:

ks init my-model-server
cd my-model-server
ks registry add kubeflow github.com/kubeflow/kubeflow/tree/master/kubeflow
ks pkg install kubeflow/tf-serving@74629b7

Setting up environment for Kubeflow:

ks env add azure
ks env set azure --namespace ${NAMESPACE}

Generating the template:

ks generate tf-serving ${MODEL_COMPONENT} --name=${MODEL_NAME}

Overriding parameters with our own values:

ks param set --env azure ${MODEL_COMPONENT} modelServerImage $MODEL_SERVER_IMAGE
ks param set --env azure ${MODEL_COMPONENT} modelPath $MODEL_PATH
ks param set --env azure ${MODEL_COMPONENT} s3Enable true
ks param set --env azure ${MODEL_COMPONENT} s3SecretName serving-creds
ks param set --env azure ${MODEL_COMPONENT} s3SecretAccesskeyidKeyName accessKeyID
ks param set --env azure ${MODEL_COMPONENT} s3SecretSecretaccesskeyKeyName secretAccessKey
ks param set --env azure ${MODEL_COMPONENT} s3Endpoint $S3_ENDPOINT
ks param set --env azure ${MODEL_COMPONENT} s3AwsRegion us-east-1
ks param set --env azure ${MODEL_COMPONENT} s3UseHttps $S3_USE_HTTPS --as-string
ks param set --env azure ${MODEL_COMPONENT} s3VerifySsl $S3_VERIFY_SSL --as-string
ks param set --env azure ${MODEL_COMPONENT} serviceType LoadBalancer

Deploying TensorFlow Serving to our cluster:

ks apply azure -c ${MODEL_COMPONENT}

After deploying, you should see a deployment and service in your cluster. You can verify with the following:

kubectl get pods -n ${NAMESPACE}

kubectl get svc -n ${NAMESPACE}

Exercise 3: Using a client to query TensorFlow Serving Model Server

In this exercise, we'll use a client to query the TensorFlow Serving model server.

cd 9-serving

If you don't have virtualenv installed, you can install with:

pip install virtualenv

Setting up our virtual environment:

virtualenv venv
source venv/bin/activate
pip install -r requirements.txt

Starting our query from the client:

export TF_MODEL_SERVER_HOST=$(kubectl get svc ${MODEL_NAME} -n ${NAMESPACE} --template="{{range .status.loadBalancer.ingress}}{{.ip}}{{end}}")

export TF_MNIST_IMAGE_PATH=data/7.png

python mnist_client.py

If everything is working correctly, you should see the output from the model and the inference.

Sample output:

outputs {
  key: "classes"
  value {
    dtype: DT_UINT8
    tensor_shape {
      dim {
        size: 1
      }
    }
    int_val: 7
  }
}
outputs {
  key: "predictions"
  value {
    dtype: DT_FLOAT
    tensor_shape {
      dim {
        size: 1
      }
      dim {
        size: 10
      }
    }
    float_val: 0.0
    float_val: 0.0
    float_val: 0.0
    float_val: 0.0
    float_val: 0.0
    float_val: 0.0
    float_val: 0.0
    float_val: 1.0
    float_val: 0.0
    float_val: 0.0
  }
}


............................
............................
............................
............................
............................
............................
............................
..............@@@@@@........
..........@@@@@@@@@@........
........@@@@@@@@@@@@........
........@@@@@@@@.@@@........
........@@@@....@@@@........
................@@@@........
...............@@@@.........
...............@@@@.........
...............@@@..........
..............@@@@..........
..............@@@...........
.............@@@@...........
.............@@@............
............@@@@............
............@@@.............
............@@@.............
...........@@@..............
..........@@@@..............
..........@@@@..............
..........@@................
............................
Your model says the above number is... 7!

Next Step

10 - Going Further