openvino_docker_usage.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# OpenVINO Docker Usage\n",
    "\n",
    "## Goal\n",
    "This tutorial will guide you through step-by-step instructions using OpenVINO docker container\n",
    "* Step 1: Download a Keras-Tensorflow model and Freeze the model\n",
    "* Step 2: Download OpenVINO docker container\n",
    "* Step 3: Convert the Tensorflow Frozen model into OpenVINO model using the OpenVINO model optimizer via Docker\n",
    "* Step 4: Run Inference using OpenVINO Inference Engine via Docker\n",
    "* Step 5: Benchmark with OpenVINO benchmark app via Docker\n",
    "\n",
    "## Prerequisites\n",
    "1. **Install** Tensorflow and Keras.\n",
    "1. **Install** Docker CE\n",
    "To install docker on  Linux:\n",
    "\n",
    "```\n",
    "curl -fsSL https://get.docker.com -o get-docker.sh &&\n",
    "sudo sh get-docker.sh && \n",
    "sudo usermod -aG docker $USER\n",
    "```\n",
    "\n",
    "\n",
    "- Click [here](https://docs.docker.com/install/linux/docker-ce/ubuntu) for complete Ubuntu instructions. For other OS platforms, see [here](https://docs.docker.com/install/).\n",
    "- Setup docker to be used as a non-root user, to run `docker` commands without `sudo` . \n",
    "```sudo usermod -aG docker `whoami` ```\n",
    "- Exit and restart your SSH session so that your username is in effect in the docker group.\n",
    "- After exiting and restarting your SSH session, you should be able to run `docker` commands without `sudo`.\n",
    "- Test installation by running ```\tdocker run hello-world ```\n",
    "\n",
    "**NOTE**: If your machine is behind a proxy, See **HTTP/HTTPS proxy** section [here](https://github.com/ravi9/misc-readmes/blob/master/install-docker-ubuntu.md#configure-proxy-for-docker-if-you-are-behind-a-proxy)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: Download a Keras-Tensorflow model and Freeze the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow_core/__init__.py:1467: The name tf.estimator.inputs is deprecated. Please use tf.compat.v1.estimator.inputs instead.\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "# import the necessary packages\n",
    "\n",
    "import tensorflow as tf\n",
    "from tensorflow.python.framework import graph_util\n",
    "from tensorflow.python.framework import graph_io\n",
    "import tensorflow.keras.backend as K\n",
    "import keras\n",
    "import numpy as np\n",
    "import shutil, sys, os\n",
    "from pathlib import Path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:74: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.\n",
      "\n",
      "WARNING:tensorflow:From /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.\n",
      "\n",
      "WARNING:tensorflow:From /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.\n",
      "\n",
      "WARNING:tensorflow:From /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.\n",
      "\n",
      "WARNING:tensorflow:From /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:184: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.\n",
      "\n",
      "WARNING:tensorflow:From /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:186: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.\n",
      "\n",
      "WARNING:tensorflow:From /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:190: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.\n",
      "\n",
      "WARNING:tensorflow:From /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead.\n",
      "\n",
      "WARNING:tensorflow:From /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead.\n",
      "\n",
      "WARNING:tensorflow:From /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead.\n",
      "\n",
      "WARNING:tensorflow:From /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:133: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.\n",
      "\n",
      "WARNING:tensorflow:From /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:3976: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead.\n",
      "\n",
      "WARNING:tensorflow:From /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:3980: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "keras_model = keras.applications.inception_v3.InceptionV3(include_top=True, weights='imagenet', input_tensor=None, input_shape=None, pooling=None, classes=1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Freeze the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/ec2-user/SageMaker/ravi//frozen_model exists already. Deleting the folder\n",
      "WARNING:tensorflow:OMP_NUM_THREADS is no longer used by the default Keras config. To configure the number of threads, use tf.config.threading APIs.\n"
     ]
    }
   ],
   "source": [
    "pwd = !pwd\n",
    "model_output_path = pwd[0] + \"/\"\n",
    "frozen_model_name = 'keras_tf_model_frozen.pb'\n",
    "frozen_model_path = model_output_path + \"/frozen_model\"\n",
    "\n",
    "if os.path.isdir(frozen_model_path):\n",
    "    print (frozen_model_path, \"exists already. Deleting the folder\")\n",
    "    shutil.rmtree(frozen_model_path)\n",
    "\n",
    "tf.get_logger().setLevel('INFO')\n",
    "\n",
    "#K.clear_session()\n",
    "K._LEARNING_PHASE = tf.constant(0)\n",
    "K.set_learning_phase(False)\n",
    "K.set_learning_phase(0)\n",
    "K.set_image_data_format('channels_last')\n",
    "\n",
    "# if you have an exiting Keras model, then do the following to load.\n",
    "# keras_model_path = model_output_path + model_fname\n",
    "# frozen_model_name = str(Path(keras_model_path).name) + '.pb'\n",
    "# keras_model = tf.keras.models.load_model(keras_model_path, compile=False)\n",
    "\n",
    "num_output = len(keras_model.outputs)\n",
    "\n",
    "predictions = [None] * num_output\n",
    "pred_node_names = [None] * num_output\n",
    "\n",
    "for i in range(num_output):\n",
    "    pred_node_names[i] = 'output_node' + str(i)\n",
    "    predictions[i] = tf.identity(keras_model.outputs[i], name=pred_node_names[i])\n",
    "\n",
    "sess = K.get_session()\n",
    "sess.run(tf.global_variables_initializer())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Make Sure the the output node name is the same, below you can see `output_node0` is the same"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[<tf.Tensor 'output_node0:0' shape=(?, 1000) dtype=float32>]\n",
      "['output_node0']\n"
     ]
    }
   ],
   "source": [
    "print(predictions)\n",
    "print(pred_node_names)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " --- Extracting default graph...\n",
      " --- Converting variables to constants...\n",
      "WARNING:tensorflow:From <ipython-input-5-be0a9f70b0f4>:5: convert_variables_to_constants (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.compat.v1.graph_util.convert_variables_to_constants`\n",
      "WARNING:tensorflow:From /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow_core/python/framework/graph_util_impl.py:277: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.compat.v1.graph_util.extract_sub_graph`\n",
      "INFO:tensorflow:Froze 378 variables.\n",
      "INFO:tensorflow:Converted 378 variables to const ops.\n",
      " --- Removing training nodes...\n",
      "WARNING:tensorflow:From <ipython-input-5-be0a9f70b0f4>:7: remove_training_nodes (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.compat.v1.graph_util.remove_training_nodes`\n",
      " --- Saving Frozen graph...\n",
      "Model input name =  input_1\n",
      "Model input shape =  (?, 299, 299, 3)\n",
      "Number of model outputs =  1\n",
      "Model output name =  predictions/Softmax\n",
      "Model output shape =  (?, 1000)\n",
      "TensorFlow Frozen model is saved in: /home/ec2-user/SageMaker/ravi//frozen_model/keras_tf_model_frozen.pb\n"
     ]
    }
   ],
   "source": [
    "with sess.as_default():\n",
    "    print(' --- Extracting default graph...')\n",
    "    def_graph = sess.graph.as_graph_def()\n",
    "    print(' --- Converting variables to constants...')\n",
    "    constant_graph = tf.compat.v1.graph_util.convert_variables_to_constants(sess, def_graph, pred_node_names)\n",
    "    print(' --- Removing training nodes...')\n",
    "    infer_graph = tf.compat.v1.graph_util.remove_training_nodes(constant_graph) \n",
    "    print(' --- Saving Frozen graph...')\n",
    "    graph_io.write_graph(infer_graph, frozen_model_path, frozen_model_name, as_text=False)\n",
    "\n",
    "#print(\"\\nModel input = \", keras_model_path)\n",
    "print(\"Model input name = \", keras_model.input.op.name)\n",
    "print(\"Model input shape = \", keras_model.input.shape)\n",
    "print(\"Number of model outputs = \", num_output)\n",
    "print(\"Model output name = \", keras_model.output.op.name)\n",
    "print(\"Model output shape = \", keras_model.output.shape)\n",
    "print(\"TensorFlow Frozen model is saved in: {}/{}\".format(frozen_model_path, frozen_model_name))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Download OpenVINO docker container"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using default tag: latest\n",
      "latest: Pulling from iotgvmc/openvino\n",
      "Digest: sha256:273c3429bd14f12da6aabee0c4c2afb69fc7b031265fb61c0325328208bdd1a0\n",
      "Status: Image is up to date for iotgvmc/openvino:latest\n"
     ]
    }
   ],
   "source": [
    "# Download OpenVINO container\n",
    "!docker pull iotgvmc/openvino"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Convert the Tensorflow Frozen model into OpenVINO model using the OpenVINO model optimizer via Docker"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "docker run -it -v /home/ec2-user/SageMaker/ravi/:/mnt/ iotgvmc/openvino               mo_tf.py               \n",
      "--input_model /mnt/frozen_model/keras_tf_model_frozen.pb               \n",
      "--input_shape [1,299,299,3]               \n",
      "--data_type FP32               \n",
      "--output_dir /mnt//IR_models/FP32/                \n",
      "--model_name ov_inceptionv3\n",
      "Model Optimizer arguments:\n",
      "Common parameters:\n",
      "\t- Path to the Input Model: \t/mnt/frozen_model/keras_tf_model_frozen.pb\n",
      "\t- Path for generated IR: \t/mnt//IR_models/FP32/\n",
      "\t- IR output name: \tov_inceptionv3\n",
      "\t- Log level: \tERROR\n",
      "\t- Batch: \tNot specified, inherited from the model\n",
      "\t- Input layers: \tNot specified, inherited from the model\n",
      "\t- Output layers: \tNot specified, inherited from the model\n",
      "\t- Input shapes: \t[1,299,299,3]\n",
      "\t- Mean values: \tNot specified\n",
      "\t- Scale values: \tNot specified\n",
      "\t- Scale factor: \tNot specified\n",
      "\t- Precision of IR: \tFP32\n",
      "\t- Enable fusing: \tTrue\n",
      "\t- Enable grouped convolutions fusing: \tTrue\n",
      "\t- Move mean values to preprocess section: \tFalse\n",
      "\t- Reverse input channels: \tFalse\n",
      "TensorFlow specific parameters:\n",
      "\t- Input model in text protobuf format: \tFalse\n",
      "\t- Path to model dump for TensorBoard: \tNone\n",
      "\t- List of shared libraries with TensorFlow custom layers implementation: \tNone\n",
      "\t- Update the configuration file with input/output node names: \tNone\n",
      "\t- Use configuration file used to generate the model with Object Detection API: \tNone\n",
      "\t- Use the config file: \tNone\n",
      "Model Optimizer version: \t2020.2.0-60-g0bc66e26ff\n",
      "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint8 = np.dtype([(\"qint8\", np.int8, 1)])\n",
      "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint8 = np.dtype([(\"quint8\", np.uint8, 1)])\n",
      "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint16 = np.dtype([(\"qint16\", np.int16, 1)])\n",
      "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint16 = np.dtype([(\"quint16\", np.uint16, 1)])\n",
      "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint32 = np.dtype([(\"qint32\", np.int32, 1)])\n",
      "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  np_resource = np.dtype([(\"resource\", np.ubyte, 1)])\n",
      "/usr/local/lib/python3.6/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint8 = np.dtype([(\"qint8\", np.int8, 1)])\n",
      "/usr/local/lib/python3.6/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint8 = np.dtype([(\"quint8\", np.uint8, 1)])\n",
      "/usr/local/lib/python3.6/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint16 = np.dtype([(\"qint16\", np.int16, 1)])\n",
      "/usr/local/lib/python3.6/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint16 = np.dtype([(\"quint16\", np.uint16, 1)])\n",
      "/usr/local/lib/python3.6/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint32 = np.dtype([(\"qint32\", np.int32, 1)])\n",
      "/usr/local/lib/python3.6/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  np_resource = np.dtype([(\"resource\", np.ubyte, 1)])\n",
      "\n",
      "[ SUCCESS ] Generated IR version 10 model.\n",
      "[ SUCCESS ] XML file: /mnt//IR_models/FP32/ov_inceptionv3.xml\n",
      "[ SUCCESS ] BIN file: /mnt//IR_models/FP32/ov_inceptionv3.bin\n",
      "[ SUCCESS ] Total execution time: 39.52 seconds. \n",
      "[ SUCCESS ] Memory consumed: 774 MB. \n",
      "\n",
      "OpenVINO model saved in: /home/ec2-user/SageMaker/ravi/IR_models/FP32/\n"
     ]
    }
   ],
   "source": [
    "# Convert TF Frozen graph into OpenVINO IR using OV model optimizer via Docker container.\n",
    "\n",
    "frozen_model = frozen_model_path + \"/\" + frozen_model_name\n",
    "ovData_type = \"FP32\"\n",
    "ovModel_path = \"IR_models/\" + ovData_type + \"/\"\n",
    "docker_mnt_path = \"/mnt/\"\n",
    "docker_frozen_model_path = docker_mnt_path + \"frozen_model/\" + frozen_model_name\n",
    "docker_OvModel_Path = docker_mnt_path + \"/\" + ovModel_path\n",
    "ovModel_name = \"ov_inceptionv3\"\n",
    "\n",
    "model_inp_shape = keras_model.input.get_shape().as_list()\n",
    "#set batch size\n",
    "model_inp_shape[0] = 1\n",
    "model_inp_shape = str(model_inp_shape).replace(\" \", \"\")\n",
    "\n",
    "if not os.path.exists(frozen_model):\n",
    "    print(frozen_model + ' doesn\\'t exist. Please make sure you have a trained keras to TF frozen model')\n",
    "\n",
    "docker_mo_cmd = \"docker run -it -v {}:{} iotgvmc/openvino \\\n",
    "              mo_tf.py \\\n",
    "              --input_model {} \\\n",
    "              --input_shape {} \\\n",
    "              --data_type {} \\\n",
    "              --output_dir {}  \\\n",
    "              --model_name {}\".format(model_output_path, docker_mnt_path, docker_frozen_model_path, \n",
    "                                               model_inp_shape, ovData_type, docker_OvModel_Path, ovModel_name )\n",
    "\n",
    "print(\"\\n--\".join(docker_mo_cmd.split(\"--\")))\n",
    "\n",
    "# Run the OpenVINO model optimizer via Docker container\n",
    "!{docker_mo_cmd}\n",
    "\n",
    "if os.path.exists(model_output_path+ovModel_path):\n",
    "    print(\"\\nOpenVINO model saved in: {}{}\".format(model_output_path, ovModel_path))\n",
    "    # Update permissions of the files created by docker. Change owner/group to the current user.\n",
    "    update_permissions_cmd = \"sudo chown $USER:$USER -R {}/{}\".format(model_output_path, ovModel_path)\n",
    "    !{update_permissions_cmd}\n",
    "\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 5: Benchmark with OpenVINO benchmark app via Docker\n",
    "#### [Click here for more info on OpenVINO Benchmark Tool](https://docs.openvinotoolkit.org/latest/_inference_engine_tools_benchmark_tool_README.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "docker run -it -v /home/ec2-user/SageMaker/ravi/:/mnt/ iotgvmc/openvino               python3 /opt/intel/openvino/deployment_tools/tools/benchmark_tool/benchmark_app.py               \n",
      "--path_to_model /mnt//IR_models/FP32/ov_inceptionv3.xml               \n",
      "--number_streams 1               -nireq 1               \n",
      "--time 30\n",
      "[Step 1/11] Parsing and validating input arguments\n",
      "[Step 2/11] Loading Inference Engine\n",
      "[ INFO ] InferenceEngine:\n",
      "         API version............. 2.1.42025\n",
      "[ INFO ] Device info\n",
      "         CPU\n",
      "         MKLDNNPlugin............ version 2.1\n",
      "         Build................... 42025\n",
      "\n",
      "[Step 3/11] Reading the Intermediate Representation network\n",
      "[ INFO ] Read network took 197.34 ms\n",
      "[Step 4/11] Resizing network to match image sizes and given batch\n",
      "[ INFO ] Network batch size: 1\n",
      "[Step 5/11] Configuring input of the model\n",
      "[Step 6/11] Setting device configuration\n",
      "[Step 7/11] Loading the model to the device\n",
      "[ INFO ] Load network took 663.51 ms\n",
      "[Step 8/11] Setting optimal runtime parameters\n",
      "[Step 9/11] Creating infer requests and filling input blobs with images\n",
      "[ INFO ] Network input 'input_1' precision U8, dimensions (NCHW): 1 3 299 299\n",
      "[ WARNING ] No input files were given: all inputs will be filled with random values!\n",
      "[ INFO ] Infer Request 0 filling\n",
      "[ INFO ] Fill input 'input_1' with random values (image is expected)\n",
      "[Step 10/11] Measuring performance (Start inference asyncronously, 1 inference requests using 1 streams for CPU, limits: 30000 ms duration)\n",
      "[Step 11/11] Dumping statistics report\n",
      "Count:      332 iterations\n",
      "Duration:   30167.16 ms\n",
      "Latency:    90.36 ms\n",
      "Throughput: 11.01 FPS\n"
     ]
    }
   ],
   "source": [
    "# Mount the directory on the docker.\n",
    "# Use the OpenVINO benchmark_app to benchmark\n",
    "# The benchmark_app runs the inference for 30sec with batch_size=1\n",
    "docker_ovModel_xml = docker_OvModel_Path + ovModel_name + \".xml\"\n",
    "number_streams = 1\n",
    "bench_runtime_sec = 30\n",
    "\n",
    "docker_bench_cmd = \"docker run -it -v {}:{} iotgvmc/openvino \\\n",
    "              python3 /opt/intel/openvino/deployment_tools/tools/benchmark_tool/benchmark_app.py \\\n",
    "              --path_to_model {} \\\n",
    "              --number_streams {} \\\n",
    "              -nireq 1 \\\n",
    "              --time {}\".format(model_output_path, docker_mnt_path, docker_ovModel_xml, \n",
    "                                               number_streams, bench_runtime_sec )\n",
    "\n",
    "print(\"\\n--\".join(docker_bench_cmd.split(\"--\")))\n",
    "\n",
    "# Run the OpenVINO benchmark app via Docker container\n",
    "!{docker_bench_cmd}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_tensorflow_p36",
   "language": "python",
   "name": "conda_tensorflow_p36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}