xgboost-high-level-apis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train and deploy on Kubeflow from Notebooks\n",
    "\n",
    "This notebook introduces you to using Kubeflow Fairing to train and deploy a model to Kubeflow on Google Kubernetes Engine (GKE), and Google Cloud ML Engine. This notebook demonstrate how to:\n",
    " \n",
    "* Train an XGBoost model in a local notebook,\n",
    "* Use Kubeflow Fairing to train an XGBoost model remotely on Kubeflow,\n",
    "* Use Kubeflow Fairing to train an XGBoost model remotely on Cloud ML Engine,\n",
    "* Use Kubeflow Fairing to deploy a trained model to Kubeflow, and\n",
    "* Call the deployed endpoint for predictions.\n",
    "\n",
    "To learn more about how to run this notebook locally, see the guide to [training and deploying on GCP from a local notebook][gcp-local-notebook].\n",
    "\n",
    "[gcp-local-notebook]: https://kubeflow.org/docs/fairing/gcp-local-notebook/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set up your notebook for training an XGBoost model\n",
    "\n",
    "Import the libraries required to train this model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: joblib in /opt/conda/lib/python3.6/site-packages (0.13.2)\n",
      "\u001b[33mYou are using pip version 19.0.1, however version 19.1.1 is available.\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\n",
      "Requirement already satisfied: sklearn in /opt/conda/lib/python3.6/site-packages (0.0)\n",
      "Requirement already satisfied: scikit-learn in /opt/conda/lib/python3.6/site-packages (from sklearn) (0.21.1)\n",
      "Requirement already satisfied: numpy>=1.11.0 in /opt/conda/lib/python3.6/site-packages (from scikit-learn->sklearn) (1.16.2)\n",
      "Requirement already satisfied: scipy>=0.17.0 in /opt/conda/lib/python3.6/site-packages (from scikit-learn->sklearn) (1.2.1)\n",
      "Requirement already satisfied: joblib>=0.11 in /opt/conda/lib/python3.6/site-packages (from scikit-learn->sklearn) (0.13.2)\n",
      "\u001b[33mYou are using pip version 19.0.1, however version 19.1.1 is available.\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!pip3 install --user joblib\n",
    "!pip3 install --user sklearn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<module 'ames' from '/home/jovyan/git_jlewi-kubecon-demo/ames.py'>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import argparse\n",
    "import logging\n",
    "import os\n",
    "import joblib\n",
    "import sys\n",
    "from pathlib import Path\n",
    "import pandas as pd\n",
    "from sklearn.metrics import mean_absolute_error\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.impute import SimpleImputer\n",
    "from xgboost import XGBRegressor\n",
    "import ames\n",
    "from importlib import reload\n",
    "\n",
    "reload(ames)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Adding /home/jovyan/git_jlewi-kubecon-demo/fairing to path\n"
     ]
    }
   ],
   "source": [
    "fairing_code = os.path.join(Path.home(), \"git_jlewi-kubecon-demo\", \"fairing\")\n",
    "\n",
    "if os.path.exists(fairing_code):    \n",
    "    logging.info(\"Adding %s to path\", fairing_code)\n",
    "    sys.path = [fairing_code] + sys.path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "logging.basicConfig(format='%(message)s')\n",
    "logging.getLogger().setLevel(logging.INFO)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define a class for your model, with methods for training and prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "class HousingServe(object):    \n",
    "    def __init__(self):\n",
    "        self.train_input = \"ames_dataset/train.csv\"\n",
    "        self.n_estimators = 50\n",
    "        self.learning_rate = 0.1\n",
    "        self.model_file = \"trained_ames_model.dat\"\n",
    "        self.model = None\n",
    "\n",
    "    def train(self):\n",
    "        (train_X, train_y), (test_X, test_y) = ames.read_input(self.train_input)\n",
    "        model = ames.train_model(train_X,\n",
    "                                 train_y,\n",
    "                                 test_X,\n",
    "                                 test_y,\n",
    "                                 self.n_estimators,\n",
    "                                 self.learning_rate)\n",
    "\n",
    "        ames.eval_model(model, test_X, test_y)\n",
    "        ames.save_model(model, self.model_file)\n",
    "\n",
    "    def predict(self, X, feature_names):\n",
    "        \"\"\"Predict using the model for given ndarray.\"\"\"\n",
    "        if not self.model:\n",
    "            self.model = joblib.load(self.model_file)\n",
    "        # Do any preprocessing\n",
    "        prediction = self.model.predict(data=X)\n",
    "        # Do any postprocessing\n",
    "        return [[prediction.item(0), prediction.item(0)]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train an XGBoost model in a notebook\n",
    "\n",
    "Call `HousingServe().train()` to train your model, and then evaluate and save your trained model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0]\tvalidation_0-rmse:177514\n",
      "Will train until validation_0-rmse hasn't improved in 40 rounds.\n",
      "[1]\tvalidation_0-rmse:161858\n",
      "[2]\tvalidation_0-rmse:147237\n",
      "[3]\tvalidation_0-rmse:134132\n",
      "[4]\tvalidation_0-rmse:122224\n",
      "[5]\tvalidation_0-rmse:111538\n",
      "[6]\tvalidation_0-rmse:102142\n",
      "[7]\tvalidation_0-rmse:93392.3\n",
      "[8]\tvalidation_0-rmse:85824.6\n",
      "[9]\tvalidation_0-rmse:79667.6\n",
      "[10]\tvalidation_0-rmse:73463.4\n",
      "[11]\tvalidation_0-rmse:68059.4\n",
      "[12]\tvalidation_0-rmse:63350.5\n",
      "[13]\tvalidation_0-rmse:59732.1\n",
      "[14]\tvalidation_0-rmse:56260.7\n",
      "[15]\tvalidation_0-rmse:53392.6\n",
      "[16]\tvalidation_0-rmse:50770.8\n",
      "[17]\tvalidation_0-rmse:48107.8\n",
      "[18]\tvalidation_0-rmse:45923.9\n",
      "[19]\tvalidation_0-rmse:44154.2\n",
      "[20]\tvalidation_0-rmse:42488.1\n",
      "[21]\tvalidation_0-rmse:41263.3\n",
      "[22]\tvalidation_0-rmse:40212.8\n",
      "[23]\tvalidation_0-rmse:39089.1\n",
      "[24]\tvalidation_0-rmse:37691.1\n",
      "[25]\tvalidation_0-rmse:36875.2\n",
      "[26]\tvalidation_0-rmse:36276.2\n",
      "[27]\tvalidation_0-rmse:35444.1\n",
      "[28]\tvalidation_0-rmse:34831.5\n",
      "[29]\tvalidation_0-rmse:34205.4\n",
      "[30]\tvalidation_0-rmse:33831.9\n",
      "[31]\tvalidation_0-rmse:33183.6\n",
      "[32]\tvalidation_0-rmse:33019.4\n",
      "[33]\tvalidation_0-rmse:32680\n",
      "[34]\tvalidation_0-rmse:32438.5\n",
      "[35]\tvalidation_0-rmse:32130.4\n",
      "[36]\tvalidation_0-rmse:31644.2\n",
      "[37]\tvalidation_0-rmse:31248.9\n",
      "[38]\tvalidation_0-rmse:31059.8\n",
      "[39]\tvalidation_0-rmse:30862.4\n",
      "[40]\tvalidation_0-rmse:30754\n",
      "[41]\tvalidation_0-rmse:30561.6\n",
      "[42]\tvalidation_0-rmse:30416.9\n",
      "[43]\tvalidation_0-rmse:30156.4\n",
      "[44]\tvalidation_0-rmse:29852.9\n",
      "[45]\tvalidation_0-rmse:29486.6\n",
      "[46]\tvalidation_0-rmse:29158.8\n",
      "[47]\tvalidation_0-rmse:29017\n",
      "[48]\tvalidation_0-rmse:28973.9\n",
      "[49]\tvalidation_0-rmse:28787.7\n",
      "Best RMSE on eval: %.2f with %d rounds 28787.720703 50\n"
     ]
    }
   ],
   "source": [
    "HousingServe().train()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set up Kubeflow Fairing for training and predictions\n",
    "\n",
    "Import the `fairing` library and configure the environment that your training or prediction job will run in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import fairing\n",
    "\n",
    "# Setting up google container repositories (GCR) for storing output containers\n",
    "# You can use any docker container registry istead of GCR\n",
    "GCP_PROJECT = fairing.cloud.gcp.guess_project_name()\n",
    "DOCKER_REGISTRY = 'gcr.io/{}/fairing-job'.format(GCP_PROJECT)\n",
    "PY_VERSION = \".\".join([str(x) for x in sys.version_info[0:3]])\n",
    "BASE_IMAGE = 'python:{}'.format(PY_VERSION)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train an XGBoost model remotely on Kubeflow\n",
    "\n",
    "Import the `TrainJob` and `KubeflowGKEBackend` classes. Kubeflow Fairing packages the `HousingServe` class, the training data, and the training job's software prerequisites as a Docker image. Then Kubeflow Fairing deploys and runs the training job on Kubeflow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "rerun_training = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:fairing.ml_tasks.tasks:Using preprocessor: <class 'fairing.preprocessors.function.FunctionPreProcessor'>\n",
      "WARNING:fairing.ml_tasks.tasks:Using docker registry: gcr.io/code-search-demo/fairing-job\n",
      "WARNING:fairing.ml_tasks.tasks:Using builder: <class 'fairing.builders.cluster.cluster.ClusterBuilder'>\n",
      "WARNING:root:/opt/conda/lib/python3.6/site-packages/fairing/__init__.py already exists in Fairing context, skipping...\n",
      "WARNING:root:/opt/conda/lib/python3.6/site-packages/fairing/__init__.py already exists in Fairing context, skipping...\n",
      "WARNING:fairing.kubernetes.manager:Waiting for fairing-builder-cqs62 to start...\n",
      "WARNING:fairing.kubernetes.manager:Waiting for fairing-builder-cqs62 to start...\n",
      "WARNING:fairing.kubernetes.manager:Waiting for fairing-builder-cqs62 to start...\n",
      "INFO:fairing.kubernetes.manager:Pod started running True\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[36mINFO\u001b[0m[0000] Downloading base image python:3.6.7\n",
      "ERROR: logging before flag.Parse: E0517 18:36:50.906574       1 metadata.go:142] while reading 'google-dockercfg' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg\n",
      "ERROR: logging before flag.Parse: E0517 18:36:50.910509       1 metadata.go:159] while reading 'google-dockercfg-url' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg-url\n",
      "2019/05/17 18:36:50 No matching credentials were found, falling back on anonymous\n",
      "\u001b[36mINFO\u001b[0m[0000] Executing 0 build triggers\n",
      "\u001b[36mINFO\u001b[0m[0000] Unpacking rootfs as cmd RUN if [ -e requirements.txt ];then pip install --no-cache -r requirements.txt; fi requires it.\n",
      "\u001b[36mINFO\u001b[0m[0020] Taking snapshot of full filesystem...\n",
      "\u001b[36mINFO\u001b[0m[0028] Skipping paths under /dev, as it is a whitelisted directory\n",
      "\u001b[36mINFO\u001b[0m[0028] Skipping paths under /etc/secrets, as it is a whitelisted directory\n",
      "\u001b[36mINFO\u001b[0m[0028] Skipping paths under /kaniko, as it is a whitelisted directory\n",
      "\u001b[36mINFO\u001b[0m[0028] Skipping paths under /proc, as it is a whitelisted directory\n",
      "\u001b[36mINFO\u001b[0m[0028] Skipping paths under /sys, as it is a whitelisted directory\n",
      "\u001b[36mINFO\u001b[0m[0028] Skipping paths under /var/run, as it is a whitelisted directory\n",
      "\u001b[36mINFO\u001b[0m[0038] WORKDIR /app/\n",
      "\u001b[36mINFO\u001b[0m[0038] cmd: workdir\n",
      "\u001b[36mINFO\u001b[0m[0038] Changed working directory to /app/\n",
      "\u001b[36mINFO\u001b[0m[0038] Creating directory /app/\n",
      "\u001b[36mINFO\u001b[0m[0038] Taking snapshot of files...\n",
      "\u001b[36mINFO\u001b[0m[0038] ENV FAIRING_RUNTIME 1\n",
      "\u001b[36mINFO\u001b[0m[0038] Using files from context: [/kaniko/buildcontext/app/requirements.txt]\n",
      "\u001b[36mINFO\u001b[0m[0038] COPY /app//requirements.txt /app/\n",
      "\u001b[36mINFO\u001b[0m[0038] Taking snapshot of files...\n",
      "\u001b[36mINFO\u001b[0m[0038] RUN if [ -e requirements.txt ];then pip install --no-cache -r requirements.txt; fi\n",
      "\u001b[36mINFO\u001b[0m[0038] cmd: /bin/sh\n",
      "\u001b[36mINFO\u001b[0m[0038] args: [-c if [ -e requirements.txt ];then pip install --no-cache -r requirements.txt; fi]\n",
      "Collecting google-cloud-storage (from -r requirements.txt (line 1))\n",
      "  Downloading https://files.pythonhosted.org/packages/34/7c/e2f563c1af1d9042b1a4e03dfcdd73fc31a5236fd108440b4981b4eaea8d/google_cloud_storage-1.16.0-py2.py3-none-any.whl (65kB)\n",
      "Collecting pandas (from -r requirements.txt (line 2))\n",
      "  Downloading https://files.pythonhosted.org/packages/19/74/e50234bc82c553fecdbd566d8650801e3fe2d6d8c8d940638e3d8a7c5522/pandas-0.24.2-cp36-cp36m-manylinux1_x86_64.whl (10.1MB)\n",
      "Collecting joblib (from -r requirements.txt (line 3))\n",
      "  Downloading https://files.pythonhosted.org/packages/cd/c1/50a758e8247561e58cb87305b1e90b171b8c767b15b12a1734001f41d356/joblib-0.13.2-py2.py3-none-any.whl (278kB)\n",
      "Collecting numpy (from -r requirements.txt (line 4))\n",
      "  Downloading https://files.pythonhosted.org/packages/c1/e2/4db8df8f6cddc98e7d7c537245ef2f4e41a1ed17bf0c3177ab3cc6beac7f/numpy-1.16.3-cp36-cp36m-manylinux1_x86_64.whl (17.3MB)\n",
      "Collecting xgboost (from -r requirements.txt (line 5))\n",
      "  Downloading https://files.pythonhosted.org/packages/6a/49/7e10686647f741bd9c8918b0decdb94135b542fe372ca1100739b8529503/xgboost-0.82-py2.py3-none-manylinux1_x86_64.whl (114.0MB)\n",
      "Collecting sklearn (from -r requirements.txt (line 6))\n",
      "  Downloading https://files.pythonhosted.org/packages/1e/7a/dbb3be0ce9bd5c8b7e3d87328e79063f8b263b2b1bfa4774cb1147bfcd3f/sklearn-0.0.tar.gz\n",
      "Collecting seldon-core (from -r requirements.txt (line 7))\n",
      "  Downloading https://files.pythonhosted.org/packages/31/44/51cac9c7df386fbd740ae9c24b2d28380974d137c5c1532eae0534c9bb67/seldon_core-0.2.7.1-py3-none-any.whl\n",
      "Collecting google-auth>=1.2.0 (from google-cloud-storage->-r requirements.txt (line 1))\n",
      "  Downloading https://files.pythonhosted.org/packages/c5/9b/ed0516cc1f7609fb0217e3057ff4f0f9f3e3ce79a369c6af4a6c5ca25664/google_auth-1.6.3-py2.py3-none-any.whl (73kB)\n",
      "Collecting google-api-core<2.0.0dev,>=1.6.0 (from google-cloud-storage->-r requirements.txt (line 1))\n",
      "  Downloading https://files.pythonhosted.org/packages/32/df/1af46c943d13332e0f113b8d1478d9d466a39db5a2c9edf1e533b720421a/google_api_core-1.11.0-py2.py3-none-any.whl (66kB)\n",
      "Collecting google-cloud-core<2.0dev,>=1.0.0 (from google-cloud-storage->-r requirements.txt (line 1))\n",
      "  Downloading https://files.pythonhosted.org/packages/34/ba/251d9b6a1695d25d9f081a3db537b0dfb15edaf2037b423e4a98280df7f9/google_cloud_core-1.0.0-py2.py3-none-any.whl\n",
      "Collecting google-resumable-media>=0.3.1 (from google-cloud-storage->-r requirements.txt (line 1))\n",
      "  Downloading https://files.pythonhosted.org/packages/e2/5d/4bc5c28c252a62efe69ed1a1561da92bd5af8eca0cdcdf8e60354fae9b29/google_resumable_media-0.3.2-py2.py3-none-any.whl\n",
      "Collecting python-dateutil>=2.5.0 (from pandas->-r requirements.txt (line 2))\n",
      "  Downloading https://files.pythonhosted.org/packages/41/17/c62faccbfbd163c7f57f3844689e3a78bae1f403648a6afb1d0866d87fbb/python_dateutil-2.8.0-py2.py3-none-any.whl (226kB)\n",
      "Collecting pytz>=2011k (from pandas->-r requirements.txt (line 2))\n",
      "  Downloading https://files.pythonhosted.org/packages/3d/73/fe30c2daaaa0713420d0382b16fbb761409f532c56bdcc514bf7b6262bb6/pytz-2019.1-py2.py3-none-any.whl (510kB)\n",
      "Collecting scipy (from xgboost->-r requirements.txt (line 5))\n",
      "  Downloading https://files.pythonhosted.org/packages/72/4c/5f81e7264b0a7a8bd570810f48cd346ba36faedbd2ba255c873ad556de76/scipy-1.3.0-cp36-cp36m-manylinux1_x86_64.whl (25.2MB)\n",
      "Collecting scikit-learn (from sklearn->-r requirements.txt (line 6))\n",
      "  Downloading https://files.pythonhosted.org/packages/90/c7/401c231c445fb6fad135e92197da9c3e77983de169ff1887cc18af94498d/scikit_learn-0.21.1-cp36-cp36m-manylinux1_x86_64.whl (6.7MB)\n",
      "Collecting Flask-OpenTracing==0.2.0 (from seldon-core->-r requirements.txt (line 7))\n",
      "  Downloading https://files.pythonhosted.org/packages/1d/c4/0546b854a3f42af9ef959df9bd1108903698e175e7a07c057cdfaeeef718/Flask_OpenTracing-0.2.0-py2.py3-none-any.whl\n",
      "Collecting flatbuffers (from seldon-core->-r requirements.txt (line 7))\n",
      "  Downloading https://files.pythonhosted.org/packages/c9/84/adf5837f96c39990bc55afdfddf460b38b4562f50341359afa32e4a98de7/flatbuffers-1.11-py2.py3-none-any.whl\n",
      "Collecting pyyaml (from seldon-core->-r requirements.txt (line 7))\n",
      "  Downloading https://files.pythonhosted.org/packages/9f/2c/9417b5c774792634834e730932745bc09a7d36754ca00acf1ccd1ac2594d/PyYAML-5.1.tar.gz (274kB)\n",
      "Collecting jaeger-client==3.13.0 (from seldon-core->-r requirements.txt (line 7))\n",
      "  Downloading https://files.pythonhosted.org/packages/c8/a2/e9bd04cd660cbdffe0598173be068be23099fbd68e7a4a89b74440509130/jaeger-client-3.13.0.tar.gz (77kB)\n",
      "Collecting requests (from seldon-core->-r requirements.txt (line 7))\n",
      "  Downloading https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl (57kB)\n",
      "Collecting flask-cors (from seldon-core->-r requirements.txt (line 7))\n",
      "  Downloading https://files.pythonhosted.org/packages/65/cb/683f71ff8daa3aea0a5cbb276074de39f9ab66d3fbb8ad5efb5bb83e90d2/Flask_Cors-3.0.7-py2.py3-none-any.whl\n",
      "Collecting redis (from seldon-core->-r requirements.txt (line 7))\n",
      "  Downloading https://files.pythonhosted.org/packages/ac/a7/cff10cc5f1180834a3ed564d148fb4329c989cbb1f2e196fc9a10fa07072/redis-3.2.1-py2.py3-none-any.whl (65kB)\n",
      "Collecting tensorflow (from seldon-core->-r requirements.txt (line 7))\n",
      "  Downloading https://files.pythonhosted.org/packages/77/63/a9fa76de8dffe7455304c4ed635be4aa9c0bacef6e0633d87d5f54530c5c/tensorflow-1.13.1-cp36-cp36m-manylinux1_x86_64.whl (92.5MB)\n",
      "Collecting grpcio-opentracing (from seldon-core->-r requirements.txt (line 7))\n",
      "  Downloading https://files.pythonhosted.org/packages/db/82/2fcad380697c3dab25de76ee590bcab3eb9bbfb4add916044d7e83ec2b10/grpcio_opentracing-1.1.4-py3-none-any.whl\n",
      "Collecting opentracing<2,>=1.2.2 (from seldon-core->-r requirements.txt (line 7))\n",
      "  Downloading https://files.pythonhosted.org/packages/06/c2/90b35a1abdc639a5c6000d8202c70217d60e80f5b328870efb73fda71115/opentracing-1.3.0.tar.gz\n",
      "Collecting protobuf (from seldon-core->-r requirements.txt (line 7))\n",
      "  Downloading https://files.pythonhosted.org/packages/5a/aa/a858df367b464f5e9452e1c538aa47754d467023850c00b000287750fa77/protobuf-3.7.1-cp36-cp36m-manylinux1_x86_64.whl (1.2MB)\n",
      "Collecting grpcio (from seldon-core->-r requirements.txt (line 7))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  Downloading https://files.pythonhosted.org/packages/0a/9d/8bd5d0e516b196f59f1c4439b424b8d4fa62d492a4b531aae322d2d82a7b/grpcio-1.20.1-cp36-cp36m-manylinux1_x86_64.whl (2.1MB)\n",
      "Collecting flask (from seldon-core->-r requirements.txt (line 7))\n",
      "  Downloading https://files.pythonhosted.org/packages/9a/74/670ae9737d14114753b8c8fdf2e8bd212a05d3b361ab15b44937dfd40985/Flask-1.0.3-py2.py3-none-any.whl (92kB)\n",
      "Collecting six>=1.9.0 (from google-auth>=1.2.0->google-cloud-storage->-r requirements.txt (line 1))\n",
      "  Downloading https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl\n",
      "Collecting pyasn1-modules>=0.2.1 (from google-auth>=1.2.0->google-cloud-storage->-r requirements.txt (line 1))\n",
      "  Downloading https://files.pythonhosted.org/packages/91/f0/b03e00ce9fddf4827c42df1c3ce10c74eadebfb706231e8d6d1c356a4062/pyasn1_modules-0.2.5-py2.py3-none-any.whl (74kB)\n",
      "Collecting cachetools>=2.0.0 (from google-auth>=1.2.0->google-cloud-storage->-r requirements.txt (line 1))\n",
      "  Downloading https://files.pythonhosted.org/packages/39/2b/d87fc2369242bd743883232c463f28205902b8579cb68dcf5b11eee1652f/cachetools-3.1.0-py2.py3-none-any.whl\n",
      "Collecting rsa>=3.1.4 (from google-auth>=1.2.0->google-cloud-storage->-r requirements.txt (line 1))\n",
      "  Downloading https://files.pythonhosted.org/packages/02/e5/38518af393f7c214357079ce67a317307936896e961e35450b70fad2a9cf/rsa-4.0-py2.py3-none-any.whl\n",
      "Requirement already satisfied: setuptools>=34.0.0 in /usr/local/lib/python3.6/site-packages (from google-api-core<2.0.0dev,>=1.6.0->google-cloud-storage->-r requirements.txt (line 1)) (40.6.2)\n",
      "Collecting googleapis-common-protos!=1.5.4,<2.0dev,>=1.5.3 (from google-api-core<2.0.0dev,>=1.6.0->google-cloud-storage->-r requirements.txt (line 1))\n",
      "  Downloading https://files.pythonhosted.org/packages/81/ca/295515dff932fc41cdcb7c80babbba7fe96643d16ce34211fff0ecd48528/googleapis-common-protos-1.5.10.tar.gz\n",
      "Collecting threadloop<2,>=1 (from jaeger-client==3.13.0->seldon-core->-r requirements.txt (line 7))\n",
      "  Downloading https://files.pythonhosted.org/packages/d3/1d/8398c1645b97dc008d3c658e04beda01ede3d90943d40c8d56863cf891bd/threadloop-1.0.2.tar.gz\n",
      "Collecting thrift (from jaeger-client==3.13.0->seldon-core->-r requirements.txt (line 7))\n",
      "  Downloading https://files.pythonhosted.org/packages/c6/b4/510617906f8e0c5660e7d96fbc5585113f83ad547a3989b80297ac72a74c/thrift-0.11.0.tar.gz (52kB)\n",
      "Collecting tornado<5,>=4.3 (from jaeger-client==3.13.0->seldon-core->-r requirements.txt (line 7))\n",
      "  Downloading https://files.pythonhosted.org/packages/e3/7b/e29ab3d51c8df66922fea216e2bddfcb6430fb29620e5165b16a216e0d3c/tornado-4.5.3.tar.gz (484kB)\n",
      "Collecting chardet<3.1.0,>=3.0.2 (from requests->seldon-core->-r requirements.txt (line 7))\n",
      "  Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)\n",
      "Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests->seldon-core->-r requirements.txt (line 7))\n",
      "  Downloading https://files.pythonhosted.org/packages/39/ec/d93dfc69617a028915df914339ef66936ea976ef24fa62940fd86ba0326e/urllib3-1.25.2-py2.py3-none-any.whl (150kB)\n",
      "Collecting idna<2.9,>=2.5 (from requests->seldon-core->-r requirements.txt (line 7))\n",
      "  Downloading https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl (58kB)\n",
      "Collecting certifi>=2017.4.17 (from requests->seldon-core->-r requirements.txt (line 7))\n",
      "  Downloading https://files.pythonhosted.org/packages/60/75/f692a584e85b7eaba0e03827b3d51f45f571c2e793dd731e598828d380aa/certifi-2019.3.9-py2.py3-none-any.whl (158kB)\n",
      "Collecting astor>=0.6.0 (from tensorflow->seldon-core->-r requirements.txt (line 7))\n",
      "  Downloading https://files.pythonhosted.org/packages/35/6b/11530768cac581a12952a2aad00e1526b89d242d0b9f59534ef6e6a1752f/astor-0.7.1-py2.py3-none-any.whl\n",
      "Collecting gast>=0.2.0 (from tensorflow->seldon-core->-r requirements.txt (line 7))\n",
      "  Downloading https://files.pythonhosted.org/packages/4e/35/11749bf99b2d4e3cceb4d55ca22590b0d7c2c62b9de38ac4a4a7f4687421/gast-0.2.2.tar.gz\n",
      "Collecting absl-py>=0.1.6 (from tensorflow->seldon-core->-r requirements.txt (line 7))\n",
      "  Downloading https://files.pythonhosted.org/packages/da/3f/9b0355080b81b15ba6a9ffcf1f5ea39e307a2778b2f2dc8694724e8abd5b/absl-py-0.7.1.tar.gz (99kB)\n",
      "Collecting keras-preprocessing>=1.0.5 (from tensorflow->seldon-core->-r requirements.txt (line 7))\n",
      "  Downloading https://files.pythonhosted.org/packages/c0/bf/0315ef6a9fd3fc2346e85b0ff1f5f83ca17073f2c31ac719ab2e4da0d4a3/Keras_Preprocessing-1.0.9-py2.py3-none-any.whl (59kB)\n",
      "Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.6/site-packages (from tensorflow->seldon-core->-r requirements.txt (line 7)) (0.32.2)\n",
      "Collecting termcolor>=1.1.0 (from tensorflow->seldon-core->-r requirements.txt (line 7))\n",
      "  Downloading https://files.pythonhosted.org/packages/8a/48/a76be51647d0eb9f10e2a4511bf3ffb8cc1e6b14e9e4fab46173aa79f981/termcolor-1.1.0.tar.gz\n",
      "Collecting keras-applications>=1.0.6 (from tensorflow->seldon-core->-r requirements.txt (line 7))\n",
      "  Downloading https://files.pythonhosted.org/packages/90/85/64c82949765cfb246bbdaf5aca2d55f400f792655927a017710a78445def/Keras_Applications-1.0.7-py2.py3-none-any.whl (51kB)\n",
      "Collecting tensorboard<1.14.0,>=1.13.0 (from tensorflow->seldon-core->-r requirements.txt (line 7))\n",
      "  Downloading https://files.pythonhosted.org/packages/0f/39/bdd75b08a6fba41f098b6cb091b9e8c7a80e1b4d679a581a0ccd17b10373/tensorboard-1.13.1-py3-none-any.whl (3.2MB)\n",
      "Collecting tensorflow-estimator<1.14.0rc0,>=1.13.0 (from tensorflow->seldon-core->-r requirements.txt (line 7))\n",
      "  Downloading https://files.pythonhosted.org/packages/bb/48/13f49fc3fa0fdf916aa1419013bb8f2ad09674c275b4046d5ee669a46873/tensorflow_estimator-1.13.0-py2.py3-none-any.whl (367kB)\n",
      "Collecting click>=5.1 (from flask->seldon-core->-r requirements.txt (line 7))\n",
      "  Downloading https://files.pythonhosted.org/packages/fa/37/45185cb5abbc30d7257104c434fe0b07e5a195a6847506c074527aa599ec/Click-7.0-py2.py3-none-any.whl (81kB)\n",
      "Collecting Jinja2>=2.10 (from flask->seldon-core->-r requirements.txt (line 7))\n",
      "  Downloading https://files.pythonhosted.org/packages/1d/e7/fd8b501e7a6dfe492a433deb7b9d833d39ca74916fa8bc63dd1a4947a671/Jinja2-2.10.1-py2.py3-none-any.whl (124kB)\n",
      "Collecting Werkzeug>=0.14 (from flask->seldon-core->-r requirements.txt (line 7))\n",
      "  Downloading https://files.pythonhosted.org/packages/9f/57/92a497e38161ce40606c27a86759c6b92dd34fcdb33f64171ec559257c02/Werkzeug-0.15.4-py2.py3-none-any.whl (327kB)\n",
      "Collecting itsdangerous>=0.24 (from flask->seldon-core->-r requirements.txt (line 7))\n",
      "  Downloading https://files.pythonhosted.org/packages/76/ae/44b03b253d6fade317f32c24d100b3b35c2239807046a4c953c7b89fa49e/itsdangerous-1.1.0-py2.py3-none-any.whl\n",
      "Collecting pyasn1<0.5.0,>=0.4.1 (from pyasn1-modules>=0.2.1->google-auth>=1.2.0->google-cloud-storage->-r requirements.txt (line 1))\n",
      "  Downloading https://files.pythonhosted.org/packages/7b/7c/c9386b82a25115cccf1903441bba3cbadcfae7b678a20167347fa8ded34c/pyasn1-0.4.5-py2.py3-none-any.whl (73kB)\n",
      "Collecting h5py (from keras-applications>=1.0.6->tensorflow->seldon-core->-r requirements.txt (line 7))\n",
      "  Downloading https://files.pythonhosted.org/packages/30/99/d7d4fbf2d02bb30fb76179911a250074b55b852d34e98dd452a9f394ac06/h5py-2.9.0-cp36-cp36m-manylinux1_x86_64.whl (2.8MB)\n",
      "Collecting markdown>=2.6.8 (from tensorboard<1.14.0,>=1.13.0->tensorflow->seldon-core->-r requirements.txt (line 7))\n",
      "  Downloading https://files.pythonhosted.org/packages/f5/e4/d8c18f2555add57ff21bf25af36d827145896a07607486cc79a2aea641af/Markdown-3.1-py2.py3-none-any.whl (87kB)\n",
      "Collecting mock>=2.0.0 (from tensorflow-estimator<1.14.0rc0,>=1.13.0->tensorflow->seldon-core->-r requirements.txt (line 7))\n",
      "  Downloading https://files.pythonhosted.org/packages/05/d2/f94e68be6b17f46d2c353564da56e6fb89ef09faeeff3313a046cb810ca9/mock-3.0.5-py2.py3-none-any.whl\n",
      "Collecting MarkupSafe>=0.23 (from Jinja2>=2.10->flask->seldon-core->-r requirements.txt (line 7))\n",
      "  Downloading https://files.pythonhosted.org/packages/b2/5f/23e0023be6bb885d00ffbefad2942bc51a620328ee910f64abe5a8d18dd1/MarkupSafe-1.1.1-cp36-cp36m-manylinux1_x86_64.whl\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Installing collected packages: six, pyasn1, pyasn1-modules, cachetools, rsa, google-auth, chardet, urllib3, idna, certifi, requests, protobuf, googleapis-common-protos, pytz, google-api-core, google-cloud-core, google-resumable-media, google-cloud-storage, numpy, python-dateutil, pandas, joblib, scipy, xgboost, scikit-learn, sklearn, opentracing, click, MarkupSafe, Jinja2, Werkzeug, itsdangerous, flask, Flask-OpenTracing, flatbuffers, pyyaml, tornado, threadloop, thrift, jaeger-client, flask-cors, redis, astor, gast, absl-py, keras-preprocessing, grpcio, termcolor, h5py, keras-applications, markdown, tensorboard, mock, tensorflow-estimator, tensorflow, grpcio-opentracing, seldon-core\n",
      "  Running setup.py install for googleapis-common-protos: started\n",
      "    Running setup.py install for googleapis-common-protos: finished with status 'done'\n",
      "  Running setup.py install for sklearn: started\n",
      "    Running setup.py install for sklearn: finished with status 'done'\n",
      "  Running setup.py install for opentracing: started\n",
      "    Running setup.py install for opentracing: finished with status 'done'\n",
      "  Running setup.py install for pyyaml: started\n",
      "    Running setup.py install for pyyaml: finished with status 'done'\n",
      "  Running setup.py install for tornado: started\n"
     ]
    }
   ],
   "source": [
    "from fairing import TrainJob\n",
    "from fairing.backends import KubeflowGKEBackend\n",
    "train_job = TrainJob(HousingServe, BASE_IMAGE, input_files=['ames_dataset/train.csv', \"ames.py\", \n",
    "                                                            \"requirements.txt\"],\n",
    "                     docker_registry=DOCKER_REGISTRY, backend=KubeflowGKEBackend())\n",
    "\n",
    "if rerun_training:\n",
    "    train_job.submit()\n",
    "else:\n",
    "    # Print out cached output. This is solely for demo purposes since training can take a while\n",
    "    with open(\"train_output.txt\") as hf:        \n",
    "        print(hf.read())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "apiVersion: batch/v1\r\n",
      "kind: Job\r\n",
      "metadata:\r\n",
      "  creationTimestamp: \"2019-05-17T01:25:23Z\"\r\n",
      "  generateName: fairing-job-\r\n",
      "  labels:\r\n",
      "    fairing-deployer: job\r\n",
      "    fairing-id: a483e1be-7842-11e9-85d6-0a580a00012d\r\n",
      "  name: fairing-job-j5bh6\r\n",
      "  namespace: kubeflow\r\n",
      "  resourceVersion: \"13095243\"\r\n",
      "  selfLink: /apis/batch/v1/namespaces/kubeflow/jobs/fairing-job-j5bh6\r\n",
      "  uid: a48de4d8-7842-11e9-8964-42010a8e00ff\r\n",
      "spec:\r\n",
      "  backoffLimit: 6\r\n",
      "  completions: 1\r\n",
      "  parallelism: 1\r\n",
      "  selector:\r\n",
      "    matchLabels:\r\n",
      "      controller-uid: a48de4d8-7842-11e9-8964-42010a8e00ff\r\n",
      "  template:\r\n",
      "    metadata:\r\n",
      "      creationTimestamp: null\r\n",
      "      labels:\r\n",
      "        controller-uid: a48de4d8-7842-11e9-8964-42010a8e00ff\r\n",
      "        fairing-deployer: job\r\n",
      "        fairing-id: a483e1be-7842-11e9-85d6-0a580a00012d\r\n",
      "        job-name: fairing-job-j5bh6\r\n",
      "      name: fairing-deployer\r\n",
      "    spec:\r\n",
      "      containers:\r\n",
      "      - command:\r\n",
      "        - python\r\n",
      "        - /app/function_shim.py\r\n",
      "        - --serialized_fn_file\r\n",
      "        - /app/pickled_fn.p\r\n",
      "        env:\r\n",
      "        - name: FAIRING_RUNTIME\r\n",
      "          value: \"1\"\r\n",
      "        - name: GOOGLE_APPLICATION_CREDENTIALS\r\n",
      "          value: /etc/secrets/user-gcp-sa.json\r\n",
      "        image: gcr.io/code-search-demo/fairing-job/fairing-job:A3482C9E\r\n",
      "        imagePullPolicy: IfNotPresent\r\n",
      "        name: model\r\n",
      "        resources: {}\r\n",
      "        securityContext:\r\n",
      "          runAsUser: 0\r\n",
      "        terminationMessagePath: /dev/termination-log\r\n",
      "        terminationMessagePolicy: File\r\n",
      "        volumeMounts:\r\n",
      "        - mountPath: /etc/secrets\r\n",
      "          name: user-gcp-sa\r\n",
      "          readOnly: true\r\n",
      "        workingDir: /app\r\n",
      "      dnsPolicy: ClusterFirst\r\n",
      "      restartPolicy: Never\r\n",
      "      schedulerName: default-scheduler\r\n",
      "      securityContext: {}\r\n",
      "      terminationGracePeriodSeconds: 30\r\n",
      "      volumes:\r\n",
      "      - name: user-gcp-sa\r\n",
      "        secret:\r\n",
      "          defaultMode: 420\r\n",
      "          secretName: user-gcp-sa\r\n",
      "status:\r\n",
      "  completionTime: \"2019-05-17T01:26:06Z\"\r\n",
      "  conditions:\r\n",
      "  - lastProbeTime: \"2019-05-17T01:26:06Z\"\r\n",
      "    lastTransitionTime: \"2019-05-17T01:26:06Z\"\r\n",
      "    status: \"True\"\r\n",
      "    type: Complete\r\n",
      "  startTime: \"2019-05-17T01:25:23Z\"\r\n",
      "  succeeded: 1\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl get jobs -o yaml fairing-job-j5bh6"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploy the trained model to Kubeflow for predictions\n",
    "\n",
    "Import the `PredictionEndpoint` and `KubeflowGKEBackend` classes. Kubeflow Fairing packages the `HousingServe` class, the trained model, and the prediction endpoint's software prerequisites as a Docker image. Then Kubeflow Fairing deploys and runs the prediction endpoint on Kubeflow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:fairing.ml_tasks.tasks:Using preprocessor: <class 'fairing.preprocessors.function.FunctionPreProcessor'>\n",
      "WARNING:fairing.ml_tasks.tasks:Using docker registry: gcr.io/code-search-demo/fairing-job\n",
      "INFO:root:Using ClusterBuilder\n",
      "WARNING:fairing.ml_tasks.tasks:Using builder: <class 'fairing.builders.cluster.cluster.ClusterBuilder'>\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:fairing.ml_tasks.tasks:Using preprocessor: <class 'fairing.preprocessors.function.FunctionPreProcessor'>\n",
      "WARNING:fairing.ml_tasks.tasks:Using docker registry: gcr.io/code-search-demo/fairing-job\n",
      "INFO:root:Using ClusterBuilder\n",
      "WARNING:fairing.ml_tasks.tasks:Using builder: <class 'fairing.builders.cluster.cluster.ClusterBuilder'>\n",
      "INFO:root:Building the docker image.\n",
      "WARNING:root:/home/jovyan/git_jlewi-kubecon-demo/fairing/fairing/__init__.py already exists in Fairing context, skipping...\n",
      "INFO:root:Creating docker context: /tmp/fairing.context.tar.gz\n",
      "INFO:root:Adding files to context: {'trained_ames_model.dat', 'requirements.txt'}\n",
      "WARNING:root:/home/jovyan/git_jlewi-kubecon-demo/fairing/fairing/__init__.py already exists in Fairing context, skipping...\n",
      "INFO:root:Context: /tmp/fairing.context.tar.gz, Adding /home/jovyan/git_jlewi-kubecon-demo/fairing/fairing/__init__.py at /app/fairing/__init__.py\n",
      "INFO:root:Context: /tmp/fairing.context.tar.gz, Adding /home/jovyan/git_jlewi-kubecon-demo/fairing/fairing/runtime_config.py at /app/fairing/runtime_config.py\n",
      "INFO:root:Context: /tmp/fairing.context.tar.gz, Adding trained_ames_model.dat at /app/trained_ames_model.dat\n",
      "INFO:root:Context: /tmp/fairing.context.tar.gz, Adding requirements.txt at /app/requirements.txt\n",
      "INFO:root:Context: /tmp/fairing.context.tar.gz, Adding /home/jovyan/git_jlewi-kubecon-demo/fairing/fairing/functions/function_shim.py at /app/function_shim.py\n",
      "INFO:root:Context: /tmp/fairing.context.tar.gz, Adding /opt/conda/lib/python3.6/site-packages/cloudpickle/__init__.py at /app/cloudpickle/__init__.py\n",
      "INFO:root:Context: /tmp/fairing.context.tar.gz, Adding /opt/conda/lib/python3.6/site-packages/cloudpickle/cloudpickle.py at /app/cloudpickle/cloudpickle.py\n",
      "INFO:root:Context: /tmp/fairing.context.tar.gz, Adding /tmp/tmpppb7k99n at /app/pickled_fn.p\n",
      "INFO:root:Context: /tmp/fairing.context.tar.gz, Adding /tmp/tmpih2offwk at /app/HousingServe.py\n",
      "INFO:root:Context: /tmp/fairing.context.tar.gz, Adding /tmp/Dockerfile at Dockerfile\n",
      "WARNING:fairing.kubernetes.manager:Waiting for fairing-builder-knsrp to start...\n",
      "WARNING:fairing.kubernetes.manager:Waiting for fairing-builder-knsrp to start...\n",
      "INFO:fairing.kubernetes.manager:Pod started running True\n",
      "INFO[0000] Downloading base image python:3.6.7\n",
      "ERROR: logging before flag.Parse: E0517 01:29:25.475011       1 metadata.go:142] while reading 'google-dockercfg' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg\n",
      "ERROR: logging before flag.Parse: E0517 01:29:25.479962       1 metadata.go:159] while reading 'google-dockercfg-url' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg-url\n",
      "2019/05/17 01:29:25 No matching credentials were found, falling back on anonymous\n",
      "INFO[0000] Executing 0 build triggers\n",
      "INFO[0000] Unpacking rootfs as cmd RUN if [ -e requirements.txt ];then pip install --no-cache -r requirements.txt; fi requires it.\n",
      "INFO[0019] Taking snapshot of full filesystem...\n",
      "INFO[0028] Skipping paths under /dev, as it is a whitelisted directory\n",
      "INFO[0028] Skipping paths under /etc/secrets, as it is a whitelisted directory\n",
      "INFO[0028] Skipping paths under /kaniko, as it is a whitelisted directory\n",
      "INFO[0028] Skipping paths under /proc, as it is a whitelisted directory\n",
      "INFO[0028] Skipping paths under /sys, as it is a whitelisted directory\n",
      "INFO[0029] Skipping paths under /var/run, as it is a whitelisted directory\n",
      "INFO[0038] WORKDIR /app/\n",
      "INFO[0038] cmd: workdir\n",
      "INFO[0038] Changed working directory to /app/\n",
      "INFO[0038] Creating directory /app/\n",
      "INFO[0038] Taking snapshot of files...\n",
      "INFO[0038] ENV FAIRING_RUNTIME 1\n",
      "INFO[0038] Using files from context: [/kaniko/buildcontext/app/requirements.txt]\n",
      "INFO[0038] COPY /app//requirements.txt /app/\n",
      "INFO[0038] Taking snapshot of files...\n",
      "INFO[0038] RUN if [ -e requirements.txt ];then pip install --no-cache -r requirements.txt; fi\n",
      "INFO[0038] cmd: /bin/sh\n",
      "INFO[0038] args: [-c if [ -e requirements.txt ];then pip install --no-cache -r requirements.txt; fi]\n",
      "Collecting pandas (from -r requirements.txt (line 1))\n",
      "  Downloading https://files.pythonhosted.org/packages/19/74/e50234bc82c553fecdbd566d8650801e3fe2d6d8c8d940638e3d8a7c5522/pandas-0.24.2-cp36-cp36m-manylinux1_x86_64.whl (10.1MB)\n",
      "Collecting joblib (from -r requirements.txt (line 2))\n",
      "  Downloading https://files.pythonhosted.org/packages/cd/c1/50a758e8247561e58cb87305b1e90b171b8c767b15b12a1734001f41d356/joblib-0.13.2-py2.py3-none-any.whl (278kB)\n",
      "Collecting numpy (from -r requirements.txt (line 3))\n",
      "  Downloading https://files.pythonhosted.org/packages/c1/e2/4db8df8f6cddc98e7d7c537245ef2f4e41a1ed17bf0c3177ab3cc6beac7f/numpy-1.16.3-cp36-cp36m-manylinux1_x86_64.whl (17.3MB)\n",
      "Collecting xgboost (from -r requirements.txt (line 4))\n",
      "  Downloading https://files.pythonhosted.org/packages/6a/49/7e10686647f741bd9c8918b0decdb94135b542fe372ca1100739b8529503/xgboost-0.82-py2.py3-none-manylinux1_x86_64.whl (114.0MB)\n",
      "Collecting sklearn (from -r requirements.txt (line 5))\n",
      "  Downloading https://files.pythonhosted.org/packages/1e/7a/dbb3be0ce9bd5c8b7e3d87328e79063f8b263b2b1bfa4774cb1147bfcd3f/sklearn-0.0.tar.gz\n",
      "Collecting seldon-core (from -r requirements.txt (line 6))\n",
      "  Downloading https://files.pythonhosted.org/packages/31/44/51cac9c7df386fbd740ae9c24b2d28380974d137c5c1532eae0534c9bb67/seldon_core-0.2.7.1-py3-none-any.whl\n",
      "Collecting python-dateutil>=2.5.0 (from pandas->-r requirements.txt (line 1))\n",
      "  Downloading https://files.pythonhosted.org/packages/41/17/c62faccbfbd163c7f57f3844689e3a78bae1f403648a6afb1d0866d87fbb/python_dateutil-2.8.0-py2.py3-none-any.whl (226kB)\n",
      "Collecting pytz>=2011k (from pandas->-r requirements.txt (line 1))\n",
      "  Downloading https://files.pythonhosted.org/packages/3d/73/fe30c2daaaa0713420d0382b16fbb761409f532c56bdcc514bf7b6262bb6/pytz-2019.1-py2.py3-none-any.whl (510kB)\n",
      "Collecting scipy (from xgboost->-r requirements.txt (line 4))\n",
      "  Downloading https://files.pythonhosted.org/packages/7f/5f/c48860704092933bf1c4c1574a8de1ffd16bf4fde8bab190d747598844b2/scipy-1.2.1-cp36-cp36m-manylinux1_x86_64.whl (24.8MB)\n",
      "Collecting scikit-learn (from sklearn->-r requirements.txt (line 5))\n",
      "  Downloading https://files.pythonhosted.org/packages/90/c7/401c231c445fb6fad135e92197da9c3e77983de169ff1887cc18af94498d/scikit_learn-0.21.1-cp36-cp36m-manylinux1_x86_64.whl (6.7MB)\n",
      "Collecting tensorflow (from seldon-core->-r requirements.txt (line 6))\n",
      "  Downloading https://files.pythonhosted.org/packages/77/63/a9fa76de8dffe7455304c4ed635be4aa9c0bacef6e0633d87d5f54530c5c/tensorflow-1.13.1-cp36-cp36m-manylinux1_x86_64.whl (92.5MB)\n",
      "Collecting flask (from seldon-core->-r requirements.txt (line 6))\n",
      "  Downloading https://files.pythonhosted.org/packages/7f/e7/08578774ed4536d3242b14dacb4696386634607af824ea997202cd0edb4b/Flask-1.0.2-py2.py3-none-any.whl (91kB)\n",
      "Collecting flask-cors (from seldon-core->-r requirements.txt (line 6))\n",
      "  Downloading https://files.pythonhosted.org/packages/65/cb/683f71ff8daa3aea0a5cbb276074de39f9ab66d3fbb8ad5efb5bb83e90d2/Flask_Cors-3.0.7-py2.py3-none-any.whl\n",
      "Collecting requests (from seldon-core->-r requirements.txt (line 6))\n",
      "  Downloading https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl (57kB)\n",
      "Collecting protobuf (from seldon-core->-r requirements.txt (line 6))\n",
      "  Downloading https://files.pythonhosted.org/packages/5a/aa/a858df367b464f5e9452e1c538aa47754d467023850c00b000287750fa77/protobuf-3.7.1-cp36-cp36m-manylinux1_x86_64.whl (1.2MB)\n",
      "Collecting jaeger-client==3.13.0 (from seldon-core->-r requirements.txt (line 6))\n",
      "  Downloading https://files.pythonhosted.org/packages/c8/a2/e9bd04cd660cbdffe0598173be068be23099fbd68e7a4a89b74440509130/jaeger-client-3.13.0.tar.gz (77kB)\n",
      "Collecting pyyaml (from seldon-core->-r requirements.txt (line 6))\n",
      "  Downloading https://files.pythonhosted.org/packages/9f/2c/9417b5c774792634834e730932745bc09a7d36754ca00acf1ccd1ac2594d/PyYAML-5.1.tar.gz (274kB)\n",
      "Collecting redis (from seldon-core->-r requirements.txt (line 6))\n",
      "  Downloading https://files.pythonhosted.org/packages/ac/a7/cff10cc5f1180834a3ed564d148fb4329c989cbb1f2e196fc9a10fa07072/redis-3.2.1-py2.py3-none-any.whl (65kB)\n",
      "Collecting grpcio (from seldon-core->-r requirements.txt (line 6))\n",
      "  Downloading https://files.pythonhosted.org/packages/0a/9d/8bd5d0e516b196f59f1c4439b424b8d4fa62d492a4b531aae322d2d82a7b/grpcio-1.20.1-cp36-cp36m-manylinux1_x86_64.whl (2.1MB)\n",
      "Collecting opentracing<2,>=1.2.2 (from seldon-core->-r requirements.txt (line 6))\n",
      "  Downloading https://files.pythonhosted.org/packages/06/c2/90b35a1abdc639a5c6000d8202c70217d60e80f5b328870efb73fda71115/opentracing-1.3.0.tar.gz\n",
      "Collecting flatbuffers (from seldon-core->-r requirements.txt (line 6))\n",
      "  Downloading https://files.pythonhosted.org/packages/c9/84/adf5837f96c39990bc55afdfddf460b38b4562f50341359afa32e4a98de7/flatbuffers-1.11-py2.py3-none-any.whl\n",
      "Collecting Flask-OpenTracing==0.2.0 (from seldon-core->-r requirements.txt (line 6))\n",
      "  Downloading https://files.pythonhosted.org/packages/1d/c4/0546b854a3f42af9ef959df9bd1108903698e175e7a07c057cdfaeeef718/Flask_OpenTracing-0.2.0-py2.py3-none-any.whl\n",
      "Collecting grpcio-opentracing (from seldon-core->-r requirements.txt (line 6))\n",
      "  Downloading https://files.pythonhosted.org/packages/db/82/2fcad380697c3dab25de76ee590bcab3eb9bbfb4add916044d7e83ec2b10/grpcio_opentracing-1.1.4-py3-none-any.whl\n",
      "Collecting six>=1.5 (from python-dateutil>=2.5.0->pandas->-r requirements.txt (line 1))\n",
      "  Downloading https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl\n",
      "Collecting termcolor>=1.1.0 (from tensorflow->seldon-core->-r requirements.txt (line 6))\n",
      "  Downloading https://files.pythonhosted.org/packages/8a/48/a76be51647d0eb9f10e2a4511bf3ffb8cc1e6b14e9e4fab46173aa79f981/termcolor-1.1.0.tar.gz\n",
      "Collecting tensorflow-estimator<1.14.0rc0,>=1.13.0 (from tensorflow->seldon-core->-r requirements.txt (line 6))\n",
      "  Downloading https://files.pythonhosted.org/packages/bb/48/13f49fc3fa0fdf916aa1419013bb8f2ad09674c275b4046d5ee669a46873/tensorflow_estimator-1.13.0-py2.py3-none-any.whl (367kB)\n",
      "Collecting gast>=0.2.0 (from tensorflow->seldon-core->-r requirements.txt (line 6))\n",
      "  Downloading https://files.pythonhosted.org/packages/4e/35/11749bf99b2d4e3cceb4d55ca22590b0d7c2c62b9de38ac4a4a7f4687421/gast-0.2.2.tar.gz\n",
      "Collecting astor>=0.6.0 (from tensorflow->seldon-core->-r requirements.txt (line 6))\n",
      "  Downloading https://files.pythonhosted.org/packages/35/6b/11530768cac581a12952a2aad00e1526b89d242d0b9f59534ef6e6a1752f/astor-0.7.1-py2.py3-none-any.whl\n",
      "Collecting keras-applications>=1.0.6 (from tensorflow->seldon-core->-r requirements.txt (line 6))\n",
      "  Downloading https://files.pythonhosted.org/packages/90/85/64c82949765cfb246bbdaf5aca2d55f400f792655927a017710a78445def/Keras_Applications-1.0.7-py2.py3-none-any.whl (51kB)\n",
      "Collecting absl-py>=0.1.6 (from tensorflow->seldon-core->-r requirements.txt (line 6))\n",
      "  Downloading https://files.pythonhosted.org/packages/da/3f/9b0355080b81b15ba6a9ffcf1f5ea39e307a2778b2f2dc8694724e8abd5b/absl-py-0.7.1.tar.gz (99kB)\n",
      "Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.6/site-packages (from tensorflow->seldon-core->-r requirements.txt (line 6)) (0.32.2)\n",
      "Collecting keras-preprocessing>=1.0.5 (from tensorflow->seldon-core->-r requirements.txt (line 6))\n",
      "  Downloading https://files.pythonhosted.org/packages/c0/bf/0315ef6a9fd3fc2346e85b0ff1f5f83ca17073f2c31ac719ab2e4da0d4a3/Keras_Preprocessing-1.0.9-py2.py3-none-any.whl (59kB)\n",
      "Collecting tensorboard<1.14.0,>=1.13.0 (from tensorflow->seldon-core->-r requirements.txt (line 6))\n",
      "  Downloading https://files.pythonhosted.org/packages/0f/39/bdd75b08a6fba41f098b6cb091b9e8c7a80e1b4d679a581a0ccd17b10373/tensorboard-1.13.1-py3-none-any.whl (3.2MB)\n",
      "Collecting Jinja2>=2.10 (from flask->seldon-core->-r requirements.txt (line 6))\n",
      "  Downloading https://files.pythonhosted.org/packages/1d/e7/fd8b501e7a6dfe492a433deb7b9d833d39ca74916fa8bc63dd1a4947a671/Jinja2-2.10.1-py2.py3-none-any.whl (124kB)\n",
      "Collecting Werkzeug>=0.14 (from flask->seldon-core->-r requirements.txt (line 6))\n",
      "  Downloading https://files.pythonhosted.org/packages/9f/57/92a497e38161ce40606c27a86759c6b92dd34fcdb33f64171ec559257c02/Werkzeug-0.15.4-py2.py3-none-any.whl (327kB)\n",
      "Collecting click>=5.1 (from flask->seldon-core->-r requirements.txt (line 6))\n",
      "  Downloading https://files.pythonhosted.org/packages/fa/37/45185cb5abbc30d7257104c434fe0b07e5a195a6847506c074527aa599ec/Click-7.0-py2.py3-none-any.whl (81kB)\n",
      "Collecting itsdangerous>=0.24 (from flask->seldon-core->-r requirements.txt (line 6))\n",
      "  Downloading https://files.pythonhosted.org/packages/76/ae/44b03b253d6fade317f32c24d100b3b35c2239807046a4c953c7b89fa49e/itsdangerous-1.1.0-py2.py3-none-any.whl\n",
      "Collecting certifi>=2017.4.17 (from requests->seldon-core->-r requirements.txt (line 6))\n",
      "  Downloading https://files.pythonhosted.org/packages/60/75/f692a584e85b7eaba0e03827b3d51f45f571c2e793dd731e598828d380aa/certifi-2019.3.9-py2.py3-none-any.whl (158kB)\n",
      "Collecting idna<2.9,>=2.5 (from requests->seldon-core->-r requirements.txt (line 6))\n",
      "  Downloading https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl (58kB)\n",
      "Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests->seldon-core->-r requirements.txt (line 6))\n",
      "  Downloading https://files.pythonhosted.org/packages/39/ec/d93dfc69617a028915df914339ef66936ea976ef24fa62940fd86ba0326e/urllib3-1.25.2-py2.py3-none-any.whl (150kB)\n",
      "Collecting chardet<3.1.0,>=3.0.2 (from requests->seldon-core->-r requirements.txt (line 6))\n",
      "  Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)\n",
      "Requirement already satisfied: setuptools in /usr/local/lib/python3.6/site-packages (from protobuf->seldon-core->-r requirements.txt (line 6)) (40.6.2)\n",
      "Collecting threadloop<2,>=1 (from jaeger-client==3.13.0->seldon-core->-r requirements.txt (line 6))\n",
      "  Downloading https://files.pythonhosted.org/packages/d3/1d/8398c1645b97dc008d3c658e04beda01ede3d90943d40c8d56863cf891bd/threadloop-1.0.2.tar.gz\n",
      "Collecting thrift (from jaeger-client==3.13.0->seldon-core->-r requirements.txt (line 6))\n",
      "  Downloading https://files.pythonhosted.org/packages/c6/b4/510617906f8e0c5660e7d96fbc5585113f83ad547a3989b80297ac72a74c/thrift-0.11.0.tar.gz (52kB)\n",
      "Collecting tornado<5,>=4.3 (from jaeger-client==3.13.0->seldon-core->-r requirements.txt (line 6))\n",
      "  Downloading https://files.pythonhosted.org/packages/e3/7b/e29ab3d51c8df66922fea216e2bddfcb6430fb29620e5165b16a216e0d3c/tornado-4.5.3.tar.gz (484kB)\n",
      "Collecting mock>=2.0.0 (from tensorflow-estimator<1.14.0rc0,>=1.13.0->tensorflow->seldon-core->-r requirements.txt (line 6))\n",
      "  Downloading https://files.pythonhosted.org/packages/05/d2/f94e68be6b17f46d2c353564da56e6fb89ef09faeeff3313a046cb810ca9/mock-3.0.5-py2.py3-none-any.whl\n",
      "Collecting h5py (from keras-applications>=1.0.6->tensorflow->seldon-core->-r requirements.txt (line 6))\n",
      "  Downloading https://files.pythonhosted.org/packages/30/99/d7d4fbf2d02bb30fb76179911a250074b55b852d34e98dd452a9f394ac06/h5py-2.9.0-cp36-cp36m-manylinux1_x86_64.whl (2.8MB)\n",
      "Collecting markdown>=2.6.8 (from tensorboard<1.14.0,>=1.13.0->tensorflow->seldon-core->-r requirements.txt (line 6))\n",
      "  Downloading https://files.pythonhosted.org/packages/f5/e4/d8c18f2555add57ff21bf25af36d827145896a07607486cc79a2aea641af/Markdown-3.1-py2.py3-none-any.whl (87kB)\n",
      "Collecting MarkupSafe>=0.23 (from Jinja2>=2.10->flask->seldon-core->-r requirements.txt (line 6))\n",
      "  Downloading https://files.pythonhosted.org/packages/b2/5f/23e0023be6bb885d00ffbefad2942bc51a620328ee910f64abe5a8d18dd1/MarkupSafe-1.1.1-cp36-cp36m-manylinux1_x86_64.whl\n",
      "Installing collected packages: six, python-dateutil, numpy, pytz, pandas, joblib, scipy, xgboost, scikit-learn, sklearn, termcolor, mock, absl-py, tensorflow-estimator, protobuf, gast, astor, h5py, keras-applications, grpcio, keras-preprocessing, Werkzeug, markdown, tensorboard, tensorflow, MarkupSafe, Jinja2, click, itsdangerous, flask, flask-cors, certifi, idna, urllib3, chardet, requests, tornado, threadloop, thrift, opentracing, jaeger-client, pyyaml, redis, flatbuffers, Flask-OpenTracing, grpcio-opentracing, seldon-core\n",
      "  Running setup.py install for sklearn: started\n",
      "    Running setup.py install for sklearn: finished with status 'done'\n",
      "  Running setup.py install for termcolor: started\n",
      "    Running setup.py install for termcolor: finished with status 'done'\n",
      "  Running setup.py install for absl-py: started\n",
      "    Running setup.py install for absl-py: finished with status 'done'\n",
      "  Running setup.py install for gast: started\n",
      "    Running setup.py install for gast: finished with status 'done'\n",
      "  Running setup.py install for tornado: started\n",
      "    Running setup.py install for tornado: finished with status 'done'\n",
      "  Running setup.py install for threadloop: started\n",
      "    Running setup.py install for threadloop: finished with status 'done'\n",
      "  Running setup.py install for thrift: started\n",
      "    Running setup.py install for thrift: finished with status 'done'\n",
      "  Running setup.py install for opentracing: started\n",
      "    Running setup.py install for opentracing: finished with status 'done'\n",
      "  Running setup.py install for jaeger-client: started\n",
      "    Running setup.py install for jaeger-client: finished with status 'done'\n",
      "  Running setup.py install for pyyaml: started\n",
      "    Running setup.py install for pyyaml: finished with status 'done'\n",
      "Successfully installed Flask-OpenTracing-0.2.0 Jinja2-2.10.1 MarkupSafe-1.1.1 Werkzeug-0.15.4 absl-py-0.7.1 astor-0.7.1 certifi-2019.3.9 chardet-3.0.4 click-7.0 flask-1.0.2 flask-cors-3.0.7 flatbuffers-1.11 gast-0.2.2 grpcio-1.20.1 grpcio-opentracing-1.1.4 h5py-2.9.0 idna-2.8 itsdangerous-1.1.0 jaeger-client-3.13.0 joblib-0.13.2 keras-applications-1.0.7 keras-preprocessing-1.0.9 markdown-3.1 mock-3.0.5 numpy-1.16.3 opentracing-1.3.0 pandas-0.24.2 protobuf-3.7.1 python-dateutil-2.8.0 pytz-2019.1 pyyaml-5.1 redis-3.2.1 requests-2.22.0 scikit-learn-0.21.1 scipy-1.2.1 seldon-core-0.2.7.1 six-1.12.0 sklearn-0.0 tensorboard-1.13.1 tensorflow-1.13.1 tensorflow-estimator-1.13.0 termcolor-1.1.0 threadloop-1.0.2 thrift-0.11.0 tornado-4.5.3 urllib3-1.25.2 xgboost-0.82\n",
      "You are using pip version 18.1, however version 19.1.1 is available.\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\n",
      "INFO[0102] Taking snapshot of full filesystem...\n",
      "INFO[0103] Skipping paths under /dev, as it is a whitelisted directory\n",
      "INFO[0103] Skipping paths under /etc/secrets, as it is a whitelisted directory\n",
      "INFO[0103] Skipping paths under /kaniko, as it is a whitelisted directory\n",
      "INFO[0103] Skipping paths under /proc, as it is a whitelisted directory\n",
      "INFO[0103] Skipping paths under /sys, as it is a whitelisted directory\n",
      "INFO[0104] Skipping paths under /var/run, as it is a whitelisted directory\n",
      "INFO[0137] Using files from context: [/kaniko/buildcontext/app]\n",
      "INFO[0137] COPY /app/ /app/\n",
      "INFO[0138] Taking snapshot of files...\n",
      "2019/05/17 01:31:43 existing blob: sha256:687ed2fb2a0d7da5503478759fd00c23970b65d02b317119b3fb9025038a2594\n",
      "2019/05/17 01:31:43 existing blob: sha256:0c1db95989906f161007d8ef2a6ef6e0ec64bc15bf2c993fd002edbdfc7aa7df\n",
      "2019/05/17 01:31:43 existing blob: sha256:a8c5303780550b746a4781e5e4cd893121d8019e971414a2a1273d54486b4eb9\n",
      "2019/05/17 01:31:43 existing blob: sha256:5d71636fb824265e30ff34bf20737c9cdc4f5af28b6bce86f08215c55b89bfab\n",
      "2019/05/17 01:31:43 existing blob: sha256:d6341e30912f12f56e18564a3b582853f65376766f5f9d641a68a724ed6db88f\n",
      "2019/05/17 01:31:43 existing blob: sha256:620aea26e85367b08cdf1f6768491fb44df6a2a71f7d663f835b1692e849c3ee\n",
      "2019/05/17 01:31:43 existing blob: sha256:087a57faf9491b1b82a83e26bc8cc90c90c30e4a4d858b57ddd5b4c2c90095f6\n",
      "2019/05/17 01:31:43 existing blob: sha256:2eeb5ce9b9240a928b0a799f9f2601027e2c6b7525394ae5c371f124058489d7\n",
      "2019/05/17 01:31:43 existing blob: sha256:54f7e8ac135a5f502a6ee9537ef3d64b1cd2fa570dc0a40b4d3b6f7ac81e7486\n",
      "2019/05/17 01:31:44 pushed blob sha256:26a37378d28f500d839feb76614c96b9d1ea7912e31a3c8d6c4312531af5aaca\n",
      "2019/05/17 01:31:45 pushed blob sha256:ef98a909bfb34df036edead47cf25d38dc82acb86221679650d37740e15f7ca9\n",
      "2019/05/17 01:31:45 pushed blob sha256:18f6c6750eeaf64863a663cfddb3e90e380cd5b21b57039ad166fc081498ae95\n",
      "2019/05/17 01:31:45 pushed blob sha256:75dc8788653ea0a5fdc18c7de39953d516a15ce1dc720e4d3a3b82492d07edeb\n",
      "2019/05/17 01:32:27 pushed blob sha256:bb7bf3474200cabb676bb4df5e0f8a20c4bbdc46e3821f7c8a9f090e73c208d9\n",
      "2019/05/17 01:32:28 gcr.io/code-search-demo/fairing-job/fairing-job:21A35F1E: digest: sha256:8aca44766b98e383a04bb7af9d2a274455a4c2758c50302717ce965b4727edcd size: 2390\n",
      "INFO:root:Deploying the endpoint.\n",
      "WARNING:fairing.deployers.serving.serving:Deployment fairing-deployer-gqpq8 launched.\n",
      "WARNING:fairing.deployers.serving.serving:In cluster Endpoint http://fairing-service-p7zjs.kubeflow.svc.cluster.local launched.\n",
      "WARNING:fairing.ml_tasks.tasks:Prediction endpoint: http://fairing-service-p7zjs.kubeflow.svc.cluster.local\n"
     ]
    }
   ],
   "source": [
    "rerun_deploy = False\n",
    "from fairing import PredictionEndpoint\n",
    "from fairing.backends import KubeflowGKEBackend\n",
    "endpoint = PredictionEndpoint(HousingServe, BASE_IMAGE, input_files=['trained_ames_model.dat', \"requirements.txt\"],\n",
    "                              docker_registry=DOCKER_REGISTRY, backend=KubeflowGKEBackend())\n",
    "\n",
    "if rerun_deploy:\n",
    "    endpoint.create()\n",
    "else:\n",
    "    endpoint.url = \"http://fairing-service-p7zjs.kubeflow.svc.cluster.local:5000/predict\"\n",
    "    # Print out cached output. This is solely for demo purposes since training can take a while\n",
    "    with open(\"deploy_output.txt\") as hf:        \n",
    "        print(hf.read())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "apiVersion: extensions/v1beta1\r\n",
      "kind: Deployment\r\n",
      "metadata:\r\n",
      "  annotations:\r\n",
      "    deployment.kubernetes.io/revision: \"1\"\r\n",
      "  creationTimestamp: \"2019-05-17T01:32:33Z\"\r\n",
      "  generateName: fairing-deployer-\r\n",
      "  generation: 1\r\n",
      "  labels:\r\n",
      "    fairing-deployer: serving\r\n",
      "    fairing-id: a4bd1e10-7843-11e9-85d6-0a580a00012d\r\n",
      "  name: fairing-deployer-gqpq8\r\n",
      "  namespace: kubeflow\r\n",
      "  resourceVersion: \"13097036\"\r\n",
      "  selfLink: /apis/extensions/v1beta1/namespaces/kubeflow/deployments/fairing-deployer-gqpq8\r\n",
      "  uid: a4be5a12-7843-11e9-8964-42010a8e00ff\r\n",
      "spec:\r\n",
      "  progressDeadlineSeconds: 600\r\n",
      "  replicas: 1\r\n",
      "  revisionHistoryLimit: 10\r\n",
      "  selector:\r\n",
      "    matchLabels:\r\n",
      "      fairing-deployer: serving\r\n",
      "      fairing-id: a4bd1e10-7843-11e9-85d6-0a580a00012d\r\n",
      "  strategy:\r\n",
      "    rollingUpdate:\r\n",
      "      maxSurge: 25%\r\n",
      "      maxUnavailable: 25%\r\n",
      "    type: RollingUpdate\r\n",
      "  template:\r\n",
      "    metadata:\r\n",
      "      creationTimestamp: null\r\n",
      "      labels:\r\n",
      "        fairing-deployer: serving\r\n",
      "        fairing-id: a4bd1e10-7843-11e9-85d6-0a580a00012d\r\n",
      "      name: fairing-deployer\r\n",
      "    spec:\r\n",
      "      containers:\r\n",
      "      - command:\r\n",
      "        - seldon-core-microservice\r\n",
      "        - HousingServe\r\n",
      "        - REST\r\n",
      "        - --service-type=MODEL\r\n",
      "        - --persistence=0\r\n",
      "        env:\r\n",
      "        - name: FAIRING_RUNTIME\r\n",
      "          value: \"1\"\r\n",
      "        image: gcr.io/code-search-demo/fairing-job/fairing-job:21A35F1E\r\n",
      "        imagePullPolicy: IfNotPresent\r\n",
      "        name: model\r\n",
      "        resources: {}\r\n",
      "        securityContext:\r\n",
      "          runAsUser: 0\r\n",
      "        terminationMessagePath: /dev/termination-log\r\n",
      "        terminationMessagePolicy: File\r\n",
      "        workingDir: /app\r\n",
      "      dnsPolicy: ClusterFirst\r\n",
      "      restartPolicy: Always\r\n",
      "      schedulerName: default-scheduler\r\n",
      "      securityContext: {}\r\n",
      "      terminationGracePeriodSeconds: 30\r\n",
      "status:\r\n",
      "  availableReplicas: 1\r\n",
      "  conditions:\r\n",
      "  - lastTransitionTime: \"2019-05-17T01:33:15Z\"\r\n",
      "    lastUpdateTime: \"2019-05-17T01:33:15Z\"\r\n",
      "    message: Deployment has minimum availability.\r\n",
      "    reason: MinimumReplicasAvailable\r\n",
      "    status: \"True\"\r\n",
      "    type: Available\r\n",
      "  - lastTransitionTime: \"2019-05-17T01:32:33Z\"\r\n",
      "    lastUpdateTime: \"2019-05-17T01:33:15Z\"\r\n",
      "    message: ReplicaSet \"fairing-deployer-gqpq8-645fcc87b7\" has successfully progressed.\r\n",
      "    reason: NewReplicaSetAvailable\r\n",
      "    status: \"True\"\r\n",
      "    type: Progressing\r\n",
      "  observedGeneration: 1\r\n",
      "  readyReplicas: 1\r\n",
      "  replicas: 1\r\n",
      "  updatedReplicas: 1\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl get deploy -o yaml fairing-deployer-gqpq8"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Call the prediction endpoint\n",
    "\n",
    "Create a test dataset, then call the endpoint on Kubeflow for predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "(train_X, train_y), (test_X, test_y) = read_input(\"ames_dataset/train.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'{\"data\":{\"names\":[\"t:0\",\"t:1\"],\"tensor\":{\"shape\":[1,2],\"values\":[165164.875,165164.875]}},\"meta\":{}}\\n'\n"
     ]
    }
   ],
   "source": [
    "import pprint\n",
    "test_X\n",
    "result = endpoint.predict_nparray(test_X)\n",
    "pprint.pprint(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Clean up the prediction endpoint\n",
    "\n",
    "Delete the prediction endpoint created by this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# endpoint.delete()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}