demo/2-distributed-mnist.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 入门云原生AI - 2. 运行分布式mnist\n",
    "接下来我们介绍如何通过arena提交，运维，管理一个分布式训练任务。通过arena，我们管理分布式训练任务，可以拥有像管理单机应用一样方便，快捷的体验。\n",
    "在这个示例中，我们将演示：\n",
    "\n",
    "* 下载并准备数据\n",
    "* 利用Arena提交分布式的训练任务,并且查看训练任务状态和日志\n",
    "* 通过TensorBoard查看训练任务\n",
    "\n",
    "> 前提：请先完成文档中的[共享存储配置](../docs/setup/SETUP_NAS.md)，当前${HOME}就是其中`training-data`的数据卷对应目录。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1.下载TensorFlow样例源代码到${HOME}/models目录"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cloning into '/root/models/tensorflow-sample-code'...\n",
      "remote: Enumerating objects: 242, done.\u001b[K\n",
      "remote: Counting objects: 100% (242/242), done.\u001b[K\n",
      "remote: Compressing objects: 100% (112/112), done.\u001b[K\n",
      "remote: Total 242 (delta 93), reused 242 (delta 93)\u001b[K\n",
      "Receiving objects: 100% (242/242), 11.25 MiB | 0 bytes/s, done.\n",
      "Resolving deltas: 100% (93/93), done.\n",
      "Checking connectivity... done.\n"
     ]
    }
   ],
   "source": [
    "! if [ ! -d \"${HOME}/models/tensorflow-sample-code\" ] ; then \\\n",
    "  git clone \"https://code.aliyun.com/xiaozhou/tensorflow-sample-code.git\" \"${HOME}/models/tensorflow-sample-code\"; \\\n",
    "fi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2.下载mnist数据到${HOME}/dataset/mnist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "100 1610k    0 1610k    0     0  2453k      0 --:--:-- --:--:-- --:--:-- 2450k\n",
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "100  4542    0  4542    0     0  12110      0 --:--:-- --:--:-- --:--:-- 12144\n",
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "100 9680k    0 9680k    0     0  12.4M      0 --:--:-- --:--:-- --:--:-- 12.4M\n",
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "100 28881    0 28881    0     0  72028      0 --:--:-- --:--:-- --:--:-- 72022\n"
     ]
    }
   ],
   "source": [
    "! mkdir -p ${HOME}/dataset/mnist && \\\n",
    "  cd ${HOME}/dataset/mnist && \\\n",
    "  curl -O https://code.aliyun.com/xiaozhou/tensorflow-sample-code/raw/master/data/t10k-images-idx3-ubyte.gz && \\\n",
    "  curl -O https://code.aliyun.com/xiaozhou/tensorflow-sample-code/raw/master/data/t10k-labels-idx1-ubyte.gz && \\\n",
    "  curl -O https://code.aliyun.com/xiaozhou/tensorflow-sample-code/raw/master/data/train-images-idx3-ubyte.gz && \\\n",
    "  curl -O https://code.aliyun.com/xiaozhou/tensorflow-sample-code/raw/master/data/train-labels-idx1-ubyte.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3.创建训练结果${HOME}/output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "! mkdir -p ${HOME}/output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4.查看目录结构, 其中`dataset`是数据目录，`models`是模型代码目录，`output`是训练结果目录。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/root\r\n",
      "|-- dataset\r\n",
      "|   `-- mnist\r\n",
      "|       |-- t10k-images-idx3-ubyte.gz\r\n",
      "|       |-- t10k-labels-idx1-ubyte.gz\r\n",
      "|       |-- train-images-idx3-ubyte.gz\r\n",
      "|       `-- train-labels-idx1-ubyte.gz\r\n",
      "|-- models\r\n",
      "|   `-- tensorflow-sample-code\r\n",
      "|       |-- README.md\r\n",
      "|       |-- data\r\n",
      "|       |-- mnist-tf\r\n",
      "|       |-- models\r\n",
      "|       |-- mpijob\r\n",
      "|       `-- tfjob\r\n",
      "`-- output\r\n",
      "\r\n",
      "10 directories, 5 files\r\n"
     ]
    }
   ],
   "source": [
    "! tree -I ai-starter -L 3 ${HOME}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "5.检查可用GPU资源"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME                                   IPADDRESS      ROLE    GPU(Total)  GPU(Allocated)\r\n",
      "cn-zhangjiakou.i-8vb2knpxzlk449e7lugx  192.168.0.209  <none>  1           0\r\n",
      "cn-zhangjiakou.i-8vb2knpxzlk449e7lugy  192.168.0.210  <none>  1           0\r\n",
      "cn-zhangjiakou.i-8vb2knpxzlk449e7lugz  192.168.0.208  <none>  1           0\r\n",
      "cn-zhangjiakou.i-8vb7yuo831zjzijo9sdw  192.168.0.205  master  0           0\r\n",
      "cn-zhangjiakou.i-8vbezxqzueo7662i0dbq  192.168.0.204  master  0           0\r\n",
      "cn-zhangjiakou.i-8vbezxqzueo7681j4fav  192.168.0.206  master  0           0\r\n",
      "-----------------------------------------------------------------------------------------\r\n",
      "Allocated/Total GPUs In Cluster:\r\n",
      "0/3 (0%)  \r\n"
     ]
    }
   ],
   "source": [
    "! arena top node"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "6.通过Arena提交训练任务, 这里`training-data`在配置[共享存储时](../docs/setup/SETUP_NAS.md)创建.   \n",
    "`--data=training-data:/training`将其映射到训练任务的`/training`目录。而`/training`目录下的子目录`/training/models/tensorflow-sample-code`就是步骤1拷贝源代码的位置，`/training`目录下的子目录`/training/dataset/mnist`就是步骤2下载数据的位置, `/training`目录下的子目录`/training/output`就是步骤3创建的训练结果输出的位置。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: JOB_NAME=tf-distributed-mnist\n",
      "configmap/tf-distributed-mnist-tfjob created\n",
      "configmap/tf-distributed-mnist-tfjob labeled\n",
      "service/tf-distributed-mnist-tensorboard created\n",
      "deployment.extensions/tf-distributed-mnist-tensorboard created\n",
      "tfjob.kubeflow.org/tf-distributed-mnist created\n",
      "\u001b[36mINFO\u001b[0m[0004] The Job tf-distributed-mnist has been submitted successfully \n",
      "\u001b[36mINFO\u001b[0m[0004] You can run `arena get tf-distributed-mnist --type tfjob` to check the job status \n"
     ]
    }
   ],
   "source": [
    "# Set the Job Name\n",
    "%env JOB_NAME=tf-distributed-mnist\n",
    "# Submit a training job \n",
    "# using code and data from Data Volume\n",
    "!arena submit tf \\\n",
    "             --name=$JOB_NAME \\\n",
    "             --ps=1 \\\n",
    "             --workers=2 \\\n",
    "             --gpus=1 \\\n",
    "             --data=training-data:/training \\\n",
    "             --tensorboard \\\n",
    "             --psImage=tensorflow/tensorflow:1.5.0 \\\n",
    "             --image=tensorflow/tensorflow:1.5.0-gpu-py3 \\\n",
    "             --logdir=/training/output/distributed-mnist \\\n",
    "             \"python /training/models/tensorflow-sample-code/tfjob/docker/v1alpha2/distributed-mnist/main.py --max_steps 10000 --data_dir /training/dataset/mnist --log_dir /training/output/distributed-mnist\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> `Arena`命令的`--logdir`指定`tensorboard`从训练任务的指定目录读取event\n",
    "> 完整参数可以参考[命令行文档](https://github.com/kubeflow/arena/blob/master/docs/cli/arena_submit_tfjob.md)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "7.检查模型训练状态，当任务状态从`Pending`转为`Running`后就可以查看日志和GPU使用率了。这里`-e`为了方便检查任务`Pending`的原因。通常看到`[Pulling] pulling image \"tensorflow/tensorflow:1.5.0-gpu-py3\"`代表容器镜像过大，导致任务处于`Pending`。这时可以重复执行下列命令直到任务状态变为`Running`。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "STATUS: PENDING\r\n",
      "NAMESPACE: default\r\n",
      "TRAINING DURATION: 0s\r\n",
      "\r\n",
      "NAME                  STATUS   TRAINER  AGE  INSTANCE                       NODE\r\n",
      "tf-distributed-mnist  PENDING  TFJOB    0s   tf-distributed-mnist-ps-0      N/A\r\n",
      "tf-distributed-mnist  PENDING  TFJOB    0s   tf-distributed-mnist-worker-0  N/A\r\n",
      "tf-distributed-mnist  PENDING  TFJOB    0s   tf-distributed-mnist-worker-1  N/A\r\n",
      "\r\n",
      "Your tensorboard will be available on:\r\n",
      "192.168.0.206:31963   \r\n",
      "\r\n",
      "Events: \r\n",
      "INSTANCE  TYPE  AGE  MESSAGE\r\n",
      "--------  ----  ---  -------\r\n",
      "                         \r\n",
      "                         \r\n",
      "                         \r\n"
     ]
    }
   ],
   "source": [
    "! arena get $JOB_NAME -e"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "8.实时检查日志，此时可以通过调整`--tail=`的数值展示输出的行数。默认为显示全部日志。\n",
    "如果想要实时查看日志，可以增加`-f`参数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2019-02-26T07:28:59.06252925Z WARNING:tensorflow:From /training/models/tensorflow-sample-code/tfjob/docker/mnist/main.py:40: read_data_sets (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\r\n",
      "2019-02-26T07:28:59.062611786Z Instructions for updating:\r\n",
      "2019-02-26T07:28:59.062616602Z Please use alternatives such as official/mnist/dataset.py from tensorflow/models.\r\n",
      "2019-02-26T07:28:59.102090755Z WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:260: maybe_download (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.\r\n",
      "2019-02-26T07:28:59.102120053Z Instructions for updating:\r\n",
      "2019-02-26T07:28:59.102123749Z Please write your own downloading logic.\r\n",
      "2019-02-26T07:28:59.106943556Z WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:262: extract_images (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\r\n",
      "2019-02-26T07:28:59.106955986Z Instructions for updating:\r\n",
      "2019-02-26T07:28:59.106959188Z Please use tf.data to implement this functionality.\r\n",
      "2019-02-26T07:28:59.705225214Z WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:267: extract_labels (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\r\n",
      "2019-02-26T07:28:59.705261581Z Instructions for updating:\r\n",
      "2019-02-26T07:28:59.705266844Z Please use tf.data to implement this functionality.\r\n",
      "2019-02-26T07:28:59.710114243Z WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:110: dense_to_one_hot (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\r\n",
      "2019-02-26T07:28:59.710131028Z Instructions for updating:\r\n",
      "2019-02-26T07:28:59.710134306Z Please use tf.one_hot on tensors.\r\n",
      "2019-02-26T07:28:59.775920928Z WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:290: DataSet.__init__ (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\r\n",
      "2019-02-26T07:28:59.775952959Z Instructions for updating:\r\n",
      "2019-02-26T07:28:59.775956287Z Please use alternatives such as official/mnist/dataset.py from tensorflow/models.\r\n",
      "2019-02-26T07:29:00.062051404Z WARNING:tensorflow:From /training/models/tensorflow-sample-code/tfjob/docker/mnist/main.py:119: softmax_cross_entropy_with_logits (from tensorflow.python.ops.nn_ops) is deprecated and will be removed in a future version.\r\n",
      "2019-02-26T07:29:00.062085896Z Instructions for updating:\r\n",
      "2019-02-26T07:29:00.062089719Z \r\n",
      "2019-02-26T07:29:00.062092246Z Future major versions of TensorFlow will allow gradients to flow\r\n",
      "2019-02-26T07:29:00.062095209Z into the labels input on backprop by default.\r\n",
      "2019-02-26T07:29:00.062097781Z \r\n",
      "2019-02-26T07:29:00.062100188Z See `tf.nn.softmax_cross_entropy_with_logits_v2`.\r\n",
      "2019-02-26T07:29:00.062114298Z \r\n",
      "2019-02-26T07:29:00.219646519Z 2019-02-26 07:29:00.219376: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA\r\n",
      "2019-02-26T07:29:00.395096237Z 2019-02-26 07:29:00.394842: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\r\n",
      "2019-02-26T07:29:00.396207408Z 2019-02-26 07:29:00.396052: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1411] Found device 0 with properties: \r\n",
      "2019-02-26T07:29:00.396224182Z name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285\r\n",
      "2019-02-26T07:29:00.396227903Z pciBusID: 0000:00:09.0\r\n",
      "2019-02-26T07:29:00.396230866Z totalMemory: 15.89GiB freeMemory: 15.60GiB\r\n",
      "2019-02-26T07:29:00.396234207Z 2019-02-26 07:29:00.396088: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1490] Adding visible gpu devices: 0\r\n",
      "2019-02-26T07:29:00.747280828Z 2019-02-26 07:29:00.747034: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] Device interconnect StreamExecutor with strength 1 edge matrix:\r\n",
      "2019-02-26T07:29:00.74731476Z 2019-02-26 07:29:00.747091: I tensorflow/core/common_runtime/gpu/gpu_device.cc:977]      0 \r\n",
      "2019-02-26T07:29:00.747318377Z 2019-02-26 07:29:00.747101: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0:   N \r\n",
      "2019-02-26T07:29:00.747676956Z 2019-02-26 07:29:00.747525: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1103] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15117 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:09.0, compute capability: 6.0)\r\n",
      "2019-02-26T07:29:06.348085472Z 2019-02-26 07:29:06.347875: I tensorflow/stream_executor/dso_loader.cc:151] successfully opened CUDA library libcupti.so.9.0 locally\r\n"
     ]
    }
   ],
   "source": [
    "! arena logs --tail=50 $JOB_NAME"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "9.查看实时训练的GPU使用情况"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INSTANCE NAME                  GPU(Device Index)  GPU(Duty Cycle)  GPU(Memory MiB)          STATUS   NODE\r\n",
      "tf-distributed-mnist-ps-0      N/A                N/A              N/A                      Running  192.168.0.208\r\n",
      "tf-distributed-mnist-worker-0  0                  9%               551.0MiB / 16276.2MiB    Running  192.168.0.210\r\n",
      "tf-distributed-mnist-worker-1  0                  6%               1092.0MiB / 16276.2MiB   Running  192.168.0.208\r\n"
     ]
    }
   ],
   "source": [
    "! arena top job $JOB_NAME"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "10.通过TensorBoard查看训练趋势。您可以使用 `192.168.1.117:30670` 访问 Tensorboard。如果您通过笔记本电脑无法直接访问 Tensorboard，可以考虑在您的笔记本电脑使用 `sshuttle`。例如：`sshuttle -r root@41.82.59.51 192.168.0.0/16`。其中`41.82.59.51`为集群内某个节点的外网IP，且该外网IP可以通过ssh访问。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "STATUS: RUNNING\r\n",
      "NAMESPACE: default\r\n",
      "TRAINING DURATION: 45s\r\n",
      "\r\n",
      "NAME                  STATUS   TRAINER  AGE  INSTANCE                       NODE\r\n",
      "tf-distributed-mnist  RUNNING  TFJOB    45s  tf-distributed-mnist-ps-0      192.168.0.208\r\n",
      "tf-distributed-mnist  RUNNING  TFJOB    45s  tf-distributed-mnist-worker-0  192.168.0.210\r\n",
      "tf-distributed-mnist  RUNNING  TFJOB    45s  tf-distributed-mnist-worker-1  192.168.0.208\r\n",
      "\r\n",
      "Your tensorboard will be available on:\r\n",
      "192.168.0.206:30308   \r\n"
     ]
    }
   ],
   "source": [
    "! arena get $JOB_NAME"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](2-1-tensorboard.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "11.查看模型训练产生的结果, 在`output`下生成了训练结果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/root\r\n",
      "|-- dataset\r\n",
      "|   `-- mnist\r\n",
      "|       |-- t10k-images-idx3-ubyte.gz\r\n",
      "|       |-- t10k-labels-idx1-ubyte.gz\r\n",
      "|       |-- train-images-idx3-ubyte.gz\r\n",
      "|       `-- train-labels-idx1-ubyte.gz\r\n",
      "|-- models\r\n",
      "|   `-- tensorflow-sample-code\r\n",
      "|       |-- README.md\r\n",
      "|       |-- data\r\n",
      "|       |-- mnist-tf\r\n",
      "|       |-- models\r\n",
      "|       |-- mpijob\r\n",
      "|       `-- tfjob\r\n",
      "`-- output\r\n",
      "    `-- distributed-mnist\r\n",
      "        |-- test\r\n",
      "        `-- train\r\n",
      "\r\n",
      "13 directories, 5 files\r\n"
     ]
    }
   ],
   "source": [
    "! tree -I ai-starter -L 3 ${HOME}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "12.删除已经完成的任务"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "service \"tf-distributed-mnist-tensorboard\" deleted\n",
      "deployment.extensions \"tf-distributed-mnist-tensorboard\" deleted\n",
      "tfjob.kubeflow.org \"tf-distributed-mnist\" deleted\n",
      "configmap \"tf-distributed-mnist-tfjob\" deleted\n",
      "\u001b[36mINFO\u001b[0m[0004] The Job tf-distributed-mnist has been deleted successfully \n"
     ]
    }
   ],
   "source": [
    "! arena delete $JOB_NAME"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "恭喜！您已经使用 `arena` 成功运行了训练作业，而且还能轻松检查 Tensorboard。\n",
    "\n",
    "总结，希望您通过本次演示了解：\n",
    "1. 如何准备代码和数据，并将其放入数据卷中\n",
    "2. 如何在分布式训练任务中引用数据卷，并且使用其中的代码和数据\n",
    "3. 如何利用arena管理您的分布式训练任务。\n",
    "\n",
    "以上是使用`Arena`在云上进行机器学习的例子，您可以修改代码`${HOME}/models/tensorflow-sample-code/tfjob/docker/v1alpha2/distributed-mnist/main.py`重新提交，从而实现模型开发的效果。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}