lab_guide.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Welcome to the stream analytics lab \n",
    "\n",
    "The purpose of this lab is to acquaint you with some basic techniques in stream analytics. We will study two sketches, the Count-min sketch and the AMS sketch. Our goal will be to construct some sketches, perform some queries on them and determine experimentally their error and probabilistic characteristics.\n",
    "\n",
    "Our lab excercises will be performed completely through your web browser, in the Jupyter environment (the web page you are reading right now!) and will be done in Python. If you are unfamiliar with the Python programming language, do not worry too much; for one thing, Python is probably the easiest language to learn, and you can always get help from the lab instructors.\n",
    "\n",
    "## First steps\n",
    "\n",
    "First, you need to make a copy of this page, so that you can begin to edit it.\n",
    "\n",
    "In the **File** menu (on the top left of the screen) select the 'Make a Copy...' item\n",
    "\n",
    "** __DO THIS NOW__ **\n",
    "\n",
    "If all went well, the name at the top of the screen has now changed from `lab_guide` to something like `lab_guide-CopyNN`.  Using again the **File** menu, you can rename the page to something friendlier, e.g.\n",
    "`lab_guide-AlanTuring` (assuming you __are__ Alan...)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting around this page\n",
    "\n",
    "We are now ready to start with the lab. First, we need to familiarize ourselves with Jupyter. This page is organized in a number of cells. Some cells (like the one you are reading) are text cells, and some cells are \n",
    "Python code. \n",
    "\n",
    "You can click on any cell and edit its contents. Once you have edited a cell, you can type **Ctrl-Enter** to execute the cell.  Text cells execute by formatting their text nicely (using github markup). Python cells execute the Python code in a background interpreter and display all kinds of output.\n",
    "\n",
    "For example, in the next cell, we are executing a Python expression:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "1+1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Loading the python file to use\n",
    "\n",
    "There is some code that you can use for the lab. It is included in file lab.py. You can see the source code at the following link (https://github.com/vsamtuc/ssch17/blob/master/lab.py)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from lab import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let us create some data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([4, 9, 3, 2, 6, 6, 2, 7, 5, 0, 2, 7, 1, 3, 3, 5, 9, 7, 4, 9])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = make_random_data(10, 20)\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We created an array of 20 random integers, each in the range 0 to 9 (inclusive).\n",
    "\n",
    "Let us now create the frequency vector for this data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({0: 1, 1: 1, 2: 3, 3: 3, 4: 2, 5: 2, 6: 2, 7: 3, 9: 3})"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dfv = frequency_vector(data)\n",
    "dfv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sketching with families of hash functions\n",
    "\n",
    "A sketch (either Count-min or AMS) requires some family of hash functions. The number of hash functions in the family is called the **depth** of the sketch. The number of hash functions is $\\Theta(\\log \\frac{1}{\\delta})$ where $\\delta$ is the probability of failure for a query.\n",
    "\n",
    "Also, a sketch has a **width**, which is the number of counters per depth used. The width is $\\Theta(\\frac{1}{\\epsilon})$ for CM-sketches and $\\Theta(\\frac{1}{\\epsilon^2})$ for AMS sketches, where $\\epsilon$ is the accuracy of the sketch.\n",
    "\n",
    "The two quantities, width and depth, combine into a projection object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "width =  10\n",
      "depth =  3\n",
      "hash family =  <lab.hash_family object at 0x7ff7880ea9e8>\n",
      "Hashing key 4 =  [6 6 9]\n"
     ]
    }
   ],
   "source": [
    "proj = projection(10,3)  # width = 10, depth = 3\n",
    "print('width = ', proj.width)\n",
    "print('depth = ', proj.depth)\n",
    "print('hash family = ', proj.hf)\n",
    "\n",
    "print(\"Hashing key 4 = \", proj.hash(4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### CM sketches\n",
    "\n",
    "Now, using the projection we can create a CM sketch for our data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The array for the sketch\n",
      " [[ 2.  0.  2.  3.  0.  3.  6.  1.  0.  3.]\n",
      " [ 5.  1.  3.  0.  2.  1.  2.  6.  0.  0.]\n",
      " [ 0.  1.  4.  0.  3.  2.  2.  3.  0.  5.]]\n",
      "Locations for key 5 [0 0 6]\n",
      "CM sketch counts for key 5 =  [ 2.  5.  2.]\n",
      "True count for key 5 =  2\n"
     ]
    }
   ],
   "source": [
    "cmsk = make_count_min_sketch(proj, dfv)\n",
    "\n",
    "print(\"The array for the sketch\\n\",cmsk.vec)\n",
    "\n",
    "print(\"Locations for key 5\", cmsk.proj.hash(5))\n",
    "print(\"CM sketch counts for key 5 = \", cmsk.vec[range(cmsk.proj.depth), cmsk.proj.hash(5)])\n",
    "print(\"True count for key 5 = \", dfv[5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### AMS sketches\n",
    "\n",
    "We can also create a FastAGMS sketch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The array for the sketch\n",
      " [[ 2.  0. -2.  3.  0.  3. -6.  1.  0.  3.]\n",
      " [ 5. -1.  3.  0.  2. -1.  2. -6.  0.  0.]\n",
      " [ 0. -1.  2.  0.  3. -2.  2.  3.  0.  1.]]\n",
      "Locations for key 5 [0 0 6]\n",
      "CM sketch counts for key 5 =  [ 2.  5.  2.]\n",
      "True count for key 5 =  2\n"
     ]
    }
   ],
   "source": [
    "amsk = make_ams_sketch(proj, dfv)\n",
    "\n",
    "print(\"The array for the sketch\\n\", amsk.vec)\n",
    "\n",
    "print(\"Locations for key 5\", amsk.proj.hash(5))\n",
    "print(\"CM sketch counts for key 5 = \", amsk.vec[range(cmsk.proj.depth), cmsk.proj.hash(5)])\n",
    "print(\"True count for key 5 = \", dfv[5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Of course, AMS sketches are not very good at point queries. Let us see if our sketch does better in computing the second moment of the frequency vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Inner product estimate of AMS sketch = 72.0\n",
      "Exact inner product from freq.vector = 50\n"
     ]
    }
   ],
   "source": [
    "print(\"Inner product estimate of AMS sketch =\", amsk @ amsk)\n",
    "print(\"Exact inner product from freq.vector =\", frequency_vector_inner(dfv, dfv))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Populating the interactive namespace from numpy and matplotlib\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/minos/school/lib/python3.5/site-packages/IPython/core/magics/pylab.py:160: UserWarning: pylab import has clobbered these variables: ['sqrt']\n",
      "`%matplotlib` prevents importing * from pylab and numpy\n",
      "  \"\\n`%matplotlib` prevents importing * from pylab and numpy\"\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(array([ 8.,  0.,  0.,  0.,  0.,  1.,  0.,  0.,  0.,  1.]),\n",
       " array([ 0. ,  0.2,  0.4,  0.6,  0.8,  1. ,  1.2,  1.4,  1.6,  1.8,  2. ]),\n",
       " <a list of 10 Patch objects>)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAD8CAYAAABXe05zAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADutJREFUeJzt3X+MHPV9xvHnqW0gJghMvG0RYM5IiAiiJrgrlBBEAySN\nMQ1u1P5hVKqQUl3TJim0VSsipKbtPyVS1aZVq1YnSpMo1CQ4UCUEUtxiFKXUpmvHYBvjBBuH4NJ4\ngfDDqQSBfvrHzoX1sXv7Xbwz5w+8X9LJszOzu4+/N35uPLNz44gQACCPn1roAACA8VDcAJAMxQ0A\nyVDcAJAMxQ0AyVDcAJAMxQ0AyVDcAJAMxQ0AySyu40WXL18eU1NTdbw0ALwhbd269amIaJWsW0tx\nT01NqdPp1PHSAPCGZPt7petyqAQAkqG4ASAZihsAkqG4ASAZihsAkikqbtu/Z3uX7Z2219s+ru5g\nAIDBRha37VMl/a6kdkS8Q9IiSevqDgYAGKz0UMliSW+xvVjSUkn/XV8kAMB8RhZ3RByQ9BeSHpf0\npKTnIuKeuoMBAAYbeeWk7WWS1kpaKelZSbfZvioivjhnvWlJ05K0YsWK1x1o6vqvv+7nHon9N16+\nIO8LAOMqOVTyfkmPRUQ3In4s6XZJF8xdKSJmIqIdEe1Wq+hyewDA61BS3I9LerftpbYt6VJJu+uN\nBQAYpuQY9xZJGyRtk7Sjes5MzbkAAEMU/XbAiPi0pE/XnAUAUIArJwEgGYobAJKhuAEgGYobAJKh\nuAEgGYobAJKhuAEgGYobAJKhuAEgGYobAJKhuAEgGYobAJKhuAEgGYobAJKhuAEgGYobAJKhuAEg\nmZHFbfts29v7vp63fV0T4QAArzXy1mURsUfSuyTJ9iJJByTdUXMuAMAQ4x4quVTS3oj4Xh1hAACj\njVvc6yStryMIAKBMcXHbPkbSFZJuG7J82nbHdqfb7U4qHwBgjnH2uC+TtC0ifjBoYUTMREQ7Itqt\nVmsy6QAArzFOcV8pDpMAwIIrKm7bx0v6gKTb640DABhl5McBJSkifiTpbTVnAQAU4MpJAEiG4gaA\nZChuAEiG4gaAZChuAEiG4gaAZChuAEiG4gaAZChuAEiG4gaAZChuAEiG4gaAZChuAEiG4gaAZChu\nAEiG4gaAZChuAEim9NZlJ9neYPsR27ttv6fuYACAwYpuXSbpryV9IyJ+1fYxkpbWmAkAMI+RxW37\nREkXSbpakiLiJUkv1RsLADBMyaGSlZK6kv7J9rdt31Td9f0wtqdtd2x3ut3uxIMCAHpKinuxpFWS\n/j4izpP0I0nXz10pImYioh0R7VarNeGYAIBZJcX9hKQnImJL9XiDekUOAFgAI4s7Iv5H0vdtn13N\nulTSw7WmAgAMVfqpkk9KuqX6RMk+SR+tLxIAYD5FxR0R2yW1a84CACjAlZMAkAzFDQDJUNwAkAzF\nDQDJUNwAkAzFDQDJUNwAkAzFDQDJUNwAkAzFDQDJUNwAkAzFDQDJUNwAkAzFDQDJUNwAkAzFDQDJ\nUNwAkEzRHXBs75f0gqRXJL0cEdwNBwAWSOk9JyXp4oh4qrYkAIAiHCoBgGRKizsk3WN7q+3pQSvY\nnrbdsd3pdruTSwgAOExpcV8YEaskXSbp47YvmrtCRMxERDsi2q1Wa6IhAQCvKiruiDhQ/XlQ0h2S\nzq8zFABguJHFbft42yfMTkv6RUk76w4GABis5FMlPyPpDtuz6/9zRHyj1lQAgKFGFndE7JP0zgay\nAAAK8HFAAEiG4gaAZChuAEiG4gaAZChuAEiG4gaAZChuAEiG4gaAZChuAEiG4gaAZChuAEiG4gaA\nZChuAEiG4gaAZChuAEiG4gaAZChuAEimuLhtL7L9bdt31hkIADC/cfa4r5W0u64gAIAyRcVt+zRJ\nl0u6qd44AIBRSve4PyvpjyT937AVbE/b7tjudLvdiYQDALzWyOK2/UuSDkbE1vnWi4iZiGhHRLvV\nak0sIADgcCV73O+VdIXt/ZJulXSJ7S/WmgoAMNTI4o6IT0XEaRExJWmdpHsj4qrakwEABuJz3ACQ\nzOJxVo6I+yTdV0sSAEAR9rgBIBmKGwCSobgBIBmKGwCSobgBIBmKGwCSobgBIBmKGwCSobgBIBmK\nGwCSobgBIBmKGwCSobgBIBmKGwCSobgBIBmKGwCSKblZ8HG2H7D9oO1dtv+0iWAAgMFK7oDzoqRL\nIuKQ7SWSvmX77ojYXHM2AMAAI4s7IkLSoerhkuor6gwFABiu6Bi37UW2t0s6KGljRGypNxYAYJii\n4o6IVyLiXZJOk3S+7XfMXcf2tO2O7U632510TgBAZaxPlUTEs5I2SVo9YNlMRLQjot1qtSaVDwAw\nR8mnSlq2T6qm3yLpA5IeqTsYAGCwkk+VnCLp87YXqVf0X46IO+uNBQAYpuRTJQ9JOq+BLACAAlw5\nCQDJUNwAkAzFDQDJUNwAkAzFDQDJUNwAkAzFDQDJUNwAkAzFDQDJUNwAkAzFDQDJUNwAkAzFDQDJ\nUNwAkAzFDQDJUNwAkAzFDQDJlNxz8nTbm2w/bHuX7WubCAYAGKzknpMvS/qDiNhm+wRJW21vjIiH\na84GABhg5B53RDwZEduq6Rck7ZZ0at3BAACDjXWM2/aUejcO3lJHGADAaMXFbfutkr4i6bqIeH7A\n8mnbHdudbrc7yYwAgD5FxW17iXqlfUtE3D5onYiYiYh2RLRbrdYkMwIA+pR8qsSS/lHS7oj4y/oj\nAQDmU7LH/V5Jvy7pEtvbq681NecCAAwx8uOAEfEtSW4gCwCgAFdOAkAyFDcAJENxA0AyFDcAJENx\nA0AyFDcAJENxA0AyFDcAJENxA0AyFDcAJENxA0AyFDcAJENxA0AyFDcAJENxA0AyFDcAJENxA0Ay\nJfecvNn2Qds7mwgEAJhfyR735yStrjkHAKDQyOKOiG9KeqaBLACAAhzjBoBkRt7lvZTtaUnTkrRi\nxYpJvSwwUVPXf31B3nf/jZcvyPu+Wb3Rv88T2+OOiJmIaEdEu9VqTeplAQBzcKgEAJIp+Tjgekn/\nKels20/Yvqb+WACAYUYe446IK5sIAgAow6ESAEiG4gaAZChuAEiG4gaAZChuAEiG4gaAZChuAEiG\n4gaAZChuAEiG4gaAZChuAEiG4gaAZChuAEiG4gaAZChuAEiG4gaAZChuAEimqLhtr7a9x/ajtq+v\nOxQAYLiSe04ukvR3ki6TdI6kK22fU3cwAMBgJXvc50t6NCL2RcRLkm6VtLbeWACAYUqK+1RJ3+97\n/EQ1DwCwAEbe5b2U7WlJ09XDQ7b3vM6XWi7pqcmkKufPjFxlQXIVINd42L7GQ64x+DNHlOuM0hVL\nivuApNP7Hp9WzTtMRMxImil942FsdyKifaSvM2nkGg+5xkOu8bzZc5UcKvkvSWfZXmn7GEnrJH21\n3lgAgGFG7nFHxMu2PyHpXyUtknRzROyqPRkAYKCiY9wRcZeku2rOMuuID7fUhFzjIdd4yDWeN3Uu\nR0QT7wMAmBAueQeAZBor7lGXzds+1vaXquVbbE/1LftUNX+P7Q82nOv3bT9s+yHb/277jL5lr9je\nXn1N9IRtQa6rbXf73v83+5Z9xPZ3q6+PNJzrr/oyfcf2s33L6hyvm20ftL1zyHLb/psq90O2V/Ut\nq3O8RuX6tSrPDtv3235n37L91fzttjsN53qf7ef6vl9/3Lestl+BUZDrD/sy7ay2qZOrZXWO1+m2\nN1VdsMv2tQPWaW4bi4jav9Q7qblX0pmSjpH0oKRz5qzzO5L+oZpeJ+lL1fQ51frHSlpZvc6iBnNd\nLGlpNf3bs7mqx4cWcLyulvS3A557sqR91Z/LqullTeWas/4n1TuZXet4Va99kaRVknYOWb5G0t2S\nLOndkrbUPV6FuS6YfT/1fq3Elr5l+yUtX6Dxep+kO490G5h0rjnrfkjSvQ2N1ymSVlXTJ0j6zoB/\nk41tY03tcZdcNr9W0uer6Q2SLrXtav6tEfFiRDwm6dHq9RrJFRGbIuJ/q4eb1fsce92O5NcMfFDS\nxoh4JiJ+KGmjpNULlOtKSesn9N7ziohvSnpmnlXWSvpC9GyWdJLtU1TveI3MFRH3V+8rNbd9lYzX\nMLX+CowxczW5fT0ZEduq6Rck7dZrryBvbBtrqrhLLpv/yToR8bKk5yS9rfC5debqd416P1FnHWe7\nY3uz7V+eUKZxcv1K9V+yDbZnL5I6KsarOqS0UtK9fbPrGq8Sw7IfTb/SYe72FZLusb3VvSuTm/Ye\n2w/avtv2udW8o2K8bC9Vr/y+0je7kfFy7zDueZK2zFnU2DY2sUve3+hsXyWpLekX+mafEREHbJ8p\n6V7bOyJib0ORviZpfUS8aPu31PvfyiUNvXeJdZI2RMQrffMWcryOarYvVq+4L+ybfWE1Xj8taaPt\nR6o90iZsU+/7dcj2Gkn/Iumsht67xIck/UdE9O+d1z5ett+q3g+L6yLi+Um+9jia2uMuuWz+J+vY\nXizpRElPFz63zlyy/X5JN0i6IiJenJ0fEQeqP/dJuk+9n8KN5IqIp/uy3CTp50ufW2euPus057+x\nNY5XiWHZ6xyvIrZ/Tr3v4dqIeHp2ft94HZR0hyZ3iHCkiHg+Ig5V03dJWmJ7uY6C8arMt33VMl62\nl6hX2rdExO0DVmluG6vjQP6AA/uL1Tsgv1KvntA4d846H9fhJye/XE2fq8NPTu7T5E5OluQ6T72T\nMWfNmb9M0rHV9HJJ39WETtIU5jqlb/rDkjbHqydCHqvyLaumT24qV7Xe29U7UeQmxqvvPaY0/GTb\n5Tr8xNEDdY9XYa4V6p23uWDO/OMlndA3fb+k1Q3m+tnZ7596Bfh4NXZF20BduarlJ6p3HPz4psar\n+rt/QdJn51mnsW1sYoNd8Bdfo96Z2L2Sbqjm/Zl6e7GSdJyk26qN+AFJZ/Y994bqeXskXdZwrn+T\n9ANJ26uvr1bzL5C0o9pwd0i6puFcfy5pV/X+myS9ve+5v1GN46OSPtpkrurxn0i6cc7z6h6v9ZKe\nlPRj9Y4hXiPpY5I+Vi23ejcE2Vu9f7uh8RqV6yZJP+zbvjrV/DOrsXqw+j7f0HCuT/RtX5vV94Nl\n0DbQVK5qnavV+8BC//PqHq8L1TuG/lDf92rNQm1jXDkJAMlw5SQAJENxA0AyFDcAJENxA0AyFDcA\nJENxA0AyFDcAJENxA0Ay/w/zDgwnhrKzdgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7ff76d1a4710>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%pylab inline\n",
    "import pandas as pd\n",
    "\n",
    "# this is a point query for CM sketches\n",
    "def cm_query(cmsk, x):\n",
    "    return np.min(cmsk.vec[range(cmsk.proj.depth), cmsk.proj.hash(x)])\n",
    "\n",
    "\n",
    "# Let us compute the difference between the counts from CM sketches and the freq.vector\n",
    "\n",
    "keys = pd.Series(list(range(10)))\n",
    "fvcounts = pd.Series([dfv[key] for key in keys])\n",
    "cmcounts = pd.Series([cm_query(cmsk, key) for key in keys ])\n",
    "dcount = cmcounts - fvcounts\n",
    "hist(dcount)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Working with 'big data'\n",
    "\n",
    "We can now experiment with a larger dataset, obtained from the WorldCup site: (http://ita.ee.lbl.gov/html/contrib/WorldCup.html)\n",
    "\n",
    "The following code loads the some 7 million records from a file and selects one of the attributes as key"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(897256801,    2703,  241,        259, 0,  66, 1, 99)\n",
      " (897256801,  423679,  323,          0, 0, 141, 1, 99)\n",
      " (897256801,  797852,  219,          0, 0,  77, 1, 99) ...,\n",
      " (897310251,  796717, 1356,      12349, 0,  66, 1, 99)\n",
      " (897310251, 1209201,    5,       1504, 0,  66, 1, 99)\n",
      " (897310251,   12300,  184, 4294967295, 0,  77, 1, 99)]\n",
      "[('timestamp', '>u4'), ('clientID', '>u4'), ('objectID', '>u4'), ('size', '>u4'), ('method', 'u1'), ('status', 'u1'), ('type', 'u1'), ('server', 'u1')]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([   2703,  423679,  797852, ...,  796717, 1209201,   12300], dtype=uint32)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "WCUP = read_wcup_native('wc_day44_1')\n",
    "\n",
    "print(WCUP)\n",
    "print(WCUP.dtype)\n",
    "\n",
    "wcdata = WCUP['clientID']\n",
    "wcdata"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  How to continue\n",
    "\n",
    "You can now proceed as above, repeating the experiments with a large dataset. Your goals are:\n",
    " * Create a frequency vector for the data in the `wcdata` array\n",
    " * Create a CM sketch and plot the distribution of error in point queries\n",
    " * Using AMS sketches, create a plot of the distribution of error in estimation of the norm, for AMS sketches of    different widths"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}