turicreate_sframes_intro.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "turicreate-sframes-intro.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "[View in Colaboratory](https://colab.research.google.com/github/jagatfx/turicreate-colab/blob/master/turicreate_sframes_intro.ipynb)"
      ]
    },
    {
      "metadata": {
        "id": "MRYC2Vn02lMd",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Introduction to Turi Create SFrames\n",
        "\n",
        "https://github.com/apple/turicreate/blob/master/userguide/sframe/sframe-intro.md"
      ]
    },
    {
      "metadata": {
        "id": "qo2KqxVs2q8B",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "SFrames are the primary data structure for extracting data from other sources for use in Turi Create.\n",
        "\n",
        "They are similar to Pandas Dataframes but do not need to be loaded as a whole into RAM, so are not constrained by the RAM of the machine running the code. This makes it a scalable data structure. It is column immutable and supports out-of-core processing.\n",
        "\n",
        "SFrames can extract data from the following static file formats:\n",
        "\n",
        "*   CSV\n",
        "*   JSON\n",
        "*   SQL databases"
      ]
    },
    {
      "metadata": {
        "id": "Ew1_T9B94_ip",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Turi Create and GPU Setup"
      ]
    },
    {
      "metadata": {
        "id": "DCSsL_vs5BHF",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!apt install libnvrtc8.0\n",
        "!pip uninstall -y mxnet-cu80 && pip install mxnet-cu80==1.1.0\n",
        "!pip install turicreate"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "-nVQfLiK6U_7",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Google Drive Access\n",
        "\n",
        "You will be asked to click a link to generate a secret key to access your Google Drive. \n",
        "\n",
        "Copy and paste secret key it into the space provided with the notebook."
      ]
    },
    {
      "metadata": {
        "id": "y5a7kHU26Vm9",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import os.path\n",
        "from google.colab import drive\n",
        "\n",
        "# mount Google Drive to /content/drive/My Drive/\n",
        "if os.path.isdir(\"/content/drive/My Drive\"):\n",
        "  print(\"Google Drive already mounted\")\n",
        "else:\n",
        "  drive.mount('/content/drive')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "dTWuiXji9cEK",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Fetch Data"
      ]
    },
    {
      "metadata": {
        "id": "_q-u2IbDPA2Q",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import os.path\n",
        "import urllib.request\n",
        "import tarfile\n",
        "import zipfile\n",
        "import gzip\n",
        "from shutil import copy\n",
        "\n",
        "def fetch_remote_datafile(filename, remote_url):\n",
        "  if os.path.isfile(\"./\" + filename):\n",
        "    print(\"already have \" + filename + \" in workspace\")\n",
        "    return\n",
        "  print(\"fetching \" + filename + \" from \" + remote_url + \"...\")\n",
        "  urllib.request.urlretrieve(remote_url, \"./\" + filename)\n",
        "\n",
        "def cache_datafile_in_drive(filename):\n",
        "  if os.path.isfile(\"./\" + filename) == False:\n",
        "    print(\"cannot cache \" + filename + \", it is not in workspace\")\n",
        "    return\n",
        "  \n",
        "  data_drive_path = \"/content/drive/My Drive/Colab Notebooks/data/\"\n",
        "  if os.path.isfile(data_drive_path + filename):\n",
        "    print(\"\" + filename + \" has already been stored in Google Drive\")\n",
        "  else:\n",
        "    print(\"copying \" + filename + \" to \" + data_drive_path)\n",
        "    copy(\"./\" + filename, data_drive_path)\n",
        "  \n",
        "\n",
        "def load_datafile_from_drive(filename, remote_url=None):\n",
        "  data_drive_path = \"/content/drive/My Drive/Colab Notebooks/data/\"\n",
        "  if os.path.isfile(\"./\" + filename):\n",
        "    print(\"already have \" + filename + \" in workspace\")\n",
        "  elif os.path.isfile(data_drive_path + filename):\n",
        "    print(\"have \" + filename + \" in Google Drive, copying to workspace...\")\n",
        "    copy(data_drive_path + filename, \".\")\n",
        "  elif remote_url != None:\n",
        "    fetch_remote_datafile(filename, remote_url)\n",
        "  else:\n",
        "    print(\"error: you need to manually download \" + filename + \" and put in drive\")\n",
        "    \n",
        "def extract_datafile(filename, expected_extract_artifact=None):\n",
        "  if expected_extract_artifact != None and (os.path.isfile(expected_extract_artifact) or os.path.isdir(expected_extract_artifact)):\n",
        "    print(\"files in \" + filename + \" have already been extracted\")\n",
        "  elif os.path.isfile(\"./\" + filename) == False:\n",
        "    print(\"error: cannot extract \" + filename + \", it is not in the workspace\")\n",
        "  else:\n",
        "    extension = filename.split('.')[-1]\n",
        "    if extension == \"zip\":\n",
        "      print(\"extracting \" + filename + \"...\")\n",
        "      data_file = open(filename, \"rb\")\n",
        "      z = zipfile.ZipFile(data_file)\n",
        "      for name in z.namelist():\n",
        "          print(\"    extracting file\", name)\n",
        "          z.extract(name, \"./\")\n",
        "      data_file.close()\n",
        "    elif extension == \"gz\":\n",
        "      print(\"extracting \" + filename + \"...\")\n",
        "      if filename.split('.')[-2] == \"tar\":\n",
        "        tar = tarfile.open(filename)\n",
        "        tar.extractall()\n",
        "        tar.close()\n",
        "      else:\n",
        "        data_zip_file = gzip.GzipFile(filename, 'rb')\n",
        "        data = data_zip_file.read()\n",
        "        data_zip_file.close()\n",
        "        extracted_file = open('.'.join(filename.split('.')[0:-1]), 'wb')\n",
        "        extracted_file.write(data)\n",
        "        extracted_file.close()\n",
        "    elif extension == \"tar\":\n",
        "      print(\"extracting \" + filename + \"...\")\n",
        "      tar = tarfile.open(filename)\n",
        "      tar.extractall()\n",
        "      tar.close()\n",
        "    elif extension == \"csv\":\n",
        "      print(\"do not need to extract csv\")\n",
        "    else:\n",
        "      print(\"cannot extract \" + filename)\n",
        "      \n",
        "def load_cache_extract_datafile(filename, expected_extract_artifact=None, remote_url=None):\n",
        "  load_datafile_from_drive(filename, remote_url)\n",
        "  extract_datafile(filename, expected_extract_artifact)\n",
        "  cache_datafile_in_drive(filename)\n",
        "  "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ja-WPIYCPBo2",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 71
        },
        "outputId": "e3a55a49-3595-4033-8fc4-c29f4e636047"
      },
      "cell_type": "code",
      "source": [
        "load_cache_extract_datafile(\"song_data.csv.zip\", \"song_data.csv\", \"https://static.turi.com/datasets/millionsong/song_data.csv\")"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "already have song_data.csv.zip in workspace\n",
            "files in song_data.csv.zip have already been extracted\n",
            "song_data.csv.zip has already been stored in Google Drive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "0cOYIVEQPRJM",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 71
        },
        "outputId": "718819df-2d6f-487a-ec33-5f613a84af32"
      },
      "cell_type": "code",
      "source": [
        "load_cache_extract_datafile(\"10000.txt.zip\", \"10000.txt\", \"https://static.turi.com/datasets/millionsong/10000.txt\")"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "already have 10000.txt.zip in workspace\n",
            "files in 10000.txt.zip have already been extracted\n",
            "10000.txt.zip has already been stored in Google Drive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "lBLwgPKlPdGs",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 71
        },
        "outputId": "a89500cb-6460-4b86-d8a3-df697dcb3582"
      },
      "cell_type": "code",
      "source": [
        "load_cache_extract_datafile(\"loc-gowalla_totalCheckins.txt.gz\", \"loc-gowalla_totalCheckins.txt\", \"https://snap.stanford.edu/data/loc-gowalla_totalCheckins.txt.gz\")"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "already have loc-gowalla_totalCheckins.txt.gz in workspace\n",
            "files in loc-gowalla_totalCheckins.txt.gz have already been extracted\n",
            "loc-gowalla_totalCheckins.txt.gz has already been stored in Google Drive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "_-x_YK5h9DfS",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Setup Turi Create"
      ]
    },
    {
      "metadata": {
        "id": "R80hcNX19F7X",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import mxnet as mx\n",
        "import turicreate as tc"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "JMpUas6Y9IHc",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Use all GPUs (default)\n",
        "tc.config.set_num_gpus(-1)\n",
        "\n",
        "# Use only 1 GPU\n",
        "#tc.config.set_num_gpus(1)\n",
        "\n",
        "# Use CPU\n",
        "#tc.config.set_num_gpus(0)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "zi7jvSvE27eP",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Sample Data\n",
        "\n",
        "The Million Song Dataset is a freely-available collection of audio features and metadata for a million contemporary popular music tracks.\n",
        "\n",
        "https://labrosa.ee.columbia.edu/millionsong/\n",
        "\n",
        "The first table contains metadata about each song in the database. Here's how we load it into an SFrame:"
      ]
    },
    {
      "metadata": {
        "id": "Z1c84n0o2Znq",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 233
        },
        "outputId": "87a21db1-2574-4da4-c40f-05b1cc002af8"
      },
      "cell_type": "code",
      "source": [
        "songs = tc.SFrame.read_csv(\"./song_data.csv\")"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<pre>Finished parsing file /content/song_data.csv</pre>"
            ],
            "text/plain": [
              "Finished parsing file /content/song_data.csv"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<pre>Parsing completed. Parsed 100 lines in 1.86884 secs.</pre>"
            ],
            "text/plain": [
              "Parsing completed. Parsed 100 lines in 1.86884 secs."
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "------------------------------------------------------\n",
            "Inferred types from first 100 line(s) of file as \n",
            "column_type_hints=[str,str,str,str,int]\n",
            "If parsing fails due to incorrect types, you can correct\n",
            "the inferred type list above and pass it to read_csv in\n",
            "the column_type_hints argument\n",
            "------------------------------------------------------\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<pre>Read 637410 lines. Lines per second: 490463</pre>"
            ],
            "text/plain": [
              "Read 637410 lines. Lines per second: 490463"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<pre>Finished parsing file /content/song_data.csv</pre>"
            ],
            "text/plain": [
              "Finished parsing file /content/song_data.csv"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<pre>Parsing completed. Parsed 1000000 lines in 1.49576 secs.</pre>"
            ],
            "text/plain": [
              "Parsing completed. Parsed 1000000 lines in 1.49576 secs."
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "lgWs2Lbo3fF_",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 296
        },
        "outputId": "2b0ecc17-dd31-406a-9e89-c35212171886"
      },
      "cell_type": "code",
      "source": [
        "songs.head()"
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\"><table frame=\"box\" rules=\"cols\">\n",
              "    <tr>\n",
              "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">song_id</th>\n",
              "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">title</th>\n",
              "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">release</th>\n",
              "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">artist_name</th>\n",
              "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">year</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOQMMHC12AB0180CB8</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Silent Night</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Monster Ballads X-Mas</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Faster Pussy cat</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2003</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOVFVAK12A8C1350D9</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Tanssi vaan</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Karkuteillä</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Karkkiautomaatti</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1995</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOGTUKN12AB017F4F1</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">No One Could Ever</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Butter</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Hudson Mohawke</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2006</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOBNYVR12A8C13558C</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Si Vos Querés</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">De Culo</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Yerba Brava</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2003</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOHSBXH12A8C13B0DF</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Tangle Of Aspens</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Rene Ablaze Presents<br>Winter Sessions ...</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Der Mystic</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOZVAPQ12A8C13B63C</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Symphony No. 1 G minor<br>\"Sinfonie ...</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Berwald: Symphonies Nos.<br>1/2/3/4 ...</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">David Montgomery</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOQVRHI12A6D4FB2D7</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">We Have Got Love</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Strictly The Best Vol. 34</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Sasha / Turbulence</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOEYRFT12AB018936C</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2 Da Beat Ch'yall</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Da Bomb</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Kris Kross</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1993</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOPMIYT12A6D4F851E</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Goodbye</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Danny Boy</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Joseph Locke</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOJCFMH12A8C13B0C2</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Mama_ mama can't you see<br>? ...</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">March to cadence with the<br>US marines ...</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">The Sun Harbor's Chorus-<br>Documentary Recordings ...</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
              "    </tr>\n",
              "</table>\n",
              "[10 rows x 5 columns]<br/>\n",
              "</div>"
            ],
            "text/plain": [
              "Columns:\n",
              "\tsong_id\tstr\n",
              "\ttitle\tstr\n",
              "\trelease\tstr\n",
              "\tartist_name\tstr\n",
              "\tyear\tint\n",
              "\n",
              "Rows: 10\n",
              "\n",
              "Data:\n",
              "+--------------------+-------------------------------+\n",
              "|      song_id       |             title             |\n",
              "+--------------------+-------------------------------+\n",
              "| SOQMMHC12AB0180CB8 |          Silent Night         |\n",
              "| SOVFVAK12A8C1350D9 |          Tanssi vaan          |\n",
              "| SOGTUKN12AB017F4F1 |       No One Could Ever       |\n",
              "| SOBNYVR12A8C13558C |         Si Vos Querés         |\n",
              "| SOHSBXH12A8C13B0DF |        Tangle Of Aspens       |\n",
              "| SOZVAPQ12A8C13B63C | Symphony No. 1 G minor \"Si... |\n",
              "| SOQVRHI12A6D4FB2D7 |        We Have Got Love       |\n",
              "| SOEYRFT12AB018936C |       2 Da Beat Ch'yall       |\n",
              "| SOPMIYT12A6D4F851E |            Goodbye            |\n",
              "| SOJCFMH12A8C13B0C2 |   Mama_ mama can't you see ?  |\n",
              "+--------------------+-------------------------------+\n",
              "+-------------------------------+-------------------------------+------+\n",
              "|            release            |          artist_name          | year |\n",
              "+-------------------------------+-------------------------------+------+\n",
              "|     Monster Ballads X-Mas     |        Faster Pussy cat       | 2003 |\n",
              "|          Karkuteillä          |        Karkkiautomaatti       | 1995 |\n",
              "|             Butter            |         Hudson Mohawke        | 2006 |\n",
              "|            De Culo            |          Yerba Brava          | 2003 |\n",
              "| Rene Ablaze Presents Winte... |           Der Mystic          |  0   |\n",
              "| Berwald: Symphonies Nos. 1... |        David Montgomery       |  0   |\n",
              "|   Strictly The Best Vol. 34   |       Sasha / Turbulence      |  0   |\n",
              "|            Da Bomb            |           Kris Kross          | 1993 |\n",
              "|           Danny Boy           |          Joseph Locke         |  0   |\n",
              "| March to cadence with the ... | The Sun Harbor's Chorus-Do... |  0   |\n",
              "+-------------------------------+-------------------------------+------+\n",
              "[10 rows x 5 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 21
        }
      ]
    },
    {
      "metadata": {
        "id": "SlHixuEd3MI6",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "No options are needed for the simplest case, as the SFrame parser infers column types. Of course, there are many options you may need to specify when importing a csv file. Some of the more common options come in to play when we load the usage data of users listening to these songs online:"
      ]
    },
    {
      "metadata": {
        "id": "hzAkDYbR3Mk1",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 107
        },
        "outputId": "d682e0eb-56fd-4e3b-eaa4-cdbbee857031"
      },
      "cell_type": "code",
      "source": [
        "usage_data = tc.SFrame.read_csv(\"./10000.txt\",\n",
        "                                header=False,\n",
        "                                delimiter='\\t',\n",
        "                                column_type_hints={'X3':int})"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<pre>Finished parsing file /content/10000.txt</pre>"
            ],
            "text/plain": [
              "Finished parsing file /content/10000.txt"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<pre>Parsing completed. Parsed 100 lines in 1.70624 secs.</pre>"
            ],
            "text/plain": [
              "Parsing completed. Parsed 100 lines in 1.70624 secs."
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<pre>Read 844838 lines. Lines per second: 741447</pre>"
            ],
            "text/plain": [
              "Read 844838 lines. Lines per second: 741447"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<pre>Finished parsing file /content/10000.txt</pre>"
            ],
            "text/plain": [
              "Finished parsing file /content/10000.txt"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<pre>Parsing completed. Parsed 2000000 lines in 1.49596 secs.</pre>"
            ],
            "text/plain": [
              "Parsing completed. Parsed 2000000 lines in 1.49596 secs."
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "1hgMj9Mq3gCC",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 415
        },
        "outputId": "7523fd11-2573-418f-bc5f-8f627583f254"
      },
      "cell_type": "code",
      "source": [
        "usage_data.head()"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\"><table frame=\"box\" rules=\"cols\">\n",
              "    <tr>\n",
              "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">X1</th>\n",
              "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">X2</th>\n",
              "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">X3</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">b80344d063b5ccb3212f76538<br>f3d9e43d87dca9e ...</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOAKIMP12A8C130995</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">b80344d063b5ccb3212f76538<br>f3d9e43d87dca9e ...</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOBBMDR12A8C13253B</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">b80344d063b5ccb3212f76538<br>f3d9e43d87dca9e ...</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOBXHDL12A81C204C0</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">b80344d063b5ccb3212f76538<br>f3d9e43d87dca9e ...</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOBYHAJ12A6701BF1D</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">b80344d063b5ccb3212f76538<br>f3d9e43d87dca9e ...</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SODACBL12A8C13C273</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">b80344d063b5ccb3212f76538<br>f3d9e43d87dca9e ...</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SODDNQT12A6D4F5F7E</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">b80344d063b5ccb3212f76538<br>f3d9e43d87dca9e ...</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SODXRTY12AB0180F3B</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">b80344d063b5ccb3212f76538<br>f3d9e43d87dca9e ...</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOFGUAY12AB017B0A8</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">b80344d063b5ccb3212f76538<br>f3d9e43d87dca9e ...</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOFRQTD12A81C233C0</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">b80344d063b5ccb3212f76538<br>f3d9e43d87dca9e ...</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOHQWYZ12A6D4FA701</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
              "    </tr>\n",
              "</table>\n",
              "[10 rows x 3 columns]<br/>\n",
              "</div>"
            ],
            "text/plain": [
              "Columns:\n",
              "\tX1\tstr\n",
              "\tX2\tstr\n",
              "\tX3\tint\n",
              "\n",
              "Rows: 10\n",
              "\n",
              "Data:\n",
              "+-------------------------------+--------------------+----+\n",
              "|               X1              |         X2         | X3 |\n",
              "+-------------------------------+--------------------+----+\n",
              "| b80344d063b5ccb3212f76538f... | SOAKIMP12A8C130995 | 1  |\n",
              "| b80344d063b5ccb3212f76538f... | SOBBMDR12A8C13253B | 2  |\n",
              "| b80344d063b5ccb3212f76538f... | SOBXHDL12A81C204C0 | 1  |\n",
              "| b80344d063b5ccb3212f76538f... | SOBYHAJ12A6701BF1D | 1  |\n",
              "| b80344d063b5ccb3212f76538f... | SODACBL12A8C13C273 | 1  |\n",
              "| b80344d063b5ccb3212f76538f... | SODDNQT12A6D4F5F7E | 5  |\n",
              "| b80344d063b5ccb3212f76538f... | SODXRTY12AB0180F3B | 1  |\n",
              "| b80344d063b5ccb3212f76538f... | SOFGUAY12AB017B0A8 | 1  |\n",
              "| b80344d063b5ccb3212f76538f... | SOFRQTD12A81C233C0 | 1  |\n",
              "| b80344d063b5ccb3212f76538f... | SOHQWYZ12A6D4FA701 | 1  |\n",
              "+-------------------------------+--------------------+----+\n",
              "[10 rows x 3 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "metadata": {
        "id": "NJzJVDKF3SGO",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "The header and delimiter options are needed because this particular csv file does not provide column names in its first line, and the values are separated by tabs, not commas. The column_type_hints keeps the SFrame csv parser from attempting to infer the datatype of each column, which it does by default. For a full list of options when parsing csv files, check our [API Reference](https://apple.github.io/turicreate/docs/api/generated/turicreate.SFrame.read_csv.html#turicreate.SFrame.read_csv)."
      ]
    },
    {
      "metadata": {
        "id": "-8jl3ffU3k6v",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Here we might want to rename columns from the default names:"
      ]
    },
    {
      "metadata": {
        "id": "Tpx2vEVP3mvw",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 449
        },
        "outputId": "1b4ff3dd-a88c-43db-a809-b6c539b00c4c"
      },
      "cell_type": "code",
      "source": [
        "usage_data.rename({'X1':'user_id', 'X2':'song_id', 'X3':'listen_count'})"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\"><table frame=\"box\" rules=\"cols\">\n",
              "    <tr>\n",
              "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">user_id</th>\n",
              "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">song_id</th>\n",
              "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">listen_count</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">b80344d063b5ccb3212f76538<br>f3d9e43d87dca9e ...</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOAKIMP12A8C130995</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">b80344d063b5ccb3212f76538<br>f3d9e43d87dca9e ...</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOBBMDR12A8C13253B</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">b80344d063b5ccb3212f76538<br>f3d9e43d87dca9e ...</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOBXHDL12A81C204C0</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">b80344d063b5ccb3212f76538<br>f3d9e43d87dca9e ...</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOBYHAJ12A6701BF1D</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">b80344d063b5ccb3212f76538<br>f3d9e43d87dca9e ...</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SODACBL12A8C13C273</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">b80344d063b5ccb3212f76538<br>f3d9e43d87dca9e ...</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SODDNQT12A6D4F5F7E</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">b80344d063b5ccb3212f76538<br>f3d9e43d87dca9e ...</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SODXRTY12AB0180F3B</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">b80344d063b5ccb3212f76538<br>f3d9e43d87dca9e ...</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOFGUAY12AB017B0A8</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">b80344d063b5ccb3212f76538<br>f3d9e43d87dca9e ...</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOFRQTD12A81C233C0</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">b80344d063b5ccb3212f76538<br>f3d9e43d87dca9e ...</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOHQWYZ12A6D4FA701</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
              "    </tr>\n",
              "</table>\n",
              "[2000000 rows x 3 columns]<br/>Note: Only the head of the SFrame is printed.<br/>You can use print_rows(num_rows=m, num_columns=n) to print more rows and columns.\n",
              "</div>"
            ],
            "text/plain": [
              "Columns:\n",
              "\tuser_id\tstr\n",
              "\tsong_id\tstr\n",
              "\tlisten_count\tint\n",
              "\n",
              "Rows: 2000000\n",
              "\n",
              "Data:\n",
              "+-------------------------------+--------------------+--------------+\n",
              "|            user_id            |      song_id       | listen_count |\n",
              "+-------------------------------+--------------------+--------------+\n",
              "| b80344d063b5ccb3212f76538f... | SOAKIMP12A8C130995 |      1       |\n",
              "| b80344d063b5ccb3212f76538f... | SOBBMDR12A8C13253B |      2       |\n",
              "| b80344d063b5ccb3212f76538f... | SOBXHDL12A81C204C0 |      1       |\n",
              "| b80344d063b5ccb3212f76538f... | SOBYHAJ12A6701BF1D |      1       |\n",
              "| b80344d063b5ccb3212f76538f... | SODACBL12A8C13C273 |      1       |\n",
              "| b80344d063b5ccb3212f76538f... | SODDNQT12A6D4F5F7E |      5       |\n",
              "| b80344d063b5ccb3212f76538f... | SODXRTY12AB0180F3B |      1       |\n",
              "| b80344d063b5ccb3212f76538f... | SOFGUAY12AB017B0A8 |      1       |\n",
              "| b80344d063b5ccb3212f76538f... | SOFRQTD12A81C233C0 |      1       |\n",
              "| b80344d063b5ccb3212f76538f... | SOHQWYZ12A6D4FA701 |      1       |\n",
              "+-------------------------------+--------------------+--------------+\n",
              "[2000000 rows x 3 columns]\n",
              "Note: Only the head of the SFrame is printed.\n",
              "You can use print_rows(num_rows=m, num_columns=n) to print more rows and columns."
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "metadata": {
        "id": "ymSWtQwz3pxd",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "SFrames can be saved as a csv file or in the SFrame binary format. If your SFrame is saved in binary format loading it is instantaneous, so we won't ever have to parse that file again. Here, the default is to save in binary format, and we supply the name of a directory to be created which will hold the binary files:"
      ]
    },
    {
      "metadata": {
        "id": "rxJsNce-3ZRL",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "usage_data.save('./music_usage_data.sframe')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "gLcdT8Cv3tfb",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Loading is then very fast:"
      ]
    },
    {
      "metadata": {
        "id": "_FUN-pBF3vTy",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "same_usage_data = tc.load_sframe('./music_usage_data.sframe')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ze0lxR9k3zRA",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Data Types\n",
        "\n",
        "An SFrame is made up of columns of a contiguous type, a number of datatypes are supported:\n",
        "\n",
        "*   int (signed 64-bit integer)\n",
        "*   float (double-precision floating point)\n",
        "*   str (string)\n",
        "*   array.array (1-D array of doubles)\n",
        "*   list (arbitrarily list of elements)\n",
        "*   dict (arbitrary dictionary of elements)\n",
        "*   datetime.datetime (datetime with microsecond precision)\n",
        "*   image (image)"
      ]
    },
    {
      "metadata": {
        "id": "HUYoOHQjCxaZ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Memory Intensive Example\n",
        "\n",
        "https://blog.usejournal.com/python-for-big-data-computation-on-a-single-computer-c232046df3c3\n",
        "\n",
        "The data we will use for our experiment comes from the (now inexistent) Gowalla social networking site. Two data nice data sets coming from this site are available here. We will be looking at the biggest one, which contains the event-log of “check-ins” of Gowalla’s users to a set of locations. This data set contains 6.44 million records, each containing a single check-in and just a few columns, of which we will pick only 3: user_id, location_id and checkin_ts (the second-resolution timestamp of the check-in event).\n",
        "\n",
        "https://snap.stanford.edu/data/loc-gowalla.html"
      ]
    },
    {
      "metadata": {
        "id": "YfHt3wIDEAYS",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "The problem and its (theoretical) solution\n",
        "We will use Turi Create to attack what could be termed the “stalker-stalkee detection problem” on this data set. In this problem, we are asked to identify pairs of users (E, R) that maximize the ‘stalking measure between E and R’. The stalking measure between E and R is defined as the number of distinct locations where there was ever a check-in by user E (the stalkEE) followed by a check-in by user R (the stalkER).\n",
        "\n",
        "The first thing is to index the check-ins by location_id (remember that in pandas a single value for a key can refer to more than one row). This will make the following computation easier.\n",
        "\n",
        "Then comes the tricky part, for each location we want to consider all pairs of check-ins where the check-in time stamp of the first user in the pair strictly precedes that of the second user. So generate chin_pairs, a data frame containing all pairs of check-ins for the same location and then filter it to enforce the conditions just described, to generate pairs_filtered.\n",
        "\n",
        "However, trying to run a naïve Pandas solution on a laptop or PC with the amount of RAM that is usual these days, (say 16GB), will result in a MemoryError exception. With Turi Create and SFrames we do not have such problems."
      ]
    },
    {
      "metadata": {
        "id": "tj8qH8pxEqqg",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 269
        },
        "outputId": "49646044-ff47-41fa-af77-ac1b4c27a0df"
      },
      "cell_type": "code",
      "source": [
        "checkins = ( tc.SFrame.read_csv( 'loc-gowalla_totalCheckins.txt',                  \n",
        "                                 delimiter='\\t', header=False )\n",
        "                .rename( {'X1': 'user_id', 'X2' : 'checkin_ts',\n",
        "                          'X3': 'lat', 'X4' : 'lon',\n",
        "                          'X5': 'location_id'} )\n",
        "  [[\"user_id\", \"location_id\", \"checkin_ts\"]] )"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Finished parsing file /content/loc-gowalla_totalCheckins.txt"
            ],
            "text/html": [
              "<pre>Finished parsing file /content/loc-gowalla_totalCheckins.txt</pre>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Parsing completed. Parsed 100 lines in 1.49398 secs."
            ],
            "text/html": [
              "<pre>Parsing completed. Parsed 100 lines in 1.49398 secs.</pre>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "------------------------------------------------------\n",
            "Inferred types from first 100 line(s) of file as \n",
            "column_type_hints=[int,str,float,float,int]\n",
            "If parsing fails due to incorrect types, you can correct\n",
            "the inferred type list above and pass it to read_csv in\n",
            "the column_type_hints argument\n",
            "------------------------------------------------------\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Read 870755 lines. Lines per second: 228086"
            ],
            "text/html": [
              "<pre>Read 870755 lines. Lines per second: 228086</pre>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Read 2588975 lines. Lines per second: 271390"
            ],
            "text/html": [
              "<pre>Read 2588975 lines. Lines per second: 271390</pre>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Read 4301430 lines. Lines per second: 281586"
            ],
            "text/html": [
              "<pre>Read 4301430 lines. Lines per second: 281586</pre>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Finished parsing file /content/loc-gowalla_totalCheckins.txt"
            ],
            "text/html": [
              "<pre>Finished parsing file /content/loc-gowalla_totalCheckins.txt</pre>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Parsing completed. Parsed 6442892 lines in 19.7667 secs."
            ],
            "text/html": [
              "<pre>Parsing completed. Parsed 6442892 lines in 19.7667 secs.</pre>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "owEASksxFHiK",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 245
        },
        "outputId": "1370e5e6-5833-4ced-a991-4bac8453dc5e"
      },
      "cell_type": "code",
      "source": [
        "checkins.head()"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\"><table frame=\"box\" rules=\"cols\">\n",
              "    <tr>\n",
              "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">user_id</th>\n",
              "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">location_id</th>\n",
              "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">checkin_ts</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">22847</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2010-10-19T23:55:27Z</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">420315</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2010-10-18T22:17:43Z</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">316637</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2010-10-17T23:42:03Z</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">16516</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2010-10-17T19:26:05Z</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5535878</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2010-10-16T18:50:42Z</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">15372</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2010-10-12T23:58:03Z</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">21714</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2010-10-12T22:02:11Z</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">420315</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2010-10-12T19:44:40Z</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">153505</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2010-10-12T15:57:20Z</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">420315</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2010-10-12T15:19:03Z</td>\n",
              "    </tr>\n",
              "</table>\n",
              "[10 rows x 3 columns]<br/>\n",
              "</div>"
            ],
            "text/plain": [
              "Columns:\n",
              "\tuser_id\tint\n",
              "\tlocation_id\tint\n",
              "\tcheckin_ts\tstr\n",
              "\n",
              "Rows: 10\n",
              "\n",
              "Data:\n",
              "+---------+-------------+----------------------+\n",
              "| user_id | location_id |      checkin_ts      |\n",
              "+---------+-------------+----------------------+\n",
              "|    0    |    22847    | 2010-10-19T23:55:27Z |\n",
              "|    0    |    420315   | 2010-10-18T22:17:43Z |\n",
              "|    0    |    316637   | 2010-10-17T23:42:03Z |\n",
              "|    0    |    16516    | 2010-10-17T19:26:05Z |\n",
              "|    0    |   5535878   | 2010-10-16T18:50:42Z |\n",
              "|    0    |    15372    | 2010-10-12T23:58:03Z |\n",
              "|    0    |    21714    | 2010-10-12T22:02:11Z |\n",
              "|    0    |    420315   | 2010-10-12T19:44:40Z |\n",
              "|    0    |    153505   | 2010-10-12T15:57:20Z |\n",
              "|    0    |    420315   | 2010-10-12T15:19:03Z |\n",
              "+---------+-------------+----------------------+\n",
              "[10 rows x 3 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "metadata": {
        "id": "0vdQZ__rFQ8I",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Next, generate the pairs of check-ins that satisfy the conditions of our detection algorithms."
      ]
    },
    {
      "metadata": {
        "id": "nFoEzuYlMQgv",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import datetime\n",
        "import dateutil.parser"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "1wEhpuFVFUJC",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "chin_ps = ( checkins.join(checkins, on='location_id').rename( {'checkin_ts': 'checkin_ts_ee', 'checkin_ts.1': 'checkin_ts_er', 'user_id': 'stalkee' , 'user_id.1': 'stalker' } ) )"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "phw55yidIWg_",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "chin_ps['time_diff'] = (chin_ps['checkin_ts_er'].apply(dateutil.parser.parse) - chin_ps['checkin_ts_ee'].apply(dateutil.parser.parse)) / 86400"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "t9OQPWvlFuIE",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 245
        },
        "outputId": "dbb4b625-e100-487c-d104-66b0f819062c"
      },
      "cell_type": "code",
      "source": [
        "# pairs_filtered = chin_ps[ (chin_ps['checkin_ts_ee'] < chin_ps['checkin_ts_er']) & (chin_ps['stalkee'] != chin_ps['stalker']) ]\n",
        "pairs_filtered = chin_ps[ (chin_ps['time_diff'] > 0.0) & (chin_ps['time_diff'] < 1.0) & (chin_ps['stalkee'] != chin_ps['stalker']) ]\n",
        "pairs_filtered.head()"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\"><table frame=\"box\" rules=\"cols\">\n",
              "    <tr>\n",
              "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">stalkee</th>\n",
              "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">location_id</th>\n",
              "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">checkin_ts_ee</th>\n",
              "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">stalker</th>\n",
              "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">checkin_ts_er</th>\n",
              "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">time_diff</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">7</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">420315</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2010-10-18T20:24:42Z</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2010-10-18T22:17:43Z</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0784837962963</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">7</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">420315</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2010-10-18T15:08:58Z</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2010-10-18T22:17:43Z</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.297743055556</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">31</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">420315</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2010-10-18T14:00:53Z</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2010-10-18T22:17:43Z</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.345023148148</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">66</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">420315</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2010-10-18T18:59:11Z</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2010-10-18T22:17:43Z</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.13787037037</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">327</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">420315</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2010-10-18T21:21:12Z</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2010-10-18T22:17:43Z</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0392476851852</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">327</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">420315</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2010-10-18T14:05:59Z</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2010-10-18T22:17:43Z</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.341481481481</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">342</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">420315</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2010-10-18T14:10:40Z</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2010-10-18T22:17:43Z</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.338229166667</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">350</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">420315</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2010-10-18T19:28:34Z</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2010-10-18T22:17:43Z</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.117465277778</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">456</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">420315</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2010-10-18T16:00:08Z</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2010-10-18T22:17:43Z</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.262210648148</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">515</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">420315</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2010-10-18T11:42:06Z</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2010-10-18T22:17:43Z</td>\n",
              "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.441400462963</td>\n",
              "    </tr>\n",
              "</table>\n",
              "[10 rows x 6 columns]<br/>\n",
              "</div>"
            ],
            "text/plain": [
              "Columns:\n",
              "\tstalkee\tint\n",
              "\tlocation_id\tint\n",
              "\tcheckin_ts_ee\tstr\n",
              "\tstalker\tint\n",
              "\tcheckin_ts_er\tstr\n",
              "\ttime_diff\tfloat\n",
              "\n",
              "Rows: 10\n",
              "\n",
              "Data:\n",
              "+---------+-------------+----------------------+---------+----------------------+\n",
              "| stalkee | location_id |    checkin_ts_ee     | stalker |    checkin_ts_er     |\n",
              "+---------+-------------+----------------------+---------+----------------------+\n",
              "|    7    |    420315   | 2010-10-18T20:24:42Z |    0    | 2010-10-18T22:17:43Z |\n",
              "|    7    |    420315   | 2010-10-18T15:08:58Z |    0    | 2010-10-18T22:17:43Z |\n",
              "|    31   |    420315   | 2010-10-18T14:00:53Z |    0    | 2010-10-18T22:17:43Z |\n",
              "|    66   |    420315   | 2010-10-18T18:59:11Z |    0    | 2010-10-18T22:17:43Z |\n",
              "|   327   |    420315   | 2010-10-18T21:21:12Z |    0    | 2010-10-18T22:17:43Z |\n",
              "|   327   |    420315   | 2010-10-18T14:05:59Z |    0    | 2010-10-18T22:17:43Z |\n",
              "|   342   |    420315   | 2010-10-18T14:10:40Z |    0    | 2010-10-18T22:17:43Z |\n",
              "|   350   |    420315   | 2010-10-18T19:28:34Z |    0    | 2010-10-18T22:17:43Z |\n",
              "|   456   |    420315   | 2010-10-18T16:00:08Z |    0    | 2010-10-18T22:17:43Z |\n",
              "|   515   |    420315   | 2010-10-18T11:42:06Z |    0    | 2010-10-18T22:17:43Z |\n",
              "+---------+-------------+----------------------+---------+----------------------+\n",
              "+-----------------+\n",
              "|    time_diff    |\n",
              "+-----------------+\n",
              "| 0.0784837962963 |\n",
              "|  0.297743055556 |\n",
              "|  0.345023148148 |\n",
              "|  0.13787037037  |\n",
              "| 0.0392476851852 |\n",
              "|  0.341481481481 |\n",
              "|  0.338229166667 |\n",
              "|  0.117465277778 |\n",
              "|  0.262210648148 |\n",
              "|  0.441400462963 |\n",
              "+-----------------+\n",
              "[10 rows x 6 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 11
        }
      ]
    },
    {
      "metadata": {
        "id": "lFli73MVF6S3",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "final_result = ( pairs_filtered[['stalkee', 'stalker', 'location_id']]\n",
        "                    .unique()\n",
        "                    .groupby( ['stalkee', 'stalker'], {\"location_count\": agg.COUNT })\n",
        "                    .topk( 'location_count', k=5 )\n",
        "                    .materialize() )"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "m3fxFqCbF8zt",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "print( final_result )"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "zg8dAd69A0eU",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Articles, Repositories, etc\n",
        "\n",
        "*   https://medium.com/@nilotic2/a-guide-to-turi-create-a72f53f26721\n",
        "*   https://blog.usejournal.com/python-for-big-data-computation-on-a-single-computer-c232046df3c3\n",
        "*   https://github.com/onmyway133/Avengers"
      ]
    }
  ]
}