code/cmd-optimze.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# User Interaction Model for CMD Structure Optimization"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "## Model Description\n",
    "### Summary\n",
    "The purpose of this study is to create a simple User Interaction Model to evaluate alternative approaches to optimizing a user interface.\n",
    "### Model\n",
    "$T = M(S, T_C, T_M, C_A, C_G)$\n",
    "\n",
    "Where $S$ is a command execution scenario expressed as a vector of \n",
    "commands, $T_C$ is the time required to switch between commands of the same group, $T_M$ is the time required to switch between commands of different groups, $C_A$ is a set of commands always available at the desktop, $C_G$ is a set of command groups, and T is the time required to execute $S$.\n",
    "\n",
    "All approaches can be expressed by varying $C_A$ and $C_G$. You need to define these for each approach. A user's expertise level is expressed through $T_C$ and $T_M$.\n",
    "\n",
    "### Calculating the Time Needed\n",
    "In this simplified version of the User Interaction Model, we assert that only 2 weights are needed:\n",
    "* Same group: $T_s$ = command1 to command2\n",
    "* Different group: $T_d$ = command1 change-group command2\n",
    "\n",
    "### Approached to be evaluated\n",
    "* <b>ALL</b> Put all commands on the screen (actually possible with new extra wide LCD screens). This establishes the theoretical optimum.\n",
    "* <b>NAIVE</b> Naive: make the most frequently executed commands always available.\n",
    "* <b>GROUP</b> Group heuristics: group together the most used common commands (e.g. delete and move) by doing away the class-verb method for selecting them.\n",
    "* <b>MRU-B</b> User optimized, based on the most recently used (MRU) commands of a particular user executes more frequently during a training period (batch).\n",
    "* <b>MRU-O</b> User optimized, based on the most recently used (MRU) commands of a particular user executes more frequently during continuously adjusted (online).\n",
    "* <b>OPT</b> Optimized: run a (stochastic) optimization algorithm to select those commands whose combination yields the fastest user interaction time. This differs from NAIVE in that it takes into account the actual switching between entities.\n",
    "* <b>CLUSTER</b> Clustered, based on 2-3 clusters of users. The clustering is performed based on the commands they execute. A stochastic optimization similar to OPT is ran to select the commands executed in each cluster.\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Assumptions\n",
    "1. We assume that cognitive processes do not affect the time taken to complete a move or a scenario.\n",
    "2. We assume that all moves need equal time to be carried on, regardless their entity, type or any other characteristic. That time differs only by the expertise of the user.\n",
    "3. We assume that a scenario will always need the same time to be completed, if the experiment remains the same."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Import\n",
    "### Experiment Data and Weights\n",
    "\n",
    "The file format is: init_time, click-on-group_name/click-on-command_group-name_command-name\n",
    "\n",
    "1497807650.013000 en_objects <br>\n",
    "1497807651.223000 vm_object_add <br>\n",
    "1497807655.943000 vm_object_get_attributes <br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "import csv\n",
    "import operator\n",
    "import random\n",
    "\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "\n",
    "from scipy import stats\n",
    "from sklearn import datasets, linear_model\n",
    "from sklearn.metrics import mean_squared_error, r2_score\n",
    "from sklearn.cluster import KMeans\n",
    "from heapq import nlargest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'a': {'s': 5.3476588221157302, 'c': 11.093731189286837}, 'e': {'s': 4.2311320709732341, 'c': 7.300849064341131}}\n"
     ]
    }
   ],
   "source": [
    "DATA_FOLDER = \"\" # main data folder\n",
    "\n",
    "AMATEURS = [\"2017-06-18_20-46\", \"2017-06-19_11-42\",\n",
    "            \"2017-06-19_12-07\", \"2017-06-19_12-18\",\n",
    "            \"2017-06-19_12-46\", \"2017-06-19_13-04\",\n",
    "            \"2017-06-19_13-45\"]\n",
    "\n",
    "EXPERTS = [\"2017-06-19_18-02\", \"2017-06-19_18-12\",\n",
    "           \"2017-06-19_18-20\", \"2017-06-19_19-30\"]\n",
    "\n",
    "\n",
    "def cleanse(filename):\n",
    "    '''\n",
    "    Get a list of useful data from an experiment file.\n",
    "    '''\n",
    "    moves = []\n",
    "    with open(filename) as test_input:\n",
    "        for line in test_input:\n",
    "            move = line.split(\"\\n\")[0]\n",
    "            moves.append([\n",
    "                float(move.split(\" \")[0]),  # Time\n",
    "                move.split(\" \")[1].split(\"_\")[0]  # Action Type\n",
    "            ])\n",
    "\n",
    "    return moves\n",
    "\n",
    "\n",
    "def count_time(moves):\n",
    "    '''\n",
    "    Calculate the time needed for every command clicked in an\n",
    "    experiment data list.\n",
    "    '''\n",
    "    action_times = []\n",
    "    starting_time = float(moves.pop(0)[0])\n",
    "\n",
    "    command_type = 0  # type 0 means the action involved changing the en\n",
    "                      # type 1 means the action remained in the same en\n",
    "    for move in moves:\n",
    "        if move[1] == \"vm\":\n",
    "\n",
    "            action_times.append([\n",
    "                move[0] - starting_time,  # time taken to since last command\n",
    "                command_type                      # type of command\n",
    "            ])\n",
    "\n",
    "            starting_time = move[0]\n",
    "            command_type = 1\n",
    "        else:\n",
    "            command_type = 0\n",
    "\n",
    "    return action_times\n",
    "\n",
    "\n",
    "def input_all():\n",
    "    '''\n",
    "    Import all instances, diversified in amateur and expert users.\n",
    "    '''\n",
    "    amateur = [np.array(count_time(cleanse(DATA_FOLDER + filename))) \n",
    "            for filename in AMATEURS]\n",
    "    expert = [np.array(count_time(cleanse(DATA_FOLDER + filename)))\n",
    "            for filename in EXPERTS]\n",
    "    \n",
    "    return amateur, expert\n",
    "\n",
    "def get_avarages(amateur, expert):\n",
    "    '''\n",
    "    Get the weights for the model.\n",
    "    '''\n",
    "    a_s, a_c, e_s, e_c = [],[],[],[]\n",
    "    \n",
    "    for case in amateur:\n",
    "        a_s += [time[0] for time in case if time[1] == 1]\n",
    "        a_c += [time[0] for time in case if time[1] == 0]\n",
    "        \n",
    "    for case in expert:\n",
    "        e_s += [time[0] for time in case if time[1] == 1]\n",
    "        e_c += [time[0] for time in case if time[1] == 0]\n",
    "    \n",
    "    weight = {}\n",
    "    weight['a'] = {'s': stats.trim_mean(a_s, 0.1),\n",
    "                  'c': stats.trim_mean(a_c, 0.1)}\n",
    "    weight['e'] = {'s': stats.trim_mean(e_s, 0.1),\n",
    "                  'c': stats.trim_mean(e_c, 0.1)}\n",
    "    \n",
    "    return weight\n",
    "\n",
    "amateur_times, expert_times = input_all()\n",
    "weight = get_avarages(amateur_times, expert_times)\n",
    "print(weight)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Real Scenarios for Valuation\n",
    "Import scenarios from real data to valuate the different approaches. The file format is:<br>\n",
    "session-id, user-id, initiation-time(starting from 0), command-group, command-id, command-action:\n",
    "\n",
    "102:94:393:41:280:<br>\n",
    "103:95:0:8:27:<br>\n",
    "103:95:8:8:175:<br>\n",
    "103:95:51:8:69:5<br>\n",
    "103:95:72:8:70:4<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['8', '27', -1], ['8', '28', -1], ['8', '28', -1], ['8', '27', -1], ['8', '29', '2'], ['10', '30', -1], ['10', '31', -1]]\n"
     ]
    }
   ],
   "source": [
    "DATA_FILE = \"\"\n",
    "\n",
    "realdata_file = open(DATA_FILE, \"r\")\n",
    "\n",
    "def get_scenarios(data):\n",
    "    '''\n",
    "    Get real scenarios and their real execution time.\n",
    "    '''\n",
    "    # Initialize Variables\n",
    "    S = []  # Scenarios\n",
    "    T = []  # Times\n",
    "    U = []  # Users\n",
    "    session = '1'\n",
    "    scenario = []\n",
    "    time_temp = -1\n",
    "    user_temp = -1\n",
    "\n",
    "    for line in data:\n",
    "        line = line.strip('\\n')\n",
    "        line = line.split(':')\n",
    "        \n",
    "        # New Session\n",
    "        if session != line[1]:\n",
    "\n",
    "            S.append(scenario)\n",
    "            scenario = []\n",
    "            session = line[1]\n",
    "            T.append(time_temp)\n",
    "            U.append(user_temp)\n",
    "        \n",
    "        # Group\n",
    "        group = line[5]\n",
    "        if (group == ''):\n",
    "            group = -1\n",
    "        \n",
    "        scenario.append([line[3], line[4], group])\n",
    "        time_temp = line[2]\n",
    "        user_temp = line[1]\n",
    "    \n",
    "    return S, T, U\n",
    "\n",
    "S, T, U = get_scenarios(realdata_file)\n",
    "scenarios = len(S)\n",
    "T = [int(time) for time in T]\n",
    "print(S[1])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Data Segmentation\n",
    "All: 50% training, 50% evaluation.\n",
    "\n",
    "Short: 1% training, 50% evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ALL:  91136\n",
      "Short: 1822\n"
     ]
    }
   ],
   "source": [
    "ALL = int(scenarios/2)\n",
    "SHORT = int(scenarios/100)\n",
    "\n",
    "print('ALL: ', ALL)\n",
    "print('Short:', SHORT)\n",
    "\n",
    "# Guide:\n",
    "# S[:ALL] all for training\n",
    "# S[ALL:] all for evaluation\n",
    "#\n",
    "# S[:SHORT] short for training\n",
    "# S[ALL:] short for evaluation"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Writing Results to CSV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "NAME = '../scoreboard.csv'\n",
    "METHOD = ['', '', 'ALL', 'NAIVE', 'GROUP', 'MRUB', 'MRUO',\n",
    "         'OPT(KS)', 'OPT(GA)', 'CLUSTER', 'time']\n",
    "\n",
    "def p(score,s,t):\n",
    "    '''\n",
    "    Get percentages evaluates with the correct CURRENT.\n",
    "    '''\n",
    "    return str(round((float(c_score[t]) - float(score)) / \\\n",
    "                     float(c_score[t]) * 100,2)) + '%'\n",
    "\n",
    "def rt(score):\n",
    "    '''\n",
    "    Calculate the relative time score for execution time.\n",
    "    '''\n",
    "    total = n_time + g_time + mrub_time + mruo_time + opt_time \\\n",
    "            + ga_time + cl_time\n",
    "    return str(round(score/total*100,2)) + '%'\n",
    "\n",
    "def initialize_csv():\n",
    "    \n",
    "    Results = []\n",
    "    Results.append(['182273 scenarios', '', '', '', '1822 scenarios', '', '', ''])\n",
    "    Results.append(['Method', 'Amateurs', 'Experts', 'Training Time', 'Amateurs',\n",
    "                    'Experts', 'Time', 'Relative Time'])\n",
    "    \n",
    "    Results.append(['ALL', '-', '-', '-', '-', '-', '-', '-']) #2\n",
    "    Results.append(['NAIVE', '-', '-', '-', '-', '-', '-', '-']) #3\n",
    "    Results.append(['GROUP', '-', '-', '-', '-', '-', '-', '-']) #4\n",
    "    Results.append(['MRUB', '-', '-', '-', '-', '-', '-', '-']) #5\n",
    "    Results.append(['MRUO', '-', '-', '-', '-', '-', '-', '-']) #6\n",
    "    Results.append(['OPT(KS)', '-', '-', '-', '-', '-', '-', '-']) #7\n",
    "    Results.append(['OPT(GA)', '-', '-', '-', '-', '-', '-', '-']) #8\n",
    "    Results.append(['CLUSTER', '-', '-', '-', '-', '-', '-', '-']) #9\n",
    "    \n",
    "    with open(NAME, 'w', newline='') as csvfile:\n",
    "        w = csv.writer(csvfile)\n",
    "        for row in Results:\n",
    "            w.writerow(row)\n",
    "    print(NAME + ' initialized')\n",
    "\n",
    "def write_csv(value, meth, exp, size):\n",
    "    \n",
    "    # Input\n",
    "    Results = []\n",
    "    with open(NAME, 'r') as csvfile:\n",
    "        r = csv.reader(csvfile)\n",
    "        for row in r:\n",
    "            Results.append(row)\n",
    "    \n",
    "    # Edit\n",
    "    if (exp == 'a'): ex = 1 \n",
    "    else: ex = 2\n",
    "    if (size == 'a'): size = 0 \n",
    "    else: size = 3\n",
    "    pos = ex + size\n",
    "    meth = METHOD.index(meth)\n",
    "    \n",
    "    if (exp == 'time'):\n",
    "        pos = 3 + size\n",
    "        Results[meth][pos] = value\n",
    "    else:\n",
    "        Results[meth][pos] = p(value, 'a', exp) \n",
    "    \n",
    "    # Output\n",
    "    with open(NAME, 'w', newline='') as csvfile:\n",
    "        w = csv.writer(csvfile)\n",
    "        for row in Results:\n",
    "            w.writerow(row)\n",
    "\n",
    "#initialize_csv()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate Different Approaches"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 0. CURRENT\n",
    "Calculation of the actual time of command execution with the current setup of the application. The results will differ from those from the real data, as \n",
    "1. the real data include time spent on designing and thinking for the way a command is used\n",
    "2. the weights used here make the assumption that the time of thinking of moving from command to command is the same for all combinations.\n",
    "\n",
    "Therefore, the $s$ weight is used whenever the command used belongs to the same entity as the previous one and the $c$ when otherwise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Amateurs: 19.56 minutes per scenario\n",
      "Experts: 14.54 minutes per scenario\n"
     ]
    }
   ],
   "source": [
    "def get_weight_CURRENT(expertise, prv, cur):\n",
    "    '''\n",
    "    @param expertise: a for amateurs and e for experts\n",
    "    @param prv: previous command group and id\n",
    "    @param cur: current command group and id\n",
    "    The s weight is used whenever the command used belongs to the \n",
    "    same entity as the previous one and the c when otherwise.\n",
    "    ''' \n",
    "    condition = 'c'\n",
    "    if (prv[0] == cur[0]):\n",
    "        condition = 's'\n",
    "    return weight[expertise][condition]\n",
    "\n",
    "\n",
    "def get_score_CURRENT(S):\n",
    "    '''\n",
    "    For every command, add the appropriate weight to the total score.\n",
    "    '''\n",
    "    a_score = 0 # amateur score\n",
    "    e_score = 0 # expert score\n",
    "    prev_com = ['-1', '-1']\n",
    "    \n",
    "    for scenario in S:\n",
    "        for cur_com in scenario:\n",
    "            a_score += get_weight_CURRENT('a', prev_com, cur_com)\n",
    "            e_score += get_weight_CURRENT('e', prev_com, cur_com)\n",
    "            prev_com = cur_com\n",
    "        prev_com = ['-1', '-1']\n",
    "        \n",
    "    global c_score\n",
    "    c_score = {'a': round(a_score/60/scenarios,2),\n",
    "           'e': round(e_score/60/scenarios,2)}\n",
    "    \n",
    "    score = 'Amateurs: ' + str(c_score['a'])\\\n",
    "            + ' minutes per scenario' + '\\n' + \\\n",
    "            'Experts: ' + str(c_score['e'])\\\n",
    "            + ' minutes per scenario'\n",
    "    \n",
    "    return score\n",
    "\n",
    "\n",
    "print(get_score_CURRENT(S))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Evaulation of CURRENT\n",
    "To find out how these predictions differ from the real data, we plot the actual and predicted data, using the weight for expert users. Those who produces the real data were experts as well:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_predictions_CURRENT(S):\n",
    "    '''\n",
    "    For every command, add the appropriate weight to the total score.\n",
    "    '''\n",
    "    global predictions\n",
    "    \n",
    "    predictions = []\n",
    "    prev_com = ['-1', '-1']\n",
    "    \n",
    "    for scenario in S:\n",
    "        prediction = 0\n",
    "        for cur_com in scenario:\n",
    "            prediction += get_weight_CURRENT('e', prev_com, cur_com)\n",
    "            prev_com = cur_com\n",
    "        prev_com = ['-1', '-1']\n",
    "        predictions.append(prediction/60)\n",
    "                               \n",
    "    return predictions\n",
    "\n",
    "def get_comparison_CURRENT(T, P):\n",
    "    '''\n",
    "    Plots CURRENT predictions and real execution time.\n",
    "    '''\n",
    "    TvsP = str(round(sum(T)/sum(P),2))\n",
    "    plt.plot(T, 'b+')\n",
    "    plt.plot(P, 'g*')\n",
    "    plt.ylim([0,300])\n",
    "    plt.ylabel('Execution time (minutes)')\n",
    "    plt.xlabel('Scenario ID')\n",
    "    plt.title('Real execution time vs CURRENT\\n'\\\n",
    "             +' Real time is '+ TvsP\\\n",
    "             +' times bigger')\n",
    "    plt.legend(['Real Numbers', 'Predictions'])\n",
    "    plt.show()\n",
    "\n",
    "def CURRENT_REAL_regression(X, Y):\n",
    "    \n",
    "    X = np.array(X).reshape(-1, 1)\n",
    "    X_train = X[:-int(len(X)*0.9)]\n",
    "    X_test = X[-int(len(X)*0.9):]\n",
    "    \n",
    "    Y = np.array(Y).reshape(-1, 1)\n",
    "    Y_train = Y[:-int(len(Y)*0.9)]\n",
    "    Y_test = Y[-int(len(Y)*0.9):]\n",
    "    \n",
    "    regr = linear_model.LinearRegression()\n",
    "    regr.fit(X_train,Y_train)\n",
    "    print('Coefficient: %.2f times' % regr.coef_)\n",
    "    print('Intercept: %.2f seconds' % regr.intercept_)\n",
    "    #print('Mean squared error: %.2f'\n",
    "    #     % mean_squared_error(Y_test, regr.predict(X_test)))\n",
    "    print('Variance score: %.2f'\n",
    "         % r2_score(Y_test, regr.predict(X_test)))\n",
    "    "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. ALL\n",
    "Put all commands on the screen (actually possible with new extra wide LCD screens). This establishes the theoretical optimum.\n",
    "\n",
    "If every command is on the screen, we assert that the time needed for each one will be the same and equal to the trimmed mean for commands in the same group ($A_S, E_S$)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 0.85\n",
      "Amateurs: 1574.7 minutes per scenario, -7950.6% less than CURRENT\n",
      "Experts: 1245.92 minutes per scenario, -8468.91% less than CURRENT\n"
     ]
    }
   ],
   "source": [
    "def get_weight_ALL(expertise):\n",
    "    '''\n",
    "    @param expertise: a for amateurs and e for experts\n",
    "    All commands have the same, non changing weight, as they are\n",
    "    all available on the screen.\n",
    "    '''\n",
    "    return weight[expertise]['s']\n",
    "\n",
    "def get_score_ALL(S):\n",
    "    '''\n",
    "    All are the same, so we count how many commands are used \n",
    "    in the scenario and multiply with the appropriate *s* weight.\n",
    "    '''\n",
    "    global a_score\n",
    "    a_score = {}\n",
    "    a_score['a'] = 0 # amateur score\n",
    "    a_score['e'] = 0 # expert score\n",
    "    \n",
    "    for scenario in S:\n",
    "        a_score['a'] += len(scenario) * get_weight_ALL('a')\n",
    "        a_score['e'] += len(scenario) * get_weight_ALL('e')\n",
    "    \n",
    "    for t in ['a', 'e']:\n",
    "        a_score[t] = a_score[t] / SHORT / 60\n",
    "\n",
    "a_time = time.time()\n",
    "\n",
    "get_score_ALL(S[SHORT:]) #All has no training, so we just evaluate with ALL:\n",
    "\n",
    "a_time = round(time.time()-a_time,2)\n",
    "print('time:',a_time)\n",
    "\n",
    "# Comparison\n",
    "\n",
    "print('Amateurs: ' + str(round(a_score['a'],2)) + ' minutes per scenario, ' \\\n",
    "     + str(round((c_score['a']-float(a_score['a']))/c_score['a']*100,2)) + '% less than CURRENT'\\\n",
    "     + '\\nExperts: ' +  str(round(a_score['e'],2)) + ' minutes per scenario, '\\\n",
    "     + str(round((c_score['e']-float(a_score['e']))/c_score['e']*100,2)) + '% less than CURRENT')\n",
    "\n",
    "write_csv(value=a_score['a'], exp='a', meth='ALL', size='a')\n",
    "write_csv(value=a_score['e'], exp='e', meth='ALL', size='a')\n",
    "write_csv(value=a_score['a'], exp='a', meth='ALL', size='s')\n",
    "write_csv(value=a_score['e'], exp='e', meth='ALL', size='s')\n",
    "write_csv(value='-', exp='time', meth='ALL', size='a')\n",
    "write_csv(value='-', exp='time', meth='ALL', size='s')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. NAIVE\n",
    "Make the most frequently executed commands always available.\n",
    "\n",
    "First, a list of the most frequently used commands is calculated. Then,\n",
    "If a command is inside this list, or the group of its previous command\n",
    "is the same, the $s$ weight will be used. Otherwise, the $c$\n",
    "weight will be used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xm4HGWZ/vHvTQIkQMKSBNkJIIssGiHgwhZQgUEEZwQVBdkEZPgpriCMGnBgZAdHZQQEEgFRkF1BNiHgAtkA2QUR2UlYE3YCz++P921S6Zw+p87hVB3SfX+uq6+uvZ7qqq6n6q2qtxQRmJlZ51pooAMwM7OB5URgZtbhnAjMzDqcE4GZWYdzIjAz63BOBGZmHc6JwPqdpLskjatpXi9KWr2OeXUTwwGSnsqxjBjIWLpT53rpb5JC0nsHOo525UTQRNJDkmZIWrzQ7cuSbmgaTpIelHR3F9O4IY+zoqQ5ktboYpiLJR2fm0PSS3lH0vgcXMHi9TtJEyQdWewWEetFxA0VzOsGSV9umtcSEfFgf8+rFzEtDJwIbJNjeWagYulJVevFEknjJD060HH0hRNB1wYBB/UwzBbAssDqkjbuaoCIeAy4Dti92F3SMsD2wMRC5w/kHUnjc2yfo7c6vQcYAtzV2xHzwYT/g21A0uCBjuEdiQh/Ch/gIeC7wLPAUrnbl4EbmoY7EzgXuAj4aVO/G4Av5+YvAP9o6v+fwK2F9gDeWzK+RYHjgYeBp4CfA0Nzv0OAW4DBuf0A0g5qSG7/MPAX4HngdmBcYbrLAGcBjwPPAZfk7nsCf2qKIYD3AvsBbwCvAy8Clxd+w48X4j05T/fx3Lxo7jcOeBT4FjADeALYq8VyHwW8Cbya5/XT5t8OmACcAlyZh/kzsFye53PAvcAHC9NcAbgQmAn8E/haod8mwFRgVv6dT+wiprWAl3IMLwJ/zN0/CkwBXsjfH23aNo7Ksb3S1XrPv9+3gb/lafymsQ5z/32BB0jb6GXACrn7/wHHN03rUuCbXayXQcBhwD+A2cA0YOXcbx3gmjz9+4DPFqa3PXB3Hucx4Nst1tcawB+BZ4CnSf+VpXqxjN/J28PjwN508x8BlgTOyMM/BhwJDCr8JhcWhj2GdHAm5m5/h+UYHwK+WPK/1hj3EOBJ4IK8Pt/K28KLpO2rx+3o3fAZ8ADebZ/Gn4W0gz8yd5snEQCL5RW7PfCZvBEtUuh/A3MTwdC8oW9W6P9X4OuF9t4kgpNIf/5lgGHA5cCPcr+FgBuBw4E1STu/D+Z+K+Y/5fZ5uE/k9lG5/+/zn3FpYGFgy9x9T1okgtw8ofE7Nf+GufmHwM2ks6dRpET037nfOGBOHmbhHNvLwNItlv3t37WbWJ4GNiIdpf+RtIP/EmnHdyRwfeG3mgb8AFgEWB14ENi2sI52z81LAB9uEdPoHEMj+S6Tf/fdgcHArrl9RGEZHgbWy/0XbrENTibtSJYB7gG+kvttnZdxQ9KO6ifAjbnfFsAjgHL70qSd0wpdrJfvAHcAa5N2ih8ARgCL52nsleP7YJ7funm8J4DNC9PfsMXv8l7SNrZoXu83AieXXMbtSDvN9XM8v6L7RHAxcGoedtk83f0L/9W/k7bjzfOyrNS0/Z2Y49ySlNjXLvFfa4x7TB53aO72aFNspbajgf4MeADvtg9zE8H6pB34KOZPBLuRjiIHk3Y4LwD/Xuh/A4UdFvAL4LTcvCbpCHrZQv8gJZbnC59tu4hNeUNdo9DtI8A/C+2jSUdy9wCHFrofApzdNL2rgD2A5UlHMvPtgHnnieAfwPaFftsCD+XmcaQd1eBC/xmt/izNv2uLWE4v9PsqcE+hfQPg+dz8IeDhpmkdCpyVm28EjgBG9rC9jGbeRLA7MLlpmL8CexaW4YcltsHdCu3HAj/PzWcAxxb6LUE6Kxudt4+HgS1yv33JZyldrJf7gJ26mPfngJuaup0KjM/NDwP7A8N7+b/6NPOeBXe3jGcCRxf6rUWLREAqmnuNfKSeu+1KTviFdf0s8C9g10L3caSd+eKFbucD36eH/1oe93XmPYsZx/yJoNR2NNAfl0+2EBF3Ar8jFRM12wM4PyLmRMSrpOKFPbqZ3ERgF0lDSDuKqyJiRtMwG0bEUoXPVV1MZxTpCGeapOclPQ/8IXdvxP0QcD1px/Czwrir5hieL4y7GSkJrAw8GxHPdbMMfbUC6Q/Y8K/creGZiJhTaH+ZtHPrq6cKza900d6Y9qrACk2/x2GkHQvAPqQd0L2SpkjaoeT8m5eX3L5iof2REtN5stBc/E3mmX5EvEg6s1sx0p7n16QdIaRiyXNbTH9lUpJutirwoabf5YukIjZIZ8DbA/+SNEnSR7qauKT3SPq1pMckzQLOAUb2YhmLv1Hz79kc78LAE4V4TyWdGQAQEbeQzvZE2tEXPRcRLzXNawVK/NeAmfn/352+bke1ciLo3njSUdXbf2JJK5FOz3eT9KSkJ4Gdge0lNW/oDX8iHZHsRDqbmNhiuJ48TdqZrVdIGEtGxNs7TkmfJB25XAccVxj3EdIZQTHZLB4RR+d+y0haqot5vkT6QzSmv1xT/+gh5sdJf9aGVXK3vuhpXr3xCOnorvh7DIuI7QEi4v6I2JW0QzkG+G3xTrJuNC8vpGV+rND+TpZjnunnmEYUpn8esLOkVUlHwhe2mM4jpHL8rrpPavpdloiIAwAiYkpE7ET6XS5h/h1rw/+QlnODiBhO2u5VchmfICWqhlW6GfYR0hnByEK8wyNivcYAkg4kFd88DjTfjbd003ptbJ89/teYfz3Ot17fwXZUKyeCbkTEA6Ry868VOu9OKnNcGxiTP2uRLhzt2jyNPJ0AfknaEJYilTX2JZ63gNOBkyQtC5BvUd02N48kFUN9mXSG8ilJ2+fRz8nt20oaJGlIvt1tpYh4gnSB9RRJS0taWNIWebzbgfUkjclnNIc3hfUUqXy9lfOA70kaleP7QY6lL3qaV29MBmZLOkTS0PybrN+4A0zSbpJG5d/8+TzOWyWmewWwlqQvSBos6XPAuqSzy/5wHrBXXh+Lkna4t+QzQSLiVtJO7BekM8/nW0znF8B/S1oz3730/vwMxO9y/Lvn7WBhSRtLep+kRSR9UdKSEfEGqTiz1W8yjHTB9AVJK5KuSZR1PrCnpHUlLUY6IOtS3navBk6QNFzSQpLWkLQlgKS1SNeGdiP9dw+WNKZpMkfkZdsc2AG4oKf/WgtPASMkLdno8A62o1o5EfTsh6SLUA17AKdExJPFD+mOgu6Kh35JOtr4TUS81kX/25ueIzi5xXQOId0xcnM+5b6WlJQATgMujYgrIt3Pvg/wC0kjIuIR0hnJYaTrG4+Q/pyNbWB3UlnzvaRy+q8DRMTf829wLXA/6eym6Axg3Xz6fEkX8R5Jumvib6SLk9Nzt774Melo9zlJ/9vHaQAQEW+S/vRjSBeUGzvPxp94O+AuSS/m+X4+Il4pMd1n8nS/RSqyORjYISKefifxFqZ/LakM+0LSkfMawOebBvsV6TrXr7qZ1ImkHe7VpB36GaRy9tnANnmaj5OKbxoXRCFtJw/lbe8rpGKjrhxBuqD9AulGhIt6sYxXku70+iNpW/9jD6N8iXTB/27ShfnfAsvnWzrPAY6JiNsj4n7S9n92TqLk5XsuL+u5pAvW9+Z+3f3Xuor7XlKifjD/H1agj9tR3Rp3F5iZdRSlp6zPiYiVBjqWgeYzAjOzDldZIpC0sqTrJd2tVMfJQbn7MpKukXR//l66qhjMzKxnlRUNSVoeWD4ipksaRnp459Ok+9KfjYijJX2XdO/6IZUEYWZmParsjCAinoiI6bl5NukBpxVJFywbt09OJCUHMzMbILVcLJY0mvSE3fqkpzmXyt1FeqBjvvvXJe1HqsuGxRdffKN11lmn8jjNzNrJtGnTno6IUT0NV3kikLQEMAk4KiIukvR8cccv6bmI6PY6wdixY2Pq1KmVxmlm1m4kTYuIsT0NV+ldQ0p1tV8InBsRjfuIn8rXDxrXEZqrWjAzsxpVedeQSA+p3BMRJxZ6XcbcB6/2IFWTa2ZmA6TKlylsSnoK8Q5Jt+VuhwFHA+dL2odUwdNnK4zBzMx6UFkiiIg/0bqSqY9VNV8zM+sdP1lsZtbhnAjMzDqcE4GZWYdzIjAz63BV3jW0wDjiiCMqnf748S3fq2FmNuB8RmBm1uGcCMzMOpwTgZlZh3MiMDPrcE4EZmYdrse7hiSNBTYHVgBeAe4EromI5yqOzczMatDyjEDSXpKmA4cCQ4H7SFVGbwZcK2mipFXqCdPMzKrS3RnBYsCmEfFKVz0ljQHWBB6uIjAzM6tHy0QQET/rbsSIuK27/mZmtmDo8WKxpJUkXSxppqQZki6UtFIdwZmZWfXK3DV0FumtYsuTLhhfnruZmVkbKJMIRkXEWRExJ38mAKMqjsvMzGpSJhE8I2k3SYPyZzfgmaoDMzOzepRJBHuT3iv8JPAEsDOwV5VBmZlZfXp8oCwi/gXsWEMsZmY2AFomAkkHR8Sxkn4CRHP/iPhapZGZmVktujsjuCd/T60jEDMzGxjdPVB2uaRBwAYR8e0aYzIzsxp1e7E4It4ENq0pFjMzGwBl3ll8m6TLgAuAlxodI+KiyqIyM7PalEkEQ0jPDWxd6BaAE4GZWRsoc/uonxkwM2tjZV5MMwTYB1iPdHYAQETsXWFcZmZWkzJPFp8NLAdsC0wCVgJmVxmUmZnVp0wieG9EfB94KSImAp8EPlRtWGZmVpcyieCN/P28pPWBJYFlqwvJzMzqVOauodMkLQ18n/RegiWAH1QalZmZ1abMXUO/yI2TgNWrDcfMzOpW5q6hRYHPAKOLw0fED6sLy8zM6lKmaOhS4AVgGvBateGYmVndyiSClSJiu95OWNKZwA7AjIhYP3cbA/yc9DzCHOA/I2Jyb6dt78wRRxxR6fTHjx9f6fTNrH+VuWvoL5I26MO0JwDNCeRY4IiIGEO64HxsH6ZrZmb9qMwZwWbAnpL+SSoaEhAR8f7uRoqIGyWNbu4MDM/NSwKP9ypaMzPrd2USwb/14/y+Dlwl6XjS2chHWw0oaT9gP4BVVlmlH0MwM7OiHouG8juLVwa2zs0vlxmvhQOAb0TEysA3gDO6me9pETE2IsaOGjWqj7MzM7Oe9LhDlzQeOAQ4NHdaGDinj/Pbg7nVV18AbNLH6ZiZWT8pc2T/78CO5JfSRMTjwLA+zu9xYMvcvDVwfx+nY2Zm/aTMNYLXIyIkBYCkxctMWNJ5wDhgpKRHgfHAvsCPJQ0GXiVfAzAzs4FTJhGcL+lUYClJ+wJ7A6f3NFJE7Nqi10a9iM/MzCpWpq6h4yV9ApgFrA38ICKuqTwyMzOrRZm6hlYDbmrs/CUNlTQ6Ih6qOjgzM6temaKhC5j3fv83c7eNK4nIrB+5Og2znpW5a2hwRLzeaMnNi1QXkpmZ1alMIpgpacdGi6SdgKerC8nMzOpUpmjoK8C5kn5KqmfoEeBLlUZlZma1KXPX0D+AD0taIre/WHlUZmZWmzJVTBwkaTjpyeKTJU2XtE31oZmZWR3KXCPYOyJmAdsAI4DdgaMrjcrMzGpTJhEof28P/DIi7ip0MzOzBVyZRDBN0tWkRHCVpGHAW9WGZWZmdSlz19A+wBjgwYh4WdIIYK9qwzIzs7qUuWvoLWB6of0Z4JkqgzIzs/r09U1jZmbWJpwIzMw6XLeJQNIgSffWFYyZmdWv20QQEW8C90lapaZ4zMysZmXuGloauEvSZPJ7iwEiYsfWo5jZQOi0arc7bXmrUiYRfL/yKMzMbMCUuX10kqRVgTUj4lpJiwGDqg/NzMzqUKbSuX2B3wKn5k4rApdUGZSZmdWnzO2jBwKbkl5eT0TcDyxbZVBmZlafMongteKrKiUNBqK6kMzMrE5lEsEkSYcBQyV9gvTi+surDcvMzOpSJhF8F5gJ3AHsD1wBfK/KoMzMrD6lKp2TNBG4hVQkdF9EuGjIzKxN9JgIJH0S+DnwD9ILaVaTtH9EXFl1cGZmVr0yD5SdAGwVEQ8ASFoD+D3gRGBm1gbKXCOY3UgC2YPA7IriMTOzmpU5I5gq6QrgfNI1gl2AKZL+AyAiLqowPjMzq1iZRDAEeArYMrfPBIYCnyIlBicCM7MFWJm7hvx+YjOzNuY3lJmZdTgnAjOzDldZIpB0pqQZku5s6v5VSfdKukvSsVXN38zMyilTDfVBkoYrOUPSdEnblJj2BGC7pmltBewEfCAi1gOO70vQZmbWf8qcEewdEbOAbUivrdwdOLqnkSLiRuDZps4HAEdHxGt5mBm9C9fMzPpbmUSg/L09cHZE3FXo1ltrAZtLukXSJEkbt5yptJ+kqZKmzpw5s4+zMzOznpRJBNMkXU1KBFdJGga81cf5DQaWAT4MfAc4X1KXSSUiTouIsRExdtSoUX2cnZmZ9aTMA2X7AGOAByPiZUkjgL4+W/AocFGuvXSypLeAkaSH1MzMbACUOSMIYF3ga7l9cdLTxn1xCbAVgKS1gEWAp/s4LTMz6wdlEsEpwEeAXXP7bOBnPY0k6Tzgr8Dakh6VtA9wJrB6vqX018AefreBmdnAKlM09KGI2FDSrQAR8ZykRXoaKSJ2bdFrt94EaGZm1SpzRvCGpEHkF9ZLGkXfLxabmdm7TJlE8L/AxcCyko4C/gT8qNKozMysNmVqHz1X0jTgY6TnBz4dEfdUHpmZmdWizDuLz46I3YF7u+hmZmYLuDJFQ+sVW/L1go2qCcfMzOrWMhFIOlTSbOD9kmZJmp3bZwCX1hahmZlVqmUiiIgfRcQw4LiIGB4Rw/JnREQcWmOMZmZWoTLPEVwpaYvmjrl2UTMzW8CVSQTfKTQPATYBpgFbVxKRmZnVqszto58qtktaGTi5sojMzKxWfXlV5aPA+/o7EDMzGxhlniP4Cbl6CVLiGANMrzIoMzOrT5lrBFMLzXOA8yLizxXFY2ZmNStzjWBiHYGYmdnAKFM0tClwOLBqHl5ARMTq1YZmZmZ1KFM0dAbwDdIto29WG46ZmdWtTCJ4ISKurDwSMzMbEGUSwfWSjgMuAl5rdIwI3zlkZtYGSr2qMn+PLXQL/GSxmVlbKHPX0FZ1BGJmZgOjZSKQtFtEnCPpm131j4gTqwvLzMzq0t0ZweL5e1gdgZiZ2cBomQgi4tT8fUR94ZiZWd3KPFC2GvBVYHRx+IjYsbqwzMysLmXuGrqE9FDZ5cBb1YZjZmZ1K5MIXo2I/608EjMzGxBlEsGPJY0HrsYPlPXJpEmTKp3+lltuWen0zay9lUkEGwC7kx4gaxQN+YEyM7M2USYR7AKsHhGvVx2MmZnVr8yrKu8Elqo6EDMzGxhlzgiWAu6VNIV5rxH49lEzszZQJhGMrzwKMzMbMN3VNaRIWt7y0himmtDMzKwO3V0juF7SVyWtUuwoaRFJW0uaCOxRbXhmZla17oqGtgP2Bs7L1Uw8DwwlJY+rgZMj4tbqQzQzsyp1V+ncq8ApwCmSFgZGAq9ExPNlJizpTGAHYEZErN/U71vA8cCoiHi6r8Gbmdk7V+b2USLijYh4omwSyCaQzirmIWllYBvg4V5My8zMKlIqEfRFRNwIPNtFr5OAg0lPJ5uZ2QCrLBF0RdJOwGMRcXuJYfeTNFXS1JkzZ9YQnZlZZyqVCCStKunjuXmopF6/tUzSYsBhwA/KDB8Rp0XE2IgYO2rUqN7OzszMSuoxEUjaF/gtcGrutBLpHQW9tQawGnC7pIfydKZLWq4P0zIzs35S5sniA4FNgFsAIuJ+Scv2dkYRcQfw9ng5GYz1XUNmZgOrTNHQa8WaRyUNpsSFXknnAX8F1pb0qKR9+h6mmZlVpcwZwSRJhwFDJX0C+E/Sayu7FRG79tB/dKkIbYFT5Yt4/BIes/5X5ozgu8BM4A5gf+AK4HtVBmVmZvXp8YwgIt4CTs8fMzNrM2XuGtpB0q2SnpU0S9JsSbPqCM7MzKpX5hrBycB/AHe4ymkzs/ZT5hrBI8CdTgJmZu2pzBnBwcAVkiYx76sqT6wsKrMFRJV3SIHvkrJ6lEkERwEvAkOARaoNx8zM6lYmEazQ/D4BMzNrH2WuEVwhaZvKIzEzswFRJhEcAPxB0iu+fdTMrP2UeaCs11VOm5nZgqNlIpC0TkTcK2nDrvpHxPTqwjIzs7p0d0bwTWA/4IQu+gWwdSURmZlZrVomgojYLzf+W0S8WuwnaUilUZnZu1KnPTfRKctb5mLxX0p2MzOzBVB31wiWA1YkvYfgg4Byr+HAYjXEZmZmNejuGsG2wJ6kdwufwNxEMIv0EnozM2sD3V0jmAhMlPSZiLiwxpjMzKxGPV4jcBIwM2tvZS4Wm5lZG3MiMDPrcGVqH0XSR4HRxeEj4pcVxWRmZjXqMRFIOhtYA7gNeDN3DsCJwMysDZQ5IxgLrOtXVZqZtacy1wjuBJarOhAzMxsYZc4IRgJ3S5rMvO8s3rGyqMzMrDZlEsHhVQdhZmYDp8yLaSZJeg+wce40OSJmVBuWmZnVpcdrBJI+C0wGdgE+C9wiaeeqAzMzs3qUKRr6L2DjxlmApFHAtcBvqwzMzMzqUeauoYWaioKeKTmemZktAMqcEfxB0lXAebn9c8AV1YVkZmZ1KnOx+DuSPgNsmjudFhEXVxuWmZnVpVRdQ7kqaldHbWbWhlqW9Uv6U/6eLWlW4TNb0qyeJizpTEkzJN1Z6HacpHsl/U3SxZKW6p/FMDOzvmqZCCJis/w9LCKGFz7DImJ4iWlPALZr6nYNsH5EvB/4O3BoH+M2M7N+UuY5grPLdGsWETcCzzZ1uzoi5uTWm0nvQzYzswFU5jbQ9YotkgYDG/XDvPcGrmzVU9J+kqZKmjpz5sx+mJ2ZmXWlu2sEh0qaDby/eH0AeAq49J3MVNJ/AXOAc1sNExGnRcTYiBg7atSodzI7MzPrRnfXCH4UEcOA45quD4yIiD6X7UvaE9gB+KLfcWBmNvDK3D56paQtmjvmawC9Imk74GBgy4h4ubfjm5lZ/yuTCL5TaB4CbAJMA7bubiRJ5wHjgJGSHgXGk+4SWhS4RhLAzRHxld6HbWZm/aXMk8WfKrZLWhk4ucR4u3bR+YzyoZmZWR36Unnco8D7+jsQMzMbGD2eEUj6CdC4qLsQMAaYXmVQZmZWnzLXCKYWmucA50XEnyuKx8zMalYmEfwWeDUi3gSQNEjSYr7rx8ysPZS5RnAdMLTQPpT0hjIzM2sDZRLBkIh4sdGSmxerLiQzM6tTmUTwkqQNGy2SNgJeqS4kMzOrU5lrBF8HLpD0OCBgOdLrKs3MrA2UeaBsiqR1gLVzp/si4o1qwzIzs7qUeR/BYsAhwEERcScwWtIOlUdmZma1KHON4CzgdeAjuf0x4MjKIjIzs1qVSQRrRMSxwBsA+fkBVRqVmZnVpkwieF3SUHI1E5LWAF6rNCozM6tNmbuGxgN/AFaWdC6wKbBnlUGZmVl9ytw1dI2k6cCHSUVCB0XE05VHZmZmtShz19A+EfFMRPw+In4HPCdpfA2xmZlZDcpcI/iYpCskLS9pPeBmYFjFcZmZWU3KFA19QdLngDuAl4AvuBpqM7P2UaZoaE3gIOBC4F/A7vkhMzMzawNlioYuB74fEfsDWwL3A1MqjcrMzGpT5vbRTSJiFkBEBHCCpMurDcvMzOrS8oxA0sEAETFL0i5NvfesMigzM6tPd0VDny80H9rUb7sKYjEzswHQXSJQi+au2s3MbAHVXSKIFs1dtZuZ2QKqu4vFH5A0i3T0PzQ3k9uHVB6ZmZnVomUiiIhBdQZiZmYDo8xzBGZm1sacCMzMOpwTgZlZh3MiMDPrcE4EZmYdzonAzKzDORGYmXU4JwIzsw5XWSKQdKakGZLuLHRbRtI1ku7P30tXNX8zMyunyjOCCcxfS+l3gesiYk3gutxuZmYDqLJEEBE3As82dd4JmJibJwKfrmr+ZmZWTpk3lPWn90TEE7n5SeA9rQaUtB+wX259UdJ9VQdXoZHA0wMdRI06aXk7aVnBy7ugWbXMQHUngrdFREhqWZ11RJwGnFZjSJWRNDUixg50HHXppOXtpGUFL2+7qvuuoackLQ+Qv2fUPH8zM2tSdyK4DNgjN+8BXFrz/M3MrEmVt4+eB/wVWFvSo5L2AY4GPiHpfuDjub0TtEURVy900vJ20rKCl7ctKcJvnTQz62R+stjMrMM5EZiZdTgnggpJ2k7SfZIekNT2T1FLekjSHZJukzR1oOPpb51WbUqL5T1c0mN5Hd8mafuBjLG/SFpZ0vWS7pZ0l6SDcve2Xb9FTgQVkTQI+Bnwb8C6wK6S1h3YqGqxVUSMadN7ryfQWdWmTGD+5QU4Ka/jMRFxRc0xVWUO8K2IWBf4MHBg/r+28/p9mxNBdTYBHoiIByPideDXpCo2bAHVadWmtFjethQRT0TE9Nw8G7gHWJE2Xr9FTgTVWRF4pND+aO7WzgK4VtK0XEVIJyhdbUob+aqkv+Wio7YrKpE0GvggcAsdsn6dCKw/bRYRY0jFYQdK2mKgA6pTpHux2/1+7P8DVgfGAE8AJwxsOP1L0hLAhcDXI2JWsV87r18nguo8BqxcaF8pd2tbEfFY/p4BXEwqHmt3HVVtSkQ8FRFvRsRbwOm00TqWtDApCZwbERflzh2xfp0IqjMFWFPSapIWAT5PqmKjLUlaXNKwRjOwDXBn92O1hY6qNqWxU8z+nTZZx5IEnAHcExEnFnp1xPr1k8UVyrfWnQwMAs6MiKMGOKTKSFqddBYAqVbbX7Xb8uZqU8aRqiZ+ChgPXAKcD6wC/Av4bES0xQXWFss7jlQsFMBDwP6FMvQFlqTNgJuAO4C3cufDSNcJ2nL9FjkRmJl1OBcNmZl1OCcCM7MO50RgZtbhnAjMzDqcE4GZWYdzIliASQpJJxTavy3p8H6a9gRJO/fHtHqYzy6S7pF0fS/GebHKmOoi6S/5e3Sxhs8Ww46T9LsW/R6SNLKKGN9t2mXdv9s4ESzYXgP+4922E5A0uBeD7wPsGxFbVRXPu1VEfHSgYzADJ4IF3RzSO1W/0dyj+Yi+cSSVjywnSbpU0oOSjpb0RUmT87sE1ihM5uOSpkpXedsNAAAGDUlEQVT6u6Qd8viDJB0naUqueGz/wnRvknQZcHcX8eyap3+npGNytx8AmwFnSDquafglJF0naXoeb76aWyX9TNKOufliSWfm5r0lHZWbL8mV4N3VqAgv9z+5MJ19JZ2Un47+vaTbc5yf62KeXcaVf8cDC8Mdns/QWi5HV0e3+ezgpjz8dEnFZDE8x3efpJ9Lmu//K2m3vC5vk3SqUnXozcNsLOkveTknSxomaYiks3KMt0raKg+7Z/4Nr8lnHv9P0jfzMDdLWiYPd0P+DafmM7yNJV2kVI//kYV5z7c+Gr+FpKNyTDdLek/uvpqkv+a4itNZXtKNeTnvlLR583JaL0SEPwvoB3gRGE56wnNJ4NvA4bnfBGDn4rD5exzwPLA8sCip/qMjcr+DgJML4/+BdLCwJqn21CHAfsD38jCLAlOB1fJ0XwJW6yLOFYCHgVGkp47/CHw697sBGNvFOIOB4bl5JPAAcx+AbCzL54HjcvNk4ObcfBawbW5eJn8PJVWHMAJYAvgHsHDu9xdgA+AzwOmFGJYsGxeptspJheHuJtU1VWY5RgN35ubFgCG5eU1gamG9vUqq8G0QcE1j/eb1PxJ4H3B5YblOAb7UFP8iwIPAxrl9eI7xW6Sn3wHWyetrCLBnjnlYXn8vAF/Jw51EqpytsR6PKWxHjzN3G3sUGNFqfeT2AD6Vm49l7jZ2WWMZgAMLv9m3gP/KzYOAYQP9f1yQPz4jWMBFqiHxl8DXejHalEj1r79G2iFenbvfQdopNZwfEW9FxP2kncc6pDqEviTpNtLj9yNIOyyAyRHxzy7mtzFwQ0TMjIg5wLlATzWTCvgfSX8DriVV4d1cBfBNwOZKLxC5m7kVhH2EtHMH+Jqk24GbSTvmNSPiRVIy2kHSOqQd5x15+T8h6RhJm0fEC2XjiohbgWUlrSDpA8BzEfFIyeUoWhg4XdIdwAWklxo1TI70fos3gfNIZ1NFHwM2Aqbk9fMxUuIoWht4IiKmQNp+8jrZDDgnd7uXVJ3CWnmc6yNidkTMJCWCy3P35u3lskL3uwrb2IPMrYBxvvWRu78ONK6BTCtMd9O8rABnF+Y1BdhL6ZrYBpHeIWB91JuyXHv3OhmYTjoSbphDLvrLRQiLFPq9Vmh+q9D+FvNuE831jwRpx/bViLiq2EPSONIZQX/5IukIdKOIeEPSQ6Qj1LnBRDwmaSnSW7RuBJYBPks6apydY/o48JGIeFnSDYVp/IJUl8y95N8tIv4uaUNge+BISdcBVwGn5nF+kOfRKq4LgJ2B5YDflF2OJt8g1evzAdL6e7W4yE3DNrcLmBgRh3Yz/b4ou7281sUwbw/Xw/p4I/LhPfAm3W+HRMSNStWcfxKYIOnEiPhlbxfMEp8RtIFIlWCdT7rw2vAQ6egQYEfSkWZv7SJpIaXrBqsD95F2jAcoVdmLpLWUahvtzmRgS0kjc5n1rsCkHsZZEpiRd55bAau2GO5m4OukRHATqXjspsI0nss7nXVIryAEICJuIR2RfoF8xClpBeDliDgHOA7YMCJuibmvZbysh7h+Qyqu2pmUFHqzHMXlfiJSNc+7k4o9GjbJZeYLAZ8D/tQ07nXAzpKWzcuzjKTm+d0HLC9p4zzMMKWL+zeRkhaS1iJVsnZfD7H2Vsv10Y0/k35TGvHlGFcFnoqI00lJfcN+jrWjOBG0jxNI5cQNp5N2vreTikr6crT+MGknfiWpXPhV0p/ubmC60i2Pp9LDmWWk2im/C1wP3A5Mi4ieqvM9Fxibi0i+RDpy78pNwOCIeIB0VrQMcxPBH0hHovcAR5OSRtH5wJ8j4rncvgEwORerjAeOZH4t44qIu0hl6Y/F3Bo5yy5HwynAHnm9rcO8620K8FPSaxT/ydzaXhvzvxv4HnB1Loq6hlROXxzmdVIS+UmexzWko/JTgIVynL8B9szFOv2pp/XRlYNILzm6g3nf8DcOuF3SraTl+XE/x9pRXPuodSyl+/JPiojrBjoWs4HkMwLrOJKWkvR34BUnATOfEZiZdTyfEZiZdTgnAjOzDudEYGbW4ZwIzMw6nBOBmVmH+/80dCtdN1SiiQAAAABJRU5ErkJggg==",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1fe25949dd8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ex:  [14.539999999999999, 13.782772836163664, 13.491561879499317, 13.136129351781083, 13.136072647246625]\n",
      "Am:  [19.559999999999999, 18.151107632595821, 17.606002256847582, 16.94068326002138, 16.940577117220617]\n"
     ]
    }
   ],
   "source": [
    "DISPLAYED_FREQ_COMMANDS = [0,5,10,15,20]\n",
    "TOTAL_COMMANDS = 800\n",
    "\n",
    "def get_most_frequent_commands(S,f):\n",
    "    '''\n",
    "    Returns a list with the f most\n",
    "    frequent commands.\n",
    "    '''\n",
    "    # Initialize freq dictionary\n",
    "    commands_freq = {}\n",
    "    for i in range(1,TOTAL_COMMANDS+1):\n",
    "        commands_freq[i] = 0\n",
    "            \n",
    "    # Compute frequencies\n",
    "    for scenario in S:\n",
    "        for c in scenario:\n",
    "            commands_freq[int(c[1])] += 1\n",
    "    \n",
    "    most_freq = sorted(commands_freq.items(), key=operator.itemgetter(1))\n",
    "    \n",
    "    return [com[0] for com in most_freq[-f:]]\n",
    "\n",
    "def get_weight_NAIVE(expertise, prv, cur):\n",
    "    '''\n",
    "    @param expertise: a for amateurs and e for experts\n",
    "    @param prv: previous command group and id\n",
    "    @param cur: current command group and id\n",
    "    If a command is inside this list, or the group of its previous command\n",
    "    is the same, the 'same' weight will be used. Otherwise, the 'changed'\n",
    "    weight will be used.\n",
    "    ''' \n",
    "    condition = 'c'\n",
    "    if (prv[0] == cur[0]):\n",
    "        condition = 's'\n",
    "    if (int(cur[1]) in freq_coms):\n",
    "        condition = 's'\n",
    "    return weight[expertise][condition]\n",
    "\n",
    "\n",
    "def get_score_NAIVE(S):\n",
    "    '''\n",
    "    For every command, add the appropriate weight to thte total score.\n",
    "    '''\n",
    "    a_score = 0 # amateur score\n",
    "    e_score = 0 # expert score\n",
    "    prev_com = ['-1', '-1']\n",
    "    \n",
    "    for scenario in S:\n",
    "        for cur_com in scenario:\n",
    "            a_score += get_weight_NAIVE('a', prev_com, cur_com)\n",
    "            e_score += get_weight_NAIVE('e', prev_com, cur_com)\n",
    "            prev_com = cur_com\n",
    "        prev_com = ['-1', '-1']\n",
    "                               \n",
    "    return [a_score/60/len(S), e_score/60/len(S)]\n",
    "\n",
    "\n",
    "def get_comparison_NAIVE(St, Se):\n",
    "    '''\n",
    "    Plots the execution time for amateurs and experts\n",
    "    for different amounts of comamnds available on desktop. \n",
    "    '''\n",
    "    # Get scores\n",
    "    global freq_coms, n_score\n",
    "    naive_a, naive_e = [],[]\n",
    "    for f in DISPLAYED_FREQ_COMMANDS[1:]:\n",
    "        #if (f == 20):\n",
    "        #    global n_time\n",
    "        #    n_time = time.time()\n",
    "        freq_coms = get_most_frequent_commands(St,f)\n",
    "        #if (f == 20):\n",
    "        #    n_time = round(time.time()-n_time,2)\n",
    "        temp = get_score_NAIVE(Se)\n",
    "        naive_a.append(temp[0])\n",
    "        naive_e.append(temp[1])\n",
    "    \n",
    "    # Add CURRENT\n",
    "    naive_a = [c_score['a']] + naive_a\n",
    "    naive_e = [c_score['e']] + naive_e\n",
    "    \n",
    "    # n_score\n",
    "    n_score = {'a' : naive_a[4], 'e' : naive_e[4]}\n",
    "    \n",
    "    # Plot Scores\n",
    "    x = np.array(DISPLAYED_FREQ_COMMANDS)\n",
    "    ax = plt.subplot(111)\n",
    "    ax.bar(x-1, naive_a, width=2, color='grey', align='center')\n",
    "    ax.bar(x+1, naive_e, width=2, color='silver', align='center')\n",
    "    plt.ylim([10,20])\n",
    "    plt.xlim([-4,24])\n",
    "    plt.title('NAIVE execution times for novices and experts')\n",
    "    plt.ylabel('Execution time (minutes per scenario)')\n",
    "    plt.xlabel('Number of always-available commands')\n",
    "    #plt.legend(['Amateurs', 'Experts'])\n",
    "    plt.savefig(\"1.eps\", format=\"eps\")\n",
    "    plt.show()\n",
    "    print('Ex: ', naive_e)\n",
    "    print('Am: ', naive_a)\n",
    "\n",
    "\n",
    "get_comparison_NAIVE(S[:SHORT],S[SHORT:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "write_csv(value=n_score['a'], exp='a', meth='NAIVE', size='a')\n",
    "write_csv(value=n_score['e'], exp='e', meth='NAIVE', size='a')\n",
    "write_csv(value=n_time, exp='time', meth='NAIVE', size='a')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Therefore we se how the time significantly decreases when adding up to common commands to the screen and that it stabilizes at around 15 available commands.\n",
    "\n",
    "Therefore, comparing the $D=15$ version of **NAIVE**, we get the following comparisson with **CURRENT**."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### 3. GROUP\n",
    "Group heuristics: group together the most used common commands (e.g. delete and move) by doing away the class-verb method for selecting them.\n",
    "\n",
    "We are using the labels we created on some of the commands:\n",
    "<u>what command group numbers mean?</u>\n",
    "\n",
    "When one of these exist in a command, we assume that it is available on the desktop. Therefore for them the $s$ weight is used, as well as for those who belong to the same entity. For the others, the $c$ weight is used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "a = 0\n",
    "b = 0\n",
    "\n",
    "for scenario in S:\n",
    "    prv = [-1, -1, -1]\n",
    "    for move in scenario:\n",
    "        if (int(move[2]) in [1,2,3,4,5,6,7]):\n",
    "            if (prv[0] == move[0]):\n",
    "                a += 1\n",
    "            else:\n",
    "                b += 1\n",
    "        prv = move\n",
    "\n",
    "print(a, b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "GROUP_ID = [1, 2, 3, 4, 5, 6, 7]\n",
    "\n",
    "\n",
    "def get_weight_GROUP(expertise, prv, cur):\n",
    "    '''\n",
    "    @param expertise: a for amateurs and e for experts\n",
    "    @param prv: previous command group and id\n",
    "    @param cur: current command group and id\n",
    "    @param g: group id to be evaluated or all together\n",
    "    If a command is in one of the special groupings, \n",
    "    or the group of its previous commandis the same, \n",
    "    the 'same' weight will be used. Otherwise, the 'changed'\n",
    "    weight will be used.\n",
    "    ''' \n",
    "    condition = 'c'\n",
    "    if (int(cur[2]) >= 0):\n",
    "        condition = 's'\n",
    "    if (prv[0] == cur[0]):\n",
    "        condition = 's'\n",
    "    return weight[expertise][condition]\n",
    "\n",
    "\n",
    "def get_score_GROUP(S):\n",
    "    '''\n",
    "    For every command, add the appropriate weight to thte total score.\n",
    "    '''\n",
    "    a_score = 0 # amateur score\n",
    "    e_score = 0 # expert score\n",
    "    prev_com = ['-1', '-1']\n",
    "    \n",
    "    for scenario in S:\n",
    "        for cur_com in scenario:\n",
    "            a_score += get_weight_GROUP('a', prev_com, cur_com)\n",
    "            e_score += get_weight_GROUP('e', prev_com, cur_com)\n",
    "            prev_com = cur_com\n",
    "        prev_com = ['-1', '-1']\n",
    "        \n",
    "    #score = 'Amateurs: ' + str(round(a_score,2)) + \\\n",
    "    #        '\\nExperts: ' + str(round(e_score,2))\n",
    "                               \n",
    "    return [str(round(a_score/60/len(S),2)),\n",
    "            str(round(e_score/60/len(S),2))]\n",
    "\n",
    "# Print Results\n",
    "print('Aggregate GROUP score:')\n",
    "score = get_score_GROUP(S[:SHORT])\n",
    "print('Amateurs: ' + score[0] + ' minutes per scenario, ' \\\n",
    "     + str(round((c_score['a']-float(score[0]))/c_score['a']*100,2)) + '% less than CURRENT'\\\n",
    "     + '\\nExperts: ' +  score[1] + ' minutes per scenario, '\\\n",
    "     + str(round((c_score['e']-float(score[1]))/c_score['e']*100,2)) + '% less than CURRENT')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "g_score = {'a' : score[0], 'e' : score[1]}\n",
    "write_csv(value=g_score['a'], exp='a', meth='GROUP', size='a')\n",
    "write_csv(value=g_score['e'], exp='e', meth='GROUP', size='a')\n",
    "write_csv(value=g_score['a'], exp='a', meth='GROUP', size='s')\n",
    "write_csv(value=g_score['e'], exp='e', meth='GROUP', size='s')\n",
    "write_csv(value=0, exp='time', meth='GROUP', size='a')\n",
    "write_csv(value=0, exp='time', meth='GROUP', size='s')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Examinging the impact of each command grouping separately, as they have relatively the same frequency, all have a small impact that is almost the same between all of them. Therefore, no command grouping shows to be more potent than the others and displaying all the 7 grouped commands on the screen is suggested.\n",
    "\n",
    "In comparison to <b>NAIVE</b>, <b>GROUP</b> is more efficient than displaying frequent commands, as NAIVE, even for 10 commands needs additional 0.5 and 1,6 minutes per scenario.\n",
    "\n",
    "Addionally, while 22% of the commands are grouped, the benefit of using this technique is minimal (less than 1%). This could be because the user who clicks a grouped command (as delete x, edit x, ...) may be almost always already on this entity."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### 4. MRU-B\n",
    "User optimized, based on the most recently used (MRU) commands of a particular user, when executed more frequently during a training period (batch).\n",
    "\n",
    "Since we concluded that 15 commands on desktop achieve an almost optimal results and can fit on the users screen, we set $D = 15$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "DISPLAYED_FREQ_COMMANDS = [0,5,10,15,20]\n",
    "TOTAL_COMMANDS = 800\n",
    "\n",
    "def get_most_frequent_commands_MRUB(S,f):\n",
    "    '''\n",
    "    Returns a dictionary with the f most\n",
    "    frequent commands for every user.\n",
    "    '''\n",
    "    # Initialize freq dictionary\n",
    "    commands_freq = {}\n",
    "    for user in U:\n",
    "        commands_freq[user] = {}\n",
    "        for i in range(1,TOTAL_COMMANDS+1):\n",
    "            commands_freq[user][i] = 0\n",
    "            \n",
    "    # Compute frequencies\n",
    "    for user, scenario in enumerate(S):\n",
    "        for c in scenario:\n",
    "            commands_freq[U[user]][int(c[1])] += 1\n",
    "    \n",
    "    most_freq = {}\n",
    "    most_freq_list = {}\n",
    "    for user in U:\n",
    "        most_freq[user] = sorted(commands_freq[user].items(), key=operator.itemgetter(1))\n",
    "        most_freq[user] = most_freq[user][-f:]\n",
    "        most_freq[user] = [most_freq[user][i][0] for i in range(0,f)]\n",
    "        \n",
    "    return most_freq\n",
    "\n",
    "def get_weight_MRUB(expertise, prv, cur, user):\n",
    "    '''\n",
    "    @param expertise: a for amateurs and e for experts\n",
    "    @param prv: previous command group and id\n",
    "    @param cur: current command group and id\n",
    "    If a command is inside this list, or the group of its previous command\n",
    "    is the same, the 'same' weight will be used. Otherwise, the 'changed'\n",
    "    weight will be used.\n",
    "    ''' \n",
    "    condition = 'c'\n",
    "    if (prv[0] == cur[0]):\n",
    "        condition = 's'\n",
    "    if (int(cur[1]) in freq_coms[user]):\n",
    "        condition = 's'\n",
    "    return weight[expertise][condition]\n",
    "\n",
    "\n",
    "def get_score_MRUB(S):\n",
    "    '''\n",
    "    For every command, add the appropriate weight to thte total score.\n",
    "    '''\n",
    "    a_score = 0 # amateur score\n",
    "    e_score = 0 # expert score\n",
    "    prev_com = ['-1', '-1']\n",
    "    \n",
    "    for user, scenario in enumerate(S):\n",
    "        for cur_com in scenario:\n",
    "            a_score += get_weight_MRUB('a', prev_com, cur_com, U[user])\n",
    "            e_score += get_weight_MRUB('e', prev_com, cur_com, U[user])\n",
    "            prev_com = cur_com\n",
    "        prev_com = ['-1', '-1']\n",
    "        \n",
    "    #score = 'Amateurs: ' + str(round(a_score,2)) + \\\n",
    "    #        '\\nExperts: ' + str(round(e_score,2))\n",
    "                               \n",
    "    return [a_score/60/len(S), e_score/60/len(S)]\n",
    "\n",
    "\n",
    "def get_comparison_MRUB(St,Se):\n",
    "    '''\n",
    "    Plots the execution time for amateurs and experts\n",
    "    for different amounts of comamnds available on desktop. \n",
    "    '''\n",
    "    # Get scores\n",
    "    global freq_coms, mrub_score\n",
    "    mrub_a, mrub_e = [],[]\n",
    "    for f in DISPLAYED_FREQ_COMMANDS[1:]:\n",
    "        freq_coms = get_most_frequent_commands_MRUB(St,f)\n",
    "        tt = get_score_MRUB(Se)\n",
    "        mrub_a.append(tt[0])\n",
    "        mrub_e.append(tt[1])\n",
    "    \n",
    "    # Add CURRENT\n",
    "    mrub_a = [c_score['a']] + mrub_a\n",
    "    mrub_e = [c_score['e']] + mrub_e\n",
    "    \n",
    "    # Get score for CURRENT comparison\n",
    "    mrub_score = {'a': mrub_a[3], 'e': mrub_e[3]}\n",
    "    \n",
    "    # Plot Scores\n",
    "    x = np.array(DISPLAYED_FREQ_COMMANDS)\n",
    "    ax = plt.subplot(111)\n",
    "    ax.bar(x-1, mrub_a, width=2, color='b', align='center')\n",
    "    ax.bar(x+1, mrub_e, width=2, color='g', align='center')\n",
    "    plt.ylim([10,20])\n",
    "    plt.xlim([-4,24])\n",
    "    plt.title('MRU-B execution times for experts and amateurs')\n",
    "    plt.ylabel('Score (minutes per scenario)')\n",
    "    plt.xlabel('Number of most common commands available on desktop')\n",
    "    plt.legend(['Amateurs', 'Experts'])\n",
    "    plt.show()\n",
    "    print('Am: ', mrub_e)\n",
    "    print('Ex: ', mrub_a)\n",
    "\n",
    "          \n",
    "# Chart Comparison          \n",
    "get_comparison_MRUB(S,S)\n",
    "\n",
    "# CURRENT Comparison\n",
    "print('\\nCURRENT and NAIVE comparison with MRUB with 15 displayed commands:')\n",
    "print('Amateurs: ' + str(round(mrub_score['a'],2)) + ' minutes per scenario, ' \\\n",
    "     + str(round((c_score['a']-float(mrub_score['a']))/c_score['a']*100,2)) + '% less than CURRENT, '\\\n",
    "     + str(round((n_score['a']-float(mrub_score['a']))/n_score['a']*100,2)) + '% less than NAIVE' \\\n",
    "     + '\\nExperts: ' +  str(round(mrub_score['e'],2)) + ' minutes per scenario, '\\\n",
    "     + str(round((c_score['e']-float(mrub_score['e']))/c_score['e']*100,2)) + '% less than CURRENT, '\\\n",
    "     + str(round((n_score['e']-float(mrub_score['e']))/n_score['e']*100,2)) + '% less than NAIVE')\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "For the best-case **MRU-B**, where we the system for the whole dataset, we see results similar to the **NAIVE** approach, that bring a significant decrease in time for 15 displayed commands and a 0.17'' additional decrease for 20 commands, which would be the maximum to display on a typical application.\n",
    "For 15 displayed commands, **MRU-B** is 15.61% and 11.25% quicker than **CURRENT** for amateurs and experts respectively and  2.26% and 1.55% quicker than **NAIVE**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "write_csv(value=mrub_score['a'], exp='a', meth='MRUB', size='a')\n",
    "write_csv(value=mrub_score['e'], exp='e', meth='MRUB', size='a')\n",
    "write_csv(value=mrub_time, exp='time', meth='MRUB', size='a')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### 5. MRU-O\n",
    "User optimized, based on the most recently used (MRU) commands of a particular user, executed more frequently during continuously adjusted (online).\n",
    "\n",
    "* **Option 1**: continuously calculate the frequencies for the user.\n",
    "* **Option 2**: continuously calculate the frequencies for the user, starting anew for every session.\n",
    "* **Option 3**: continuously calculate the frequencies for the user, starting a new for every session but using a discunted memory system.\n",
    "\n",
    "\n",
    "We have seen from **MRU-B** that for $D=20$ desktop-available commands we achieve maximum performance. We keep this stable to better compare the above options.\n",
    "\n",
    "**Option 1** is evaluated below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "D = 20  # commands available always on desktop\n",
    "\n",
    "def initialize_most_frequent_commands_MRUO(S,D):\n",
    "    '''\n",
    "    Returns a dictionary with the f most\n",
    "    frequent commands for every user.\n",
    "    '''\n",
    "    commands_freq = {}\n",
    "    for user in U:\n",
    "        commands_freq[user] = {}\n",
    "        for i in range(1,TOTAL_COMMANDS+1):\n",
    "            commands_freq[user][i] = 0\n",
    "\n",
    "    return commands_freq\n",
    "\n",
    "\n",
    "def update_most_frequent_commands_MRUO(user, command):\n",
    "    '''\n",
    "    Gets a user and a command and updates its frequency.\n",
    "    ''' \n",
    "    \n",
    "    commands_freq[user][int(command[1])] += 1\n",
    "    \n",
    "\n",
    "\n",
    "def is_in_most_frequent_commands_MRUO(user, command):\n",
    "    '''\n",
    "    Gets a user and a command and returns true if the\n",
    "    command is one of the current D most frequent.\n",
    "    '''   \n",
    "    freq_coms = nlargest(D, commands_freq[user].items(),\n",
    "                        operator.itemgetter(1))\n",
    "    \n",
    "    if (command in [c[0] for c in freq_coms]):\n",
    "        return True\n",
    "    else:\n",
    "        return False\n",
    "\n",
    "\n",
    "def get_weight_MRUO(expertise, prv, cur, user):\n",
    "    '''\n",
    "    @param expertise: a for amateurs and e for experts\n",
    "    @param prv: previous command group and id\n",
    "    @param cur: current command group and id\n",
    "    If a command is inside this list, or the group of its previous command\n",
    "    is the same, the 'same' weight will be used. Otherwise, the 'changed'\n",
    "    weight will be used.\n",
    "    ''' \n",
    "    condition = 'c'\n",
    "    if (prv[0] == cur[0]):\n",
    "        condition = 's'\n",
    "    if (is_in_most_frequent_commands_MRUO(user, int(cur[1]))):\n",
    "        condition = 's'\n",
    "    return weight[expertise][condition]\n",
    "\n",
    "\n",
    "def get_score_MRUO(S):\n",
    "    '''\n",
    "    For every command, add the appropriate weight to thte total score.\n",
    "    '''\n",
    "    a_score = 0 # amateur score\n",
    "    e_score = 0 # expert score\n",
    "    prev_com = ['-1', '-1']\n",
    "    global commands_freq, mruo_time\n",
    "    mruo_time = 0\n",
    "    commands_freq = initialize_most_frequent_commands_MRUO(S, D)\n",
    "    \n",
    "    for user, scenario in enumerate(S):\n",
    "        #print(str(user) + \"out of\" + str(scenarios))\n",
    "        for cur_com in scenario:\n",
    "            a_score += get_weight_MRUO('a', prev_com, cur_com, U[user])\n",
    "            e_score += get_weight_MRUO('e', prev_com, cur_com, U[user])\n",
    "            prev_com = cur_com\n",
    "            temp_time = time.time()\n",
    "            update_most_frequent_commands_MRUO(U[user], cur_com)\n",
    "            mruo_time += round(time.time()-temp_time,2)\n",
    "        prev_com = ['-1', '-1']\n",
    "    \n",
    "    global mruo_score\n",
    "    mruo_score = {'a' : round(a_score/60/len(S),2),\n",
    "                  'e' : round(e_score/60/len(S),2)}\n",
    "    print(mruo_score)\n",
    "\n",
    "# Execution\n",
    "get_score_MRUO(S[ALL:])\n",
    "print(mruo_score)\n",
    "print(mruo_time)\n",
    "\n",
    "write_csv(value=mruo_score['a'], exp='a', meth='MRUO', size='a')\n",
    "write_csv(value=mruo_score['e'], exp='e', meth='MRUO', size='a')\n",
    "write_csv(value=mruo_score['a'], exp='a', meth='MRUO', size='s')\n",
    "write_csv(value=mruo_score['e'], exp='e', meth='MRUO', size='s')\n",
    "write_csv(value=mruo_time, exp='time', meth='MRUO', size='a')\n",
    "write_csv(value=mruo_time, exp='time', meth='MRUO', size='s')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### 6. OPT\n",
    "Optimized: run a (stochastic) optimization algorithm to select those commands whose combination yields the fastest user interaction time. This differs from NAIVE in that it takes into account the actual switching between entities"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "**Greedy Knapsack** (Dantzig, 1957): $D$ identical places for $D$ identical-sized items with different values. Order all items with the value they creat by saving time, by ordering them in ascending order. The first $D$ items should be chosen. Additionally and on a stohastic basis, we evaluate only the $F$ most frequent commands, as we assume that the $TOTAL\\_COMMANDS - F$ are not likely to be that significant in lowering the execution time of the scenarios, if they appear on desktop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "D = 20  # Number of available commands on desktop\n",
    "F = 300  # Number of most-frequent commands to have their score evaluated\n",
    "TOTAL_COMMANDS = 800  # The maximum number of commands\n",
    "\n",
    "def evaluate_singles(S,D):\n",
    "    '''\n",
    "    Calculates a sorted list of the most time-saving commands\n",
    "    to be always available on desktop and returns the D most\n",
    "    valuable.\n",
    "    '''\n",
    "    \n",
    "    # score for amateurs only as both groups would give\n",
    "    # the same results, since they're weight lines are\n",
    "    # parallel.\n",
    "    \n",
    "    gain = {}\n",
    "    freq = get_most_frequent_commands(S,F)\n",
    "    \n",
    "    for command in freq:\n",
    "        gain[command] = 0\n",
    "        for scenario in S:\n",
    "            prev_group = -1\n",
    "            for move in scenario:\n",
    "                \n",
    "                if (command == int(move[1])) and prev_group != move[0]:\n",
    "                    gain[command] += 1\n",
    "                prev_group = move[0]\n",
    "\n",
    "    \n",
    "    score_list = [0 for i in range(TOTAL_COMMANDS+1)]\n",
    "    \n",
    "    for i in range(TOTAL_COMMANDS+1):\n",
    "        if i in gain.keys():\n",
    "            score_list[i] = gain[i]\n",
    "        else:\n",
    "            score_list[i] = 0\n",
    "            \n",
    "    sortedscore = np.array(score_list)\n",
    "    return sortedscore.argsort()[-D:]\n",
    "\n",
    "\n",
    "def evaluate_commmands(S, commands):\n",
    "    '''\n",
    "    Given a set of Scenarios and the commands that remain on desktop\n",
    "    It returns the score of that solution.\n",
    "    '''\n",
    "    score = {'a': 0, 'e': 0}\n",
    "    commands = commands.tolist()\n",
    "    \n",
    "    for scenario in S:\n",
    "        prev_group = -1\n",
    "        for move in scenario:\n",
    "            # Find Weight Type\n",
    "            if prev_group == move[0] or int(move[1]) in commands:\n",
    "                w_type = 's'\n",
    "            else:\n",
    "                w_type = 'c'\n",
    "            \n",
    "            # Add Score\n",
    "            score['a'] += weight['a'][w_type]\n",
    "            score['e'] += weight['e'][w_type]\n",
    "            \n",
    "            prev_group = move[0]\n",
    "    \n",
    "    return score\n",
    "    \n",
    "def optimization(St,Se):\n",
    "    \n",
    "    global opt_score, opt_time\n",
    "    \n",
    "    # Training\n",
    "    opt_time = time.time()\n",
    "    a = evaluate_singles(St,D)\n",
    "    opt_time = round(time.time()-opt_time,2)\n",
    "    \n",
    "    # Evaluation\n",
    "    opt_score = evaluate_commmands(Se, a)\n",
    "    for t in ['a', 'e']:\n",
    "        opt_score[t] = round(opt_score[t]/60/len(Se),2)\n",
    "    print('done.')\n",
    "\n",
    "#optimization(S[:ALL],S[ALL:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "write_csv(value=opt_score['a'], exp='a', meth='OPT(KS)', size='a')\n",
    "write_csv(value=opt_score['e'], exp='e', meth='OPT(KS)', size='a')\n",
    "write_csv(value=opt_time, exp='time', meth='OPT(KS)', size='a')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Difference with NAIVE\n",
    "\n",
    "* **NAIVE** 20 commands: [72, 71, 73, 28, 53, 44, 81, 85, 35, 47, 48, 63, 57, 62, 27, 50, 41, 42, 38, 40]\n",
    "* **OPT** 20 commands: [135, 11, 147, 228, 119, 12, 1, 30, 121, 108, 74, 86, 4, 57, 85, 81, 35, 27, 38, 40]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#NAIVE = [72, 71, 73, 28, 53, 44, 81, 85, 35, 47, 48, 63, 57, 62, 27, 50, 41, 42, 38, 40]\n",
    "#OPT = [135, 11, 147, 228, 119, 12, 1, 30, 121, 108, 74, 86, 4, 57, 85, 81, 35, 27, 38, 40]\n",
    "\n",
    "# Comparison\n",
    "print('\\nCURRENT and NAIVE comparison with OPT with 20 displayed commands:')\n",
    "print('Amateurs: ' + str(round(opt_score['a'],2)) + ' minutes per scenario, '\\\n",
    "     + str(round((c_score['a']-float(opt_score['a']))/c_score['a']*100,2)) + '% less than CURRENT, '\\\n",
    "     + str(round((n_score['a']-float(opt_score['a']))/n_score['a']*100,2)) + '% less than NAIVE, ' \\\n",
    "     + '\\nExperts: ' +  str(round(opt_score['e'],2)) + ' minutes per scenario, '\\\n",
    "     + str(round((c_score['e']-float(opt_score['e']))/c_score['e']*100,2)) + '% less than CURRENT, '\\\n",
    "     + str(round((n_score['e']-float(opt_score['e']))/n_score['e']*100,2)) + '% less than NAIVE, ')\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**OPT / Stohastic (Genetic) Optimization**\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "D = 20  # Number of available commands on desktop\n",
    "\n",
    "TOTAL_COMMANDS = 800  # The maximum number of commands\n",
    "\n",
    "# Sampling\n",
    "\n",
    "\n",
    "# Genetic Algorithm Variables\n",
    "Y = 1500  # Number of generations (years)\n",
    "PS = 50  # Population Size\n",
    "CR = 0.6  # Crossover Rate\n",
    "#S_EVAL = 1000  # To randomly evaluate commands on a subset of S\n",
    "#S_st = random.sample(St,S_EVAL)\n",
    "M = 0.01  # Mutated propability\n",
    "E = int(PS * (1-CR))  # Elitist members\n",
    "geneSet = [i for i in range(1,TOTAL_COMMANDS+1)]\n",
    "population = generate_initial_population(D,PS)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "def generate_one(D):\n",
    "    '''\n",
    "    Generates a random solution with D displayed commands.\n",
    "    '''\n",
    "    member = random.sample(geneSet,D)\n",
    "    return member\n",
    "\n",
    "\n",
    "def generate_initial_population(D,PS):\n",
    "    '''\n",
    "    Create initial population of size PS and D genes.\n",
    "    '''\n",
    "    population = [generate_one(D) for i in range(PS)]\n",
    "    print('Initial population created.')\n",
    "    return population\n",
    "\n",
    "\n",
    "def get_fitness(population,S):\n",
    "    '''\n",
    "    Evaluate the fitness of a solution using the OPY\n",
    "    evaluation method.\n",
    "    '''\n",
    "    fitness = [evaluate_commmands(S,np.array(member))['a']/len(S)/60\n",
    "              for member in population]\n",
    "    return fitness\n",
    "\n",
    "\n",
    "def mutate(m_population,D,M):\n",
    "    '''\n",
    "    Mutate a population, given on a mutation rate.\n",
    "    '''\n",
    "    for m in range(len(m_population)):\n",
    "        for g in range(D):\n",
    "            r = random.random()\n",
    "            if (r <= M):\n",
    "                newGene, alternate = random.sample(geneSet, 2)\n",
    "                if(m_population[m][g] == newGene):\n",
    "                    m_population[m][g] = alternate\n",
    "                else:\n",
    "                    m_population[m][g] = newGene\n",
    "    return m_population\n",
    "\n",
    "\n",
    "def make_child(mom, dad):\n",
    "    '''\n",
    "    Create a child from two members.\n",
    "    '''\n",
    "    child = []\n",
    "    for i in range(len(mom)):\n",
    "        child.append(random.choice([mom[i],dad[i]]))\n",
    "    return child\n",
    "\n",
    "\n",
    "def crossover(population,fitness,C):\n",
    "    cross_pop = []\n",
    "    for i in range(int(C)):\n",
    "        mom,dad = roulette_wheel_pop(population, get_propability_list(fitness))\n",
    "        \n",
    "        i = [i for i in range(D)]\n",
    "        r = random.choice(i)\n",
    "        \n",
    "        child = mom[:r] + dad[r:]\n",
    "        cross_pop.append(child)\n",
    "    return cross_pop\n",
    "\n",
    "def get_propability_list(fitness):\n",
    "    '''\n",
    "    Get propability list for a roulete selection.\n",
    "    '''\n",
    "    total_fit = float(sum(fitness))\n",
    "    relative_fit = [f/total_fit for f in fitness]\n",
    "    propabilities = [sum(relative_fit[:i+1])\n",
    "                    for i in range(len(relative_fit))]\n",
    "    return propabilities\n",
    "\n",
    "\n",
    "def roulette_wheel_pop(population, propabilities, number=2):\n",
    "    '''\n",
    "    Selects randomly a member of their population,\n",
    "    giving higher priority to those with better fitness.\n",
    "    '''\n",
    "    chosen = []\n",
    "    for n in range(number):\n",
    "        r = random.random()\n",
    "        for (i, member) in enumerate(population):\n",
    "            if (r <= propabilities[i]):\n",
    "                chosen.append(member)\n",
    "                break\n",
    "    return chosen\n",
    "\n",
    "\n",
    "def elitist(population,fitness):\n",
    "    '''\n",
    "    Get the member who survives.\n",
    "    '''\n",
    "    index, value = min(enumerate(fitness),\n",
    "                       key=operator.itemgetter(1))\n",
    "    elitist = population[index]\n",
    "    population.pop(index)\n",
    "    fitness.pop(index)\n",
    "    \n",
    "    return population, fitness, elitist\n",
    "   \n",
    "    \n",
    "def evolution_process(S,D,PS,E,Y):\n",
    "    '''\n",
    "    Run the evolution process.\n",
    "    @param S Scenarios\n",
    "    @param D Displayed commands = No of genes per member\n",
    "    @param PS Population Size\n",
    "    @param E Number of Elitists that survive\n",
    "    @param Y evolution iterations (years)\n",
    "    '''\n",
    "    # 0. Initialize Population\n",
    "    population = generate_initial_population(D,PS)\n",
    "\n",
    "    for i in range(Y):\n",
    "        # 1. Evaluate Population Members\n",
    "        S_st = random.sample(S,3500)\n",
    "        fitness = get_fitness(population,S_st)\n",
    "\n",
    "        # 2. Crossover\n",
    "        new_population = crossover(population, fitness, PS-E)\n",
    "\n",
    "        # 3. Mutations\n",
    "        mut_population = mutate(new_population,D,M)\n",
    "        # 4. Elitists\n",
    "        elits = []\n",
    "        for j in range(E):\n",
    "            population, fitness, elit = elitist(population,fitness)\n",
    "            elits.append(elit)\n",
    "            mut_population.append(elit)\n",
    "        \n",
    "        \n",
    "        # 5. Update Population\n",
    "        population = new_population\n",
    "        #print(population)\n",
    "        print(i)\n",
    "    #print(elits[0],evaluate_commmands(S,np.array(elits[0]))['a']/scenarios/60)\n",
    "    \n",
    "    return elits[0]\n",
    "\n",
    "ga_time = time.time()\n",
    "ga_list = evolution_process(S[:ALL],D,PS,E,Y)\n",
    "ga_time = round(time.time()-ga_time,2)\n",
    "print(ga_time)\n",
    "\n",
    "ga_score = evaluate_commmands(S[ALL:],np.array(ga_list))\n",
    "for t in ['a', 'e']:\n",
    "    ga_score[t] = ga_score[t]/60/len(S[ALL:])\n",
    "print(ga_list,ga_score)   \n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2800 Scenarios \n",
    "\n",
    "**1st try, 796 iterations:**\n",
    "[118, 427, 543, 69, 383, 121, 38, 51, 181, 85, 245, 368, 133, 15, 217, 29, 508, 369, 40, 312] 17.5054769999\n",
    "\n",
    "**2nd try, 1500 iterations:**\n",
    "[219, 40, 387, 507, 505, 432, 27, 9, 356, 368, 401, 369, 343, 481, 38, 193, 227, 525, 569, 534] 17.3805623832\n",
    "\n",
    "**3rd try, 2000 iterations:**\n",
    "[147, 81, 27, 40, 121, 85, 86, 57, 135, 119, 108, 12, 35, 228, 4, 1, 224, 38, 74, 30] 15.8104998208\n",
    "\n",
    "#### 300000 Scenarios \n",
    "\n",
    "** 1200 iterations **\n",
    "[32, 1, 35, 86, 119, 38, 121, 40, 147, 108, 127, 30, 74, 27, 4, 12, 224, 81, 57, 85] 16.1607420376"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "print(ga_score)\n",
    "write_csv(value=ga_score['a'], exp='a', meth='OPT(GA)', size='s')\n",
    "write_csv(value=ga_score['e'], exp='e', meth='OPT(GA)', size='s')\n",
    "write_csv(value=ga_time, exp='time', meth='OPT(GA)', size='s')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ga_score = evaluate_commmands(S,np.array([\n",
    "    119, 38, 40, 35, 85, 224, 30, 27, 11, 74, 86, 12, 228, 57, 127, 121, 1, 81, 4, 108]))\n",
    "for t in ['a', 'e']:\n",
    "    ga_score[t] = ga_score[t]/60/scenarios\n",
    "\n",
    "print('\\nCURRENT and OPT comparison with OPT with 20 displayed commands:')\n",
    "print('Amateurs: ' + str(round(ga_score['a'],2)) + ' minutes per scenario, '\\\n",
    "     + str(round((c_score['a']-float(ga_score['a']))/c_score['a']*100,2)) + '% less than CURRENT, '\\\n",
    "     + str(round((ga_score['a']-float(opt_score['a']))/ga_score['a']*100,2)) + '% more than Knapsack OPT, ' \\\n",
    "     + '\\nExperts: ' +  str(round(ga_score['e'],2)) + ' minutes per scenario, '\\\n",
    "     + str(round((c_score['e']-float(ga_score['e']))/c_score['e']*100,2)) + '% less than CURRENT, '\\\n",
    "     + str(round((ga_score['e']-float(opt_score['e']))/ga_score['e']*100,2)) + '% less than Knapsack OPT, ')\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7. CLUSTER\n",
    "Clustered, based on 2-3 clusters of users. The clustering is performed based on the commands they execute. A stochastic optimization similar to OPT is ran to select the commands executed in each cluster.\n",
    "\n",
    "A simple way to cluster the users would be to base the clustering on the frequency of the commands they use. However, we want to choose the commands to be displayed always on desktop the time in a way that they minimize the time taken for a scenario to be completed. \n",
    "\n",
    "A command can have a 'same-group' time or a 'changing-group' time. therefore, we need to display the frequent commands that are used when user change groups. Therefore, a more appropriate clustering can be done on:\n",
    "the frequency of commands clicked, counting only if the group was changed immediately before. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TOTAL_COMMANDS = 800\n",
    "\n",
    "D = 20\n",
    "CN = 3  # Number of Clusters\n",
    "\n",
    "def get_user_command_frequencies(S,U):\n",
    "    '''\n",
    "    Creates a numpy array with the command frequency\n",
    "    for every user.\n",
    "    In the output array, 0th index corresponds to the\n",
    "    1st user, as it is in the User (U) list.\n",
    "    '''\n",
    "    #print(S[scenario][move][command],U[user])\n",
    "    # Initialize\n",
    "    U = [int(u) for u in U]\n",
    "    Cf = [[0 for command in range(TOTAL_COMMANDS)] \n",
    "          for user in range(max(U))]\n",
    "    \n",
    "    # Get Frequencies\n",
    "    for user,scenario in enumerate(S):\n",
    "        for move in scenario:\n",
    "            Cf[U[user]-1][int(move[1])-1] = Cf[U[user]-1][int(move[1])-1] + 1\n",
    "    \n",
    "    return Cf\n",
    "        \n",
    "\n",
    "def get_clusters(Cf,CN):\n",
    "    '''\n",
    "    Return the clustered users, as a list of their labels.\n",
    "    KMeans is used for clustering.\n",
    "    '''\n",
    "    # Cluster\n",
    "    X = np.matrix(Cf)\n",
    "    kmeans = KMeans(n_clusters=CN).fit(X)\n",
    "    L =  kmeans.labels_\n",
    "    return L\n",
    "\n",
    "\n",
    "def get_clustered_scenarios(S,L,U,l):\n",
    "    '''\n",
    "    Get the scenarios that correspond to a specific cluster\n",
    "    of users.\n",
    "    '''\n",
    "    S_cluster = []\n",
    "    \n",
    "    for user,scenario in enumerate(S):\n",
    "        if(L[int(U[user])-1] == l):\n",
    "            S_cluster.append(scenario)\n",
    "    \n",
    "    return S_cluster\n",
    "\n",
    "def get_clusters_OPT(St,Se,U,CN,D):\n",
    "    '''\n",
    "    Evaluate CLUSTER by aggregating the OPT scores\n",
    "    of the clusters.\n",
    "    '''\n",
    "    global cl_score, L, cl_time\n",
    "    \n",
    "    cl_time = time.time()\n",
    "    \n",
    "    # Get Clusters\n",
    "    Cf = get_user_command_frequencies(St,U[:len(St)])\n",
    "    L = get_clusters(Cf,CN)\n",
    "    \n",
    "    counter = {}\n",
    "    for i  in range(CN):\n",
    "        counter[i] = 0\n",
    "    for user in L:\n",
    "        counter[i] += 1\n",
    "    common, value = max(enumerate(counter), key=operator.itemgetter(1))\n",
    "    \n",
    "    L = list(L)\n",
    "    for user in U[len(St):]:\n",
    "        L.append(int(common))\n",
    "    L = np.array(L)\n",
    "    \n",
    "    S_clustered = [get_clustered_scenarios(St,L,U,l)\n",
    "                  for l in range(CN)]\n",
    "    \n",
    "    # Get Cluster scores\n",
    "    a = []\n",
    "    a = [evaluate_singles(S_clustered[l],D)\n",
    "        for l in range(CN)]\n",
    "    \n",
    "    cl_time = round(time.time()-cl_time,2)\n",
    "    \n",
    "    # Evaluate\n",
    "    cl_score_temp = [evaluate_commmands(Se, a[l])\n",
    "                 for l in range(CN)]\n",
    "    \n",
    "    # Aggregate\n",
    "    cl_score = {'a': 0, 'e': 0}\n",
    "    for l in range(CN):\n",
    "        for t in ['a', 'e']:\n",
    "            cl_score[t] += cl_score_temp[l][t]\n",
    "    \n",
    "    # Export\n",
    "    for t in ['a', 'e']:\n",
    "        cl_score[t] = round(cl_score[t]/60/len(Se),2)\n",
    "        \n",
    "    print('done.')\n",
    "\n",
    "get_clusters_OPT(S[:SHORT],S[ALL:],U,CN,D)\n",
    "print(cl_time)\n",
    "#write_csv(value=opt_score['a'], exp='a', meth='OPT(KS)', size='a')\n",
    "#write_csv(value=opt_score['e'], exp='e', meth='OPT(KS)', size='a')\n",
    "write_csv(value=cl_score['a'], exp='a', meth='CLUSTER', size='s')\n",
    "write_csv(value=cl_score['e'], exp='e', meth='CLUSTER', size='s')\n",
    "#write_csv(value=opt_time, exp='time', meth='OPT(KS)', size='a')\n",
    "write_csv(value=cl_time, exp='time', meth='CLUSTER', size='s')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "write_csv(value=cl_score['a'], exp='a', meth='CLUSTER', size='a')\n",
    "write_csv(value=cl_score['e'], exp='e', meth='CLUSTER', size='a')\n",
    "write_csv(value=cl_time, exp='time', meth='CLUSTER', size='a')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}