tensor_flaw_seq2sql.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Seq2Sql\n",
    "\n",
    "Задача перевода запросов на натуральном языке в машинный код аналогична задаче перевода \"последовательность-в-последовательность\" \n",
    "\n",
    "<img src=\"files/lstm.png\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset: SQL!!\n",
    "На самом деле много sql'eй\n",
    "\n",
    "Все они находятся в папке \"Data\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**1) Посмотрим, что за данные**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<sqlite3.Cursor at 0x7f5620913260>"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "CURRENT_PATH = '/home/vika/seq2sql_botcamp/WikiSQL/data/'\n",
    "\n",
    "\n",
    "import sqlite3\n",
    " \n",
    "conn = sqlite3.connect(CURRENT_PATH + \"dev.db\") # или :memory: чтобы сохранить в RAM\n",
    "cursor = conn.cursor()\n",
    "\n",
    "cursor.execute('SELECT col0 FROM table_1_10015132_9 WHERE col3=\"Guard\"')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[]\n"
     ]
    }
   ],
   "source": [
    "print(cursor.fetchall())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>question</th>\n",
       "      <th>query</th>\n",
       "      <th>db</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>What are the description and credit of the cou...</td>\n",
       "      <td>SELECT T4.crs_description ,  T4.crs_credit FRO...</td>\n",
       "      <td>college_1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>What is highest rating for the most recent mov...</td>\n",
       "      <td>SELECT max(T1.stars) ,  T2.year FROM Rating AS...</td>\n",
       "      <td>movie_1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>What is the average latitude and longitude of ...</td>\n",
       "      <td>SELECT avg(lat) ,  avg(long) FROM station WHER...</td>\n",
       "      <td>bike_1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>What are the names of the airports in the city...</td>\n",
       "      <td>SELECT name FROM airports WHERE city  =  'Goroka'</td>\n",
       "      <td>flight_4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>Who are the friends of Alice that are doctors?</td>\n",
       "      <td>SELECT T2.friend FROM Person AS T1 JOIN Person...</td>\n",
       "      <td>network_2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Unnamed: 0                                           question  \\\n",
       "0           0  What are the description and credit of the cou...   \n",
       "1           1  What is highest rating for the most recent mov...   \n",
       "2           2  What is the average latitude and longitude of ...   \n",
       "3           3  What are the names of the airports in the city...   \n",
       "4           4     Who are the friends of Alice that are doctors?   \n",
       "\n",
       "                                               query         db  \n",
       "0  SELECT T4.crs_description ,  T4.crs_credit FRO...  college_1  \n",
       "1  SELECT max(T1.stars) ,  T2.year FROM Rating AS...    movie_1  \n",
       "2  SELECT avg(lat) ,  avg(long) FROM station WHER...     bike_1  \n",
       "3  SELECT name FROM airports WHERE city  =  'Goroka'   flight_4  \n",
       "4  SELECT T2.friend FROM Person AS T1 JOIN Person...  network_2  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**2) Попробуем выполнить запрос.**\n",
    "\n",
    "Третья колонка - это индекс базы данных\n",
    "\n",
    "Напоминаю, что базы данных лежат в папке  database, для локальных бд используется пакет sqlite3\n",
    "\n",
    "Если ваши бд лежет где-то в сети, можно использовать следующий код:\n",
    "    \n",
    "*import _mysql*\n",
    "\n",
    "*db=_mysql.connect(\"localhost\",\"user\", \"pass\", \"base\")*\n",
    "\n",
    "*db.query(r\"SELECT field1, field2 FROM table1 WHERE field1 < 5\")*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 264,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3701"
      ]
     },
     "execution_count": 264,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(df['db'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Return the average total amount purchased and total value purchased for the supplier who supplies the greatest number of products. department_store\n"
     ]
    }
   ],
   "source": [
    "print (df['question'].values[23], df['db'].values[23])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(27702.7975, 33126.1)]\n"
     ]
    }
   ],
   "source": [
    "print(cursor.fetchall())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 214,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<sqlite3.Cursor at 0x7f43bef149d0>"
      ]
     },
     "execution_count": 214,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import sqlite3\n",
    " \n",
    "conn = sqlite3.connect(CURRENT_PATH + \"database/movie_1/movie_1.sqlite\") # или :memory: чтобы сохранить в RAM\n",
    "cursor = conn.cursor()\n",
    "\n",
    "# Создание таблицы\n",
    "cursor.execute(\"\"\"SELECT max ( T1 . stars ) , T2 . year FROM Rating AS T1 JOIN Movie AS T2 ON T1 . mID = T2 . mID WHERE T2 . year = ( SELECT max ( YEAR ) FROM Movie )\"\"\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 248,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<sqlite3.Cursor at 0x7f4215e69180>"
      ]
     },
     "execution_count": 248,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import sqlite3\n",
    " \n",
    "conn = sqlite3.connect(CURRENT_PATH + \"database/college_1/college_1.sqlite\") # или :memory: чтобы сохранить в RAM\n",
    "cursor = conn.cursor()\n",
    "\n",
    "# Создание таблицы\n",
    "cursor.execute(\"\"\"select T4.crs_description, T4 . crs_credit FROM CLASS AS T1 JOIN enroll AS T2 ON T1.class_code = T2.class_code JOIN student AS T3 ON T3.stu_num = T2.stu_num JOIN course AS T4 ON T4.crs_code = T1.crs_code WHERE T3.stu_lname = 'Smithson' \"\"\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 249,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('Accounting I', 3.0), ('Intro. to Microcomputing', 3.0), ('Intro. to Statistics', 3.0)]\n"
     ]
    }
   ],
   "source": [
    "print(cursor.fetchall())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SELECT avg ( lat ) , avg ( long ) FROM station WHERE city = \" San Jose \""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Переведем sql.csv to train, test.txt*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3701"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('data/train.txt', 'wb') as the_file:\n",
    "    for ind, qwest in enumerate(df['question'].values[1:3000]):\n",
    "        the_file.write(qwest.replace('   ', ' ').replace('  ', ' ').encode('utf-8'))\n",
    "        the_file.write('\\t'.encode('utf-8'))\n",
    "        the_file.write(df['query'].replace('   ', ' ').replace('  ', ' ').values[ind].encode('utf-8'))\n",
    "        the_file.write('\\t'.encode('utf-8'))\n",
    "        the_file.write(df['db'].replace('   ', ' ').replace('  ', ' ').values[ind].encode('utf-8'))\n",
    "        the_file.write('\\n'.encode('utf-8'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/vika/seq2sql_botcamp\n"
     ]
    }
   ],
   "source": [
    "!pwd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('data/test.txt', 'wb') as the_file:\n",
    "    for ind, qwest in enumerate(df['question'].values[3000:]):\n",
    "        the_file.write(qwest.replace('  ', ' ').replace('  ', ' ').encode('utf-8'))\n",
    "        the_file.write('\\t'.encode('utf-8'))\n",
    "        the_file.write(df['query'].replace('   ', ' ').replace('  ', ' ').values[ind].encode('utf-8'))\n",
    "        the_file.write('\\t'.encode('utf-8'))\n",
    "        the_file.write(df['db'].replace('   ', ' ').replace('  ', ' ').values[ind].encode('utf-8'))\n",
    "        the_file.write('\\n'.encode('utf-8'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Словари\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Обычно: модель работает со словарями слов, которые возможно встретить во входных и выходных последовательностях.\n",
    "* Cловарь каждому слову присваивает номер, по номеру слова словарь дает слово. Т.е класс \"словарь\" содержит стурктуры word2index, index2word. Например, в util_tf.py класс Vocab\n",
    "* на вход модели дается не набор слов, в набор номеров слов в словаре. \n",
    "\n",
    "## BPE!\n",
    "**Byte Pair Encoding.**\n",
    "\n",
    "Токенезирует на уровне символов (т.е. побуквенно), заменяет часто встречающиеся символы одним токеном. Затем вместо словаря слов появляется словарь токенов."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 178,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "from nltk.tokenize import WordPunctTokenizer\n",
    "from subword_nmt.learn_bpe import learn_bpe\n",
    "from subword_nmt.apply_bpe import BPE\n",
    "tokenizer = WordPunctTokenizer()\n",
    "\n",
    "def tokenize(x):\n",
    "    return ' '.join(tokenizer.tokenize(x))\n",
    "\n",
    "def preprocess(dataset_type, datadir, source_voc_type, dist_voc_type): #создает bpe-словарики из слов в словарях sourcevoc(это все возможные слова, \n",
    "    # которые могут встретиться в вопросах на натуральном языке по этой базе данных), targetvoc(символы математического языка запроса)\n",
    "    with open(\"{}.{}\".format(dataset_type,source_voc_type), 'wb') as f_src,  open(\"{}.{}\".format(dataset_type, dist_voc_type), 'wb') as f_dst:\n",
    "        with open('{}.txt'.format(dataset_type), encoding='utf-8') as f and open('{}_db.txt'.format(dataset_type), encoding='utf-8') as f1:\n",
    "            my_list = f.readlines()\n",
    "            for line in my_list:\n",
    "                try:\n",
    "                    src_line, dst_line, db_type = line.strip().split('\\t')\n",
    "                    f_src.write((tokenize(src_line) + '\\n').encode('utf-8'))\n",
    "                    f_dst.write((tokenize(dst_line) + '\\n').encode('utf-8'))\n",
    "                except Exception as e:\n",
    "                    print (line)\n",
    "                    print (line.strip().split('\\t', 4))\n",
    "                    print (len(line.strip().split('\\t')))\n",
    "                    \n",
    "        \n",
    "    bpe = {}\n",
    "    for v_type in [source_voc_type, dist_voc_type]:\n",
    "        a = open('./{}.'.format(dataset_type) + v_type)\n",
    "        b = open('bpe_rules.' + v_type, 'w')\n",
    "        print(a)\n",
    "        learn_bpe(open('./{}.'.format(dataset_type) + v_type), open('bpe_rules.' + v_type, 'w'), num_symbols=8000)\n",
    "        bpe[v_type] = BPE(open('./bpe_rules.' + v_type))\n",
    "        with open('{}.bpe.'.format(dataset_type) + v_type, 'w') as f_out:\n",
    "            for line in open('{}.'.format(dataset_type) + v_type):\n",
    "                f_out.write(bpe[v_type].process_line(line.strip()) + '\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 261,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'s@@ e@@ le@@ c@@ t t@@ 1 . m@@ ar@@ k@@ e@@ t@@ i@@ n@@ g@@ _@@ r@@ e@@ g@@ i@@ o@@ n@@ @@@ @ '"
      ]
     },
     "execution_count": 261,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bpe_qu = BPE(open('./bpe_rules.qu'))\n",
    "bpe_qu.process_line('select t1 . marketing_region@@ ')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 260,
   "metadata": {},
   "outputs": [],
   "source": [
    "bpe_qu.vocab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "metadata": {},
   "outputs": [
    {
     "ename": "FileNotFoundError",
     "evalue": "[Errno 2] No such file or directory: 'train.txt'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mFileNotFoundError\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-148-2dc18b017fe6>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mpreprocess\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'train'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'data'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'qw'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'qu'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m<ipython-input-146-68c3fd771a02>\u001b[0m in \u001b[0;36mpreprocess\u001b[0;34m(dataset_type, datadir, source_voc_type, dist_voc_type)\u001b[0m\n\u001b[1;32m     13\u001b[0m     \u001b[0;31m# которые могут встретиться в вопросах на натуральном языке по этой базе данных), targetvoc(символы математического языка запроса)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     14\u001b[0m     \u001b[0;32mwith\u001b[0m \u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"{}.{}\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdataset_type\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0msource_voc_type\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'wb'\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mf_src\u001b[0m\u001b[0;34m,\u001b[0m  \u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"{}.{}\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdataset_type\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdist_voc_type\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'wb'\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mf_dst\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 15\u001b[0;31m         \u001b[0;32mwith\u001b[0m \u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'{}.txt'\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdataset_type\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mencoding\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'utf-8'\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     16\u001b[0m             \u001b[0mmy_list\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreadlines\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     17\u001b[0m             \u001b[0;32mfor\u001b[0m \u001b[0mline\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mmy_list\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mFileNotFoundError\u001b[0m: [Errno 2] No such file or directory: 'train.txt'"
     ]
    }
   ],
   "source": [
    "preprocess('train', 'data', 'qw', 'qu')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'a' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-143-3f786850e387>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0ma\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m: name 'a' is not defined"
     ]
    }
   ],
   "source": [
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {},
   "outputs": [],
   "source": [
    "!export LC_ALL=en_US.UTF-8\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "no pair has frequency >= 2. Stopping\n",
      "no pair has frequency >= 2. Stopping\n"
     ]
    }
   ],
   "source": [
    "preprocess('test', 'data', 'qw', 'qu')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**определим train и test и посмотрим на данные**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 229,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "inp: what is highest rating for the most recent movie and when was it released ?\n",
      "out: select max ( t1 . stars ) , t2 . year from rating as t1 join movie as t2 on t1 . mid = t2 . mid where t2 . year = ( select max ( year ) from movie )\n",
      "db: movie_1\n",
      "\n",
      "inp: what is the average latitude and longitude of stations located in san jose city ?\n",
      "out: select avg ( lat ) , avg ( long ) from station where city = \"sanjose\"\n",
      "db: bike_1\n",
      "\n",
      "inp: what are the names of the airports in the city of goroka ?\n",
      "out: select name from airports where city = 'goroka'\n",
      "db: flight_4\n",
      "\n"
     ]
    }
   ],
   "source": [
    "train_inp = np.array(open('./train.bpe.qw', encoding = 'utf-8').read().split('\\n'))\n",
    "train_out = np.array(open('./train.bpe.qu').read().split('\\n'))\n",
    "test_inp = np.array(open('./test.bpe.qw').read().split('\\n'))\n",
    "test_out = np.array(open('./test.bpe.qu').read().split('\\n'))\n",
    "test_db = np.array(open('./test_db.txt').read().split('\\n'))\n",
    "train_db = np.array(open('./train_db.txt').read().split('\\n'))\n",
    "\n",
    "import re\n",
    "\n",
    "for i in range(3):\n",
    "    print('inp:', train_inp[i])\n",
    "    res = re.sub(r'\\\".*?\\\"', lambda x: ''.join(x.group(0).split()), train_out[i])\n",
    "    res = re.sub(r'\\'.*?\\'', lambda x: ''.join(x.group(0).split()), res)\n",
    "    print('out:', res)\n",
    "    print('db:',train_db[i], end='\\n\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 230,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from util_tf import Vocab\n",
    "inp_voc = Vocab.from_lines(train_inp)\n",
    "out_voc = Vocab.from_lines(train_out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**to_matrix, to_line**\n",
    "* у класса vocab есть функция to_matrix, которая из батча вопросов на естественном языке генерит батч чисел, где число - индекс соответсвующего слова в словаре.\n",
    "\n",
    "* И функция to_lines, которая батч чисел переводит обратно\n",
    "\n",
    "* Важно! модель предсказывает батч чисел"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 231,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "lines\n",
      "['show all allergy types .', 'what are all the labels ?', 'how many students exist ?', 'how many staff in total ?', 'how many vehicles exist ?']\n",
      "\n",
      "words to ids (0 = bos, 1 = eos):\n",
      "[[   0 1563  242  247 1774   21    1    1]\n",
      " [   0 1842  296  242 1708 1034  175    1]\n",
      " [   0  925 1123 1650  741  175    1    1]\n",
      " [   0  925 1123 1611  947 1739  175    1]\n",
      " [   0  925 1123 1810  741  175    1    1]]\n",
      "\n",
      "back to words\n",
      "['show all allergy types .', 'what are all the labels ?', 'how many students exist ?', 'how many staff in total ?', 'how many vehicles exist ?']\n"
     ]
    }
   ],
   "source": [
    "batch_lines = sorted(train_inp, key=len)[5:10]\n",
    "batch_ids = inp_voc.to_matrix(batch_lines)\n",
    "batch_lines_restored = inp_voc.to_lines(batch_ids)\n",
    "\n",
    "print(\"lines\")\n",
    "print(batch_lines)\n",
    "print(\"\\nwords to ids (0 = bos, 1 = eos):\")\n",
    "print(batch_ids)\n",
    "print(\"\\nback to words\")\n",
    "print(batch_lines_restored)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Модель\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 232,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import keras.layers as L\n",
    "from util_tf import infer_length, infer_mask"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 233,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class BasicModel:\n",
    "    def __init__(self, name, inp_voc, out_voc, emb_size=64, hid_size=128):\n",
    "        \n",
    "        self.name, self.inp_voc, self.out_voc = name, inp_voc, out_voc\n",
    "\n",
    "        with tf.variable_scope(name):\n",
    "            self.emb_inp = L.Embedding(len(inp_voc), emb_size)\n",
    "            self.emb_out = L.Embedding(len(out_voc), emb_size)\n",
    "            self.enc0 = tf.nn.rnn_cell.GRUCell(hid_size)\n",
    "\n",
    "            self.dec_start = L.Dense(hid_size)\n",
    "            self.dec0 = tf.nn.rnn_cell.GRUCell(hid_size)\n",
    "            self.logits = L.Dense(len(out_voc))\n",
    "\n",
    "            # prepare to translate_lines\n",
    "            self.inp = tf.placeholder('int32', [None, None])\n",
    "            self.initial_state = self.prev_state = self.encode(self.inp)\n",
    "            self.prev_tokens = tf.placeholder('int32', [None])\n",
    "            self.next_state, self.next_logits = self.decode(self.prev_state, self.prev_tokens)\n",
    "        \n",
    "        self.weights = tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES, scope=name)\n",
    "\n",
    "    def encode(self, inp, **flags):\n",
    "        \"\"\"\n",
    "        Получает на вход батч, прогоняет его через emb_layer (теперь каждому числу-слову соответсвует вектор)\n",
    "        возвращает начальное состояние для rnn-ки декодера\n",
    "        \"\"\"\n",
    "        inp_lengths = infer_length(inp, self.inp_voc.eos_ix)\n",
    "        inp_emb = self.emb_inp(inp)\n",
    "        with tf.variable_scope('enc0'):\n",
    "            _, enc_last = tf.nn.dynamic_rnn(\n",
    "                              self.enc0, inp_emb,\n",
    "                              sequence_length=inp_lengths,\n",
    "                              dtype = inp_emb.dtype)\n",
    "        dec_start = self.dec_start(enc_last)\n",
    "        print (\"dec_start.shape\", dec_start.shape)\n",
    "        return [dec_start]\n",
    "\n",
    "    def decode(self, prev_state, prev_tokens, **flags):\n",
    "        \"\"\"\n",
    "        берет предыдущее состояние, возвращает вероятности для всех символов выходного словаря и состояние для следующей ячейки\n",
    "        \"\"\"\n",
    "        [prev_dec] = prev_state\n",
    "        prev_emb = self.emb_out(prev_tokens[:,None])[:,0]\n",
    "        with tf.variable_scope('dec0'):\n",
    "            new_dec_out, new_dec_state = self.dec0(prev_emb, prev_dec)\n",
    "        output_logits = self.logits(new_dec_out)\n",
    "        return [new_dec_state], output_logits\n",
    "\n",
    "    def translate_lines(self, inp_lines, max_len=100):\n",
    "        state = sess.run(self.initial_state, {self.inp: inp_voc.to_matrix(inp_lines)})\n",
    "        outputs = [[self.out_voc.bos_ix] for _ in range(len(inp_lines))]\n",
    "        all_states = [state]\n",
    "        finished = [False] * len(inp_lines)\n",
    "\n",
    "        for t in range(max_len):\n",
    "            state, logits = sess.run([self.next_state, self.next_logits], {**dict(zip(self.prev_state, state)),\n",
    "                                           self.prev_tokens: [out_i[-1] for out_i in outputs]})\n",
    "            next_tokens = np.argmax(logits, axis=-1)\n",
    "            all_states.append(state)\n",
    "            for i in range(len(next_tokens)):\n",
    "                outputs[i].append(next_tokens[i])\n",
    "                finished[i] |= next_tokens[i] == self.out_voc.eos_ix\n",
    "        return out_voc.to_lines(outputs), all_states"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 234,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/anaconda/lib/python3.6/site-packages/tensorflow/python/client/session.py:1662: UserWarning: An interactive session is already active. This can cause out-of-memory errors in some cases. You must explicitly call `InteractiveSession.close()` to release resources held by the other session(s).\n",
      "  warnings.warn('An interactive session is already active. This can '\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dec_start.shape (?, 128)\n"
     ]
    }
   ],
   "source": [
    "tf.reset_default_graph()\n",
    "sess = tf.InteractiveSession()\n",
    "\n",
    "model = BasicModel('model', inp_voc, out_voc)\n",
    "sess.run(tf.global_variables_initializer())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loss \n",
    "\n",
    "$$ L = {\\frac1{|D|}} \\sum_{X, Y \\in D} \\sum_{y_t \\in Y} - \\log p(y_t \\mid y_1, \\dots, y_{t-1}, X, \\theta) $$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 235,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def compute_logits(model, inp, out, **flags):\n",
    "   \n",
    "    batch_size = tf.shape(inp)[0]\n",
    "    \n",
    "    first_state = model.encode(inp)\n",
    "    \n",
    "    first_logits = tf.log(tf.one_hot(tf.fill([batch_size], model.out_voc.bos_ix),\n",
    "                                     len(model.out_voc)) + 1e-30)\n",
    "    \n",
    "    def step(prev_tupl, y_prev):\n",
    "        prev_state, logits = prev_tupl\n",
    "        next_state, logits = model.decode(prev_state, y_prev)\n",
    "        \n",
    "        return [next_state, logits]\n",
    "    \n",
    "    h_seq, logits_seq = tf.scan(step, elems=tf.transpose(out)[:-1], initializer=[first_state, first_logits])\n",
    "    \n",
    "    logits_seq = tf.concat([first_logits[None], logits_seq], axis = 0)\n",
    "    #print (\"inp shape\", inp.shape)\n",
    "    #print (\"logits_seq.shape\", logits_seq.shape)\n",
    "    # Make sure you convert logits_seq from [time, batch, voc_size] to [batch, time, voc_size]\n",
    "    logits_seq = tf.transpose(logits_seq, (1, 0, 2))\n",
    "    \n",
    "    #print (\"logits_seq.shape\", logits_seq.shape)\n",
    "    \n",
    "    return logits_seq"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 236,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from util_tf import select_values_over_last_axis\n",
    "\n",
    "def compute_loss(model, inp, out, **flags):\n",
    "    mask = infer_mask(out, out_voc.eos_ix)    \n",
    "    logits_seq = compute_logits(model, inp, out, **flags)\n",
    "    #print (\"logits_seq.shape\", logits_seq.shape)\n",
    "    soft_max = tf.nn.log_softmax(logits_seq, -1)\n",
    "    #print (\"soft max shape\", soft_max.shape)\n",
    "    #print (\"out shape\", out.shape)\n",
    "    #print (\"out\", sess.run(out[2]))\n",
    "    symbols = select_values_over_last_axis(soft_max, out)\n",
    "    #print (\"symbols.shape\", symbols.shape)\n",
    "    symbols = symbols*mask\n",
    "    \n",
    "    return -tf.reduce_sum(symbols)/tf.reduce_sum(mask)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 254,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['what is highest rating for the most recent movie and when was it released ?',\n",
       "       'what is the average latitude and longitude of stations located in san jose city ?',\n",
       "       'what are the names of the airports in the city of goroka ?',\n",
       "       'who are the friends of alice that are doc@@ tors ?',\n",
       "       'what are the appelations for wines produced after 2008 but not in the central coast area ?',\n",
       "       'show the average price range of hotels that have 5 star ratings and al@@ low pe@@ ts .',\n",
       "       'find the number of female students ( with f sex ) living in smith hall',\n",
       "       'what are the staff ids and genders of all staffs whose job title is department manager ?',\n",
       "       'what are the names of all the video games and their types in alphabetical order ?',\n",
       "       'what are the number of rooms for each bed type ?',\n",
       "       'what are the top three apartment types in terms of the average room count ? give me the',\n",
       "       'what is the average account balance of customers with credit score below 50 for the different account types ?',\n",
       "       'what is the total revenue of all companies whose ma@@ in office is at tokyo or taiwan ?'],\n",
       "      dtype='<U236')"
      ]
     },
     "execution_count": 254,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_inp[:13]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 245,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['select max ( t1 . stars ) , t2 . year from rating as t1 join movie as t2 on t1 . mid = t2 . mid where t2 . year = ( select max ( year ) from movie )',\n",
       "       'select avg ( lat ) , avg ( long ) from station where city = \" san jose \"',\n",
       "       \"select name from airports where city = ' goroka '\"], dtype='<U602')"
      ]
     },
     "execution_count": 245,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_out[:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 239,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dec_start.shape (3, 128)\n",
      "Loss: 7.0708547\n"
     ]
    }
   ],
   "source": [
    "dummy_inp = tf.constant(inp_voc.to_matrix(train_inp[:3]))\n",
    "dummy_out = tf.constant(out_voc.to_matrix(train_out[:3]))\n",
    "dummy_loss = sess.run(compute_loss(model, dummy_inp, dummy_out))\n",
    "print(\"Loss:\", dummy_loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3 запроса длиной 16 символов максимально, на каждом символе - 70 вероятностей"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "out определяет индекс элемента в реальной последовательности. По этому индексу из logits вытаскивается значение и суммируется. Чем больше значение по реальному индексу(ег овероятность) тем лучше"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training loop\n",
    "\n",
    "Training encoder-decoder models isn't that different from any other models: sample batches, compute loss, backprop and update"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 240,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dec_start.shape (?, 128)\n"
     ]
    }
   ],
   "source": [
    "inp = tf.placeholder('int32', [None, None])\n",
    "out = tf.placeholder('int32', [None, None])\n",
    "\n",
    "loss = compute_loss(model, inp, out)\n",
    "\n",
    "train_step = tf.train.AdamOptimizer().minimize(loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 241,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from IPython.display import clear_output\n",
    "from tqdm import tqdm, trange\n",
    "metrics = {'train_loss': []}\n",
    "\n",
    "sess.run(tf.global_variables_initializer())\n",
    "batch_size = 32"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 242,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "32"
      ]
     },
     "execution_count": 242,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch_size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 243,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAArwAAAEICAYAAAC9CSe9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzs3Xd4lFXax/HvmUlvJEASOqH3HqUKQVQQ7Ou6lrWu4q5rW9urrr0gW3R1Letid3cVXbsiuCCGpoD03gm9hZZe57x/zGSSIR3SmPw+15VrZ57nzDNnciR75+R+7ttYaxERERER8VeO+p6AiIiIiEhtUsArIiIiIn5NAa+IiIiI+DUFvCIiIiLi1xTwioiIiIhfU8ArIiIiIn5NAa+IiIiI+DUFvCIitcQY87ox5tFTvMa7xphnampOIiKNUUB9T0BEpKEyxqQAN1trZ53M6621v63ZGYmIyMnQDq+IyEkwxmjDQETkNKGAV0SkDMaYfwHtgK+NMRnGmAeMMdYY8xtjzE5gtmfcf40x+40xx40xc40xvUpcw5uOYIxJMsbsNsbca4w5aIzZZ4y58STmdYsxZosx5ogx5itjTCvPcWOM+Zvn2mnGmNXGmN6ec+ONMeuMMenGmD3GmPtq4FskInLaUMArIlIGa+21wE7gQmttBPCx59QooAcw1vN8OtAFiAOWAf+p4LItgCZAa+A3wKvGmJiqzskYczbwHHAF0BLYAUz1nD4PGAl09bzHFcBhz7m3gFuttZFAbzzBuohIY6E/yYmIVM8T1trMoifW2reLHhtjngCOGmOaWGuPl/HafOApa20B8K0xJgPoBiys4ntfA7xtrV3meb+HPO+X4Ll2JNAdWGytXX/C+/Y0xqy01h4Fjlbx/URE/IJ2eEVEqmdX0QNjjNMYM9kYs9UYkwakeE41L+e1hz3BbpEsIKIa790K964uANbaDNy7uK2ttbOBV4BXgYPGmCnGmCjP0F8A44Edxpg5xpih1XhPEZHTngJeEZHy2UqOXQ1cDJyDO40gwXPc1NJ89gLti54YY8KBZsAeAGvt3621g4CeuFMb7vcc/9laezHutIsvKE7PEBFpFBTwioiU7wDQsYLzkUAu7l3WMGBSLc/nQ+BGY0x/Y0yw5/0WWWtTjDFnGGMGG2MCgUwgB3AZY4KMMdd40izygTTAVcvzFBFpUBTwioiU7zngEWPMMeDyMs6/jzvFYA+wjqrn4p4UTz3gR4FPgX1AJ+BKz+ko4A3c+bk7cAfhf/GcuxZI8aRd/BZ3LrCISKNhrC3rL3YiIiIiIv5BO7wiIiIi4tcU8IqI1DNjzFpPc4sTv5R6ICJSA5TSICIiIiJ+rVYaTzRv3twmJCTUxqUrlJmZSXh4eJ2/r9QtrbP/0xo3DlrnxkHr3DjU1zovXbo01VobW9m4Wgl4ExISWLJkSW1cukLJyckkJSXV+ftK3dI6+z+tceOgdW4ctM6NQ32tszFmR+WjlMMrIiIiIn5OAa+IiIiI+DUFvCIiIiLi1xTwioiIiIhfU8ArIiIiIn5NAa+IiIiI+DUFvCIiIiLi1/wm4C0odDFndz6FLnWOExEREZFifhPwvv/TDt5Zk8eHi3fW91REREREpAGpNOA1xnQzxqwo8ZVmjLm7LiZXHfvTcgBIy8mv55mIiIiISENSacBrrd1ore1vre0PDAKygM9rfWbVlJ1XCMCfZ2z0PhYRERERqW5Kwxhgq7W2Sn2L64q1lnmbD3mfP/+/jfU4GxERERFpSIy1Vb/JyxjzNrDMWvtKGecmAhMB4uPjB02dOrXGJlkVh7NdPDA3i0JrGNYqgIl9g+v0/aXuZGRkEBERUd/TkFqkNW4ctM6Ng9a5caivdR49evRSa21iZeOqHPAaY4KAvUAva+2BisYmJibaJUuWVOm6NSk5OZm/rnYSGxHMOzeeWefvL3UjOTmZpKSk+p6G1CKtceOgdW4ctM6NQ32tszGmSgFvdVIazse9u1thsFvfWjUJZe+xnPqehoiIiIg0ENUJeK8CPqytidSUVtGh7D2WXd/TEBEREZEGokoBrzEmHDgX+Kx2p3PqWkWHkJ5boPJkIiIiIgJUMeC11mZaa5tZa4/X9oROVcsmoQDsU1qDiIiIiOBHndaKxIQFAXA8Wzu8IiIiIuKHAW9kSAAA6UppEBERERH8MOCN8AS8GbkF9TwTEREREWkI/C7gjQwu2uFVwCsiIiIifhjwhnsC3kzt8IqIiIgIfhjwhgQ6AcjJd9XzTERERESkIfC7gNfpMAQ5HWTnF9b3VERERESkAfC7gBcgONBBjgJeEREREcFPA97QQKcCXhEREREB/DXgDXIqpUFEREREAD8NeEMCtMMrIiIiIm7+GfAGOclWlQYRERERwV8D3gDdtCYiIiIibn4Z8IYGKaVBRERERNz8M+ANdJKdp4BXRERERPw04A0JdJJToIBXRERERPw44M3O001rIiIiIuK3Aa+DXOXwioiIiAh+GvCGBqrxhIiIiIi4+W3AW+Cy5BcqrUFERESksfPLgDck0Amg0mQiIiIiUrWA1xgTbYz5xBizwRiz3hgztLYndipCgtwBr9IaRERERCSgiuNeAmZYay83xgQBYbU4p1MW6tnhzVV7YREREZFGr9KA1xjTBBgJ3ABgrc0D8mp3WqcmJNC9ca0dXhEREREx1tqKBxjTH5gCrAP6AUuBu6y1mSeMmwhMBIiPjx80derUWplwRTIyMoiIiGDFwQJeXJbLY0ND6NjEWefzkNpVtM7iv7TGjYPWuXHQOjcO9bXOo0ePXmqtTaxsXFVSGgKAgcAd1tpFxpiXgAeBR0sOstZOwR0Yk5iYaJOSkqo96VOVnJxMUlISgVtSYdkievXpz+COzep8HlK7itZZ/JfWuHHQOjcOWufGoaGvc1VuWtsN7LbWLvI8/wR3ANxgFVVpUEqDiIiIiFQa8Fpr9wO7jDHdPIfG4E5vaLBCvWXJdNOaiIiISGNX1SoNdwD/8VRo2AbcWHtTOnVFN62pDq+IiIiIVCngtdauACpNCG4oQlWHV0REREQ8/LPTWoA6rYmIiIiIm18GvNrhFREREZEifhnwBgcU5fDqpjURERGRxs4vA15jDCGBDqU0iIiIiIh/BrzgLk2WnaeAV0RERKSx89uANyTQqR1eEREREfHfgDc00Kmb1kRERETEfwNe9w6vbloTERERaez8OODVTWsiIiIi4scBb2iQUhpERERExJ8DXt20JiIiIiL4ccAbrJvWRERERAQ/DnhDA53k6qY1ERERkUbPbwPekECHdnhFRERExH8DXnVaExERERHw84A3p6AQa219T0VERERE6pHfBrzBgU6shdwC5fGKiIiINGZ+G/CGBjoBdOOaiIiISCPntwFviCfg1Y1rIiIiIo2b3wa8oUHuj6aAV0RERKRxC6jKIGNMCpAOFAIF1trE2pxUTShKaVC3NREREZHGrUoBr8doa21qrc2khgUrpUFERERE8OeUBu3wioiIiAhVD3gt8D9jzFJjzMTanFBNCVHAKyIiIiKAqUpjBmNMa2vtHmNMHDATuMNaO/eEMROBiQDx8fGDpk6dWhvzrVBGRgYREREA7E538ciCbG7rH8yZLaqTuSENXcl1Fv+kNW4ctM6Ng9a5caivdR49evTSqtxbVqVI0Fq7x/O/B40xnwNnAnNPGDMFmAKQmJhok5KSqjvnU5acnEzR++48nAULfqBjl+4kDWpT53OR2lNyncU/aY0bB61z46B1bhwa+jpXmtJgjAk3xkQWPQbOA9bU9sROVZOwQACOZObW80xEREREpD5VZYc3HvjcGFM0/gNr7YxanVUNaBIaSGRwAHuOZtf3VERERESkHlUa8FprtwH96mAuNS42KpjUzLz6noaIiIiI1CO/LUsGEBUSSHpOQX1PQ0RERETqkV8HvJEhAaRl59f3NERERESkHvl1wOve4VXAKyIiItKY+XfAGxqglAYRERGRRs6vA97IkEAOpueSnaduayIiIiKNlV8HvOv3pQHwpxkb6nkmIiIiIlJf/DrgPZTubjqx60hWPc9EREREROqLXwe8j0zoCUDv1k28x6at2sfeY2pGISIiItJY+HXAO7hjUwBe+n4zP2w8SKHL8vsPlnHFP3+q55mJiIiISF3x64A30Fn88W5852ey8903r+3RDq+IiIhIo+HXAe+JcjwBb4DD1PNMRERERKSuNKqAt6gmb36hreeZiIiIiEhd8fuAd3CHpt7H/0je4n38c8qR+piOiIiIiNQxvw94fzGojffxgbRc7+M1e44D4HJZXv5+M4fSc0lJzSStRCviFbuOkfDgNLYdyqi7CYuIiIhIjfL7gPfCvq28jzNzi9sML91xlMnTN7By9zGen7mJBz5ZSdJfk7n8Hz96x3y3dj8A36zaV3cTFhEREZEaFVDfE6htoUFOfpXYlo+W7GLJjqPe40VBbHRYIAD7jucAsOlABkcz84gICSA2IhiA1IxcREREROT05Pc7vAB/urwvQc6yP+rk6e62wxv2p3uPDXh6Jvd+vJKdJ9mh7R/JW5m/OfWkXisiIiIiNatRBLwAm549v1rjv1q5l3d/TDmp9/rTjA38+q1FgDuN4vEv15CVV1DJq0RERESkNjSagBfgpuEdSGgWxthe8Sd9jZe/31xq93broQxyCwrLHP/W/O2899MO3lmQctLvKSIiIiInz+9zeEt67MKePHZhT3YdyeLHrYe9dXkrs2FfOj9tPczny3fz8ZLdAMRGBvP5bcModFnGPD+Hywa25oUr+vu8LjO3gLwCF+CuBiEiIiIida9R7fAWads0jNVPjPU+79UqiltHdix3/OKUI1z1xkJvsAtwKD2XRduOMOovyQB8tmwP4Nu2uNfj33EkKw+AnIJCEh6cxtvzt9fkRxERERGRSlQ54DXGOI0xy40x39TmhOpDoNPBQ+N7VPt1W0+oz/vejykMnzzb59jMdQcAePWHrYA7xUFERERE6k51dnjvAtbX1kTqU782Tco9FxRQ/reo5I4vwONfra30vUruAIuIiIhI7atSwGuMaQNMAN6s3enUj4cnuHd3Fz88ptS5eQ+MZtKlfUodN6Zq9XkPpZcek/DgtJOYpYiIiIicDGNt5TdTGWM+AZ4DIoH7rLUXlDFmIjARID4+ftDUqVNreKqVy8jIICIiosrjb5iRCcC748K9x6y15LvgnuQsMvLhzfPCSM22PDjPd2d2QodApm3P52T9pncQPZs5aRbaKNOoT0l111lOP1rjxkHr3DhonRuH+lrn0aNHL7XWJlY2rtIqDcaYC4CD1tqlxpik8sZZa6cAUwASExNtUlK5Q2tNcnIy1XnfrzofIzIkkA7Nw0ud+6ZfJkt3HOWcQW0AyI7eTqfYCD5dtpsvV+zlrAHdmbZ9NQCPTOjBM9Oql+3x1po872tvPqv8G+aktOqus5x+tMaNg9a5cdA6Nw4NfZ2rUpZsOHCRMWY8EAJEGWP+ba39de1Orfb1bRNd7rmE5uEklAiEbxzeAYAzEppy4/AORAQXf+uGdWoOQNumoZzdLY73ftpR5Tk8M2091w1NqDBXWEREREROXqVRlrX2IWttG2ttAnAlMNsfgt2TFRrkpH/baDrFhjOuVwt6t46ieWQQAAPaxuB0VD9wPeopXSYiIiIiNa9RNZ6oScYYXr92kPf5Z7cNo0eLKF6YubHa1xo2eTa3JXXi3vO61eQURURERIRqNp6w1iaXdcOawMB2MYQGORnbqwUAE0s0smgdHVrhawtdlpdnb6nV+YmIiIg0VkocrWGJCU3ZNmk8D5doZDHtzhFcOqB1pa/NL3SVWcZMRERERE6eAt5a4HAYAB69oCedYsOJDAnkwn4tK31d3yf+xxnPzuJoZh5pOfnsVZMKERERkVOmHN5a9JsRHfjNCHd1h7O7x3uPv3zVALanZvLCzE0+47PzCwG4+f0lHEjLYffRbFImT6i7CYuIiIj4IQW8deidG84gKjSQQe1j2H88p1TAW2T5zqO4Ku8HIiIiIiJVoIC3Do3uHud9HB0WWO44BbsiIiIiNUc5vPUkJNDp8/zL3w8vc5xL0a+IiIjIKdEObz166cr+dI2PpHuLSIwxXDe0Pe+f0KUtt8DFgbQcsvIK6dkqqp5mKiIiInL6UsBbjy7u71uqzJaxmZuakUvSX5MBdAObiIiIyElQSkMDMqRjs1LHvl29rx5mIiIiIuI/FPA2IBP6tuS7u0f6HCtQDq+IiIjIKVHA28B0jY/web49NbOeZiIiIiLiHxTwNjDGGIaWSG34ZOnuepyNiIiIyOlPAW8D9O5NZ/DGdYmljj/37fp6mI2IiIjI6U0BbwMUHOAkLjK41PF/zt1WD7MREREROb0p4G2gggK0NCIiIiI1QVFVAxXoNGUeP56VX8czERERETm9KeBtoDrHRfLmdYmYE+LeS15bgLWWvAIXBYWu+pmciIiIyGlEAW8Ddk7PeGbfm0SLqBDvse2pmazcfZyuj0xn4NMz63F2IiIiIqcHBbwNXIfm4Sx8eIzPsTfmuW9eS8spqI8piYiIiJxWFPCeJq4d0t77eNoqtRsWERERqSoFvKeJpy/pXebx3o9/V8czERERETm9VBrwGmNCjDGLjTErjTFrjTFP1sXEpGoycpXWICIiIlKRgCqMyQXOttZmGGMCgfnGmOnW2oW1PDcRERERkVNW6Q6vdcvwPA30fNlanZVUi8qTiYiIiJTPWFt57GqMcQJLgc7Aq9ba/ytjzERgIkB8fPygqVOn1vBUK5eRkUFERESdv29duWFGJgCdmjiICzP8tK8QgH+cE8aRbEtYIIQFGoLLaVrhL/x9nUVr3FhonRsHrXPjUF/rPHr06KXW2sTKxlUlpQFrbSHQ3xgTDXxujOltrV1zwpgpwBSAxMREm5SUVP1Zn6Lk5GTq433rzIxpAIzq3Z79admwbz8AiYOHccazs7zDNjw9juU7j9G2aShtYsLqZaq1ye/XWbTGjYTWuXHQOjcODX2dqxTwFrHWHjPG/ACMA9ZUNl5qR1iQk29X7/c+v+TVBT7n7/xwOf9bdwCAlMkT6nRuIiIiIg1NVao0xHp2djHGhALnAhtqe2JSvpvP6sA1g9t5n+85lu1zfub6A3U9JREREZEGqyo7vC2B9zx5vA7gY2vtN7U7LSnLskfPxVpLdFgQF/RtxX8W7SxzXJDTQW6BbmQTERERgSoEvNbaVcCAOpiLVKJpeJD3cWiQs9xxQQEKeEVERESKqNPaaSoksPylCw4oHQwXuix7T0h9EBEREWkMFPCepkIDy9/hLXQV7+4eSMsB4MVZmxg2ebaCXhEREWl0FPCepkIqCHiPZuV7H09btQ+A5I2HADickVe7ExMRERFpYBTwnqbiIoN5/MKePP/LfhWOe+qbdYA7pQHA4VnxdXvTSMvJL+9lIiIiIn5DAe9pyhjDjcM70DwyuNKxz/9vo7d02bq9aVhrGf/3eVz31uLanqaIiIhIvVPAe5rr2Dy80jEvz97C8Wz3bu79n6yiwLPbu2LXsVqdm4iIiEhDoID3NNe2aRgbnh7HZ7cNY0z3uCq9pmTJsr/N3ESeSpiJiIiIH1PA6wdCAp0MbBfDWzecQevo0ErHL9p22Pv4pe8389+lu2pzeiIiIiL1SgGvn0m+P4nZ946qcMxv3lvi87zohjYRERERf6SA188EOh0Vliwri9NhAEjPyffW7RURERHxFwp4/VCA01RrfKCnVtnYv81l8KTva2NKIiIiIvVGAa8fCnBUb1kDnIb8Qhd7j5/87u7uo1mc/9I8DqXnnvQ1RERERGqDAl4/VN4O76D2MWUeL3BZlqQcPaX3fHt+Cuv3pfHlij2ndB0RERGRmqaA1w85jDvgDQty5/JGhwWy8KEx3HJWxzLH5xW4SDmcWeE1X0vewoeLd1b63lb3v4mIiEgDE1DfE5CaF+jZ4R3WqRl/vrwfQQEOIoID6FEQWeb4R75Y4/P8+rcX895NZ/oc+/OMjQBcdWY7Vu0+RvOIYFp5SqD9sOEgby/YXtMfQ0RERKRGKOD1Q8EBTmbdM4rW0aGEBhVXbGjfLJzW0aHeNsPlmbPpUIXnL3plAU6HYeuk8QDc+O7P3nMWbfGKiIhIw6KUBj/VOS7CJ9gtMnFk2WkNJ9pxOJOEB6eR8OA0Zq474D2+cX86oNq9IiIicvpQwNvIXD8sgfAyAuETjfpLsvfxLe8XN6oY++Jcn3HLd/re7KYcXhEREWloFPA2QsZzU9uZCU25LakTUSEBRAZXP7vF5bJc+tqPPsdqK9597Ms1fLFcFSBERESk+pTD2whdM7gd/5y7jX/dfCbBAU4eGNcdgIJCF53/OL3K1zmUUbrm7uTpG5g8fQPn9oxnyrWDvMH1qXr/px28/9MOLhnQukauJyIiIo2Hdngbof8b150NT48jOMA3tSHAWfX/HLrERbC/gkYVM9cdYMmO0rV9V+0+RkZuQdUnKyIiInKKKo1wjDFtjTE/GGPWGWPWGmPuqouJSe1xOAwhgZXn8VZk88EMLn51QYVjsvIKueDleQx6eiZfrthDZm4BF72ygLs+XH5K713Ss9PWkfDgtBq7noiIiPifqmzpFQD3Wmt7AkOA3xtjetbutKS+/HpIO+/jlY+f5308+bI+1b7W3E2HWLMnjcOZefzhoxXetsNr96YBsOdYNn2e+I4lKUfKfP3x7Hxe/n5zhRUh3pin+r8iIiJSsUoDXmvtPmvtMs/jdGA9oERKP/XMJcWBbZPQQO/jSwa0pnlEcLWu9db84mDUZSHpr8kARIa4U8fv+WgF6TkFXP76T2W+/qmv1/H8zE3M2XSwWu8rIiIiUpKx1agjZYxJAOYCva21aSecmwhMBIiPjx80derUmptlFWVkZBAREVHn7+tvDme7yCmE1hEObpjhbjn87rhwrLXc+F1WjbzHPYOC+XRzPjvSXADc1i+YL7bkMa5DICPbuAPtP/+czbrDLm7qHcTba/Lcx0aG8sDcbJ4YGkJCE6fP/MR/6N9y46B1bhy0zo1Dfa3z6NGjl1prEysbV+WA1xgTAcwBnrXWflbR2MTERLtkyZKKhtSK5ORkkpKS6vx9/dncTYcwBs7qEgvgky/boXk4D57fnVv/tfSkrl1R17fwICcx4UHsPup7/uYRHXhz/nauGdyOZy/t453PtknjcThqpiJEbdl5OIu2TUNrrHKFP9O/5cZB69w4aJ0bh/paZ2NMlQLeKt2Wb4wJBD4F/lNZsCv+ZWTXWG+wCzDrnlHexz/cl0TPllFVus7D47uXOlZRi+PMvMJSwS7Am540iYJC31/U8l0u3p6/nWNZ7p3gHzYcZM+xbL5ZtZcZa/ZXaY4AmbkF7DuezY9bUsktKKzy6yqzctcxRv7lB/69cEeNXVNERESqptI6vMa9HfUWsN5a+0LtT0kass5xESx/9FxvabGqVHu4aXgHrjqzHZO+3VBj88gpKOSDRTu9zxdvP8JT36zj55QjnN+nJXd+uJym4UEcyXQHwBueHuczV2stX6zYw/DOzYmLDPEev3LKQlbvOQ7AdUPb89TFvVm07TAdYsO94/ILXczbfIizu8dXeb7bUjMA+DnlKNcOTTjpzy0iIiLVV5XGE8OBa4HVxpgVnmMPW2u/rb1pSUMWEx5ETHgQAKEl2hT3aBnF+n1pRAYHkO4JiJuEBvLYhT2pTq54VXy5Yi9frtjrfb5sxzEAdh/N5k5P2bOiYBfg+f9tJKlbHGFBTppHBLP1UAZ/+GglAM9d1geHgW9W7fMGuwCbD7iD1F9NWUh8VDCLHj4HgBdnbeLVH7bywS2DGdapeZXma1Aag4iISH2pNOC11s4H/b+1lC0koDgr5umLe9G3TTQBDkOPx2aQW+Dyljar7bzVv83aBOATsJb0xrztPiXMnv9lP+/jhz5bXeZrHA68gfqBtFwKCt032G075L5R7mhmfpmv23IwnQCHg4Tm4WTkFnDRy/M5q4s7MFb6roiISN1Ta2E5JQFOBy2iQrhtdCcSE5p6j8//v7M5np1X7uueuaQ3YUFO7vl4pc/xP/+iLw98uqrW5lvEWYUb3BZsOcyyncXd4k5su1zeJc55YS4AKZMnsGzHUbalZrItNfPkJysiIiKnRK2F5ZQtfHgM152QlxobGUznuMgyx182sDWXDWxNiyYhPscv6d+KSwf6lnieOLIj4UFOPrhlMAsfGlNjc777oxVlHm8dHerz/Bf/KLtGMBTv1hYUunhp1uZSLZPzC10UuFw+x6qa2ZG88WCFrZtFRESk6rTDK3XuhSv6+zzv26YJsRHB3De2G4FOB1/dPpyLXllAoNPw8PgePDy+B4A3paA2hQVVp+Wyodsj08ktcM/rSGYut43u7D3b5YQd4ZK+XLGHQKeD8X1a8s6C7czZdIi3rj8Dp8NgreWGd34mLjKY5y7rw9nd41TKTERE5BQo4JU688N9ST7Pi0qa3XteN0Z1LS591rKJe5f1xLSDAKeDRyb04Llv11NYs/fAeW0+mFHlsXmFLm+wC/DeTzt476eKy459tXIvt5zVkbumuneYn7q4F09+vQ6AuZsPMbpbHD9uPQzAwfRcfvOeu571jLvPonuLKF6ctYl/L9zJV7cPp9UJu9F1bd3eNEKDnHRorqYfIiLSsCnglTpzYmAUHRZEyuQJpcYFOt2BbqCjdMbNzWd1pEX2DoJa92BiOQ0vFv9xDONfmkdqRukc4s5xEWypRlBbkaJqENV14SvzvY8f+3Kt9/H01ft44JNVHErPLfWacS/O83n+u38vpX/baN77aQef3TaMaav2MbJrLIEOQ9OIILq3cP8y8ey0dT61lI9k5tHUU2Gjug6m51BQaL2B9vi/u+dU1hpWRaHL8sXyPVwyoHWVcqpFREROlgJeaXCCA9xpBef1alHm+Yggw4jucQBc2K8VS1OO8OXtI8jKKyA1I4+4yBCWPHIuWw9lsPNwFje++7P3tQa4oG9Lvlm1D4CEZmGkHM7ikQk9eGbaegDuPbcrCc3DuaOcgLZTbDhbD9X8TWgfL9ld5bErdx9n5W53RYrPl+3hXwt38Nb84ioUG58Zh7XF1Sm++P1wLnk1WBgxAAAgAElEQVR1AQCf3TaMge1iSM1w7yD/5fK+dI0vzrfenprpbeAxoF2M9/jwybPJL7QnHeCe6OMlu3jos9Ucz87nphEdauSaIiIiZVHAKw1OaJCTnx46m2bhweWOCXA6ygi8gmnfrHgXuVNsBJ1iI7hsYGs+W7bHe/yVqwfyzappDOnYlOev6M/CrYfp17YJz0xbz9COzbhjTBf+t7b87myXDWzDofRc3v0x5WQ/Yo36Vxnd27o9MsPneVGwC+6ub//6aQefL3d/Tz5ZupvjWflM6NuSkV1jGf3XZO/Ykt/jfE8eSUGhiwDnqd/vWlQn+WAZO9oAuQWFuFy+tZ5FREROhgJeaZCK8nhrQvum7iDY6TDeG+ZWPHYuIYFOQgKd/GJQG8A3uCu6SeyMhBhGdI7lQHoOadn5fLNqHzFhQZzXM77BBLzVVZQzXGTK3G0AfLRkF2ufHOtzzlqLMcanYsSuo9nsK9EWeu3e4zz37Qbmb0nlnB5xpBzO4tlLejO4Y7My33/n4SyW7zrqTWMoqnVsrSU1I4/YSPcvOmOen8Puo9k1tqMsIiKNlwJe8Xu/H92Jbi0iGNurhTeQjQ6rOI+1Zyt3DuyNwzswvk9LwL3jOKh9DFcktsHpMFw9uB2fLt3tc+MawK8S2/LRkl2lrvnOjWdw4zs/06tVFGv3ptXER6txvR7/zuf5Le8vYcq1iQx57nvvsflbUnn0izXe5xP+XpyTPGv9QcDdnW7JI+fwwaKdNAkN5Noh7Xl59hbO6BDD1W8sAuA3njSGuZtT+e/TMzmenU+hyzLrnlGAu2tekTfnbePFWZtZc0JALiIiUhUKeMXvBTgdjOvdslqvaR0dWmpnMTjAyY3Di3NNJ13ah0mX9mHBllSueXMRN4/owB1juhAVElAq4H1kQg+Susby0pX9Ob93SwKdBmvh55QjRIQE8Pb8FD5dVn4Ob4DDUOCqpdIUFZi1/iAdH/btIl4y2K1I4jOzvI/7tY32dsMrUuj5POv3+Qb/UxfvZM6a4mB3+c6j3vzqnPxCQgKLUxystXy1ci9RoYGM6hLLZf/4kajQQN6/6cwqzVFERBoHBbwip2h45+bl/tn9g1sGM7BdjDdIu7h/cWMNY/D+2f/5K/rRu3UUC7akAjC+T0umr9nPzHUHCHQaNj87HoCej80gK6+w3LlMurQPD39edqvk9246k+vfXlz9D1gDXpi5qdSx8lJC3ixx8x3Apa/96H389DfreOKiXgR6coi/W7vfW+ItsX0MK3Yd83ltbkEhG/al069tNJsOpJOb76JPmyaAO1h+LXkrlw1sXWYKTaHLYoBJ366nTUwoVw1uR3CAkye/XsvwTs1JzcjljA5N6RQbUeXvg4iI1A8FvCK1aFin5lUee+PwDj47yBf3b02nh7/lwfN7eI/97Vf9udVTjm3VE+dxNDOPUX9J9p6vqLrXqK6xjOoay5xNhwgLclYYONe0uZsO1ch1/rNoJ1+t2MvqJ8fy8c+7WLG7OMBdsqO4DfTSHUfIL7T84aMV7DueQ+voUPZ48o4/vGUIQzs1Y+OBdP7y3UZmrjvAA2O7Mayze62+X3+A/EIXD362mmNZ+d5rfrxkN9PuHME7C1J4Z0EK4G5Usu6pceXO1+Wy9H7iO1o0CWH2vUk18j0QEZHqU8ArUgtm3H0Wa/acWp6u02FK7RyP7dWCPq2bMKZHHFEhgUSFBBIZHEB6ibbGbWJCffJfAWLCAgF447pEsvMKeXP+Nl6eveWU5lcTBndoyqLtR6r1mvTcAt6ev52nvllX7pgTW0LvKXGT3ZxNh2gSGsjHnrSTFbuOcfWbiyp933X70rw3+BUp+qXBWsu7P6YwoU9L4qKKW2ZvS80kK6+QbYcyyS90eXema5O1lkXbjzC4Q1Nvznp6Tj7frz/IJQNaV/JqERH/VPs/fUUaoe4torjcU/2hpn19xwjuPqer9/l/fzeUi/q1AtzpFbec1RGA2MhgnrmkNy9d2Z8vfj8cgKAAB03CArnn3K68c+MZ3muUzHl9ZEIPn2oNRR3xihqCFPlNJbVz4yLLLytX9Pp3bzyTsb3ivcfuGtOlwtcUqSjYrczrc7Yy/u/zTqrKxnPTN5R5fNXu4zz59TrOnPQ9f5qxgYtemU/Cg9N8drZnbzhY5fdZknKk3AYpRzPzvG22X/5+M1dO8Q3uP1m6myunLOSrlXu9x/74+Rru/mgFa/Ycr/IcRET8iXZ4RU5z3VtE8ferBvD3qwYA8ItBbfhxayoPjOtebn6pMYbR3eLY8uz5OIzB4TDcP7Ybf/luIzcMSyDA6eCxC3oyd/MhDqS56+R+fttwMnMLmLv5EMM6NWd45+a0jQllytxtNI8MZs2e45S8r+6DW4ZwzgtzOLdnPDPXHfB5/xeu6MeEvi0JDnDyz2sTcbks+S4XwQFOXvp+c6n5JraP8UlZqI5B7WNYepKvrYrr317MLxOLf7n5R/JW7+OSgfmt/1rKrxLbMn3NPlY94f6FYv7mVNbuPc6PWw/zzCW9ads0DGstl7/uDmLXPDmWiODiH9N5BS4GPD2Tqwe3Y9KlfXjekxs9Ze5WgpwOzukZ722KMnvDQW/O+F7PDndNpLFs3J9OdFgg8SV2skVEGjoFvCJ+JiI4gH9em1ilsSUbSPx+dGduS+rk/TP4TSM6cNOIDpz/UnFb48Edm/nU171heAdu8OQdn/PCHJ9dybioYGbdM5I2MWEs3HaYtXvT+Mt3GwF3846SHA5DsMO3wcSHtwyhU2w4C7cfYVyvFnR9ZHqFn+UXA9sQFuQs1YijZ8uoWg1452w6xJwq5igXVe+4+b2fvSXcivzy9Z9Y+PAY1u9L9x7728xNPHpBT+/zDE/qypfL9zDp0j7e45O+de88vzx7CxP6uiuSfLliL/3aRPPrIe3xLCkuW/VKH7uPZlFQaEloHs6Wgxn87t9L+ejWoYx9cS5Oh2HanSNIyy7gzA5Nq3xNEZH6ooBXRLyKgt2SimLiMk75aBsTypaDGbx69UAWbT9MZHAAUXHulsVJ3eJI6hbH4u1HGN0ttsLrfHPHCGYu+JmhndyBdVG6xrZJ43n0yzX8Z9FO79gnL+rF41+tBdyVLgAsln8v3MnNIzoQFuTkmiHtGdAumpdnb2Fc7xYM69SMa9+qn2oVRU4MdgH2p+Xw+pytTC6RNvHW/O0+Ae/uo1lA2esEcDgzj/d/Kg74n/pmHTPW7ufnFHfA/+s3FzHznlHERgZzp6d19vg+LSl0ufjVGe3YcTiTsS/O5cNbhnirY6RMnsArszez+WCGNy2j0GUZ9+I873kRkYZOAa+IVOiVqwbyzoLtdG8RVeG4F381gAVbUxnfp6V3l/FE71WhPm7v1k1IjSv9o8nhMPzf+d1pFR3KrSM7ciQzj7ioEFo0CSErr/imvSvPaMeSlKNcM6Q9HZq7u+xdNrCNz67y8kfPJSo0kE4lagxfNqA1ny0vbkFdMhXji98P92nP/IuBbSqsm3yyJpeRI/zG3G1Mmr6e83u34NvV7pbXGbkF3P7Bsipdc3GJmwILXJb7/ruSvcey2efpnlcUxHaKjWD2hoPk5Lt49YfiGxof+WI1q3a7c39DA0u3eX7q63VcOqA1fdo0oaDQRVZ+IV+v3MvVZ7YrNzAXEalrxlbjT1xVlZiYaJcsWVLj161McnIySUlJdf6+Ure0zv6vrtb4+rcXM2fTIa5IbMOfL+/Hom2HSWgeTnxUCHkFLm8aRcrkCcxYs59RXWMJDXIHfQfSchg86ftS13zxV/25+6MV3ud3n9OFF2dtLlVNoyGKDgv0KcVWHSmTJ9Dnie9Iz3F/xg9uGVxpWb6idT6amUdugYsWTYrzgtNy8glwGMKCtC9zutPP7MahvtbZGLPUWltpHp9+kohIo/X6rwfxw8aD3vbRJfOTgwIcLHv0XPYdd9/wNa53C5/XNg0vuz111/hIUiZPICU1k2vfXsSlA1pzVpdYwJYql1bk3nO7em9A+/mP55CTX8g9H6/wpiLUlZMNdsFdDq0o2AV3ZYjJl/Vh0vQN/PXyvhzNyucv320gPiqE28/uzLer9tHbacnJL2TwpO/JK3RxVpfmzNucylMX9+KxL9fSIiqEhQ+PqYmPRkZuAZ8t2821Q9r77DznFbg4lp1HXKTvTXhfrtjD9tRMn4ooInL6UsArIo1WaJDTG+yWpWl4ULmBbaDTwaonzuOK13+iRZMQ3rgukeU7j9GzlTv1I6F5OPMeOBuA9s3C2Z6aWeoa7954Bt1aRBLgcDB/SyqTf9GXWE85t96tm/BzylH++st+3Pffld7XDO7QlEcv6MkFL8/3udaPD57N/C2pPPDJqup9E2pIh4d8W1BvT83kV1MWAnDu3+bSPCKI1Iw8AL5ZtQ8Ap4HCmTO8r5m32d1p8LEv3XnZ+9NyfK65avcxWkWHkl/oYsXOY5zvWTtrLfuO59AqunTHvCLPTlvPh4t30r5ZOKO6xjJt1T6OZOWxcNthpq3ax7ZJ43GU6NxS1MGvZMC7aNthkjcd4v/Gda/Gd0ZEGoJKA15jzNvABcBBa23v2p+SiMjpISokkBl3j/Q+r6hiQbumYfxyUBtuGtGBrLwCjmTmk9Qtznv+o1uH+ox/8PzujO3VgiEdm/HU12tJ8+yeFo178Pzu3pzfP47vQavoULJOSJl46PzubDyQzmfL9vD8L/sxqH0MSX9NLnN+8VHB3hJ0taEo2C2psAoZdZsPpBMS6KRt0zAuemWBz7mRXWO5LakT363dzzsLUph97yg6lijF9+WKPWzYn87VZ7bjWJb7/dOy3bvYvz8hB3rnkSz+NGMDDmP4w7m+9aBnrTvA8ex87vX84nHH2Z29qRZT5m5l68FMnriolzfdRUQanqrs8L4LvAK8X7tTERHxX06H4S+/7Ffl8cEBToZ4Uixeu2YQ/164g2uHtvee/+2oTozt1YKWTUII8dxMNqZHPM9+u56v7xhBt/hI75/uX7iiv8+1LxvYms+WuW/Qu3xQGz5Zupsbh3dg+pr9ZOTke2v5At40g5uGd+DtBdu9x5+4sCev/LCV1IzaC5LBvTtcnrmbDvk09zj7+Tn8YmAbbhvdiWmr9vGCJ03kH8lbvTcwpmbk8nQZjUumzNvG9DXumwK3ldiN/2rlXm9FiyK/eXcJT1/Sm85xEd6ScB8t2cX4Pi24fXQXpq3ey51juhAcoABYpKGoNOC11s41xiTU/lRERKQsI7o0Z0SX0jeAFQVxRdo2DWPzs+MrvNamZ84nwGG8Ae8dZ3fGYeC6oe357ahOWGv5csVeRnePo0loIE98tZZ5m1NpHRPKV7cPp0loIE6HoU1MGN1aRHHVGwu91y6rrXVd+3TZ7jIraBSllDz5ddld+j4oUe5u/b7ituAnBrsAP207zDkvzOHKM9r6HP929X5vJY2N+zN48/ry76PJL3Qx/qV53HJWR64ocZ28Ahd/m7WJ25I6ERkSWOp1eQUunA6D01FxBYzdR7P4dOke7hzTGWMMhS7Lb/+9lJtHdPDJVRdpLKpUpcET8H5TUUqDMWYiMBEgPj5+0NSpU2toilWXkZFBRETZnaXEf2id/Z/WuPbd/F0mBRbeHhuGo4LyYanZLl5dnssfBoUQFVx63Izt+UzdmEfXGAc39Arm4fnlB7x9Y53sTndxJKfmqwM1RC+ODiW/EKZvz+fqHkGkZluaBBtCAwzHcy13/eCuqzxpRCitItwFr+ftzuetNXkMaelkVJtAejRzsifDRZADYsMc3DAjkwFxTu4aWLrT3e9mZdIp2sl9iSE8/mM2O9Jc3msfyXFxT3I2kYHw8pjwUq+tTfr33DjU1zqPHj26SlUaaizgLUllyaQ2aZ39n9a49q3bm0bypoPcltT5lK6TmVvAn2Zs4P6x3YgMCWTx9iP0ad2EBVtSufn9JXxwy2A+/nkXX6zYy4anxxES6OR4Vj79nvofAFuePZ+nvlnn0zCjSMfm4eTkF7L3eE6pc6ezSZf24ZtVe/lx62HvsWl3juBYVj7XvLnIZ+z/jevOn2aUrs9868iO3De2G4GezjBvzN3Gs9+uB9wl4hIenAbAK1cPYHzvllzy2gJW7T5OdFgg8x4YzU3v/szjF/aid+smpa6dk19IWnY+cTXQPvrfC3ewd/tmHrjqnArH5RYUkpPnoklY6V1tOT2oLJmIiDQ4PVtFeStKnIrw4ACeurh4L6Toxr1zesZ7u7ANah/Dved18+YalwxqApwOnrq4N18s30NaTgErHjuXf8zZyoeLdjL7viSe/99GXp69hZJaR4fy2jUDeeCTVWw8kE5i+xiW7DhKXGQwB9OrllN8ZkJTzuzQlFd+2FL54Ao4He50gep4+PPVpY5N+Pv8MkZSZrAL8M+52+gUF8FXK/Yyf0uqzzlXifnc/sFy4iLXeb8vx7Ly6fOE+5eNq6YsxOEw3HdeV/61cAdDOzbjUEauNy2jZBe9fcezycl3+aTRFBS6WLn7OIPax5T7WR/5Yg0A/fvu57xeLcodd8v7S5m76VCVO/el5eQT5HR4/5sSqYwCXhERqVXBAe4qCycKcBQ/fu+mM/nPop1EhQTy0Pk9eOj8HgDcOaYLY3rEs3LXMVbuOsYlA1ozsqu7PfWHE4ew52g23VpEcueHy7n73C7M25TKhz/v5LyeLcjJL+SGYQk8+fVafth4yFsa7b7zunL72V1YtfuYT8CbMnkCgyfNqrBaxeu/HsSOw5lMnrEBa+H+sd3K7JBXF8orQdfxYd8SceX9ElDUCOVRTxm4TQcyfM7n5BcS6HSw60iWt7rHx7cOZWC7aGZvOMjMdQf479LdfHf3SNJz8jmcmceQjs1oEur+hWbD/uJc6In/WsrqJ87j8S/XsulgOt/ccZbPexXdfPjxkl1ckdgWl8uSnltAVEhAmR37+j7xP/q0bsLXd4wo87OJnKgqZck+BJKA5saY3cDj1tq3antiIiLiv+beP5oVS4r/fD+gXQwD2pXeKQx0OujfNpr+baNLnStZJ/n1awcB0L1FFLeM7Ogz7vVrB5GdV0hWXiHvLNjOzWe5z/dtE82wTs34ceth5j0wGoBPfzeM/Z6avsMmzwZgzZNj6f34d0BxA5K/zdpETr6LhGanlg/rMFDNDWKaRwTXenUMgO6Pzih17Ip//kRwgIPcApf32NgXfStprHriPCKDAxj34jyf40U7y+CundzhoW+5fXRn2jcr/mXovR9TuKhfK25+b4l353rmH0bSJT6S41n5HMrIoXNcJACr9xyv8mfZdiiDnHwX+YUujHGv/YkKCl28NX871w1NIDjAwcJthxnaqVm9tch2uSypmbmlmqLIyXFUNsBae5W1tqW1NtBa20bBroiInKp2zcLKvAmuNgQHOIkOC6JVdCh/nNDT58/gH9wyhJTJE7w70G1iwkhMaEqr6FAWPTyG5PuSiAguvTd0cb/WAPRt04SHx5duRLG4ih3itj03gQ1Pj+O/vx1a4bj//WEkLZuE0CIqhNn3jeIP9dgBrmSwW5ZN+9NLNSI5UVHpt1d+2ML9JXaqR3WN5f8+XeWTplFUmu5XU37inBfmMmvdAe+57amZTF+9j73Hsvndv5ey+6j7RsDdR7N44JOV5Hnmevbzcxj/93lc/OoCLnplAcez88nOK+T2D5axZs9x9h/P4aXvN/Pc9A1c/85iPli8k6vfXORN76jMofRc/rVwR5kl76rCWsuaEgF8XoGLX7+1iDOf/Z79x3PIyitg3d60Cq4glVFKg4iISBniS9y0Ne3OERiKA/RJl/XhmiHtaBUdysSRnVi+8xjT1+zn69tHEBLoIC4qhFVPnMeKnce47u3FPDCuG7cldSY1I5f7/ruS5I3FHdtCAp2ckdCU9U+N4/YPlvH9hoMArH1yLIMnfU9GbgFd4yP56aHiIPqmEQn0bduEQe1jWLbjKDe887PP3M9MaMrilCPe5x2bh5OZV8CBtFz6tWkCxrBy1zHA3do6O7+Q15K31sj37fLXy26hXdKY5+eUeXza6n3sOJxV6vg9H61gw/50AG5+v/im+NEnNFIpqqVc5OMlu3m+jPrXv/rnT97rFXX+K7J4+xEWb3d/7zbsT2NUt1hc1pKTV0h6bgGRIQH89l9LWbMnjbxCF1snjeeMZ2d5X985LoKHPlvNuzee4W0u8/Q363hr/nafHOWCQhdvzNvOdUPbM33Nfu7770reuj6RMT3ieW76eu9NjYfSc3niq7XMWOv+bNufG3/Su857j2Uza/0Brhua4D325Ndr+c/CnWx69vyTuubpQgGviIhIJXq18q1m4HQYnz+Lv3r1QLLyC312g6NCAhnZNZa594+mTYy77XHziGDevfHMMt8jNMjJG9clknI4k80HMwgPDmDmPSMJcJT+Y2xkSCCjPcFU9xbFNx/eNaYLL32/GWPgv78dSmp6Lr/7zzLyXS5iwoI4kJbLf24ZQkRwgLeSwx1juvCqJ5f596M7sedoNl+s2FvqPSf0acm01ftKHa/MBR0D+WZbfpXGlhXsAny2fE+137fIvSVacxcpCnYr8/LsLaVumjzRayfc+PjQZ+6bEm9452cuG9iafcdy+GmbO3idtmofE/q6W2J/u2Y/f5qxga9X7mWdp/bz2wu2M6h9jM9u77HsPGatL97VzsgtKFWjeemOo7w4axN/vrwvMWFB5d7Md+eHy1my4yiju8XRJiYUYwzvLEip8PMdzsilWUSwz7EN+9M4lpXvbY5zOqg0pUFEREQq5nCYMlMfwJ2+4aikUUTJ63SMjWCsp6JByyahxEYGV/iapuFBNAkN5OWrBvD70Z25uH8rJv+iL2ckNGW4p2FJeFAAb16fyLs3nuGd57k947050NcPS+DXQ9rx21GdvOXI7j7Ht8VyoNPw9g2+1Z8Sy6nQ8NKVxd39usY4WPJI6bJkA9sV/8IQGXL67r897+noV5bPlu3xBrvgbmmd8OA0DqXn8s857h31dSUanSzYcpj+T83k55Sj3mPXvrWYghKJ3rM9fwEAd8OUjfvT+ePnq5m3OZWhz82m+6MzGP+Sb/50kez8QsBdKaTDQ99WWGEkJ7+QP83YwKBnZrF8p3s+uQWF3D11OeNenMeVUxZy6WsLKCisOMWloTh9/wsTERERggIcrHz8PO/zl64c4H0cFRLI368awIC20bSJCaNNTPENYm9cVxy8RgQH8MwlfQC459yutGsaxjWD2zGyayxtokO59LUfuXVUJ7q3iOTWUR3ZsC+dOZsOMaxzc64flsDo7nFsPZjBxa8u4OzucVzcvzUT+rRk5roDhKRuoHlEsPfP+S/M3MTfv9/MhxOH8OCnq/l8+R5WPHYe21MzOOeF4hvgggIcrH9qHDe8s5h5m31LrwH0axvN7aM78+gXa9ifdnrVai6ZAlFdd01dQfOIYB79Yo03FzosyHdHd92+NO8O/h/H9+BwZh5nd49j1xH3DnrR9/NAie9bwoPTuLBfK16+agCXvraAjfvTycpzB8h7j+UQFpTOKz9s4euVxbv/y3ce40hmHvvTcsjKb9gNZRTwioiI+LGL+rWq1viQQCe/HtIegIGeyhkLHjzbe/6h83vwzoLtzNl0iF6tory70f3aRpN8X5K36kKA08H5fVqSnLzR5/r3nNuVe85133T3p1/05dELeuJ0GDrHRXLLWR04mpVP+6ZhXDcsAafD8K/fDOZQei4vzNxISKCTxPZNiY8Kpk+bJgQHODG483pbNQkhMMCBtbDzSBZnJjTl8kFteGHmJp6+pDdfr9zLVytLp2qcjk5sUFIUmJalqCHJ63NK52i/luybjvH1yr30bxvN8p3HfI5/u3pfueksX6zYw59mbGR8hwAqbmxevxTwioiISLVcNzSBTrERnOVJmSiS0Lx6ZdqCAhw0DQjyPv/jhJ5ljouNDOa5y/qWee6cnvFsfGYcBoPTYXAYSMsuICrUXcP3ijPaAu4UjpW7j7HjcBav/3oQi7cf4dNlu5l25wje/2kH7/2YwlldYnnl6gFllmSD4kYjQQEOrLXkF1oS28dwTs/4UvWYH7ugJ099s45RXd3XHDLpezJPCEwjgwO89ZBLGtG5OYkJMbw4azPgTh3p1iKS/yzaWc538uT8e2Hp65VVaaKi3O1J37o/d9uIhp0lq4BXREREqsXpMN4GIA1BcIDvn/TLa1H80cShrNlznHN6xjOudwseu9AdYD88vgcPj+/hHffXX/ZjSMem5OQX0r5ZOF8s30P7ZuG0bBLC2c8n89nvhtGjZRQz1uzn/N4tcDgMlw1ozXs/pTC2Vwt6tozyVlm4bGBrIkMCWfvUOLYdyuCBT1axZMdRLh/Uhr/+sp839aDIrHtG0TkuAoDze7fkWFYegz03h43r3YJr31pcpe/J/WO7MaprLBe8XHYXv1PVLDyIw5l53uddYxTwioiIiNS7Fk1CaNGk8kYOlw9q4/P8l4ltvY83P1v8h/uiigsAcVEh3D+2uCbzyK6xzL53FB1jI7zHOsZGMHXiELanZtIl3t1A46+/7MffZm7ieHY+kSEB3mAXoFuLSJ95nNUlls9vG8ah9FwOpOfSq1UUd01dzq4j2Xxwy2CufqNkM5doerQsruAxqmssczwd7W5L6sRZXWK56o2F3vMjOjf3qX/8zCW9iQwJ4K6pKwC4dVRHCgstb87fDkCPllHe8ZcNbE10iG8aREOjgFdERESkFpQMdosEOB3eYBfcwfXlg9qQk19IVcrrntiRcMZdI8ktcNE0PIg59yfRvlk4qRm5NPeUEivqzPfqNQO5/78reXh8D2+jlX9eOwiXy/K7/yzjlpEdee6yPrw5bxuz1h/kmsHtMMZ4A94Hx3XHGOMNeM/uHsefL+9L03B3GbTk5OST+RbVGQW8IiIiIvWsvNq5lQkPDiDcU7muvafVdfMSdXP/94eRHM/OJyI4gH/8epDPa4tuOITVWH8AAAURSURBVNzw9Djv+z95cW+evLh4zKx7RrFq9zFvs4sPbxlC8saD3DAsocrl9hoCBbwiIiIifqppeJC33nJ5Kgq2O8dF+KRZDO3UjKGdTp+GE0UadoaxiIiIiMgpUsArIiIiIn5NAa+IiIiI+DUFvCIiIiLi1xTwioiIiIhfU8ArIiIiIn5NAa+IiIiI+DUFvCIiIiLi14y1tuYvaswhYEeNX7hyzYHUSkfJ6U7r7P+0xo2D1rlx0Do3DvW1zu2ttbGVDaqVgLe+GGOWWGsT63seUru0zv5Pa9w4aJ0bB61z49DQ11kpDSIiIiLi1xTwioiIiIhf87eAd0p9T0DqhNbZ/2mNGwetc+OgdW4cGvQ6+1UOr4iIiIjIifxth1dERERExIcCXhERERHxa34R8BpjxhljNhpjthhjHqzv+cipMcakGGNWG2NWGGOWeI41NcbMNMZs9vxvjOe4Mcb83bP2q4wxA+t39lIeY8zbxpiDxpg1JY5Ve12NMdd7xm82xlxfH59FylfOOj9hjNnj+Te9whgzvsS5hzzrvNEYM7bEcf1cb6CMMW2NMT8YY9YZY9YaY+7yHNe/Zz9SwTqfnv+erbWn9RfgBLYCHYEgYCXQs77npa9TWtMUoPkJx/4MPOh5/CDwJ8/j8cB0wABDgEX1PX99lbuuI4GBwJqTXVegKbDN878xnscx9f3Z9FXpOj8B3FfG2J6en9nBQAfPz3Knfq437C+gJTDQ8zgS2ORZS/179qOvCtb5tPz37A87vGcCW6y126y1ecBU4OJ6npPUvIuB9zyP3wMuKXH8feu2EIg2xrSsjwlKxay1c4EjJxyu7rqOBWZaa49Ya48CM4FxtT97qapy1rk8FwNTrbW51trtwBbcP9P1c70Bs9bus//f3h27RhFFURz+HYxaqIUgbBEFg6SPVoIiqRYsbSQWKiJokRTWNoI2NtpaiOlUCGgwRVD/Aw2KoDGNhWJCTApBBavEa/HewhKyIRvCZuZxvmbevBl2Bw53uPBmdiPe5/EfYA7ox/VclA1y7qTS9VxCw9sPfG/bn2fjQKz6Angt6Z2ka3muERGLefwDaOSx86+3bnN13vU1lpezx1tL3Tjn2pN0FDgOvMH1XKw1OUMN67mEhtfKczoiTgBngVFJZ9oPRlo78e/pFca5Fu0BcAwYAhaBezt7ObYdJO0HngE3IuJ3+zHXcznWybmW9VxCw7sAHGnbP5znrKYiYiFvl4FJ0nLIUutRhbxdzqc7/3rrNlfnXUMRsRQRqxHxD3hIqmlwzrUlaTepCXocEc/ztOu5MOvlXNd6LqHhnQEGJQ1I2gOMAFM7fE22RZL2STrQGgNN4BMp09YbvJeBF3k8BVzKbwGfBH61LalZ9XWb6yugKelgXkZr5jmrsDXP1Z8j1TSknEck7ZU0AAwCb/F9vdIkCXgEzEXE/bZDrueCdMq5rvXc1+sv3G4RsSJpjFQku4DxiJjd4cuyrWsAk6nO6AOeRMRLSTPAhKSrwDfgfD5/mvQG8BfgL3Cl95dsmyHpKTAMHJI0D9wC7tJFrhHxU9Id0g0U4HZEbPYFKeuBDjkPSxoiLXF/Ba4DRMSspAngM7ACjEbEav4c39er6xRwEfgo6UOeu4nruTSdcr5Qx3r2XwubmZmZWdFKeKTBzMzMzKwjN7xmZmZmVjQ3vGZmZmZWNDe8ZmZmZlY0N7xmZmZmVjQ3vGZmZmZWNDe8ZmZmZla0/xaIVVBbQPXHAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 864x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean loss=0.593\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 2500/2500 [04:41<00:00,  8.89it/s]\n"
     ]
    }
   ],
   "source": [
    "for _ in trange(2500):\n",
    "    step = len(metrics['train_loss']) + 1\n",
    "    batch_ix = np.random.randint(len(train_inp), size=batch_size)\n",
    "    feed_dict = {\n",
    "        inp: inp_voc.to_matrix(train_inp[batch_ix]),\n",
    "        out: out_voc.to_matrix(train_out[batch_ix]),\n",
    "    }\n",
    "    \n",
    "    loss_t, _ = sess.run([loss, train_step], feed_dict)\n",
    "    metrics['train_loss'].append((step, loss_t))\n",
    "    \n",
    "    if step % 100 == 0:\n",
    "        \n",
    "        clear_output(True)\n",
    "        plt.figure(figsize=(12,4))\n",
    "        for i, (name, history) in enumerate(sorted(metrics.items())):\n",
    "            plt.subplot(1, len(metrics), i + 1)\n",
    "            plt.title(name)\n",
    "            plt.plot(*zip(*history))\n",
    "            plt.grid()\n",
    "        plt.show()\n",
    "        print(\"Mean loss=%.3f\" % np.mean(metrics['train_loss'][-10:], axis=0)[1], flush=True)\n",
    "        \n",
    "# Note: it's okay if bleu oscillates up and down as long as it gets better on average over long term (e.g. 5k batches)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Посмотрим, что вышло"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 263,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "inp_line display the id for those employees who did two or more jobs in the past .\n",
      "real query select employee_id from job_history group by employee_id having count (*) >= 2\n",
      "generated query select statement_id , last_name from employees order by birth_date desc limit 1\n",
      "\n",
      "inp_line what are the last names and ages of the students who are allergic to m@@ il@@ k and cat ?\n",
      "real query select lname , age from student where stuid in ( select stuid from has_allergy where allergy = \" mil@@ k \" intersect select stuid from has_allergy where allergy = \" cat \")\n",
      "generated query select count (*) from faculty\n",
      "\n",
      "inp_line which cities have 2 to 4 parks ?\n",
      "real query select city from park group by city having count (*) between 2 and 4 ;\n",
      "generated query select count (*) from video_games\n",
      "\n",
      "inp_line what are the names of the different artists from banglade@@ sh who never received a rating higher than a 7 ?\n",
      "real query select distinct artist_name from artist where country = \" bangladesh \" except select distinct artist_name from song where rating > 7\n",
      "generated query select distinct building_description from apartment_buildings\n",
      "\n",
      "inp_line for each payment method , how many payments were made ?\n",
      "real query select payment_method_code , count (*) from customer_payments group by payment_method_code ;\n",
      "generated query select distinct t1 . name from physician as t1 join prescribes as t2 on t1 . employeeid = t2 . physician join procedures as t3 on t3 . code = t2 . treatment where t1 . name = \" john wen \"\n",
      "\n",
      "inp_line return the name of the mar@@ k@@ e@@ ting reg@@ ion the store ro@@ b d@@ in@@ n@@ ing is located in .\n",
      "real query select t1 . marketing_region@@ _name from marketing_region@@ s as t1 join stores as t2 on t1 . marketing_region_code = t2 . marketing_region_code where t2 . store_name = \" ro@@ b d@@ in@@ n@@ ing \"\n",
      "generated query select t1 . name from customer as t1 join loan as t2 on t1 . cust_id = t2 . cust_id group by t1 . cust_name order by count (*) desc limit 1\n",
      "\n",
      "inp_line which trip started from the station with the largest doc@@ k count ? give me the trip id .\n",
      "real query select t1 . id from trip as t1 join station as t2 on t1 . start_station_id = t2 . id order by t2 . do@@ ck@@ _@@ count desc limit 1\n",
      "generated query select count (*) from products where product_category_code = \" seeds \"\n",
      "\n",
      "inp_line who performed the song named \" b@@ ad@@ lan@@ ds \"? show the first name and the last name .\n",
      "real query select t2 . firstname , t2 . lastname from performance as t1 join band as t2 on t1 . bandmate = t2 . id join songs as t3 on t3 . songid = t1 . songid where t3 . title = \" b@@ ad@@ lan@@ ds \"\n",
      "generated query select count (*) from student where age = 18\n",
      "\n",
      "inp_line find the number of customers in the banks at new york city .\n",
      "real query select sum ( no_of_customers ) from bank where city = ' new york city '\n",
      "generated query select t1 . name from genres as t1 join tracks as t2 on t1 . id = t2 . genre_id where t1 . name = \" rock \" and t1 . name = \" jazz \"\n",
      "\n",
      "inp_line what are the names of students who have taken statistics courses ?\n",
      "real query select t3 . name from course as t1 join takes as t2 on t1 . course_id = t2 . course_id join student as t3 on t2 . id = t3 . id where t1 . dept_name = ' statistics '\n",
      "generated query select distinct name from races where year between 2014 and 2017\n",
      "\n",
      "inp_line what are the names of all the dorms that can ac@@ com@@ date more than 300 students ?\n",
      "real query select dorm_name from dorm where student_@@ capacity > 300\n",
      "generated query select t1 . name from medication as t1 join student_course_enrolment as t2 on t1 . id = t2 . genre_id where t1 . name = \" john wen \"\n",
      "\n",
      "inp_line show each gender code and the corresponding count of guests sorted by the count in descending order .\n",
      "real query select gender_code , count (*) from guests group by gender_code order by count (*) desc\n",
      "generated query select decor from reservations where zip_code = ' good ' group by zip_code order by count (*) desc limit 1 ;\n",
      "\n",
      "inp_line what is the average base price of rooms , for each bed type ?\n",
      "real query select bedtype , avg ( baseprice ) from rooms group by bedtype ;\n",
      "generated query select distinct t1 . state from college as t1 join tryout as t2 on t1 . cname = t2 . cname where t2 . ppos = ' goalie '\n",
      "\n",
      "inp_line find the ids of the employees who does not work in those departments where some employees works whose man@@ a@@ ger id with@@ in the range 100 and 2@@ 00 .\n",
      "real query select * from employees where department_id not in ( select department_id from departments where manager_id between 100 and 2@@ 00 )\n",
      "generated query select t1 . first_name , t1 . last_name from employees as t1 join customers as t2 on t1 . customer_id = t2 . customer_id join addresses as t3 on t2 . address_id = t3 . address_id group by t2 . customer_id order by count (*) desc limit 1\n",
      "\n",
      "inp_line find the name of students who didn ' t tak@@ e any course from biology department .\n",
      "real query select name from student where id not in ( select t1 . id from takes as t1 join course as t2 on t1 . course_id = t2 . course_id where t2 . dept_name = ' biology ')\n",
      "generated query \n",
      "\n"
     ]
    }
   ],
   "source": [
    "for inp_line, query, generated_query in zip(test_inp[:50], test_out[:50], model.translate_lines(test_inp[::50])[0]):\n",
    "    print(\"inp_line\", inp_line)\n",
    "    print(\"real query\", query)\n",
    "    print(\"generated query\", generated_query)\n",
    "    print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 250,
   "metadata": {},
   "outputs": [],
   "source": [
    "candidate_list = []\n",
    "reference_list = []\n",
    "\n",
    "for inp_line, query, generated_query in zip(test_inp, test_out, model.translate_lines(test_inp)[0]):\n",
    "    candidate_list.append(generated_query)\n",
    "    reference_list.append(query)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Metrcic: accuracy\n",
    "+1 к accuracy, если последовательность символов в запросе полностью совпадает с оригиналом"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 251,
   "metadata": {},
   "outputs": [],
   "source": [
    "def is_all_same(c1, c2):\n",
    "    if len(c1) == len(c2):\n",
    "        all_same = True\n",
    "        for j in range(len(c1)):\n",
    "            if c1[j] != c2[j]:\n",
    "                all_same = False\n",
    "                break\n",
    "        return all_same\n",
    "    else:\n",
    "        return False\n",
    "\n",
    "\n",
    "def compute_accuracy(candidate_list, reference_list):\n",
    "    if len(candidate_list) != len(reference_list):\n",
    "        print(\"candidate list has length {}, reference list has length {}\\n\".format(len(candidate_list), len(reference_list)))\n",
    "\n",
    "    len_min = min(len(candidate_list), len(reference_list))\n",
    "    c = 0\n",
    "    for i in range(len_min):\n",
    "        #print(candidate_list[i])\n",
    "        #print(reference_list[i])\n",
    "        if is_all_same(candidate_list[i], reference_list[i]):\n",
    "            #print(\"above was all same\")\n",
    "            c = c+1\n",
    "    return c/float(len_min)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 252,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.007132667617689016"
      ]
     },
     "execution_count": 252,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "compute_accuracy(candidate_list, reference_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Attention"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "После работы энкодера есть последовательность его скрытых состояний, каждое из которых относиться к токену входной последовательности $ h^e_0, h^e_1, h^e_2, ..., h^e_T$ \n",
    "Также есть скрытое состояние декодера с прошлого шага $h^d$,\n",
    "\n",
    "* Считаем вот такое\n",
    "$$a_t = linear_{out}(tanh(linear_{e}(h^e_t) + linear_{d}(h_d)))$$\n",
    "* Считаем вес, с которым каждое слово во входной последовательности повлияет на декодер state.\n",
    "$$ p_t = {{e ^ {a_t}} \\over { \\sum_\\tau e^{a_\\tau} }} $$\n",
    "\n",
    "* Добавляем ко внутренему состоянию декодера на предыдущем шаге взвешенную сумму внутренних состояний энкодера.\n",
    "$$ attn = \\sum_t p_t \\cdot h^e_t $$\n",
    "\n",
    "Чуть подробнее тут(https://distill.pub/2016/augmented-rnns/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class AttentionLayer:\n",
    "    def __init__(self, name, enc_size, dec_size, hid_size, activ=tf.tanh,):\n",
    "\n",
    "        self.name = name\n",
    "        self.enc_size = enc_size \n",
    "        self.dec_size = dec_size \n",
    "        self.hid_size = hid_size \n",
    "        self.activ = activ       \n",
    "\n",
    "        with tf.variable_scope(name):\n",
    "            \n",
    "            self.linear_enc = L.Dense(hid_size)\n",
    "            self.linear_dec = L.Dense(hid_size)\n",
    "            self.out = L.Dense(1)\n",
    "\n",
    "            # prepare to translate_lines\n",
    "            self.inp = tf.placeholder('int32', [None, None])\n",
    "\n",
    "    def __call__(self, enc, dec, inp_mask):\n",
    "        with tf.variable_scope(self.name):\n",
    "            \n",
    "            \n",
    "            adds = self.linear_enc(enc) + self.linear_dec(dec)[:, None, :]\n",
    "            tanh = tf.tanh(adds)\n",
    "            logits = self.out(adds)[:, :, 0]\n",
    "            print (logits.shape)\n",
    "\n",
    "            \n",
    "           \n",
    "            a = tf.equal(inp_mask, 0.0)\n",
    "            b = logits\n",
    "            c = tf.fill(tf.shape(logits), -1e9)\n",
    "            \n",
    "            logits = tf.where(a, logits, tf.fill(tf.shape(logits), -1e9))\n",
    "            \n",
    "            probs = tf.nn.softmax(logits)\n",
    "            \n",
    "           \n",
    "            attn = tf.reduce_sum(tf.multiply(enc,probs), axis=2)\n",
    "            attn = tf.reduce_sum(probs[:, :, None] * enc, axis=1)\n",
    "            \n",
    "            return attn, probs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Seq2seq с вниманием(attention)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class AttentiveModel(BasicModel):\n",
    "    def __init__(self, name, inp_voc, out_voc,\n",
    "                 emb_size=64, hid_size=128, attn_size=128):\n",
    "        self.name = name\n",
    "        self.inp_voc = inp_voc\n",
    "        self.out_voc = out_voc\n",
    "\n",
    "        with tf.variable_scope(name):\n",
    "            \n",
    "            \n",
    "            self.emb_inp = L.Embedding(len(inp_voc), emb_size)\n",
    "            self.emb_out = L.Embedding(len(out_voc), emb_size)\n",
    "            self.enc0 = tf.nn.rnn_cell.GRUCell(hid_size)\n",
    "            \n",
    "            self.attn_layer = AttentionLayer(\"attn1\", emb_size, emb_size, hid_size)\n",
    "\n",
    "            self.dec_start = L.Dense(hid_size)\n",
    "            self.dec0 = tf.nn.rnn_cell.GRUCell(hid_size)\n",
    "            self.logits = L.Dense(len(out_voc))\n",
    "\n",
    "            self.inp = tf.placeholder('int32', [None, None])\n",
    "            self.initial_state = self.prev_state = self.encode(self.inp)\n",
    "            self.prev_tokens = tf.placeholder('int32', [None])\n",
    "            self.next_state, self.next_logits = self.decode(self.prev_state, self.prev_tokens)\n",
    "        \n",
    "        self.weights = tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES, scope=name)\n",
    "        \n",
    "\n",
    "    def encode(self, inp, **flags):\n",
    "        # encode input sequence, create initial decoder states\n",
    "        inp_lengths = infer_length(inp, self.inp_voc.eos_ix)\n",
    "        inp_mask_ = infer_mask(inp, self.inp_voc.eos_ix)\n",
    "        inp_emb = self.emb_inp(inp)\n",
    "        with tf.variable_scope('enc0'):\n",
    "            _, enc_last = tf.nn.dynamic_rnn(\n",
    "                              self.enc0, inp_emb,\n",
    "                              sequence_length=inp_lengths,\n",
    "                              dtype = inp_emb.dtype)\n",
    "        \n",
    "        print (\"enc_last shape \", _.shape)\n",
    "        dec_start = self.dec_start(enc_last)\n",
    "        print (\"enc_last shape \", dec_start.shape)\n",
    "        \n",
    "        #!!! Attention мы добавляем в энкодер. Вот тут вся разница\n",
    "        first_attn, first_attn_probas = self.attn_layer(inp_emb, dec_start, inp_mask_)\n",
    "        \n",
    "        first_state = [dec_start, inp_emb, inp_mask_, first_attn_probas]\n",
    "        return first_state\n",
    "\n",
    "    def decode(self, prev_state, prev_tokens, **flags):\n",
    "       \n",
    "        [dec_start, inp_emb, inp_mask_, first_attn_probas] = prev_state\n",
    "        \n",
    "        \n",
    "        next_attn_response, next_attn_probas = self.attn_layer(inp_emb, dec_start, inp_mask_)\n",
    "        \n",
    "        prev_emb = self.emb_out(prev_tokens[:,None])[:,0]\n",
    "        \n",
    "        print (\"shapes\", prev_emb.shape, next_attn_response.shape)\n",
    "        \n",
    "        with tf.variable_scope('dec0'):\n",
    "            aaa = tf.concat([next_attn_response, prev_emb], axis=-1)\n",
    "            new_dec_out, new_dec_state = self.dec0(aaa, dec_start)\n",
    "        output_logits = self.logits(new_dec_out)\n",
    "\n",
    "        next_state = [new_dec_state, inp_emb, inp_mask_, next_attn_probas]\n",
    "        return next_state, output_logits\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "enc_last shape  (?, ?, 128)\n",
      "enc_last shape  (?, 128)\n",
      "(?, ?)\n",
      "(?, ?)\n",
      "shapes (?, 64) (?, 64)\n"
     ]
    }
   ],
   "source": [
    "tf.reset_default_graph()\n",
    "sess = tf.InteractiveSession()\n",
    "model = AttentiveModel('model_attn', inp_voc, out_voc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training attentive model\n",
    "\n",
    "We'll reuse the infrastructure you've built for the regular model. I hope you didn't hard-code anything :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "enc_last shape  (?, ?, 128)\n",
      "enc_last shape  (?, 128)\n",
      "(?, ?)\n",
      "(?, ?)\n",
      "shapes (?, 64) (?, 64)\n"
     ]
    }
   ],
   "source": [
    "inp = tf.placeholder('int32', [None, None])\n",
    "out = tf.placeholder('int32', [None, None])\n",
    "\n",
    "loss = compute_loss(model, inp, out)\n",
    "train_step = tf.train.AdamOptimizer().minimize(loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "metrics = {'train_loss': []}\n",
    "sess.run(tf.global_variables_initializer())\n",
    "batch_size = 32"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAArwAAAEICAYAAAC9CSe9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xl4lNXd//H3mZmsZAMCYSeALLIvUVEQg6IiWK3a+ljXWlvaPtrt1+qjtla7Ka3WVqvVqrWbW61b3RXEKLiwy75DIBAghJB9me38/pjJkJAVSDLJzOd1Xbm4t5n7mznMXJ+cOfe5jbUWEREREZFI5Qh3ASIiIiIi7UmBV0REREQimgKviIiIiEQ0BV4RERERiWgKvCIiIiIS0RR4RURERCSiKfCKiIiISERT4BURaSfGmMeNMXed5HP83Rjz67aqSUQkGrnCXYCISGdljMkFvmmtXXgij7fWfqdtKxIRkROhHl4RkRNgjFGHgYhIF6HAKyLSCGPMv4BBwBvGmHJjzG3GGGuMuckYswdYFDzuP8aYA8aYEmPMx8aYMXWeIzQcwRiTbYzZa4z5sTGmwBiz3xhz4wnU9S1jzHZjTJEx5nVjTL/gdmOM+UPwuUuNMeuMMWOD++YYYzYaY8qMMfuMMT9pg5dIRKTLUOAVEWmEtfY6YA/wJWttEvBicNc5wKnAhcH1d4DhQG9gFfBsM0/bB0gF+gM3AY8aY7q3tiZjzLnAfcCVQF9gN/BCcPcFwAxgRPAcVwKHg/v+CnzbWpsMjCUY1kVEooW+khMROT73WGsralestU/XLhtj7gGOGGNSrbUljTzWA/zSWusF3jbGlAMjgc9bee5rgKettauC57sjeL7M4HMnA6OAZdbaTcecd7QxZo219ghwpJXnExGJCOrhFRE5Pnm1C8YYpzFmvjFmhzGmFMgN7kpv4rGHg2G3ViWQdBzn7kegVxcAa205gV7c/tbaRcAjwKNAgTHmCWNMSvDQK4A5wG5jzEfGmDOP45wiIl2eAq+ISNNsC9uuBi4FZhEYRpAZ3G7aqZ58YHDtijGmG9AT2AdgrX3YWjsFGE1gaMOtwe3LrbWXEhh28RpHh2eIiEQFBV4RkaYdBIY2sz8ZqCHQy5oI3NvO9TwP3GiMmWiMiQueb6m1NtcYc5ox5gxjTAxQAVQDfmNMrDHmmuAwCw9QCvjbuU4RkU5FgVdEpGn3AT8zxhQDX2lk/z8JDDHYB2yk9WNxT0hwPuC7gJeB/cAw4Krg7hTgSQLjc3cTCOH3B/ddB+QGh118h8BYYBGRqGGsbewbOxERERGRyKAeXhERERGJaAq8IiJhZozZELy5xbE/GnogItIGNKRBRERERCJau9x4Ij093WZmZrbHUzeroqKCbt26dfh5pWOpnSOf2jg6qJ2jg9o5OoSrnVeuXFlore3V0nHtEngzMzNZsWJFezx1s3JycsjOzu7w80rHUjtHPrVxdFA7Rwe1c3QIVzsbY3a3fJTG8IqIiIhIhFPgFREREZGIpsArIiIiIhFNgVdEREREIpoCr4iIiIhENAVeEREREYloCrwiIiIiEtEiJvC+tHIvH+V5wl2GiIiIiHQyERN4//vFPnL2esNdhoiIiIh0MhETeHsnx1NSY8NdhoiIiIh0Mq0OvMYYpzFmtTHmzfYs6ERlpMRxpNpSXOkOdykiIiIi0okcTw/vD4BN7VXIybpgTB8s8O76A+EuRUREREQ6kVYFXmPMAGAu8FT7lnPiJgxIJdYJ2wrKw12KiIiIiHQire3h/SNwG+Bvx1pOijGG7nGGA6XV4S5FRERERDoRV0sHGGMuBgqstSuNMdnNHDcPmAeQkZFBTk5OW9XYaj3j/KzecSAs55aOU15erjaOcGrj6KB2jg5q5+jQ2dvZWNv8zAbGmPuA6wAvEA+kAK9Ya69t6jFZWVl2xYoVbVlnq3z/yfd5fYeHnffOweEwHX5+6Rg5OTlkZ2eHuwxpR2rj6KB2jg5q5+gQrnY2xqy01ma1dFyLQxqstXdYawdYazOBq4BFzYXdcEp0BUJuuVvz8YqIiIhIQMTMwwuQEBP4t6xagVdEREREAlocw1uXtTYHyGmXStpAbQ9vWbUHSAhvMSIiIiLSKURWD28wvquHV0RERERqRVjgrdvDKyIiIiISsYFXPbwiIiIiEhBRgTcxOKShVIFXRERERIIiKvAmxGhIg4iIiIjUF1GBN9YBLofRkAYRERERCYmowGuMITneRbkCr4iIiIgERVTgBUiKd2lIg4iIiIiERFzgTY6L0ZAGEREREQmJuMCbEOuk2usLdxkiIiIi0klEXOCNczmo8fjDXYaIiIiIdBIRGXjdPgVeEREREQmIuMAbqx5eEREREakj4gJvnMtJjcbwioiIiEhQBAZeB26venhFREREJCDyAm+MgxoFXhEREREJirjAG+t0KvCKiIiISEjEBd64GA1pEBEREZGjIi/wBqcl8/ttuEsRERERkU4g4gJvrCvwK2kuXhERERGBCAy8cS4ngMbxioiIiAgQkYE38CtpLl4RERERgQgOvLpwTUREREQgAgNvbKiHV4FXRERERCIw8IbG8HoUeEVEREQkEgNvTOBXqtYYXhEREREhAgNvfLCHt9qjwCsiIiIikRh4gz28GtIgIiIiIhCRgVc9vCIiIiJyVMQF3oTawKsxvCIiIiJCBAbeoz28GtIgIiIiIhEZeIOzNGhIg4iIiIgQkYFXPbwiIiIiclTEBd7aWwurh1dEREREIAIDrzGGOJdDF62JiIiICBCBgRcCwxo0D6+IiIiIQMQGXgdVbvXwioiIiEjEBl6nhjSIiIiICBCpgdfl1EVrIiIiIgJEauCNcWhaMhEREREBWhF4jTHxxphlxpg1xpgNxphfdERhJyMuRj28IiIiIhLQmh7eGuBca+0EYCIw2xgztX3LOjmBMbzq4RURERGRVgReG1AeXI0J/th2reokxbsc1KiHV0REREQAY23L2dUY4wRWAqcAj1pr/6+RY+YB8wAyMjKmvPDCC21casvKy8tJSkri8TXV7Crx89sZiR1eg7S/2naWyKU2jg5q5+igdo4O4WrnmTNnrrTWZrV0nKs1T2at9QETjTFpwKvGmLHW2vXHHPME8ARAVlaWzc7OPv6qT1JOTg7Z2dm8U7iW3IpDhKMGaX+17SyRS20cHdTO0UHtHB06ezsf1ywN1tpi4ENgdvuU0zYSYp1Uur3hLkNEREREOoHWzNLQK9izizEmATgf2NzehZ2MlIQYymq8+PydeqixiIiIiHSA1gxp6Av8IziO1wG8aK19s33LOjlpCTFYC2XVHtISY8NdjoiIiIiEUYuB11q7FpjUAbW0me7dYgA4UqnAKyIiIhLtIvJOa2kJgZBbXOkOcyUiIiIiEm6RGXgTAz28xZWeMFciIiIiIuEWoYE32MNbpR5eERERkWgXkYG3e7CHd2N+aZgrEREREZFwi8jAmxIfCLxPLt4V5kpEREREJNwiMvA6HCa0/J8VeWGsRERERETCLSIDL0B6UhwAt760NsyViIiIiEg4RWzgvXPOqHCXICIiIiKdQMQG3tqpyQCqPb4wViIiIiIi4RSxgTfe5QwtH67Q9GQiIiIi0SpiA2/3bkdvKbxk26EwViIiIiIi4RSxgffUvin85bopACzYWBDmakREREQkXCI28AJcOKYPM0f2YuGmgxrHKyIiIhKlIjrwAhwsrQFg1F3vhrkSEREREQmHiA+8CbFHL17bkF8SxkpEREREJBwiPvA+eOWE0PJDC7eFsRIRERERCYeID7yDe3ZjQPcEAN7feBBrbZgrEhEREZGOFPGBF8DpMKHl5blHwliJiIiIiHS0qAi8Xz8rM7T83WdWcvd/17Nur8bzioiIiESDqAi8N04bwo575wCBu67947PdXP/00jBXJSIiIiIdISoCL9Qf1gBwpNJDYXlNmKoRERERkY4SNYG3MVm/XhjuEkRERESknUVV4F3xs1nce9m4ettqvLoDm4iIiEgki6rAm54Ux9VnDOLpr2eFthVVuMNYkYiIiIi0t6gKvLXOHZURWt5eUB7GSkRERESkvUVl4K3rmc93h7sEEREREWlHURt4dwanKXtvw0F2H64IczUiIiIi0l6iNvA66kxTVlLlCWMlIiIiItKeojbw1nXJI59w56vrwl2GiIiIiLSDqA68r/7vWaHl55buCWMlIiIiItJeojrwJsQ6w12CiIiIiLSzqA68TlP/dsOvr8kPUyUiIiIi0l6iOvBmpMbXW3/9i31hqkRERERE2ktUB96U+Bhy588Nra/aUxzGakRERESkPUR14K3VKzkO0G2GRURERCKRAi9w/1fGh5Y3HygNYyUiIiIi0tYUeIHskb25dGI/AGb/cXGYqxERERGRtqTAG5SREt/yQSIiIiLS5SjwBsXHaE5eERERkUjUYuA1xgw0xnxojNlojNlgjPlBRxTW0RIUeEVEREQikqsVx3iBH1trVxljkoGVxpgF1tqN7Vxbh4p1qbNbREREJBK1mPKstfuttauCy2XAJqB/exfW0aad0jO0/Mn2wjBWIiIiIiJtyVhrW3+wMZnAx8BYa23pMfvmAfMAMjIyprzwwgttV2UrlZeXk5SUdEKPdfss8xZUAnDeIBfXjY5ry9KkDZ1MO0vXoDaODmrn6KB2jg7haueZM2eutNZmtXRca4Y0AGCMSQJeBn54bNgFsNY+ATwBkJWVZbOzs1tfbRvJycnhpM674C0ABg0cQHb2mLYpStrcSbezdHpq4+igdo4Oaufo0NnbuVUDV40xMQTC7rPW2lfat6Tw+yJPtxgWERERiRStmaXBAH8FNllrH2z/ksLnvR/OAGD1HgVeERERkUjRmh7eacB1wLnGmC+CP3Paua6wGNknObR8PGObRURERKTzas0sDUustcZaO95aOzH483ZHFBcOkwalAXDfO5vDXImIiIiItAVNPnuMm7NPAeCJj3eql1dEREQkAijwHmP8gNTQstevwCsiIiLS1SnwHqN3SjwXjM4AoMbrD3M1IiIiInKyFHgbMX14OgA1Hl+YKxERERGRk6XA24g4V+BlUQ+viIiISNenwNuIOJcTgGr18IqIiIh0eQq8jajt4S2v8Ya5EhERERE5WQq8jaidnOHOV9eFtxAREREROWkKvI3wB+ffXb+vlILS6jBXIyIiIiInQ4G3EXPH9Q0tn37vB2GsREREREROlgJvIxwOQ89usaF13XFNREREpOtS4G1CfIwztKzpyURERES6LgXeJtTO1ABQ6db0ZCIiIiJdlQJvE3YWVoSWK92ankxERESkq1LgbQXdgEJERESk61LgbcKtF44MLWtIg4iIiEjXpcDbhK9MGRBaVuAVERER6boUeJuQmhATWq7SkAYRERGRLkuBtwnxMU7+duNpAFSph1dERESky1LgbcbQ9G4A/O+zq3BrLl4RERGRLkmBtxkJdW4+cc8bG8JYiYiIiIicKAXeZqTUGce7fFdRGCsRERERkROlwNuM+BgnZw9PB8BvbZirEREREZETocDbghhn4CUqq9bd1kRERES6IgXeFiTGBsbxFld5wlyJiIiIiJwIBd4W3P2lMaFlj08zNYiIiIh0NQq8LeiVHMeEAam4vX4eXLA13OWIiIiIyHFS4G2FmuAcvEt3Hg5zJSIiIiJyvBR4W2HK4O4ArNpTzIGS6jBXIyIiIiLHQ4G3FX42d3RoubC8JoyViIiIiMjxUuBthfiYoy+Ty2nCWImIiIiIHC8F3lYw5mjIffTDHWGsRERERESOlwLvcXpjTX64SxARERGR46DAKyIiIiIRTYH3BMx/ZzO3vbQm3GWIiIiISCso8LbSk9dnhZYf/2gHL67Yy6ur93LJI0vCWJWIiIiItMQV7gK6ioQYZ4NtP/p3oJfX57c4HZq9QURERKQzUg9vKw3r3a3Jfe7gndhEREREpPNR4G2lvqkJ/OMbpze6T4FXREREpPNqMfAaY542xhQYY9Z3REGdWY/E2Ea313h9HVyJiIiIiLRWa3p4/w7Mbuc6uoQhvRof1lCjHl4RERGRTqvFwGut/Rgo6oBaOr2kOBd/v/E0Zo/pU2+726fAKyIiItJZGWttywcZkwm8aa0d28wx84B5ABkZGVNeeOGFNiqx9crLy0lKSmr389y3tIotR46G3F+eFc+glIazOEj76Kh2lvBRG0cHtXN0UDtHh3C188yZM1daa7NaOq7NpiWz1j4BPAGQlZVls7Oz2+qpWy0nJ4eOOO+8he/UWx8/aQoTB6a1+3kloKPaWcJHbRwd1M7RQe0cHTp7O2uWhhOQkRJXb72oooZqjy5cExEREemMFHhPwDM3nVFv/Rt/X8FZ8xeFqRoRERERaU5rpiV7HvgMGGmM2WuMuan9y+rcBvdsOFtDUYUbry5eExEREel0WjNLw9estX2ttTHW2gHW2r92RGGdXe78udxw5uB627YfKg9TNSIiIiLSFA1pOAk//9KYeuuf7TgcpkpEREREpCkKvCfB6TBcPL5vaP1QWU0YqxERERGRxijwniSXw4SWP952iEsfWcLh8vrBt8brUxgWERERCRMF3pPkdBx9CdfvK2XN3hKeX7aH0mpPaPvNz67mtN8sDEd5IiIiIlFPgfck1fbwmqMdvTzw/lbG3/N+aH3hpoMAmsVBREREJAwUeE/S8IzAbfS+dfbQFo/derAcay0+/9HbOf/lox2s2nOk3eoTERERiXZtdmvhaPWNaUMY1SeFtMQYnvh4Z719Z933Afkl1aH1OQ8vZu74vry1dj+58+cCcN87mwFC6yIiIiLSttTDe5IcDsP04enEuhq+lHXDbq231u4HwFqLv05Pr4iIiIi0DwXeNjK8dxJ3zhnFhAGprTq+tNrLnIcXt3NVIiIiIqIhDW3EGMO8GcP41tlDeXvdASrcXm57aW2TxxdVuNl8oKwDKxQRERGJTurhbWPGGOaO70uM0zR73Fce+7SDKhIRERGJbgq87SQpLqbZ/Ycr3I1ut9ay41B5e5QkIiIiEpUUeNvJrFN78+jVk1t9/Bd5xQD8Z8Vezvv9Rzz+0Y72Kk1EREQkqijwtpPaoQ2r7zqfVXedz6+/PLbZ47/86CcA7CysAGD+O5vZkF8S2u/2+nn4g21Ue3ztV7SIiIhIBFLgbWfdu8XSo1ss547q3eKxXp+f+JijTbL1YBnr9wVC7wvL9/Dggq08ltN8z+/ibYf4IHhnNxERERHRLA0dpl9aAo9dM5mpQ3uyYvcR3liTz+tr8usdc8pP36m3/qN/rwECN6Wo7dmtqPE2e57r/ros9BgRERERUeDtUBeN6wvA+aMzOH90RoPA2xzD0Vkf3F4/l/35EyYOTCM+xsldF49u81pFREREIoUCbxjlzp9LjdfHyJ+92+xxK3cXkV9SBcBTS3ZxsKyGDfmlbMgvBeCOi0bhcjY+OmXJtkImDkojKU5NLSIiItFJY3jDLM7lJDm++TB6xWOf8bdPckPrbxzTM3zJI4EL3jYGA3CtvKJKrv3rUn766rq2KVZERESkC1Lg7QT8fntSj9+4v5Q1ecW8v/FAaNv2gjLO/t2HACzbVcTK3UWhfYfKanjm8914fX4eeG8Lf9EUaCIiIhLB9D13J3CSeReAS4PTmtVavac4tLy/pJorHvuMX146hp//dwMOEzhn39R4HvlwOwDfPmcYAMWVbirdPvqlJZx8USIiIiKdgHp4O4E5wYvZHr92MjNH9mLN3ReE9j37zTNO6DmfXbqnwbaf/3cDcDRgl1UfnfFhRW6gB/ic+3M4a/4iADJvf4v739t8QucXERER6SwUeDuB+VeMY9lPz2P22L787cbTSU2IYcLANACmnZLO1WcMOnrs5eNa9Zy1d25rTkmVJ7T88qp99bbVeAPToD36YWC4g7WWg6XVrNxdxLNLd1NR48Xntxwqq2n0uQtKq8m8/S3eXX+g0f0Ar67ey9Kdh1v1+4iIiIicKAXeTiDG6aB3cny9bS9+eyrr7gn09P5w1nDSk2J56/vT+Z/TBtI9MQaAHffOOanzHql0h5afX7aHef9cEVovrao/3+/Lq/Zxxr0fcMVjn/HTV9cz5u73uPWlNZz2m4WNzg1cO4PEU4t3cqTCTbXHx31vb6K8zrE/+vca/ueJz0/qdwBYu7eYqfd+QHGd30dERESklgJvJxWYvSEQbHsnx7PiZ+czpl8qxhg+u+M8lt15Hk7H0bl5//mN04/7HH9cuK3e+vsbj96hre7d2h7+YBuLNje8e9srwV7hxgKvNzhuYsXuI0z61QJeXb2Pv3y8k4c/2Nbg2NZye/2Nbn9k0XYOlFazdFdRo/s7g9zCCjJvf6vR11FERETalwJvFxQf46R3SqBHeGCPwMVlM0b04onrprDgRzP4bvawkz7H7a8cncrswQVbeXtd00MTnlu2h20Hy0Lr//osl1ueW1XvmNqp1z7acoj9JVXc/OzR/XMeWszmA6XsPVJJpdtLUYWb4ko3+cVV3Pi3ZXy6vZCbn13FiJ+902BKNgATzP22DS7+ay+r844A8N8vWn+zEREREWkbmqWhi3v7+2dT5Q6Mt71gTB8A/m/2KCYP6s6oPsmhqcl+NGsEf1i4tV1q+OPCbfxx4TZe/PaZnD6kB3cFL46rq3aIxJaDZZz/4Mf1hjZs3F/Knz/cwetr8jl9SA+W7SqiR7dYzhrWkw+3HOLDLYdCx76+Jp/Cwmr+sOET/nzNZPqnJdS5C13HJd6SKg+lVR4G9kjssHOKiIjIiVEPbxeXHB8T6u2t6/zRGQzskciu++aw5P9m8oNZw8mdPzd0G+IrJg/gR7NGAJCaEFPvsRMGpJ5QLVf+5TMyb3+r0X2r9xwJLZc3MgRi0eYCIDBnMEBRhZt3GrngbcHGg6wu8LEmr5hp8xdR5fbx7obAcd95ZhVFFW4yb3+L7z2/moLS6mbrfWnlXnYfrgitu71+rvvrUlbVqbUpF/9pceiPiVrVHh8fbi7gcHnDC/nq3hpaREREOpZ6eCOcMYYB3Y/2Ql6ZNYBPtxfy4wtG0C8tgdOGdGdMv1Q27S/lquAFZK/dPI1HFm3n7BG9yNlSwKLNBazdW9LguV+/ZVroLm8t+c/Kvc3ubywE+1oxQfGpP69/W+bXVgfGFb+xJp831uTz0a3ZxDgd7Cuu4rTMHry/4QAHy2romxLPT/6zBoBPbz+XfmkJ7CwsZ/G2QvaXVLPw/53T7Hnziqrqre88VM6sBz/Cb2FUn2Te/eGMevvtcfQ+H6lw071bbKuPFxERkeYp8EaZ5PgY/vr100LrZw1LB2Dq0J5cPrk/izYXYIzhe+cNB2DiwDR+OGsECzce5JvBWRzmju/Lb68YT0KMs+N/gRb88s2N9dbPuT8ntPzk9VnM+9fKBo957Yt9XDF5AP/8bDcA8TENv/jw+vyUVXubDKLn/v6j0PLmA2UN9vuD19tV1HhZubuIKYN7NPo8G/NLmfPwYh68cgKXTx4Q2l5S5SE5zoXDoZ5iERGR46XAKyEPXjmxyX2zRmfw0FUTSU+KY9op6aHt35g2hKc/2cXQ9G7sLKzgF5eM4Yu8Yl4N9rS2pfSkWArLT3zqsW/VmXatrt+9u4XfvbsltL5+XylFFW6S411UeXyUVnmY/tsPiXM5uOOiUbywPC90bKXby6/f2tTgOVfkFpEY62J0vxQAfhzsTV64qYCFmwp483vTGdYriUq3lwOl1YzpFxhGUjvE4rUv8pk6tCclVR4uemhx6Hk3/2o28cE/NNxeP7EuB9ZaHnh/C5MGdifG5eCcEb1O+DWq65PthWzML+VbM4ZSXuPliz3FTB+e3vIDRUREOhkFXmm1Syf2b7Dt9otGMW/GUDJS4oDAEIobgIvH92VbQTnfOWcY89/ZzOMf7Qg9Zt6MobgchhunDSFnSwG3vrSWh66ayPp9JTy5eFe95592Sk8+2R64OcWvvzyO7zzTsIe2PUz+1YIG22q8fu55o34P8pRfLaTK42tw7Fce/wyA3PlzG705x8V/WsLlk/rzSvAPg6Q4F+U13tC46o+3HuKs+Yu4cVpmvce9smof54/OYO+RSi7786f0S41nSmaPerNX5M6fy7q9JXy+8zArdhfxvXOHM6hnIinx9cdq11VS5WFDfgm7D1fy1SkDcDoM1zy1FIDrzhzMD1/4goWbDrL8p7PolRzX5PM0ZXtBGYN7diPGeXyXDew+XMGgHokY034929UeX+iPCBERiUwKvHJSYl0O+qQ2vGjuvFMzOO/UDCAQir80oS9xLgdgOKV3Uui4r2YN5KtZA4FAoK7y+PjPir3UBOfcffabU8m8/S3OGNKD2WP7sOEXF/LQyzlMHDeGB97fwuAeiTx45USuePxTdh6qaFBHrUsn9gtNCTb/8nGhadfG9U9l3b6G45Nbq7GwW9dHWw9xw9PLGt33Sp1e8NoxzMfOpFH3bngAd766jjtfPTplXH5JNfnHTNW2IrcoFLgB3tsQmPv3ga9OoKLGS0ZKHL1T4pk0MA2/BafDcMtzq1i8rRCADzYVkBJ/9KPhF29sCF3It2jzQU7tm8K4/qn1Qqjfb9lXXNVg1ooHF2yltMrD3z/N5cZpmWQN7sHMUb2IdzlbHJ6xbFcRV/7lM3715bFcN3Vws8eeqBdX5HHbS2v5+NaZDOqpGTdERCKVse0weWlWVpZdsaLxr4/bU05ODtnZ2R1+XmlbxZVu1u4tYeKgNFLiY/D4/DiMCd1oo7F2rvb4uOW5VVwzdTBbD5Tx1ayBvL1uPz97bT3fP284P5o1nIOlNfRJjaes2sO4e94HYNlPz+P033zQYk2PXTOZ7z67qsXjOoP4GAfVnsZv0lHX8N5JbCsoP+HQ/5MLRvDA+1v5/rmnUOXx8eTiXXx+x3n87LX1jOmXwg/OG87QO99u9LFJcS7W3n0Bp9+7kNsuHMXlk/vjdJhQiM7JyWFfwhB++up6AL6bPYzbLhyJMYbPdx5m/b4Sbpo+hLyiKircXob26sbirYVkpidySu/kFmv3+vzc/NwqPt1xmLJqL09dn8Ws0Rmh/ZVuL4mx6g9ob/rMjg5q5+gQrnY2xqy01ma1dJw+0aXTSUuMZUZtJh+2AAATqElEQVSdcait+Ro8PsbJUzcELsabObI3ANdOHcw1ZwwKhajanujk+Bg2/vJCNuSX0js5nm+dPYQnF+/ize9NZ3DPxFAY/uT2c3lk0XaeX7aHqUN7NnreN783nYRYJ4/l7OCllXuZN2MoT3y8s8Fxc8f35axhPfliTzFvrM1vEEjPHp4e6mE9Wa0JuwDbCsoBTriH+4H3A73RDy/aXmfbFhZuOsjCTQd5qJm76pXXeLnssU8pLHdz28true3ltcwd35dHr57M+n0l3L+8isvPPPp7PJazg9lj+nDzc6vYeyQwQ0b3xNjQ2Oi6VvxsFvExTua/s4mx/VK56vRB9fZf//Sy0B9VtTYfKCXW5WDGiF5syC9h7sNLePzaKYzsk0z/tARiXc3/H7TWsvdIwx7ullhrsZYGvd1en5/th8oZ1SfluJ5PREQapx5e6XLaup1r3wO1wfirj3/KqX1T+OWlY+sdt2DjQRZtPsjVpw9meEYSTocJhfEar4/9xdUs3l7IXa+t5/ozB3PLuafw2up93DR9aL3bQENgnuGcLQVcNLYva/cWc8bQnri9fn6/YAs3TR/CtU8tZevBcp7/1lT+tGgbn+44fEK/2wWjM+rdMvp4JcQ4Wxy20ZYSY51Uult3vrTEGIorPS0fCGT2TMThMPz6y2O5+smlTR43oHtCKFDXumRCP+7+0mh6JsWxJq+YkioPM0b04pqnPueT7YdZfdf5TAqO+X7mpjOYPDiNI5Ue9h2p4sq/fMZPglMAXj55AFsPlrHzUAWTB6fRKymOv3y8k/nvbOa1m6fx01fX8eT1WfRLS+DF5Xnc9vJaxg9I5WunD8Lnt1w7dTBr8ooxBsYPSKtX46l3vctlk/tz72XjWvV6hEtuYQV9UuNDY6b1mR0d1M7RobP38CrwSpfTmdvZ4/Pz8sq9fDVrYIOQezyODeHlNV5cDhMKCi8uz6PS7eXr04YAcLC0mh2HyjlzaE+shVF3vYvb5+ep67NC08lBYLzurFN7h8b11nX5pP6kJcayIb+EpcEbgLxxy3S+9MiSBse2NAxiaK9uzY6p7upumXkKj3y4veUDj1P2yF5MG5bOb95uOPPHgh/N4Pw/fBxaf/TqyUwYmEpCjJMpv14IwPbfXITb5+dPi7bz4eYCNh8o4zvnDOPlVXt5+obTGDcglXfXH+A7z6zk2+cM5fbZo3jog230TIrj8kn9iXE6+PVbG/nf7FPqjc3fmF/K88v2cNqQHryzbj+TBqXxzelDj2uavIoaL2Pufo/LJ/Xnwf8JzAjT1u/lg6XVdItzkRTXPl9e+v0Wj99PnEsXOR6PzvyZLW1HgbcD6U0VHdTOLfP4/Ow7UkVmejd+9eZGzh+dwdBe3XAaQ8+ko7MsWGup8vjwW+qFhENlNXh8fvqlJWCtpbTay57DleQdqeS0zB71ZmpYv6+Ei/+0hMmD0li1p5jfXTGeyyf3579f5NOjWyw3/n15g/rmjuvLW+v2AzDr1N4s3FTQ5O/y2yvG8cznexoN2KP6JLP5QBkzRvTi4vF9ue2ltQDceuFI7n9vS4PjI911UwezYvcRNu0vbXT/53ecx9T7Wh6zfs6IXnzr7KH84o0N7CmqDF1E2pjfXTGev3y8g5kje4fmqq6o8fKvz3fzh/+ZQI3Hz1vr9jPr1Azufn0DfVPj+d65w/H6/fx7ySbuvOw0PthUwKRBacwYHhhS8vW/Lcft8/PBj8/B67PkFVWys7Cc5blHGNwjkdlj+/Dcsj3cffEYUhNj8PktS3cd5uonl3Jq3xTe+cHZbNpfSl5RJReM6cNFDy2mtMrD1WcM4qbpQ4iPcbImr5ix/VMb/cN0e0E5+4qr6JcaT4zTwS/e2EBZtZfM9G68tHIvu+6bgzGGdXtLSE+OpW9qQrOvZ0mVh8XbDnHx+H4tvvattb2gnBqvLzSlYUfKL67iX5/v5uLxfVt1fn1mRwcF3g6kN1V0UDt3PhvySxiZkYzrmPHWfr/lN29v4qrTBjI8I5ncwgpiXQ76pSUw5VcLOFzhJnf+XHYfruCllXu54axM0pPiyMnJYcaMcyip8tC9WyyVbi/bDpbzyY5CfvfuFm6/aBRj+6UydWgPiircodtrf77zMMnxLsb0S8VaS1GFm12FFSzYdJDhvZP5zVsbeezaKZRXe/ls52FO6Z3E0p2HqXD7+Ma0Iby4Io8Lx/Th/Y0HeGXV0Vk0Yl0O3M2Evt9cNjZ0gV1dv7x0DD//74Z62yYMTGNNXnGTzzWgewK/vHQM3/h7x3+GRptrzhjEnHF9MSbQi93YnNqNeecHZ/PGmnz+nBOYbvF3V4zn7BHp/GnRdp5buof7vzKexFgXL67I46Oth0KP+965p7A8t4hrpw4me2Rvlu06jM8fuBU8BKbhO1zhpqjczTf/uYJ7vjQ69C1OjdeHtYS+5am9jfuOe+fgMIEZR95cu587LjqVOQ8vZvyAVL5+ViaXTeqPMYa8okpueX41T14/hV5Jcbh9flwOB4cravhgUwFfO2as+7GKKty4vX76pMYz68GP2B68BmDWqRn0T4sn70gVKfEuSqo8OB0OJgxIZcGmg6Gx8s996wymDulJYUUNvZPrz+xTWF5DXlElkwZ1b9XrX8taG5pppjPz+Pw4jcEdvAC7pWsCuqqICLzGmNnAQ4ATeMpaO7+54xV4pT2pnSNDRY0XC41+/dxcG3t9/nozOrQXj89PpdtHYqwTpzHsK65i9+FK7n9vMw98dQJ90xLYc7gydHORY4eh5BZWMLhnIv9ensfmA2XcNH1I6KI2ay27CitIT44jJT4Gay0FZTWcce8HfP+84fy/80dQWF5DVnCoQq2/3XgacS5Hs+OQj/XXG7JIT4rj0keP3gZ8wsA0qtxeZo/pw6TB3bnxbw174aVz6J4Yw8g+yazbW0KF20e/1HjyS6qP6zla+iMLYFivbgzqkciHWwIB/aKxfdh8oIxdhR03NGn2mD6kJsTw7xV5jB+QyrRT0lmTV8z4AWncOC2Tj7YeYnjvJIorPRwsrWZnYQVPfLyTpDgXt144kksm9KOgrIbVe46wZm8xIzOSuWRif0qqPPzqzY1cffogeqfE8d6GA3yRV8z5p2Ywe2xfqjw+nv18NxeN68uOgnJue3ktd84ZRUmVh093HCYtIYaMlHheWbUPt8/P3PF9+crkAby7/gApCS4SY11Ue3zEuhws2lxAt1gXKQkuJg5MC13cmzW4Oyt2B6Z3fOyayQzrnURCjDP0mXC4vIaUhJhGL9LefbiCKo+PUX1SQn/41AZnn9/itxaXw/DR1kOcPqQHibEucgsryEzv1ujr7PX5OVLpOaE51ZvT5QOvMcYJbAXOB/YCy4GvWWs3NvUYBV5pT2rnyBetbVxQVk2PxNh6PeUfbT3EnqLADUFqe/fyiirJSIkn1uXgjlfWkRzv4urTBzG459GbdPj8FrfXT0Ls0fGmeUWVVLp9jOxTf+q2HYfK6RbrIiHWyfsbDvCVKQMYckdgSrmvn5WJwxh+/qXRZN7+FkPSu/HAV8dT4/Fz1inplFR6uPTRJUwZ3IP5V4xjybZCJg/qzp8WbePsEb2ocvtCN4z5242nse1gGY5gjWX7d1Lg6s2hsppGh7XcOWcUmw+UMXVoT8b2S6VfWjxvrdvPgZJqrAVjYPfhSl5fk8/Zw9O5afoQvh4M72cN6xm62POb04fw1JJdPHr1ZP71eS6f7yxqsg1qx2ffMvMUnA5DelIsdx3TSy/SnkZkJNE9MZYN+aWhOdrbwtj+gTnUn1929G6hsS4HPbvFsr+kmgHdE0iKc7H5QFlof5zLwelDerDvSBXpSXGUVnvYfKCM04f0YFz/VMYPSGV03xS2FZTjzd/MJRfObLN6W6stA++ZwD3W2guD63cAWGvva+oxCrzSntTOkU9t3DkVV7qJdTnabI7iuu2cX1xF98RYEmKdFJbX4DSG7t1iW/c8WwrIyuxBUpyLt9buZ83eYn44azjvbzjIlMHdG50urqTKQ0FpNf3SEnA6DM8u3cO5o3ozJL1bo3ffK6ny0C3WyZLthcx/ZzPfzR5G9ojepCYG5gq//70tzB7bh8mDulNQWo3TYeieGEtxlYfUhJjQ1+5en5/nl+fxr89yOWdEL4b1SmLCwDRG9UnGbwMX3v3js1ymDOrO1GE9OVLh5pPthznv1N5sPVjGA+9t4bJJ/bn+zEye/mQXNV4/RRVurpg8gBeDPaNHKj1MHdqD3Ycr+WR7IRMHpnHrS2uJczk4d1Rvymu83Dgtk4LSGjYfKOPvn+aGfs/TMruTPbI3B0qqSYxzcu0Zg0mKc7Est4iU+Bje33iApDgX/dMSmDGiV/D/g5N1e0tIjA3cMXL8gFSWbC/kjCE9MBh+/uyHXHHOJIqrPHy05RAvr9rLJRP6Eety8N8v9uHx1c8hA3skkFcUGCJhjGlwA57GDO6ZSEVNYIaXwvKGd7c8XpdP7l9vWBMEpo/ctL+swfPPHNmLJdsLG/wexxrbP4U+KfHNXrPQlc3OdPH4dy7s8PO2ZeD9CjDbWvvN4Pp1wBnW2luOOW4eMA8gIyNjygsvvHCitZ+w8vJykpKSWj5QujS1c+RTG0cHtXPH8fotDkOod72uGp8lzmkorbGkxLX9UKGTbWdrbeibC7+1+CzUeKFbDI0Obar2Wso9lgSXIcYBsc76xxyp9uP2QYwTesTXTi1pcTrAVWc8sLUWC3j9sLnIx7h0J8YYyt0Wr7UkxZjQa+q3lgMVln5JjlCdBqj2gbXgOqaOAxV+yj2WoakOSmssZZ5AG2074md3qZ/p/V2M7OHAbyG31E+Cy+A0kBJr8PoD3274/Ba3H/wWUuMMH+Z5mNLbhc9CpSfwOuWW+nE5YFo/F6sLfPitpaDScmY/F0v2eenTzTCqhxOvHzx+8PgspW7L7lI/e8r8zBoUQ0qcYeNhHwZwGijzWDx+SHQZNhf5qPRYBqY4uLi/hx6pHf9+njlzZsfeeMJa+wTwBAR6eMPRO6Neoeigdo58auPooHaODpHQzheEu4BWuKiF/bOPWb+qjc/f2du5NZcK7gMG1lkfENwmIiIiItLptSbwLgeGG2OGGGNiCfxR8Hr7liUiIiIi0jZaHNJgrfUaY24B3iMwLdnT1lpdsioiIiIiXUKrxvBaa98G3m7nWkRERERE2lxk3u5DRERERCRIgVdEREREIpoCr4iIiIhENAVeEREREYloLd5p7YSe1JhDwO42f+KWpQOFYTivdCy1c+RTG0cHtXN0UDtHh3C182Brba+WDmqXwBsuxpgVrbm9nHRtaufIpzaODmrn6KB2jg6dvZ01pEFEREREIpoCr4iIiIhEtEgLvE+EuwDpEGrnyKc2jg5q5+igdo4OnbqdI2oMr4iIiIjIsSKth1dEREREpB4FXhERERGJaBEReI0xs40xW4wx240xt4e7Hjk5xphcY8w6Y8wXxpgVwW09jDELjDHbgv92D243xpiHg22/1hgzObzVS1OMMU8bYwqMMevrbDvudjXG3BA8fpsx5oZw/C7StCba+R5jzL7ge/oLY8ycOvvuCLbzFmPMhXW263O9kzLGDDTGfGiM2WiM2WCM+UFwu97PEaSZdu6a72drbZf+AZzADmAoEAusAUaHuy79nFSb5gLpx2z7HXB7cPl24LfB5TnAO4ABpgJLw12/fpps1xnAZGD9ibYr0APYGfy3e3C5e7h/N/202M73AD9p5NjRwc/sOGBI8LPcqc/1zv0D9AUmB5eTga3BttT7OYJ+mmnnLvl+joQe3tOB7dbandZaN/ACcGmYa5K2dynwj+DyP4Av19n+TxvwOZBmjOkbjgKledbaj4GiYzYfb7teCCyw1hZZa48AC4DZ7V+9tFYT7dyUS4EXrLU11tpdwHYCn+n6XO/ErLX7rbWrgstlwCagP3o/R5Rm2rkpnfr9HAmBtz+QV2d9L803iHR+FnjfGLPSGDMvuC3DWrs/uHwAyAguq/27tuNtV7V313VL8Ovsp2u/6kbt3OUZYzKBScBS9H6OWMe0M3TB93MkBF6JPNOttZOBi4CbjTEz6u60ge9ONJ9ehFG7RrTHgGHARGA/8PvwliNtwRiTBLwM/NBaW1p3n97PkaORdu6S7+dICLz7gIF11gcEt0kXZa3dF/y3AHiVwNchB2uHKgT/LQgervbv2o63XdXeXZC19qC11met9QNPEnhPg9q5yzLGxBAIQc9aa18Jbtb7OcI01s5d9f0cCYF3OTDcGDPEGBMLXAW8Huaa5AQZY7oZY5Jrl4ELgPUE2rT2Ct4bgP8Gl18Hrg9eBTwVKKnzlZp0fsfbru8BFxhjuge/RrsguE06sWPG1V9G4D0NgXa+yhgTZ4wZAgwHlqHP9U7NGGOAvwKbrLUP1tml93MEaaqdu+r72dXRJ2xr1lqvMeYWAm8SJ/C0tXZDmMuSE5cBvBp4n+ECnrPWvmuMWQ68aIy5CdgNXBk8/m0CVwBvByqBGzu+ZGkNY8zzQDaQbozZC9wNzOc42tVaW2SM+RWBD1CAX1prW3uBlHSAJto52xgzkcBX3LnAtwGstRuMMS8CGwEvcLO11hd8Hn2ud17TgOuAdcaYL4Lb7kTv50jTVDt/rSu+n3VrYRERERGJaJEwpEFEREREpEkKvCIiIiIS0RR4RURERCSiKfCKiIiISERT4BURERGRiKbAKyIiIiIRTYFXRERERCLa/wdC394pUXh0pQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 864x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean loss=0.016\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 2500/2500 [07:48<00:00,  5.33it/s]\n"
     ]
    }
   ],
   "source": [
    "for _ in trange(2500):\n",
    "    step = len(metrics['train_loss']) + 1\n",
    "    batch_ix = np.random.randint(len(train_inp), size=batch_size)\n",
    "    feed_dict = {\n",
    "        inp: inp_voc.to_matrix(train_inp[batch_ix]),\n",
    "        out: out_voc.to_matrix(train_out[batch_ix]),\n",
    "    }\n",
    "    \n",
    "    loss_t, _ = sess.run([loss, train_step], feed_dict)\n",
    "    metrics['train_loss'].append((step, loss_t))\n",
    "    \n",
    "    if step % 100 == 0:\n",
    "        \n",
    "        clear_output(True)\n",
    "        plt.figure(figsize=(12,4))\n",
    "        for i, (name, history) in enumerate(sorted(metrics.items())):\n",
    "            plt.subplot(1, len(metrics), i + 1)\n",
    "            plt.title(name)\n",
    "            plt.plot(*zip(*history))\n",
    "            plt.grid()\n",
    "        plt.show()\n",
    "        print(\"Mean loss=%.3f\" % np.mean(metrics['train_loss'][-10:], axis=0)[1], flush=True)\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Посмотрим, что вышло -2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '']"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.translate_lines('Where is New York?')[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "inp_line where is c0\n",
      "real query ( lambda $ 0 e ( loc : t c0 $ 0 ) )\n",
      "generated query ( lambda $ 0 e ( loc : t c0 $ 0 ) )\n",
      "\n",
      "inp_line river in s0\n",
      "real query ( lambda $ 0 e ( and ( river : t $ 0 ) ( loc : t $ 0 s0 ) ) )\n",
      "generated query ( population : i c0 )\n",
      "\n",
      "inp_line where are mountain\n",
      "real query ( lambda $ 0 e ( exists $ 1 ( and ( mountain : t $ 1 ) ( loc : t $ 1 $ 0 ) ) ) )\n",
      "generated query ( count $ 0 ( and ( state : t $ 0 ) ( next_to : t s0 $ 0 ) ) )\n",
      "\n",
      "inp_line where is c0\n",
      "real query ( lambda $ 0 e ( loc : t c0 $ 0 ) )\n",
      "generated query ( argmax $ 0 ( and ( city : t $ 0 ) ( loc : t $ 0 s0 ) ) ( size : i $ 0 ) )\n",
      "\n",
      "inp_line what state border s0\n",
      "real query ( lambda $ 0 e ( and ( state : t $ 0 ) ( next_to : t $ 0 s0 ) ) )\n",
      "generated query ( argmax $ 0 ( river : t $ 0 ) ( count $ 1 ( and ( state : t $ 1 ) ( loc : t r0 $ 1 ) ) )\n",
      "\n",
      "inp_line what state border s0\n",
      "real query ( lambda $ 0 e ( and ( state : t $ 0 ) ( next_to : t $ 0 s0 ) ) )\n",
      "generated query ( argmax $ 0 ( and ( place : t $ 0 ) ( loc : t $ 0 ( the $ 1 ( and ( state : t $ 1 ) ( capital : t $ 1 c0 ) ) ) ) ) ( elevation : i $ 0 ) )\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for inp_line, query, generated_query in zip(test_inp[:50], test_out[:50], model.translate_lines(test_inp[::50])[0]):\n",
    "    print(\"inp_line\", inp_line)\n",
    "    print(\"real query\", query)\n",
    "    print(\"generated query\", generated_query)\n",
    "    print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [],
   "source": [
    "candidate_list = []\n",
    "reference_list = []\n",
    "\n",
    "for inp_line, query, generated_query in zip(test_inp, test_out, model.translate_lines(test_inp)[0]):\n",
    "    candidate_list.append(generated_query)\n",
    "    reference_list.append(query)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5836298932384342"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "compute_accuracy(candidate_list, reference_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "чуть лучше"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## А что не так?\n",
    "\n",
    "* Машинные запросы \"жестче\", чем реальная речь. То есть если человек понимает магиста Йоду, машина вот такой запрос \n",
    "\n",
    "$) \\lambda $ 0 e ( loc : t 0 $ c0 ( )$\n",
    "\n",
    "вместо\n",
    "\n",
    "$( \\lambda $ 0 e ( loc : t c0 $/$$ 0 ) )$\n",
    "\n",
    "понять уже не сможет\n",
    "\n",
    "* Более того, одна ошибка - и запрос не выполнится\n",
    "\n",
    "* Вышестоящая моделька выучиласть открывать/закрывать скобки, но обычно в seq2seq моделях это проблема.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Seq2Tree model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* У логического выражения иерархическая структура\n",
    "\n",
    "<img src=\"files/hierarical.png\">\n",
    "\n",
    "* В отличи от seq2seq, где каждый последующий в пространстве шаг зависит от предыдущего и неявно от всего входа(если attention), в иерархической структуре дети должны зависеть только от родителей и бабушек с дедушками, но не от сиблингов и тем более от \"двоюродных братьев\". Время вылета в запросе не зависит от города вылета\n",
    "\n",
    "* Поэтому генерация проходит следующим образом:\n",
    "\n",
    "В набор символов добавляется новый символ n, что означает начало дерева. \n",
    "\n",
    "Если выпал символ n, в следующее состояние передается не скрытое состояние n, а сумма всех состояний, сгенерированных на этом уровне до этого момента\n",
    "<img src=\"files/tree_generate.png\">\n",
    "\n",
    "* Loss - такой же, accuracy - так же"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## SQL запросы (WikiSQL)\n",
    "\n",
    "* Qw: What record company did conductor Mikhail Snitko record for after 1996?\n",
    "\n",
    "  Qu: SELECT Record Company WHERE (Year of Recording > 1996) AND (Conductor = Mikhail Snitko)\n",
    "  \n",
    "* Aggregation operation: SVG, COUNT, MIN, MAX, SUM (\"How many\" -> COUNT)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## RL approach\n",
    "\n",
    "<img src=\"files/rl.png\">\n",
    "\n",
    "* предсказываем aggregation operation как возможный ответ классификатора ($Loss_{agg}$ - cross-entropy)\n",
    "* предсказываем название колонки (SELECT) как возможный ответ классификатора ($Loss_{sel}$ - cross-entropy)\n",
    "* предсказываем WHERE clause как последовательность. \n",
    "  \n",
    "  Эта последовательность должна выполняться. \n",
    "  \n",
    "  Если предсказанная последовательность невыполнима, reward = -2, если выполнима, но дает неверный ответ, -1; если выполнима и дает верный ответ, +1\n",
    "  \n",
    "  $Loss_{where}$ = -среднее reward по батчу\n",
    "  \n",
    "* Loss = $Loss_{agg}$ + $Loss_{sel}$ + $Loss_{where}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Почему reward, а не кросс-энтропия?**\n",
    "\n",
    "запросы\n",
    "\n",
    "**SELECT name FROM insurance WHERE age > 18 AND gender = \"male\"** \n",
    "\n",
    "**SELECT name FROM insurance WHERE gender = \"male\" AND age > 18**\n",
    "\n",
    " дают одинаковый верный результат, но один из них, например, совпадает с ответом из датасета, а другой нет. Вопорос на естественном языке **не взаимно однозначно** связан с его SQL-формой."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Coarse-to-fine"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Qwestion: What record company did conductor Mikhail Snitko record for after 1996?**\n",
    "\n",
    "**Expected query: SELECT Record Company WHERE (Year of Recording > 1996) AND (Conductor = Mikhail Snitko)**\n",
    "\n",
    "* Энкодим входной вопрос на естественном языке\n",
    "\n",
    "* Агррегатор и название колонки - классификатор на LSTM_encoder\n",
    "\n",
    "* декодим скетч \"WHERE > AND = \"\n",
    "\n",
    "* енкодим скетч\n",
    "\n",
    "* декодим полный WHERE clause \"SELECT Record Company WHERE (Year of Recording > 1996) AND (Conductor = Mikhail Snitko)\"\n",
    "\n",
    "* Profit!!!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Дальнейшие работы: генерировать where clause, которых нет в трейне"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}