Week 02/Week_02_Word_Embeddings_(Part_1).ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Week 02 - Word Embeddings (Part 1).ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "metadata": {
        "id": "wWzU9hqUomdU",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!wget -O quora.zip -qq --no-check-certificate \"https://drive.google.com/uc?export=download&id=1ERtxpdWOgGQ3HOigqAMHTJjmOE_tWvoF\"\n",
        "!unzip quora.zip\n",
        "!pip install -q --upgrade nltk gensim bokeh pandas\n",
        "\n",
        "import nltk\n",
        "nltk.download('punkt')\n",
        "nltk.download('stopwords')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "VXl16AOdtlDk",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Словные эмбеддинги"
      ]
    },
    {
      "metadata": {
        "id": "ErZ_TOu0vAOR",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "*NB. Ноутбук в большой степени основан на [первом задании из релевантного ШАДовского курса](https://github.com/yandexdataschool/nlp_course/tree/master/week1_embeddings).*\n",
        "\n",
        "Все видели такие картинки (я надеюсь):\n",
        "![embeddings relations](https://www.tensorflow.org/images/linear-relationships.png)\n",
        "*From [Vector Representations of Words, Tensorflow tutorial](https://www.tensorflow.org/tutorials/representation/word2vec)*\n",
        "\n",
        "Сегодня будем заниматься такими моделями.\n",
        "\n",
        "Начнём утро с визуализаций. Идём на сайт [http://rusvectores.org/ru/](http://rusvectores.org/ru/) и смотрим, что умеют обученные модели для русского.\n",
        "\n",
        "Обратите внимание на разделы *Похожие слова* и *Калькулятор*, а также на набор моделей, которые в них можно выбирать."
      ]
    },
    {
      "metadata": {
        "id": "FkxsGQqZNjxj",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Тренируем простую модель"
      ]
    },
    {
      "metadata": {
        "id": "PaOn69Bg1hH-",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Просто так смотреть на чужие модели, конечно, неприкольно - поэтому будем постепенно приближаться к решению конкретной задачи: [Quora Question Pairs at kaggle](https://www.kaggle.com/c/quora-question-pairs):"
      ]
    },
    {
      "metadata": {
        "id": "K-X7I7nc1gyS",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "\n",
        "quora_data = pd.read_csv('train.csv')\n",
        "\n",
        "quora_data"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "p13HdkzWKtKe",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Поучим на этих текстах Word2vec из gensim. \n",
        "\n",
        "Для начала объединим все тексты."
      ]
    },
    {
      "metadata": {
        "id": "Mchv4fS_21OX",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "\n",
        "quora_data.question1 = quora_data.question1.replace(np.nan, '', regex=True)\n",
        "quora_data.question2 = quora_data.question2.replace(np.nan, '', regex=True)\n",
        "\n",
        "texts = list(pd.concat([quora_data.question1, quora_data.question2]).unique())\n",
        "texts[:10]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "-hZMFAmvK5b7",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Для токенизации проще всего воспользоваться `nltk` (он быстрее `spacy`, но может быть хуже в отдельных случаях)."
      ]
    },
    {
      "metadata": {
        "id": "LTxolf8nLM-n",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from nltk.tokenize import word_tokenize\n",
        "\n",
        "word_tokenize(texts[0])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "fuJceE4JLRxK",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**Задание** Приведите все тексты к нижнему регистру и токенизируйте их."
      ]
    },
    {
      "metadata": {
        "id": "a7XbnSdt4REg",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "tokenized_texts = <do smth>\n",
        "\n",
        "assert all(isinstance(row, (list, tuple)) for row in tokenized_texts), \\\n",
        "    \"please convert each line into a list of tokens\"\n",
        "assert all(all(isinstance(tok, str) for tok in row) for row in tokenized_texts), \\\n",
        "    \"please convert each line into a list of tokens\"\n",
        "\n",
        "is_latin = lambda tok: all('a' <= x.lower() <= 'z' for x in tok)\n",
        "assert all(not is_latin(token) or token.islower() for tokens in tokenized_texts for token in tokens),\\\n",
        "    \"please lowercase each line\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "irl7RotC5C_B",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "print([' '.join(row) for row in tokenized_texts[:2]])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "9kj4dC3iLdwH",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Потренируем небольшую модель на полученных текстах:"
      ]
    },
    {
      "metadata": {
        "id": "9GNuiLio8M25",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from gensim.models import Word2Vec\n",
        "\n",
        "model = Word2Vec(tokenized_texts, \n",
        "                 size=32,      # embedding vector size\n",
        "                 min_count=5,  # consider words that occured at least 5 times\n",
        "                 window=5).wv  # define context as a 5-word window around the target word"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "JclToDJMNwTy",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Изучаем полученную модель"
      ]
    },
    {
      "metadata": {
        "id": "JBVR2kY7LkCs",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Ура, теперь можно делать то же, что было на `rusvectores`.\n",
        "\n",
        "Получить вектор для слова:"
      ]
    },
    {
      "metadata": {
        "id": "Jk6Fgraj-j3c",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "model.get_vector('anything')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "qHKRy7HyLuxH",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Найти наиболее близкие слова:"
      ]
    },
    {
      "metadata": {
        "id": "toyNzyTB-p70",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "model.most_similar('bread')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "A510z5gTL00E",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Или даже так:"
      ]
    },
    {
      "metadata": {
        "id": "j2A_DF5E-ucq",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "model.most_similar(positive=['coder', 'money'], negative=['brain'])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "H-_uKG4vNIJv",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "И так, конечно:"
      ]
    },
    {
      "metadata": {
        "id": "l_mzQgi4L474",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "model.most_similar([model.get_vector('politician') - model.get_vector('power') + model.get_vector('honesty')])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "y5BAEyMyL2kx",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**Задание** Поищите аналогии самостоятельно. "
      ]
    },
    {
      "metadata": {
        "id": "1FYuh4DKN1Fd",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Визуализируем модель"
      ]
    },
    {
      "metadata": {
        "id": "OdT0eIEiN4Ja",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Посмотрим на проекции первых тысячи самых частотных слов."
      ]
    },
    {
      "metadata": {
        "id": "d1e9yS0zBr6j",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "words = sorted(model.vocab.keys(), \n",
        "               key=lambda word: model.vocab[word].count,\n",
        "               reverse=True)[:1000]\n",
        "\n",
        "print(words[::100])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "0Yk5pgMXOESS",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**Задание** Постройте матрицу из эмбеддингов этих слов:"
      ]
    },
    {
      "metadata": {
        "id": "kQu724f2CAh0",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "word_vectors = model.vectors[[model.vocab[word].index for word in words]]\n",
        "\n",
        "assert isinstance(word_vectors, np.ndarray)\n",
        "assert word_vectors.shape == (len(words), model.vectors.shape[1])\n",
        "assert np.isfinite(word_vectors).all()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "f7cgxin-OTvK",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### PCA\n",
        "\n",
        "Простейший линейный метод сокращения размерностей - __P__rincipial __C__omponent __A__nalysis.\n",
        "\n",
        "PCA ищет оси, при проекции на которые данные будут иметь наибольший разброс.\n",
        "\n",
        "![pca](https://i.stack.imgur.com/Q7HIP.gif)\n",
        "*From [https://stats.stackexchange.com/a/140579](https://stats.stackexchange.com/a/140579)*\n",
        "\n",
        "В результате, можно взять проекции на несколько первых компонент - и сохранить как можно больше информации, сократив размерность.\n",
        "\n",
        "Красивые визуализации можно найти [здесь](http://setosa.io/ev/principal-component-analysis/).\n",
        "\n",
        "**Задание** Воспользуйтесь PCA из sklearn, а потом центрируйте и нормируйте результат."
      ]
    },
    {
      "metadata": {
        "id": "V0fQKZw2Css4",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "<imports>\n",
        "\n",
        "def get_pca_projection(word_vectors):\n",
        "    <code>"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "a_9VtLl9CviW",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "word_vectors_pca = get_pca_projection(word_vectors)\n",
        "\n",
        "assert word_vectors_pca.shape == (len(word_vectors), 2), \"there must be a 2d vector for each word\"\n",
        "assert max(abs(word_vectors_pca.mean(0))) < 1e-5, \"points must be zero-centered\"\n",
        "assert max(abs(1 - word_vectors_pca.std(0))) < 1e-5, \"points must have unit variance\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "jGyvhrk7Rlyt",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Визуализируем то, что получилось:"
      ]
    },
    {
      "metadata": {
        "id": "1U58YxF3Cx0W",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import bokeh.models as bm, bokeh.plotting as pl\n",
        "from bokeh.io import output_notebook\n",
        "\n",
        "def draw_vectors(x, y, radius=10, alpha=0.25, color='blue',\n",
        "                 width=600, height=400, show=True, **kwargs):\n",
        "    \"\"\" draws an interactive plot for data points with auxilirary info on hover \"\"\"\n",
        "    output_notebook()\n",
        "    \n",
        "    if isinstance(color, str): \n",
        "        color = [color] * len(x)\n",
        "    data_source = bm.ColumnDataSource({ 'x' : x, 'y' : y, 'color': color, **kwargs })\n",
        "\n",
        "    fig = pl.figure(active_scroll='wheel_zoom', width=width, height=height)\n",
        "    fig.scatter('x', 'y', size=radius, color='color', alpha=alpha, source=data_source)\n",
        "\n",
        "    fig.add_tools(bm.HoverTool(tooltips=[(key, \"@\" + key) for key in kwargs.keys()]))\n",
        "    if show: \n",
        "        pl.show(fig)\n",
        "    return fig"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "qBljy2hCC1qX",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "draw_vectors(word_vectors_pca[:, 0], word_vectors_pca[:, 1], token=words)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "VOIU8uXnSItf",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### TSNE\n",
        "\n",
        "Более интересный и сложный (нелинейный) метод для визуализации высокоразмерных пространств - TSNE. Подробно посмотреть на него можно [здесь](https://distill.pub/2016/misread-tsne/) (ещё более красивые картинки!).\n",
        "\n",
        "**Задание** Как и с PCA - воспользуйтесь TSNE из sklearn и нормируйте + центрируйте результат."
      ]
    },
    {
      "metadata": {
        "id": "F-nlN4_aDF9G",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from sklearn.manifold import TSNE\n",
        "\n",
        "def get_tsne_projection(word_vectors):\n",
        "    <your code>"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "D_YFR_rYDK2n",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "word_tsne = get_tsne_projection(word_vectors)\n",
        "draw_vectors(word_tsne[:, 0], word_tsne[:, 1], color='green', token=words)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "sz6uHYMbSjuE",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Эмбеддим фразы"
      ]
    },
    {
      "metadata": {
        "id": "L4GEypE4SokX",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Сейчас для тестов будет полезно пользоваться фиксированными эмбеддингами. Загрузим уже обученную модель.  \n",
        "(посмотреть, какие вообще модели и корпуса доступны, можно вызовом `api.info()`)"
      ]
    },
    {
      "metadata": {
        "id": "GUDRtumXXF3S",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import gensim.downloader as api\n",
        "\n",
        "model = api.load('glove-twitter-100')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "eypMhlOSXFWN",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Простой и дешевый способ посчитать эмбеддинг фразы, когда есть эмбеддинги слов - усреднить.\n",
        "\n",
        "Вот этим и займемся: токенизируйте и приведите к нижнему регистру фразу, усредните эмбеддинги тех слов, для которых они подсчитаны.\n",
        "\n",
        "**Задание** Напишите функцию для подсчета эмбеддинга фразы."
      ]
    },
    {
      "metadata": {
        "id": "F76HGRGDEPVq",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def get_phrase_embedding(model, phrase):    \n",
        "    vector = np.zeros([model.vector_size], dtype='float32')\n",
        "    \n",
        "    <write me>\n",
        "    \n",
        "    return vector"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "i12XsqnIXfko",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "vector = get_phrase_embedding(model, \"I'm very sure. This never happened to me before...\")\n",
        "\n",
        "assert np.allclose(vector[::10],\n",
        "                   np.array([ 0.30757686, -0.05861897,  0.143751  , -0.11104885, -0.96929336,\n",
        "                             -0.21928601,  0.21652265,  0.14978765,  1.4842536 ,  0.017826  ],\n",
        "                              dtype=np.float32))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Yl2nBZG0WAUx",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Подсчитаем вектора всех фраз."
      ]
    },
    {
      "metadata": {
        "id": "40LezFEJFwv3",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "text_vectors = np.array([get_phrase_embedding(model, phrase) for phrase in tokenized_texts])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "nBj8XMZvWFTv",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "И научимся искать ближайшие!\n",
        "\n",
        "**Задание** По строке найти k ближайших к ней вопросов."
      ]
    },
    {
      "metadata": {
        "id": "fjw7kTQ-FP11",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from sklearn.metrics.pairwise import cosine_similarity\n",
        "\n",
        "def find_nearest(model, text_vectors, texts, query, k=10):\n",
        "    <write me, please>"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "c2yK0twNGWaQ",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "results = find_nearest(model, text_vectors, texts, query=\"How do i enter the matrix?\", k=10)\n",
        "\n",
        "print('\\n'.join(results))\n",
        "\n",
        "assert len(results) == 10 and isinstance(results[0], str)\n",
        "assert results[1] == 'How do I get to the dark web?'\n",
        "assert results[4] == 'What can I do to save the world?'"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "AutTfxbDGhku",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "find_nearest(model, text_vectors, texts, query=\"How does Trump?\", k=10)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "47Ff7heKGiGV",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "find_nearest(model, text_vectors, texts, query=\"Why don't i ask a question myself?\", k=10)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "5BhDEF11ZnTY",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Начинаем классификацию"
      ]
    },
    {
      "metadata": {
        "id": "2D_LEV8cm0z3",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Bag-of-words"
      ]
    },
    {
      "metadata": {
        "id": "jGge73gDid99",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**Задание** Соберем для начала токенизированные вопросы."
      ]
    },
    {
      "metadata": {
        "id": "FJTjqdgiih7O",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "tokenized_question1 = ...\n",
        "tokenized_question2 = ..."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "0VBcipE7Ztkc",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**Задание** Посчитайте косинусную близость между вопросами.\n",
        "\n",
        "Проще всего реализовать функцию ее вычисления руками:\n",
        "$$\\text{cosine_similarity}(x, y) = \\frac{x^{T} y}{||x||\\cdot ||y||}$$"
      ]
    },
    {
      "metadata": {
        "id": "ItJLR_ENHGtI",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "<do smth>\n",
        "\n",
        "cosine_similarities = <and calc the result>"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "NgCVueJ4Z3DQ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Подберём порог близости, при котором будем считать тексты одинаковыми.\n",
        "\n",
        "**Задание** Реализуйте функцию `accuracy`, вычисляющую точность классификации при заданном пороге."
      ]
    },
    {
      "metadata": {
        "id": "41hRIb-KSA3F",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import matplotlib.pyplot as plt\n",
        "%matplotlib inline\n",
        "\n",
        "def accuracy(cosine_similarities, threshold):\n",
        "    return <accuracy>\n",
        "\n",
        "thresholds = np.linspace(0, 1, 100, endpoint=False)\n",
        "plt.plot(thresholds, [accuracy(cosine_similarities, th) for th in thresholds])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "kOxJSE38msmX",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "И запустим оптимизацию:"
      ]
    },
    {
      "metadata": {
        "id": "R3LVCJwqSgZz",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from scipy.optimize import minimize_scalar\n",
        "\n",
        "res = minimize_scalar(lambda x: -accuracy(cosine_similarities, x), bounds=(0.5, 0.99), method='bounded')\n",
        "best_threshold = res.x\n",
        "print('Threshold = {:.5f}, Accuracy = {:.2%}'.format(best_threshold, accuracy(cosine_similarities, best_threshold)))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "H8yFlaffm34A",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Tf-idf weights"
      ]
    },
    {
      "metadata": {
        "id": "-cpeSa-4m64H",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Вместо того, чтобы тупо усреднять вектора, можно усреднять их с учетом весов - поможет в этом уже знакомые tf-idf.\n",
        "\n",
        "**Задание** Посчитайте взвешенные вектора вопросов. Используйте `TfidfVectorizer`\n",
        "\n",
        "`TfidfVectorizer` возвращает матрицу `(samples_count, words_count)`. А наши эмбеддинги имеют размерность `(words_count, embedding_dim)`. Значит, их можно просто перемножить. Тогда каждая фраза - последовательность слов $w_1, \\ldots, w_k$ - преобразуется в вектор $\\sum_i \\text{idf}(w_i) \\cdot \\text{embedding}(w_i)$. Этот вектор, вероятно, стоит нормировать на число слов $k$.\n",
        "\n",
        "**Задание** Кроме tf-idf можно добавить фильтрацию стоп-слов и пунктуации.  \n",
        "Стоп-слова можно взять из `nltk`:\n",
        "```python\n",
        "from nltk.corpus import stopwords\n",
        "stop_words = stopwords.words('english')\n",
        "```"
      ]
    },
    {
      "metadata": {
        "id": "UoYjVExHd_OC",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import re\n",
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "from nltk.corpus import stopwords\n",
        "from string import punctuation\n",
        "\n",
        "<calculate tf-idf weighted vectors>"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "34Di8jXxoBxg",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Посмотрим, что получается после фильтрации и векторизации:"
      ]
    },
    {
      "metadata": {
        "id": "k52NtVJKf1Wl",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "for col in tfidf_question1[0].tocoo().col:\n",
        "    print(model.index2word[col], end=' ')\n",
        "\n",
        "print('\\n' + ' '.join(tokenized_question1[0]))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "3fMqY5tOoL9r",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**Задание** Посчитайте качество с новыми векторами."
      ]
    },
    {
      "metadata": {
        "id": "Chf2qy7uhbPF",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "cosine_similarities = <calc it somehow>\n",
        "\n",
        "res = minimize_scalar(lambda x: -accuracy(cosine_similarities, x), bounds=(0.8, 0.99), method='bounded')\n",
        "best_threshold = res.x\n",
        "print('Threshold = {:.5f}, Accuracy = {:.2%}'.format(best_threshold, accuracy(cosine_similarities, best_threshold)))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "qXi24B78sbsg",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Посмотрим внутрь обучения словных эмбеддингов"
      ]
    },
    {
      "metadata": {
        "id": "6GSDgSXfswp7",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Ключевая идея - слово можно определить по контексту, в котором оно встречается:\n",
        "![contexts](https://image.ibb.co/mnQ2uz/2018_09_17_21_07_08.png)\n",
        "*From [cs224n, Lecture 2](http://web.stanford.edu/class/cs224n/lectures/lecture2.pdf)*\n",
        "\n",
        "Смотреть, как всё учится, будем здесь: [https://ronxin.github.io/wevi/](https://ronxin.github.io/wevi/)."
      ]
    },
    {
      "metadata": {
        "id": "xvqrFUS6vVhh",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Запилим пословный машинный перевод!"
      ]
    },
    {
      "metadata": {
        "id": "1CXcr-ypzGXg",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!wget -O ukr_rus.train.txt -qq --no-check-certificate \"https://drive.google.com/uc?export=download&id=1vAK0SWXUqei4zTimMvIhH3ufGPsbnC_O\"\n",
        "!wget -O ukr_rus.test.txt -qq --no-check-certificate \"https://drive.google.com/uc?export=download&id=1W9R2F8OeKHXruo2sicZ6FgBJUTJc8Us_\"\n",
        "!wget -O fairy_tale.txt -qq --no-check-certificate \"https://drive.google.com/uc?export=download&id=1sq8zSroFeg_afw-60OmY8RATdu_T1tej\"\n",
        "\n",
        "# Install the PyDrive wrapper & import libraries.\n",
        "# This only needs to be done once per notebook.\n",
        "!pip install -U -q PyDrive\n",
        "from pydrive.auth import GoogleAuth\n",
        "from pydrive.drive import GoogleDrive\n",
        "from google.colab import auth\n",
        "from oauth2client.client import GoogleCredentials\n",
        "\n",
        "# Authenticate and create the PyDrive client.\n",
        "# This only needs to be done once per notebook.\n",
        "auth.authenticate_user()\n",
        "gauth = GoogleAuth()\n",
        "gauth.credentials = GoogleCredentials.get_application_default()\n",
        "drive = GoogleDrive(gauth)\n",
        "\n",
        "downloaded = drive.CreateFile({'id': '1d7OXuil646jUeDS1JNhP9XWlZogv6rbu'})\n",
        "downloaded.GetContentFile('cc.ru.300.vec.zip')\n",
        "\n",
        "downloaded = drive.CreateFile({'id': '1yAqwqgUHtMSfGS99WLGe5unSCyIXfIxi'})\n",
        "downloaded.GetContentFile('cc.uk.300.vec.zip')\n",
        "\n",
        "!unzip cc.ru.300.vec.zip\n",
        "!unzip cc.uk.300.vec.zip"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "7RqUeOXxws8y",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Напишем простенькую реализацию модели машинного перевода.\n",
        "\n",
        "Идея основана на статье [Word Translation Without Parallel Data](https://arxiv.org/pdf/1710.04087.pdf). У авторов в репозитории еще много интересного: [https://github.com/facebookresearch/MUSE](https://github.com/facebookresearch/MUSE).\n",
        "\n",
        "А мы будем переводить с украинского на русский.\n",
        "\n",
        "![](https://raw.githubusercontent.com/yandexdataschool/nlp_course/master/resources/blue_cat_blue_whale.png)   \n",
        "*синій кіт* vs. *синій кит*"
      ]
    },
    {
      "metadata": {
        "id": "jjPj9FTRry0U",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from gensim.models import KeyedVectors\n",
        "\n",
        "ru_emb = KeyedVectors.load_word2vec_format(\"cc.ru.300.vec\")\n",
        "uk_emb = KeyedVectors.load_word2vec_format(\"cc.uk.300.vec\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "7rGx4TXWFJ65",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Посмотрим на пару серпень-август (являющихся переводом)"
      ]
    },
    {
      "metadata": {
        "id": "FkHer36xyh4n",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "ru_emb.most_similar([ru_emb[\"август\"]])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "1RSDixWvylEP",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "uk_emb.most_similar([uk_emb[\"серпень\"]])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "iwmm3YQ1yl1U",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "ru_emb.most_similar([uk_emb[\"серпень\"]])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "lAsW7oxszE_I",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def load_word_pairs(filename):\n",
        "    uk_ru_pairs = []\n",
        "    uk_vectors = []\n",
        "    ru_vectors = []\n",
        "    with open(filename, \"r\", encoding='utf8') as inpf:\n",
        "        for line in inpf:\n",
        "            uk, ru = line.rstrip().split(\"\\t\")\n",
        "            if uk not in uk_emb or ru not in ru_emb:\n",
        "                continue\n",
        "            uk_ru_pairs.append((uk, ru))\n",
        "            uk_vectors.append(uk_emb[uk])\n",
        "            ru_vectors.append(ru_emb[ru])\n",
        "    return uk_ru_pairs, np.array(uk_vectors), np.array(ru_vectors)\n",
        "\n",
        "\n",
        "uk_ru_train, X_train, Y_train = load_word_pairs(\"ukr_rus.train.txt\")\n",
        "uk_ru_test, X_test, Y_test = load_word_pairs(\"ukr_rus.test.txt\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "9z6ts7DC0XmN",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Учим маппинг из одного пространства эмбеддингов в другое\n",
        "\n",
        "У нас есть пары слов, соответствующих друг другу, и их эмбеддинги. Найдем преобразование из одного пространства в другое, чтобы приблизить известные нам слова:\n",
        "\n",
        "$$W^*= \\arg\\min_W ||WX - Y||_F, \\text{где} ||*||_F - \\text{норма Фробениуса}$$\n",
        "\n",
        "Эта функция очень похожа на линейную регрессию (без биаса).\n",
        "\n",
        "**Задание** Реализуйте её - воспользуйтесь `LinearRegression` из sklearn с `fit_intercept=False`:"
      ]
    },
    {
      "metadata": {
        "id": "fraTOQtu1YWI",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "mapping = ..."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "PrzRk3ja1b_6",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Проверим, куда перейдет `серпень`:"
      ]
    },
    {
      "metadata": {
        "id": "Quax6HnF1aON",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "august = mapping.predict(uk_emb[\"серпень\"].reshape(1, -1))\n",
        "ru_emb.most_similar(august)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ih1GLNZt1nZX",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Должно получиться, что в топе содержатся разные месяцы, но август не первый.\n",
        "\n",
        "Будем мерять percision top-k с k = 1, 5, 10.\n",
        "\n",
        "**Задание** Реализуйте следующую функцию:"
      ]
    },
    {
      "metadata": {
        "id": "JnmrLp9y2gNI",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def precision(pairs, mapped_vectors, topn=1):\n",
        "    \"\"\"\n",
        "    :args:\n",
        "        pairs = list of right word pairs [(uk_word_0, ru_word_0), ...]\n",
        "        mapped_vectors = list of embeddings after mapping from source embedding space to destination embedding space\n",
        "        topn = the number of nearest neighbours in destination embedding space to choose from\n",
        "    :returns:\n",
        "        precision_val, float number, total number of words for those we can find right translation at top K.\n",
        "    \"\"\"\n",
        "    assert len(pairs) == len(ru_vectors)\n",
        "    num_matches = 0\n",
        "    for i, (_, ru) in enumerate(pairs):\n",
        "        <write code here>\n",
        "    precision_val = num_matches / len(pairs)\n",
        "    return precision_val"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "-1NIvhSH2olG",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "assert precision([(\"серпень\", \"август\")], august, topn=5) == 0.0\n",
        "assert precision([(\"серпень\", \"август\")], august, topn=9) == 1.0\n",
        "assert precision([(\"серпень\", \"август\")], august, topn=10) == 1.0"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "9Ml_w1Tl2r7Y",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "assert precision(uk_ru_test, X_test) == 0.0\n",
        "assert precision(uk_ru_test, Y_test) == 1.0"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "-d9KQHMr2tx8",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "precision_top1 = precision(uk_ru_test, mapping.predict(X_test), 1)\n",
        "precision_top5 = precision(uk_ru_test, mapping.predict(X_test), 5)\n",
        "\n",
        "assert precision_top1 >= 0.635\n",
        "assert precision_top5 >= 0.813"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "JNbDTP502urT",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Улучшаем маппинг\n",
        "\n",
        "Можно показать, что маппинг лучше строить ортогональным:\n",
        "$$W^*= \\arg\\min_W ||WX - Y||_F \\text{, где: } W^TW = I$$\n",
        "\n",
        "Искать его можно через SVD:\n",
        "$$X^TY=U\\Sigma V^T\\text{, singular value decompostion}$$\n",
        "\n",
        "$$W^*=UV^T$$\n",
        "\n",
        "**Задание** Реализуйте эту функцию."
      ]
    },
    {
      "metadata": {
        "id": "9de8XZ_F3v53",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def learn_transform(X_train, Y_train):\n",
        "    \"\"\" \n",
        "    :returns: W* : float matrix[emb_dim x emb_dim] as defined in formulae above\n",
        "    \"\"\"\n",
        "    <write code there>"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "8WeCadzN382y",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "W = learn_transform(X_train, Y_train)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "p6qaMb0E3-f9",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "ru_emb.most_similar([np.matmul(uk_emb[\"серпень\"], W)])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "_Nn58crh4AH0",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "assert precision(uk_ru_test, np.matmul(X_test, W)) >= 0.653\n",
        "assert precision(uk_ru_test, np.matmul(X_test, W), 5) >= 0.824"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "lqgcYk-c4DE5",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Пишем переводчик"
      ]
    },
    {
      "metadata": {
        "id": "hwi70fP6FaAN",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Реализуем простой пословный переводчик - для каждого слова будем искать его ближайшего соседа в общем пространстве эмбеддингов. Если слова нет в эмбеддингах - просто копируем его."
      ]
    },
    {
      "metadata": {
        "id": "0etAHUks4JOr",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "with open(\"fairy_tale.txt\", \"r\") as in f:\n",
        "    uk_sentences = [line.rstrip().lower() for line in in f]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "JK_FJGmn4N7V",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def translate(sentence):\n",
        "    \"\"\"\n",
        "    :args:\n",
        "        sentence - sentence in Ukrainian (str)\n",
        "    :returns:\n",
        "        translation - sentence in Russian (str)\n",
        "\n",
        "    * find ukrainian embedding for each word in sentence\n",
        "    * transform ukrainian embedding vector\n",
        "    * find nearest russian word and replace\n",
        "    \"\"\"\n",
        "    <implement it!>"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "H47pbFyk4P6D",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "assert translate(\".\") == \".\"\n",
        "assert translate(\"1 , 3\") == \"1 , 3\"\n",
        "assert translate(\"кіт зловив мишу\") == \"кот поймал мышку\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "PAVWK7mE4RYU",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "for sentence in uk_sentences:\n",
        "    print(\"src: {}\\ndst: {}\\n\".format(sentence, translate(sentence)))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "pDnIPVLg4Sg7",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Сдача задания\n",
        "\n",
        "[Форма для сдачи](https://goo.gl/forms/GGjrH7axdGJr6yTp2)  \n",
        "[Опрос](https://goo.gl/forms/3QRwLTmLgBzl5VVm2)"
      ]
    },
    {
      "metadata": {
        "id": "_5GrChTeFqIg",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Дополнительные материалы"
      ]
    },
    {
      "metadata": {
        "id": "HwffxpbmFwDh",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Почитать\n",
        "### База:  \n",
        "[On word embeddings - Part 1, Sebastian Ruder](http://ruder.io/word-embeddings-1/)  \n",
        "[Deep Learning, NLP, and Representations, Christopher Olah](http://colah.github.io/posts/2014-07-NLP-RNNs-Representations/)  \n",
        "\n",
        "### Как кластеризовать смыслы многозначных слов:  \n",
        "[Making Sense of Word Embeddings (2016), Pelevina et al](http://anthology.aclweb.org/W16-1620)    \n",
        "\n",
        "### Как оценивать эмбеддинги\n",
        "[Evaluation methods for unsupervised word embeddings (2015), T. Schnabel](http://www.aclweb.org/anthology/D15-1036)  \n",
        "[Intrinsic Evaluation of Word Vectors Fails to Predict Extrinsic Performance (2016), B. Chiu](https://www.aclweb.org/anthology/W/W16/W16-2501.pdf)  \n",
        "[Problems With Evaluation of Word Embeddings Using Word Similarity Tasks (2016), M. Faruqui](https://arxiv.org/pdf/1605.02276.pdf)  \n",
        "[Improving Reliability of Word Similarity Evaluation by Redesigning Annotation Task and Performance Measure (2016), Oded Avraham, Yoav Goldberg](https://arxiv.org/pdf/1611.03641.pdf)  \n",
        "[Evaluating Word Embeddings Using a Representative Suite of Practical Tasks (2016), N. Nayak](https://cs.stanford.edu/~angeli/papers/2016-acl-veceval.pdf)  \n",
        "\n",
        "\n",
        "## Посмотреть\n",
        "[Word Vector Representations: word2vec, Lecture 2, cs224n](https://www.youtube.com/watch?v=ERibwqs9p38)"
      ]
    }
  ]
}