bigram-nn-LM.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Char-level Language Modeling - Part 2\n",
    "In this series, we will be building char-level bi-gram language models:\n",
    "* ~~[Bi-gram count based language model](https://github.com/swechhasingh/nlp-from-scratch/blob/main/bigram-LM.ipynb)~~\n",
    "* **Bi-gram neural network based language model**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn.functional as F\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We would like to formulate the bigram NN-LM as a supervised learning, given an input character, the neural network will output the next character.<br>\n",
    "For this, we need a dataset of pairs of (X, y), X is input character and y is the next character (target/label)<br>\n",
    "In a bigram model, X (input) will be the first character of bigram (context of one character) and y (target) will be the second character of bigram.\n",
    "\n",
    "We will be using the same [names.txt](https://github.com/karpathy/makemore/blob/master/names.txt) used in Andrej Karpathy's makemore lecture series."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"names.txt\",'r') as file:\n",
    "    names = file.read().splitlines()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['emma', 'olivia', 'ava', 'isabella', 'sophia']"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "names[:5]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Let's create the dataset for NN model:\n",
    "First, create character to index mapping and index to character mapping (same as bigram count model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vocab_size=27 and characters in vocab: .abcdefghijklmnopqrstuvwxyz\n",
      "chartoi={'.': 0, 'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5, 'f': 6, 'g': 7, 'h': 8, 'i': 9, 'j': 10, 'k': 11, 'l': 12, 'm': 13, 'n': 14, 'o': 15, 'p': 16, 'q': 17, 'r': 18, 's': 19, 't': 20, 'u': 21, 'v': 22, 'w': 23, 'x': 24, 'y': 25, 'z': 26}\n",
      "itochar={0: '.', 1: 'a', 2: 'b', 3: 'c', 4: 'd', 5: 'e', 6: 'f', 7: 'g', 8: 'h', 9: 'i', 10: 'j', 11: 'k', 12: 'l', 13: 'm', 14: 'n', 15: 'o', 16: 'p', 17: 'q', 18: 'r', 19: 's', 20: 't', 21: 'u', 22: 'v', 23: 'w', 24: 'x', 25: 'y', 26: 'z'}\n"
     ]
    }
   ],
   "source": [
    "# let's create a character vocabulary for names dataset and also add \".\", the start token to the vocab\n",
    "char_vocab = ['.'] + sorted(set(\"\".join(names)))\n",
    "vocab_size = len(char_vocab)\n",
    "print(f\"{vocab_size=} and characters in vocab: {''.join(char_vocab)}\")\n",
    "\n",
    "# character to index mapping\n",
    "chartoi = {}\n",
    "# index to character mapping\n",
    "itochar = {}\n",
    "for i, c in enumerate(char_vocab):\n",
    "    chartoi[c] = i\n",
    "    itochar[i] = c\n",
    "print(f\"{chartoi=}\")\n",
    "print(f\"{itochar=}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "For word '.emma.': X=[0, 5, 13, 13, 1] and y=[5, 13, 13, 1, 0]\n",
      "No. of total examples in X: 228146\n",
      "X[:10]=tensor([ 0,  5, 13, 13,  1,  0, 15, 12,  9, 22]) and y[:10]=[5, 13, 13, 1, 0, 15, 12, 9, 22, 9]\n"
     ]
    }
   ],
   "source": [
    "# compute all the bigrams for names\n",
    "X = []\n",
    "y = [] \n",
    "for i, name in enumerate(names):\n",
    "    name = '.' + name + '.'\n",
    "    for ch1, ch2 in zip(name[:-1],name[1:]):\n",
    "        X.append(chartoi[ch1])\n",
    "        y.append(chartoi[ch2])\n",
    "    if i == 0:\n",
    "        print(f\"For word '{name}': {X=} and {y=}\")\n",
    "print(f\"No. of total examples in X: {len(X)}\")\n",
    "\n",
    "X = torch.tensor(X)\n",
    "Y= torch.tensor(y)\n",
    "\n",
    "print(f\"{X[:10]=} and {y[:10]=}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our dataset contains **228146** samples."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Giving an integer as input to a neural network doesn't make sense, so we will convert the integer index to a one-hot vector representation. Our vocabulary has 27 (`vocab_size`) unique character, therefore we need a one-hot vector of size 27. If the character has index *i*, then the one-hot vector will have *1* at *i-th* position and all the remaining positions will be zeros."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x7f79c8a42ca0>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAACbCAYAAABLYdHrAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/P9b71AAAACXBIWXMAAAsTAAALEwEAmpwYAAAJSElEQVR4nO3dX4xcZR3G8e9jW8CCRhA0UKoFgyaEGCAbomKIwT8IGtFETUkw4E29EAVjougN3pgYgwQvDElFDEaEGEAlhliIQtSbylIa/q0gwQqlFVASBYy0ys+LnZp1u+2e1TkzrzPfT9LszDln9vzevtOn7757zjupKiRJ7XrFuAuQJB2cQS1JjTOoJalxBrUkNc6glqTGGdSS1LjVfXzTo49aVRvWr+l8/KP3r+2jDEn6v/F3XmRPvZSl9vUS1BvWr+E3W9Z3Pv6c407towxJ+r+xtX5+wH2dpj6SvD/JI0keS3L50CqTJC1r2aBOsgr4FnAucDJwQZKT+y5MkjSvy4j6DOCxqnq8qvYANwHn91uWJGmfLkG9DnhywfOdg22SpBHoEtRL/RZyv5WckmxKMptk9tk///N/r0ySBHQL6p3Awks4jgd2LT6oqjZX1UxVzRzz2lXDqk+Spl6XoL4HOCnJCUkOATYCt/VbliRpn2Wvo66qfyS5BNgCrAKuq6qHeq9MkgR0vOGlqm4Hbu+5FknSEnq5M/HR+9dO5d2GW3ZtX9Hx0/h3JGnlXJRJkhpnUEtS4wxqSWqcQS1JjTOoJalxBrUkNc6glqTGGdSS1DiDWpIaZ1BLUuMMaklqnEEtSY3rZVGmaeUiS+1Y6QJZYP+pXY6oJalxBrUkNW7ZoE6yPsldSeaSPJTk0lEUJkma12WO+h/A56tqW5JXAfcmubOqHu65NkkSHUbUVbW7qrYNHj8PzAHr+i5MkjRvRXPUSTYApwFbe6lGkrSfzpfnJTkCuAW4rKr+usT+TcAmgMNYO7QCJWnadRpRJ1nDfEjfUFW3LnVMVW2uqpmqmlnDocOsUZKmWperPgJ8B5irqqv6L0mStFCXEfWZwCeAs5NsH/w5r+e6JEkDy85RV9WvgYygFknSErwzUZIaZ1BLUuMMaklqnEEtSY0zqCWpcQa1JDXOoJakxhnUktQ4g1qSGmdQS1LjDGpJapxBLUmNM6glqXGdP+GlT1t2bV/xa8457tSh16HJ4ftDk8QRtSQ1zqCWpMZ1Duokq5Lcl+SnfRYkSfpPKxlRXwrM9VWIJGlpXT+F/HjgA8C1/ZYjSVqs64j6auALwMv9lSJJWsqyQZ3kg8AzVXXvMsdtSjKbZHYvLw2tQEmadl1G1GcCH0qyA7gJODvJ9xcfVFWbq2qmqmbWcOiQy5Sk6bVsUFfVl6rq+KraAGwEflFVF/ZemSQJ8DpqSWreim4hr6q7gbt7qUSStKRe1vp481v/xpYt2zsf77oMknRgTn1IUuMMaklqnEEtSY0zqCWpcQa1JDXOoJakxhnUktQ4g1qSGmdQS1LjDGpJapxBLUmNM6glqXG9LMr06P1rXWhJE2/Lru0rfo3/LvTfcEQtSY0zqCWpcZ2COslrktyc5LdJ5pK8ve/CJEnzus5RfxP4WVV9NMkhwNoea5IkLbBsUCd5NXAWcDFAVe0B9vRbliRpny5THycCzwLfTXJfkmuTHL74oCSbkswmmd3LS0MvVJKmVZegXg2cDlxTVacBLwKXLz6oqjZX1UxVzazh0CGXKUnTq0tQ7wR2VtXWwfObmQ9uSdIILBvUVfVH4MkkbxlsejfwcK9VSZL+retVH58Bbhhc8fE48Mn+SpIkLdQpqKtqOzDTbymSpKX0staHNGwrXVdjFGtquG6HRsVbyCWpcQa1JDXOoJakxhnUktQ4g1qSGmdQS1LjDGpJapxBLUmNM6glqXEGtSQ1zqCWpMYZ1JLUOBdl0n5cAElqiyNqSWqcQS1JjesU1Ek+l+ShJA8muTHJYX0XJkmat2xQJ1kHfBaYqapTgFXAxr4LkyTN6zr1sRp4ZZLVwFpgV38lSZIW6vIp5E8BVwJPALuBv1TVHYuPS7IpyWyS2b28NPxKJWlKdZn6OBI4HzgBOA44PMmFi4+rqs1VNVNVM2s4dPiVStKU6jL18R7g91X1bFXtBW4F3tFvWZKkfboE9RPA25KsTRLg3cBcv2VJkvbpMke9FbgZ2AY8MHjN5p7rkiQNdLqFvKquAK7ouRZJ0hJc60P7cV0NjVuL682Mk7eQS1LjDGpJapxBLUmNM6glqXEGtSQ1zqCWpMYZ1JLUOINakhpnUEtS4wxqSWqcQS1JjTOoJalxqarhf9PkWeAPS+w6GvjT0E/YPts9XWz3dBlWu99YVccstaOXoD6QJLNVNTOyEzbCdk8X2z1dRtFupz4kqXEGtSQ1btRBPa0f4WW7p4vtni69t3ukc9SSpJVz6kOSGjeSoE7y/iSPJHksyeWjOGcLkuxI8kCS7Ulmx11Pn5Jcl+SZJA8u2HZUkjuT/G7w9chx1tiHA7T7K0meGvT79iTnjbPGYUuyPsldSeaSPJTk0sH2ie7vg7S79/7ufeojySrgUeC9wE7gHuCCqnq41xM3IMkOYKaqJv7a0iRnAS8A36uqUwbbvg48V1VfG/wHfWRVfXGcdQ7bAdr9FeCFqrpynLX1JcmxwLFVtS3Jq4B7gQ8DFzPB/X2Qdn+cnvt7FCPqM4DHqurxqtoD3AScP4LzaoSq6pfAc4s2nw9cP3h8PfNv6olygHZPtKraXVXbBo+fB+aAdUx4fx+k3b0bRVCvA55c8HwnI2pcAwq4I8m9STaNu5gxeH1V7Yb5NznwujHXM0qXJLl/MDUyUVMACyXZAJwGbGWK+ntRu6Hn/h5FUGeJbdNyqcmZVXU6cC7w6cGPyZp81wBvAk4FdgPfGGs1PUlyBHALcFlV/XXc9YzKEu3uvb9HEdQ7gfULnh8P7BrBeceuqnYNvj4D/Ij5aaBp8vRgXm/f/N4zY65nJKrq6ar6Z1W9DHybCez3JGuYD6sbqurWweaJ7++l2j2K/h5FUN8DnJTkhCSHABuB20Zw3rFKcvjgFw4kORx4H/DgwV81cW4DLho8vgj4yRhrGZl9YTXwESas35ME+A4wV1VXLdg10f19oHaPor9HcsPL4HKVq4FVwHVV9dXeTzpmSU5kfhQNsBr4wSS3O8mNwLuYX0nsaeAK4MfAD4E3AE8AH6uqifrF2wHa/S7mfwwuYAfwqX1zt5MgyTuBXwEPAC8PNn+Z+fnaie3vg7T7Anrub+9MlKTGeWeiJDXOoJakxhnUktQ4g1qSGmdQS1LjDGpJapxBLUmNM6glqXH/Au8ipHqJRSxRAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# pytorch has a in-built function for one-hot representation, torch.nn.functional.one_hot(tensor, num_classes=-1)\n",
    "# convert integer to float data type for nn input\n",
    "\n",
    "X_enc = F.one_hot(X, num_classes=27).float()\n",
    "\n",
    "# visualize the one-hot representation of the inputs\n",
    "plt.imshow(X_enc[:10])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Let's define our neural network architecture: \n",
    "We will use a single linear layer of 27 (`vocab_size`) hidden units (neurons) without bias and a output softmax layer.<br>\n",
    "**One hidden layer**: 27 hidden units and takes an input one-hot vector of dimension 27, so the weight matrix, W, will be of shape (27x27).<br>\n",
    "**Weight initialization**: Initialize the weight matrix, W, with random numbers from a standard normal distribution with mean 0 and variance 1. If W is initialized with all zeros or with some constant $\\alpha$, then all the hidden units will learn the same features.<br>\n",
    "We are using 27 hidden units (neurons), so that we can compare this W matrix with the `probs` matrix from bigram count-based LM in Part-1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number parameter in NN: 729\n"
     ]
    }
   ],
   "source": [
    "print(f\"Total number parameter in NN: {27*27}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5, 1.0, 'W (Initialized with Std. Normal distribution)')"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAARIAAAEICAYAAACTenveAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/P9b71AAAACXBIWXMAAAsTAAALEwEAmpwYAAAifklEQVR4nO3deZRkdXn/8fdT1dvsO7MyCzCyyuaIRFY1cSFRMCpKjKL+EvRE/OmJJ4nrETVG44lbJNGgcoQoKlFR8hNURBwQARlwgGEGGGBmYJhh9n3trnp+f9zbUDP0fb7dc2umuvHzOqdPd9dT995v3ap66tatp56vuTsiImVUWj0AERn6lEhEpDQlEhEpTYlEREpTIhGR0pRIRKS0Q5JIzOx2MzvlAJZ7q5n9MoifZWYP93Nd7zCz3zb8v93MjhjomBLb+I2Z/U3JdXzdzD4exC8zs++U2cYAx3NIt3cw7f8Y6Mf1l5vZn+Z/f8TMvtnEsTzz+DOzb5vZPzdx3eFjaADreZ2Zfb8/1w0TiZl92Mxu2O+ypQWXvaVgHa8Ftrn7H/L/+/3AdPfvuvsrG9blZnZUQ/w2dz+6P+vqY90j3f3xA1n2YHL397j7pwHM7FwzW1lmfWZ2vpktNLOtZrbezG42s9l5rKlJIl+fm9mbGi5ryy+b3azttIK7/4u7J18k+vti0qzHX1/JsfExVIa7Xw+cYGYnpq6bOiK5FTjDzKoAZjYFaAdO3e+yo/Lr9uU9wH/3c+zSRHnSvRr4IDAGmAP8J1A/iJvdCHyq9/FRhpm1NWE8g8oQvE3fAy5JXSmVSO4mSxwn5/+fDdwCPLzfZY+5+6r9FzazDuDlwPyiDeSvVu/Jj2o2mdl/mJnlsWeyrZn1Jqr78sPCN+//im1mHzKzx8xsm5ktNrPXJ7Z7lJlNy9fX+7PTzLzheu8ysyX52H5hZrMaYn9mZg+Z2RYzuxywgm11mdkuM5uY//8xM+sxs9H5//9sZl/O//52/v8I4EagcXzT8lV2mNnV+e180MzmFdzMk4Fl7n6zZ7a5+4/c/QkzezXwEeDN+brvy7c/x8zm5+u+CZhYtA8L/BzYC/x1wb4Yk499nZmtyPdFJY+9w7K3wV8ys43AZfn++E8zuzEf5+1mNsXMvpzfJw9Zw9vmgTwG+hjb2/IxbTCzj+4Xe+boLb8/v5Nfb7OZ3W1mk83sM8BZwOX5WC/Pr+9m9l4zWwosbbjsqIZNTDSzm/Jxz+99nJnZ7Py6bQ1j+Y2Z/Y2ZHQt8HfiTfHub8/g+b5XM7G/N7FEz22hm1zc8jsLnX+43wJ+n9l2YSNx9L3AXWbIg/30b8Nv9Lis6GpkL1N09dXj+F8CLgZOAC4FX9TGW3u2dlB8W/qCP9TxGdkeOAT4JfMfMpkYbdvdV+fpGuvtI4Drg+wBmdgHZk+0vgUlkt/17eWwi8CPgY2RPtseAMwq2sZssKZ+TX3Q2sKLh+mezX7J19x3Aa4DG8fUm69flYxwLXA9cXnDz7gWOyZ+YLzOzkQ3r/znwL8AP8nWflIeuAe7Jb9OngYsL1l3EgY8DnzCz9j7iXyW7f44g2x9vB97ZEH8J8DhwGPCZ/LILeXY/7wHuyG/bROCHwBcblh/wYwDAzI4Dvga8DZgGTABmFFz94nz9h+fXew+wy90/SvYYuTTfp5c2LHNBftuOK1jnW8n290RgIfDd1JjdfUm+7Tvy7Y3t43a9HPgs2T6cSva42/+8R/T8WwLM7n3RK9Kfk63zeTZpnEW2o27b77KiI46xwLZ+bONz7r7Z3Z8gO+I5uR/LPIe7/0+eGOp5olkKnNbf5c3sn4BjgHflF70b+Ky7L3H3HrIn3sn5q8V5wGJ3/6G7dwNfBp4OVj8fOCd/ZTkR+Pf8/y6yO/G2AdzU37r7De5eI3vbeFJfV8rfg58LTAeuBdbnr1Yj+7q+mc3Mx/Jxd9/j7rcC/zuAcfVu93pgHbDPuQLL3u68GfhwfnS0HPgC2ZO31yp3/6q797j7rvyy69z9njwhXwfsdver89v/A+CZI5ISj4E3Av/P3W919z1kybDoLWA3WQI5yt1r+di2Jtb/WXff2HCb9vezhm1/lOwo4/B+jDvlrcCV7n5vvu4P5+ue3XCd6PnX+/wdG22kP4nkVuBMMxsHTHL3pcDvgJfml51A8RHJJmBUP7bR+ATcCfT5QE8xs7dbdmJxc36YdwL9PDQ3s9cA7wcuaLizZwFfaVjfRrK3L9PJXrWe7F3es28/Pkmx+WRP6lOBB4CbyF6RTwcedff1/byZ8Nz91WUF773d/U53v9DdJ5El/bPJHqh9mQZsyo+Geq0YwLgafSzfTlfDZROBjv3WuYJsf/bqax+uafh7Vx//P/N4KfEY2P/+3AFsKLjufwO/AL5vZqvM7PMFR1+NosfGPnF33072WJtWfPV+m0bD/s7XvYF993n0/Ot9/m6ONtKfRHIH2WHcJcDt+WC2Aqvyy1a5+7KCZZcCZmbTC+JNkx8lfAO4FJiQH+YtouC8xX7LHg1cBVzo7o13+JPAu919bMPPMHf/HbCa7NC2dx3W+H8ffgccDbwemO/ui4GZZO8/i47omvrVbHe/G/gx2ZOrr/WvBsZZdn6m18wD3NZNwKPA3zVcvJ7s1XxWw2UzgacaFz2Q7UG5xwDPvT+Hkx11PIe7d7v7J939OOClZG8N3p4Yf+p2NW57JDCe7DnWm9SHN1x3ygDWu4qG/Z3ftxPYd59HjgWWp464kokkf3VeAPw9+x5+/za/rOhohPyQ/1c8e26grDVk7637MoJsp64DMLN38uwTplD+3u+nwMfcff8ag68DHzaz4/PrjrFnP9r8GXC8mf1lfjTwf9n3Dt6Hu+8kO/fwXp5NHL8je/tUlEjWABPMbEzqdvTFzM7MT7Qdlv9/DNn5lTsb1j/b8pOd7r6C7L7+pJl1mNmZwGsPZNu5jwL/2PtP/lbkWuAzZjYqf+L/PdCsj6AP6DGQ+yHwF/k+6wA+RcHzIz/f9ML8rdpWsuRYy8PRYzRyXsO2Pw3c5e5Puvs6sif9X5tZ1czeBRzZsNwaYEa+XF+uAd5pZiebWSfZ2/O78reV/XEO2Un/UH8L0uaTnfxqfKLdll9WmEhy/8W+74HLuAy4Kj9svbAxkL/Cf4HsCGoN8ELyI6iEU8mOFL5oDZ/e5Ou8DvhXskPYrWSvbq/JY+uBNwGfIztUnNuP7c0n+xTs9w3/j6JgH7r7Q2Qndx/Pb/NAD3U3kyWOB/Lb9HOycwyfz+P/k//eYGb35n//FdlJwY3AJ8g+Pn5Gvn/O6s/G3f12nr2tvd5H9ir7ONnj6Rrgyv7fpHB7B/oYwN0fJEvy15AdnWwCij4kmEKWeLaSnYycz7PJ8CvAG/NPQP59AMO/hmx/bwReRHZuo9ffAv9A9jg7nuwFqNevgQeBp83sOW+P3f1msvM9P8pv15FAnzVfBS4iew6HzA9BYyPLPsJ9n+dFaSIy+FlWTPo2d78wed1DkUhE5PlNX9oTkdKUSESkNCUSESmtJV8gau8c4Z3DxxfGq1uLiv9yFpcFeL0Wxq0tcbPr8XkjryXW31H0SVzjNuJ1kBrj3u443pGoj0otn1Af0RXGj54T19c99MSkMF7ZvCOMW2e8jz1x+6yrM4wDsHdvvI1a/N1Ha0/cB/V4+e458fL1evHzoHvdZmpbd/SnfqYpmpJILPsC2FeAKvBNd/9cdP3O4eM56eXvL4yP/NXieHuJJ1l9W1yVX53YZ53RM3xP/ACqbYmrodtmFH1Fo2EbWxPfHJgSP9F8WVwoaYfHnxT7ytXx9hMn4fecfmwY//XV3wrjZ7333WF8+E/2/9R4X9WZs8N4fUVcb2VHHxnGAWxZvI7a9jjZtU0pLCsCwLdvD+Orvxh/RWjn7uJk+uSHvh4u22yl39rkRTn/QVZfcRxwUf4FKBH5I9GMcySnkX1X5PH828LfB85vwnpFZIhoRiKZzr5fSFrJvl8IAsDMLjGzBWa2oHtPfEgnIkNLMxJJXyd0nvMG292vcPd57j6vvfOAvtwrIoNUMxLJSvb91usMsm8cisgfiWYkkruBuZa16Osg+0LQ9U1Yr4gMEU35ro2ZnUfWIaxK1o3pM9H1x1Qm+Old5xXGvacn3F7q49va+qJ+NJnKnESLjY2bw/DeF84O4213PBivH9K1Kj1xHUT9jD6boj2jeueiePmXxN+ur/w+8RF8qg4jUSNRGR33u+pZsy5e/7z4g8G2tVvi5XsSdTyA74rrmWobN4Xx6oTiWikAJowLw7Z7T7z9p4o/wr+z5xdsrW8cWnUk7n4DcEPyiiLyvKQSeREpTYlEREpTIhGR0pRIRKQ0JRIRKU2JRERKa82ExtUKNnJEYdi3JL5iX43np65Ojb++XVuemEG0En/83rEoMWdUZ7rXRX3nzvgKFuf4tq27w7gnlt9y5LA4/qoXhfHZn74njFenTQ7jVOLx7f7zePvDbrw3jNcTrSb6Uz+16c2nhvEJNz4ab2NaYl6uh4umg8qNGB7HBxEdkYhIaUokIlKaEomIlKZEIiKlKZGISGlKJCJSmhKJiJTWkrl/R4+a7i8+5e8K4233PRYuX98R12DUzo57dXQ+lJhmYGri8/9FS8Nwat4bgMoJc8N4PbGN6vix8QYSdSS1DRvjxdsTdRinHB0vf89DcTzRzyR1H6dufz1Vi5SoFQLwPXE/kMqouKdKPTHdRGV4XCeSqjWK5k+6c8+NbK1vOGT9SHREIiKlKZGISGlKJCJSmhKJiJSmRCIipSmRiEhpSiQiUlpr+pFs30Xl9vsKw5Xp0+Llu+M5X9qWxXOieC2ec8X/EM9LY4l+I5WudB+J+v1xnUVyTpTJca1Lz5iueP3b4jqL5Lw1D8U9WWrde+PlE3UcbbNmhPGeFXFPmUpi/JXDErVCQH1Mcc8cAB5PjGFkPDVtqs4kJZrfydYc2qe2jkhEpDQlEhEpTYlEREpTIhGR0pRIRKQ0JRIRKU2JRERKa0kdiXW00zZlemF8x4lTw+VH3NMTxmsTR4fxpR+IawiO+XBxnweA+voNYbwyMVEDAslaGEv0qtg9Ja5R6Fz0ZBj3xLwyqX4mqV4dbbMOj9dfj/vgpOpEtr9hXhgf85vH482nakSAysa41qZ21Mww7gsXx+tP9SPZsSOOTwge5xviuZ+arSmJxMyWA9uAGtDj7vG9LCLPK808InmZu69v4vpEZIjQORIRKa1ZicSBX5rZPWZ2SZPWKSJDRLPe2pzh7qvM7DDgJjN7yN1vbbxCnmAuAeiqxk1zRWRoacoRibuvyn+vBa4DTuvjOle4+zx3n9dRHdaMzYrIIFE6kZjZCDMb1fs38EpgUdn1isjQ0Yy3NpOB68ysd33XuPvPowW8o429s4prOYYviHtdvP/2W8L4l056zgHRPo7+5GFhnMRcP9F8IgC1VU/H6wc2v/GUMD7+92vDeMf6xJwnFvf78GpcZ+A9ca2OdyfiiXlp1rzhBWF88g/j5YevjfudeKLfij+4KYwD+Oy4FsYeWR7GK2PHxsuPid/i26ziWisA27arOJjoudNspROJuz8OxDNSicjzmj7+FZHSlEhEpDQlEhEpTYlEREpTIhGR0pRIRKQ0JRIRKc08UXx1MIyujPfT215VGE81fLExceMiEsVUK99yRBif9s0HwrjvjYuhquPGhnGAWqI5UqroLaXnRUeH8e5RcQlR17rdYdy6a2HcFz0Sxitz4qZAJBov1ZfHjZuqUyeH8Z4nnoq3D1RGJB6HMxMTuT0VFybWdwQFZYC1x/eR14rvgzv33MjW+oa4KrGJdEQiIqUpkYhIaUokIlKaEomIlKZEIiKlKZGISGlKJCJSWosmyOqgMjNoGrM2rrGImiIBdDwWf34/aWFcI+G74s/3k5NH9aepTKKxEInGRPVdiTqPxBg6b1wQLz8sboeZqqVJ1QL50+vCeP34OfH6VyRKJBJ1KG0z46ZBAE+9dkYYn3rlfWE8VQtUnRI32PJhnWG8vjSYBOwQ14fpiERESlMiEZHSlEhEpDQlEhEpTYlEREpTIhGR0pRIRKS0ltSR0NMT1orsPm1uuHjnXXGvC+/qCuPtm+IaDNoSuyVRA1LbsDFeHqiOH5u8TiT5CnDfo/Hys+LJn1L9OupnnRjG/bb7w7idemwc74nrYKwrrrHYPXtCGO984IkwDjDt2ngf2qiRYby2IZ6EqxL0EwFgW1wrY23txcHuQ9aKBNARiYg0gRKJiJSmRCIipSmRiEhpSiQiUpoSiYiUpkQiIqW1po7EKtBZXAcw7KG4n4jPmBLG64/HNQK2c2e8/lQvh73dYbg6ckS8PFDbuDlex9Hx3Dv29Pp4A9XEvDBPrw3jfvoJ8ervXBzHE/PK1O6La4E4Ma4lqm3ZGsY71u4I4/XNW+LtA9UZU8O4b9ocLz8tsQ9WrQnjlRFxT5iolsZ6BmkdiZldaWZrzWxRw2XjzewmM1ua/x53cIYpIoPZQN7afBt49X6XfQi42d3nAjfn/4vIH5l+JxJ3vxXYv/b7fOCq/O+rgAuaMywRGUrKnmyd7O6rAfLfhU0ozewSM1tgZgv21hM9UUVkSDlkn9q4+xXuPs/d53VU4pNIIjK0lE0ka8xsKkD+O/4oQESel8omkuuBi/O/LwZ+WnJ9IjIEWbJmoveKZt8DzgUmAmuATwA/Aa4FZgJPAG9y92QzjtE23l9SfWVhvC1Rg9CzanU81kS/EDvuqDDuS4L5QoDKrMScKKk5awDbE88LU1sd1xh4d088hCNmhvHdRyT6ddy+JN5+T2L7h02Kl0/MHeSpeXsS8+ZQSdRRjI57iQD4k6viMSTqTNgc17pYou9Nz9OJOpNgH9y562dsqa0/ZMUk/S5Ic/eLCkKvaNJYRGSIUom8iJSmRCIipSmRiEhpSiQiUpoSiYiUpkQiIqW1pB+JmVHpKJ6Tw3fEvSSqx8R1IPWly8J4Zd3mMO5jx8TxVB+JSXGNBoC3x7s+VWOQqiOpJ2oghtXieWM8Uadh40aHcXbEdSIPfyzuNzL7f+OeLx23Pxhvvz2Y8wWoWLrE4ul3nhLGp94Yz/2z+8TZYbzr4bgeqnrUnDAezQ3FoW1HoiMSESlPiURESlMiEZHSlEhEpDQlEhEpTYlEREpTIhGR0lozr021io0aVRiuzyxs/ZotvnF7GK9MGB/G65PGhnGWxfUBNqwrjPvWeHwA2859QRgfmahVsUS/DT8pXn/9/qXx+kfEdST+RLyP6ok+N0f/Vzz3T2puIubGNRb1xYnb1486ksO+cXe8jUStT8eGTWF82yuODeOj7osfA7U9ewpjXu9fn6Fm0RGJiJSmRCIipSmRiEhpSiQiUpoSiYiUpkQiIqUpkYhIaS2pI+ke38mqtxT3o5h21aJw+e3nHBPGR96fmPdmQzzfSM/2uA7E2uJeF9WJcR0LwMhfLQ7jqZ4mtafjSQ2rS1eG8VSVwZ6T4jqNzvvini++Pe4ps/FF8e0btyKuUyERt1PjGo36A3GdCUAlqHUCqO/cGcdPPDKMj/hl/Djv2V1cJwJgiZ42h5KOSESkNCUSESlNiURESlMiEZHSlEhEpDQlEhEpTYlEREozT/SNOBhGVyb46Z2vKYzXTj8uXL5y2/1hvHrkrDBuu/eG8VSNhh17RBhffkG6jmT2F+4L494Tz1vjQS8KgPqZJ4fx9rXbwjhbEvFdu+PtJ8ZXGR3Pi+Pb4u1bV2cYr23eEsbbpkwO4wAk+s7UV8f9QlL7wDo6SsV3BP1MFt7yFbZvWnnIZrfp9xGJmV1pZmvNbFHDZZeZ2VNmtjD/Oe/gDFNEBrOBvLX5NvDqPi7/krufnP/c0JxhichQ0u9E4u63AhsP4lhEZIhqxsnWS83s/vytz7iiK5nZJWa2wMwWdHv8/lpEhpayieRrwJHAycBq4AtFV3T3K9x9nrvPa7f4JJaIDC2lEom7r3H3mrvXgW8ApzVnWCIylJRKJGY2teHf1wPx96JF5Hmp3w0NzOx7wLnARDNbCXwCONfMTiZrb7EceHc/14UFc4J0LF8fLu/jxsQbqMT50ffGdSRWjZe3p+I6kznXxPUDACRqBEjU91Qmxv08Kn+I+214Yh+l5n2pbY17ulQnx3MTUauF4fru+Dya1epxvDOuM6lvTdTJAJ6YlybFqtUwXntx3DOlfUk8t8/wny0sjFW6d4XLNlu/E4m7X9THxd9q4lhEZIhSibyIlKZEIiKlKZGISGlKJCJSmhKJiJSmRCIipbVkYgzv6sCPDeZNWRf3krAxcS+LnocfjZcPalggXYNgo0aG8dryJ8M4QGVC3LOkvimuYagkahRS/Uzs6HjemvrieB9SSWx/6sQwvvnY+D4ctzDePxtPjetoJsyP74PaurhWCYBKuXYeduwLwnjbgofC+FPvOTWMT//Ow8Xb3nRojxF0RCIipSmRiEhpSiQiUpoSiYiUpkQiIqUpkYhIaUokIlJaS+pIbG8PlRXFc4L4uMScJ6sT884k6kCq06bE60/0qvDEnCn9sef4w8N49YipYdzvfjCOJ+pIqtvjfh898xJzCz24LIxTj/uFjPvlI/HyCWN/GPfqYNzYMJzqtwJQ39uduELcU8WWrgjjqfto2jfiuY92vOz4wlj91vg50Gw6IhGR0pRIRKQ0JRIRKU2JRERKUyIRkdKUSESkNCUSESmtJXUk1Gr4tqBWY8eOcPHKYXGvi72z4l4VdteSMO4vnBsv/0A8Z0yq3wlAx91xHUVl/NgwXrPEa0CiTmLNK+I6lck3xDUQtdTcQI8lerJMj2t56svi5SsjhoXxnjWJWqOT4zoZAKvG+9AWJfredCX62kybHMe74zqT0z51d2Fs6V/tDJdtNh2RiEhpSiQiUpoSiYiUpkQiIqUpkYhIaUokIlKaEomIlNaSOpIJx+/mndcV13J86wXxnCtsiOd8ad8e16Hs+NMTw/jwX90fbz9h5aXxfCQA0796Txj3jvYwXj18WhjfcmpcozDx2/H2mTAuDFeGdYXx7hceEca3zYprLMZv3BzGSc3rc3p8H9fb0q+h7Uvinic2Ka5n8uHxPqqPiuPcF9caLTpreGFs185BOq+NmR1uZreY2RIze9DM3p9fPt7MbjKzpfnv+BEoIs87A0lbPcAH3f1Y4HTgvWZ2HPAh4GZ3nwvcnP8vIn9E+p1I3H21u9+b/70NWAJMB84HrsqvdhVwQZPHKCKD3AG9kTKz2cApwF3AZHdfDVmyAQ4rWOYSM1tgZgu2bYy/QyAiQ8uAE4mZjQR+BHzA3bf2dzl3v8Ld57n7vFHjW/NdQRE5OAaUSMysnSyJfNfdf5xfvMbMpubxqUD8tUsRed4ZyKc2BnwLWOLuX2wIXQ9cnP99MfDT5g1PRIaCgbzHOAN4G/CAmS3ML/sI8DngWjP7P8ATwJuaOkIRGfTM3Q/5Rsd0TPaXTrmoMF7fGBec1XfGTVtW/eNLw/iMyxeG8dTkTiv+IS44m3153DgJwPfsia9w1MwwbMtXhfFaoijPKnHTnmqq2GpHfB/YiOJiKYD6lsTptcR94LU4vuWN8X007ucPx9sn/TijHj93KmNGhXHfuSuMb3ldXFTXFXxocc8dX2XblpXpWcCaRCXyIlKaEomIlKZEIiKlKZGISGlKJCJSmhKJiJSmRCIipbXmSy8Vg6BxT2qCqfo5p4Tx6V9aEG8/0TSonqjxmPOdp+Lld8X1AQCp+p3auHgCqOqS3WG8bWrc2Ki2fkMY3/Dy2WF87LX3hvHqqJFh3GZNj+Nb4zqYWmICrPG/i++jWjRBW+8YOuPmS96T+PJpKp6YxGzsDYvDeFQrZDVNkCUiQ4wSiYiUpkQiIqUpkYhIaUokIlKaEomIlKZEIiKltaSOZO/Ydp54Q/EETzO+Htc4VG6LJ7Cy4+aGce+Kb3Z1xZowzq64hsOOnBUvD/gjy8J4+0NxHURqAq361rhOopKo8xi+pjuMk+hnUlu7PozXZ04K49VlT8abHzki3v7TcZ3J6vedFsYBZvwkvg/YHdcb1dbF+8DnHRfG21bGz4O2ieMLY7ayI1y22XREIiKlKZGISGlKJCJSmhKJiJSmRCIipSmRiEhpSiQiUlpL6kg6Nuzh8KsfLYzX9+4Nl68M64rjGzaH8Z6n4zqR+ouOD+PVp+L6ANYk4kA1qAEA8MQ+2H3msWG889dxrU1Kx/wHSi2Px/POtD0Y19FweHGdEQCbE/1E5sbzAg1fE48PwDfE8yuR6FfSffZJYbzj7kfCeC31PDgiuI2JXifNpiMSESlNiURESlMiEZHSlEhEpDQlEhEpTYlEREpTIhGR0lozr021AqOL+2HYlq3x8ok5YVI1GI99/vQwPveyRA3G6FFhuPsHw+Plge5/mxLGh90S13F0zV8Ub6Aav0ak5g5y4l4bKTYsnpenPvfweAWLiuuMIN1PxRY/HsZ3fjSuMwEYszTRV2bhw2F481FxT5DJC+Jaj+4/ieuZ2u9ZWhzcm+gn02T9PiIxs8PN7BYzW2JmD5rZ+/PLLzOzp8xsYf5z3sEbrogMRgM5IukBPuju95rZKOAeM7spj33J3f+t+cMTkaGg34nE3VcDq/O/t5nZEiCed1FE/igc0MlWM5sNnALclV90qZndb2ZXmtm4gmUuMbMFZrZgby09N66IDB0DTiRmNhL4EfABd98KfA04EjiZ7IjlC30t5+5XuPs8d5/XUY1PxInI0DKgRGJm7WRJ5Lvu/mMAd1/j7jV3rwPfANLtuUXkeWUgn9oY8C1gibt/seHyqQ1Xez2Q+FxSRJ5vzBM1Gc9c0exM4DbgAaC3mcNHgIvI3tY4sBx4d35ittDoEdP89OMuKYxXVq4Lx1LfvCWMp+pIqqPiOhDv6Qnj1ONeFjZmdLw8JD/n33vSnDDefufieAyJOg7fFZ+nssS8MST2UXIfJPpleGLuIJ8U93OpbNsRL9+ZnvfFdsZjoFYLw8vedUQYn3pHXKvTuSCoEwGYUVyLdMdjV7Jl1+pD1pRkIJ/a/Bboa2A3NG84IjIUqUReREpTIhGR0pRIRKQ0JRIRKU2JRERKUyIRkdL6XUfS1I2arQNWNFw0EUhPBtM6Gl85g318MPjHONDxzXL3SQdrMPtrSSJ5ziDMFrj7vFaPo4jGV85gHx8M/jEO9vHprY2IlKZEIiKlDZZEckWrB5Cg8ZUz2McHg3+Mg3p8g+IciYgMbYPliEREhjAlEhEpraWJxMxebWYPm9mjZvahVo6lL2a23MweyKfZWNDq8QDkfXHXmtmihsvGm9lNZrY0/91n39wWjm/QTFkSTKsyKPbhUJ32pWXnSMysCjwC/BmwErgbuMjd4449h5CZLQfmufugKVQys7OB7cDV7n5CftnngY3u/rk8IY9z938aROO7DNg+GKYsyTv6TW2cVgW4AHgHg2AfBuO7kEGyD/vSyiOS04BH3f1xd98LfB84v4XjGRLc/VZg434Xnw9clf99FdkDryUKxjdouPtqd783/3sb0DutyqDYh8H4BrVWJpLpwJMN/69k8O0wB35pZveYWXFvyNab3NveMv99WIvH05fklCWH2n7Tqgy6fXgg0760SisTSV9tGwfbZ9FnuPupwGuA9+aH7TJw/Zqy5FDqY1qVQeVAp31plVYmkpVA40zSM4BVLRpLn9x9Vf57LXAdg3eqjTW93fzz32tbPJ59DLYpS/qaVoVBtA+H4rQvrUwkdwNzzWyOmXUAbwGub+F49mFmI/KTXZjZCOCVDN6pNq4HLs7/vhj4aQvH8hyDacqSomlVGCT7cKhO+9LSytb8I6wvA1XgSnf/TMsGsx8zO4LsKASybvvXDIbxmdn3gHPJvla+BvgE8BPgWmAm8ATwJndvyQnPgvGdywCnLDmI4yuaVuUuBsE+bOa0L4eSSuRFpDRVtopIaUokIlKaEomIlKZEIiKlKZGISGlKJCJSmhKJiJT2/wGUGSe5IFspMQAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# for reproduciblity of results\n",
    "g = torch.Generator().manual_seed(2147483647)\n",
    "#weight matrix (learnable weights)\n",
    "W = torch.randn((vocab_size,vocab_size), requires_grad=True, generator=g)\n",
    "\n",
    "# visualize the weight matrix = log counts, therefore we need to take exponent to compare it with `probs` table\n",
    "plt.imshow(W.exp().detach().numpy())\n",
    "plt.title(\"W (Initialized with Std. Normal distribution)\")\n",
    "# plt.axis('off');"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After training the NN, this weight matrix visualiztion should look similar to counts matrix in bigram count-based LM.<br>\n",
    "**W, weight matrix = log counts or logits**, therefore we need to take exponent to compare it with `probs` table\n",
    "#### To learn the weight matrix, W, we will use iterative gradient based optimization method like gradient descent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.4901304244995117 2.4901304244995117\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f79e88fef40>]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/P9b71AAAACXBIWXMAAAsTAAALEwEAmpwYAAAdkklEQVR4nO3de3hc9X3n8fd3rpJGkmVbI99v3C8GDFEckhJCgBaHbEPSJltyabMkKaVlt6TbZzfNdrf78LRpdzdJl6Y0oQTSwCbbhE3YXNiQhBJuTsAgE2PjGzi+Y2NLtmVbsm4z890/5kgejSVrZI88njOf1/PMozPn/HTm+wP7M8e/8zvnmLsjIiLVL1LpAkREpDwU6CIiIaFAFxEJCQW6iEhIKNBFREIiVqkPbm1t9cWLF1fq40VEqtLq1au73D091raKBfrixYvp6Oio1MeLiFQlM9sx3rYJh1zMrM7MXjSzV8xsvZndPUabaWb2w4I2t51u0SIiMjmlHKEPANe7e4+ZxYGVZva4u79Q0OZOYIO7/6aZpYHNZvZNdx+ciqJFROREEwa65y8l7QnexoNX8eWlDjSZmQGNwEEgU8Y6RURkAiXNcjGzqJmtAfYDT7j7qqIm9wIXA3uAdcBd7p4bYz+3m1mHmXV0dnaeXuUiIjJKSYHu7ll3XwbMB5ab2dKiJjcBa4C5wDLgXjNrHmM/97t7u7u3p9NjnqQVEZFTNKl56O7eDTwNrCjadBvwqOdtAbYBF5WjQBERKU0ps1zSZtYSLNcDNwKbiprtBG4I2swCLgS2lrVSERE5qVKO0OcAT5nZWuAl8mPoj5nZHWZ2R9DmL4F3mNk64EngM+7eNRUFb37zKF/4yWYO9moCjYhIoVJmuawFrhxj/X0Fy3uA3yhvaWPb1tXDvU9t4ebL5jAjlTgTHykiUhWq7l4uqWT+O6h3ULMiRUQKVW2g9wwo0EVEClVdoDcOH6Er0EVERqm6QE8p0EVExlR1gd6YGB5yyVa4EhGRs0vVBXoqGQV0hC4iUqzqAj0WjZCMRRToIiJFqi7QIX9iVLNcRERGq8pATyVjOkIXESlStYGuk6IiIqNVZaA3JqM6QhcRKVKVgZ5KxnTpv4hIkaoNdJ0UFREZrSoDvTERo6dfgS4iUqg6A71Os1xERIpVZaDnx9Cz5HJe6VJERM4aVRnojcHl/8eGNHVRRGRYVQa67rgoInKiqgz0Rj3kQkTkBFUZ6KmEjtBFRIpVZ6DrCF1E5ARVGejHH0Onk6IiIsMmDHQzqzOzF83sFTNbb2Z3j9PuOjNbE7R5pvylHqeHXIiInChWQpsB4Hp37zGzOLDSzB539xeGG5hZC/BlYIW77zSztqkpN08nRUVETjRhoLu7Az3B23jwKr6i5yPAo+6+M/id/eUsspimLYqInKikMXQzi5rZGmA/8IS7rypqcgEw3cyeNrPVZvZ74+zndjPrMLOOzs7OUy66IRHFTIEuIlKopEB396y7LwPmA8vNbGlRkxjwFuC9wE3AfzGzC8bYz/3u3u7u7el0+pSLNjNSCT3kQkSk0KRmubh7N/A0sKJo027gx+7e6+5dwLPAFeUocDwpPeRCRGSUUma5pIOTnphZPXAjsKmo2feBd5pZzMwagLcBG8tc6yipZIwePeRCRGREKbNc5gAPmVmU/BfAI+7+mJndAeDu97n7RjP7MbAWyAEPuPurU1Y1+Zkuuie6iMhxpcxyWQtcOcb6+4refx74fPlKO7lUQvdEFxEpVJVXikL+IReahy4iclz1BroeFC0iMkrVBnp+loumLYqIDKviQNeQi4hIoaoN9MZEjMFMjqFsrtKliIicFao20HU/FxGR0ao20HXHRRGR0ao20FN6yIWIyChVHOj5h1zoCF1EJK9qA71RY+giIqNUbaDrpKiIyGhVG+g6KSoiMlrVBrqO0EVERqviQNdJURGRQlUb6MlYlHjU9Bg6EZFA1QY65IddNOQiIpJX3YGuh1yIiIyo6kBv0kMuRERGVHWgp/SQCxGREVUf6DopKiKSV9WB3piMagxdRCRQ1YGuk6IiIsdNGOhmVmdmL5rZK2a23szuPknbt5pZ1sw+WN4yx6bH0ImIHBcroc0AcL2795hZHFhpZo+7+wuFjcwsCvx34CdTUOeYGoN56O6OmZ2pjxUROStNeITueT3B23jw8jGa/jvgu8D+8pV3cqlkjJxD/5CeKyoiUtIYuplFzWwN+bB+wt1XFW2fB3wAuG+C/dxuZh1m1tHZ2XmKJR/XqPu5iIiMKCnQ3T3r7suA+cByM1ta1OQe4DPuftI5hO5+v7u3u3t7Op0+lXpH0R0XRUSOK2UMfYS7d5vZ08AK4NWCTe3At4Jx7FbgZjPLuPv3ylTnmFK6J7qIyIgJA93M0sBQEOb1wI3kT36OcPclBe2/Djw21WEO0BQE+tF+BbqISClH6HOAh4JZLBHgEXd/zMzuAHD3k46bT6WZjUkADvQOVKoEEZGzxoSB7u5rgSvHWD9mkLv7vzn9skrT2pgAoOuoAl1EpKqvFJ3ekCAaMTp7FOgiIlUd6JGIMTOVoOvoYKVLERGpuKoOdIDWxqSO0EVECEGgp5uSdCnQRUSqP9BbG5M6KSoiQhgCvSlBV88g7mPdXkZEpHZUfaCnG5MMZnMc6dPFRSJS26o/0JvyFxfpxKiI1LrqD/TgatFOjaOLSI2r+kBvDY7QNdNFRGpd9Qe6jtBFRIAQBHpLfZxYxHSELiI1r+oDPRIxZjYmFOgiUvOqPtAhuPxfQy4iUuNCEej5y/91gy4RqW2hCPTWRt3PRUQkVIGuy/9FpJaFItDTTUmGss7hvqFKlyIiUjGhCPThR9HpxKiI1LJQBPrI5f8aRxeRGhaOQB+5/F8zXUSkdoUi0HX5v4hICYFuZnVm9qKZvWJm683s7jHafNTM1gavX5jZFVNT7tim6fJ/ERFiJbQZAK539x4ziwMrzexxd3+hoM024F3ufsjM3gPcD7xtCuodUyRiehSdiNS8CQPd85O7e4K38eDlRW1+UfD2BWB+uQosVWtTQidFRaSmlTSGbmZRM1sD7AeecPdVJ2n+SeDxcfZzu5l1mFlHZ2fnpIs9mbSuFhWRGldSoLt71t2XkT/yXm5mS8dqZ2bvJh/onxlnP/e7e7u7t6fT6VMseWy6QZeI1LpJzXJx927gaWBF8TYzuxx4ALjF3Q+Uo7jJaG1KcqBnkFxOl/+LSG0qZZZL2sxaguV64EZgU1GbhcCjwO+6+2tTUOeE0o1JMjld/i8itauUWS5zgIfMLEr+C+ARd3/MzO4AcPf7gL8AZgJfNjOAjLu3T1HNYxp+tmhnzwDTU4kz+dEiImeFUma5rAWuHGP9fQXLnwI+Vd7SJmf4fi5dRwe4YFZTJUsREamIUFwpCjC7uQ6AN7r7KlyJiEhlhCbQF8xoIBoxth/orXQpIiIVEZpAj0cjLJzRwLYuBbqI1KbQBDrAktYUWzsV6CJSm0IX6NsP9GouuojUpFAF+jnpFP1DOd480l/pUkREzrhQBfqS1hSAxtFFpCaFKtDPaW0EYKsCXURqUKgCfVZzkvp4lG06MSoiNShUgW5mLGlNsa2rZ+LGIiIhE6pAB1iSTmkMXURqUugC/ZzWFLsO9TGYyVW6FBGRMyp0gb6kNUU25+w6dKzSpYiInFGhDHRAJ0ZFpOaEN9A1ji4iNSZ0gd7SkGBGKqG56CJSc0IX6ICmLopITQpxoOsIXURqS2gDfd+RAXoHMpUuRUTkjAlloJ+jE6MiUoNCGehL0vlA14lREakl4Qz01hSJaIT1bxyudCkiImfMhIFuZnVm9qKZvWJm683s7jHamJl9ycy2mNlaM7tqasotTTIW5ZK5zfxyV3clyxAROaNKOUIfAK539yuAZcAKM7u6qM17gPOD1+3AV8pZ5KlYtqCFdbsPk8nqni4iUhsmDHTPG57UHQ9exQ/tvAV4OGj7AtBiZnPKW+rkLFvQQt9Qltf2aT66iNSGksbQzSxqZmuA/cAT7r6qqMk8YFfB+93BuuL93G5mHWbW0dnZeYoll2bZghYA1mjYRURqREmB7u5Zd18GzAeWm9nSoiY21q+NsZ/73b3d3dvT6fSki52MRTMbmN4QZ82uQ1P6OSIiZ4tJzXJx927gaWBF0abdwIKC9/OBPadT2OkyM65Y0KIjdBGpGaXMckmbWUuwXA/cCGwqavYD4PeC2S5XA4fdfW+5i52sZQtaeH1/D0f7hypdiojIlCvlCH0O8JSZrQVeIj+G/piZ3WFmdwRtfgRsBbYAXwX+aEqqnaRlC1pwh3W7NR9dRMIvNlEDd18LXDnG+vsKlh24s7ylnb6RE6O7u3nHea2VLUZEZIqF8krRYS0NCZa0plizs7vSpYiITLlQBzrkj9LX7Oom/48IEZHwqolA3390gL2H+ytdiojIlKqJQAddYCQi4Rf6QL94TjP18SjP/+pApUsREZlSoQ/0RCzCNee38rNN+zWOLiKhFvpAB7jhojbe6O7TjbpEJNRqItDffVEbAE9u2lfhSkREpk5NBPqs5jqWzmvmqU37K12KiMiUqYlAB7j+olms3nGIQ72DlS5FRGRK1FCgt5FzeOa1qb0Pu4hIpdRMoF8+bxqtjQme1LCLiIRUzQR6JGK8+8I2ntm8X88ZFZFQqplAB7jh4jaO9GdYvUNPMRKR8KmpQL/m/DTxqPGT9Zq+KCLhU1OB3piM8euXzOL//nI3A5lspcsRESmrmgp0gFvfupBDx4b4qY7SRSRkai7QrzmvlXkt9Xz7pV2VLkVEpKxqLtAjEeN33rqAlVu62HngWKXLEREpm5oLdIAPtc8nYvDtjp2VLkVEpGxqMtDnTKvnugvb+D8duzUnXURCoyYDHeDWty5g/9EBntqsWwGISDhMGOhmtsDMnjKzjWa23szuGqPNNDP7oZm9ErS5bWrKLZ93X9RGW1OSr63cVulSRETKopQj9Azwp+5+MXA1cKeZXVLU5k5gg7tfAVwHfNHMEmWttMzi0Qi3X3sOz289wKqtejydiFS/CQPd3fe6+8vB8lFgIzCvuBnQZGYGNAIHyX8RnNU+dvUi0k1J/ue/vFbpUkRETtukxtDNbDFwJbCqaNO9wMXAHmAdcJe7n3C20cxuN7MOM+vo7Kz82HVdPMofvutcXth6UA+RFpGqV3Kgm1kj8F3g0+5+pGjzTcAaYC6wDLjXzJqL9+Hu97t7u7u3p9PpUy66nD7ytoW0BUfpeoi0iFSzkgLdzOLkw/yb7v7oGE1uAx71vC3ANuCi8pU5deriUf7ounN5cdtBntdYuohUsVJmuRjwILDR3f92nGY7gRuC9rOAC4Gt5Spyqt26fCGzm+v4b49vIpvTUbqIVKdSjtB/Dfhd4HozWxO8bjazO8zsjqDNXwLvMLN1wJPAZ9y9a4pqLru6eJTP3nwRa3cf5hsv7Kh0OSIipyQ2UQN3XwnYBG32AL9RrqIq4X1XzOU7q3fz+Z9sZsXS2cxqrqt0SSIik1KzV4oWMzP+6v1LGcrmuPuH6ytdjojIpCnQCyyameKPbzifH617k59t0v3SRaS6KNCL/P47z+H8tkY+++g6DvQMVLocEZGSKdCLJGIR7rl1GYeODfEnj7xCTrNeRKRKKNDHcOncafzX37yEZ1/r5CvP/KrS5YiIlESBPo6PLF/I+66Yyxd/upkXdMGRiFQBBfo4zIy//q3LWDwzxb/93y/rcXUictZToJ9EYzLGVz/eTibnfPyfXtRJUhE5qynQJ3BuupEHP97Onu4+PvFQB8cGz/q7AotIjVKgl+Ati2bwpQ9fybrd3dz5zZcZyGQrXZKIyAkU6CW66dLZfO4Dl/HU5k4+9VAHfYMKdRE5uyjQJ+HDyxfyP377clZu6eK2r79Iz4CGX0Tk7KFAn6R//dYF3PM7y3hp+yE+9sAqunSiVETOEgr0U3DLsnl8+aNXsenNI9xy78/ZuLf4AU4iImeeAv0U3XTpbB75g7eTyeX44Fd+wRMbdDMvEaksBfppuHx+C9+/8xrObWvk9x/u4G9+tJHBzAnPxhYROSMU6Kdp9rQ6HvmDt/ORty3kH5/dygfv+wXbu3orXZaI1CAFehnUxaP89Qcu4ysfvYrtXb2890vP8fDz23WnRhE5oxToZfSey+bw+Kev5apF0/mL76/nQ//4PK/vO1rpskSkRijQy2xeSz0Pf2I5X/zQFWzt7OHmLz3HXz22gcPHhipdmoiEnAJ9CpgZv/2W+fzLv38Xv3XlfB78+Tbe9YWn+PrPt+mkqYhMGXOvzDhve3u7d3R0VOSzz7QNe47wuR9t4OdbDjCvpZ4/vO5cPtQ+n2QsWunSRKTKmNlqd28fa9uER+hmtsDMnjKzjWa23szuGqfddWa2JmjzzOkWHSaXzG3mG598Gw9/YjltzUn+8/de5d2ff5oHV27jaL+GYkSkPCY8QjezOcAcd3/ZzJqA1cD73X1DQZsW4BfACnffaWZt7r7/ZPutpSP0Qu7Oc6938fc/e52Xth+iKRnj1uUL+NjVi1g0M1Xp8kTkLHeyI/TYRL/s7nuBvcHyUTPbCMwDNhQ0+wjwqLvvDNqdNMxrmZlx7QVprr0gzSu7unlw5Ta+9vPtfPW5bVxzXisfXr6QX79kFomYTm+IyORMagzdzBYDzwJL3f1Iwfp7gDhwKdAE/J27PzzG798O3A6wcOHCt+zYseN0ag+NNw/380jHLr790i7e6O5jWn2cmy+bwweunEf7oulEIlbpEkXkLHGyI/SSA93MGoFngM+5+6NF2+4F2oEbgHrgeeC97v7aePur1SGXk8nmnOde7+R7v3yDn6zfR99QltnNddx06SxWLJ3D8iUziCrcRWraaQ25BDuIA98Fvlkc5oHdQJe79wK9ZvYscAUwbqDLiaIR47oL27juwjZ6BzI8sWEfP1q3l2+9tIuHnt9BS0Ocd12Q5vqL2rj2/DTTU4lKlywiZ5FSTooa8BBw0N0/PU6bi4F7gZuABPAicKu7vzrefnWEXrregQxPb+7kyU37eGZzJwd6BzGDS+c2c815ad5x7kzesmg6qWRJ388iUsVOa8jFzK4BngPWAcNXxfwnYCGAu98XtPsPwG1Bmwfc/Z6T7VeBfmqyOWft7m6ee72LlVu6eHnHITI5JxYxls6bxvIlM7hqYQtXLZxOW3NdpcsVkTIryxh6uSnQy6N3IMPqHYdYte0Aq7YeZO3uwwxm89+781rquXz+NC6f38IV86dxydxmWho0TCNSzU57DF3OXqlkbGQaJMBAJsv6PUd4ecchfrmrm3W7D/P4q2+OtJ/XUs8lc5u5aHYTF85u4qLZTSyamSIe1TRJkWqnQA+ZZCzKVQunc9XC6SPrDvUO8uqew6zfc4T1e46wYc9hnty4j+G7+8YixpLWFOe1NXJuupFz0inOSTeyZGaKaQ3xCvVERCZLgV4DpqcSvPP8NO88Pz2yrn8oy5b9PWx+8yhbOnt4fV8PG/ce4acb9pEtuI97S0OcRTNTLJrRwMIZDSyYUc+C6Q3Mm17PnGn1ugBK5CyiQK9RdfEoS+dNY+m8aaPWD2Zy7Dx4jK2dPew4cIztB3rZceAYa3Z18//W7R0V9mbQ1pRkbks9c1vqmdNcx+xpdcyZVs/saUnamupoa07qJmQiZ4gCXUZJxCKc19bIeW2NJ2zLZHPs6e5nd/cxdh/q441Dfezp7mPP4T7Wv5EfxukfOvH2wNMb4rQ11ZFuSpJuStLamKC1MUlrY5KZwfKMVIIZqQR1cYW/yKlSoEvJYtEIC2c2sHBmw5jb3Z3DfUPsPdzPviP515uHB+js6afz6AD7jw6wfXsvnUcHGBjnvvANiSgzUgmmNySYnkowvSHO9IYE0+rjtDTkX9Pqj7+a6+I018f1RSCCAl3KyMxoaUjQ0pDg4jnN47Zzd3oGMnT1DHKwd4CunkEO9Axy6NggB3vzr0PHBjl0bIjtXb10HxvkSH/mpJ+diEZoro/RVBenqS6WfyXzy411MZqS+Z+pZIzG4DW83JCIkgp+NiRiur2CVC0FupxxZhYEb5wlraXdMjibyx/9D7+GQ/5w3xBH+oY40j/Ekb4MR/qH6OnPcLR/iP1HBugZyNDTn6FnMEOpl1zUxSM0JIYDPkp9IkZDPL9cl4jSEI9Sn4hSH49SN2o5Ql2wri4epS5W+D5CMnb8ZzIW0U3XpOwU6FIVohEbGWc/Fbmcc2woS+9AhqP9GXoHMvQOZugdyHJsMEPPQIZjA1l6BjL0Be36BrMcG8xybCjLsYEM+44OcWwwS99glr6h/M/xho5KEY8adbEoySDkE7EIyViERCxCIhohGc//TMQiJGLRkeXCNvFgXTxqo9bFYxESUcsvj7QzYpHh9/ltsaiRiEaIBcvxSH5bNGLk7/oh1USBLjUhErGRoZZZ448GTVo25wxkjod8/1CO/qEsA5njy8fXHf85vH0wWB7IHF8eXt8/lONIX4bBTI7B7PD2HIOZ7Mj73BRe6B2P5r8AYkH4RyNGPGL58I8YsejwF0T+CyAWybcpbB8raD/8Plrwyr8v2h41oja6fSRYjlh+/xEb/jyI2Oh9Rq2gfeT4vo63y/8rcWT9yO8E+wp+f3hfZoxaPpu/6BToIqchGrFgeKYyf5WyOWcomw/6oWz+NZjJMZT1kS+CTDZ4X7A8lM2RyeUYyhxfn8kNL3u+Xc5H2mdyueCzfKRtJlifyflIHZlcjr6h4++zw9tyOXI58u2zTtbz6wuXs1P57VRGEcsH/8gXgeUPGCLDy8G2keWRL5T8ezP48PKFfOqd55S9NgW6SBXLH3FGQzHLx4eD3QuCPutkck6uIPSzOR/5EskWbitokxvez/DySDvI+ej9lbQ+2F/OR9fpzkjbwuWc54f5ch78Xs5xyK93J92UnJL/hgp0ETkrWDCcEgN0J+hTo+u2RURCQoEuIhISCnQRkZBQoIuIhIQCXUQkJBToIiIhoUAXEQkJBbqISEiYl3oLunJ/sFknsOMUf70V6CpjOdWiFvtdi32G2ux3LfYZJt/vRe6eHmtDxQL9dJhZh7u3V7qOM60W+12LfYba7Hct9hnK228NuYiIhIQCXUQkJKo10O+vdAEVUov9rsU+Q232uxb7DGXsd1WOoYuIyImq9QhdRESKKNBFREKi6gLdzFaY2WYz22Jmf1bpeqaCmS0ws6fMbKOZrTezu4L1M8zsCTN7Pfg5vdK1lpuZRc3sl2b2WPC+FvrcYmbfMbNNwf/zt9dIv/8k+PP9qpn9s5nVha3fZvY1M9tvZq8WrBu3j2b22SDbNpvZTZP9vKoKdDOLAv8AvAe4BPiwmV1S2aqmRAb4U3e/GLgauDPo558BT7r7+cCTwfuwuQvYWPC+Fvr8d8CP3f0i4Ary/Q91v81sHvDHQLu7LwWiwK2Er99fB1YUrRuzj8Hf8VuBS4Pf+XKQeSWrqkAHlgNb3H2ruw8C3wJuqXBNZefue9395WD5KPm/4PPI9/WhoNlDwPsrUuAUMbP5wHuBBwpWh73PzcC1wIMA7j7o7t2EvN+BGFBvZjGgAdhDyPrt7s8CB4tWj9fHW4BvufuAu28DtpDPvJJVW6DPA3YVvN8drAstM1sMXAmsAma5+17Ihz7QVsHSpsI9wH8EcgXrwt7nc4BO4J+CoaYHzCxFyPvt7m8AXwB2AnuBw+7+U0Le78B4fTztfKu2QLcx1oV23qWZNQLfBT7t7kcqXc9UMrN/Bex399WVruUMiwFXAV9x9yuBXqp/mGFCwbjxLcASYC6QMrOPVbaqijvtfKu2QN8NLCh4P5/8P9NCx8zi5MP8m+7+aLB6n5nNCbbPAfZXqr4p8GvA+8xsO/mhtOvN7BuEu8+Q/zO9291XBe+/Qz7gw97vG4Ft7t7p7kPAo8A7CH+/Yfw+nna+VVugvwScb2ZLzCxB/gTCDypcU9mZmZEfU93o7n9bsOkHwMeD5Y8D3z/TtU0Vd/+su89398Xk/7/+zN0/Roj7DODubwK7zOzCYNUNwAZC3m/yQy1Xm1lD8Of9BvLnisLebxi/jz8AbjWzpJktAc4HXpzUnt29ql7AzcBrwK+AP690PVPUx2vI/1NrLbAmeN0MzCR/Vvz14OeMStc6Rf2/DngsWA59n4FlQEfw//t7wPQa6ffdwCbgVeB/Acmw9Rv4Z/LnCIbIH4F/8mR9BP48yLbNwHsm+3m69F9EJCSqbchFRETGoUAXEQkJBbqISEgo0EVEQkKBLiISEgp0EZGQUKCLiITE/wcZZu5dioIIhwAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "epochs = 100\n",
    "n_samples = X_enc.shape[0]\n",
    "# learning rate\n",
    "lr = 50\n",
    "nlls = []\n",
    "# to keep track of min nll loss\n",
    "min_nll = float('inf')\n",
    "\n",
    "for _ in range(epochs):\n",
    "    # equivalent to `torch.nn.Linear` with no bias\n",
    "    logits = X_enc @ W\n",
    "    # shape of logits = (n_samples x vocab_size)\n",
    "    # softmax layer\n",
    "    counts = logits.exp()\n",
    "    probs = counts / counts.sum(dim=-1, keepdims=True)\n",
    "    # loss (average negative log likelihood)\n",
    "    nll = -probs[torch.arange(n_samples), Y].log().mean() + 0.01*(W**2).mean()\n",
    "\n",
    "    # set gradients to zero and perform backpropagation\n",
    "    W.grad = None\n",
    "    nll.backward()\n",
    "\n",
    "    # parameter update, inplace update\n",
    "    W.data += -lr * W.grad\n",
    "\n",
    "    if nll < min_nll:\n",
    "        min_nll = nll.item()\n",
    "\n",
    "    nlls.append(nll.item())\n",
    "\n",
    "print(min_nll, nlls[-1])\n",
    "plt.plot(nlls)\n",
    "\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.4735162258148193 2.4735162258148193\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f79b85b2f70>]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/P9b71AAAACXBIWXMAAAsTAAALEwEAmpwYAAAdVElEQVR4nO3deZCc9X3n8fe3j+k5NSNpRreEJBCnAMnIXMYYAzYCx8GbSrbwxhcxq2XjJDjr2sVJdlPFurb2sOMjhR0twbHxxrHLATYGNmBrHY6wGIkBiwEhhHWAJCSkGZ1z9kx3f/eP55lRazSj6ZF61Orn+byquvo5fk/39wfSpx/9nsvcHRERqX6JShcgIiLloUAXEYkIBbqISEQo0EVEIkKBLiISEalKfXFra6svXry4Ul8vIlKVXn755S53bxtrXcUCffHixbS3t1fq60VEqpKZvTPeOg25iIhEhAJdRCQiFOgiIhGhQBcRiQgFuohIRCjQRUQiQoEuIhIRVRfoW97r5ms/28LB3sFKlyIiclapukDf0dXD/U9vZd/RgUqXIiJyVqm6QG/IBBe39mRzFa5EROTsUnWB3jgc6AMKdBGRYlUX6E212kMXERlL1QV6YyYNKNBFREabMNDNrNbMNpjZq2a2yczuG6NNs5k9XtTmzqkpFxprNeQiIjKWUm6fmwVudPceM0sDz5vZk+7+YlGbLwBvuPvHzawN2GJmP3T3sp9bWJ9OAtCtPXQRkeNMGOju7kBPOJsOXz66GdBkZgY0AgeBKUncRMJozKToVaCLiBynpDF0M0ua2UZgP7DO3dePanI/cBGwB3gNuMfdC2N8zhozazez9s7OzlMuujGT0pCLiMgoJQW6u+fdfQWwALjSzJaPanILsBGYB6wA7jezaWN8zgPuvsrdV7W1jfkEpZI01qZ0UFREZJRJneXi7oeBZ4DVo1bdCTzqga3ADuDCchQ4loZMSmPoIiKjlHKWS5uZtYTTdcDNwJujmu0EbgrbzAYuALaXtdIiTZkUPQNDU/XxIiJVqZSzXOYCD5lZkuAH4Cfu/oSZ3Q3g7muBrwDfN7PXAAPudfeuqSq6MZOiszs7VR8vIlKVSjnLpQNYOcbytUXTe4CPlre08WkMXUTkRFV3pSgEe+jdGnIRETlO1QZ6TzZHcIq8iIhAtQZ6bYqCQ/9QvtKliIicNaoz0HVPdBGRE1RloDfpBl0iIieoykBvqNEeuojIaFUZ6LqFrojIiaoz0DWGLiJygqoMdD2GTkTkRFUZ6A3aQxcROUFVBvrwkEu3xtBFREZUZaBnUgnSSdMeuohIkaoMdDM9hk5EZLSqDHQI77ioIRcRkRFVG+gNNXpqkYhIsaoN9CbtoYuIHKdqA70xk6J3UIEuIjKsegO9Nq09dBGRItUb6JmkxtBFRIpUcaBrDF1EpNiEgW5mtWa2wcxeNbNNZnbfOO1uMLONYZtny1/q8RozafqH8uTyhan+KhGRqpAqoU0WuNHde8wsDTxvZk+6+4vDDcysBfgOsNrdd5rZrKkp95jhW+j2DuZprqvaf2iIiJTNhEnogZ5wNh2+Rj+d+V8Bj7r7znCb/WWtcgxNukGXiMhxStq1NbOkmW0E9gPr3H39qCbnA9PN7Bkze9nMPjPO56wxs3Yza+/s7DytwkfuuKhxdBERoMRAd/e8u68AFgBXmtnyUU1SwBXAx4BbgP9kZueP8TkPuPsqd1/V1tZ2WoWPPLUoO3RanyMiEhWTGnx298PAM8DqUat2A0+5e6+7dwHPAZeXo8DxHHtqUX4qv0ZEpGqUcpZLW3jQEzOrA24G3hzV7KfAB80sZWb1wFXA5jLXepwmPVdUROQ4pZzlMhd4yMySBD8AP3H3J8zsbgB3X+vum83sKaADKAAPuvvrU1Y1xU8t0pCLiAiUEOju3gGsHGP52lHzXwW+Wr7STk5PLRIROV7VnsDdqNMWRUSOU7WBnkwY9TVJPbVIRCRUtYEO4f1cFOgiIkAEAl1j6CIigeoO9FrtoYuIDKvuQM+kNIYuIhKq+kDXkIuISKDqA11DLiIigeoOdI2hi4iMqO5ADx9D5z769uwiIvFT3YFemyJXcLI5PYZORKS6A12X/4uIjIhGoOtMFxGRiAS69tBFRKo70Jtq0wAc7dc90UVEqjrQWxtrAOjqHaxwJSIilVflgZ4BoKs7W+FKREQqr6oDvbkuTSphdPUo0EVEqjrQEwljZmONAl1EhBIC3cxqzWyDmb1qZpvM7L6TtH2/meXN7LfLW+b4WhszdGrIRURk4odEA1ngRnfvMbM08LyZPenuLxY3MrMk8N+Bn01BneNqbczQ1aODoiIiE+6he6AnnE2Hr7FunvKHwCPA/vKVN7Eg0LWHLiJS0hi6mSXNbCNBWK9z9/Wj1s8H/gWwdoLPWWNm7WbW3tnZeYolH6+1qYYDPYO6QZeIxF5Jge7ueXdfASwArjSz5aOafBO4193zE3zOA+6+yt1XtbW1nUq9J2hrzDCYL3C0X1eLiki8lTKGPsLdD5vZM8Bq4PWiVauAH5sZQCtwm5nl3P0fylTnuIbPRe/sydJcn57qrxMROWuVcpZLm5m1hNN1wM3Am8Vt3H2Juy9298XAw8Dvn4kwh6KLizSOLiIxV8oe+lzgofAslgTwE3d/wszuBnD3k46bT7XWpvDyfwW6iMTchIHu7h3AyjGWjxnk7v650y+rdLr8X0QkUNVXigJMr68hmTCdiy4isVf1gZ5MGDMadPm/iEjVBzro4iIREYhMoNfQqSEXEYm5SAR6W2NGB0VFJPYiEeitTRk6e7K6/F9EYi0agd5Yw2CuQLceFi0iMRaRQNe56CIi0Qp0HRgVkRiLWKBrD11E4isaga77uYiIRCPQZ9TXYKYxdBGJt0gEeiqZYEa9Li4SkXiLRKCDLv8XEYlOoDfpBl0iEm/RCXTtoYtIzEUr0LsHdfm/iMRWZAK9rSlD/1Ce3sF8pUsREamIyAS6Lv8XkbiLUKDr4iIRibcJA93Mas1sg5m9amabzOy+Mdr8rpl1hK8XzOzyqSl3fG1NwR76fu2hi0hMpUpokwVudPceM0sDz5vZk+7+YlGbHcCH3P2Qmd0KPABcNQX1juucmQ0AvH2g90x+rYjIWWPCQPfgtJGecDYdvnxUmxeKZl8EFpSrwFI1ZlK0NWXY0alAF5F4KmkM3cySZrYR2A+sc/f1J2n+eeDJcT5njZm1m1l7Z2fnpIudyJLWBnZ0KdBFJJ5KCnR3z7v7CoI97yvNbPlY7czswwSBfu84n/OAu69y91VtbW2nWPL4lirQRSTGJnWWi7sfBp4BVo9eZ2aXAQ8Ct7v7gXIUN1lLWhs40DvIkf6hSny9iEhFlXKWS5uZtYTTdcDNwJuj2iwCHgU+7e5vTUGdJVnSGh4Y1V66iMRQKXvoc4GnzawDeIlgDP0JM7vbzO4O2/w5MBP4jpltNLP2Kar3pJa2BYGuYRcRiaNSznLpAFaOsXxt0fRdwF3lLW3yFs6oJ2GwXYEuIjEUmStFATKpJPOn12kPXURiKVKBDrCktZEdXT0TNxQRiZjIBfrS1gZ2dPbqNroiEjuRC/QlrQ30Dubp1E26RCRmIhnogG4BICKxE91A14FREYmZyAX6vJY6alIJBbqIxE7kAj2ZMM6ZUa9z0UUkdiIX6BAMu+jyfxGJm2gGelsD7xzoI1/QqYsiEh+RDPSlrQ0M5gvsOdxf6VJERM6YSAb6ktZGQPd0EZF4iWSgnxvedfHX+7orXImIyJkTyUCf2ZhhXnMtHbuPVLoUEZEzJpKBDnDpgmZee1eBLiLxEdlAv2xBCzu6evU4OhGJjQgHejMAr2svXURiIrKBfun8INA1ji4icRHZQG+pr+GcmfV07D5c6VJERM6ICQPdzGrNbIOZvWpmm8zsvjHamJn9pZltNbMOM3vf1JQ7OZfOb9YeuojERil76FngRne/HFgBrDazq0e1uRVYFr7WAH9VziJP1WULmnn3cD8H9LALEYmBCQPdA8MP6UyHr9E3Sbkd+EHY9kWgxczmlrfUybtsQQsAHTowKiIxUNIYupklzWwjsB9Y5+7rRzWZD+wqmt8dLquoS+ZNwwxe07CLiMRASYHu7nl3XwEsAK40s+WjmthYm41eYGZrzKzdzNo7OzsnXexkNdWmWdraoAOjIhILkzrLxd0PA88Aq0et2g0sLJpfAOwZY/sH3H2Vu69qa2ubXKWn6PIFLTowKiKxUMpZLm1m1hJO1wE3A2+OavYY8JnwbJergSPuvrfcxZ6KSxc0s787y3tHBipdiojIlEqV0GYu8JCZJQl+AH7i7k+Y2d0A7r4W+EfgNmAr0AfcOUX1TtrwFaMduw8zp3lOhasREZk6Ewa6u3cAK8dYvrZo2oEvlLe08rh4bjPppPHyO4f46CUKdBGJrsheKTqsribJykXTeX5rV6VLERGZUpEPdIDrl7Wyac9RXWAkIpEWi0C/bllwRo320kUkymIR6JfOb6a5Ls3zv1agi0h0xSLQkwnj2nNn8vzWLoLjtyIi0ROLQAe4blkre48MsK2zt9KliIhMidgE+gfPC8fRfz31txwQEamE2AT6opn1LJpRrwOjIhJZsQl0CIZdXtx+kKF8odKliIiUXawC/fplrfRkc2zcdbjSpYiIlF2sAv2ac1tJGDy7RePoIhI9sQr05ro015w7k8c79uj0RRGJnFgFOsDtK+bzzoE+DbuISOTELtBXL59DTSrBTzee8PwNEZGqFrtAn1ab5qYLZ/FExx5yOttFRCIkdoEOwbBLV8+gzkkXkUiJZaB/+MI2ptWmNOwiIpESy0DPpJLcdulcfrbpPfoGc5UuR0SkLGIZ6BAMu/QN5ln3xr5KlyIiUhaxDfSrlsxgfksdP9qws9KliIiUxYSBbmYLzexpM9tsZpvM7J4x2jSb2eNm9mrY5s6pKbd8Egnjc9cu5sXtB3VOuohEQil76DngS+5+EXA18AUzu3hUmy8Ab7j75cANwF+YWU1ZK50Cn7xqEdNqU/zPZ7dVuhQRkdM2YaC7+153fyWc7gY2A/NHNwOazMyARuAgwQ/BWa0xk+LT15zDU5veY3tnT6XLERE5LZMaQzezxcBKYP2oVfcDFwF7gNeAe9z9hKt2zGyNmbWbWXtn59lxg6zPXbuEdDLBX//z9kqXIiJyWkoOdDNrBB4BvujuR0etvgXYCMwDVgD3m9m00Z/h7g+4+yp3X9XW1nbKRZdTW1OG375iAY+8/C77jw5UuhwRkVNWUqCbWZogzH/o7o+O0eRO4FEPbAV2ABeWr8ypteaDS8kVCnz3+R2VLkVE5JSVcpaLAd8FNrv718dpthO4KWw/G7gAqJoxjMWtDfzm5fP43gtvs/NAX6XLERE5JaXsoX8A+DRwo5ltDF+3mdndZnZ32OYrwLVm9hrwC+Bed6+qG6V8+daLSCWMr/yfNypdiojIKUlN1MDdnwdsgjZ7gI+Wq6hKmNNcyx/ceB7/46ktPPtWJx86/+wY4xcRKVVsrxQdy+evW8KS1gbue2wTgzndWldEqosCvUgmleTPP34x27t6+Zv/pwOkIlJdFOijfPiCWXzk4tl8fd1bbN47+uxMEZGzlwJ9DP/1ty6luS7NH/7oV7q9rohUDQX6GFobM3zjX65gW2cP//lxnfUiItVBgT6O65a18m8/dC4/fmkXj7+qJxuJyNlPgX4Sf/yR83nfohbufaSD13YfqXQ5IiInpUA/iXQywdpPXcH0+hru/P5L7Dqoq0hF5OylQJ/ArGm1PPR772coX+Cz39vAod7BSpckIjImBXoJzpvVxIOfXcXuQ/383kMvcXRgqNIliYicQIFeovcvnsFf3rGS13Yf4Xf/er321EXkrKNAn4TVy+fwwGeuYMu+bu544EX2d+v+6SJy9lCgT9KNF87m+597P7sO9fE7a3/J1v3dlS5JRARQoJ+Sa89r5W/vuorebI5PfPsFfrF5X6VLEhFRoJ+q9y2azk//4DoWt9Zz1w/a+fbTWykUvNJliUiMKdBPw/yWOv7+31zLxy+bx1d/toVPfXc9e4/0V7osEYkpBfppqqtJ8q07VvDffutSfrXzMLd84zkee3UP7tpbF5EzS4FeBmbGHVcu4h/v+SBL2xr5ox/9irseateVpSJyRinQy2hJawMP330Nf3rbhfxy+wE+8o1n+fbTWxkYyle6NBGJAQV6maWSCdZcfy7/9999iBvOn8VXf7aFG7/2DH/fvou8DpqKyBSaMNDNbKGZPW1mm81sk5ndM067G8xsY9jm2fKXWl3mtdSx9tNX8Hf/+iramjL8+4c7uPVbz/H4q3sU7CIyJWyig3dmNheY6+6vmFkT8DLwCXd/o6hNC/ACsNrdd5rZLHfff7LPXbVqlbe3t592B6qBu/Pk6+/xFz/fwrbOXhbPrOfuD53LJ1bOpzadrHR5IlJFzOxld1815rrJno1hZj8F7nf3dUXLfh+Y5+7/sdTPiVOgD8sXnJ9veo/vPLON1949woyGGu54/0I+dfU5zGupq3R5IlIFyhboZrYYeA5Y7u5Hi5Z/E0gDlwBNwLfc/QdjbL8GWAOwaNGiK955553SexEh7s4L2w7w/RfeHrnK9IYLZvE7VyzgpotmU5PSoQ0RGVtZAt3MGoFngf/i7o+OWnc/sAq4CagDfgl8zN3fGu/z4riHPpZdB/v4uw07efSV3ew7mmV6fZrbLp3Lb1w2jyuXzCCZsEqXKCJnkZMFeqrED0gDjwA/HB3mod1Al7v3Ar1m9hxwOTBuoEtg4Yx67l19IV/6yPn889YuHn55N4+8spsfrt/JrKYMt1wyh5svns3VS2eQSWm8XUTGV8pBUQMeAg66+xfHaXMRcD9wC1ADbADucPfXx/tc7aGPrzeb45/e3M8THXt47q0u+ofyNGZSXHdeK9ef38b157eyYHp9pcsUkQo43T30DwCfBl4zs43hsj8FFgG4+1p332xmTwEdQAF48GRhLifXkEnx8cvn8fHL5zEwlOeFbV2se2Mfz27p5KlN7wGweGY915w7k6uXzuSqJTOZ01xb4apFpNImfZZLuWgPffLcnW2dPTz7VhcvbO1iw46DdGdzQHCjsFWLp3PFOdO5fEELF85t0hCNSASV9bTFclGgn758wdm05wgvvX2Il985SPvbh9jfnQWgJpngorlNXDyvmeXzp3HJvGbOn91IfU1Jh01E5CylQI8Jd2fvkQFe3XWYjbsO07H7CJv2HOHoQLAXbwaLZzZwwewmls1uZNnsJs5ra2RpW4MucBKpEqd9lotUBzNjXksd81rquPXSuUAQ8rsP9bNpz1G2vNfNln1HeXNvN+s27xu5BYEZzGuuY2lbA0taGzhnZgNLWutZNKOeBdPrFfYiVUKBHnFmxsIZ9SycUc/q5XNGlmdzed7u6uOtfd1s7+xle1cP2zt7+d+vvDsyLj9s9rQMC6fXs2B6HfOn1zG/pZ65LbXMb6ljbnMtTbXpM90tERmDAj2mMqkkF8xp4oI5Tcctd3cO9Q2xo6uXXQf72Bm+dh/qo/2dQzzesfeEm4s1ZlLMaa5lzrRaZk3LBO9NGWZNq6WtKcOspgytjRkaMvrjJjKV9DdMjmNmzGioYUZDDVecM/2E9bl8gc6eLHsO97P7UD/7jg6w98gAew8PsK97gBe39bC/O0tujDtK1tckaW3MMLOxhpkNNcxsyDC9IZge/s7pDTVMr0/TUl/DtNoUwWUQIlIKBbpMSiqZYG5zHXOb67jinLHbFArOob5B9ndn2d+dpbM7S1dPlq7uLJ09WQ72DvLu4QE6dh/hYO/gmOEPkEwYzXVpWurSNNeH70WvaXVpptWmmVaXoqk2TVNt8XtKp21K7CjQpewSCWNmY4aZjRkumnvytu7O0YEcB3sHOdQ3yOG+QQ70DHKkf4jDfUMc6gumj/QP0dUzyLbOXo70D3F0YIiJTtCqSSZorE3RmCl61aZoyKRozCRpqAmmGzJJ6muK3mtS1NUkg/l0MF1fk6QunSShe+vIWUyBLhVlZiN73EtoKHm7QsHpGcxxtH+Io/05ugeG6B7IcXRgiJ5sbmS6N5ujZyBHTzZ4dXZn2dHVS082R182R+/g5B4PmEklRsK9tijoa9PH3mvTiZHpTDhfmwrnU4mRNpnUsfdM2KYmlSCTSpAJl+vmbDIZCnSpSomEBcMttWk4cai/ZIWC0z+Up3cwR182T082F8xnc/QP5ukbzNM3lKcvXD68rH8oPzI/MJQf+bHI5gr0h+sHhvJkc4XT6mcqYSMhXzP8SgZhPzxdvHx4Op0MtkknbWR+uE165HVs3ch8MkEqnB7eJpWwkTappJFOJEinjFQiaKfjHGcPBbrEWiJh4bBLKriTf5kVCs5gvsDAUJ6BocJIyA+EgT+YL5AdKjCQyzOYK4y0GcwXwvlg+XC7bO7YumwueO8bzHG4P5geeeWdoXxhZNupfOxhKmEjQZ9MHgv64WWpomXJhI38YKQSiZFtU2G7ZMLCZcG64flk8fbh/Mj6cZaPvMzCuo7NB9skSJqRSEAqkSCZ4IRliQTB9kWfl7Bj33e2/Zgp0EWmUCJh1CaSFb84K18IAr445HP54MdmqGh6MBdMD+WPtSmezhUKDOWdXLjdULhsePvge4L1ufA7i7cbrmNgqECukCc3sk3QPl/wkfb5gpML54PpAmfb43jNCH8AbCT4E0YY/sGPRMKO/QgMr//klYu464NLy16PAl0kBoIwqfwPy+kqFJy8H/thKBQIflDCH4PhH4Hh6aF8gYIfv+xYuwL5AseWuVMItx/+npHpojb5ovWjt80XoODH6igUPJgfXu9BH1obM1Py30eBLiJVI5EwEhjpJFX/4zQV9PBKEZGIUKCLiESEAl1EJCIU6CIiEaFAFxGJCAW6iEhEKNBFRCJCgS4iEhEVe0i0mXUC75zi5q1AVxnLqRZx7Hcc+wzx7Hcc+wyT7/c57t421oqKBfrpMLP28Z56HWVx7Hcc+wzx7Hcc+wzl7beGXEREIkKBLiISEdUa6A9UuoAKiWO/49hniGe/49hnKGO/q3IMXURETlSte+giIjKKAl1EJCKqLtDNbLWZbTGzrWb25UrXMxXMbKGZPW1mm81sk5ndEy6fYWbrzOzX4ftpPB757GRmSTP7lZk9Ec7Hoc8tZvawmb0Z/j+/Jib9/uPwz/frZvYjM6uNWr/N7G/MbL+ZvV60bNw+mtmfhNm2xcxumez3VVWgm1kS+DZwK3Ax8Ekzu7iyVU2JHPAld78IuBr4QtjPLwO/cPdlwC/C+ai5B9hcNB+HPn8LeMrdLwQuJ+h/pPttZvOBPwJWuftyIAncQfT6/X1g9ahlY/Yx/Dt+B3BJuM13wswrWVUFOnAlsNXdt7v7IPBj4PYK11R27r7X3V8Jp7sJ/oLPJ+jrQ2Gzh4BPVKTAKWJmC4CPAQ8WLY56n6cB1wPfBXD3QXc/TMT7HUoBdWaWAuqBPUSs3+7+HHBw1OLx+ng78GN3z7r7DmArQeaVrNoCfT6wq2h+d7gsssxsMbASWA/Mdve9EIQ+MKuCpU2FbwL/ASgULYt6n5cCncD3wqGmB82sgYj3293fBb4G7AT2Akfc/edEvN+h8fp42vlWbYFuYyyL7HmXZtYIPAJ80d2PVrqeqWRmvwHsd/eXK13LGZYC3gf8lbuvBHqp/mGGCYXjxrcDS4B5QIOZfaqyVVXcaedbtQX6bmBh0fwCgn+mRY6ZpQnC/Ifu/mi4eJ+ZzQ3XzwX2V6q+KfAB4DfN7G2CobQbzexviXafIfgzvdvd14fzDxMEfNT7fTOww9073X0IeBS4luj3G8bv42nnW7UF+kvAMjNbYmY1BAcQHqtwTWVnZkYwprrZ3b9etOox4LPh9GeBn57p2qaKu/+Juy9w98UE/1//yd0/RYT7DODu7wG7zOyCcNFNwBtEvN8EQy1Xm1l9+Of9JoJjRVHvN4zfx8eAO8wsY2ZLgGXAhkl9srtX1Qu4DXgL2Ab8WaXrmaI+XkfwT60OYGP4ug2YSXBU/Nfh+4xK1zpF/b8BeCKcjnyfgRVAe/j/+x+A6THp933Am8DrwP8CMlHrN/AjgmMEQwR74J8/WR+BPwuzbQtw62S/T5f+i4hERLUNuYiIyDgU6CIiEaFAFxGJCAW6iEhEKNBFRCJCgS4iEhEKdBGRiPj/Xyf0lUWJmgoAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "epochs = 100\n",
    "n_samples = X_enc.shape[0]\n",
    "# learning rate\n",
    "lr = 50\n",
    "nlls = []\n",
    "# to keep track of min nll loss\n",
    "min_nll = float('inf')\n",
    "\n",
    "import torch.nn as nn\n",
    "\n",
    "model = nn.Linear(vocab_size, vocab_size, bias=False)\n",
    "torch.nn.init.normal_(model.weight, mean=0.0, std=1.0)\n",
    "\n",
    "optimizer = torch.optim.SGD(model.parameters(), lr=lr)\n",
    "loss = nn.CrossEntropyLoss()\n",
    "\n",
    "for _ in range(epochs):\n",
    "    # shape of logits = (n_samples x vocab_size)\n",
    "    logits = model(X_enc)\n",
    "    \n",
    "    # combination of LogSoftmax and NLLLoss\n",
    "    nll = loss(logits, Y)\n",
    "    # set gradients to zero and perform backpropagation\n",
    "    optimizer.zero_grad()\n",
    "    nll.backward()\n",
    "    # parameter update\n",
    "    optimizer.step()\n",
    "    \n",
    "    if nll < min_nll:\n",
    "        min_nll = nll.item()\n",
    "\n",
    "    nlls.append(nll.item())\n",
    "\n",
    "print(min_nll, nlls[-1])\n",
    "plt.plot(nlls)\n",
    "\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor(3.8470, grad_fn=<MaxBackward1>),\n",
       " tensor(-2.4338, grad_fn=<MinBackward1>))"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "W.max(),W.min()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Visualize the learned weight matrix, W"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5, 1.0, 'W (learned weights)')"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAEICAYAAACZA4KlAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/P9b71AAAACXBIWXMAAAsTAAALEwEAmpwYAAAbyUlEQVR4nO2deZBkVZXGvy+zll6B3jeabpYOlCFkmYKBARGHGAViBIwYCDFEXFsjRCHGGbf5AxzHGDQGDXVcBgRpcWGIQAVHVJDBZocu9qVB6KaB3rfqfanKzDN/5GsoinrnZFVWZSZ9v19ERWa98+695y1fvuXcey7NDEKI/Z9Csx0QQjQGiV2IRJDYhUgEiV2IRJDYhUgEiV2IRJDYmwDJ+0gel32/guTPmu1TBMnTSa4c5TZ+T/LiGtf9M8lPjECbnSSfIzm93rpaHYl9CJD8MsnbBix7IWfZB3LqeB+A7Wb22Ci6+pbEzM4ys0X11kNyPkkj2VZDm3sBXAfgi/W22+pI7EPjbgCnkCwCAMmZANoBHD9g2RHZuoPxaQA3jLajtZzo4jV+AeBikp3NdmQ0kdiHxhJUxX1s9v9pAO4C8PyAZcvMbPXAwiQ7APwdgMV5DZA8ieT9JLeQfILk6f1sHyW5lOR2kstJfqqf7XSSK0l+keRaAD/JHhFuIvnTrMwzJLv6lZlN8maSG0i+RPJz/WxjSV5PsofkswBOcHz+KsnvZd/bSe4k+c1+9ewhOamG7Xvt1pxkkeRVJDdmvl0yyNV6XvZItJ3k7SSnZsv3/dBuIbmD5MkkjyC5mOTWrM7/2VeJma0E0APgpLxt3B+Q2IeAmfUCeAhVQSP7vAfAvQOW5V3VFwCoZCfXmyA5B8DvAPw7gMkA/hnAzSSnZausB/APAA4A8FEA3yZ5fL8qZmbl5gFYmC07B8CNAA4CcCuA/8raKgD4LYAnAMwBcAaAy0i+Nyt3OYDDs7/3AvCepRcDOD37fgKAtQDelf1/MoDnzaynhu3rzycBnIXqj+jxAM4bZJ0PZvthOoCOrD7g9WNxkJlNMLMHAHwNwO0AJgE4GMD3BtS1FMAxzja+5ZHYh85ivH4yvRNVsd8zYFnelfsgANuduj8E4DYzu83MKmZ2B4BuAGcDgJn9zsyWWZXFqJ687+xXvgLgcjPba2a7s2X3ZvWVUX182HdCnwBgmpn9m5n1mtlyANcA2Peu4QIAXzezzWb2KoDvOn4/AGABySnZfrgWwBySE1AV/b794W7fAC4A8B0zW2lmPQCuHGSdn5jZX7JtvQmv310NRh+qP4KzzWyPmd07wL4d1eOz3yKxD527AZya3ZZOM7MXANwP4G+zZUcj/8reA2CiU/c8AOdnt7hbSG4BcCqAWQBA8iySD5LcnNnOBjC1X/kNZrZnQJ1r+33fBWBMdis8D8DsAW19BcCMbN3ZAF7tV/blPKczsXWjKuzTUBX3/QBOwRvF7m7fAAa2/+og6wzctgl5PgL4AgACeDh7nPnYAPtEAFuc8m959BJn6DwA4EBUb5PvAwAz20ZydbZstZm9lFP2BQAkOcfMVg1ifxXADWb2yYGG7OXRzQA+DOAWM+sj+RtUT+B9DGUI46sAXjKzBTn2NQDmAngm+/+QoL7FqL6POA7VdxuLUb39PxGv//jlbl9O+wf3+39uDWX28ab9YGZrUX00AMlTAfyJ5N1m9mK2ytsBXDWENt5y6Mo+RPpdxf4J1dv3fdybLcu7qsPM+gD8Ca8/zw7kZwDeR/K92QuqMdmLt4NRfSbtBLABQInkWQDeU8emPAxgW/ZCb2zW3tEk972IuwnAl0lOytr/bFDfYlR/iJ7N3m38GcAnUP1B2VDD9g3kJgCXkpxD8iAMLTS2AdVHmsP2LSB5fr92elD9QShntjmovkN4cAhtvOWQ2IfHYlRfCvV/7rsnW5Yr9oz/BnDRYIbs2fhcVG+nN6B6JfwXAAUz2w7gc6iKoAfVl1O3DncDsmf496H6nPsSgI0AfozqXQsAfBXVW/eXUH03EIUL7wcwFq9v/7MA9vT7392+Qeq7Jmv3SQCPAbgNQAmZQINt2wXg6wDuyx4XTkL1HcVDJHegut8u7XcH9kEAi7KY+34Llbyi8ZC8F8Bn1bGmdrI7mR+Z2bwRrrcT1YjEaWa2fiTrbjUkdtGSkBwL4N2oXt1noPq+4kEzu6yZfr2VkdhFS0JyHKqPS28DsBvV+PylZratqY69hZHYhUgEvaATIhEaGmfvYKeNwfj8FcaPdctzT6/fAH1zvIJ/l2Nl/0UwYwcA+utEd1psDw4Zg9/vcsm3F4p1lbdKxS8f7aNo+9ui7Q/qD/2Lj3Odp1FMsA107LsrO9BrewZdoS6xkzwTwHcAFAH82MwG69L4GmMwHn/DM/JXeMc73PaKzw/Wiar/CsGJHgnB/BOhsmWrX310IgJAwfchOtGKM/1h19bZ4be/eYtr5kSvExpgm3pce2XXLr/94BhYn/+DXpw8WDf6ftV3tPv179rt2gGg3ONvY3ScrRKovRJcNNr9Y+ht44O7/jfXNuzbeFaHdH4f1cEKRwG4kORRw61PCDG61PPMfiKAF81sedZj6kZUO0wIIVqQesQ+B28cnLAyW/YGSC4k2U2yuw/7dQclIVqaesQ+2EuAwQYgXG1mXWbW1Y79OhGIEC1NPWJfiTeORDoYwJuyswghWoN6xL4E1YQFh7KabukDqGNghhBidBl26M3MSiQvAfBHVENv15nZM26hieNQPuH4XHPbPU+6xeMIqQ87/JCG9QUx5FJ9dgBomznDtdteP/RUetkPP0ZhoeK0qa7dera49vLbgnEoDz/l2+uEE8a59sr6ja7devviRqK+BsXAXqrv3VThoANde3nT5lyb18+hrji7md2G6tBDIUSLo+6yQiSCxC5EIkjsQiSCxC5EIkjsQiSCxC5EIjR0PDt37UXHY8ty7ZVjjvQrePIF1xyOQw7i6Bs/8teufcq1D/v1B0NkAaAUxIHb5gw2X8LrcLs3oUy8D2y3P8SzsmOna8eSp11z8YADXHs5qJ8Ffyz38oveNPziDRzytVdcey0UOv1u3ZUgVh8OgY2O0c7gGARDZPPQlV2IRJDYhUgEiV2IRJDYhUgEiV2IRJDYhUgEiV2IRGhonN0qFVR278m1753q543vCNIMh/HNkh8fnbbETxVdGWZ88w0+nHy0b1+2xrcHMdq2ObNde3ndBtcejsmPcppPmeSXj/oJlPw0zPP+o9svH81wFOWVB1DZk3+O1lJHtA1h+d5gfoRhoiu7EIkgsQuRCBK7EIkgsQuRCBK7EIkgsQuRCBK7EInQ2PHs7e0ozp6Zay+N9/NxB5MRhzHoaJzyjkMnuvZxjwcO1BDDLTzsp9bHkYf79vV+nLy0KpiUJ5q2uoZtcHH6UQAI51+P2i9O9/Pel1b7/RRqod7x6OE+DPZBPXnjvbnhdWUXIhEkdiESQWIXIhEkdiESQWIXIhEkdiESQWIXIhEaGmcvTWzH+tPzx1tPu/V5t3w5mjc7ouD/th3wxDrXXorip1EMG4BVgjjzxp6wDteFYO7wnef4ufEn3rPcte855hC//Xv8vPL1xqDLsya79mI0Hn9vPHd6ees2114YF8wRH+V9D85jjvXzOrjH2Nl9dYmd5AoA2wGUAZTMrKue+oQQo8dIXNnfbWb+NCdCiKajZ3YhEqFesRuA20k+QnLhSDgkhBgd6r2NP8XMVpOcDuAOks+Z2d39V8h+BBYCQMf4IBmhEGLUqOvKbmars8/1AH4N4MRB1rnazLrMrKttzPh6mhNC1MGwxU5yPMmJ+74DeA+AIO4ihGgW9dzGzwDwa1bjpm0AfmFmf3Ab296L6f+3Mte+8Wx/fvYpv33WtZe37XDt4bzYne2uvRDEPyt74hhuYbwfoy0d4ed9LwYxYCv7c8SP/+1jrr0c7KP2O/3AS+FAf3526/CzEkTzxxd2+7n/yxs2uXaYv3+AGuZnD8bsR+PhEfSFqHjj1aPypfx+DMMWu5ktB3DMcMsLIRqLQm9CJILELkQiSOxCJILELkQiSOxCJILELkQiSOxCJEJDk1egVEbF6fQwaWmQHD/oNBMSJI6ojPc7U0QdPqLEEQBQPvow196+Yr1rLwXJF0IfAjstStDhdzyKOpwgmmAhgFv9cyDafivFnWqiJCcs1DeRhvX2+itE2+DtQyf5h67sQiSCxC5EIkjsQiSCxC5EIkjsQiSCxC5EIkjsQiRCQ+Pse+d34qVvHJ5r793lJzZY8NEaYqQOFoV424LfvmACgyg5BgDwgSdce++px7r2wqrVvg9BX4LC2CB5RBADZjDJQ+Egv69EaZ3fjyDCdtSXoIRtfj8BAHGcO5roo4YEGR6FiRNde7lneBOJ6MouRCJI7EIkgsQuRCJI7EIkgsQuRCJI7EIkgsQuRCI0NM7esdYw/5tOjNL8sdBBdDOcxAEVP/5ZWO9PwFCKYszB5AIAUAnGoxfu8+PwEYWOYKKLYBKHysZgkoUj5vvlX3rVLx/0VQgJYuBts2e69krPlrCJShDLj8fMB9sYnEdRnN6dhKKUb9KVXYhEkNiFSASJXYhEkNiFSASJXYhEkNiFSASJXYhEaGicnXt6wWeX59p//+L9bvkzD+ly7ZUgrzvo/7ZtPdGP0U586WW//d4+v/0afChOmezay5s21+VDJRgPH8aQn3netfd8+CTXPumnD9bVfrT9bgwateUcKC7wc/uXX1wxqj7UNT+CE+IPr+wkryO5nuTT/ZZNJnkHyReyz0nD904I0QhquY2/HsCZA5Z9CcCdZrYAwJ3Z/0KIFiYUu5ndDWDgvdO5ABZl3xcBOG9k3RJCjDTDfUE3w8zWAED2OT1vRZILSXaT7O4N+r4LIUaPUX8bb2ZXm1mXmXV1cMxoNyeEyGG4Yl9HchYAZJ/1pQwVQow6wxX7rQAuzr5fDOCWkXFHCDFahHF2kr8EcDqAqSRXArgcwJUAbiL5cQCvADi/ttbaUJgxLdd8+I2fdosfUfJjtDH+OOHi3jrHWteQL7xtzmzXXpnijzfHhg2+veDHqQvjxvnt79rl1x8wbfEq1+4MtwYQx6ALY/xHQYvGy0c53xHH0aPjbJX6no7b5s917aWgv0duvdEKZnZhjumMYbUohGgK6i4rRCJI7EIkgsQuRCJI7EIkgsQuRCJI7EIkQkPHs5cmtmPDaflx5s5DtvsVBDHkKP4Z5ZXv+Mwav/7f+OZaKAXjyV+8MneYAQDgiA8FDQT7YOMF73DtU25Y4lcfjcWe5s/PjpeDvPIBUd79aP51FoKc7QCK06a69vKGja69ljHzHivP8ftizPzO8OLsurILkQgSuxCJILELkQgSuxCJILELkQgSuxCJILELkQgNjbMXd1cw+dn8nNgf/MK9bvk/jvHjj9bb69ore/wYbe8PZrn2NtQXIwYQ5o0f92Qwx3ydTL7ezwkQzd9e3urPYc/eYMR6lJc+Gs/e2enaozh8LVSCbaw3jh6dAwff/Iprj3IC5KEruxCJILELkQgSuxCJILELkQgSuxCJILELkQgSuxCJ0NA4+4T5O3Hyjx/JtS/6wdlu+em7/PnbGcRg22b645THv+rnTI9ykkc52YE4L/v07iBOzHg8tlu8o8O1l7dsde17zzrBtY9bsty1V0rDjRJn5YM4epSzIDpHgLgvQUiUu978OL0F54g7h30987MLIfYPJHYhEkFiFyIRJHYhEkFiFyIRJHYhEkFiFyIRGhpn37lsDB76x7fl2mes6PYraAvcjXKaB/m+sXadbw9i3JXdu/3yANjux7nb7nrUryCK4Qa59YtTp7j2Ss8W175run8MxgRx+jD3fyUYz/5XR7p2rtvkV9/T47cPoDA2mAM+6CtgfUFfgiC3f7kn2Ide+Xri7CSvI7me5NP9ll1BchXJx7M/vzeMEKLp1HIbfz2AMwdZ/m0zOzb7u21k3RJCjDSh2M3sbgCbG+CLEGIUqecF3SUkn8xu8yflrURyIclukt295fiZVggxOgxX7D8EcDiAYwGsAXBV3opmdrWZdZlZV0dxdJMpCiHyGZbYzWydmZXNrALgGgAnjqxbQoiRZlhiJ9k/5/L7ATydt64QojUI4+wkfwngdABTSa4EcDmA00kei2pUbwWAT9XSmPX2obJiZa69sGC+W7783DLf12ju7Sgn+fy5fvHlfj7vWub+Lhx2iGuvrPBz04cx3IDyuvV+/RU/jj9p0QOundGY/uAY2F7fzg3+u+LypvrfJUfjyeslmkM+isMHw+FzCcVuZhcOsvja4TUnhGgW6i4rRCJI7EIkgsQuRCJI7EIkgsQuRCJI7EIkQkPHs7NYRGHyQbl2e3mVW74wJpibO4qPRuPRgxh32H4N49nDOHqQF53BmP4oTs4gr7rt2OHaw7z1wfzrhXFB+3XmLCh0+DHsSm+fXz+AtZee7Npnfvch34fxfl+Dyk7/PC1OC3IObN6Sb+zLPz66sguRCBK7EIkgsQuRCBK7EIkgsQuRCBK7EIkgsQuRCA2Ns6O9DTYrf470nfPnu8XH3rLErz/ISe7Oaw2gcuJRfvn7Hvfbr2Hu9Gg8+uaP+THeydf548kjHxjZg30U4fWjAIDSy34/g4hi4J+V/bHgtWzf7B/5ufsrQW77yvbtYRsebPf7Clhfr2PM72ehK7sQiSCxC5EIErsQiSCxC5EIErsQiSCxC5EIErsQidDQOHulo4Ddc8bn2jnMfNivlQ/ytlspGMsczX0eUUP5Xe/vcu1Tu/35w/0oMgAGv9/t9R1yi/K+R/XXOT97ecdO1942d7ZffuoBfvsA8NwK315Df4p6qGzdNvz265mfXQixfyCxC5EIErsQiSCxC5EIErsQiSCxC5EIErsQidDYvPElQ+em/Lzo606Y4JYfE9Qf5UyPYtB7pvt54cdG8dUoxg2guNuPlJcO8LeyEMWpA2yPn5c+iqNH27jryPx8BQDQGcxxH8WwiwsOde3lZStce2G8n7ceAColP+dALce5HjhrumsvvJLfX4R76sgbT3IuybtILiX5DMlLs+WTSd5B8oXsc1JUlxCiedTyE1UC8HkzezuAkwB8huRRAL4E4E4zWwDgzux/IUSLEordzNaY2aPZ9+0AlgKYA+BcAIuy1RYBOG+UfBRCjABDevggOR/AcQAeAjDDzNYA1R8EAIM+aJBcSLKbZHdfn9+vWQgxetQsdpITANwM4DIzC3rqv46ZXW1mXWbW1d6ePwhGCDG61CR2ku2oCv3nZvarbPE6krMy+ywA60fHRSHESFDL23gCuBbAUjP7Vj/TrQAuzr5fDOCWkXdPCDFS1BJnPwXARQCeIvl4tuwrAK4EcBPJjwN4BcD5o+KhEGJEoNWbsGEIHFicaidNOGfY5aNJ7AsdfnJ9FIIbmWgSiaD9KHlGLbCjw/dh926/fJu/D4pTJ/v1B8khOM7vlFLesMm1w8L0Gy5th87zqw+SZ1SiTj2oIclJQHQMo4lCwvqd8+zB0h+xrbJ50BXUXVaIRJDYhUgEiV2IRJDYhUgEiV2IRJDYhUgEiV2IRGho8goQoJOcYMVnj3aLH/KNh117Za+fmCFi7edOdu0zv/eQaw8TP9TA8uuOcO2HXvhUXT6U1q5z7Yz6GuzY4drbZs9y7eV1fq/qKAFJ+ZWVdZWvKc4fJacI6gjj6EH54mQ/NUR502an7nyTruxCJILELkQiSOxCJILELkQiSOxCJILELkQiSOxCJEKD4+wFYGz+JAgHrAjil0EMORpHHMWQ5/zJH4tdCcqjEO/OKAZ7xNf8vgLlIAYcjqmnP96dwXhwBMcgiqOzza/feoOx5OEEDb5/hQn+RCTVKvw6omMYjYcvdPqTkUQTZdArv7eOSSKEEPsHErsQiSCxC5EIErsQiSCxC5EIErsQiSCxC5EIDY2zW0cbKgdPy7VP/sNf3PJRjBnBWObKXj/ve3HVWr/+iBrGs0exfm7e6toLY/wYbZRXvtAZ5FUPykdE22clP0Zdb5y/OMGfT5AT4zh71FcgojDWz62Pit+fpLLFPwe8/iRevghd2YVIBIldiESQ2IVIBIldiESQ2IVIBIldiESQ2IVIhIbG2fsOKGLVGQfm2ssd+TYAmPv1B/wGgvHkxUl+Pm4U/d++cN7ucKw1APPrqGzb7tuj+dmDOPeed/m5+ccsftq1r/nk8a599s+Wuvbyli2uHUEcvjjFn1+eY/LzJQBAef1Gv33UMOa+HMTJ6+yrENLbm2syJyd9eHaSnEvyLpJLST5D8tJs+RUkV5F8PPs7ezh+CyEaQy1X9hKAz5vZoyQnAniE5B2Z7dtm9p+j554QYqQIxW5mawCsyb5vJ7kUwJzRdkwIMbIM6QUdyfkAjgOwb9KzS0g+SfI6koM+EJNcSLKbZHd51876vBVCDJuaxU5yAoCbAVxmZtsA/BDA4QCORfXKf9Vg5czsajPrMrOu4jh/kIIQYvSoSewk21EV+s/N7FcAYGbrzKxs1dd/1wA4cfTcFELUSy1v4wngWgBLzexb/Zb3n5v3/QD8mI0QoqnU8jb+FAAXAXiK5OPZsq8AuJDksajOCL0CwKeiijo29+GQm/Ln13728vyx7rVgwfzs5SDf996z/Bhy523OvNg1EuW23/2uo3wfft/t2qP5ycc+vMy1V8wvP+P7/hz1mHyQaw7Huwf+204/J0Fl6zbXXhPtQV73UtDXoc3PzR9RnOr3JfC2keX863ctb+PvBTDYiPjborJCiNZB3WWFSASJXYhEkNiFSASJXYhEkNiFSASJXYhEoAVx1RFtjNwA4OV+i6YCiAcYN49W9w9ofR/lX30M1b95ZjZoh5WGiv1NjZPdZtbVNAcCWt0/oPV9lH/1MZL+6TZeiESQ2IVIhGaL/eomtx/R6v4Bre+j/KuPEfOvqc/sQojG0ewruxCiQUjsQiRCU8RO8kySz5N8keSXmuFDBMkVJJ/K0mT7g8gb4891JNeTfLrfsskk7yD5QvYZJMZvio8tkXLcSYneMvtwtNO2N/yZnWQRwF8A/D2AlQCWALjQzJ5tqCMBJFcA6DKzluhwQfI0ADsA/NTMjs6WfRPAZjO7MvvRnGRmX2wxH68AsKPZKcezzEqz+qdEB3AegI+gRfah4+MFGIF92Iwr+4kAXjSz5WbWC+BGAOc2wY+3FGZ2N4CBqXLOBbAo+74I1ROjaeT42BKY2RozezT7vh3AvpToLbMPHR9HhGaIfQ6AV/v9vxKtmYfeANxO8hGSC5vtTA4zsrz++/L7T2+yP3mEKccbyYCU6C25D4eTtj2iGWIfLMVVK8b/TjGz4wGcBeAz2S2qGDo1pRxvFIOkRG85hpu2PaIZYl8JYG6//w8GsLoJfriY2erscz2AX6M1U2Wv25flN/tc32R/3kQrpRwfLCU6Wmwfjmba9maIfQmABSQPJdkB4AMAbm2CH7mQHJ+9IAHJ8QDeg9ZMlX0rgIuz7xcDuKWJvgxKq6Qcz0uJjhbah6Oett3MGv4H4GxU38gvA/CvzfAh8O8wAE9kf8+0go8AfonqLVwfqndHHwcwBcCdAF7IPie3oI83AHgKwJOoCmtWk3w7FdXHxScBPJ79nd1K+9DxcUT2obrLCpEI6kEnRCJI7EIkgsQuRCJI7EIkgsQuRCJI7EIkgsQuRCL8P4FHXZOJAkZFAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.imshow(W.exp().detach().numpy())\n",
    "plt.title(\"W (learned weights)\")\n",
    "# plt.axis('off');"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Generate new words using learned weight matrix, W."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mor.\n",
      "axx.\n",
      "minaymoryles.\n",
      "kondlaisah.\n",
      "anchshizarie.\n",
      "odaren.\n",
      "iadcesh.\n",
      "h.\n",
      "jhinatien.\n",
      "egvonn.\n"
     ]
    }
   ],
   "source": [
    "# for reproduciblity of results\n",
    "g = torch.Generator().manual_seed(2147483647)\n",
    "n = 10 # generate n words\n",
    "for i in range(n):\n",
    "    # start token (idx)\n",
    "    curr_idx = 0\n",
    "    curr_word = ''\n",
    "    while True:\n",
    "        x_enc = F.one_hot(torch.tensor([curr_idx]), num_classes=27).float()\n",
    "        logits = model(x_enc)#x_enc @ W #log counts\n",
    "        counts = logits.exp()\n",
    "        prob = counts / counts.sum(dim=-1, keepdims=True)\n",
    "        # sample index of next character\n",
    "        next_char_ix = torch.multinomial(prob, num_samples=1, replacement=True, generator=g).item()\n",
    "        curr_char = itochar[next_char_ix]\n",
    "        curr_word += curr_char\n",
    "        curr_idx = next_char_ix\n",
    "        if curr_char == '.':\n",
    "            print(curr_word)\n",
    "            break"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Words generated by neural network LM are similar to bigram count-based LM. The next steps to further improve NN-LM would be to use non-linearity to increase model complexity and also experimenting with N-grams (longer context)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "ml_env_3.9",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.15 (main, Nov 24 2022, 08:29:02) \n[Clang 14.0.6 ]"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "02a75899b11e31bd85ccc905c76a128ad2bb11b6b68df04d39be7741d54ebc47"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}