NOTEBOOK.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Homework №2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This homework will be dedicated to **ASR & Co**.\n",
    "\n",
    "In general, you may implement any ASR model that was discussed in the lecture,\n",
    "but we recommend to implement **QuartzNet**.\n",
    "\n",
    "## **Important aspects (model)**\n",
    "1) Pay attention on different length of utterances. P.S. **masking**.\n",
    "    \n",
    "2) A good ASR is a robust ASR, so we ask you to implement and use at least **4 types of augmentations** (P.S. 2 seminar).\n",
    "\n",
    "3) Also, to get better quality, we ask you to implement a **beam search** for better decoding.\n",
    "\n",
    "4) (Bonus) As a bonus you can use **BPE** instead of Char. You can use SentencePiece, HuggingFace or YouTokenToMe.\n",
    "\n",
    "5) (Bonus) As a bonus you can take pretrained **LM** (or train yourself) and fusing LM with ASR.\n",
    "    Way of fusing you may choose yourself.\n",
    "\n",
    "## **Important aspects (code)**\n",
    "1) You already know about pytorch-lighting (I hope :)) but you are not allowed to use it in this homework.\n",
    "\n",
    "2) Try to write code more structurally and cleanly !\n",
    "\n",
    "3) Good logging of experiments save your nerves and time,\n",
    "    so we ask you to use **W&B** and log at least loss, WER, CER and pairs (audio -- recognized text).\n",
    "    **Do not remove** the logs until we have checked your work and given you a grade!\n",
    "\n",
    "4) We also ask you to organize your code in github repo with Docker and setup.py. You can use my template https://github.com/markovka17/dl-start-pack.\n",
    "\n",
    "5) Your work **must be** reproducable, so fix seed, save the weights of model, and etc.\n",
    "\n",
    "6) In the end of your work write inference utils. Anyone should be able to take your weight, load it into the model and run it on some audio track.\n",
    "\n",
    "## Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "CommonVoice Mozilla: https://commonvoice.mozilla.org/en/datasets\n",
    "Он весит 50 гигабайт. В нем значительно больше коротких записей, что должно ускорить сходимость методов на нем.\n",
    "Можно обучиться на нем всем или просто отщипнуть себе кусочек.\n",
    "\n",
    "Все еще настоятельно рекомендую препроцессить датасет и выкидывать все записи длиннее N-секунд (а если вы учите не CTC, то стоит еще дополнительно выкидывать все записи длиннее  K символов), чтобы максимизировать размер батча."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### config"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://github.com/NVIDIA/NeMo/blob/main/examples/asr/conf/quartznet_15x5.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Начинаем решение"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import wandb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 172,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Populating the interactive namespace from numpy and matplotlib\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/kirill/.local/lib/python3.6/site-packages/IPython/core/magics/pylab.py:160: UserWarning: pylab import has clobbered these variables: ['random', 'test', 'f']\n",
      "`%matplotlib` prevents importing * from pylab and numpy\n",
      "  \"\\n`%matplotlib` prevents importing * from pylab and numpy\"\n"
     ]
    }
   ],
   "source": [
    "import string\n",
    "import re\n",
    "import math\n",
    "import random\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torchaudio\n",
    "import librosa\n",
    "import asrtoolkit\n",
    "import torch_optimizer\n",
    "\n",
    "\n",
    "\n",
    "from tqdm import tqdm\n",
    "from torch.utils.data import DataLoader\n",
    "from torch.nn.utils.rnn import pad_sequence\n",
    "from torch.optim.lr_scheduler import StepLR\n",
    "from torch.optim.lr_scheduler import CosineAnnealingLR\n",
    "from torch import distributions\n",
    "\n",
    "from collections import Counter\n",
    "from IPython import display as display_\n",
    "%pylab inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 221,
   "metadata": {},
   "outputs": [],
   "source": [
    "BATCH_SIZE = 80\n",
    "NUM_EPOCHS = 10\n",
    "N_MELS     = 64"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 222,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "device(type='cpu')"
      ]
     },
     "execution_count": 222,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "device"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Датасет"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 223,
   "metadata": {},
   "outputs": [],
   "source": [
    "class TrainDataset(torch.utils.data.Dataset):\n",
    "    \"\"\"Custom competition dataset.\"\"\"\n",
    "\n",
    "    def __init__(self, csv_file, transform=None):\n",
    "        \"\"\"\n",
    "        Args:\n",
    "            csv_file (string): Path to the csv file with annotations.\n",
    "            root_dir (string): Directory with all the images.\n",
    "            transform (callable, optional): Optional transform to be applied on a sample.\n",
    "        \"\"\"\n",
    "        self.answers = pd.read_csv(csv_file, '\\t')\n",
    "        self.transform = transform\n",
    "\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.answers)\n",
    "\n",
    "\n",
    "    def __getitem__(self, idx):\n",
    "        if torch.is_tensor(idx):\n",
    "            idx = idx.tolist()\n",
    "\n",
    "        utt_name = 'cv-corpus-5.1-2020-06-22/en/clips/' + self.answers.loc[idx, 'path']\n",
    "        utt = torchaudio.load(utt_name)[0].squeeze()\n",
    "        \n",
    "        if len(utt.shape) != 1:\n",
    "            print(utt.shape)\n",
    "            print(utt)\n",
    "            utt = utt[1]\n",
    "            \n",
    "        answer = self.answers.loc[idx, 'sentence']\n",
    "\n",
    "        if self.transform:\n",
    "            utt = self.transform(utt)\n",
    "\n",
    "        sample = {'utt': utt, 'answer': answer}\n",
    "        return sample"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 224,
   "metadata": {},
   "outputs": [],
   "source": [
    "class TestDataset(torch.utils.data.Dataset):\n",
    "    \"\"\"Custom competition dataset.\"\"\"\n",
    "\n",
    "    def __init__(self, csv_file, transform=None):\n",
    "        \"\"\"\n",
    "        Args:\n",
    "            csv_file (string): Path to the csv file with annotations.\n",
    "            root_dir (string): Directory with all the images.\n",
    "            transform (callable, optional): Optional transform to be applied on a sample.\n",
    "        \"\"\"\n",
    "        self.names = pd.read_csv(csv_file, '\\t')\n",
    "        self.transform = transform\n",
    "\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.names)\n",
    "\n",
    "\n",
    "    def __getitem__(self, idx):\n",
    "        if torch.is_tensor(idx):\n",
    "            idx = idx.tolist()\n",
    "\n",
    "        utt_name = 'cv-corpus-5.1-2020-06-22/en/clips/' + self.names.loc[idx, 'path']\n",
    "        utt = torchaudio.load(utt_name)[0].squeeze()\n",
    "  \n",
    "\n",
    "        if self.transform:\n",
    "            utt = self.transform(utt)\n",
    "\n",
    "        sample = {'utt': utt}\n",
    "        return sample"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Аугментации"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 225,
   "metadata": {},
   "outputs": [],
   "source": [
    "def transform_tr(wav):\n",
    "    aug_num = torch.randint(low=0, high=3, size=(1,)).item()\n",
    "    augs = [\n",
    "        lambda x: x,\n",
    "        lambda x: (x + distributions.Normal(0, 0.01).sample(x.size())).clamp_(-1, 1),\n",
    "        lambda x: torchaudio.transforms.Vol(.1)(x)\n",
    "    ]\n",
    "    \n",
    "    return augs[aug_num](wav)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Визуализация"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 226,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "def viz(wav):\n",
    "    figsize(20, 5)\n",
    "    plot(wav)\n",
    "    plt.show()\n",
    "\n",
    "    display_.display(display_.Audio(wav, rate=48000, normalize=False))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Класс для кодировки текста (0 - blank_token)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 227,
   "metadata": {},
   "outputs": [],
   "source": [
    "class TextTransform:\n",
    "    def __init__(self):\n",
    "        self.char_dict = {}\n",
    "        self.index_dict = {}\n",
    "        \n",
    "        self.char_dict['\\''] = 0\n",
    "        self.index_dict[0] = '\\''\n",
    "        self.char_dict[' '] = 1\n",
    "        self.index_dict[1] = ' '\n",
    "        for i, let in enumerate(string.ascii_lowercase):\n",
    "            self.index_dict[i + 2] = let\n",
    "            self.char_dict[let] = i + 2\n",
    "            \n",
    "    def text_to_int(self, text):\n",
    "        labels = []\n",
    "        for let in text:\n",
    "            labels.append(self.char_dict[let])\n",
    "        return labels\n",
    "    \n",
    "    def int_to_text(self, labels):\n",
    "        text = []\n",
    "        for num in labels:\n",
    "            text.append(self.index_dict[num])\n",
    "        return text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Collate_fn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 243,
   "metadata": {},
   "outputs": [],
   "source": [
    "#win_len=1024, hop_len=256\n",
    "# counting len of MelSpec before doing it (cause of padding)\n",
    "def mel_len(x):\n",
    "    return int(x // 256) + 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 229,
   "metadata": {},
   "outputs": [],
   "source": [
    "def preprocess_data(data):\n",
    "    text_transform = TextTransform()\n",
    "    wavs = []\n",
    "    input_lens = []\n",
    "    labels = []\n",
    "    label_lens = []\n",
    "    \n",
    "    for el in data:\n",
    "        wavs.append(el['utt'])\n",
    "        input_lens.append(math.ceil(mel_len(el['utt'].shape[0]) / 2))    # cause of stride 2\n",
    "        label = torch.Tensor(text_transform.text_to_int(el['answer']))\n",
    "        labels.append(label)\n",
    "        label_lens.append(len(label))\n",
    "\n",
    "    wavs = pad_sequence(wavs, batch_first=True)\n",
    "    labels = pad_sequence(labels, batch_first=True)\n",
    "    \n",
    "    return wavs, input_lens, labels, label_lens    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 230,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "all train+val samples: 435947\n"
     ]
    }
   ],
   "source": [
    "# Loading data and loaders\n",
    "my_dataset = TrainDataset(csv_file='train_preprocessed.tsv', transform=transform_tr)\n",
    "print('all train+val samples:', len(my_dataset))\n",
    "test_dataset = TestDataset(csv_file='cv-corpus-5.1-2020-06-22/en/test.tsv', transform=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "100k files ~ 10GB"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "95_percentile_len = 412416.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Отсортированные по длинне звука индексы объектов. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 236,
   "metadata": {},
   "outputs": [],
   "source": [
    "# sorted indexes\n",
    "with open('sorted.npy', 'rb') as f:\n",
    "    s = np.load(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 237,
   "metadata": {},
   "outputs": [],
   "source": [
    "to_save = s[:100000][:, 0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 239,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_dataset = torch.utils.data.Subset(my_dataset, to_save)\n",
    "train_set, val_set = torch.utils.data.random_split(my_dataset, [85000, 15000])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Даталоадеры"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 241,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_loader = DataLoader(train_set, batch_size=BATCH_SIZE,\n",
    "                          shuffle=True, collate_fn=preprocess_data, drop_last=True,\n",
    "                          num_workers=0, pin_memory=True)\n",
    "\n",
    "val_loader = DataLoader(val_set, batch_size=BATCH_SIZE,\n",
    "                        shuffle=True, collate_fn=preprocess_data, drop_last=True,\n",
    "                        num_workers=0, pin_memory=True)\n",
    "\n",
    "test_dataloader = DataLoader(test_dataset, batch_size=BATCH_SIZE, shuffle=False, drop_last=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Модель"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Фикс сидов и смотрим кол-во параметров"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 245,
   "metadata": {},
   "outputs": [],
   "source": [
    "def set_seed(seed):\n",
    "    torch.backends.cudnn.deterministic = True\n",
    "    torch.manual_seed(seed)\n",
    "    torch.cuda.manual_seed(seed)\n",
    "    random.seed(seed)\n",
    "    np.random.seed(seed)\n",
    "set_seed(21)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 281,
   "metadata": {},
   "outputs": [],
   "source": [
    "def count_parameters(model):\n",
    "    model_parameters = filter(lambda p: p.requires_grad, model.parameters())\n",
    "    return sum([np.prod(p.size()) for p in model_parameters])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Вспомогательные блоки"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 247,
   "metadata": {},
   "outputs": [],
   "source": [
    "def conv_bn_act(in_size, out_size, kernel_size, stride=1, dilation=1):\n",
    "    return nn.Sequential(\n",
    "        nn.Conv1d(in_size, out_size, kernel_size, stride, dilation=dilation),\n",
    "        nn.BatchNorm1d(out_size),\n",
    "        nn.ReLU()\n",
    "    )\n",
    "\n",
    "\n",
    "def sepconv_bn(in_size, out_size, kernel_size, stride=1, dilation=1, padding=None):\n",
    "    if padding is None:\n",
    "        padding = (kernel_size-1)//2\n",
    "    return nn.Sequential(\n",
    "        torch.nn.Conv1d(in_size, in_size, kernel_size, \n",
    "                        stride=stride, dilation=dilation, groups=in_size,\n",
    "                        padding=padding),\n",
    "        torch.nn.Conv1d(in_size, out_size, kernel_size=1),\n",
    "        nn.BatchNorm1d(out_size)\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 248,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Основной блок B_i\n",
    "class QnetBlock(nn.Module):\n",
    "    def __init__(self, in_size, out_size, kernel_size, stride=1,\n",
    "                R=5):\n",
    "        super().__init__()\n",
    "        \n",
    "        self.layers = nn.ModuleList(sepconv_bn(in_size, out_size, kernel_size, stride))\n",
    "        for i in range(R - 1):\n",
    "            self.layers.append(nn.ReLU())\n",
    "            self.layers.append(sepconv_bn(out_size, out_size, kernel_size, stride))\n",
    "        self.layers = nn.Sequential(*self.layers)\n",
    "        \n",
    "        self.residual = nn.ModuleList()\n",
    "        self.residual.append(torch.nn.Conv1d(in_size, out_size, kernel_size=1))\n",
    "        self.residual.append(torch.nn.BatchNorm1d(out_size))\n",
    "        self.residual = nn.Sequential(*self.residual)\n",
    "    \n",
    "    def forward(self, x):\n",
    "        return F.relu(self.residual(x) + self.layers(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 249,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "class QuartzNet(nn.Module):\n",
    "    def __init__(self, n_mels, num_classes):\n",
    "        super().__init__()\n",
    "                    \n",
    "        self.c1 = sepconv_bn(n_mels, 256, kernel_size=33, stride=2)\n",
    "                  \n",
    "        self.blocks = nn.Sequential(\n",
    "                #         in   out  k   s  R\n",
    "                QnetBlock(256, 256, 33, 1, R=5),\n",
    "                QnetBlock(256, 256, 39, 1, R=5),\n",
    "                QnetBlock(256, 512, 51, 1, R=5),\n",
    "                QnetBlock(512, 512, 63, 1, R=5),\n",
    "                QnetBlock(512, 512, 75, 1, R=5)\n",
    "        )\n",
    "        \n",
    "        self.c2 = sepconv_bn(512, 512, kernel_size=87, dilation=2, padding=86)\n",
    "        \n",
    "        self.c3 = conv_bn_act(512, 1024, kernel_size=1)\n",
    "        \n",
    "        self.c4 = conv_bn_act(1024, num_classes, kernel_size=1)\n",
    "        \n",
    "        self.init_weights()\n",
    "\n",
    "\n",
    "    def init_weights(self):\n",
    "        pass\n",
    "\n",
    "\n",
    "    def forward(self, x):\n",
    "        c1 = F.relu(self.c1(x))\n",
    "        blocks = self.blocks(c1)\n",
    "        c2 = F.relu(self.c2(blocks))\n",
    "        c3 = self.c3(c2)\n",
    "        return self.c4(c3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### обратим внимание, что c1 и c2 тоже separable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Обучение "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 242,
   "metadata": {},
   "outputs": [],
   "source": [
    "melspec = torchaudio.transforms.MelSpectrogram(\n",
    "    sample_rate=16000,           ### 22050, 48000\n",
    "    n_fft=1024,\n",
    "    hop_length=256,\n",
    "    n_mels=N_MELS                ### 64,    80\n",
    ").to(device)\n",
    "\n",
    "\n",
    "# with augmentations\n",
    "melspec_transforms = nn.Sequential(\n",
    "    torchaudio.transforms.MelSpectrogram(sample_rate=16000, n_fft=1024, hop_length=256,  n_mels=N_MELS),\n",
    "    torchaudio.transforms.FrequencyMasking(freq_mask_param=15),\n",
    "    torchaudio.transforms.TimeMasking(time_mask_param=35),\n",
    ").to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 251,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_epoch(model, optimizer, dataloader, CTCLoss, device, melspec_transforms):\n",
    "    model.train()\n",
    "    \n",
    "    losses = []\n",
    "    \n",
    "    for i, (wavs, wavs_len, answ, answ_len) in tqdm(enumerate(dataloader)):\n",
    "        wavs, answ = wavs.to(device), answ.to(device)\n",
    "        \n",
    "        trans_wavs = torch.log(melspec_transforms(wavs) + 1e-9)\n",
    "        \n",
    "        optimizer.zero_grad()\n",
    "            \n",
    "        output = model(trans_wavs)\n",
    "        output = F.log_softmax(output, dim=1)\n",
    "        output = output.transpose(0, 1).transpose(0, 2)        \n",
    "        \n",
    "        loss = CTCLoss(output, answ, wavs_len, answ_len)\n",
    "        loss.backward()        \n",
    "        \n",
    "        torch.nn.utils.clip_grad_norm_(model.parameters(), 15)\n",
    "        optimizer.step()\n",
    "        losses.append(loss.item())\n",
    "        if i % 100 == 0:\n",
    "            wandb.log({'mean_train_loss':loss})\n",
    "            preds, targets = decoder_func(output, answ, answ_len, del_repeated=False)\n",
    "            wandb.log({\"CER_train\": cer(targets[0], preds[0])})\n",
    "            wandb.log({\"WER_train\": wer(targets[0], preds[0])})\n",
    "        \n",
    "    return np.mean(losses)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 252,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train(model, opt, train_dl, scheduler, CTCLoss, device, n_epochs, val_dl=None):\n",
    "    for epoch in range(n_epochs):\n",
    "        print(\"Epoch {} of {}\".format(epoch, n_epochs), 'LR', scheduler.get_last_lr())\n",
    "        \n",
    "        mean_loss = train_epoch(model, opt, train_dl, CTCLoss, device)\n",
    "        print('MEAN EPOCH LOSS IS', mean_loss)\n",
    "        \n",
    "        scheduler.step()\n",
    "        \n",
    "        if (val_dl != None):\n",
    "            test(model, opt, val_dl, CTCLoss, device)\n",
    "            \n",
    "        torch.save({\n",
    "            'epoch': epoch,\n",
    "            'model_state_dict': model.state_dict(),\n",
    "            'optimizer_state_dict': opt.state_dict(),\n",
    "            'scheduler_state_dict': scheduler.state_dict()\n",
    "        }, 'epoch_5_and_'+str(epoch))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 253,
   "metadata": {},
   "outputs": [],
   "source": [
    "def decoder_func(output, answ, answ_lens, blank_label=0, del_repeated=True):    \n",
    "    decoded_preds, decoded_targs = [], []\n",
    "    \n",
    "    text_transform = TextTransform()\n",
    "\n",
    "    batch_freqs = torch.argmax(output, dim=2).transpose(0, 1)\n",
    "        \n",
    "    for i, freqs in enumerate(batch_freqs):\n",
    "        preds = []\n",
    "        \n",
    "        decoded_targs.append(\n",
    "            text_transform.int_to_text(answ[i][:answ_lens[i]].tolist())\n",
    "        )\n",
    "        \n",
    "        for j, num in enumerate(freqs):\n",
    "            if num != blank_label:\n",
    "                if del_repeated and j != 0 and num == freqs[j-1]:\n",
    "                    continue\n",
    "                preds.append(num.item())\n",
    "        decoded_preds.append(text_transform.int_to_text(preds))\n",
    "\n",
    "    return decoded_preds, decoded_targs    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 254,
   "metadata": {},
   "outputs": [],
   "source": [
    "def cer(target, pred):\n",
    "    cer_res = asrtoolkit.cer(''.join(target), ''.join(pred))\n",
    "    return cer_res\n",
    "\n",
    "def wer(target, pred):\n",
    "    wer_res = asrtoolkit.wer(''.join(target), ''.join(pred))\n",
    "    return wer_res"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 282,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test(model, optimizer, dataloader, CTCLoss, device, melspec, bs_width=None):\n",
    "    model.eval()\n",
    "    \n",
    "    cers, wers, cers_bs, wers_bs = [], [], [], []\n",
    "    losses = []\n",
    "    \n",
    "    with torch.no_grad():\n",
    "        for i, (wavs, wavs_len, answ, answ_len) in enumerate(dataloader):\n",
    "            wavs, answ = wavs.to(device), answ.to(device)\n",
    "\n",
    "            trans_wavs = torch.log(melspec(wavs) + 1e-9)\n",
    "\n",
    "            output = model(trans_wavs)\n",
    "            if bs_width != None:\n",
    "                output_bs = F.softmax(output, dim=1).transpose(0, 1).transpose(0, 2)\n",
    "                preds_bs, targets_bs = beam_search_decoding(output_bs, answ, answ_len, width=bs_width)\n",
    "                \n",
    "            output = F.log_softmax(output, dim=1)\n",
    "            output = output.transpose(0, 1).transpose(0, 2)\n",
    "            loss = CTCLoss(output, answ, wavs_len, answ_len)\n",
    "            losses.append(loss.item())\n",
    "            \n",
    "            # argmax\n",
    "            preds, targets = decoder_func(output, answ, answ_len, del_repeated=True)\n",
    "            # beam search\n",
    "            \n",
    "            for i in range(len(preds)):\n",
    "                if i == 0:\n",
    "                    print('target: ', ''.join(targets[i]))\n",
    "                    print('prediction: ', ''.join(preds[i]))\n",
    "                    print('beam_search_preds: ', ''.join(preds_bs[i]))\n",
    "                    \n",
    "                cers.append(cer(targets[i], preds[i]))\n",
    "                wers.append(wer(targets[i], preds[i]))\n",
    "                cers_bs.append(cer(targets_bs[i], preds_bs[i]))\n",
    "                wers_bs.append(wer(targets_bs[i], preds_bs[i]))\n",
    "                \n",
    "        avg_cer = np.mean(cers)\n",
    "        avg_wer = np.mean(wers)\n",
    "        if bs_width != None:\n",
    "            avg_cer_bs = np.mean(cers_bs)\n",
    "            avg_wer_bs = np.mean(wers_bs)\n",
    "\n",
    "        \n",
    "        print('CER', avg_cer)\n",
    "        print('CER BS', avg_cer_bs)\n",
    "        \n",
    "        print('WER', avg_wer)\n",
    "        print('WER BS', avg_wer_bs)\n",
    "        #wandb.log({\"CER_val\": avg_cer})    \n",
    "        #wandb.log({\"WER_val\": avg_wer})\n",
    "        avg_loss= np.mean(losses)\n",
    "        print('average test loss is', avg_loss)\n",
    "        #wandb.log({'mean_VAL_loss':avg_loss})\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 256,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'wandb' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-256-0393b24b5bbf>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mwandb\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlogin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0mwandb\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0mtrain_table\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mwandb\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTable\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcolumns\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"Predicted Text\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"True Text\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'wandb' is not defined"
     ]
    }
   ],
   "source": [
    "wandb.login()\n",
    "wandb.init()\n",
    "train_table = wandb.Table(columns=[\"Predicted Text\", \"True Text\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 257,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = QuartzNet(n_mels=64, num_classes=28)\n",
    "print(count_parameters(model))\n",
    "model.to(device)\n",
    "wandb.watch(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 259,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<All keys matched successfully>"
      ]
     },
     "execution_count": 259,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#model = QuartzNet(n_mels=64, num_classes=28)\n",
    "#num_of_params = count_parameters(model)\n",
    "#model.to(device)\n",
    "#wandb.watch(model)\n",
    "#opt = torch.optim.RMSprop(model.parameters(), weight_decay=0.0001)\n",
    "#scheduler = StepLR(opt, step_size=1, gamma=0.9) \n",
    "#checkpoint = torch.load('epoch_5', map_location=torch.device('cpu'))\n",
    "#model.load_state_dict(checkpoint['model_state_dict'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 260,
   "metadata": {},
   "outputs": [],
   "source": [
    "opt = torch_optimizer.NovoGrad(\n",
    "                        model.parameters(),\n",
    "                        lr=0.01,\n",
    "                        betas=(0.8, 0.5),\n",
    "                        weight_decay=0.001,\n",
    ")\n",
    "scheduler  = CosineAnnealingLR(opt, T_max=50, eta_min=0, last_epoch=-1) # ###### TMAX = MAX NUM OF EPOCHS\n",
    "\n",
    "#opt = torch.optim.RMSprop(model.parameters(), weight_decay=0.0001)\n",
    "#scheduler = StepLR(opt, step_size=1, gamma=0.9) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 261,
   "metadata": {},
   "outputs": [],
   "source": [
    "CTCLoss = nn.CTCLoss(blank=0).to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "0it [00:00, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0 of 150 LR [0.01]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "173it [10:03,  3.52s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 180864])\n",
      "tensor([[ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -4.0159e-05,\n",
      "         -3.1214e-05,  3.6880e-06],\n",
      "        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -4.0159e-05,\n",
      "         -3.1214e-05,  3.6880e-06]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "231it [13:26,  3.48s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 200448])\n",
      "tensor([[ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -8.9087e-05,\n",
      "         -5.9199e-05, -5.9370e-05],\n",
      "        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -8.9087e-05,\n",
      "         -5.9199e-05, -5.9370e-05]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "377it [21:57,  3.47s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 205056])\n",
      "tensor([[ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -2.7344e-06,\n",
      "         -7.1339e-06, -7.7367e-05],\n",
      "        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -2.7344e-06,\n",
      "         -7.1339e-06, -7.7367e-05]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "523it [30:29,  3.47s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 275328])\n",
      "tensor([[ 0.0000,  0.0000,  0.0000,  ..., -0.0015, -0.0014, -0.0014],\n",
      "        [ 0.0000,  0.0000,  0.0000,  ..., -0.0015, -0.0014, -0.0014]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "526it [30:41,  3.63s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 200448])\n",
      "tensor([[ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -7.8108e-05,\n",
      "         -1.9141e-05, -7.8119e-06],\n",
      "        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -7.8108e-05,\n",
      "         -1.9141e-05, -7.8119e-06]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "824it [48:07,  3.49s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 270720])\n",
      "tensor([[ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -7.7933e-06,\n",
      "         -1.6287e-05, -5.2683e-05],\n",
      "        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -7.7933e-06,\n",
      "         -1.6287e-05, -5.2683e-05]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "902it [52:43,  3.91s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 131328])\n",
      "tensor([[0.0000, 0.0000, 0.0000,  ..., 0.0004, 0.0005, 0.0005],\n",
      "        [0.0000, 0.0000, 0.0000,  ..., 0.0004, 0.0005, 0.0005]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "1062it [1:02:03,  3.51s/it]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MEAN EPOCH LOSS IS 3.3313134944596086\n",
      "target w l hough in command\n",
      "prediction he si i   \n",
      "target the conservatives soon found themselves leading in the polls\n",
      "prediction thi sis  i \n",
      "target it is centered on the new haven and its suburbs\n",
      "prediction h  \n",
      "target gairs actions helped to precipitate a double dissolution\n",
      "prediction the sisi a  \n",
      "target he is buried in saint albans cathedral\n",
      "prediction h  \n",
      "target the national debt had skyrocketed\n",
      "prediction h  \n",
      "target royal scottish orchestra\n",
      "prediction he i i  \n",
      "target he attended ohio university\n",
      "prediction the sis i  \n",
      "target the show is also available as a podcast available for purchase\n",
      "prediction h \n",
      "target she is married\n",
      "prediction he e   \n",
      "target young team of basketball players winning the game\n",
      "prediction he i i  \n",
      "target two children play on the road on a snowy day\n",
      "prediction he i i \n",
      "target you know st albans sir\n",
      "prediction h  \n",
      "target the photographs show the cup in an uncleaned state\n",
      "prediction si i \n",
      "target rowses design was in streamline moderne style\n",
      "prediction h\n",
      "target meanwhile robin is part of a military column in afghanistan\n",
      "prediction h \n",
      "target something like this\n",
      "prediction he i i   \n",
      "target it is therefore not suitable for vegetarians\n",
      "prediction hs  \n",
      "target he was fired at the end of the season\n",
      "prediction hes  \n",
      "target they settled off the coast in a remote village\n",
      "prediction hs  \n",
      "target the fist bump involves the touching of two closed fists\n",
      "prediction h\n",
      "target louisville the county seat is to the southwest\n",
      "prediction she i  \n",
      "target further alerts have been issued by news agencies\n",
      "prediction she\n",
      "target she is a supporter of spanish club real madrid\n",
      "prediction he  \n",
      "target courts have been reluctant to agree to this\n",
      "prediction h\n",
      "target the fairness of trickledown economics is debatable\n",
      "prediction h  \n",
      "target he also served as chairman of the detroit edison company\n",
      "prediction the sis  i \n",
      "target the mineral andradite is named after him\n",
      "prediction h  \n",
      "target to fail doing the ordinary is a disaster\n",
      "prediction h \n",
      "target lincoln alexander above the western end of the main building\n",
      "prediction h  \n",
      "target they are endocentric environments\n",
      "prediction he si i \n",
      "target how can the data have become corrupted\n",
      "prediction o \n",
      "target only younger kids camps are held at the site\n",
      "prediction i\n",
      "target it is also mentioned in the jack kerouac book on the road\n",
      "prediction h \n",
      "target it may sometimes form stalactites\n",
      "prediction the sis i  \n",
      "target his birthhouse can be visited today\n",
      "prediction his \n",
      "target she later returned to the show as sister mary joel\n",
      "prediction he i i \n",
      "target kirk introduced legislation entitled no state bailouts s res\n",
      "prediction the cis  i \n",
      "target their services retained the name packet\n",
      "prediction the sis  i \n",
      "target this park also has a outdoor stage for concerts and other events\n",
      "prediction h \n",
      "target they are both married\n",
      "prediction o  \n",
      "target human beings show empathy most to those in need\n",
      "prediction sh \n",
      "target the acorn matures at the end of its second season\n",
      "prediction h \n",
      "target dont argue with me here\n",
      "prediction h \n",
      "target they relaunched the brand under its own company\n",
      "prediction he i i \n",
      "target they are often bilateral\n",
      "prediction he is a   \n",
      "target it is the most common variant of the flute\n",
      "prediction h \n",
      "target can you help me find the author author trailer\n",
      "prediction h  \n",
      "target after this long went on a musical hiatus broken by occasional gigs\n",
      "prediction o\n",
      "target he was also master of the masonic lodge reform\n",
      "prediction the is i a \n",
      "target man watching child in hallway\n",
      "prediction h \n",
      "target students are usually very passionate about their respective halls\n",
      "prediction si \n",
      "target b s haldane and sewall wright\n",
      "prediction he i i  \n",
      "target he is buried on the indian land not far from daisy\n",
      "prediction h \n",
      "target the comic was a modest success\n",
      "prediction h  \n",
      "target however the municipal website uses the taipei system\n",
      "prediction o\n",
      "target the teacher is reading a story to the class\n",
      "prediction he \n",
      "target read them and wonder\n",
      "prediction a  \n",
      "target doctor thomlinson gave a dwelling house with outbuildings\n",
      "prediction h \n",
      "target the streets of a chinatown look alike are lit with bright colors\n",
      "prediction hs \n",
      "target this one likely began as a temple\n",
      "prediction hs  \n",
      "target they had already entered the court\n",
      "prediction h  \n",
      "target what he did was economically justified\n",
      "prediction o\n",
      "target the province is divided in four cantons\n",
      "prediction h\n",
      "target the parallel course did not take hold however until the next century\n",
      "prediction h \n",
      "target he is best known as the drummer of system of a down\n",
      "prediction the si  i \n",
      "target at my age i need to be playing regularly\n",
      "prediction the i a  \n",
      "target no she had never seen him\n",
      "prediction o  \n",
      "target the name was later shortened to trempealeau\n",
      "prediction the is  a  \n",
      "target we should meet again in about a fortnight\n",
      "prediction the si i \n",
      "target allen in the firm allen and norris\n",
      "prediction o \n",
      "target the town common is above sea level\n",
      "prediction the sis i \n",
      "target a fireman going under the red rope\n",
      "prediction he si i \n",
      "target morris and ralph vaughan williams\n",
      "prediction o \n",
      "target she also participated in volleyball\n",
      "prediction the si  i \n",
      "target he was born in barcelona and died in bastia\n",
      "prediction the si i  \n",
      "target occasionally a bald eagle can be sighted\n",
      "prediction she \n",
      "target they both have british accents\n",
      "prediction h \n",
      "target seay also engaged in planting\n",
      "prediction see \n",
      "target anthony looked at him\n",
      "prediction he i i \n",
      "target i can send contextual information to it from other apps\n",
      "prediction the si  i \n",
      "target dino went on to form duva boxing\n",
      "prediction he i i \n",
      "target these are known as asterisms\n",
      "prediction he si i  \n",
      "target the gel strength is quantified using the bloom test\n",
      "prediction h \n",
      "target another factor is the cost of land\n",
      "prediction i\n",
      "target the bulges added to her beam and to her draught\n",
      "prediction he \n",
      "target if only we are in time he said catching his breath\n",
      "prediction the i a \n",
      "target its cover was the british passport control office\n",
      "prediction h \n",
      "target they concluded that the ship was lying in one piece\n",
      "prediction h  \n",
      "target at the same time they are symbols of action and progress\n",
      "prediction hs \n",
      "target there are anadromous and landlocked forms\n",
      "prediction h \n",
      "target the little man with a low growl stood still\n",
      "prediction h \n",
      "target it occurs as wellformed finesized crystals\n",
      "prediction the sis i \n",
      "target today it is a national historic site\n",
      "prediction the sis i  \n",
      "target i insist upon that\n",
      "prediction i \n",
      "target without treatment death may occur\n",
      "prediction he i i  \n",
      "target group of males surrounding a female sitting in a chair\n",
      "prediction os\n",
      "target she was the only daughter to survive\n",
      "prediction sh \n",
      "target he was captain of the football team\n",
      "prediction she \n",
      "target strikes are much rarer on humans since they are more aware\n",
      "prediction the sis a \n",
      "target an old lady consulting a paperback book\n",
      "prediction the i i  \n",
      "target by then half their food had been eaten\n",
      "prediction h  \n",
      "target he held this post for the duration of the war\n",
      "prediction he  \n",
      "target let me breathe\n",
      "prediction he i a  \n",
      "target i said no originally\n",
      "prediction the sis i  \n",
      "target the property is now a quality inn\n",
      "prediction he  \n",
      "target see sea dumping in karantina\n",
      "prediction he i i  \n",
      "target suddenly they could not seem to do anything right\n",
      "prediction the i a \n",
      "target how does this occur\n",
      "prediction h  \n",
      "target hes so very prompt\n",
      "prediction h  \n",
      "target race dogs run around a curve\n",
      "prediction he si i \n",
      "target in this verse herod again questions the magi\n",
      "prediction the sis i \n",
      "target his son christopher o\n",
      "prediction sh  \n",
      "target many young people spent one or more years in such programmes\n",
      "prediction h \n",
      "target the council normally meets six times each year\n",
      "prediction he si  i \n",
      "target fightings ceased at nightfall\n",
      "prediction i \n",
      "target about four miles of this road was built in saugus\n",
      "prediction she\n",
      "target and at last he yielded and came to her\n",
      "prediction he i i \n",
      "target the ballena is spanish for whale\n",
      "prediction the si i \n",
      "target its capital was fuzhou\n",
      "prediction the sis i  \n",
      "target it does not need a recoil pit\n",
      "prediction he \n",
      "target the play is currently in development\n",
      "prediction the si i  \n",
      "target tomorrow is another day\n",
      "prediction h \n",
      "target overall the team came fourth in the championship\n",
      "prediction he i i  \n",
      "target he is initially not into the politics of the police department\n",
      "prediction he  \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "target a young baby boy is about to sneeze next to his mother\n",
      "prediction h \n",
      "target i know glenn wants it to continue\n",
      "prediction o \n",
      "target another example is north sea cod\n",
      "prediction i  \n",
      "target the region is called kosovo pomoravlje\n",
      "prediction the i  a \n",
      "target a small sample is taken from the specimen to be investigated\n",
      "prediction i\n",
      "target goren comes to think of her as his white whale\n",
      "prediction o \n",
      "target zettel was born in sacramento california\n",
      "prediction she \n",
      "target the new slogan your life\n",
      "prediction h \n",
      "target the dolls are then dressed in traditional mayan style\n",
      "prediction the i i \n",
      "target i was shattered at the news she had passed away\n",
      "prediction he i i  \n",
      "target he supports expanding the current cap on charter schools\n",
      "prediction the sis  i \n",
      "target the mets previously played at dobynsbennett high school\n",
      "prediction thi sis  a \n",
      "target in december he undertook a survey of western port\n",
      "prediction h  \n",
      "target it was only that you looked odd\n",
      "prediction h  \n",
      "target it is very similar to cyan\n",
      "prediction h  \n",
      "target he is also an artists representative\n",
      "prediction the si  i \n",
      "target woods again won the fight by unanimous decision\n",
      "prediction a \n",
      "target two men were killed and two others wounded\n",
      "prediction she \n",
      "target the album was a critical and commercial failure\n",
      "prediction the i i \n",
      "target the unit has fourteen police horses\n",
      "prediction h \n",
      "target she rejects kevin branding him a phoney\n",
      "prediction she  \n",
      "target never had he been in so strait a predicament\n",
      "prediction he \n",
      "target it was nominated for a grammy award\n",
      "prediction the sis a \n",
      "target delhi metro has train stations nearby\n",
      "prediction the i a  \n",
      "target it also might have some challenge involved\n",
      "prediction h  \n",
      "target this is the story of closers in new seoul\n",
      "prediction hs \n",
      "target traditionally wedding invitations are mailed in double envelopes\n",
      "prediction the sis  i \n",
      "target how the tradition originally started is unclear\n",
      "prediction the sis  i  \n",
      "target beside the card celias letter was laid\n",
      "prediction the si i  \n",
      "target it is located westnorthwest of williamsport\n",
      "prediction the si  i \n",
      "target the person is reading a book\n",
      "prediction i \n",
      "target attorney gerry spence represented the defendants\n",
      "prediction h \n",
      "target the boats had a complement of fiftythree\n",
      "prediction the si  i \n",
      "target filiatra has several schools churches and shops\n",
      "prediction he\n",
      "target you wont tell\n",
      "prediction he i a  \n",
      "target there are no private schools in the town\n",
      "prediction h  \n",
      "target the family lives in the atlanta area\n",
      "prediction h  \n",
      "target submissions are free of charge\n",
      "prediction sh \n",
      "target he now lives in myrtle beach south carolina\n",
      "prediction he \n",
      "target the small revolutionary republic had a short life\n",
      "prediction hs \n",
      "target passenger service was never very profitable\n",
      "prediction hs  \n",
      "target the vessel was stripped of its broadcasting equipment\n",
      "prediction hs  \n",
      "target kabua began his career as a school teacher\n",
      "prediction h  \n",
      "target i blew my whistle\n",
      "prediction h  \n",
      "target i never had any such idea\n",
      "prediction h  \n",
      "target some even consider it beautiful\n",
      "prediction o  \n",
      "target the bible in the beginning\n",
      "prediction o  \n",
      "target the ball games over\n",
      "prediction he i i \n",
      "target different variations include a scraping sound on the car door\n",
      "prediction h \n",
      "target hence the settlement of parkgate was born\n",
      "prediction h \n",
      "torch.Size([2, 198144])\n",
      "tensor([[0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 4.8324e-05, 7.6901e-05,\n",
      "         3.2309e-05],\n",
      "        [0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 4.8324e-05, 7.6901e-05,\n",
      "         3.2309e-05]])\n",
      "target iceman and magma are shown together\n",
      "prediction thi sis i   \n",
      "target the rosebuds for girls and young women\n",
      "prediction h  \n",
      "target the appeal was rejected\n",
      "prediction hei  \n",
      "target prior to world war one truck production began\n",
      "prediction he i i \n",
      "target preston v mercer in command\n",
      "prediction the si i  \n",
      "target he has worked as a farmer and was a management consultant\n",
      "prediction the sis a \n",
      "target i didnt cope well that night\n",
      "prediction she \n",
      "target many mudcats have reached the majors\n",
      "prediction he i i \n",
      "target the version was more grandiloquent than the geneva bible\n",
      "prediction h  \n",
      "target p a guthrie\n",
      "prediction he i i  \n",
      "target these kinds of widgets are hosted by a widget engine\n",
      "prediction h  \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "0it [00:00, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "target this completes the proof of the claim\n",
      "prediction he i i  \n",
      "average test loss is 2.79661633496616\n",
      "Epoch 1 of 150 LR [0.009990133642141357]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "352it [20:29,  3.49s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 275328])\n",
      "tensor([[ 0.0000,  0.0000,  0.0000,  ..., -0.0015, -0.0014, -0.0014],\n",
      "        [ 0.0000,  0.0000,  0.0000,  ..., -0.0015, -0.0014, -0.0014]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "460it [26:49,  3.50s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 200448])\n",
      "tensor([[ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -8.9087e-05,\n",
      "         -5.9199e-05, -5.9370e-05],\n",
      "        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -8.9087e-05,\n",
      "         -5.9199e-05, -5.9370e-05]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "462it [26:56,  3.51s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 205056])\n",
      "tensor([[ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -2.7344e-06,\n",
      "         -7.1339e-06, -7.7367e-05],\n",
      "        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -2.7344e-06,\n",
      "         -7.1339e-06, -7.7367e-05]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "516it [30:05,  3.50s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 180864])\n",
      "tensor([[ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -4.0159e-05,\n",
      "         -3.1214e-05,  3.6880e-06],\n",
      "        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -4.0159e-05,\n",
      "         -3.1214e-05,  3.6880e-06]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "573it [33:25,  3.51s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 200448])\n",
      "tensor([[ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -7.8108e-05,\n",
      "         -1.9141e-05, -7.8119e-06],\n",
      "        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -7.8108e-05,\n",
      "         -1.9141e-05, -7.8119e-06]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "581it [33:53,  3.46s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 131328])\n",
      "tensor([[0.0000, 0.0000, 0.0000,  ..., 0.0004, 0.0005, 0.0005],\n",
      "        [0.0000, 0.0000, 0.0000,  ..., 0.0004, 0.0005, 0.0005]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "695it [40:35,  3.50s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 270720])\n",
      "tensor([[ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -7.7933e-06,\n",
      "         -1.6287e-05, -5.2683e-05],\n",
      "        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -7.7933e-06,\n",
      "         -1.6287e-05, -5.2683e-05]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "1062it [1:02:04,  3.51s/it]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MEAN EPOCH LOSS IS 2.341327511545853\n",
      "target bauer was born in graz styria\n",
      "prediction aon as sto paas senea \n",
      "target in an instant she was free\n",
      "prediction ena tan te ta \n",
      "target dont you think thats a rather risky course of action\n",
      "prediction ation tes o ome as ti col ation\n",
      "target a ripping speech he said\n",
      "prediction e re ans pa esant\n",
      "target ok so i was a little dramatic but you saw what happened\n",
      "prediction hetao as ooltion i ae sol acin\n",
      "target it is known as the polka palace\n",
      "prediction it is knoas to fol of coles \n",
      "target the couple live in highgate north london\n",
      "prediction the coe ras  in dae o monton \n",
      "target the family settled in toronto\n",
      "prediction the son e seon te taton \n",
      "target thus he employed historical perspectives to serve philology\n",
      "prediction tosin pos oo estantos e ser fooai\n",
      "target she is after all a former goldwyn girl\n",
      "prediction t as a tollas on a bal an ro \n",
      "target huave is now considered an isolate\n",
      "prediction me is nolciontea in ai \n",
      "target in short i had come down on purpose\n",
      "prediction theo a conton poues \n",
      "target and she bowed under her suffering\n",
      "prediction ani  an over soufer\n",
      "target his career was controversial both at home and in ireland\n",
      "prediction shestani was on toaon bote fomaninaon\n",
      "target the castle is built in the renaissance style\n",
      "prediction the  alte o as in dete tea ane sta \n",
      "target it is owned by townsquare media\n",
      "prediction his on bi cosca mai \n",
      "target a keepsake was often printed to commemorate the occasion\n",
      "prediction the caa was oin proanen te anar atation\n",
      "target none were feebleminded\n",
      "prediction then a sin in anat \n",
      "target but theres no romantic side to heroin  its no good\n",
      "prediction e is now o en saelan ie \n",
      "torch.Size([2, 198144])\n",
      "tensor([[0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 4.8324e-05, 7.6901e-05,\n",
      "         3.2309e-05],\n",
      "        [0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 4.8324e-05, 7.6901e-05,\n",
      "         3.2309e-05]])\n",
      "target see list of governors of texas\n",
      "prediction s aes oen coer rans o paises\n",
      "target the rest of the world kept the new zealand title waiata\n",
      "prediction theas a mor cactin ean fae laer\n",
      "target a happy couple walking down the street and holding hands\n",
      "prediction e toa i an ti aia \n",
      "target we are the party of the charter\n",
      "prediction e a e pin on thi  \n",
      "target he was buried at putney vale cemetery\n",
      "prediction he was bard i omataro sonta \n",
      "target a superactinide island is already slotted in\n",
      "prediction es som pos af pona on e or ane toenin\n",
      "target bullets can be stopped by\n",
      "prediction te ani to tlie \n",
      "target these staircases go up to a crossover\n",
      "prediction thes ais as ol fo cosoer \n",
      "target he sometimes carries a large pickax\n",
      "prediction heson tan tas a ar tacat\n",
      "target one moon of jupiter is named elara\n",
      "prediction mon monte oper is nan a lorn \n",
      "target a woman who is skiing poses for the camera\n",
      "prediction a woin a stein poa fo tin\n",
      "target the rest of the underparts are yellowish with dark flank spotting\n",
      "prediction his lass to ian e ro y oler on to lai soen\n",
      "target it is one of the larger plateau glaciers in norway\n",
      "prediction it is on te lorer fa oleaters an nori\n",
      "target im going to get you home friend\n",
      "prediction in go  tation o \n",
      "target two of merrills four children predeceased her\n",
      "prediction te wa mras or tolin proesiter \n",
      "target the clayton store has a richmond heights zip code\n",
      "prediction he caons tol as aertion in i so car ee\n",
      "target die verstopfung ist wirklich sehr hartnckig\n",
      "prediction she  fas tofos e sa or bentiio\n",
      "target he often sings in a madeup language\n",
      "prediction he als on ses o  maolinweos\n",
      "target he was a boy scout in his youth\n",
      "prediction he was avied toton iao \n",
      "target modern taxonomists recognise two genera\n",
      "prediction mein taciss wie as tantanrar\n",
      "target it is notable for its bioluminescent properties\n",
      "prediction his is nai of frris a in etien pro sees\n",
      "target not right away\n",
      "prediction mo ra ra \n",
      "target so too with its studies\n",
      "prediction soton was a sais\n",
      "target his heart grew tender for everybody\n",
      "prediction hi so re sion e aerfe\n",
      "target say no to war\n",
      "prediction s o om ma \n",
      "target dunoon may have been important\n",
      "prediction temann ma a faan an futen\n",
      "target theatre on the other hand is terrifying\n",
      "prediction te er olai is ter iin \n",
      "target this connection uses three suspension bridges\n",
      "prediction thes ontion as as toras o tentin rrers \n",
      "target the handgrips are wood\n",
      "prediction ein pre an wo \n",
      "target likewise the pakistanturkey free trade agreement is due to be signed\n",
      "prediction wa was a batanter efresan uran ti ton e sin\n",
      "target she also won the most photogenic award\n",
      "prediction she als o on e oas toftiin ona \n",
      "target im inspector bucket and you can trust me\n",
      "prediction aes teboe in centrutiy \n",
      "target for all the while they lay safe in this cache\n",
      "prediction teo e wan anema on  cat \n",
      "target christoph and christian knaut\n",
      "prediction teto an petinmot \n",
      "target she is after him who offends her\n",
      "prediction he is ai an o le ano \n",
      "target he also found work on ranches wrangling cattle\n",
      "prediction thealion o or onantis maon ca \n",
      "target the first two years of studies are preparatory years\n",
      "prediction thears tion as steis o poen ta ans \n",
      "target this rather can be called an introduction\n",
      "prediction his oleartine coor in panrecacin\n",
      "target my bad\n",
      "prediction a dins \n",
      "target papua new guinea is the largest yam market in asia\n",
      "prediction a plotin is ioe tan mortianmasin\n",
      "target soccer players are meandering over the playing field\n",
      "prediction ste aers  a mandeof e canco\n",
      "target the area was developed largely for subsistence farming\n",
      "prediction theer was enol parian freren sein sfoien \n",
      "target he was elected member of parliament from almora\n",
      "prediction he as i moer pol e som on maa \n",
      "target wrestler jumping in the air\n",
      "prediction thes wen tom pin o la \n",
      "target hardwicke lived in chichester\n",
      "prediction for wi a a tatos tar \n",
      "target how does this occur\n",
      "prediction aas i ator \n",
      "target at that time two survivors of the crash were still alive\n",
      "prediction i a u o a as e ca o sto a \n",
      "target the building was mostly used for rock concerts\n",
      "prediction thee coin was nol siis ermoc cunsiecs\n",
      "target the final result will include the result code\n",
      "prediction tefi ersol menta ers oultea \n",
      "target they also host an annual music festival called lutefest\n",
      "prediction he also es tin onsi pesen of coin pas\n",
      "target the organisers hope to get sufficient signatures for a debate in parliament\n",
      "prediction eeo naer of deucon sanetrors or te ain fone\n",
      "target she had studied at newnham college cambridge\n",
      "prediction she as end in meon olne panret \n",
      "target packages may contain other packages\n",
      "prediction ta tis ma tecian of tecites \n",
      "target he has sought a new treatment for the disease\n",
      "prediction ee s so an maa ta an in\n",
      "target whats the timeline for the project\n",
      "prediction wes e i  forto catin \n",
      "target his recent books have been big sellers\n",
      "prediction his meon bo sof in de soes\n",
      "target so again we learned a lot\n",
      "prediction sia weorn a \n",
      "target the video starts with a dark hallway lighting up\n",
      "prediction the ta is ter o the tar olwawae \n",
      "target the following makes this precise\n",
      "prediction the in an mai tresoees \n",
      "target this section became the east subdivision\n",
      "prediction thes  teion tcan i iste suteion \n",
      "target i cant swim\n",
      "prediction e fana \n",
      "target poisoning can lead to tremors and seizures\n",
      "prediction a sin o a tetonas on sasis \n",
      "target it was a thing that affected the whole family\n",
      "prediction it  was a in ofic tealommin \n",
      "target he was a native of dublin georgia\n",
      "prediction ie wa na o poe teate\n",
      "target the bay is littered with wrecks and has many dive sites\n",
      "prediction thepais ei i rres an as banie ses \n",
      "target he was also the head coach at olivet college\n",
      "prediction he sals o aetut a oee colla\n",
      "target apple upgraded the motherboard free of charge\n",
      "prediction i on main ton te bo feaoftu\n",
      "target he is currently a senior vicepresident of law asia\n",
      "prediction he s or atin an astees of roa asan \n",
      "target it is locally common\n",
      "prediction ie is noi po \n",
      "target the ship explodes and kills him\n",
      "prediction the sie o ensi \n",
      "target you know what i want\n",
      "prediction t mo woe wo \n",
      "target often deportation is their biggest fear\n",
      "prediction a is e foan o teti  tee\n",
      "target inside the pericarp of the flower is the hakini shakti\n",
      "prediction the sa te tor o  or is aton supt\n",
      "target he is also cofounder of the world rolling series\n",
      "prediction ie s as actuon ef bol boin saes \n",
      "target jupiter has a tropical savanna climate\n",
      "prediction thepenteran as to toman tin ras o vennot conete\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "target he named the tortoise after his godson\n",
      "prediction he mans te colos o  poson \n",
      "target the white dog runs across the snow\n",
      "prediction the wi ton bon e case st\n",
      "target it is southsouthwest of the crater moretus\n",
      "prediction thea sa ss asion te traof toraes\n",
      "target the song became a modest success as a result\n",
      "prediction the so in tecan an moe sota sesoein\n",
      "target the facility in sault ste\n",
      "prediction hi   an sol scat \n",
      "target she tried to participate but nobody listened\n",
      "prediction she ane proti eaed of ome lain \n",
      "target pelias kills him and takes the fleece\n",
      "prediction thewi o an pa to fis \n",
      "target they are named interindustrial standards\n",
      "prediction the a in t in ar in toscols canders\n",
      "target it is limited to twochannel output\n",
      "prediction it is onan toe tro canal apa\n",
      "target fr diese blutgrtsche mit schraubstollen gab es zurecht die rote karte\n",
      "prediction s to tas o tain non seati os caer \n",
      "target i didnt have friends\n",
      "prediction ta in  caains \n",
      "target whitehorse mountain of the north cascades\n",
      "prediction me wars on often wo ostas\n",
      "target good enough for me\n",
      "prediction e na er a \n",
      "target it is not a film for most people\n",
      "prediction it  is noen ermos tacol \n",
      "target yes ada thought she had better go with me\n",
      "prediction nes a a toin af  aean\n",
      "target will was more polite but equally sceptical\n",
      "prediction the was noola te e osct \n",
      "target the items are then stored on racks in the cloak room\n",
      "prediction the a ins an in ser an ra in theorer\n",
      "target it was named for ezra stiles seventh president of yale\n",
      "prediction it  was nan of pises tin  feeain otin\n",
      "target he lives in greenwich village in new york city\n",
      "prediction iens in torane teen ana oe see \n",
      "target ordnance in mccarran nevada\n",
      "prediction armans in nectain eoa \n",
      "target both parties win\n",
      "prediction e oi a \n",
      "target both his father and mother were cheshire landowners\n",
      "prediction es is tol ein oan wer catern anan\n",
      "target my friend said i love you\n",
      "prediction a fonse aa\n",
      "target tonic pupils can occur in neurosyphilis\n",
      "prediction aec teos a po  aa sives\n",
      "target long live the king\n",
      "prediction e on ean e ae \n",
      "target did you speak to mr norton\n",
      "prediction eis ea to wes tinmormi\n",
      "target she was survived by pip\n",
      "prediction she was aror otent \n",
      "target this behavior seems to be realistic\n",
      "prediction this paer sin o en olestic \n",
      "target the grimaldis ended up in the guelph towns around nice\n",
      "prediction the cemois on to on the tol far an mas \n",
      "target eutychius was born at theium in phrygia\n",
      "prediction he isis is or e fa in frai\n",
      "target the journal is published every thursday\n",
      "prediction the torals oo ori ti \n",
      "target his younger brother was the linguist elmer bagby atwood\n",
      "prediction thes sen ten ear asi an e a aa \n",
      "target donohue beat back the challenge\n",
      "prediction tea an bea a te ta cionont\n",
      "target translators can also be used to provide services to the user\n",
      "prediction fenaeas tommainis o to tatie i ese\n",
      "target the royal palace is known as istana besar\n",
      "prediction thea dones s mola is oe son \n",
      "target rainfall occurs mostly during the spring and autumn\n",
      "prediction ten al a poars alaean is tain ocom\n",
      "target a woman walks holding an umbrella\n",
      "prediction thewas an mo o tenonca \n",
      "target the acorn matures at the end of its second season\n",
      "prediction tea coneturs a ian ee sein ceaion\n",
      "target the almohads were thoroughly routed with huge losses\n",
      "prediction thean  ton iaeon tii ste coses\n",
      "target a rubberized track was added\n",
      "prediction a rove te wo stae \n",
      "target february the twentysecond i think sir\n",
      "prediction te tea e tan setin ain sean \n",
      "target this is the role of the demister\n",
      "prediction thes as ro wof  imento\n",
      "target the shield is supported by two men of different shades of brown\n",
      "prediction the ion is pols ote en ete cins af cron \n",
      "target a brown dog is fetching something in its mouth on a field\n",
      "prediction the pron tis tations on pan ai non orn tae \n",
      "target this led to a ladder match at new blood rising\n",
      "prediction thes lon o ae monen no lo raion \n",
      "target he was born in chambly quebec\n",
      "prediction he s o an e ommai ie \n",
      "target the level ends when all the enemies are defeated\n",
      "prediction his ote on is pine on teaas ef tantao\n",
      "target tonya survived the shooting\n",
      "prediction te as e an peuntion \n",
      "target weather in parc national de nech sar\n",
      "prediction mee n or  mauon on den man\n",
      "target he later became governor of malta\n",
      "prediction hea recin tonaf mote \n",
      "target constructions workers work on a railroad\n",
      "prediction e teshin neros  one a\n",
      "target play some thirties frank wildhorn\n",
      "prediction thes o nderi fin aon \n",
      "target rescue ships swung into action immediately\n",
      "prediction wes i se o i toactin eanaly\n",
      "target his brothers name is larry and he lived in staten island\n",
      "prediction se oe  ter ais o tos staina\n",
      "target the origin of the towns name lies in its geographic location\n",
      "prediction he arin tefom i ras in e te ori mocaion\n",
      "target the standard botanical author abbreviation pringsh\n",
      "prediction theste e ean a oler reraion prancio\n",
      "target he first started rapping that year\n",
      "prediction he as aion rroin af jar\n",
      "target he died in brighton victoria\n",
      "prediction he dad in praion ti tori \n",
      "target they were among the founders of the republican party in mississippi\n",
      "prediction the a anmo e solers afero poe on parn weesi\n",
      "target what is to become of us\n",
      "prediction es to caas \n",
      "target a solar chimney can serve many purposes\n",
      "prediction the ol onmanon ser man erses \n",
      "target both parents came from fourthgeneration south boston families\n",
      "prediction e pa san ro to tinrerion so ason tenmis\n",
      "target a man makes bubbles while a nearby child gets excited\n",
      "prediction e an ma tofes so a e ba ta i to sa  \n",
      "target he has wounded men\n",
      "prediction he  as ondato dain \n",
      "target its simple mechanism was similar to a relay\n",
      "prediction thi se nais ion was teaer to oaay\n",
      "target sir charles howard\n",
      "prediction sr ts  \n",
      "target shant said the cook\n",
      "prediction te so te toupt \n",
      "target a woman in a black dress waiting to cross the street\n",
      "prediction thelon in o aeion as  \n",
      "target this has often been an inspiration to me\n",
      "prediction thes was ao be an terraion tomma \n",
      "target a snowmobile trail runs along dakota heights park avenue\n",
      "prediction is non to o traal ron aon e croa rcain\n",
      "target he decided to try what fortune had in store for him\n",
      "prediction he eid i e tro wa forion oni tolfa \n",
      "target the episode was written by series cocreator trey parker\n",
      "prediction teea oeton a seis cocae tapate \n",
      "target meanwhile faith manages to find peace in her jail cell\n",
      "prediction the wa a pa man is eancas of teoo \n",
      "target estrela da amadora\n",
      "prediction es te a coento \n",
      "target the nearby bukit jugra is used for paragliding\n",
      "prediction then ananea i tontae es tof faresin \n",
      "target its territory is now in eastern croatia\n",
      "prediction setai ois non e sin praason \n",
      "target those who find beautiful meanings in beautiful things are the cultivated\n",
      "prediction telse on gen on aians anano ans a foltonti\n",
      "target it is in the central part of the state\n",
      "prediction i inan san te foo  state \n",
      "target isnt that name a bit authoritarian\n",
      "prediction his an dem an veoan taion \n",
      "target the melbourne cup was brews last win\n",
      "prediction the nolan to i ti a t \n",
      "target they adopted a second son robert nearing\n",
      "prediction ne a o te sian son lolo ein\n",
      "target the canberra college is situated in phillip\n",
      "prediction it can ber ran poli sitonn into\n",
      "target why i could tell you a story\n",
      "prediction wo a e to a t \n",
      "target not to have tried is the true failure\n",
      "prediction the es a tae o tis i taa \n",
      "target they had three children two boys and one daughter\n",
      "prediction ten feiolren sovaion  ae\n",
      "target when they find them they will be able to return to japan\n",
      "prediction tew e an an pea e of tor h tan tein\n",
      "target he later killed another sikh\n",
      "prediction he  wa tetion to e se \n",
      "target he sprang up from his chair\n",
      "prediction thes mra a o  cer \n",
      "target this was held to be divine retribution\n",
      "prediction thes as ons o ba in moin atoneasion \n",
      "target it is unclear whether this had any effect on the elections outcome\n",
      "prediction it is onfa on te si in nsa tonlatos o ton\n",
      "target there is also conversion mode an elimination game type\n",
      "prediction he as al so confersionlol e onation inta \n",
      "target the sleeve rings encircled the lower cuffs\n",
      "prediction thesla pras tan serboe o enecas \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "0it [00:00, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "target it is the third single and final track from the album\n",
      "prediction itis atu si an aen sac for aro\n",
      "average test loss is 2.062117064700407\n",
      "Epoch 2 of 150 LR [0.009960573506572389]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "5it [00:19,  3.91s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 180864])\n",
      "tensor([[ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -4.0159e-05,\n",
      "         -3.1214e-05,  3.6880e-06],\n",
      "        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -4.0159e-05,\n",
      "         -3.1214e-05,  3.6880e-06]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "143it [08:19,  3.49s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 275328])\n",
      "tensor([[ 0.0000,  0.0000,  0.0000,  ..., -0.0015, -0.0014, -0.0014],\n",
      "        [ 0.0000,  0.0000,  0.0000,  ..., -0.0015, -0.0014, -0.0014]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "334it [19:29,  3.51s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 270720])\n",
      "tensor([[ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -7.7933e-06,\n",
      "         -1.6287e-05, -5.2683e-05],\n",
      "        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -7.7933e-06,\n",
      "         -1.6287e-05, -5.2683e-05]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "532it [31:09,  3.49s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 200448])\n",
      "tensor([[ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -7.8108e-05,\n",
      "         -1.9141e-05, -7.8119e-06],\n",
      "        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -7.8108e-05,\n",
      "         -1.9141e-05, -7.8119e-06]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "931it [54:28,  3.50s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 205056])\n",
      "tensor([[ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -2.7344e-06,\n",
      "         -7.1339e-06, -7.7367e-05],\n",
      "        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -2.7344e-06,\n",
      "         -7.1339e-06, -7.7367e-05]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "995it [58:11,  3.49s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 131328])\n",
      "tensor([[0.0000, 0.0000, 0.0000,  ..., 0.0004, 0.0005, 0.0005],\n",
      "        [0.0000, 0.0000, 0.0000,  ..., 0.0004, 0.0005, 0.0005]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "1003it [58:41,  3.78s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 200448])\n",
      "tensor([[ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -8.9087e-05,\n",
      "         -5.9199e-05, -5.9370e-05],\n",
      "        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -8.9087e-05,\n",
      "         -5.9199e-05, -5.9370e-05]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "1062it [1:02:06,  3.51s/it]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MEAN EPOCH LOSS IS 1.9229573038338268\n",
      "target she might be intimate and sad with him\n",
      "prediction she ritecin coad safer \n",
      "target it has recently undergone a renovation\n",
      "prediction i asersonla e din iweranation \n",
      "target his position is prop\n",
      "prediction his psition mi tro \n",
      "target shall we just get a takeaway tonight\n",
      "prediction te wegia tapoi andi \n",
      "target arthur went outside for some coal\n",
      "prediction ater wanolsi fosion chald \n",
      "target it is a hardy perennial plant\n",
      "prediction the pas p hary proin lpa\n",
      "target birsen was born in ankara turkey\n",
      "prediction theriston wa bo an cono terf turtie\n",
      "target the scheringa museum of realist art is in spanbroek\n",
      "prediction the finom musiom ofvrelastar is an spogork\n",
      "target his plan became known as the ripley plan\n",
      "prediction es tin becam non as te briple ti \n",
      "target there s cyclist in street\n",
      "prediction thes sitis tan ste \n",
      "target the woman is wearing a purple shirt and runs on the beach\n",
      "prediction the women is wi of opo thar a ron opi\n",
      "target this equity could be used to launch a mobile network\n",
      "prediction thes tecin to be ues to lonton kobenaort\n",
      "target the nature of the emerald city is changed in the film\n",
      "prediction the maturin alsity is ca on thetol \n",
      "target he also donated a large amount of items for auction\n",
      "prediction he as stoi olaen ai station\n",
      "target he was also a pioneer of computer chess\n",
      "prediction he was also pinaror copi chas\n",
      "target i want to pull his head off\n",
      "prediction a wonto pois haes \n",
      "target the park center has declined somewhat\n",
      "prediction the poarcinter astedcind son \n",
      "target each species has a distinct number of rings\n",
      "prediction ipaces afs testen te no ris\n",
      "target this valley is the source of dudhi river\n",
      "prediction thes la is e sos of totut rever\n",
      "target think of the luck of it\n",
      "prediction tofor te lot afit\n",
      "target his wife survived\n",
      "prediction she is war siit \n",
      "target the latter was a standard history reference for decades\n",
      "prediction the la wastan ister reen ordecin\n",
      "target i want to become a restaurateur\n",
      "prediction o wo o inmerristron urer \n",
      "target his greatnephew was george smith patton\n",
      "prediction the cralati wais dor et pain \n",
      "target krasnodar is the economic center of southern russia\n",
      "prediction toston the rs is et o acenter o in muter\n",
      "target he was also a member of the british relay team\n",
      "prediction he was also n mamber of the proi welatinm\n",
      "target these achieved for him a brilliant success\n",
      "prediction thes ati pi on teensi  \n",
      "target services were to wrexham and seacombe\n",
      "prediction sidi is w to becton an setar\n",
      "target he needed to show legitimate income\n",
      "prediction the i tocho legiinat canton\n",
      "target kathleen kennedy townsend\n",
      "prediction he adlae cir y cho steio \n",
      "target the current principal is pauric daly\n",
      "prediction he capas pa pori dady\n",
      "target for small scene graphs this tends to suffice\n",
      "prediction co wol i los tistans to fpie \n",
      "target the children know him by no other name\n",
      "prediction e sholn kni an bier ofe a\n",
      "target everybody tried to forget the scene\n",
      "prediction thevi vai citis to eti tii \n",
      "target an ambulance is near a pine tree\n",
      "prediction a nantur ans i er apite \n",
      "target many of the amalfitans in salerno sacked that city and left\n",
      "prediction mey of ennoftntin cler o sicte ianlat\n",
      "target b cook in command\n",
      "prediction he cok in tonad \n",
      "target he was one of the great climbers\n",
      "prediction he s for cla foirst \n",
      "target both are now in the custody of english heritage\n",
      "prediction wo an n he custoiven li haice\n",
      "target she came to the girls help\n",
      "prediction shecanto hecos ho \n",
      "target the county line still follows the original riverbed\n",
      "prediction the cotrerlisto fos ereno reteda\n",
      "target airline workers standing under a plane\n",
      "prediction thear lin morcursstin andoclo\n",
      "target salad days\n",
      "prediction solan das \n",
      "target at the time he was not believed\n",
      "prediction atei i was e titn \n",
      "target these impressive results however may hide a murkier reality\n",
      "prediction the is oncresireels i e my hie aterrinca\n",
      "target most settled elsewhere in castile\n",
      "prediction o sel als sor i cusste \n",
      "target canisius mascot is the golden griffin\n",
      "prediction teiatis i asco dis tecorde prier \n",
      "target destruction of private property will not be tolerated\n",
      "prediction the cition al pae poorntedoasi\n",
      "target see commercial law\n",
      "prediction sa torr lo \n",
      "target a little college girl from a school of journalism\n",
      "prediction a yii fr is bi oun sco loten aiter \n",
      "target the more turbulence the better the degree of dispersion\n",
      "prediction the orturins te berit igra tisprtion \n",
      "target wilson as the first mayor\n",
      "prediction wolson as te fors tmaer \n",
      "target bono is a leading advocate for drug abuse prevention and treatment\n",
      "prediction oes oldelot cet topis ventoantate\n",
      "target she was born in sainteustache quebec\n",
      "prediction she is bin an sa i sa cback \n",
      "target it was tampa bays first stanley cup in franchise history\n",
      "prediction it was taobas fost stanycop in franthascister\n",
      "target it is near the villages of brackenfield ashover and clay cross\n",
      "prediction it is nateelgo ar paantiol oselvo ca cas\n",
      "target the dolly copp campground is located here\n",
      "prediction the dol co angrn fo fai yer \n",
      "target so i called them up and asked them\n",
      "prediction sico d mt in laston \n",
      "target length in kilometers is in parentheses\n",
      "prediction then tinconerers tisonn fo tceae\n",
      "target it peaked at number nineteen in new zealand\n",
      "prediction the catet noer li ta and iilind\n",
      "target brown tendered his resignation eight days later\n",
      "prediction tontome is was aition a tisdeta \n",
      "target he travelled through france to rome in italy\n",
      "prediction he toe cn fies to lolin id \n",
      "target just the room\n",
      "prediction sos folon \n",
      "target his main residence is in bel air los angeles\n",
      "prediction is na waentofm ller oapa\n",
      "target it must not be lost\n",
      "prediction it bas not te lat \n",
      "target congressman and governor john page\n",
      "prediction o wesmenin goena jo ps \n",
      "target he was best known as the why not\n",
      "prediction he was be non as bolir no\n",
      "target he has none and no anger\n",
      "prediction the has no anowinger \n",
      "target he swelled attendance at wranglers home games\n",
      "prediction hes wa cantins sa riter holtas\n",
      "target if he were to die you would be the master\n",
      "prediction teh was it e wis pat loto\n",
      "target three young guys working in a dark room\n",
      "prediction sdan gis wi in o aur\n",
      "target stype stars with masers are uncommon\n",
      "prediction e to stos wor te laos iai cone \n",
      "target he cited a desire to spend more time with his family\n",
      "prediction hes tid it thesir is pen or ciof isteny\n",
      "torch.Size([2, 198144])\n",
      "tensor([[0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 4.8324e-05, 7.6901e-05,\n",
      "         3.2309e-05],\n",
      "        [0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 4.8324e-05, 7.6901e-05,\n",
      "         3.2309e-05]])\n",
      "target ahmed he also wrote songs for shalimar pictures\n",
      "prediction orec il sore sons for shaan mo fiche\n",
      "target doctor doreen o ryan is currently principal\n",
      "prediction to ciderain l riin is cor ly cricia \n",
      "target treatment is guided by extent of disease\n",
      "prediction toipin is cii bi stno dsis \n",
      "target two complete onehour shows\n",
      "prediction the conki won o shits \n",
      "target the size of nanocrystals distinguishes them from larger crystals\n",
      "prediction the siies a nacristlels i tenicis e  form largr crustels \n",
      "target he was made a pastor in alassio\n",
      "prediction hewas paed of pastrn aastir \n",
      "target the same sentence was confirmed by the appeals chamber\n",
      "prediction the saimtins was en forned byi cis champer \n",
      "target in ireland it has a much more localised distribution\n",
      "prediction the irei was luano cos e pition \n",
      "target weather in parc national de nech sar\n",
      "prediction mader and par natinol de nat so\n",
      "target start off with the glimmering landscape\n",
      "prediction sotas wit e tbarin rasta \n",
      "target his form for north queenslands first season was mixed\n",
      "prediction his foon folof cas on forcason was mad \n",
      "target the couple did not have any children\n",
      "prediction tpe i elto pa ae teer\n",
      "target the house was assessed to be worth ninety thousand pounds prior to the flood\n",
      "prediction the has s osese wo e any talsin co piro pla\n",
      "target nine are preserved\n",
      "prediction no a tosid \n",
      "target you cannot be shown\n",
      "prediction he canot pe shiol \n",
      "target he is married and has two children\n",
      "prediction heis nar an has twonchilten \n",
      "target get me the man\n",
      "prediction to ma bei \n",
      "target in the past the german name was domschale\n",
      "prediction an pas the gomin nan was dotr \n",
      "target i stood looking at him in silence\n",
      "prediction a slr inatin ind side \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "target it was now eleven oclock\n",
      "prediction it was aot ilean apok \n",
      "target snow was observed to fall from cirrus clouds\n",
      "prediction to was olte te fiton sai stoe \n",
      "target it is the county seat of stephens county\n",
      "prediction it is a couyseto  st fo cody\n",
      "target the series was then discontinued\n",
      "prediction the ciis was tan dis ontene\n",
      "target four individuals walk down a street with two holding hands and one holding a guitar\n",
      "prediction trr aas mn orstrin o to oin ons an wonorin toto\n",
      "target after their exchange the two departed coolly\n",
      "prediction atoraer a tins sme tror air et tudin\n",
      "target made his own lake\n",
      "prediction the is ol li \n",
      "target the school has been involved in two controversial incidents to date\n",
      "prediction the sco as anfol an turcutorrershialins iits toda\n",
      "target it is insectivorous and nocturnal\n",
      "prediction it is an setibors a no to \n",
      "target it is currently open as a museum\n",
      "prediction it is curntly i anas n uan \n",
      "target the music in the album was all composed and arranged by kikuta\n",
      "prediction the musen nape was al a posin raed by ticoter\n",
      "target it is the only law school in the state of new hampshire\n",
      "prediction hi is nowte was sco  tar in ater \n",
      "target housing is of goodquality in general\n",
      "prediction owsinn is oni col tein elo \n",
      "target the crashed meteor left a gigantic crater on the ground\n",
      "prediction the cuse orof to tetet i cratorin gro \n",
      "target shew helped in organizing her funeral even purchasing the coffin\n",
      "prediction she aotin o reisaeiro is opas ton thecon\n",
      "target in the short term the reforms stabilised the economy\n",
      "prediction en e solton drfol i stals econi \n",
      "target the line is not electrified and all trains are dieselpowered\n",
      "prediction the li is morpeaefi an olta siy apali\n",
      "target it stars bob monkhouse\n",
      "prediction its ta fot lon co \n",
      "target she dies the house survives\n",
      "prediction she dis te hl sbit \n",
      "target english is also spoken as a second language by many\n",
      "prediction agi is alsospocon as a sectlawit irey\n",
      "target i shook my head wondering even what it was\n",
      "prediction i si miared wondorn eid woti was\n",
      "target this did not apply to bonus jackpots\n",
      "prediction thestian a ied gon stacco \n",
      "target no specific categories were chosen\n",
      "prediction thes piti fetitois pa toser\n",
      "target middleton served in a variety of public offices in south carolina\n",
      "prediction teaconser in teridi ofe foces in sufterline\n",
      "target the melbourne cup was brews last win\n",
      "prediction the lolen toy stas wacor \n",
      "target a small dog walks with men in the city\n",
      "prediction o wo or aas w a in the sin\n",
      "target several scenes from the book take place in his apartment\n",
      "prediction ssins son tebo tapa ins apoldr \n",
      "target auf das was jetzt kommt habe ich nmlich absolut keinen bock\n",
      "prediction the fespufsits co iis asti cando \n",
      "target the flowers are windpollinated catkins\n",
      "prediction e foba olti pi lecer a \n",
      "target while heated over a fire all ingredients blend together well\n",
      "prediction wo e tore fiar olin reiins plito git te wo\n",
      "target modern herbalists also use it to treat catarrh and kidney stones\n",
      "prediction aan halis al sus t totra coa anckin e sta\n",
      "target fortunately no lives were lost in the incident\n",
      "prediction tin ain ories we ofst natent\n",
      "target we only sell organic food\n",
      "prediction woone solor danafo \n",
      "target he formed a new government later that day\n",
      "prediction he fa an o goet wy terat da\n",
      "target since father died sir ive gone out to work\n",
      "prediction he foe di se  ten ao \n",
      "target take it away my dear\n",
      "prediction he i la igeer \n",
      "target a group of asian young adults are gathered in a classroom setting\n",
      "prediction the o atin no ai tfart iancotin\n",
      "target harris was his wifes maiden name\n",
      "prediction ares w hisis wa tewa \n",
      "target keep your temper my good fellow keep your temper\n",
      "prediction he icun o mie fe kecioumpe \n",
      "target nogger black has a logotype that looks like graffiti\n",
      "prediction wowerlu hasalobeba te  sicdoity\n",
      "target a young girl enjoys a frozen treat by the pool\n",
      "prediction the angero aas frosintra i co\n",
      "target but then belboz suddenly disappears\n",
      "prediction me o dan ollo seli i a pars\n",
      "target the band also featured al scott on bass at some gigs\n",
      "prediction the dadalsorfiton als sco on bion somgas\n",
      "target this added greatly to rural poverty\n",
      "prediction thes said briitor ro opety \n",
      "target oil and ranching remain important to denver city\n",
      "prediction a orlinrantin erninpurnto im pisi\n",
      "target also the county is home to several attractions\n",
      "prediction ose the contys hol te sevvl of tactions \n",
      "target three israeli soldiers were killed in the operation\n",
      "prediction t is er o solis worcilin aersion \n",
      "target he had not finished his job\n",
      "prediction he ha no tiniti stol \n",
      "target he was not appointed and instead tim nielsen took over\n",
      "prediction he was ta opited in ha stetan oalsontebalver\n",
      "target for example seated presses may be done while standing\n",
      "prediction tordinpll seen pe mateden no standin\n",
      "target two outer wing tanks each contained of fuel\n",
      "prediction toe wltobin tens i coa o tobo\n",
      "target you know me and i know you\n",
      "prediction healy aoer \n",
      "target she also participated in volleyball\n",
      "prediction she as to potis pua todoio \n",
      "target what are the interdependencies of the application\n",
      "prediction wo oty an to the ponn saes of tetocation \n",
      "target stolz is signed to ford models\n",
      "prediction as is fied to forodes \n",
      "target it is a bird of woodland often near water\n",
      "prediction its purd of wodent ftin a water \n",
      "target it is always written in the third person\n",
      "prediction heer wis oonn a tafutiter \n",
      "target she never officially announced her retirement\n",
      "prediction she ever of fitioll ian ows tor wet tirt \n",
      "target hebi has several coal mines\n",
      "prediction ay ha sel co lis \n",
      "target so did the others\n",
      "prediction e e tawe \n",
      "target ill be a monkeys uncle\n",
      "prediction o be won tis iwe \n",
      "target this is illustrated in the animation below\n",
      "prediction this is i sire yannation tlo \n",
      "target remittance plays a major role in economy of the area\n",
      "prediction themiics claes o mateol a to ar the ar\n",
      "target they adopted a second son robert nearing\n",
      "prediction they opop to di secinso o o leer\n",
      "target the following schools serve the town of merrimac\n",
      "prediction the folin scos  sorthe toof rarrmat\n",
      "target more relevant than anyone dared expect\n",
      "prediction wo roen te ineon gar ixspeth \n",
      "target venice was named after venice italy\n",
      "prediction the lis was nam ater ini cadly \n",
      "target the origin of the name raiganj is not exactly known\n",
      "prediction towiti ats  am bituis is nitacky do\n",
      "target it was constructed of limestone and granite\n",
      "prediction it was instute of listoin pri \n",
      "target it operates with the use of a twisting throttle\n",
      "prediction he oporis wotti us soacostin frile \n",
      "target it also maintains a blog\n",
      "prediction i also laintas a bo \n",
      "target she was drawn to him\n",
      "prediction she wa tines o i \n",
      "target josef sommer completed the principal cast\n",
      "prediction joasot somar compide he cono the cast \n",
      "target leather not being stretchy cannot fit as tightly as the others\n",
      "prediction aer no bann sety tonuci is caa eters\n",
      "target three documentaries on roberts have been released\n",
      "prediction the docimniris on roer atnelis\n",
      "target a volleyball player striking a volleyball\n",
      "prediction e falbolyr striin a falwi\n",
      "target i always played it light and breezy\n",
      "prediction a alwis platetli inbrai \n",
      "target the springdale free public library serves the borough\n",
      "prediction the spadil si oi pibi ses abar\n",
      "target there were several theories\n",
      "prediction ther wa soo teris \n",
      "target day of wonders\n",
      "prediction the of wonders \n",
      "target here is an example using a named method\n",
      "prediction he es anecanpe is inan na watenh \n",
      "target he found an empty barn\n",
      "prediction he haltorra c bid\n",
      "target i offer them mentorship\n",
      "prediction i o toten eleter ca \n",
      "target a man pulling down a screen\n",
      "prediction a lan coin donis fan \n",
      "target a construction worker is digging in front of a machine\n",
      "prediction the con fuction wakos tedinin fono fom me\n",
      "target four people on the street running a marathon\n",
      "prediction toten hi stre ponn in nara fo\n",
      "target but hes not in london\n",
      "prediction wat ke orin lnlet \n",
      "target a man laying down in the street\n",
      "prediction a mala don i sta \n",
      "target they consult maps newspapers travel books\n",
      "prediction the contl lo i tipis tae po \n",
      "target it is a close relative to the sinker\n",
      "prediction it is a co bote i the siter \n",
      "target people hanging out on park benches\n",
      "prediction tpo inin ov ar pitbnces\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "target a clean version of the album was also made\n",
      "prediction a tem bied of the olte was also mad \n",
      "target what was that something\n",
      "prediction ofos da sotin \n",
      "target he was additionally trained in luta livre and boxing\n",
      "prediction he was a dismny taed in yotey far and pusing\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "0it [00:00, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "target it is the oldest known exposed rock in the world\n",
      "prediction hes ecordes ns sti te ia\n",
      "average test loss is 1.7388111671661948\n",
      "Epoch 3 of 150 LR [0.009911436253643442]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "77it [04:28,  3.48s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 205056])\n",
      "tensor([[ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -2.7344e-06,\n",
      "         -7.1339e-06, -7.7367e-05],\n",
      "        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -2.7344e-06,\n",
      "         -7.1339e-06, -7.7367e-05]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "195it [11:22,  3.54s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 180864])\n",
      "tensor([[ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -4.0159e-05,\n",
      "         -3.1214e-05,  3.6880e-06],\n",
      "        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -4.0159e-05,\n",
      "         -3.1214e-05,  3.6880e-06]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "458it [26:47,  3.49s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 200448])\n",
      "tensor([[ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -7.8108e-05,\n",
      "         -1.9141e-05, -7.8119e-06],\n",
      "        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -7.8108e-05,\n",
      "         -1.9141e-05, -7.8119e-06]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "779it [45:32,  3.48s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 131328])\n",
      "tensor([[0.0000, 0.0000, 0.0000,  ..., 0.0004, 0.0005, 0.0005],\n",
      "        [0.0000, 0.0000, 0.0000,  ..., 0.0004, 0.0005, 0.0005]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "858it [50:09,  3.48s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 275328])\n",
      "tensor([[ 0.0000,  0.0000,  0.0000,  ..., -0.0015, -0.0014, -0.0014],\n",
      "        [ 0.0000,  0.0000,  0.0000,  ..., -0.0015, -0.0014, -0.0014]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "887it [51:51,  3.47s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 270720])\n",
      "tensor([[ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -7.7933e-06,\n",
      "         -1.6287e-05, -5.2683e-05],\n",
      "        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -7.7933e-06,\n",
      "         -1.6287e-05, -5.2683e-05]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "998it [58:21,  3.49s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 200448])\n",
      "tensor([[ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -8.9087e-05,\n",
      "         -5.9199e-05, -5.9370e-05],\n",
      "        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -8.9087e-05,\n",
      "         -5.9199e-05, -5.9370e-05]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "1062it [1:02:06,  3.51s/it]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MEAN EPOCH LOSS IS 1.6946504578527533\n",
      "target this makes sure they are red and firm for storage\n",
      "prediction this mak ster ther ro infor fer str\n",
      "target thus the stimulus intensities are based on various fibers\n",
      "prediction tes tesenniss ancensiti arpas onveris pivers\n",
      "target gamespots frank provo reviewed the virtual console version of the game\n",
      "prediction espos pren crovo ret te vr in conober it bo\n",
      "target you cant sleep\n",
      "prediction the con sr \n",
      "target the city is served by the richland school district\n",
      "prediction the sily is sre bytediction sco tister\n",
      "target a man in a blue shirt holding a protest sign\n",
      "prediction e mani  boser con rodesin\n",
      "target it was produced by lil jon\n",
      "prediction it was pridys palilc \n",
      "target a prerelease enclosure was also constructed on the somerset levels\n",
      "prediction theperrely infloter ws also contrecte on the sere o loes\n",
      "target the two railroads cross downtown\n",
      "prediction oo ralon to mon\n",
      "target in practice the act only delayed the removal of the feeding tube\n",
      "prediction itn practe se ac o l tela teroeo the teing to\n",
      "target leonardo dicaprio is staring in the new movie\n",
      "prediction te noo tecacpri a s tin on the nelonby \n",
      "target thus was momentum gained in the younger world\n",
      "prediction teswes women to de in the goumber lol\n",
      "target it will be blessed for you\n",
      "prediction he c be les  \n",
      "target the family castle has disappeared\n",
      "prediction the femmy casi has bisuperrs \n",
      "target the oldest object in the british museum is a chopping tool\n",
      "prediction te ls orec in the veris msem is a trpento\n",
      "target this attractive bird is readily approached\n",
      "prediction thes xoctf bor as mee acrrs \n",
      "target it is commonly known as robotics\n",
      "prediction its conmoly mar as erboesh \n",
      "target the town derives its name from the saint joseph river\n",
      "prediction the town eris isnam from the sinjes ofvere \n",
      "target a man laying down in the street\n",
      "prediction a man lan don stret \n",
      "target it helps him a lot\n",
      "prediction e at so wo \n",
      "target williams inc\n",
      "prediction wo ons ank \n",
      "target the color of the sulfur smoke is used to find the leak\n",
      "prediction the colra t solfrersmo tis leesd o finnila \n",
      "target wilson and john e zuccotti\n",
      "prediction wolso  an jone ecoty\n",
      "target people can upload by recording the contents of their brains on computer disks\n",
      "prediction epte a orve ar in the onen of eprins an omreebese\n",
      "target however it has become a cult film\n",
      "prediction oweve t s becomo colt sti  \n",
      "target they had one child together\n",
      "prediction ther on troga\n",
      "target the station is above mean sea level\n",
      "prediction the station wis boin me selie\n",
      "target jodi lives in boise idaho\n",
      "prediction to dever en po y igaor \n",
      "target dont try kicking against the wind\n",
      "prediction to tri tii ada the wond \n",
      "target his poetry was beautiful stuff\n",
      "prediction hesol tr was o noston\n",
      "target aluminium and bronze were used in these tokens\n",
      "prediction aeme in aas wer ls fo lis cucems \n",
      "target he is thus nicknamed river dragon\n",
      "prediction it is bosmen nin re a droenr \n",
      "target peter begs brian to teach him how to be a gentleman\n",
      "prediction to vecs bran e tetion hoto be a jianr \n",
      "target i was fined for returning the library book late\n",
      "prediction iwe spin feres rinen livery onla\n",
      "target its seat is mansfield\n",
      "prediction it sea is ma sfedr \n",
      "target after leaving the army he entered politics\n",
      "prediction afte le ene ari te entror pot\n",
      "target another major element is signal emission control\n",
      "prediction a no mater eman i silinliso\n",
      "target work hard to unleash your full potential\n",
      "prediction ter ar toris tur fol aaalr \n",
      "target whats the weather in the current place\n",
      "prediction wots te rete in the curnpis\n",
      "target he was not appointed and instead tim nielsen took over\n",
      "prediction he was not oponted an e ste tim nolsen topoper\n",
      "target but i shall not need a nurse\n",
      "prediction a so mot le enert \n",
      "target his father was a carriage maker\n",
      "prediction its ove as oturs mace\n",
      "target a dog swims retrieving a stick\n",
      "prediction a dok swons rtte in e steck \n",
      "target during this time her interest in cohen foundered\n",
      "prediction ring westi arntes con somt\n",
      "target turn on slacker radio\n",
      "prediction comon sfoorade\n",
      "target goodbye liberty he said sullenly\n",
      "prediction the fi weverty e sa sonmes \n",
      "target there is a particular focus on canadian studies\n",
      "prediction there sae fetitiar focis n cani in stei\n",
      "target he was president of the chamber of commerce\n",
      "prediction he was prrsdentof the caber of cromers\n",
      "target what are the interdependencies of the application\n",
      "prediction wot ove in to the pnnn sees of te otation\n",
      "target this system continued through the european middle ages\n",
      "prediction this siston compinu to the yearapeninalages\n",
      "target benedict asked what have you done\n",
      "prediction te de t t o e don\n",
      "target i want to play a song from the nineties by roddy woomble\n",
      "prediction i won topo som from thebay sbyr roty o\n",
      "target it does not include most subway or light rail lines\n",
      "prediction it dis onco bo suvwalyrns \n",
      "target its capital is wavre\n",
      "prediction thes cocot i lar \n",
      "target his third speech marked the silencing of the friends\n",
      "prediction thistu spanc mon the cien sin of the fris\n",
      "target er ist ein richtiger proteinjunkie\n",
      "prediction it is ten rette poti i indonty\n",
      "target a woman cutting a brown cake\n",
      "prediction a won toe a ra ti\n",
      "target i need to plug my computer into a power point\n",
      "prediction any t bewc beteto bopont\n",
      "target drought frequently precedes movements\n",
      "prediction r for wo i resa  mimes\n",
      "target there is also a risk of igniting the mixture in the crankcase\n",
      "prediction e lo o is o mo ater in te cromcss\n",
      "target a bunch of people are walking in a crowded area\n",
      "prediction a bens a bemo o roki in a cropan area\n",
      "target a surfer on a roiling wave\n",
      "prediction the srfer o e ro lin pa\n",
      "target the town is run by a mayorcouncil government system\n",
      "prediction the di is non bae con o gomin siston\n",
      "target what are the colours of the rainbow spectrum\n",
      "prediction wo o the colens o the ran o stctro \n",
      "target he retired after that season\n",
      "prediction he redid aterbasit \n",
      "target the capital of the province became temeswar\n",
      "prediction the capoa lot oens beta mesmor\n",
      "target geary has seven brothers and sisters\n",
      "prediction ter has o eotes  sistrs \n",
      "target neum is planned to be a freight port\n",
      "prediction no is plie to bea fri par\n",
      "target they are maintained at yverdon\n",
      "prediction the armantan at overdonr \n",
      "target riders can choose to sit in either a single or double ring\n",
      "prediction mo as conusd to siin eon si o tuping\n",
      "target what did you think of the curried parsnip soup\n",
      "prediction wot te the pen of te pi pomensor \n",
      "target after some time they realized the ship had left\n",
      "prediction ot te oum bi e els bishi abi\n",
      "target this from the university of pennsylvania\n",
      "prediction thes fromme iversty ponsoai \n",
      "target the original production records are considered lost\n",
      "prediction heyereto o protutonacrens acconier lost\n",
      "target a female singer on stage playing the guitar and singing\n",
      "prediction bot fe siner o stg pla it teteran sini\n",
      "target from outer space\n",
      "prediction co ater spacst \n",
      "target it is arranged by country of origin\n",
      "prediction it ise ransd my comre of ote\n",
      "target this was the third time he had received the award\n",
      "prediction its was a ports fo e arese the ar \n",
      "target there are four main supportive dragon pillars each representing a season\n",
      "prediction te acol mas apor rie cos introc teeto e sier\n",
      "target hansen was trained as a typographer\n",
      "prediction oson was trein is a tipogefor \n",
      "target how still and peaceful everything is said angela\n",
      "prediction o sti ones ol errycii secanto\n",
      "target he was a landowner and industrialist\n",
      "prediction ye alonbo er anden  trela\n",
      "target the reunion of diamond head did not last\n",
      "prediction the reun in a bencar the nowas \n",
      "target cause of death pneumonia brought on by lung cancer\n",
      "prediction olso ta  lana os on by lon ona\n",
      "target so they will remain in the eerste divisie\n",
      "prediction sele woa in the earst the day\n",
      "target she was thereafter replaced in the show by georgina hale\n",
      "prediction she was ther out tor replased o the sr byto te a hia\n",
      "target his political affiliation was with the congress party\n",
      "prediction s ple ocooffination as wit the comves pordy\n",
      "target and why not boys and ladies and gentlemen\n",
      "prediction i wi not bos an late sentu temem\n",
      "target i dont want one\n",
      "prediction igo worwon\n",
      "target com and b\n",
      "prediction o do co anbe ot\n",
      "target reese has three older siblings\n",
      "prediction ws as te althersiblin\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "target gilbert dayles what would a gentleman do\n",
      "prediction doer da wt we itentolen de\n",
      "target repeatedly people ask him where he is from\n",
      "prediction tepate y people as o wervin sco\n",
      "target the city houses sombor airport\n",
      "prediction the siy as e sonbor ar coy \n",
      "target young children stand outside on a dirt surface\n",
      "prediction yo sin en asto osi on e to su\n",
      "target we are in the milky way galaxy\n",
      "prediction nearin the me toly groloyr \n",
      "target they have worked with los condores\n",
      "prediction the ae war toth los comndonas\n",
      "target connecting with union station and olvera street\n",
      "prediction tomactin wot te min sta ion ntovearsto\n",
      "target he is married and has two children\n",
      "prediction the is mar an has to cilbren \n",
      "target but the fears proved groundless\n",
      "prediction the tofer pre craunposr \n",
      "torch.Size([2, 198144])\n",
      "tensor([[0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 4.8324e-05, 7.6901e-05,\n",
      "         3.2309e-05],\n",
      "        [0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 4.8324e-05, 7.6901e-05,\n",
      "         3.2309e-05]])\n",
      "target in the final though they would lose to sweden\n",
      "prediction in te fi e to te weres tspenr \n",
      "target it mounted eight guns\n",
      "prediction  no otg\n",
      "target seltzer refused as he had no interest in writing sequels\n",
      "prediction solse refusd os ea narentresten war oncicas \n",
      "target two years later the family returned to london\n",
      "prediction tweys later the fily eturn tolendoy \n",
      "target this gentleman is talking on the phone\n",
      "prediction the gimm was cokn in the fo\n",
      "target with their architecture id never get the speed\n",
      "prediction the ta on ceseton in mevie thestatt \n",
      "target a man paddling a kayak along the shore of a river\n",
      "prediction oan pin wo iop a on th sir fo briver\n",
      "target oh leah how can you be so cruel sobbed anne\n",
      "prediction woo yea ancny beso pr sotha\n",
      "target my lady do you object to the twilight\n",
      "prediction ila to d optec to the prinitr \n",
      "target sorry i cant see ive lost my glasses\n",
      "prediction ser iten seter neti\n",
      "target he was born and died in hamilton ontario\n",
      "prediction he was bon ano in heeton ontera\n",
      "target the father follows his wife to get the unhappy son\n",
      "prediction the foter folos oswied tot te on aeso\n",
      "target the engine raced the clutch slid into position\n",
      "prediction the antin rst eclo li in the position\n",
      "target after this he had a stint working for a jamie symonds\n",
      "prediction offto bes ad esten torking for atamesinmens\n",
      "target cats was sent to school in breda\n",
      "prediction cas wes son te scon bar \n",
      "target located in a beautiful setting not far from warsaw\n",
      "prediction woloadis coabef alseving not for from worso\n",
      "target the following year he entered the convent of san marco\n",
      "prediction the fon year vheeter i the crommin asener \n",
      "target a character part\n",
      "prediction the carrc te or \n",
      "target a dog is jumping up in the air to catch a toy\n",
      "prediction a dog i domking ap in te yar to tacch to\n",
      "target but thats not what avantasia is all about\n",
      "prediction te bas wi e tiers o or \n",
      "target i graduated last of my class\n",
      "prediction agrter i las o mi is\n",
      "target he was a member of the national academy of sciences\n",
      "prediction he was mmber of the nao aeme sies\n",
      "target from there he was sent to rome\n",
      "prediction wo te e as ero\n",
      "target sadly the inn no longer exists\n",
      "prediction saly an ain no lonersiss\n",
      "target for the record did she hurt my career\n",
      "prediction wo the roper te shecer my corer\n",
      "target the father punches desmond\n",
      "prediction the fover oco stesimnt\n",
      "target a major factor influencing the birthrate was the abortion rate\n",
      "prediction a mater fater infotin the tersfe was te eborsoa\n",
      "target the peak is so named due to the peak having four corners\n",
      "prediction the pe o semin e to thepe in foforms\n",
      "target many commented on the phenomenon\n",
      "prediction meny coer to on limont \n",
      "target she had a brother david rowlands\n",
      "prediction sheha e rotr tei forlonss \n",
      "target its not an inappropriate relationship\n",
      "prediction its po an i a proprea telationso\n",
      "target a woman wearing a black dress crosses the street\n",
      "prediction o woman wern te black tros croms s the stre \n",
      "target they were interested in collaborating for a track\n",
      "prediction theer intesta in colober in feratuk\n",
      "target the library is a fourbranch system\n",
      "prediction the liger isop foagrang siston \n",
      "target after hours the amphitheater is used for a popular music in the zoo series\n",
      "prediction affro was epefae yus fore poppo amee o thesinsars\n",
      "target baby in brown jacket crying and walking to a boy by orange car\n",
      "prediction avian bronate clon topor i a\n",
      "target the main towns on the island are podor and salde\n",
      "prediction the ma ons on aa na foto an tit\n",
      "target the skin is pure white\n",
      "prediction e asca is pro wit \n",
      "target employees joined the writers guild of america east\n",
      "prediction a i sis a rae t o rao\n",
      "target since then the camera has never left his side\n",
      "prediction sen sti the caer as never wf t sir \n",
      "target a man in a black coat is walking outdoors\n",
      "prediction a ma abrak co es bo in o sco\n",
      "target it is the thirdsmallest county in vermont by area\n",
      "prediction it s epare los can i temn a a\n",
      "target this became his most famous characteristic\n",
      "prediction the sprecam as mos fanss car or restec \n",
      "target the change to the order of succession was widely accepted\n",
      "prediction the tied to the rer suxestions wi wy esuct\n",
      "target a person kayaking down a small waterfall\n",
      "prediction a cerson ira tin donismo waerfol \n",
      "target did you ever see such roses\n",
      "prediction wo teare si ene \n",
      "target chancery justice is so very difficult to follow\n",
      "prediction so sery justi i sever te co folr\n",
      "target paul revere and the world he lived in by ester forbes\n",
      "prediction olrerved on  ro e liten y ester fors\n",
      "target trirail uses two types of passengers cars\n",
      "prediction crirres s two tie o cai is carsh \n",
      "target the multiedit module is a good place to start\n",
      "prediction tn mote a moto i ico pestr\n",
      "target no i was not attracted by her\n",
      "prediction no a wa no atrac en byrsr \n",
      "target after the fall of the empire benedetti retired to corsica\n",
      "prediction ofors e fo of eempya peni ity di tocosecar\n",
      "target six council members wanted to continue the legal fight\n",
      "prediction sixccul of eers wone e contiy lege irs \n",
      "target about of pollution flowed into the river\n",
      "prediction a bo of pollisionsloden to the rer \n",
      "target will was more polite but equally sceptical\n",
      "prediction we was norli at becosepce\n",
      "target trains terminated at the platform and changed tracks before entering it\n",
      "prediction cra emmes o he popfu unta rac forto\n",
      "target the artist can then develop images based on what is seen\n",
      "prediction the erosconnen te vol aigespae om re i se\n",
      "target the new version of the format was called sing\n",
      "prediction ely vesin fom s co sint \n",
      "target the opening recital was given by william thomas best\n",
      "prediction heopertersie wasca en by riins otespas\n",
      "target garuda then failed to pay the compensation\n",
      "prediction trge len pi pi  ompsation \n",
      "target below is a list of some troma distributed films\n",
      "prediction nolo en sons coso or  tutefiss \n",
      "target numbers are in thousands\n",
      "prediction nore ern tosnsr \n",
      "target the reviews were mixed\n",
      "prediction the reges rer maksetr \n",
      "target it is a remixed version of the song\n",
      "prediction it is remexper of the sor \n",
      "target a pleased older woman blow drying her short redhair\n",
      "prediction the prees tol e woen bocrin isi rape\n",
      "target portage lakes career center is the vocational school for springfield\n",
      "prediction oer laks toris enta i wo tocae sco osprati\n",
      "target the original footbridge remains in situ\n",
      "prediction the regjolf to pra frmons in sitor \n",
      "target he cofounded id software\n",
      "prediction epo on di ofter \n",
      "target this is the eddy current\n",
      "prediction gorte i ati cora\n",
      "target their dual presenter format has proved to be successful\n",
      "prediction ted es ontor foma spo to be sitsor \n",
      "target meetings can be set far apart\n",
      "prediction mets com be se foco\n",
      "target it is on the entertainment boost along with lifetime\n",
      "prediction the is on ing eertm i o solon te i irs \n",
      "target a thing begun is half done\n",
      "prediction the tin tego is otor \n",
      "target telemark resort in cable wisconsin\n",
      "prediction colomar res or in pabe was conson\n",
      "target of course i wanted to write them\n",
      "prediction af scor sin linde tvitemr \n",
      "target there are several types of ribs\n",
      "prediction thee aseve titorish \n",
      "target several configurations are practical\n",
      "prediction sevvel consiterasions aprackcr \n",
      "target there is no visitor center at the refuge\n",
      "prediction there is novis iesetreaftherreet\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "target several different political spectra have been proposed\n",
      "prediction sever defr pis cospctr a bepirs\n",
      "target the couples daughter ivana is also an actress\n",
      "prediction the coples oa bonmi is also acces\n",
      "target the loud crackers went off during the festival\n",
      "prediction thee log cricpers war of duin tefetiar \n",
      "target its headquarters is located in kabul\n",
      "prediction the sa coers was locater ea cophe wor\n",
      "target all but fyshwick have book shops\n",
      "prediction o t tis wri a bsat\n",
      "target he sprang excitedly to his feet\n",
      "prediction the scran ise co to e scor \n",
      "target sellers would become a longlasting close friend\n",
      "prediction so  be coe lon los in clus sri \n",
      "target my hat it has three corners\n",
      "prediction maha n has ore co\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "0it [00:00, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "target youre worth more dead than alive\n",
      "prediction ber wr te e al\n",
      "average test loss is 1.6230953842560876\n",
      "Epoch 4 of 150 LR [0.009842915805643154]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "166it [09:42,  3.48s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 200448])\n",
      "tensor([[ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -8.9087e-05,\n",
      "         -5.9199e-05, -5.9370e-05],\n",
      "        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -8.9087e-05,\n",
      "         -5.9199e-05, -5.9370e-05]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "325it [19:00,  3.50s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 131328])\n",
      "tensor([[0.0000, 0.0000, 0.0000,  ..., 0.0004, 0.0005, 0.0005],\n",
      "        [0.0000, 0.0000, 0.0000,  ..., 0.0004, 0.0005, 0.0005]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "394it [23:01,  3.49s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 270720])\n",
      "tensor([[ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -7.7933e-06,\n",
      "         -1.6287e-05, -5.2683e-05],\n",
      "        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -7.7933e-06,\n",
      "         -1.6287e-05, -5.2683e-05]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "465it [27:11,  3.48s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 200448])\n",
      "tensor([[ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -7.8108e-05,\n",
      "         -1.9141e-05, -7.8119e-06],\n",
      "        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -7.8108e-05,\n",
      "         -1.9141e-05, -7.8119e-06]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "478it [27:57,  3.50s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 275328])\n",
      "tensor([[ 0.0000,  0.0000,  0.0000,  ..., -0.0015, -0.0014, -0.0014],\n",
      "        [ 0.0000,  0.0000,  0.0000,  ..., -0.0015, -0.0014, -0.0014]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "721it [42:10,  3.46s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 180864])\n",
      "tensor([[ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -4.0159e-05,\n",
      "         -3.1214e-05,  3.6880e-06],\n",
      "        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -4.0159e-05,\n",
      "         -3.1214e-05,  3.6880e-06]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "727it [42:31,  3.49s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 205056])\n",
      "tensor([[ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -2.7344e-06,\n",
      "         -7.1339e-06, -7.7367e-05],\n",
      "        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -2.7344e-06,\n",
      "         -7.1339e-06, -7.7367e-05]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "1062it [1:02:08,  3.51s/it]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MEAN EPOCH LOSS IS 1.5611987014050313\n",
      "target anderlecht won on coin toss\n",
      "prediction  derlec wa an qintes\n",
      "target the hallway was smeared in blood\n",
      "prediction the oi was sbar inla\n",
      "target its county seat is eureka\n",
      "prediction its county sat is eriter \n",
      "target woman and baby waiting at an airport\n",
      "prediction weme wotae wate itear fo \n",
      "target he served under general arthur currie\n",
      "prediction theshere abuber ana arer puis \n",
      "target saint francis called him the knight of our round table\n",
      "prediction san fani sporvin the ie offarrol ti\n",
      "target the opposite concept is lusophilia\n",
      "prediction the opaic coensettii affiri\n",
      "target while the girls learned to play the guitar and dance\n",
      "prediction p the gos erin epla it tetar in tatss \n",
      "target see sons of odin\n",
      "prediction sa a snn ofvirtent \n",
      "target all of the fun was out of it\n",
      "prediction wi of the porin was o tolet\n",
      "target no player has won the award more than once\n",
      "prediction ne laer has o jowar moring wans\n",
      "target innis and taught canadian economic history\n",
      "prediction he is ato onlagula colicistrs \n",
      "target but even worse was to come\n",
      "prediction bat y in worc was tecomme\n",
      "target the ashley children were all roped into the business\n",
      "prediction the arshatin an woroa ro ting to evitnas\n",
      "target popularly known as nova aprile\n",
      "prediction overally now as no a apidial   \n",
      "target it also includes sable island\n",
      "prediction it also includ sap ilent \n",
      "target he spoke a good deal about death too\n",
      "prediction his pok e tiabote ass \n",
      "target this channel will not feature the pirillo vlogs\n",
      "prediction this ctle  lo fitra theparo teos\n",
      "target consider a unit cube\n",
      "prediction nsiteri o it si \n",
      "target these kinds of widgets are hosted by a widget engine\n",
      "prediction hes antebite ile t bopisivented \n",
      "target the duties of a finance minister differ between countries\n",
      "prediction the duris of fonnins in ister iffer tetramcentras\n",
      "target it was a master\n",
      "prediction it was an mastert \n",
      "target two dogs playing in a field\n",
      "prediction two dos playing in ated \n",
      "target he is hiding at my place\n",
      "prediction he is hirding at matlas\n",
      "target it met with success at home and abroad\n",
      "prediction it mex mith sucsis ac ol and abor\n",
      "target and personally i hate new jersey\n",
      "prediction then cisionly i ain jursis \n",
      "target conditions in the camp were brutal\n",
      "prediction theditions i acappo tebe \n",
      "target i know this prayer\n",
      "prediction i no i pret \n",
      "target mr brad cables you need cash\n",
      "prediction ist te wo twas in  tash \n",
      "target everybody carries a mobile phone nowadays\n",
      "prediction awebuty caris in e we foin nowidass \n",
      "target a photographer taking a picture of two men in front of a banner\n",
      "prediction baf ta thogefe taging apicto two men in fonti a ba\n",
      "target without treatment death may occur\n",
      "prediction wet takmnt det t meco\n",
      "target but as for the earrings  i will tell you\n",
      "prediction theas e heeris a witaia \n",
      "target it is located westnorthwest of williamsport\n",
      "prediction it is ricacted ist nolist o bierporte\n",
      "target northern housed the states death row for men\n",
      "prediction moten hosd a spet difers of man \n",
      "target colby is part of the colby school district\n",
      "prediction tlabe is par of the cobi scoo tistrurts \n",
      "target women in a rock band perform on stage\n",
      "prediction woein errocta ta forman stas\n",
      "target im a comedian doing radio\n",
      "prediction i a conmite a tey reh \n",
      "target his comments were endorsed by the british national party\n",
      "prediction this coarns wer in borse by the pridishartial partys \n",
      "target however they soon parted ways\n",
      "prediction te evervition ortid waest \n",
      "target it is uncommon as a given name\n",
      "prediction his i cormet is agivenas \n",
      "target she later returned to the show as sister mary joel\n",
      "prediction she waerahar of a shol i sistervary ja\n",
      "target the company was based and registered in west germany\n",
      "prediction the company was baed a medister in betarmny\n",
      "target gotta be gentle to suit me\n",
      "prediction t if echan i te sie\n",
      "target a young boy smiling next to the ocean\n",
      "prediction r ian boi maing lex to teision\n",
      "target hiroyuki nakano\n",
      "prediction he er etein tacovit \n",
      "target tenar receives word that moss is dying and wants to see her\n",
      "prediction the errese wor hat mos isfare in was tseten\n",
      "target woodlands has a long history and is known for multiple reasons\n",
      "prediction wean pasa lo istri ad is mar or mo tifesens\n",
      "target she has an older brother todd\n",
      "prediction she hasi oreter n\n",
      "target externally it is painted white\n",
      "prediction it te oly iti pantotwa\n",
      "target francis rarely reused his lead characters\n",
      "prediction tanisrata reuseitali tarites\n",
      "target not even in a single shot\n",
      "prediction the iile so a s \n",
      "target in total several hundred thousand votes were cast\n",
      "prediction an totle sevva funtit tels in vit we cast\n",
      "target horses also appear in her writings\n",
      "prediction corss also ater in herigangs\n",
      "target the concept has been compared to the monroe doctrine\n",
      "prediction the contict as bit compur to te motra dortray\n",
      "target the other bridge was for everyone else\n",
      "prediction the as ase is stifteasas \n",
      "target now it is directly under the control of the city\n",
      "prediction no is teicta af te the conthl of li sitt\n",
      "target it is used extensively for weddings\n",
      "prediction it is use texta inliffel witings\n",
      "target he took it up and spoke to the nurse\n",
      "prediction he op i o an spopciven erc\n",
      "target he signed a twoyear contract\n",
      "prediction heis ainitea cotat\n",
      "target the citys moniker is the city of friendly people\n",
      "prediction the sit is wol gares the sicof fri dipo\n",
      "target this test is no longer frequently used\n",
      "prediction this test i ar lon defriwit the ued \n",
      "target online access was by subscription\n",
      "prediction ami atta as ba sinscection\n",
      "target reviews for the album are mixed\n",
      "prediction wetidused fo te apem pomicx \n",
      "target some of the firefighters ran their hoses down to the thames\n",
      "prediction some of he fare aters raner o istarn to the ters\n",
      "target morocco advanced to the final round\n",
      "prediction raber ban to te fi we\n",
      "target here was the emporium the first port on the river\n",
      "prediction tre was e inporiand the firsc port on e river\n",
      "target after this he renames himself hoist\n",
      "prediction oteters serinan a sof qoss \n",
      "target play too much rain by the artist dloc\n",
      "prediction caa to mting ar the wartis t\n",
      "target i thank you very very much\n",
      "prediction i ten yo thery erymuch\n",
      "target scarlett began doing small jobs until they found an agent they both liked\n",
      "prediction tarlla peenducs wra tation to wit faen ita pabopa\n",
      "target it has a swiss stamp\n",
      "prediction it as aspis stendt \n",
      "target it shall not be spoken of again\n",
      "prediction the tom lot t espokanals egn \n",
      "target he attended a catholic school in vienna\n",
      "prediction she etten e the caic scooi teen\n",
      "target helmut zacharias was born in berlin\n",
      "prediction oman sacke was worn in boman\n",
      "target see filipino alphabet\n",
      "prediction se tilopinal al ofbeat \n",
      "target norrin begins dating a woman named amy\n",
      "prediction moring tecins tatin aboman nan amy\n",
      "target he died in ascot\n",
      "prediction heired en mas cart\n",
      "target a man seated in a kitchen types on a laptop\n",
      "prediction i meenie en etiten pats an a leter\n",
      "target he is now retired\n",
      "prediction he s na retird \n",
      "target israel withdrew its forces by june\n",
      "prediction this a wit tare for is pyto\n",
      "target kelly was born in boston massachusetts\n",
      "prediction caly was born in bost en a mestchasess \n",
      "target i kissed her bright hair which was all i could see\n",
      "prediction i ist to pre har wic was poli coses \n",
      "target father of three children\n",
      "prediction fother of tre chodren \n",
      "target the station was used in the making of several films\n",
      "prediction thestation was ued in the making of seer filns\n",
      "target in trying to stop her he falls\n",
      "prediction the tran destoper hefolss \n",
      "target he was born in nuremberg and died at wittenberg\n",
      "prediction he was born in nowbork and di in itn bors \n",
      "target she has a staid composure\n",
      "prediction she has a sta conportere \n",
      "target check the calorimeter\n",
      "prediction thac the coery meter\n",
      "target however these arrangements remained theoretical\n",
      "prediction owef the is arenge en werman the areica \n",
      "target the children refuse and dive back in to the pool\n",
      "prediction te shilder er fes an dai becan ofa tols \n",
      "target birsen was born in ankara turkey\n",
      "prediction therisa was born in an fee tuiti\n",
      "target the band is expected to release a new album\n",
      "prediction the bans expacted  rile in awale\n",
      "target baseball player grabbing ball in his glove\n",
      "prediction cal fla rrain boin is o \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "target it was named after lake trasimeno\n",
      "prediction it was naeed of the la trasin nenneve\n",
      "target it is usually stored cold under argon\n",
      "prediction it is usly stor co on er orgon \n",
      "target mamboundou was born in mouila\n",
      "prediction the bode was born in woila \n",
      "target it is the premier baron in the irish peerage\n",
      "prediction it is the pomer barin in the iwis purlage \n",
      "target she would never speak to him\n",
      "prediction she wt teiverstic to hine \n",
      "target she sang harold arlens a sleepin bee\n",
      "prediction she san hero orlins te swean ba\n",
      "torch.Size([2, 198144])\n",
      "tensor([[0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 4.8324e-05, 7.6901e-05,\n",
      "         3.2309e-05],\n",
      "        [0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 4.8324e-05, 7.6901e-05,\n",
      "         3.2309e-05]])\n",
      "target after three months the revolt in babylonia had ended\n",
      "prediction batti whremods tevol te bobelo iy hadende  \n",
      "target kinski began to work with director werner herzog\n",
      "prediction shens ty begete wer it terecter for ther harsa \n",
      "target postwar reconstruction was slow\n",
      "prediction tes wor y consection wa sllot \n",
      "target several other unreleased solo projects circulate\n",
      "prediction sivevel other anremy sol a progic secawat  \n",
      "target critical reaction ranged from praise to disgust\n",
      "prediction car ite viaction a janed fo paises o bicust\n",
      "target overall the team came fourth in the championship\n",
      "prediction oower thetincin forsoichafintiph \n",
      "target apple upgraded the motherboard free of charge\n",
      "prediction ipa apited the wo e wo reoftete\n",
      "target that was about all i wanted\n",
      "prediction thee was a be boln wonted \n",
      "target measurements are made daily with the aid of cameras\n",
      "prediction mershoits arema daly wit the a of cambes\n",
      "target the team plays in yellow shirts and black shorts\n",
      "prediction the heplays int ilo shors a washorts \n",
      "target after this it slowly fell off the charts\n",
      "prediction asterti it sody fel of the chorts\n",
      "target despite her rejection he never gets over her\n",
      "prediction the spie e rijection enever tut overer\n",
      "target not in the least said he\n",
      "prediction ter alolestcitiot \n",
      "target there are no invincibility items\n",
      "prediction there are no in tinsiwite ademmst \n",
      "target alberghetti was a child prodigy\n",
      "prediction  wor tit  was a cha e portity\n",
      "target a little girl wearing a princess halloween costume\n",
      "prediction a woa gol waring aprenses oeing cotin\n",
      "target the franklins had no children\n",
      "prediction the facos av no chidrens \n",
      "target hopper became the laboratorys director\n",
      "prediction aef becan the lever oristoecter \n",
      "target uraninite is a major ore of uranium\n",
      "prediction heran alit is a matarwal af eraniar\n",
      "target elderly people sit on the street playing a game\n",
      "prediction oter ipeble sta otr itan\n",
      "target well remember mother\n",
      "prediction tesoemer memote \n",
      "target shall i compare thee to a summers day\n",
      "prediction shale o peteritat a semmen sta\n",
      "target the township is mostly undeveloped and rural\n",
      "prediction the touncic is miste on devilt ad worl\n",
      "target this is also applicable to circuits\n",
      "prediction this is  also a pittebot setictss \n",
      "target the chicago outfit was now in charge\n",
      "prediction the sicon telol it is nowan chad \n",
      "target they could be sexually humiliated\n",
      "prediction seco pe sictially erin lincet \n",
      "target however his form improved toward the end of the season\n",
      "prediction thewerver is forn i fo turte entif the seson\n",
      "target the supply chain needs experienced developers\n",
      "prediction thes tartion ixs tepase if ti sus \n",
      "target the call is a doglike yip\n",
      "prediction the cor is avadiai \n",
      "target angst seems not organic to me\n",
      "prediction ins sin not ofderni tomet \n",
      "target moderate beach erosion also occurred\n",
      "prediction themoteraf becar wo an alsalcur \n",
      "target but i must tell you that we have another\n",
      "prediction te was toi tiwy aaoter \n",
      "target the supreme court disagreed\n",
      "prediction the sopintlo isecttd \n",
      "target thomas o selfridge in command\n",
      "prediction tomes o solfrirgin te mate \n",
      "target he is still an active pilot and teaches people how to fly\n",
      "prediction hes sti  ected pilit an pitespipen o tefved\n",
      "target no acknowledgement is required from the driver\n",
      "prediction te war no mit is acar om the triters \n",
      "target i just want to go to walmart\n",
      "prediction e jispen desinlala\n",
      "target since then the commune is part of harghita county\n",
      "prediction sfi o reis praof for tidectont\n",
      "target a female singer on stage playing the guitar and singing\n",
      "prediction bat feo siner i stagh plan it eteran sinin\n",
      "target fourche is french meaning fork\n",
      "prediction ffors is frec mining fork\n",
      "target wont stacy be delighted\n",
      "prediction the or ti eleraet \n",
      "target please get me the british journal of dermatology game\n",
      "prediction ts sitn it the preis shael of themmatalla jigaim\n",
      "target the more we listened the more gaily he talked\n",
      "prediction ta wo wy is n the w day he tot \n",
      "target the bureau has had four homes since its inception\n",
      "prediction the vere ws ad forvol  sinti onsections \n",
      "target at that moment the bell rang\n",
      "prediction i ta womit the bire\n",
      "target woman with redhair singing on stage with drummer in the background\n",
      "prediction whaman wit wet er sen on a sta wi comer in the baccran\n",
      "target both men on board died\n",
      "prediction bet men on bor dad\n",
      "target he acted as the secretary to the board\n",
      "prediction he ati as the sixatay to the bol\n",
      "target the refuge has many beds a kitchen and even hot water\n",
      "prediction the refti is manibetts paciction onmui fol walser\n",
      "target it talked to you\n",
      "prediction the tapto ou \n",
      "target he never married but had several affairs\n",
      "prediction the ne e motin th at se wif fants \n",
      "target it lies on the slopes of brandon hill and cullintra hill\n",
      "prediction i clas on thes loic bate hi an colentrai\n",
      "target alan vaughan williams directed\n",
      "prediction an bon wi ins strecten\n",
      "target clarke has written two books to date\n",
      "prediction carcis itnto bok tidats \n",
      "target but hes not in london\n",
      "prediction wat ei or iniotet \n",
      "target two dogs run through a field\n",
      "prediction two dos ncrai woof i\n",
      "target a group of young boys are playing baseball in a foreign country\n",
      "prediction a gi of ou pos arprang ba bal in aforin cantra\n",
      "target sandstone was a popular building material from ancient times\n",
      "prediction sanstor was apupted of the iter of o etiontans\n",
      "target water supply is sourced from lake belmore\n",
      "prediction waer suppa i sorc from la billo\n",
      "target however the love story seems out of place\n",
      "prediction cowever witastoisa arpocs \n",
      "target he gave out abundant largesse and benefactions\n",
      "prediction hedava of bn in worjuss and thet ifactionss \n",
      "target then paul often raked the fire\n",
      "prediction then pa al in wat tefoers \n",
      "target the city is in the town of viroqua\n",
      "prediction the sit is in the to of verolqa \n",
      "target he is one of the farrelly brothers\n",
      "prediction he is wonofser fa aly pales\n",
      "target he died in saint boniface\n",
      "prediction he dor in san one pas \n",
      "target these systems have also typically switched to charging by the minute\n",
      "prediction thee sistins sd tapitric charn a the mini\n",
      "target a barefooted girl is swinging high on a swing on a spring afternoon\n",
      "prediction at ther fitti dirswining pi n aslen in aspreing ftemil\n",
      "target such pieces may be captured as any other\n",
      "prediction suti e te tetaptr as aeither \n",
      "target there also are several hundred chinese\n",
      "prediction there so alsowe o o wi chanis\n",
      "target thomas bangalter was born in paris france\n",
      "prediction tani menlter was born in teri ratss \n",
      "target destruction of private property will not be tolerated\n",
      "prediction histitionalpai popo i iseteeeisi\n",
      "target he also memorised the quran\n",
      "prediction he basa me roris te coont \n",
      "target it is part of the anabar constituency\n",
      "prediction it is paely an a vore contetusya \n",
      "target your cat is dead\n",
      "prediction norapt is dide \n",
      "target he played guitar while visiting holiday camps\n",
      "prediction heprae teterwa disi in po i patss \n",
      "target the zone contained ancient christian catacombs\n",
      "prediction thesol contan anion itian caticols\n",
      "target librarians always cite their sources\n",
      "prediction teraiands o y sa the solses\n",
      "target many countries use this in their flags\n",
      "prediction tene fents islis r fis\n",
      "target it operates scheduled regional services\n",
      "prediction i opera ste e the riginal servesits\n",
      "target youll find many native species of plant in this field\n",
      "prediction the fi eilate stesiitai i ste\n",
      "target make sure you use your indicator before changing lanes\n",
      "prediction teshin a sil in theca of the fortandiplas\n",
      "target many new bulletin boards ran his offer\n",
      "prediction mei ner boa ten bos renis offo \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "target despite the show being picketed the event was transmitted as intended\n",
      "prediction this py te shobincceted he avet wi tras miti asentented\n",
      "target a woman walking with her umbrella\n",
      "prediction a woman walking wit her an boen \n",
      "target were in a cinema and we expect something epic\n",
      "prediction weriasiamol in weispec somting at bic\n",
      "target the crowd scenes were shot using cranes and helicopters\n",
      "prediction the pra sans wer shar is in rins avovems\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "0it [00:00, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "target the bill passed the senate but died in a house committee\n",
      "prediction the bil past the sennet y daed in a hos commity\n",
      "average test loss is 1.5106954778579467\n",
      "Epoch 5 of 150 LR [0.009755282581475767]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "56it [03:14,  3.46s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 205056])\n",
      "tensor([[ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -2.7344e-06,\n",
      "         -7.1339e-06, -7.7367e-05],\n",
      "        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -2.7344e-06,\n",
      "         -7.1339e-06, -7.7367e-05]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "237it [13:47,  3.48s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 270720])\n",
      "tensor([[ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -7.7933e-06,\n",
      "         -1.6287e-05, -5.2683e-05],\n",
      "        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -7.7933e-06,\n",
      "         -1.6287e-05, -5.2683e-05]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "312it [18:11,  3.51s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 200448])\n",
      "tensor([[ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -8.9087e-05,\n",
      "         -5.9199e-05, -5.9370e-05],\n",
      "        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -8.9087e-05,\n",
      "         -5.9199e-05, -5.9370e-05]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "418it [24:22,  3.48s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 180864])\n",
      "tensor([[ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -4.0159e-05,\n",
      "         -3.1214e-05,  3.6880e-06],\n",
      "        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -4.0159e-05,\n",
      "         -3.1214e-05,  3.6880e-06]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "643it [37:30,  3.44s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 200448])\n",
      "tensor([[ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -7.8108e-05,\n",
      "         -1.9141e-05, -7.8119e-06],\n",
      "        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -7.8108e-05,\n",
      "         -1.9141e-05, -7.8119e-06]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "709it [41:23,  3.53s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 131328])\n",
      "tensor([[0.0000, 0.0000, 0.0000,  ..., 0.0004, 0.0005, 0.0005],\n",
      "        [0.0000, 0.0000, 0.0000,  ..., 0.0004, 0.0005, 0.0005]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "927it [54:12,  3.49s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 275328])\n",
      "tensor([[ 0.0000,  0.0000,  0.0000,  ..., -0.0015, -0.0014, -0.0014],\n",
      "        [ 0.0000,  0.0000,  0.0000,  ..., -0.0015, -0.0014, -0.0014]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "1062it [1:02:04,  3.51s/it]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MEAN EPOCH LOSS IS 1.4634145897885051\n",
      "target the town of farmersville is on the south\n",
      "prediction the cow of fammers vi is onthe sese\n",
      "target it is found in brazil peru and bolivia\n",
      "prediction it as sonde vesel arre an bolevera\n",
      "target its all there\n",
      "prediction this at thatseste \n",
      "target he is resleeved from a backup\n",
      "prediction he is reasllyv frommo bakoe\n",
      "target they can also be used to specify temporary speed limits\n",
      "prediction the  also beused i tus hig comprry etemens\n",
      "target some languages have not been identified\n",
      "prediction som mi wges a not tei amafid\n",
      "target the debt has since been paid and the service restored\n",
      "prediction the gor osetin hi fommofover estol\n",
      "target a little girl is running\n",
      "prediction a lote gois fonninhtel \n",
      "target the chronicle consists of two parts\n",
      "prediction the cri fel consis te kwofor\n",
      "target but going home was out of the question\n",
      "prediction the doin hon was at of the qestone\n",
      "target mccanless son immediately rushed into the building\n",
      "prediction wecambis as son am bete  le resten the billinoe\n",
      "target the primary school is on headland road\n",
      "prediction the pimly scool is on headan rrod\n",
      "target similar results were produced\n",
      "prediction sim rsouts were produsedse e\n",
      "target both sides declared their leader as the rebbe\n",
      "prediction e sites dele thi teo as the ebense\n",
      "target especially not americans\n",
      "prediction teve tover me can\n",
      "target the basic engineering environment must be affordable\n",
      "prediction te besicantnern a biman mustoforliblese\n",
      "target we wish matt all the very best\n",
      "prediction wvi mat olto teeusetdet \n",
      "target its twin spires dominate the skyline of the town\n",
      "prediction its twes birs nonmana the sgili of the towne\n",
      "target the region is called kosovo pomoravlje\n",
      "prediction tr ragen is fo the sovel for moroufige\n",
      "target he was defeated in the second round by socialist marius masse\n",
      "prediction he was defete in the setalrol baseciol is man is motssese\n",
      "target her daughters survive her\n",
      "prediction i dagkers svrt\n",
      "target he was killed by a crocodile\n",
      "prediction he was kel by fotedatste \n",
      "target nip round to the garage at once\n",
      "prediction we bron to the grova at wondse \n",
      "target it came from the heart\n",
      "prediction it cam from be outlt \n",
      "target the great central railway operated a number of ships\n",
      "prediction the grit fentral realway oprated a nober of ships\n",
      "target verbal agreements are not binding\n",
      "prediction bet oro ge an nonot bidin \n",
      "target most of the songs were written by graham day\n",
      "prediction wal of the sos flor riten bi gran da\n",
      "target he released several records on yo mama\n",
      "prediction he deleed serenv ors on you wonm\n",
      "target these are fastneutron reactors\n",
      "prediction thes offeste fom le ectesee\n",
      "target the original sugar busters\n",
      "prediction fote wige no fodebastes\n",
      "target he also grew flowers and vegetables on his farm\n",
      "prediction the also gre flost avisial foms fom\n",
      "target moss played college football at the university of miami\n",
      "prediction mostfle coa fit bol o the universty miameoesese\n",
      "target the shamrocks announced that they would abandon the league\n",
      "prediction the she os in ous the tha wi a badan the lege\n",
      "target this was the first processor implemented in this process\n",
      "prediction this wis the firt proscesser inplemante in is proses\n",
      "target it isnt poetry is it interposed his father\n",
      "prediction it is an poeter ust n topos es fotherose e\n",
      "target the zone contained ancient christian catacombs\n",
      "prediction theszo contan antial fitian caticold\n",
      "target i shall like it of all things\n",
      "prediction i shan likit avall fansee\n",
      "target the legal process of resolving a dispute\n",
      "prediction thebe oup prosses on rhesonbin can disgut\n",
      "target stine was born in stillwater oklahoma\n",
      "prediction stin was born in stil water olcle olmaseleo \n",
      "target these impressive results however may hide a murkier reality\n",
      "prediction the se im freste reselns i vte ma hi them metareoncyd\n",
      "target he then spent two years travelling in the east\n",
      "prediction he vensben twoyears trovininte vese\n",
      "target the volcano has a large wide caldera\n",
      "prediction the vokanow has la logs wi fo derra\n",
      "target for this he was arrested by the federal authorities and imprisoned\n",
      "prediction obes e is aresed bo the fater aftotesumpisen\n",
      "target bless my soul alive\n",
      "prediction tles wi so aliee\n",
      "target his son maurice hutcheson succeeded him\n",
      "prediction hi son merys hadche som siceted himpsese\n",
      "target hes probably working somewhere\n",
      "prediction his provabaly boking sommorlt \n",
      "target two darkhaired girls are on a tire swing\n",
      "prediction two dorgker vrils are on a tire foing\n",
      "target the motors of the new rakes also make less noise than the older ones\n",
      "prediction the moders of the new reaks also males nos tin te ilverwons\n",
      "target the ranking is part of the story the best cities for business\n",
      "prediction the rinkin is po te story the ba sittys wo vhis messdse\n",
      "target news articles updated daily\n",
      "prediction wese iticlels o tdated dalylsle \n",
      "target here comes liz back again\n",
      "prediction herecom rors backe gae\n",
      "target thats one of robs main flaws\n",
      "prediction the wan f was men fo son\n",
      "target how can she do so\n",
      "prediction opi sei dussode\n",
      "target please find me the home at last trailer\n",
      "prediction hs fin tefom at los tate\n",
      "target a celebration in king arthurs court follows\n",
      "prediction a sote bi son in to outes scoo follows\n",
      "target then he took interest in his farm\n",
      "prediction famd he to infrist in vhis forse\n",
      "target allmusic gave the album a three star rating\n",
      "prediction oles ic thefe album af teree tar radingse\n",
      "target and ospreys\n",
      "prediction an os brinslt \n",
      "target he made his first recordings for remington\n",
      "prediction he matis faster codings forreinton\n",
      "target its county seat is oakland\n",
      "prediction this cotise is oe\n",
      "target it is composed of the seven neighbourhoods surrounding high park\n",
      "prediction it is compoed af the sedeabehds terrounding hig park\n",
      "target scared they run to craigs room\n",
      "prediction skard the rando crate timte \n",
      "target my dear i think so now\n",
      "prediction et mter has insil nole\n",
      "target the strength of iron is determined by its carbon content\n",
      "prediction the stan tef aern of the tormen bi its garben contand\n",
      "target we need to get the bottom of this\n",
      "prediction we e toget to the bi movise\n",
      "target it is less rounded than other horned lizards\n",
      "prediction it is  mest rerndev then ther horint lusitssese\n",
      "target he was born in barbastro\n",
      "prediction he was born in byverstee e\n",
      "target today i wouldnt get an academic job\n",
      "prediction to da a win gine ati amit jor\n",
      "target the accident effectively ended her modeling career\n",
      "prediction tatslen hfetoly ended a moly gre\n",
      "target this was also announced as a tricorder\n",
      "prediction this lo a la na ffer soforder\n",
      "target so with the bearded bloke\n",
      "prediction som te be te bliket \n",
      "target the ashley children were all roped into the business\n",
      "prediction the ash metin ven werar ropd din to t visnusss\n",
      "target he advocated extending slavery into california\n",
      "prediction he a the cande te xtanding fli fre in tocoo\n",
      "target it would only be like you\n",
      "prediction it wa orlete lit soe\n",
      "target a group of bystanders view the eiffel tower from a balcony\n",
      "prediction the ver wo vied bader teo te ite fa fon te vati\n",
      "target the area is covered by strict planning controls\n",
      "prediction he rre av his fpote fistrict pamin te fos\n",
      "target two young men playing instruments\n",
      "prediction two young mam fliin fanstomentel \n",
      "target very little new paving is done and the entire profession is at risk\n",
      "prediction ear evaal nopating was dounen an ti prefetion is thagres\n",
      "target he took a bite\n",
      "prediction it took a bontlt \n",
      "target rock is in this summer\n",
      "prediction w is own a solore e\n",
      "target the change to the order of succession was widely accepted\n",
      "prediction the chageed to bhe rter succescions wi ly au soppe\n",
      "target keep learning and youll always be young\n",
      "prediction he epordin  ano owis be ardne\n",
      "target certainly not said alice indignantly\n",
      "prediction sortey no sar oter sete sefise\n",
      "target he continued to work in other musical fields\n",
      "prediction it ecgee to borkin other rusiga filsoltlses \n",
      "target he made his first on screen appearance in september\n",
      "prediction he mae is first ont gren of perencs in sof comber\n",
      "target it is nonmagnetic\n",
      "prediction it is no lat netit \n",
      "target the starship nexus is our flagship\n",
      "prediction the starshi neces is a flathipse\n",
      "target sheridan was born in newark delaware\n",
      "prediction serde was born in meer tolwor\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "target jennings is a supporter of chelsea football club\n",
      "prediction jendings is a soqorter of chse fo bo clobe\n",
      "target tennessee williams because of his delicacy\n",
      "prediction tcennity ilams the casethis sdolo tisesese\n",
      "target people running or walking on a field\n",
      "prediction eople roning a walkng on a faildt \n",
      "target the crashed meteor left a gigantic crater on the ground\n",
      "prediction the crestme toon of the jegenti creter on the grondtloe \n",
      "target to tell the truth  he had forgotten them\n",
      "prediction the ferman tot e a foeedn\n",
      "target he has it\n",
      "prediction h has ite \n",
      "target the springdale free public library serves the borough\n",
      "prediction thi brodel fee poli iven sevs te bar\n",
      "target they carried out salt manufacture\n",
      "prediction tpaka  as so lne fatfor\n",
      "target the numerals glow amber\n",
      "prediction te neeros colaamberltelt \n",
      "target he was forbidden any further writing\n",
      "prediction he was frorpidin meiny forta ritdinge\n",
      "target the centre also runs a squash league\n",
      "prediction the senter also run as wisled\n",
      "target give me an example\n",
      "prediction dv me i xonblee\n",
      "target it has good capacity to recover from frost injury\n",
      "prediction the hals bo pempos teato e coloivel fros endry\n",
      "target i certainly will not\n",
      "prediction i serly bo nouttelt \n",
      "target originally a ballerina she found success as an actress\n",
      "prediction osion labearri naun she fan ifast i nentessese\n",
      "target it detonates upon contact with anything\n",
      "prediction tit dem mats fon contefo initin\n",
      "target it has the appearance of a reddishblue powder\n",
      "prediction his hast tot herins ofte lavi ne holter\n",
      "target wight led the raid on millport\n",
      "prediction w wa te bat on do orte\n",
      "target they are endocentric environments\n",
      "prediction ther at andbosen tret and vommintse\n",
      "target urry managed to overcome this problem by using powdered zinc\n",
      "prediction ere maniste of er comdis prou bi s in phater sinkse \n",
      "target but i didnt\n",
      "prediction butt i divene \n",
      "target the community lies in the eastern vogelsberg mountains\n",
      "prediction the cmeity lis ne eased ten folbs ber monthing\n",
      "target each side accused the other of provoking the violence\n",
      "prediction he sat fused the ater of folol fom the rinnits\n",
      "target later rubidium and cesium beam clocks were used\n",
      "prediction wver r biiom and ses om bel flos fenesed\n",
      "target i also tried my best\n",
      "prediction i also tri mavestte \n",
      "torch.Size([2, 198144])\n",
      "tensor([[0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 4.8324e-05, 7.6901e-05,\n",
      "         3.2309e-05],\n",
      "        [0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 4.8324e-05, 7.6901e-05,\n",
      "         3.2309e-05]])\n",
      "target he has a good sense of humor so hes able to defuse things\n",
      "prediction he dhas a brit set o hemen sovis a ebo ga fesed dains\n",
      "target the other cases are similar\n",
      "prediction he  ever teses asellmaste e \n",
      "target a skeleton crew remained aboard\n",
      "prediction ascirl en gre rema a bodle \n",
      "target we want it serving the local community\n",
      "prediction e wonte seivin  locoton munitye\n",
      "target i had been chosen to get a message across\n",
      "prediction  ha ben shosen togaen mefettho cops\n",
      "target tourists visiting a landmark taking pictures\n",
      "prediction tourists visiting alan mark titin piters\n",
      "target the eastern terminus is at the lewis and clark bridge\n",
      "prediction the estren temmins he sate leis and clack grich\n",
      "target stephen had him thrown with his horse off a cliff\n",
      "prediction stifen hardim tom rit his ors ol fite\n",
      "target its cover was the british passport control office\n",
      "prediction his covens a braed bo borpe jolloffese e\n",
      "target during his reign the suevic nation became fragmented\n",
      "prediction mager i liin lost tevi me shir the cam fi lon\n",
      "target it is used extensively for weddings\n",
      "prediction it  s used xteso liver wedings\n",
      "target she remarried wyndham strover\n",
      "prediction sheremari windm tovere \n",
      "target a man washing windows on a red brick building\n",
      "prediction a man wotion winders on ha redt frick fidn\n",
      "target the village recovered through migration from surrounding areas\n",
      "prediction the rigin cowr to miger atols a onding nenes\n",
      "target to prevent the train rolling back the brakes were applied\n",
      "prediction to preren the frin rortin bat the briks wor plid\n",
      "target she is the youngest of three children\n",
      "prediction shas the youngest of tre comdenale \n",
      "target this is done by collecting radioactive signs\n",
      "prediction this is don by telecting radyactov sins\n",
      "target this had a high rate of complications\n",
      "prediction this veat i bre at copo plationsse\n",
      "target a toddler catches droplets of water as he sits in a shallow pool\n",
      "prediction the tobin rages drople o wo a as in sittin a fholli po\n",
      "target two young kids are in a hut\n",
      "prediction tto yo kis ar inahte\n",
      "target the new shop flourished\n",
      "prediction te sha  los\n",
      "target find the tv series joy of a toy\n",
      "prediction inmat teb seis jia overto\n",
      "target but guimard is guimard\n",
      "prediction bot de marr is gemomere\n",
      "target murder is very rude\n",
      "prediction marver is bary rodet \n",
      "target he worked for the ardwick and ancoats dispensary\n",
      "prediction e go t o with ta man co isbensye \n",
      "target bullets can be stopped by\n",
      "prediction bod cam be stot byte \n",
      "target so ill go halfway to meet them\n",
      "prediction so ate harf wa two metemeoe\n",
      "target i wish to listen to some fifties music\n",
      "prediction wa lesed talesion fisel fi is nei\n",
      "target this island is a large dormant volcano\n",
      "prediction the silum as a large doman flkinalse e\n",
      "target for the ideas that i had i needed to make that move\n",
      "prediction forde a a the e mar a nete te me that no\n",
      "target the convention called on carranza to resign\n",
      "prediction the comvention colbon corans ater resin\n",
      "target the available fourcylinder engines are the same as the focus\n",
      "prediction the vable folson de engers of the sam is e fokessesese\n",
      "target a football player points\n",
      "prediction wat fo ve a fo \n",
      "target ruby chadwick is rather sentimental\n",
      "prediction rebe chadwa is rover sonto mntillt \n",
      "target teenagers are dating\n",
      "prediction to nagers ar datine e\n",
      "target playa grande has an airport\n",
      "prediction tiar groad has  amporsese\n",
      "target produced by sandy roberton\n",
      "prediction podost by sady rovetemlostset \n",
      "target it is high and in diameter\n",
      "prediction the i o ad in beainfile \n",
      "target it was followed by these days tour\n",
      "prediction it was bor by thesederis toelse \n",
      "target the motion was voted down in the lower house\n",
      "prediction the motian was worded don in teloolv\n",
      "target as a result chautauqua was forced to add a new fleet type\n",
      "prediction avesal chosiborf was bost a ave feafosese\n",
      "target the park is home to diverse flora and fauna\n",
      "prediction the partis hern two davers floro in forna\n",
      "target he left his post in december\n",
      "prediction he a tes forsdban the femderse \n",
      "target the economic model has been reworked\n",
      "prediction the eennoic moal has beenry work\n",
      "target it is often seen in towns in gardens or vacant lots\n",
      "prediction he is often seen touns in gardin lovekan lot\n",
      "target he stared morosely at the fire\n",
      "prediction his tare ma rose a the fir\n",
      "target when it appears in print people believe it\n",
      "prediction tenti apeas iti epe eiity\n",
      "target kemp hurried upstairs and rapped eagerly\n",
      "prediction tcam phare u str and roup edurlye\n",
      "target its county seat is amarillo\n",
      "prediction its county feat is mmoriate \n",
      "target hey can you please slow down\n",
      "prediction t peve loodtlteo \n",
      "target it is a remixed version of the song\n",
      "prediction it is ale mix firs nof the sonnse e\n",
      "target pauls come she exclaimed\n",
      "prediction has con shexflndolslses \n",
      "target in total several hundred thousand votes were cast\n",
      "prediction in totle sevel fundert fusan vot wer cast\n",
      "target i dont know why\n",
      "prediction i do no wi\n",
      "target neutron blankets deuterium and tritium fusion generates neutrons\n",
      "prediction the fon langets the aramman fi ten fisin inrits betros\n",
      "target no i dont\n",
      "prediction e now i doe\n",
      "target he narrated the initial teaser trailer for the game\n",
      "prediction o naras of the inisal ses i traer for the game\n",
      "target the match was decided via a penalty shoot out\n",
      "prediction the mas was d sitede ve of lty shotoe\n",
      "target a man and woman sit on a bench\n",
      "prediction a man an woman sit on a banc\n",
      "target they roost communally in loose groups of a hundred or more birds\n",
      "prediction they wossd tomeely inlo scrops of te hungete on bords\n",
      "target what are them movie times\n",
      "prediction what o bheand me ve timse\n",
      "target turner and the subject of history\n",
      "prediction toner and the suter to histreltheo \n",
      "target this was an opportunity for new talent to emerge\n",
      "prediction this was s nobecanaty finne toan t maglse e\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "target each species has a distinct number of rings\n",
      "prediction eac sapesese a desten the mer rins\n",
      "target a woman in a blue jacket is drinking tea from a cup\n",
      "prediction a woman in e ble jackit is dentking tefrom acotste \n",
      "target his father was a professor of greek\n",
      "prediction his father was befesser of greke\n",
      "target with his death the rebellion ceased\n",
      "prediction wath his da the rebol inceasedse\n",
      "target it is still in production\n",
      "prediction it s tan an bodivtione\n",
      "target it is the county seat of stephens county\n",
      "prediction it is the counti sea of  sterren condy\n",
      "target the do not resuscitate order was withdrawn\n",
      "prediction the  do not reesa ta oter was with fron\n",
      "target for legal scholars several issues are important\n",
      "prediction orevas cal as sirvelie sos ont fortents\n",
      "target they are also equipped with beach couches\n",
      "prediction they are also a grop of be couteslteo \n",
      "target a man lying on the grass next to a small lake\n",
      "prediction a man win o e taes sis momae\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "0it [00:00, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "target i am growing wary\n",
      "prediction wi m groing wanryt \n",
      "average test loss is 1.3742885117862313\n",
      "Epoch 6 of 150 LR [0.009648882429441256]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "35it [02:02,  3.49s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 200448])\n",
      "tensor([[ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -8.9087e-05,\n",
      "         -5.9199e-05, -5.9370e-05],\n",
      "        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -8.9087e-05,\n",
      "         -5.9199e-05, -5.9370e-05]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "240it [13:57,  3.48s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 131328])\n",
      "tensor([[0.0000, 0.0000, 0.0000,  ..., 0.0004, 0.0005, 0.0005],\n",
      "        [0.0000, 0.0000, 0.0000,  ..., 0.0004, 0.0005, 0.0005]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "250it [14:32,  3.47s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 205056])\n",
      "tensor([[ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -2.7344e-06,\n",
      "         -7.1339e-06, -7.7367e-05],\n",
      "        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -2.7344e-06,\n",
      "         -7.1339e-06, -7.7367e-05]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "344it [20:01,  3.49s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 275328])\n",
      "tensor([[ 0.0000,  0.0000,  0.0000,  ..., -0.0015, -0.0014, -0.0014],\n",
      "        [ 0.0000,  0.0000,  0.0000,  ..., -0.0015, -0.0014, -0.0014]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "801it [46:49,  4.08s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 200448])\n",
      "tensor([[ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -7.8108e-05,\n",
      "         -1.9141e-05, -7.8119e-06],\n",
      "        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -7.8108e-05,\n",
      "         -1.9141e-05, -7.8119e-06]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "984it [57:29,  3.49s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 180864])\n",
      "tensor([[ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -4.0159e-05,\n",
      "         -3.1214e-05,  3.6880e-06],\n",
      "        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -4.0159e-05,\n",
      "         -3.1214e-05,  3.6880e-06]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "1000it [58:25,  3.47s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 270720])\n",
      "tensor([[ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -7.7933e-06,\n",
      "         -1.6287e-05, -5.2683e-05],\n",
      "        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -7.7933e-06,\n",
      "         -1.6287e-05, -5.2683e-05]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "1062it [1:02:05,  3.51s/it]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MEAN EPOCH LOSS IS 1.3780443376515994\n",
      "target his hero being randy macho man savage\n",
      "prediction is her  ban ndy mochurman sha age\n",
      "target he died when the pourquoipas\n",
      "prediction he died fen  bash\n",
      "target achilles saves her\n",
      "prediction i cite siy ca\n",
      "target the production received both critical and popular acclaim\n",
      "prediction the erdtion is ca for fritetanand ocoactan\n",
      "target if a score line is present the knife is safe\n",
      "prediction it escr onas preset ten natecaf\n",
      "target you wont have a lawyer\n",
      "prediction ye  havmen o\n",
      "target it has also been established in the wild in colombia\n",
      "prediction it is also enistavish tom the faun o comobee\n",
      "target it is located in the vincennes historic district\n",
      "prediction   setea atataa\n",
      "target this file system has gone through ten versions\n",
      "prediction this ficistron has con to genversans\n",
      "target garforth has two railway stations\n",
      "prediction arsorhav to rran witaso\n",
      "target the community was built in four quadrants\n",
      "prediction the comminitem was bil in fol portrentse\n",
      "target you remember that time at the girls school\n",
      "prediction o e memven hat timat to gro schoo\n",
      "target the website offered adoptions and discussions\n",
      "prediction the resine noferaductions in discussions\n",
      "target pollen is collected by solitary bees\n",
      "prediction oman is colactor by sonacaryle\n",
      "target many others succumbed to shock or hypothermia\n",
      "prediction man of the sucon sha hadt fanie\n",
      "target it formerly included queensbury\n",
      "prediction i fame in fui fe\n",
      "target and theres not such another\n",
      "prediction an tein ouv setshian ouserheltbtbtete\n",
      "target the cat only grinned when it saw alice\n",
      "prediction he caet on recrue frie son ali\n",
      "target their first child melissa coleman was born there the next year\n",
      "prediction the fir cho belese camen wis born deirn an neteir\n",
      "target its then found out that its like recording while hitting the rewind button\n",
      "prediction is in tou owt tot isi procord in moti in terinin fadn\n",
      "target she returned to skating two years later\n",
      "prediction she rerturn t tuscatt toers later\n",
      "target it was created by aran rei\n",
      "prediction it mis created batter an  ra\n",
      "target he wrote four songs in this film\n",
      "prediction he rould four sons in thiscoun\n",
      "target many families go to church on sunday mornings\n",
      "prediction mmani fhan is gor te chur on sennen morning\n",
      "target dogs run out of the water toward a rocky beach\n",
      "prediction nnsen oun ta ten waten san an attede\n",
      "target eimas was born in bridgeport connecticut\n",
      "prediction ammes was born in brich park commedticerhe\n",
      "target services were to wrexham and seacombe\n",
      "prediction shete sis wer todection an setand\n",
      "target congressman from north carolina\n",
      "prediction gronershon from ma caryanna\n",
      "target she was born in germany\n",
      "prediction she was born in surmanyee\n",
      "target the school is a specialist visual and performing arts college\n",
      "prediction te cuone is spisin a fisin an poforin ascunne\n",
      "target mann was born in yorkshire\n",
      "prediction man wa fa in ashra\n",
      "target he remains a hero to the aro people\n",
      "prediction eman an arant to the arem pece\n",
      "target no paper money was issued\n",
      "prediction a fo moin efia\n",
      "target what more could you want\n",
      "prediction wat hear ten sou mat\n",
      "target it is not a science\n",
      "prediction it as in nonced fa shirans\n",
      "target the first to win two sets wins the match\n",
      "prediction a fast in trocets ans coatt\n",
      "target andrews was born in hinsdale new hampshire\n",
      "prediction anders was bor an han tern an housha\n",
      "target also along the way he established supply points and an airfield\n",
      "prediction affoa on doa te as tapish so pliflas andanarfim\n",
      "target a man wearing a safety harness is walking on a tight wire across a ravine\n",
      "prediction a man waring a safe oone is walking an a ci rie frosso revenn\n",
      "target a man wearing no shirt climbs a steep cliff\n",
      "prediction a man en noshat canc w sted trole\n",
      "target i think this may be easily arranged\n",
      "prediction i cut es ma be u fory arrangd\n",
      "target these are still manufactured by black diamond equipment\n",
      "prediction es as co man efenctud buglat cmmanqrimen\n",
      "target at what times will animated movies be showing nearby\n",
      "prediction afo ti o anamento bo bife shain ara\n",
      "target the level ends when all the enemies are defeated\n",
      "prediction laer aun wen o bunerlys ar cafeti\n",
      "target thats where the gospel ends\n",
      "prediction at fe  de vu ead\n",
      "target man cutting fish in a fish market\n",
      "prediction an catting fision o fishmarkec\n",
      "target a group of children playing a game outside\n",
      "prediction a group of shildran pranan gan outfid\n",
      "target bullets can be stopped by\n",
      "prediction oitt can be stout bi\n",
      "target only us explorers can get it\n",
      "prediction one ar eta fron tis\n",
      "target a brown dog runs with a toy in its mouth\n",
      "prediction a bron dot ran with te twoin is mnoufh\n",
      "target you have opted to receive biannual statements\n",
      "prediction e hav affi suiuvi anoefefin\n",
      "target you were rotten today\n",
      "prediction you  otin today\n",
      "target aurangabad is the headquarters and principal city\n",
      "prediction ar an abute the hadcartr and o prencical city\n",
      "target it has no antibiotic properties\n",
      "prediction it has nr hantuvone fofe\n",
      "target australia won all matches comfortably\n",
      "prediction a straon be oun an wal accus confrir to lay\n",
      "target owen county is to the south across the creek\n",
      "prediction homen coufes t fisount a frorst ofref\n",
      "target he has only just come home\n",
      "prediction heas collage counfoud\n",
      "target despite this the game is quite challenging\n",
      "prediction is pra i f tegans con chongi\n",
      "target each carried a lantern and long stave\n",
      "prediction e carde an antern andnn sce\n",
      "target many are liberal arts colleges\n",
      "prediction mani alid pro at cotee\n",
      "target senator saxby chambliss\n",
      "prediction handader sac be chanlli\n",
      "target it never occurred to me\n",
      "prediction itnever acantony\n",
      "target it is unclear if he died on the gibbet\n",
      "prediction it is unclure egied on begitec\n",
      "target the parallel course did not take hold however until the next century\n",
      "prediction the patnent cors onartec won cober an tof nexcentrury\n",
      "target another new person in her class is davy\n",
      "prediction an attan you parcin arcasis tady\n",
      "target many people seem to be taking a picture\n",
      "prediction man y people sim to be te ing a ecthur\n",
      "target he died in winterthur\n",
      "prediction it died in etcerer\n",
      "target she really is a marvel\n",
      "prediction sherrenly is ammar\n",
      "target the peasants were then quickly overwhelmed\n",
      "prediction the pasens wee anacricky of reoundote e e\n",
      "target it is a type of pyroxenite\n",
      "prediction i is ata o parcsonoung\n",
      "target the reason for buchans appointment was a mystery\n",
      "prediction therater for do cenach apanat was anetory\n",
      "target our record label said we had to finish the song and record it\n",
      "prediction abe cota aso w hav to finsh o sonaneqardect\n",
      "target he was buried in a tomb along the via appia\n",
      "prediction it s barid tan a cum oa tefea ape\n",
      "target truth is green bitter\n",
      "prediction troch is gren bedac\n",
      "target she received the life\n",
      "prediction he recuved e larh\n",
      "target the hundredweight has had many different values\n",
      "prediction ehendrer wa ha shad an e difren foues\n",
      "target the first artist associated with the group was george knapton\n",
      "prediction the first arstecessio e it the gret as jurach onnckon\n",
      "target this welding method was used to construct the pioneer zephyr\n",
      "prediction the oming eted wa susd to constrec to by anerseffer\n",
      "target reviews for the album are mixed\n",
      "prediction e efused for the auem fomit\n",
      "torch.Size([2, 198144])\n",
      "tensor([[0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 4.8324e-05, 7.6901e-05,\n",
      "         3.2309e-05],\n",
      "        [0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 4.8324e-05, 7.6901e-05,\n",
      "         3.2309e-05]])\n",
      "target maronga was a municipal official in libreville\n",
      "prediction erauntot was ununi por offision an giverohan\n",
      "target she ate some food and drank a glass of champagne\n",
      "prediction she aint som fer cantracan laofca en\n",
      "target the beach is right around the corner\n",
      "prediction the beaches rracaraun the cone\n",
      "target certificates comprised two components\n",
      "prediction he ditegat  comprarved to cobotent\n",
      "target she prefers cricket over rounders\n",
      "prediction sshi fofars crickc am eaunders\n",
      "target my husband is on his way\n",
      "prediction  ha trnt is a i forf\n",
      "target as a result the current grass field drains very well\n",
      "prediction is a esolt te curent crafil drain to vearywel\n",
      "target goodman died following a heart attack\n",
      "prediction au men da fo a n a hackat t\n",
      "target nuevo tiempo is part of the hope channel network\n",
      "prediction to en wa feanfo is fart of tehic hann meforg\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "target these animals should be housed alone because they are highly territorial\n",
      "prediction he can inoso fe hau aman soce a ha i srefiar\n",
      "target im so happy\n",
      "prediction in sav hapfi\n",
      "target please search for the thrilling cities photograph\n",
      "prediction a ser for the foring sudyis fartagraf\n",
      "target arkose is typically grey to reddish in colour\n",
      "prediction arcoces iffecytra outihan cholar\n",
      "target shabazz was born and raised in the bronx\n",
      "prediction shaa was born and vrased in he pran\n",
      "target the event developed from the ancient pentathlon\n",
      "prediction   han tdo nat a trornteention cantafon\n",
      "target a man behind either a fence or cage\n",
      "prediction a man in ben is aunt a n art to\n",
      "target it is the second rama game to be produced\n",
      "prediction it is a seco a romenantoe frodu\n",
      "target it consists of the village of ennetbaden which is a suburb of baden\n",
      "prediction it consessto the dillagof anaga an wic es a sunder o fa\n",
      "target however it also removes voice dialogue\n",
      "prediction herenacat a srreis to e stiouc\n",
      "target the location has since been lost\n",
      "prediction the ecetion has so sfren la\n",
      "target it is named for schuyler colfax\n",
      "prediction it is nan fe cila onfes\n",
      "target its county seats are plymouth and brockton\n",
      "prediction its county sint a pane an frockten\n",
      "target no further shows have been announced\n",
      "prediction no ferter shos ofenana\n",
      "target this episcopal division was the poorest in brittany\n",
      "prediction thiss cofiscupel defision wus te forest an gitten\n",
      "target none were found\n",
      "prediction nownman wat fama\n",
      "target remittance plays a major role in economy of the area\n",
      "prediction emendics clays an manjr wal in aconta of theara\n",
      "target each one is headed by a vice president\n",
      "prediction a tan is eddent b a esidint\n",
      "target i wish youd lend me half a crown\n",
      "prediction im wa sho autme ato conhlhlt \n",
      "target the place was excavated the same year\n",
      "prediction thepat was exconventeen the san year\n",
      "target the schools chief rival is aurora high school\n",
      "prediction hes cou shufried of is oora hig scoole\n",
      "target the veliduans language is dhivehi\n",
      "prediction ceden the jur anthe fren wento is coventye\n",
      "target see filipino alphabet\n",
      "prediction se cemapeno oufonect\n",
      "target the effects on the ice sheet are visible\n",
      "prediction fa fonte ishi andtiadenhe\n",
      "target male with goggles sitting on the floor with a pamphlet\n",
      "prediction nal wit gargle fatt an a four wit hispampulet\n",
      "target a team runs down the field doing some drills\n",
      "prediction a thin an stounte feou ge in sindru\n",
      "target its probably the first thing theyll think of\n",
      "prediction ipormanter t fer tam nan in crof\n",
      "target this roommate was dora bloom\n",
      "prediction tesul mate was doapbe\n",
      "target all of the team members were americans\n",
      "prediction ar ofte ten mem wu efort fommettefon\n",
      "target tonya survived the shooting\n",
      "prediction tuny ou sovit tishin thinhlhtlte e e e\n",
      "target who would believe my tale\n",
      "prediction fore ver mea\n",
      "target in antiquity it was known as catabathmus magnus\n",
      "prediction ean te retet wes noa can a batmes madii\n",
      "target horses dogs and drink had brought him there pellmell\n",
      "prediction corfi dox an trint at brotin the omow\n",
      "target financial success then follows from this\n",
      "prediction an an shal siuccessh to en follo frenis\n",
      "target anand had a controversial tenure as chief justice of india\n",
      "prediction aond fhat a contruracial cenna as  shut jutirsamarnna\n",
      "target his title gave a huge boost to chess in the netherlands\n",
      "prediction is cat on ca asut fus ches on anetterlans\n",
      "target what would you say about that\n",
      "prediction wit wan you saar bot ta\n",
      "target his brothers name is larry and he lived in staten island\n",
      "prediction potes nam is abary an te  frotuh standin na\n",
      "target a blondhaired girl sitting on a bed\n",
      "prediction a frooun tat a go shien on a fet\n",
      "target the name walter the tenth\n",
      "prediction the nan water the tench\n",
      "target walker attended great cornard upper school\n",
      "prediction boper tnangra cor ont affer sco\n",
      "target it features the influential title track and hiphop single the message\n",
      "prediction i fanchas te intementiolt otrat on hifocenel fameci\n",
      "target grady is located about south of montgomery\n",
      "prediction graandi is locatted a bout sou ta mongonverye\n",
      "target the majority voted to reject all changes\n",
      "prediction ha major y vou a tur regec tola changene\n",
      "target his parents were from oklahoma\n",
      "prediction his bar en warsrom o cahaver\n",
      "target moltke was forced to leave the military service\n",
      "prediction mo fe was folstely then minatre sherris\n",
      "target this is the oldest on the high plains\n",
      "prediction tis is the otes ondei blanse\n",
      "target however the elevator will eventually run out of memory\n",
      "prediction howaver tou agatter we laventuly rn at a mendera\n",
      "target young children stand outside on a dirt surface\n",
      "prediction yn chiltems tund on sied on e durt sonfec\n",
      "target i suppose they are he mused\n",
      "prediction i shap a  ta  emanister\n",
      "target at election time the tavern served as a voting place\n",
      "prediction alection fiden a tomen sat is abumtin flac\n",
      "target davis lawfers was born in rochester new york\n",
      "prediction i sal fracor s for in eocestndto\n",
      "target the land was donated to the jesuits\n",
      "prediction on lan wast onnated to encsse\n",
      "target he was equipped with a pace maker and made a full recovery\n",
      "prediction he was eqre trehacpas macar ana mata for recarmerry\n",
      "target i thought they were fine\n",
      "prediction i ca dol er i\n",
      "target carpenter is an atheist\n",
      "prediction cartan teri focafif\n",
      "target this was one of the reasons the cortina wars happened\n",
      "prediction this is bon f torese af the courtina wors hacpend\n",
      "target we worked around the clock\n",
      "prediction me brat oun a clak\n",
      "target the banner now on display is a modern reproduction\n",
      "prediction tha bon an ow on diplani so moten tro preduction\n",
      "target chronology of ant colony optimization algorithms\n",
      "prediction fraagy o ant cony hatoadaional bertens\n",
      "target it was a suburban riding in edmonton\n",
      "prediction it was he sepanberin grarding and admenten\n",
      "target android is the most used mobile operating system\n",
      "prediction hendera is te mifused inmobi ffer in systen\n",
      "target the blood clot had broken apart and she was clear of danger\n",
      "prediction the bla cla had burgan api fon shuas claer tange\n",
      "target these included headgear though this was not usually the primary element\n",
      "prediction es incu ed had ger to otish was not nusaly de pramer anument\n",
      "target the season ended with the clubs first overseas tour\n",
      "prediction he ses er ended wit the coferst a fcisto\n",
      "target many of their performances are for charity\n",
      "prediction many haf their performent is ar forcharty\n",
      "target keep your phone handy in case i text you with subway updates\n",
      "prediction two per ohander on ja fete so efeaanter\n",
      "target my friend said i love you\n",
      "prediction i fen sad i ot \n",
      "target he also won west coast young farmer of the year\n",
      "prediction asa ar is ca o un faner ter\n",
      "target he was a recipient of padma shri\n",
      "prediction she wa sher  shigend fo fattenashran\n",
      "target it includes all african countries\n",
      "prediction i incluts all acfrgin counturieaesele\n",
      "target it is also known as the small egyptian gerbil\n",
      "prediction it is a te noen ateemaro tetriction e erclel\n",
      "target he last played for warrington town\n",
      "prediction a grat forworts inth ta\n",
      "target a collection of short stories\n",
      "prediction i crocan a so do\n",
      "target doctor karen hamilton\n",
      "prediction dactr car and haualtamelebe\n",
      "target it reproduces by seed\n",
      "prediction i ree pordues by she\n",
      "target it was the premier bank of kerala\n",
      "prediction it was te prennuar beng fof curuee\n",
      "target he is a historian by training\n",
      "prediction ean an sae storita an fia srue\n",
      "target ben thuy is near the city of vinh\n",
      "prediction en toren is er the cityo pan\n",
      "target prior to then it was part of the suburb of campbelltown\n",
      "prediction parcagan i was par of the suver of cable towne\n",
      "target they first were in a group known as chill deal\n",
      "prediction pa fersere inagrt nownn a shal de\n",
      "target the largest of the group is yamdena\n",
      "prediction a maxist o fo go e youn dana\n",
      "target the general condition of the rural roads was poor\n",
      "prediction fechenner condition affoferan raris i faf\n",
      "target the modern term for this meaning is ultrasonic\n",
      "prediction he matden turn troah mening is oartra sarnic\n",
      "target hybrids are known\n",
      "prediction hibrek on non\n",
      "target the wainscoting has painted figures\n",
      "prediction teworin scoa as panter frino\n",
      "target its county seat is grundy\n",
      "prediction its counfe ofs crandy\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "target china began to reach its height\n",
      "prediction hanam bexounturrat ot higt\n",
      "target he has worked as a journalist and author\n",
      "prediction he has bri tos a ganais tonoha\n",
      "target clubs for cricket and football have junior sections\n",
      "prediction a aean  taosae\n",
      "target he commissioned the construction of aston hall soon after\n",
      "prediction hecremision the contructiona ha on hal shoufter\n",
      "target i know you are surprised\n",
      "prediction i nowy yar su praced\n",
      "target more recently he has written on love and caring\n",
      "prediction morres ola as wen on loven tern\n",
      "target then ill stay still\n",
      "prediction an n sta sci\n",
      "target crete was once a contender for county seat\n",
      "prediction ra was on e contender for county sek\n",
      "target ranew was born in albany georgia\n",
      "prediction rde asfrorn e albaefshurceehe\n",
      "target a soldier is holding a gun and crouched down with is mouth covered by a cloth\n",
      "prediction a shouder s hopina doun an crotc don wit is mo cono bafo\n",
      "target the two daleks inside are destroyed\n",
      "prediction ta to dollicc ansid er destrod\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "0it [00:00, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "target stillman is married to the former mara stefanski\n",
      "prediction stomen is mared to the forvermarss the founcet\n",
      "average test loss is 1.5802616645945584\n",
      "Epoch 7 of 150 LR [0.009524135262330098]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "46it [02:41,  3.47s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 131328])\n",
      "tensor([[0.0000, 0.0000, 0.0000,  ..., 0.0004, 0.0005, 0.0005],\n",
      "        [0.0000, 0.0000, 0.0000,  ..., 0.0004, 0.0005, 0.0005]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "287it [16:41,  3.49s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 200448])\n",
      "tensor([[ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -7.8108e-05,\n",
      "         -1.9141e-05, -7.8119e-06],\n",
      "        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -7.8108e-05,\n",
      "         -1.9141e-05, -7.8119e-06]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "389it [22:40,  3.49s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 200448])\n",
      "tensor([[ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -8.9087e-05,\n",
      "         -5.9199e-05, -5.9370e-05],\n",
      "        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -8.9087e-05,\n",
      "         -5.9199e-05, -5.9370e-05]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "637it [37:13,  3.47s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 275328])\n",
      "tensor([[ 0.0000,  0.0000,  0.0000,  ..., -0.0015, -0.0014, -0.0014],\n",
      "        [ 0.0000,  0.0000,  0.0000,  ..., -0.0015, -0.0014, -0.0014]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "666it [38:54,  3.51s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 180864])\n",
      "tensor([[ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -4.0159e-05,\n",
      "         -3.1214e-05,  3.6880e-06],\n",
      "        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -4.0159e-05,\n",
      "         -3.1214e-05,  3.6880e-06]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "926it [54:06,  3.47s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 205056])\n",
      "tensor([[ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -2.7344e-06,\n",
      "         -7.1339e-06, -7.7367e-05],\n",
      "        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -2.7344e-06,\n",
      "         -7.1339e-06, -7.7367e-05]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "1025it [59:53,  3.49s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 270720])\n",
      "tensor([[ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -7.7933e-06,\n",
      "         -1.6287e-05, -5.2683e-05],\n",
      "        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -7.7933e-06,\n",
      "         -1.6287e-05, -5.2683e-05]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "1062it [1:02:03,  3.51s/it]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MEAN EPOCH LOSS IS 1.3129917292271631\n",
      "target a goal task is a task of satisfying a condition\n",
      "prediction ico pro scasi ost te sodis fi ifpiodition\n",
      "target he was drawn to a debtors prison\n",
      "prediction he was cuned to the deters prison\n",
      "target suspicion eager and sharp looks out\n",
      "prediction suspision e ter a sror look outolteee\n",
      "target he believes that it was one of the most important decisions of his career\n",
      "prediction thebeli setr was ooffemis inpoedo the fisions of his coolis\n",
      "target five people are sitting together in the snow\n",
      "prediction fo peofle a siting togather in t sco\n",
      "target he assisted his mother in leading the denomination\n",
      "prediction he isisted is mother inleating the denobonation\n",
      "target it is a commercially important species\n",
      "prediction it is a commersually inmporo speasies\n",
      "target whitefish point is a designated important bird area\n",
      "prediction wi fisbo is a dasiated importen purdarea\n",
      "target dunckley was born in warwick\n",
      "prediction tokle was born in ori\n",
      "target today its catalogue has expanded to span several genres\n",
      "prediction the li e es extanded tosfen several solerstsese\n",
      "target tell me did you turn to humpty dumpty\n",
      "prediction tomy de uta to onsedamtye\n",
      "target these were used for training\n",
      "prediction thes eesed for trai\n",
      "target thats hardly enough he said anxiously\n",
      "prediction thes elenno esod antheslry\n",
      "target despite its recent adoption it has old roots\n",
      "prediction is pidits despided tote sintedotuy cas obus\n",
      "target the balloon is then perforated with a needle\n",
      "prediction the bitle isten befrytericate onr\n",
      "target he decided to try what fortune had in store for him\n",
      "prediction he dissided tocroe foryuotinstorfa\n",
      "target the city is served by thompson public school\n",
      "prediction this sitty servedd by cumsor owi scoo\n",
      "target commodus statues were thrown down\n",
      "prediction cootof stetyus wrer frondo\n",
      "target asian people walking around the streets\n",
      "prediction thesi epe wo era the se\n",
      "target a swan landing in a lake\n",
      "prediction te sworn ron l ive lake\n",
      "target scripps has been a story element in numerous fictional works\n",
      "prediction sucraps has beinistorialmen i nuvrs fitioal works\n",
      "target we see convergences of different technologies\n",
      "prediction reseecon versans is af tofricicpoliges\n",
      "target mechanoid was the vehicle to do this\n",
      "prediction licolas astete fur n titis\n",
      "target finally khitan forces withdrew\n",
      "prediction filoi gedin orces of te troesdstsrle\n",
      "target there she opened national tatar art\n",
      "prediction the heoprenmetion totor ot\n",
      "target in the company he also appears as a fan favorite\n",
      "prediction in the compay he also apears was a fan fibreel\n",
      "target he has two daughters from his first marriage\n",
      "prediction he is tododi fror is fors marage\n",
      "target abati was a satirical poet who wrote in the baroque style\n",
      "prediction oberty was e seterito po fuded in the boto stase\n",
      "target king stakes himself instead\n",
      "prediction kang staks im sel in stad\n",
      "target the council normally meets six times each year\n",
      "prediction he caunso nobely met suxrons tec ear\n",
      "target doesnt even come close\n",
      "prediction us ni comclofs\n",
      "target to prevent the train rolling back the brakes were applied\n",
      "prediction topreben the fran royvas the brat for fod\n",
      "target can you search the picture titled the accounting\n",
      "prediction pe surts tepitul titleld te conting\n",
      "target thats how he spotted me\n",
      "prediction thet ois spadedy \n",
      "target these ideas are universally accepted\n",
      "prediction he saes a inecitisete\n",
      "target it is currently a part of the tacoma public school district\n",
      "prediction it is cerely aportof ticobe obe scuo distrend\n",
      "target his form for north queenslands first season was mixed\n",
      "prediction his forl for tor crensons for season was liedsl\n",
      "target the story also features the beatles in a film clip\n",
      "prediction the story also feature s the betles in a filqert\n",
      "target he remained very popular until old age\n",
      "prediction he reain very poculerintoaelageese\n",
      "target in each chart the plaques are dimensional submanifolds\n",
      "prediction he e curtd the pixs ar themintiu sut miifors\n",
      "target three men dressed in white playing kettle drums outdoors\n",
      "prediction thre men grestin wit plang catledrems ar to\n",
      "target and even so  what right have you to pull them she asked\n",
      "prediction tink iel so wat rater ou tepover shes\n",
      "target each mast carried three sails\n",
      "prediction therest caid tro sas\n",
      "target this prompted her move to a house near reading\n",
      "prediction this pontec renliter hos e redis\n",
      "target reflecting pools and fountains spread throughout the compound\n",
      "prediction relickan pols ind fountin sprea treou the conprond\n",
      "target a lot of people walking around outside of a building\n",
      "prediction elit o peple woding aro tun sie of ar building\n",
      "target borden was however elected again for halifax\n",
      "prediction or as porbenlito the gin for holifes\n",
      "target its capital is kaya\n",
      "prediction this cotee is cieslt \n",
      "target a man in a green shirt leaning on a pole\n",
      "prediction a mon in a gre sho laing on the for\n",
      "target he is known for his signature calls of goodbye\n",
      "prediction he is kno for is sinitipos of pebo\n",
      "target the relation between such propositions\n",
      "prediction terrilation eten suchpropositionsseselob\n",
      "target i wonder if it feels lonely\n",
      "prediction themonneis a isiololyaiae\n",
      "target the western border of the suburb is marked by the gateway motorway\n",
      "prediction the we en bode soer is mortd by a kewa mode\n",
      "target this was certainly familiar before hilbert\n",
      "prediction this was sertly fennilyor be four hilberd\n",
      "target this was used among others by julian tuwim\n",
      "prediction this was sus on uniles byeian toi\n",
      "target the bridge was built by the oxford mason john townesend\n",
      "prediction the broge was boed byhe otro mason jonton end\n",
      "target they are such pros\n",
      "prediction y ars switcros\n",
      "target it is a multipurpose stadium\n",
      "prediction it is a mosite of tistadiooe\n",
      "target a public announcement followed the ceremony\n",
      "prediction the puble inasbenforlyd de serman\n",
      "target voluntary enclosure was frequent at that time\n",
      "prediction rontran closers was rrecei e tisee\n",
      "target shall we go up in the lift or shall we take the escalator\n",
      "prediction owegop in the wist or shorweta theiscatrer\n",
      "target in essence he was the life of the party\n",
      "prediction hen esince he was alatof o the paryll\n",
      "target i stood awhile in thought\n",
      "prediction pi stode wiin bod\n",
      "target however this was soon to change\n",
      "prediction however thes wat son the ja\n",
      "target fort ridgely state park is nearby\n",
      "prediction fordrily i ta o tiseba\n",
      "target two dogs playing in a field\n",
      "prediction two gogs playing in a feod\n",
      "target the borough borders franklin township and cumberland county\n",
      "prediction the boabo dis frenten tounsip af coin contry\n",
      "target this compound is then secreted by the patients body\n",
      "prediction thes commpowd is then ucreted by the pations bory\n",
      "target our state does not have a climate for job creation\n",
      "prediction ar stat is non o ecloif forjocreationaie\n",
      "target a white dog is standing on a grassy hillside\n",
      "prediction a wi do i staing on a gracivelsi\n",
      "target the project initially drew some criticism\n",
      "prediction the projectonnutially dres im creiceson\n",
      "target thats brilliant i think\n",
      "prediction abriian i tenk\n",
      "target the family attempts to get a prize at the carnival\n",
      "prediction the fally attenes tegea prie at te cornivle\n",
      "target the production later transferred to the west end\n",
      "prediction the protociollity i frosfer to te es in\n",
      "target its just the thing\n",
      "prediction its justd te ti\n",
      "target students attend meadowbrook high school\n",
      "prediction stedens atan mel rerpiscoololtetet e \n",
      "target perhaps it is paid already\n",
      "prediction reas i is pado redy\n",
      "target the reasons for the withdrawal are unclear\n",
      "prediction the resein es fotewifto foronty\n",
      "target the data are protected for confidentiality\n",
      "prediction the ate erpotected for conpredinsariy\n",
      "target its county seat is amarillo\n",
      "prediction its county seat is ummerenehlete \n",
      "target today chief occupations are fishing and tourism\n",
      "prediction today je otupations ervisiong anturson\n",
      "target i must master that\n",
      "prediction i es mesteders\n",
      "target the zoo is the largest zoo in ecuador\n",
      "prediction thise is the larces su in accudo\n",
      "target two girls riding on swings\n",
      "prediction fe togirs ridi on sins\n",
      "target a skateboarder slides down a metal railing\n",
      "prediction the ska bordersli down o mete ron\n",
      "target the ambulance is on its way\n",
      "prediction the aelans tis on ly swa\n",
      "target the link for windows drivers has been removed\n",
      "prediction the wink frorwindes drivers has beenemued\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "target oconnor charity fund\n",
      "prediction the corer hurdy fomehlt \n",
      "target the following years were not nearly as successful\n",
      "prediction thefo ears were not narly as sucessful\n",
      "target the schools mascot is the rising sun\n",
      "prediction the scoos masicot is the rising so\n",
      "target he was replaced by new lead curtis young\n",
      "prediction he was supplaed by nuele crutes guns\n",
      "target collins and major chuck yeager\n",
      "prediction cos an majer trip eater\n",
      "target he then went on to direct minor films\n",
      "prediction he te worin to tarit monfils\n",
      "target grusin is married to nan newton\n",
      "prediction rruson is married to nan muvise\n",
      "target text programs are easier to write smaller and run faster\n",
      "prediction theprorensar isi to rraloter abepesiten\n",
      "target a woman in a white shirt playing volleyball\n",
      "prediction e bomen in e wa shirt paying voyble\n",
      "target he was left in his cage\n",
      "prediction he was te lefst inus cae\n",
      "target these are available in both print and electronically\n",
      "prediction the arrawasbu trit armetor\n",
      "target he received his information from court records\n",
      "prediction there seived dis enpomation fom cortrecordsaie\n",
      "target a stickle is a hill with a prominent rocky top\n",
      "prediction e sticce is a ill with the promie ropetof\n",
      "target a couple enjoying the night lights together\n",
      "prediction  in eion oplntouna lace\n",
      "target they proved to be exceedingly popular\n",
      "prediction the proved to b ecetingly potor\n",
      "target during workups she lost two aircraft\n",
      "prediction theri wor bots she was to aircrot\n",
      "target you must find a can of worms to get this power up\n",
      "prediction hemas for a lycana woren siat thisporsitse\n",
      "target shriekback are still actively producing music\n",
      "prediction she refet for sto ectef teproduci usi\n",
      "target three sisters in matching pink dresses standing on the beach\n",
      "prediction thereiss temethi pentresss  stay onta\n",
      "target it is owned by power corporation of canada\n",
      "prediction it is aned by coo coperation wof cauter\n",
      "target a majority was required for election\n",
      "prediction the mojari els required for lation\n",
      "target both albums were released by caroline records\n",
      "prediction thes altums weleed the colyreces\n",
      "target however the higher frequency often brings advantages\n",
      "prediction howewer the iaf rrepinsy ofen bris aden dages\n",
      "target joscelin was rewarded with the county of edessa\n",
      "prediction te len was revorded with the comteovadise\n",
      "target he lived at the ropewalk nottingham\n",
      "prediction ei atrubot notinden\n",
      "target a woman wearing a black jacket is walking on a sidewalk\n",
      "prediction foove weri lex jecxes is foting on the site\n",
      "target he signed his art as c krieghoff\n",
      "prediction he sie i ar is se freko\n",
      "target she identifies as bisexual\n",
      "prediction she ten tofies atis by situ\n",
      "target dismiss her patronage from consideration\n",
      "prediction this misst er patrea fren consideration\n",
      "target he was killed in battle\n",
      "prediction he was kiled in mow\n",
      "target aristocrats shunned it as a peasant drink\n",
      "prediction oris te crap shiondi as a pisen dretoltl \n",
      "target that same year the medical schools new clinical departments were opened\n",
      "prediction the so u the meipo scoos eciiccal the prormits were open\n",
      "target oh what is the matter\n",
      "prediction o wat is the metrer\n",
      "target tala is the mistress of her own part of hell\n",
      "prediction oer is the mistrers ofronorhi\n",
      "torch.Size([2, 198144])\n",
      "tensor([[0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 4.8324e-05, 7.6901e-05,\n",
      "         3.2309e-05],\n",
      "        [0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 4.8324e-05, 7.6901e-05,\n",
      "         3.2309e-05]])\n",
      "target banner hill is a suburb of erwin\n",
      "prediction temeti is a suberbaf arry\n",
      "target his short story the flood has been translated into english\n",
      "prediction thes srostory the flo as bentranenclated in dalishagsgsl\n",
      "target it was repeated on just two occasions\n",
      "prediction it was a repeted undusto asions\n",
      "target it is an allboys school\n",
      "prediction it is an al bus school\n",
      "target a person is riding in a boat on a lake\n",
      "prediction a pursion is rady in a bo on the lae\n",
      "target admit that you have been unable to do so for the past yearandahalf\n",
      "prediction it i the u tealible to do sof or the paster an er\n",
      "target it consists of two platforms\n",
      "prediction it consis of to performs\n",
      "target he was remarkable for his humility and piety\n",
      "prediction he was reorcoble for tis umulityan piitye\n",
      "target many were unsafe and costly to maintain\n",
      "prediction the u wori sesins of secmenteestese\n",
      "target a dog leaps over a wooden fence as another is about to\n",
      "prediction the dogy covera woin fen is mother res aboptu\n",
      "target it has the largest area of unfenced common land in england\n",
      "prediction it has the nogestene of ben finc collolanled ingund\n",
      "target i was fined for returning the library book late\n",
      "prediction i res fiin te reteringded wivery ofly\n",
      "target for example seated presses may be done while standing\n",
      "prediction fer dexemple sete presimabedan mol standi\n",
      "target fox an assistant surgeon on the expedition\n",
      "prediction far an a sisted sidin on te expiditionaseoterte\n",
      "target the behaviour is rough and the fish often jump out of the water\n",
      "prediction the beever is ro in the fese ofpenjubon of the warder\n",
      "target lipscomb does not have fraternities and sororities\n",
      "prediction thitsgon dis knot ha forturtys oersorrooties\n",
      "target legend says also they had in total ten sons\n",
      "prediction legon sas also verein toin tinsons\n",
      "target a girl in a green bathing suit leaping into a lake\n",
      "prediction a girl in a gren bading su pyping inoellak\n",
      "target i can hear wilko in lots of places\n",
      "prediction i can ewil coy lofs of plaes\n",
      "target a hospital in the city was established in her memory\n",
      "prediction theete te ce was i soblition amonlo\n",
      "target he remained somewhat rueful though no longer bitter about it\n",
      "prediction he emainn somon worl for to no longger biderbai\n",
      "target highpressure lamps produce very bright bluish white light\n",
      "prediction iperiallens produ very brit buish woartlif\n",
      "target at high tide access is a difficult climb down the cliff\n",
      "prediction a rita dasiesis te diffice cri down tot cuf\n",
      "target he is usually depicted with a bag full of valuable objects\n",
      "prediction he isusual depicted with the baed for  vaive ofjuts\n",
      "target keenanbolger starred as mary flynn in the encores\n",
      "prediction kilen bolger sto is marryfle of te uncles\n",
      "target the quorum for privy council meetings is four\n",
      "prediction the corin ser prote counso redings is for\n",
      "target this set in motion a new leadership crisis which lasted almost six months\n",
      "prediction the sarmlo so in nylidisoprae swir lesed on te sulots\n",
      "target the system was never supposed to be a huge seller for atari\n",
      "prediction thi sister was no as posedto ve ay seior firs\n",
      "target alwal was a prime area under the british cantonment\n",
      "prediction ao was fron ara on the te griscran onbens\n",
      "target he moved to england\n",
      "prediction helo ticti ris\n",
      "target from bristol\n",
      "prediction from briste\n",
      "target the northern boundary is the railway line\n",
      "prediction e oder bantreised to rewalos\n",
      "target later that year ray became a christian convert\n",
      "prediction later ther year ra beca a criestu cober\n",
      "target american eagle continued the route after the merger\n",
      "prediction lotlo contonu terly anpimeite\n",
      "target that was a very interesting lesson she said\n",
      "prediction do the berry intrisilesson kesod\n",
      "target we die under your stone ceilings\n",
      "prediction le doy anlayors to seles\n",
      "target newscasts are simulcasted to both stations\n",
      "prediction heis cused ar so upested deblu stasionso\n",
      "target he is a director of dubai bank\n",
      "prediction ie is ate recter of te bi bond\n",
      "target fourteen cars participated in race\n",
      "prediction fortein corss prejusipated in os\n",
      "target a civil funeral took place at lenkom theater\n",
      "prediction the cirel vadel toplased laco saser\n",
      "target mccarron manages to deliver the infant alive and well\n",
      "prediction recaro meius to te lofvirthe infentowi e werl\n",
      "target i clung desperately to the words of peter\n",
      "prediction ilor disperilyed to the bos of peter\n",
      "target he has played twelve caps for his country\n",
      "prediction he is plete cot for is contry\n",
      "target she continued her remarks\n",
      "prediction she contiu er remarks\n",
      "target he was the son of a lutheran minister\n",
      "prediction he is the soliteliin lilisters\n",
      "target he was created by simon spurrier and artist carl critchlow\n",
      "prediction he was coreated by somin sviia ad otis col critul\n",
      "target the following day he was claimed by the washington capitals\n",
      "prediction the followingda he was cland by the wosionton capitels\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "target it tastes strongly of iodine\n",
      "prediction it a stoly ofaedise\n",
      "target dibens lives and trains in boulder colorado\n",
      "prediction dibins livves an traineds in bulder colleroa\n",
      "target the contact was limited but it was at full speed\n",
      "prediction the pontic was ebecatebeytaser fu spedy\n",
      "target three people giving a presentation before a crowd\n",
      "prediction fre pepeliing of pisicatiton e for\n",
      "target they toured around europe and north america mostly as support\n",
      "prediction the tordaroour of ben northe merita mos le i suppor\n",
      "target a celebration in king arthurs court follows\n",
      "prediction a sobasion into otis co follos\n",
      "target the group subsequently disbanded\n",
      "prediction thet te coks ubchipentexunded\n",
      "target it is eaten with bread\n",
      "prediction it is acin ri brat\n",
      "target the underfur is paler in color\n",
      "prediction the counderfer is paler in coler\n",
      "target the modern term for this meaning is ultrasonic\n",
      "prediction the moden tem for the se is orte soni\n",
      "target the toes contain lamellae\n",
      "prediction he tos conan themellot \n",
      "target peleaga is located in the park\n",
      "prediction poliorder is wocated in the bork\n",
      "target there are a few exceptions known\n",
      "prediction thee ar fe oxsuptuos knownl  \n",
      "target he also centralized control over supply\n",
      "prediction he also fenturullis contro over soppli\n",
      "target im a comedian doing radio\n",
      "prediction an te commution do radio\n",
      "target a man dressed in full santa attire with beard and glasses\n",
      "prediction the main destinful santer tir wih beed in lanses\n",
      "target john dowland visited him in italy as well\n",
      "prediction jon dowlan visideddeam anmevely is well\n",
      "target kids chasing a soccer ball\n",
      "prediction ias tusin te soffivlel\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "0it [00:00, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "target timothy played herriot in the television series\n",
      "prediction twoesepla erriat in tytobision seris\n",
      "average test loss is 1.3849328603336517\n",
      "Epoch 8 of 150 LR [0.009381533400219317]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "50it [02:53,  3.44s/it]"
     ]
    }
   ],
   "source": [
    "#train(model, opt, train_loader, scheduler, CTCLoss, device,\n",
    "#     n_epochs=NUM_EPOCHS, val_dl=val_loader)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 291,
   "metadata": {},
   "outputs": [],
   "source": [
    "def beam_search_decoding(output, answ, answ_lens, blank_label=0, width=8):\n",
    "    decoded_preds, decoded_targs  = [], []\n",
    "    \n",
    "    text_transform = TextTransform()    \n",
    "    \n",
    "    for i, mat in enumerate(output.transpose(0, 1)):\n",
    "        last = {}        \n",
    "        P_b, P_t = 1, 1\n",
    "        P_nb = 0\n",
    "        # dict       [0:prob_blank, 1:prob_not_blank, 2:prob_total, 3:list of ]\n",
    "        last[''] = [P_b, P_nb, P_t]\n",
    "                        \n",
    "        for t in range(mat.shape[0]):\n",
    "            curr = {}\n",
    "            \n",
    "            # sorting\n",
    "            cand = [(key, el) for (key, el) in last.items()] # листы pb pnb pt beam\n",
    "            sorted_cand = sorted(cand, reverse=True, key=lambda x: x[1][2]) # P_Total\n",
    "            best_beams = [key for (key, el) in sorted_cand][0:width] # лучших w штук bшк\n",
    "           \n",
    "            for beam in best_beams:\n",
    "                P_nb = 0\n",
    "                if t == 0:\n",
    "                    beam = ''\n",
    "                else:\n",
    "                    if len(beam) > 0:\n",
    "                        last_num = text_transform.text_to_int(beam[-1])\n",
    "                        P_nb = last[beam][1] * mat[t, last_num]\n",
    "\n",
    "                    P_b = last[beam][2] * mat[t, blank_label]\n",
    "\n",
    "                    if beam not in curr:\n",
    "                        curr[beam] = [P_b, P_nb, P_b+P_nb]\n",
    "                    else:\n",
    "                        curr[beam][0] += P_b\n",
    "                        curr[beam][1] += P_nb\n",
    "                        curr[beam][2] += P_b + P_nb                \n",
    "                \n",
    "                # 0 is blank\n",
    "                for c in range(1, mat.shape[1]): \n",
    "                    new_beam = beam + ''.join(text_transform.int_to_text([c]))                    \n",
    "                    \n",
    "                    if len(beam) > 0 and last_num == c:\n",
    "                        P_nb = mat[t, c] * last[beam][0]\n",
    "                    else:\n",
    "                        P_nb = mat[t, c] * last[beam][2]\n",
    "                    \n",
    "                    if new_beam not in curr:\n",
    "                        curr[new_beam] = [0, P_nb, P_nb]\n",
    "                    else:\n",
    "                        curr[new_beam][1] += P_nb \n",
    "                        curr[new_beam][2] += P_nb                \n",
    "            last = curr\n",
    "        \n",
    "        cand = [(key, el) for (key, el) in last.items()]\n",
    "        sorted_cand = sorted(cand, reverse=True, key=lambda x: x[1][2])\n",
    "        best_beam = [x[0] for x in sorted_cand][0]\n",
    "        \n",
    "        decoded_preds.append(best_beam)\n",
    "        \n",
    "        # i - номер бача\n",
    "        decoded_targs.append(\n",
    "                text_transform.int_to_text(answ[i][:answ_lens[i]].tolist())\n",
    "        )\n",
    "        \n",
    "    return decoded_preds, decoded_targs                "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 292,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "target:  he started violently\n",
      "prediction:  hs ter fi\n",
      "beam_search_preds:  this ter fise\n",
      "target:  i think it is\n",
      "prediction:  i then etis\n",
      "beam_search_preds:  bi  theng etes\n",
      "torch.Size([2, 198144])\n",
      "tensor([[0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 4.8324e-05, 7.6901e-05,\n",
      "         3.2309e-05],\n",
      "        [0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 4.8324e-05, 7.6901e-05,\n",
      "         3.2309e-05]])\n",
      "target:  a boy is sleeping in his dinner bowl\n",
      "prediction:  abi seping in as tat \n",
      "beam_search_preds:  ta bi seping in as ta nrllltttt                                                         \n",
      "target:  a woman in gray cuts cake\n",
      "prediction:  a won in gray cat\n",
      "beam_search_preds:  ta  won in gray cats\n",
      "target:  is she as good as you say\n",
      "prediction:  the se as boreses\n",
      "beam_search_preds:  the se  as boreses\n",
      "target:  flaming youth\n",
      "prediction:  flaiin ou\n",
      "beam_search_preds:  ffllain ou\n",
      "target:  list of mirmo\n",
      "prediction:  he list bof e ammo\n",
      "beam_search_preds:  the list  of e ammol\n",
      "target:  you have a couch\n",
      "prediction:  he ha a cand\n",
      "beam_search_preds:  the  hav a cand\n",
      "target:  no but suppose she was\n",
      "prediction:  tbaswas sho\n",
      "beam_search_preds:  t aswas sho\n",
      "target:  a group of hikers hike\n",
      "prediction:  the gro of hatis ha\n",
      "beam_search_preds:  the gro of hatis hag\n",
      "target:  he lost his first match\n",
      "prediction:  he  lase fet mech\n",
      "beam_search_preds:  yhe  lasse  fert metch\n",
      "target:  it was fun\n",
      "prediction:  i was fon\n",
      "beam_search_preds:  tit  was  fon\n",
      "target:  warren and son\n",
      "prediction:  warin anson\n",
      "beam_search_preds:  warrin  anson\n",
      "target:  whats up\n",
      "prediction:  watter\n",
      "beam_search_preds:  wat er\n",
      "target:  live and let live\n",
      "prediction:  tev fon tedes\n",
      "beam_search_preds:  tev fon tedess\n",
      "target:  afraid it might rain\n",
      "prediction:  ope if mi flen\n",
      "beam_search_preds:  t o pe if  mi flen\n",
      "target:  will it snow in granbury\n",
      "prediction:  lit kno in gron\n",
      "beam_search_preds:  w lit  knot in grongrr\n",
      "target:  maybe it will\n",
      "prediction:  wle i wo\n",
      "beam_search_preds:  wme i wo\n",
      "target:  tony lilly in command\n",
      "prediction:  nny leilly in cna\n",
      "beam_search_preds:  wnny leilly  in cmand\n",
      "target:  allerdings\n",
      "prediction:  ilomins\n",
      "beam_search_preds:  wilomins\n",
      "target:  he was not alone\n",
      "prediction:  he was notta lon\n",
      "beam_search_preds:  she was notta lon\n",
      "target:  the schools athletic teams are known as the red raiders\n",
      "prediction:  e atante\n",
      "beam_search_preds:  te at ante\n",
      "target:  and they passed on\n",
      "prediction:  i the pas on\n",
      "beam_search_preds:  bin the past on\n",
      "target:  you dont say so\n",
      "prediction:  do sasel\n",
      "beam_search_preds:  tdo sasel\n",
      "torch.Size([2, 200448])\n",
      "tensor([[ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -8.9087e-05,\n",
      "         -5.9199e-05, -5.9370e-05],\n",
      "        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -8.9087e-05,\n",
      "         -5.9199e-05, -5.9370e-05]])\n",
      "target:  as you wish\n",
      "prediction:  i ye ise \n",
      "beam_search_preds:  ti ye isseeeeee                                                                            \n",
      "target:  whom are you calling now\n",
      "prediction:  n i coing a\n",
      "beam_search_preds:  wn i corling na\n",
      "target:  theres the rub\n",
      "prediction:  theres ther rob\n",
      "beam_search_preds:  thereis ther roub\n",
      "target:  care for the sick\n",
      "prediction:  kar forte se\n",
      "beam_search_preds:  skar forte se\n",
      "target:  we can both hear\n",
      "prediction:   wten o\n",
      "beam_search_preds:  t wt en o\n",
      "target:  id like to think\n",
      "prediction:  i li for fin\n",
      "beam_search_preds:  bi li for fin\n",
      "target:  what is it now\n",
      "prediction:  what is it now\n",
      "beam_search_preds:  what is it now\n",
      "target:  that was my first thought\n",
      "prediction:  the is na forst tark\n",
      "beam_search_preds:  the is nav forsst ark\n",
      "target:  wie kann man sich schtzen\n",
      "prediction:  te cone shise\n",
      "beam_search_preds:  te con e shilse\n",
      "CER 48.82082145759296\n",
      "CER BS 50.6209678653425\n",
      "WER 84.74335709184193\n",
      "WER BS 88.5725581937703\n",
      "average test loss is 1.604088331713821\n"
     ]
    }
   ],
   "source": [
    "test(model, opt, val_loader, CTCLoss, device, bs_width=8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "target:  they are also used for local level polling\n",
    "\n",
    "prediction:  they are also used for loka level polling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}