AI_Writes_TBBT_Using_GPT_2_Text_Generating_Model.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "AI-Writes-TBBT-Using-GPT-2 Text-Generating Model w/ GPU",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H7LoMj4GA4n_",
        "colab_type": "text"
      },
      "source": [
        "##  AI Writes The Big Bang Theory\n",
        "\n",
        "Training advanced text-generating neural network on Big bang theory 9 seasons corpes and let AI generate additional dialogues for the beloved characters.\n",
        "\n",
        "using `gpt-2-simple`! -> GPT-2 Fine tuning implementation\n",
        "\n",
        "For more about `gpt-2-simple`, you can visit [this GitHub repository](https://github.com/minimaxir/gpt-2-simple)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qMgSwNJ5k3Mx",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 119
        },
        "outputId": "1bbcefd3-522c-4e24-c546-1790d4f5f274"
      },
      "source": [
        "!git clone --single-branch --branch data-set https://github.com/bhavita/AI-Writes-TBBT.git\n"
      ],
      "execution_count": 65,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Cloning into 'AI-Writes-TBBT'...\n",
            "remote: Enumerating objects: 209, done.\u001b[K\n",
            "remote: Counting objects: 100% (209/209), done.\u001b[K\n",
            "remote: Compressing objects: 100% (209/209), done.\u001b[K\n",
            "remote: Total 209 (delta 0), reused 205 (delta 0), pack-reused 0\u001b[K\n",
            "Receiving objects: 100% (209/209), 1.30 MiB | 4.38 MiB/s, done.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "34-MZT2vk43U",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "68ad6828-3273-4a0f-98b5-64ad2900b22a"
      },
      "source": [
        "import glob\n",
        "\n",
        "episodes_filenames = sorted(glob.glob(\"AI-Writes-TBBT/DataSet/*.txt\"))\n",
        "\n",
        "print(\"Found {} Episodes\".format(len(episodes_filenames)))"
      ],
      "execution_count": 66,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Found 202 Episodes\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "slil0SddmbqI",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "outputId": "0ef5978f-7def-40cd-c921-0f5f462fb9d1"
      },
      "source": [
        "print(\"Displaying few episodes for TBBT\")\n",
        "episodes_filenames[:9]"
      ],
      "execution_count": 84,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Displaying few episodes for TBBT\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['AI-Writes-TBBT/DataSet/1_season_episode1.txt',\n",
              " 'AI-Writes-TBBT/DataSet/1_season_episode10.txt',\n",
              " 'AI-Writes-TBBT/DataSet/1_season_episode11.txt',\n",
              " 'AI-Writes-TBBT/DataSet/1_season_episode12.txt',\n",
              " 'AI-Writes-TBBT/DataSet/1_season_episode13.txt',\n",
              " 'AI-Writes-TBBT/DataSet/1_season_episode14.txt',\n",
              " 'AI-Writes-TBBT/DataSet/1_season_episode15.txt',\n",
              " 'AI-Writes-TBBT/DataSet/1_season_episode16.txt',\n",
              " 'AI-Writes-TBBT/DataSet/1_season_episode17.txt']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 84
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Z0GCGkQCk-kN",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "with open('tbbt-collab.txt', 'w') as outfile:\n",
        "  for file in episodes_filenames:\n",
        "    with open(file) as infile:\n",
        "      outfile.write(infile.read())"
      ],
      "execution_count": 68,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jbpUcL-MlIiq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KBkpRgBCBS2_",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 170
        },
        "outputId": "302feb92-d4f2-4dc6-8fdc-6b187d399b41"
      },
      "source": [
        "%tensorflow_version 1.x\n",
        "!pip install -q gpt-2-simple\n",
        "import gpt_2_simple as gpt2\n",
        "from datetime import datetime\n",
        "from google.colab import files\n",
        "import tensorflow as tf\n"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "TensorFlow 1.x selected.\n",
            "WARNING:tensorflow:\n",
            "The TensorFlow contrib module will not be included in TensorFlow 2.0.\n",
            "For more information, please see:\n",
            "  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md\n",
            "  * https://github.com/tensorflow/addons\n",
            "  * https://github.com/tensorflow/io (for I/O related ops)\n",
            "If you depend on functionality not listed there, please file an issue.\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sUmTooTW3osf",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 357
        },
        "outputId": "c507b7f8-64d6-4ce4-ccd6-c1be4ae6769b"
      },
      "source": [
        "!nvidia-smi"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Sat Aug 29 16:56:56 2020       \n",
            "+-----------------------------------------------------------------------------+\n",
            "| NVIDIA-SMI 450.66       Driver Version: 418.67       CUDA Version: 10.1     |\n",
            "|-------------------------------+----------------------+----------------------+\n",
            "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
            "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
            "|                               |                      |               MIG M. |\n",
            "|===============================+======================+======================|\n",
            "|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |\n",
            "| N/A   43C    P8     9W /  70W |      0MiB / 15079MiB |      0%      Default |\n",
            "|                               |                      |                 ERR! |\n",
            "+-------------------------------+----------------------+----------------------+\n",
            "                                                                               \n",
            "+-----------------------------------------------------------------------------+\n",
            "| Processes:                                                                  |\n",
            "|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n",
            "|        ID   ID                                                   Usage      |\n",
            "|=============================================================================|\n",
            "|  No running processes found                                                 |\n",
            "+-----------------------------------------------------------------------------+\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0wXB05bPDYxS",
        "colab_type": "text"
      },
      "source": [
        "## Downloading GPT-2\n",
        "\n",
        "Downloading the GPT-2 model first. Using \n",
        "* `124M` (default): the \"small\" model, 500MB on disk.\n",
        "\n",
        "Other larger models have more knowledge, but take longer to finetune and longer to generate text. For simplification using 124M"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "P8wSlgXoDPCR",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 136
        },
        "outputId": "ecf939c9-560a-456d-ffb5-369593232c87"
      },
      "source": [
        "gpt2.download_gpt2(model_name=\"124M\")"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Fetching checkpoint: 1.05Mit [00:00, 268Mit/s]                                                      \n",
            "Fetching encoder.json: 1.05Mit [00:00, 80.1Mit/s]                                                   \n",
            "Fetching hparams.json: 1.05Mit [00:00, 339Mit/s]                                                    \n",
            "Fetching model.ckpt.data-00000-of-00001: 498Mit [00:02, 218Mit/s]                                   \n",
            "Fetching model.ckpt.index: 1.05Mit [00:00, 355Mit/s]                                                \n",
            "Fetching model.ckpt.meta: 1.05Mit [00:00, 157Mit/s]                                                 \n",
            "Fetching vocab.bpe: 1.05Mit [00:00, 151Mit/s]                                                       \n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZFABYX6YM7QQ",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "fa8c83d4-3ae3-4756-9f47-53a651a6f4b6"
      },
      "source": [
        "!ls"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "checkpoint  models  sample_data  tbbt-collab.txt\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6OFnPCLADfll",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "file_name = \"tbbt-collab.txt\" //corpus developed by combining the 203 episodes"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rVLSx5GQLr2v",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "text = open(file_name).read()\n"
      ],
      "execution_count": 70,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Cn0UH17rlK-M",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "3fdfa931-6716-4fa6-ba65-6a4aa68fb7ea"
      },
      "source": [
        "print ('Length of text: {} characters'.format(len(text)))"
      ],
      "execution_count": 71,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Length of text: 3254056 characters\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Av2Y1XJIlPYv",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 173
        },
        "outputId": "ff9b95b0-764b-4b5e-b11d-3c5b181338af"
      },
      "source": [
        "print(text[:500])"
      ],
      "execution_count": 74,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "\t\t\t\tScene: A corridor at a sperm bank.\n",
            "Sheldon: So if a photon is directed through a plane with two slits in it and either slit is observed it will not go through both slits. If it’s unobserved it will, however, if it’s observed after it’s left the plane but before it hits its target, it will not have gone through both slits.\n",
            "Leonard: Agreed, what’s your point?\n",
            "Sheldon: There’s no point, I just think it’s a good idea for a tee-shirt. \n",
            "Leonard: Excuse me?\n",
            "Receptionist: Hang on. \n",
            "Leonard: One acr\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aeXshJM-Cuaf",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "479e9207-9ea5-4ec3-d990-48de424a6d95"
      },
      "source": [
        "tf.reset_default_graph()\n",
        "\n",
        "\n",
        "sess = gpt2.start_tf_sess()\n",
        "\n",
        "gpt2.finetune(sess,\n",
        "              dataset='tbbt-collab.txt',\n",
        "              model_name='124M',\n",
        "              steps=1000,\n",
        "              restore_from='fresh',\n",
        "              run_name='run123',\n",
        "              print_every=10,\n",
        "              sample_every=200,\n",
        "              save_every=500\n",
        "              )"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/gpt_2_simple/src/sample.py:17: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.where in 2.0, which has the same broadcast rule as np.where\n",
            "Loading checkpoint models/124M/model.ckpt\n",
            "INFO:tensorflow:Restoring parameters from models/124M/model.ckpt\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\r  0%|          | 0/1 [00:00<?, ?it/s]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Loading dataset...\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "100%|██████████| 1/1 [00:04<00:00,  4.49s/it]\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "dataset has 975080 tokens\n",
            "Training...\n",
            "[10 | 26.25] loss=2.81 avg=2.81\n",
            "[20 | 47.60] loss=2.57 avg=2.69\n",
            "[30 | 69.12] loss=2.73 avg=2.70\n",
            "[40 | 90.84] loss=2.72 avg=2.71\n",
            "[50 | 112.67] loss=2.46 avg=2.66\n",
            "[60 | 134.63] loss=2.48 avg=2.63\n",
            "[70 | 156.73] loss=2.40 avg=2.59\n",
            "[80 | 178.97] loss=2.55 avg=2.59\n",
            "[90 | 201.35] loss=2.40 avg=2.57\n",
            "[100 | 223.84] loss=2.34 avg=2.54\n",
            "[110 | 246.34] loss=2.47 avg=2.54\n",
            "[120 | 268.87] loss=2.44 avg=2.53\n",
            "[130 | 291.94] loss=2.34 avg=2.51\n",
            "[140 | 315.86] loss=2.43 avg=2.51\n",
            "[150 | 339.01] loss=2.35 avg=2.50\n",
            "[160 | 361.96] loss=2.31 avg=2.48\n",
            "[170 | 385.20] loss=2.26 avg=2.47\n",
            "[180 | 408.27] loss=2.34 avg=2.46\n",
            "[190 | 431.02] loss=2.40 avg=2.46\n",
            "[200 | 453.75] loss=2.32 avg=2.45\n",
            "======== SAMPLE 1 ========\n",
            " The boy asked us: What the hell do you think you’re going to do with him? And I said I’m going to go out and buy a motorcycle.\n",
            "Sheldon: You know, when I was at a show, a girl got me a hand-painted motorcycle.\n",
            "Leonard: Now that was a motorcycle.\n",
            "Sheldon: I see now, you didn’t get to pick that up, did you??\n",
            "Leonard: Of course not, she was looking at you with an odd look in her mouth, so we let her go.\n",
            "Sheldon: You look like a child.\n",
            "Leonard: Oh, please.\n",
            "Sheldon: I said I saw you with a kid. Is that what she meant?\n",
            "Leonard: She said I knew what she meant.\n",
            "Sheldon: You’re a little boy.\n",
            "Bernadette: I can see that.\n",
            "Sheldon: Come on, you know what? I’m just doing a favour, you idiot.\n",
            "Scene: The apartment.\n",
            "Sheldon: You said if she were you, I wouldn’t want to live with you.\n",
            "Amy: Well, then are you telling her I’m just being too polite? I have a lot of respect for her.\n",
            "Sheldon: Okay, and that brings up another point.\n",
            "Amy: So you’re saying that I don’t have anything against you?\n",
            "Sheldon: Well, she would be an awful lady.\n",
            "Amy: Okay, why shouldn’t her have something for me?\n",
            "Sheldon: Well, now you’re saying that I have nothing for you.\n",
            "Amy: You mean, like, you know what kind of woman you are?\n",
            "Sheldon: Like you, I’m a woman.\n",
            "Bernadette: Well, now that’s the end of the conversation, Sheldon.\n",
            "Sheldon: It’s gone, Sheldon. There.\n",
            "Scene: An apartment door.\n",
            "Amy: What are you looking for?\n",
            "Sheldon: A motorcycle.\n",
            "Bernadette: No motorcycle.\n",
            "Amy: It’s just a motorcycle. It’s got a body checker.\n",
            "Sheldon: No, it’s not a bike. No, it could be an actual motorcycle.\n",
            "Bernadette: No, it’s not a motorcycle. It was stolen.\n",
            "Sheldon: It was a motorcycle. That is what it was. And it’s not a motorcycle because it was stolen.\n",
            "Scene: Penny’s apartment.\n",
            "Sheldon: It’s okay to be suspicious.\n",
            "Penny: Why can’t I check if a woman is dating someone?\n",
            "Sheldon: Good question. How come no one was looking?\n",
            "Penny: Okay, here’s my point.\n",
            "Amy: No, no, good question. Here’s your point.\n",
            "Sheldon: That is my point.\n",
            "Penny: Okay, I’m sorry. This is ridiculous. It really is.\n",
            "Sheldon: Yeah. Yeah, you’re right. Look. This is a lot of stuff going on.\n",
            "Amy: Hey, what come on? (He turns off his laptop computer)\n",
            "Sheldon: It is the same laptop you already have? It doesn’t have anything. You could write it in in a drawer, you could change it back, you could change it back.\n",
            "Amy: Okay. I’m gonna take a detour here and there. I’m gonna move this laptop to this remote lab.\n",
            "Sheldon: Yeah, I’ll move it here.\n",
            "Penny: Why, Penny, why?\n",
            "Sheldon: I have my laptop locked down. I’m moving it here.\n",
            "Penny: I don’t want to move it.\n",
            "Sheldon: And you know what, I’m going to have to move your laptop elsewhere.\n",
            "Scene: Howard’s apartment.\n",
            "Leonard: You sure you have some stuff going on here?\n",
            "Howard: Maybe a couple hundred dollars, maybe. The key was in there somewhere.\n",
            "Leonard: The key?\n",
            "Howard: Something?\n",
            "Leonard: I’m asking you, aren’t you getting paid a lot for this?\n",
            "Howard: No, I know what you’re talking about.\n",
            "Leonard: Leonard, I’m not making you this huge sum.\n",
            "Howard: Okay, this is a lot. Can we get it out of here so they can find someone else?\n",
            "Leonard: I don’t know what I want to do.\n",
            "(Scene: The apartment door. The door opens and a woman in a tux comes running in. This is Amy.)\n",
            "Sheldon: Hello\n",
            "\n",
            "[210 | 488.04] loss=2.34 avg=2.44\n",
            "[220 | 510.70] loss=2.29 avg=2.44\n",
            "[230 | 533.33] loss=2.40 avg=2.43\n",
            "[240 | 555.93] loss=2.14 avg=2.42\n",
            "[250 | 578.54] loss=2.24 avg=2.41\n",
            "[260 | 601.14] loss=2.42 avg=2.41\n",
            "[270 | 623.71] loss=2.04 avg=2.40\n",
            "[280 | 646.27] loss=2.34 avg=2.39\n",
            "[290 | 668.85] loss=2.05 avg=2.38\n",
            "[300 | 691.37] loss=2.26 avg=2.38\n",
            "[310 | 713.91] loss=2.20 avg=2.37\n",
            "[320 | 736.40] loss=2.28 avg=2.37\n",
            "[330 | 758.91] loss=2.20 avg=2.36\n",
            "[340 | 781.47] loss=2.12 avg=2.35\n",
            "[350 | 804.01] loss=2.28 avg=2.35\n",
            "[360 | 826.50] loss=2.27 avg=2.35\n",
            "[370 | 849.03] loss=2.20 avg=2.34\n",
            "[380 | 871.53] loss=2.20 avg=2.34\n",
            "[390 | 894.04] loss=2.15 avg=2.33\n",
            "[400 | 916.61] loss=2.11 avg=2.32\n",
            "======== SAMPLE 1 ========\n",
            " about you think?\n",
            "Sheldon: I suppose it was worth the wait. And that is the way your family lives, but they are both deeply flawed.\n",
            "Gablehauser: Oh, my turn. You and Spock are a couple on an amazing mission, but he doesn’t make it.\n",
            "Sheldon: And they make it. Uh, so I, I must, I, I think your story is fascinating. And of course, it’s going to run so long, the story is going to run so long. I’m trying to take you to the Moon, my friend.\n",
            "Gablehauser: You need to, you need to be very, very careful.\n",
            "Sheldon: Is, is it worth the wait?\n",
            "Gablehauser: That’s because I think this is such an extraordinary achievement, even by the standards of science fiction.\n",
            "Raj: So, where’d you, Sheldon?\n",
            "Sheldon: I’ve done it. And no one’s ever tried to catch me and Sheldon.\n",
            "Gablehauser: But it’s the other way around. It’s the world’s most incredible pleasure to live in a world completely unimpressive.\n",
            "Scene: The rooftop.\n",
            "Sheldon: And I, I was on my way to the moon.\n",
            "Penny: Ooh, you can’t go. It’s a little big, but I’m in town.\n",
            "Sheldon: It’s not big, it’s in the right place. In my case, in the right place.\n",
            "Amy: I hope you’re not a big woman in this room.\n",
            "Penny: Oh, my God.\n",
            "Penny: Well, they don’t know you’d like them.\n",
            "Amy: Yeah, well, I’ll probably be there.\n",
            "Sheldon: Is that you?\n",
            "Penny: No.\n",
            "Amy: I don’t feel like saying so.\n",
            "Scene: Amy and Bernadette’s kitchen.\n",
            "Sheldon: Okay, I hope you realize you, you’re in control.\n",
            "Bernadette: You, you did get my permission. Thank you.\n",
            "Amy: And I don’t give a damn what any of you say on social media, so I guess I, I just…\n",
            "Sheldon: Oh.\n",
            "Amy: Hey.\n",
            "Sheldon: Amy.\n",
            "Bernadette: I’m sorry what happened. But it was not surprising. So… How are you?\n",
            "Sheldon: I’m being honest. In an era when men seem so much more interested in having an emotional experience, what do you do?\n",
            "Amy: Hey, what happened?\n",
            "Sheldon: Actually, I went to a comic book store.\n",
            "Bernadette: Did you text?\n",
            "Howard: Well, I’m going to have to go do some research.\n",
            "Amy: Oh, come on. Are you going to say yes or no?\n",
            "Bernadette: I, I don’t…\n",
            "Amy: I don’t.\n",
            "Howard: Well, you know, once they open up the space capsule to the International Space Station, people will get to have space in their own homes.\n",
            "Amy: And Sheldon will come home alone for years.\n",
           
            "\n",
            "[410 | 949.74] loss=2.12 avg=2.32\n",
            "[420 | 972.24] loss=2.20 avg=2.32\n",
            "[430 | 994.74] loss=2.23 avg=2.31\n",
            "[440 | 1017.22] loss=2.09 avg=2.31\n",
            "[450 | 1039.72] loss=2.04 avg=2.30\n",
            "[460 | 1062.25] loss=2.07 avg=2.29\n",
            "[470 | 1084.75] loss=2.11 avg=2.29\n",
            "[480 | 1107.25] loss=2.17 avg=2.29\n",
            "[490 | 1129.75] loss=2.10 avg=2.28\n",
            "[500 | 1152.30] loss=2.18 avg=2.28\n",
            "Saving checkpoint/run123/model-500\n",
            "[510 | 1177.42] loss=1.95 avg=2.27\n",
            "[520 | 1200.22] loss=2.16 avg=2.27\n",
            "[530 | 1224.20] loss=2.09 avg=2.26\n",
            "[540 | 1247.62] loss=1.95 avg=2.26\n",
            "[550 | 1270.63] loss=1.97 avg=2.25\n",
            "[560 | 1293.89] loss=2.15 avg=2.25\n",
            "[570 | 1317.29] loss=1.95 avg=2.24\n",
            "[580 | 1340.48] loss=2.04 avg=2.24\n",
            "[590 | 1363.64] loss=2.06 avg=2.23\n",
            "[600 | 1386.75] loss=1.97 avg=2.23\n",
            "======== SAMPLE 1 ========\n",
            "Leonard: Okay, Sheldon, I’m just gonna slip through the door.\n",
            "Sheldon: Do it.\n",
            "Scene: The apartment.\n",
            " \n",
            "Howard: Okay, look, if the FBI decides to turn me in, I’m gonna go behind the Iron Curtain to get it handed to them.\n",
            "Sheldon: The people?\n",
            "Howard: The people are here because it’s the FBI.\n",
            "Raj: I am the FBI. They gave me your passport.\n",
            "Howard: We will get it back.\n",
            "Sheldon: Why doesn’t the FBI know that you’re here?\n",
            "Raj: You never answer those questions. You’ve never answered them. Nobody’s going to believe it.\n",
            "Howard: You really think the FBI is lying to the American people?\n",
            "Sheldon: My God, do the FBI have a badge on their boots?\n",
            "Raj: They’re just going to say the least. If he were to show up at your door and shake his hand, he’d be at home crying.\n",
            "Sheldon: If the FBI had a badge on their boots, it wouldn’t make for a great mystery.\n",
            "Raj: It would make for a great crime.\n",
            "Howard: Yeah, he should be back in the house before my door breaks down.\n",
            "Raj: Look, if the FBI wants to get our passport back, they should tell him the FBI’s going to keep it.\n",
            "Howard: I have some good news, he’s back here, and I want to tell him I love him so much.\n",
            "Sheldon: Howard! How dare you do that to my friend. And as long as he’s here, you can’t hurt my feelings.\n",
            "Scene: Penny’s front door.\n",
            "Leonard: Hey, it’s you and me, we got your back.\n",
            "Penny: What?\n",
            "Leonard: Just leave me here, I’m scared.\n",
            "Penny: You don’t say that to me, I say it to you.\n",
            "Leonard: Okay, here. Let me put this in your pocket. How can I do this without my mom?\n",
            "Penny: Okay, wait. Tell me what happens.\n",
            "Leonard: What? I don’t know.\n",
            "Penny: I’m sorry, I’m out.\n",
            "Leonard: What’s going on?\n",
            "Penny: She was out by herself a long time ago. Just, that’s it. I can see her now. She says she can’t catch a cold.\n",
            "Leonard: Oh, that isn’t so great.\n",
            "Penny: No, no, you were. You were such a sweetie. I mean, you were a little shy, but I think people recognize you when you kind of seem shy.\n",
            "Leonard: I know not all the other boys in the world would take that into account when deciding if a baby was a penguin or not.\n",
            "Penny: What am I supposed to say to that guy? He made me an ass.\n",
            "Scene: The apartment at Comic-Con.\n",
            "Like this:Like Loading...\t\t\t\t\t\t\t\t\n",
            "\t\t\t\n",
            "\t\t\t\tScene: The International Space Station.\n",
            "Sheldon: That was just an example of the kind of thing my office needs the space for. As I said, as I’m certain you would say, it’s really a good idea.\n",
            "Wil: I don’t know anyone better than Dr. Cooper. You have some sort of personality. There are many more amazing people out there than you.\n",
            "Sheldon: Okay. So you have your friend Leonard. Isn’t that great?\n",
            "Wil: So, what’s so great about him?\n",
            "Sheldon: I don’t know, I just know it when I see it.\n",
            "Scene: Penny’s door.\n",
            " \n",
            "Wil: You’re not gonna go through with it?\n",
            "\n",
            "\n",
            "[610 | 1420.10] loss=1.89 avg=2.22\n",
            "[620 | 1442.83] loss=2.03 avg=2.21\n",
            "[630 | 1465.48] loss=1.97 avg=2.21\n",
            "[640 | 1488.11] loss=1.94 avg=2.20\n",
            "[650 | 1510.75] loss=2.03 avg=2.20\n",
            "[660 | 1533.38] loss=2.22 avg=2.20\n",
            "[670 | 1555.97] loss=1.98 avg=2.20\n",
            "[680 | 1578.54] loss=1.87 avg=2.19\n",
            "[690 | 1601.12] loss=2.08 avg=2.19\n",
            "[700 | 1623.69] loss=1.96 avg=2.18\n",
            "[710 | 1646.25] loss=1.99 avg=2.18\n",
            "[720 | 1668.79] loss=1.93 avg=2.17\n",
            "[730 | 1691.36] loss=1.95 avg=2.17\n",
            "[740 | 1713.88] loss=1.84 avg=2.16\n",
            "[750 | 1736.42] loss=1.90 avg=2.16\n",
            "[760 | 1758.95] loss=1.85 avg=2.15\n",
            "[770 | 1781.50] loss=2.19 avg=2.15\n",
            "[780 | 1804.06] loss=1.88 avg=2.15\n",
            "[790 | 1826.60] loss=1.61 avg=2.14\n",
            "[800 | 1849.14] loss=1.85 avg=2.13\n",
            "======== SAMPLE 1 ========\n",
            "?\n",
            "Leonard: What?\n",
            "Sheldon: We don’t eat like this, Leonard. Now get off of me.\n",
            "Leonard: Why was I kicked in the butt?\n",
            "Sheldon: Uh, well, for one thing, we’re the ones who did kick you, and for another, you two were so cute and harmless, but you kind of killed me. Secondly, I don’t think the rest of you are really being realistic here.\n",
            "Leonard: Come on, Sheldon, you can do anything you want.\n",
            "Sheldon: No, I’ve never done that before. What do you want?\n",
            "Leonard: Okay, let’s go.\n",
            "Sheldon: Okay. Okay.\n",
            "Leonard: You? ‘Cause we’re just friends.\n",
            "Sheldon: No, no, no, this is about you.\n",
            "Leonard: Hey, I love you, but I’m always worried about you.\n",
            "Sheldon: Don’t get all freaky, Leonard. When I was in school, you had a bully named Don who could come in and get you, and then you got a bully named Don and you’re stuck with him. And now, you can come into a relationship with someone who isn’t Don, who hasn’t had Don since he was six months old.\n",
            "Leonard: What?\n",
            "Leonard: So, we just talked about the new movie we’re making and we’re going to be at the mall?\n",
            "Sheldon: You made that up.\n",
            "Leonard: Great.\n",
            "Sheldon: Oh, good, I think you’ll like it.\n",
            "Leonard: What?\n",
            "Sheldon: I don’t know. I’m being irrational, and that’s not the movie I want to see.\n",
            "Leonard: What about you?\n",
            "Sheldon: You’re a great friend.\n",
            "Leonard: What are you going to do?\n",
            "Sheldon: Don’t do that. Stop it.\n",
            "Leonard: No, no.\n",
            "Sheldon: Oh, I’m not going to do that. I’m an astrophysicist. I was the first to the ISS, and there were all these great people making the jump out at me.\n",
            "Leonard: Yeah, you went to warp speed a while back and you know the rest.\n",
            "Sheldon: I was going to have you know that I never acted like this.\n",
            "Leonard: Yeah. I knew that. I was trying to make it up to you.\n",
            "Sheldon: Well, thank you for the invitation.\n",
            "\n",
            "[810 | 1882.59] loss=1.95 avg=2.13\n",
            "[820 | 1905.09] loss=1.58 avg=2.12\n",
            "[830 | 1927.63] loss=1.88 avg=2.12\n",
            "[840 | 1950.12] loss=1.57 avg=2.11\n",
            "[850 | 1972.62] loss=1.80 avg=2.10\n",
            "[860 | 1995.13] loss=1.76 avg=2.09\n",
            "[870 | 2017.62] loss=1.78 avg=2.09\n",
            "[880 | 2040.16] loss=1.66 avg=2.08\n",
            "[890 | 2062.66] loss=1.79 avg=2.08\n",
            "[900 | 2085.20] loss=1.65 avg=2.07\n",
            "[910 | 2107.70] loss=1.70 avg=2.06\n",
            "[920 | 2130.34] loss=1.59 avg=2.06\n",
            "[930 | 2153.95] loss=1.56 avg=2.05\n",
            "[940 | 2177.74] loss=1.84 avg=2.04\n",
            "[950 | 2200.81] loss=1.82 avg=2.04\n",
            "[960 | 2223.93] loss=1.80 avg=2.04\n",
            "[970 | 2247.35] loss=1.77 avg=2.03\n",
            "[980 | 2270.66] loss=1.43 avg=2.02\n",
            "[990 | 2293.86] loss=1.76 avg=2.02\n",
            "[1000 | 2317.10] loss=1.66 avg=2.01\n",
            "Saving checkpoint/run123/model-1000\n",
            "WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/python/training/saver.py:963: remove_checkpoint (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use standard file APIs to delete files with this prefix.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ClJwpF_ACONp",
        "colab_type": "text"
      },
      "source": [
        "## Sample generation Text From The Trained Model\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4RNY6RBI9LmL",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 122
        },
        "outputId": "9d230c4b-632f-42b2-dd82-803f446e2d0f"
      },
      "source": [
        "gpt2.generate(sess,run_name='run123',\n",
        "              prefix=\"Scene: The comic book store.\\nLeonard: Who would win in Batman vs Superman fight?\",\n",
        "              temperature=0.8,\n",
        "              length=100,\n",
        "              top_p=0.9,\n",
        "              )"
      ],
      "execution_count": 93,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Scene: The comic book store.\n",
            "Leonard: Who would win in Batman vs Superman fight? Superman, Waluigi, Wonder Twins, Dr. Jekyll and Mrs. Whore?\n",
            "Sheldon: I’m sorry, Dr. Whore? I was thinking of putting a damper on the whole DC multiverse, not just the one where Superman is king.\n",
            "Leonard: Great, I’m going to start a new one.\n",
            "Sheldon: Oh, I think I saved that one for last. It’s called Jekyll and Hyde\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}