GPT2-finetuning2-345M.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "GPT-2 Fine Tuning Notebook",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/jkraybill/gpt-2/blob/finetuning/GPT2-finetuning2-345M.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EHiwt3Ww7FF6",
        "colab_type": "text"
      },
      "source": [
        "To try out GPT-2, do this:\n",
        "\n",
        "- go to the \"Runtime\" menu and click \"Change runtime type\" and make sure this is a Python 3 notebook, running with GPU hardware acceleration.\n",
        "- use the \"Files\" section to the left to upload a text file called \"corpus.txt\" which contains all the text you want to train on.\n",
        "- run the steps below in order."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JE_fFgQ8a_Dd",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import os\n",
        "import json\n",
        "import random\n",
        "import re"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "etM-i8RwbcTH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!git clone https://github.com/jkraybill/gpt-2.git"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LVTyGrhsbdep",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "cd gpt-2"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "azu6KCOHbhIy",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!pip3 install -r requirements.txt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ecmuIWxFgFBz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!sh download_model.sh 345M"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-OhJ6ka_DD94",
        "colab_type": "text"
      },
      "source": [
        "The below step encodes your corpus into \"NPZ\" tokenized format for GPT-2.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZzHsNeMNenxR",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!PYTHONPATH=src ./encode.py --in-text ../train.txt --out-npz train.txt.npz --model_name 345M\n",
        "!PYTHONPATH=src ./encode.py --in-text ../val.txt --out-npz val.txt.npz --model_name 345M"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "94oLzOq23474",
        "colab_type": "text"
      },
      "source": [
        "Training is below. I usually get usable results with \"stop_after\" anywhere from 800 to 3000, but you can try going even higher. 800 steps takes only a few minutes.\n",
        "\n",
        "\"sample_every\" controls how often you get sample output from the trained model.\n",
        "\n",
        "\"save_every\" controls how often the model is saved.\n",
        "\n",
        "\"learning_rate\" is the AI learning rate. 0.00005 is the rate I've gotten the best results with, but I think most people are running with significantly higher rates, so you could try adjusting it."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QbBJ6JoufBGQ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!PYTHONPATH=src ./trainval.py --dataset train.txt.npz --valset val.txt.npz --sample_every=1000 --save_every=25 --learning_rate=0.00005 --stop_after=60000 --model_name=345M --batch_length=512"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GUa2LujW82cy",
        "colab_type": "text"
      },
      "source": [
        "The step below simply copies your trained model to the model directory, so the output will use your training. If you don't do this, you will be running against the trained GPT-2 model without your finetuning training."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mNXhOM22fNHJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!cp -r /content/gpt-2/checkpoint/run1/* /content/gpt-2/models/345M/"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Uvtv6NUj5eXU",
        "colab_type": "text"
      },
      "source": [
        "Run the below step to generate unconditional samples (i.e. \"dream mode\").\n",
        "\n",
        "\"top_k\" controls how many options to consider per word (the larger, the more \"diverse\" the output - anything from 1 to about 50 usually works, I think values around 10 are pretty good).\n",
        "\n",
        "\"temperature\" controls the sampling of the words, from 0 to 1 where 1 is the most \"random\".\n",
        "\n",
        "\"length\" controls the number of words in each sample output.\n",
        "\n",
        "This command will run continuously until you turn it off."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2csc2bZHfrXd",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!python3 src/generate_unconditional_samples.py --top_k 20 --temperature 0.8 --length=300 --model_name=345M"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IwZl7JAn6d5y",
        "colab_type": "text"
      },
      "source": [
        "Run the command below to run in interactive / \"completion\" mode. You will get a prompt; just type in whatever prompt text you want, and the model will attempt to complete it \"nsamples\" times.\n",
        "\n",
        "\"top_k\", \"length\", and \"temperature\" work as specified above."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ugmKkFv__NI_",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!python3 src/interactive_conditional_samples.py --top_k 1 --length=30 --temperature 0.1 --nsamples 3 --model_name=345M"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}