notebooks/Conditional-Text-Generation-with-GPT-2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conditional Text Generation with GPT-2\n",
    "\n",
    "See: https://huggingface.co/gpt2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import GPT2Tokenizer, GPT2LMHeadModel\n",
    "\n",
    "# Import textwrap library to display context\n",
    "import textwrap\n",
    "wrapper = textwrap.TextWrapper(width=80) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "MODEL_NAME = 'gpt2-large'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokenizer\n",
    "\n",
    "The texts are tokenized using a byte-level version of Byte Pair Encoding (BPE) (for unicode characters) and a vocabulary size of 50,257. The inputs are sequences of 1024 consecutive tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = GPT2Tokenizer.from_pretrained(MODEL_NAME)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[15496,   995,     0]])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample_text = \"Hello world!\"\n",
    "tokenizer.encode(sample_text, return_tensors='pt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of GPT2LMHeadModel were not initialized from the model checkpoint at gpt2-large and are newly initialized: ['h.0.attn.masked_bias', 'h.1.attn.masked_bias', 'h.2.attn.masked_bias', 'h.3.attn.masked_bias', 'h.4.attn.masked_bias', 'h.5.attn.masked_bias', 'h.6.attn.masked_bias', 'h.7.attn.masked_bias', 'h.8.attn.masked_bias', 'h.9.attn.masked_bias', 'h.10.attn.masked_bias', 'h.11.attn.masked_bias', 'h.12.attn.masked_bias', 'h.13.attn.masked_bias', 'h.14.attn.masked_bias', 'h.15.attn.masked_bias', 'h.16.attn.masked_bias', 'h.17.attn.masked_bias', 'h.18.attn.masked_bias', 'h.19.attn.masked_bias', 'h.20.attn.masked_bias', 'h.21.attn.masked_bias', 'h.22.attn.masked_bias', 'h.23.attn.masked_bias', 'h.24.attn.masked_bias', 'h.25.attn.masked_bias', 'h.26.attn.masked_bias', 'h.27.attn.masked_bias', 'h.28.attn.masked_bias', 'h.29.attn.masked_bias', 'h.30.attn.masked_bias', 'h.31.attn.masked_bias', 'h.32.attn.masked_bias', 'h.33.attn.masked_bias', 'h.34.attn.masked_bias', 'h.35.attn.masked_bias', 'lm_head.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    }
   ],
   "source": [
    "model = GPT2LMHeadModel.from_pretrained(MODEL_NAME)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Text generation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_text(initial_text, model, tokenizer, display=False):\n",
    "    # Generate text\n",
    "    encoded_input = tokenizer.encode(initial_text, return_tensors='pt')\n",
    "    outputs = model.generate(\n",
    "        encoded_input,\n",
    "        do_sample=True,\n",
    "        max_length=100,\n",
    "        top_k=20,\n",
    "        top_p=1.,\n",
    "        temperature=1,\n",
    "        num_return_sequences=1)\n",
    "    \n",
    "    generated_text = []\n",
    "    for i, token_id in enumerate(outputs):\n",
    "        generated_text.append(tokenizer.decode(token_id, skip_special_tokens=True))\n",
    "\n",
    "    generated_text = ''.join(generated_text)\n",
    "\n",
    "    # Display\n",
    "    if display:\n",
    "        print('='*21)\n",
    "        print('='*6, 'INITIAL', '='*6)\n",
    "        print(initial_text)\n",
    "\n",
    "        print('='*21)\n",
    "        print('='*8, 'TEXT', '='*7)\n",
    "        print(wrapper.fill(generated_text))\n",
    "    else:\n",
    "        return generated_text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to 50256 (first `eos_token_id`) to generate sequence\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=====================\n",
      "====== INITIAL ======\n",
      "Last week, many people reported that they saw a unicorn at the park.\n",
      "=====================\n",
      "======== TEXT =======\n",
      "Last week, many people reported that they saw a unicorn at the park. That,\n",
      "however, was a misidentification of a unicorn that was actually an antelope. It\n",
      "was not the first time that people have mistaken a unicorn for a unicorn, but\n",
      "it's one that people are not quite used to seeing.  But the confusion was caused\n",
      "by a misunderstanding about what the unicorn means.  A unicorn is a mythical\n",
      "creature who lives in the forests of the eastern Himalayas. It\n"
     ]
    }
   ],
   "source": [
    "initial_text = \"Last week, many people reported that they saw a unicorn at the park.\"\n",
    "generate_text(initial_text, model, tokenizer, display=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to 50256 (first `eos_token_id`) to generate sequence\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=====================\n",
      "====== INITIAL ======\n",
      "Q: What do you think about the death penalty? A: \n",
      "=====================\n",
      "======== TEXT =======\n",
      "Q: What do you think about the death penalty? A:  I am not an advocate for the\n",
      "death penalty. I am not someone who believes that this life should be taken and\n",
      "then taken over again, which of course is the case with every case when we see a\n",
      "murder. It is my belief that a fair penalty should be used, and that if the\n",
      "evidence was not sufficient to prove that the person committed the actual crime,\n",
      "then there should be a life sentence. Q: Do you\n"
     ]
    }
   ],
   "source": [
    "initial_text = \"Q: What do you think about the death penalty? A: \"\n",
    "generate_text(initial_text, model, tokenizer, display=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to 50256 (first `eos_token_id`) to generate sequence\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=====================\n",
      "====== INITIAL ======\n",
      "In the last months, a new virus spread worldwide and changed our daily lifes. This new virus, called\n",
      "=====================\n",
      "======== TEXT =======\n",
      "In the last months, a new virus spread worldwide and changed our daily lifes.\n",
      "This new virus, called Ebola, appeared in three countries, and has killed more\n",
      "than 3,000 people in the four countries: Guinea, Liberia and Sierra Leone. The\n",
      "virus is spread through direct contact with the fluids of infected persons,\n",
      "which can include body fluids, such as blood, sweat and tears…  It is also\n",
      "spread through contaminated food and water. When Ebola arrives in a country, a\n",
      "lot of people\n"
     ]
    }
   ],
   "source": [
    "initial_text = \"In the last months, a new virus spread worldwide and changed our daily lifes. This new virus, called\"\n",
    "generate_text(initial_text, model, tokenizer, display=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to 50256 (first `eos_token_id`) to generate sequence\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=====================\n",
      "====== INITIAL ======\n",
      "The RMS Titanic was visited by divers for the first time in 14 years.\n",
      "=====================\n",
      "======== TEXT =======\n",
      "The RMS Titanic was visited by divers for the first time in 14 years. The\n",
      "British vessel was damaged to the bow in 1912 and the ship was found to be\n",
      "completely submerged.  The Titanic is also the only ship that has sunk in\n",
      "international waters.  But, according to experts, the damage to the Titanic\n",
      "would have been even worse, if the ship had not been in the water. In fact,\n",
      "according to reports, the damage could have been caused by a catastrophic\n",
      "failure of the\n"
     ]
    }
   ],
   "source": [
    "initial_text = \"The RMS Titanic was visited by divers for the first time in 14 years.\"\n",
    "generate_text(initial_text, model, tokenizer, display=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}