T5_evaluate_WMT.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "T5-evaluate-WMT.ipynb",
      "provenance": [],
      "authorship_tag": "ABX9TyOcjQrRjmpu6HWWqPnaqrlU",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/mrm8488/shared_colab_notebooks/blob/master/T5_evaluate_WMT.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GDHXGqBhMgHW",
        "colab_type": "text"
      },
      "source": [
        "***This script evaluates the multitask pre-trained checkpoint for ``t5-base`` (see paper [here](https://arxiv.org/pdf/1910.10683.pdf)) on the English to German WMT dataset. Please note that the results in the paper were attained using a model fine-tuned on translation, so that results will be worse here by approx. 1.5 BLEU points***\n",
        "\n",
        "### Intro\n",
        "\n",
        "This example shows how T5 (here the official [paper](https://arxiv.org/abs/1910.10683)) can be\n",
        "evaluated on the WMT English-German dataset.\n",
        "\n",
        "### Get the WMT Data\n",
        "\n",
        "To be able to reproduce the authors' results on WMT English to German, you first need to download \n",
        "the WMT14 en-de news datasets.\n",
        "Go on Stanford's official NLP [website](https://nlp.stanford.edu/projects/nmt/) and find \"newstest2013.en\" and \"newstest2013.de\" under WMT'14 English-German data or download the dataset directly via:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NnCPfGlCGW2Y",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%%bash\n",
        "curl -s https://nlp.stanford.edu/projects/nmt/data/wmt14.en-de/newstest2013.en > newstest2013.en\n",
        "curl -s https://nlp.stanford.edu/projects/nmt/data/wmt14.en-de/newstest2013.de > newstest2013.de"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fgybk5tMM1wu",
        "colab_type": "text"
      },
      "source": [
        "You should have 3000 sentence in each file. You can verify this by running:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6no1fQuLH7-m",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!wc -l newstest2013.en  # should give 3000"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1fGhrX0UM86a",
        "colab_type": "text"
      },
      "source": [
        "### Usage\n",
        "\n",
        "Let's check the longest and shortest sentence in our file to find reasonable decoding hyperparameters: \n",
        "\n",
        "Get the longest and shortest sentence:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "c8ausLLcID7u",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%%bash\n",
        "awk '{print NF}' newstest2013.en | sort -n | head -1 # shortest sentence has 1 word\n",
        "awk '{print NF}' newstest2013.en | sort -n | tail -1 # longest sentence has 106 words"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PiNI3MYDMtsn",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%%bash\n",
        "git clone https://github.com/huggingface/transformers.git\n",
        "pip install -q ./transformers\n",
        "pip install -q sacrebleu"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B1OkJz6UNMhT",
        "colab_type": "text"
      },
      "source": [
        "We will set our `max_length` to ~3 times the longest sentence and leave `min_length` to its default value of 0.\n",
        "We decode with beam search `num_beams=4` as proposed in the paper. Also as is common in beam search we set `early_stopping=True` and `length_penalty=2.0`.\n",
        "\n",
        "To create translation for each in dataset and get a final BLEU score, run:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HbNa2qFuKOS3",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!python /content/transformers/examples/translation/t5/evaluate_wmt.py \\\n",
        "  /content/newstest2013.en \\\n",
        "  newstest2013_de_translations.txt \\\n",
        "  /content/newstest2013.de \\\n",
        "  newsstest2013_en_de_bleu.txt \\\n",
        "  --batch_size 16"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wBnV3O20NdSw",
        "colab_type": "text"
      },
      "source": [
        "the default batch size, 16, fits in 16GB GPU memory, but may need to be adjusted to fit your system.\n",
        "\n",
        "### Where is the code?\n",
        "The core model is in `src/transformers/modeling_t5.py`. This directory only contains examples.\n",
        "\n",
        "### BLEU Scores\n",
        "\n",
        "The BLEU score is calculated using [sacrebleu](https://github.com/mjpost/sacreBLEU) by mjpost.\n",
        "To get the BLEU score we used "
      ]
    }
  ]
}