examples/Fine_tuning_AraT5.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A_kDLcjaRErP"
      },
      "source": [
        "#Fine-tuning UBC-NLP AraT5 models\n",
        "https://github.com/UBC-NLP/araT5"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K4wxJ06TSRad"
      },
      "source": [
        "##Install requirments"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "id": "YH2WCTBYSasB"
      },
      "outputs": [],
      "source": [
        "!pip install transformers accelerate nltk datasets regex sklearn sentencepiece protobuf fairscale sacrebleu rouge_score -qqq"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yIEH75xsSa9R"
      },
      "source": [
        "##Get seq2seq trainier from UBC-NLP/araT5 github"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "B7poTh9vRNUo",
        "outputId": "b73e203e-13a2-4e01-a620-44266ad83b6f"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "--2021-08-30 23:45:27--  https://raw.githubusercontent.com/UBC-NLP/araT5/main/examples/run_trainier_seq2seq_huggingface.py?token=AA4R7KKKCBPS5WX4BJSWNFTBG2OPK\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 36633 (36K) [text/plain]\n",
            "Saving to: ‘run_trainier_seq2seq_huggingface.py’\n",
            "\n",
            "\r          run_train   0%[                    ]       0  --.-KB/s               \rrun_trainier_seq2se 100%[===================>]  35.77K  --.-KB/s    in 0.001s  \n",
            "\n",
            "2021-08-30 23:45:28 (58.6 MB/s) - ‘run_trainier_seq2seq_huggingface.py’ saved [36633/36633]\n",
            "\n",
            "--2021-08-30 23:45:28--  https://raw.githubusercontent.com/UBC-NLP/araT5/main/examples/eval_squad.py?token=AA4R7KOBBXSPCDXRRHJ3RW3BG2RF4\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 7689 (7.5K) [text/plain]\n",
            "Saving to: ‘eval_squad.py’\n",
            "\n",
            "eval_squad.py       100%[===================>]   7.51K  --.-KB/s    in 0s      \n",
            "\n",
            "2021-08-30 23:45:28 (71.2 MB/s) - ‘eval_squad.py’ saved [7689/7689]\n",
            "\n"
          ]
        }
      ],
      "source": [
        "!wget https://raw.githubusercontent.com/UBC-NLP/araT5/main/examples/run_trainier_seq2seq_huggingface.py?token=AA4R7KKKCBPS5WX4BJSWNFTBG2OPK -O run_trainier_seq2seq_huggingface.py\n",
        "!wget https://raw.githubusercontent.com/UBC-NLP/araT5/main/examples/eval_squad.py?token=AA4R7KOBBXSPCDXRRHJ3RW3BG2RF4 -O eval_squad.py"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dYZSVidMTAzJ"
      },
      "source": [
        "##Fine-tuning AraT5 on title generation task\n",
        "In this example, we use **sample data from ARGEN title genration dataset** as an example for how to **fine-tunne AraT5 models**\n",
        "\n",
        "###Getting ARGEN title genration sample train and valid"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TYZh7q9USM45",
        "outputId": "3c76d26c-4a48-4a6d-be55-033dd2de82e3"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "--2021-08-30 23:45:30--  https://raw.githubusercontent.com/UBC-NLP/araT5/main/examples/ARGEn_title_genration_sample_train.tsv?token=AA4R7KKFIZA3WUAVHFI7AIDBG2QCE\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 2961751 (2.8M) [text/plain]\n",
            "Saving to: ‘ARGEn_title_genration_sample_train.tsv’\n",
            "\n",
            "ARGEn_title_genrati 100%[===================>]   2.82M  --.-KB/s    in 0.03s   \n",
            "\n",
            "2021-08-30 23:45:30 (93.2 MB/s) - ‘ARGEn_title_genration_sample_train.tsv’ saved [2961751/2961751]\n",
            "\n",
            "--2021-08-30 23:45:30--  https://raw.githubusercontent.com/UBC-NLP/araT5/main/examples/ARGEn_title_genration_sample_valid.tsv?token=AA4R7KLXSAQEF2HUWL52Y53BG2QEE\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.111.133, 185.199.108.133, 185.199.109.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.111.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 288031 (281K) [text/plain]\n",
            "Saving to: ‘ARGEn_title_genration_sample_valid.tsv’\n",
            "\n",
            "ARGEn_title_genrati 100%[===================>] 281.28K  --.-KB/s    in 0.01s   \n",
            "\n",
            "2021-08-30 23:45:30 (18.7 MB/s) - ‘ARGEn_title_genration_sample_valid.tsv’ saved [288031/288031]\n",
            "\n"
          ]
        }
      ],
      "source": [
        "!wget https://raw.githubusercontent.com/UBC-NLP/araT5/main/examples/ARGEn_title_genration_sample_train.tsv?token=AA4R7KKFIZA3WUAVHFI7AIDBG2QCE -O ARGEn_title_genration_sample_train.tsv\n",
        "!wget https://raw.githubusercontent.com/UBC-NLP/araT5/main/examples/ARGEn_title_genration_sample_valid.tsv?token=AA4R7KLXSAQEF2HUWL52Y53BG2QEE -O ARGEn_title_genration_sample_valid.tsv"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2iYH2PcUjalj"
      },
      "source": [
        "###Training the model\n",
        "**To obtain the best results, you should train the model for 22 epochs with full data (train, valid, and test)**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "SE2bzzCbVZcZ"
      },
      "outputs": [],
      "source": [
        "!python run_trainier_seq2seq_huggingface.py \\\n",
        "        --learning_rate 5e-5 \\\n",
        "        --max_target_length 128 --max_source_length 128 \\\n",
        "        --per_device_train_batch_size 8 --per_device_eval_batch_size 8 \\\n",
        "        --model_name_or_path \"UBC-NLP/AraT5-base\" \\\n",
        "        --output_dir \"/content/AraT5_FT_title_generation\" --overwrite_output_dir \\\n",
        "        --num_train_epochs 22 \\\n",
        "        --train_file \"/content/ARGEn_title_genration_sample_train.tsv\" \\\n",
        "        --validation_file \"/content/ARGEn_title_genration_sample_valid.tsv\" \\\n",
        "        --task \"title_generation\" --text_column \"document\" --summary_column \"title\" \\\n",
        "        --load_best_model_at_end --metric_for_best_model \"eval_bleu\" --greater_is_better True --evaluation_strategy step --logging_strategy step --predict_with_generate \\\n",
        "        --do_train --do_eval"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vPTUhSegjntV"
      },
      "source": [
        "###Generating titles\n",
        "(1) using trained checkpoint"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "y28duyiOpGLa"
      },
      "outputs": [],
      "source": [
        "!python run_trainier_seq2seq_huggingface.py \\\n",
        "        --learning_rate 5e-5 \\\n",
        "        --max_target_length 128 --max_source_length 128 \\\n",
        "        --per_device_train_batch_size 8 --per_device_eval_batch_size 8 \\\n",
        "        --model_name_or_path \"/content/AraT5_FT_title_generation\" \\\n",
        "        --output_dir \"/content/AraT5_FT_title_generation\" \\\n",
        "        --train_file \"/content/ARGEn_title_genration_sample_train.tsv\" \\\n",
        "        --validation_file \"/content/ARGEn_title_genration_sample_valid.tsv\" \\\n",
        "        --test_file \"/content/ARGEn_title_genration_sample_valid.tsv\" \\\n",
        "        --metric_for_best_model \"eval_bleu\" \\\n",
        "        --task \"title_generation\" --text_column \"document\" --summary_column \"title\" \\\n",
        "        --predict_with_generate --do_predict"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CPHEmbRErBpd"
      },
      "source": [
        "(2) using **UBC-NLP/AraT5-base-title-generation** model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 62,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "aGkuAu7zrB5E",
        "outputId": "b710ccb3-3661-4e1d-c561-722d73235d67"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "last_checkpoint None\n",
            "08/31/2021 00:49:07 - WARNING - __main__ -   Process rank: -1, device: cuda:0, n_gpu: 1distributed training: False, 16-bits training: False\n",
            "08/31/2021 00:49:07 - INFO - __main__ -   Training/evaluation parameters Seq2SeqTrainingArguments(output_dir='/content/AraT5-base-title-generation_output', overwrite_output_dir=False, do_train=False, do_eval=None, do_predict=True, evaluation_strategy=<IntervalStrategy.NO: 'no'>, prediction_loss_only=False, per_device_train_batch_size=8, per_device_eval_batch_size=8, per_gpu_train_batch_size=None, per_gpu_eval_batch_size=None, gradient_accumulation_steps=1, eval_accumulation_steps=None, learning_rate=5e-05, weight_decay=0.0, adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-08, max_grad_norm=1.0, num_train_epochs=3.0, max_steps=-1, lr_scheduler_type=<SchedulerType.LINEAR: 'linear'>, warmup_ratio=0.0, warmup_steps=0, logging_dir='runs/Aug31_00-49-07_ca4daf48a05c', logging_strategy=<IntervalStrategy.STEPS: 'steps'>, logging_first_step=False, logging_steps=500, save_strategy=<IntervalStrategy.STEPS: 'steps'>, save_steps=500, save_total_limit=None, no_cuda=False, seed=42, fp16=False, fp16_opt_level='O1', fp16_backend='auto', fp16_full_eval=False, local_rank=-1, tpu_num_cores=None, tpu_metrics_debug=False, debug=False, dataloader_drop_last=False, eval_steps=500, dataloader_num_workers=0, past_index=-1, run_name='/content/AraT5-base-title-generation_output', disable_tqdm=False, remove_unused_columns=True, label_names=None, load_best_model_at_end=False, metric_for_best_model='eval_bleu', greater_is_better=True, ignore_data_skip=False, sharded_ddp=[], deepspeed=None, label_smoothing_factor=0.0, adafactor=False, group_by_length=False, length_column_name='length', report_to=['tensorboard'], ddp_find_unused_parameters=None, dataloader_pin_memory=True, skip_memory_metrics=False, mp_parameters='', sortish_sampler=False, predict_with_generate=True)\n",
            "[INFO] loading from TSV\n",
            "08/31/2021 00:49:08 - WARNING - datasets.builder -   Using custom data configuration default-08d5d8b9683548db\n",
            "08/31/2021 00:49:08 - WARNING - datasets.builder -   Reusing dataset csv (/temp/AraT5_cache_dir/csv/default-08d5d8b9683548db/0.0.0/9144e0a4e8435090117cea53e6c7537173ef2304525df4a077c435d8ee7828ff)\n",
            "https://huggingface.co/UBC-NLP/AraT5-base-title-generation/resolve/main/config.json not found in cache or force_download set to True, downloading to /temp/AraT5_cache_dir/tmpacpv79py\n",
            "Downloading: 100% 687/687 [00:00<00:00, 723kB/s]\n",
            "storing https://huggingface.co/UBC-NLP/AraT5-base-title-generation/resolve/main/config.json in cache at /temp/AraT5_cache_dir/0e75043f52303252d9360339a423fdf44c6b35a20d1be95dd9f1fb88bc2f45a6.69ca0ad3e2b8c76da2f3ac6da0317a8259a2cc899d85ba72c91b5ff4db80bf09\n",
            "creating metadata file for /temp/AraT5_cache_dir/0e75043f52303252d9360339a423fdf44c6b35a20d1be95dd9f1fb88bc2f45a6.69ca0ad3e2b8c76da2f3ac6da0317a8259a2cc899d85ba72c91b5ff4db80bf09\n",
            "loading configuration file https://huggingface.co/UBC-NLP/AraT5-base-title-generation/resolve/main/config.json from cache at /temp/AraT5_cache_dir/0e75043f52303252d9360339a423fdf44c6b35a20d1be95dd9f1fb88bc2f45a6.69ca0ad3e2b8c76da2f3ac6da0317a8259a2cc899d85ba72c91b5ff4db80bf09\n",
            "Model config T5Config {\n",
            "  \"_name_or_path\": \"/project/6007993/DataBank/our_models/title_generation_AraT5_base\",\n",
            "  \"architectures\": [\n",
            "    \"T5ForConditionalGeneration\"\n",
            "  ],\n",
            "  \"d_ff\": 2048,\n",
            "  \"d_kv\": 64,\n",
            "  \"d_model\": 768,\n",
            "  \"decoder_start_token_id\": 0,\n",
            "  \"dropout_rate\": 0.1,\n",
            "  \"eos_token_id\": 1,\n",
            "  \"feed_forward_proj\": \"gated-gelu\",\n",
            "  \"gradient_checkpointing\": false,\n",
            "  \"initializer_factor\": 1.0,\n",
            "  \"is_encoder_decoder\": true,\n",
            "  \"layer_norm_epsilon\": 1e-06,\n",
            "  \"model_type\": \"t5\",\n",
            "  \"num_decoder_layers\": 12,\n",
            "  \"num_heads\": 12,\n",
            "  \"num_layers\": 12,\n",
            "  \"output_past\": true,\n",
            "  \"pad_token_id\": 0,\n",
            "  \"relative_attention_num_buckets\": 32,\n",
            "  \"transformers_version\": \"4.5.1\",\n",
            "  \"use_cache\": true,\n",
            "  \"vocab_size\": 110080\n",
            "}\n",
            "\n",
            "loading configuration file https://huggingface.co/UBC-NLP/AraT5-base-title-generation/resolve/main/config.json from cache at /temp/AraT5_cache_dir/0e75043f52303252d9360339a423fdf44c6b35a20d1be95dd9f1fb88bc2f45a6.69ca0ad3e2b8c76da2f3ac6da0317a8259a2cc899d85ba72c91b5ff4db80bf09\n",
            "Model config T5Config {\n",
            "  \"_name_or_path\": \"/project/6007993/DataBank/our_models/title_generation_AraT5_base\",\n",
            "  \"architectures\": [\n",
            "    \"T5ForConditionalGeneration\"\n",
            "  ],\n",
            "  \"d_ff\": 2048,\n",
            "  \"d_kv\": 64,\n",
            "  \"d_model\": 768,\n",
            "  \"decoder_start_token_id\": 0,\n",
            "  \"dropout_rate\": 0.1,\n",
            "  \"eos_token_id\": 1,\n",
            "  \"feed_forward_proj\": \"gated-gelu\",\n",
            "  \"gradient_checkpointing\": false,\n",
            "  \"initializer_factor\": 1.0,\n",
            "  \"is_encoder_decoder\": true,\n",
            "  \"layer_norm_epsilon\": 1e-06,\n",
            "  \"model_type\": \"t5\",\n",
            "  \"num_decoder_layers\": 12,\n",
            "  \"num_heads\": 12,\n",
            "  \"num_layers\": 12,\n",
            "  \"output_past\": true,\n",
            "  \"pad_token_id\": 0,\n",
            "  \"relative_attention_num_buckets\": 32,\n",
            "  \"transformers_version\": \"4.5.1\",\n",
            "  \"use_cache\": true,\n",
            "  \"vocab_size\": 110080\n",
            "}\n",
            "\n",
            "https://huggingface.co/UBC-NLP/AraT5-base-title-generation/resolve/main/spiece.model not found in cache or force_download set to True, downloading to /temp/AraT5_cache_dir/tmpnrurvjur\n",
            "Downloading: 100% 2.44M/2.44M [00:00<00:00, 3.80MB/s]\n",
            "storing https://huggingface.co/UBC-NLP/AraT5-base-title-generation/resolve/main/spiece.model in cache at /temp/AraT5_cache_dir/3b9209e7a0b5d9b0bd3f332650419a449b9e7bbc38341ac8e5c6acaabbe92fd9.a28e4cb40a178b4c9dc2f864056ef84f0116dff1de5b2f36f258a99dad119fa7\n",
            "creating metadata file for /temp/AraT5_cache_dir/3b9209e7a0b5d9b0bd3f332650419a449b9e7bbc38341ac8e5c6acaabbe92fd9.a28e4cb40a178b4c9dc2f864056ef84f0116dff1de5b2f36f258a99dad119fa7\n",
            "https://huggingface.co/UBC-NLP/AraT5-base-title-generation/resolve/main/special_tokens_map.json not found in cache or force_download set to True, downloading to /temp/AraT5_cache_dir/tmpnm9fiwx3\n",
            "Downloading: 100% 1.79k/1.79k [00:00<00:00, 1.43MB/s]\n",
            "storing https://huggingface.co/UBC-NLP/AraT5-base-title-generation/resolve/main/special_tokens_map.json in cache at /temp/AraT5_cache_dir/5fd6ad7f171724e1cb52e70af7a0639c977dc6908488e757e955b5c98d5d248f.c94798918c92ded6aeef2d2f0e666d2cc4145eca1aa6e1336fde07f2e13e2f46\n",
            "creating metadata file for /temp/AraT5_cache_dir/5fd6ad7f171724e1cb52e70af7a0639c977dc6908488e757e955b5c98d5d248f.c94798918c92ded6aeef2d2f0e666d2cc4145eca1aa6e1336fde07f2e13e2f46\n",
            "https://huggingface.co/UBC-NLP/AraT5-base-title-generation/resolve/main/tokenizer_config.json not found in cache or force_download set to True, downloading to /temp/AraT5_cache_dir/tmp7jdaxq6i\n",
            "Downloading: 100% 2.00k/2.00k [00:00<00:00, 1.66MB/s]\n",
            "storing https://huggingface.co/UBC-NLP/AraT5-base-title-generation/resolve/main/tokenizer_config.json in cache at /temp/AraT5_cache_dir/78a52f0d756609895805b9bd8896cbc4e3bf05a806f926be9a953fe84eab1373.b3b0277a289ef2f698a5b0198c90a24f1c671f1702ad56a9169e16f42afd2f6e\n",
            "creating metadata file for /temp/AraT5_cache_dir/78a52f0d756609895805b9bd8896cbc4e3bf05a806f926be9a953fe84eab1373.b3b0277a289ef2f698a5b0198c90a24f1c671f1702ad56a9169e16f42afd2f6e\n",
            "loading file https://huggingface.co/UBC-NLP/AraT5-base-title-generation/resolve/main/spiece.model from cache at /temp/AraT5_cache_dir/3b9209e7a0b5d9b0bd3f332650419a449b9e7bbc38341ac8e5c6acaabbe92fd9.a28e4cb40a178b4c9dc2f864056ef84f0116dff1de5b2f36f258a99dad119fa7\n",
            "loading file https://huggingface.co/UBC-NLP/AraT5-base-title-generation/resolve/main/tokenizer.json from cache at None\n",
            "loading file https://huggingface.co/UBC-NLP/AraT5-base-title-generation/resolve/main/added_tokens.json from cache at None\n",
            "loading file https://huggingface.co/UBC-NLP/AraT5-base-title-generation/resolve/main/special_tokens_map.json from cache at /temp/AraT5_cache_dir/5fd6ad7f171724e1cb52e70af7a0639c977dc6908488e757e955b5c98d5d248f.c94798918c92ded6aeef2d2f0e666d2cc4145eca1aa6e1336fde07f2e13e2f46\n",
            "loading file https://huggingface.co/UBC-NLP/AraT5-base-title-generation/resolve/main/tokenizer_config.json from cache at /temp/AraT5_cache_dir/78a52f0d756609895805b9bd8896cbc4e3bf05a806f926be9a953fe84eab1373.b3b0277a289ef2f698a5b0198c90a24f1c671f1702ad56a9169e16f42afd2f6e\n",
            "https://huggingface.co/UBC-NLP/AraT5-base-title-generation/resolve/main/pytorch_model.bin not found in cache or force_download set to True, downloading to /temp/AraT5_cache_dir/tmpk734w14l\n",
            "Downloading: 100% 1.13G/1.13G [00:28<00:00, 40.3MB/s]\n",
            "storing https://huggingface.co/UBC-NLP/AraT5-base-title-generation/resolve/main/pytorch_model.bin in cache at /temp/AraT5_cache_dir/4e59c9dc644f76f0fc67afbfc97a8ac2f37f91e1f2b17c20a4583bdd083dfa06.5d7df83a0992e5736c9095ad1a59e07547dada1969a63bf613ba2456c062f213\n",
            "creating metadata file for /temp/AraT5_cache_dir/4e59c9dc644f76f0fc67afbfc97a8ac2f37f91e1f2b17c20a4583bdd083dfa06.5d7df83a0992e5736c9095ad1a59e07547dada1969a63bf613ba2456c062f213\n",
            "loading weights file https://huggingface.co/UBC-NLP/AraT5-base-title-generation/resolve/main/pytorch_model.bin from cache at /temp/AraT5_cache_dir/4e59c9dc644f76f0fc67afbfc97a8ac2f37f91e1f2b17c20a4583bdd083dfa06.5d7df83a0992e5736c9095ad1a59e07547dada1969a63bf613ba2456c062f213\n",
            "All model checkpoint weights were used when initializing T5ForConditionalGeneration.\n",
            "\n",
            "All the weights of T5ForConditionalGeneration were initialized from the model checkpoint at UBC-NLP/AraT5-base-title-generation.\n",
            "If your task is similar to the task the model of the checkpoint was trained on, you can already use T5ForConditionalGeneration for predictions without further training.\n",
            "100% 1/1 [00:00<00:00,  6.27ba/s]\n",
            "[INFO] evlaute using  bleu score task name: title_generation\n",
            "[INFO] early_stopping_num= 20\n",
            "08/31/2021 00:49:58 - INFO - __main__ -   *** Test ***\n",
            "***** Running Prediction *****\n",
            "  Num examples = 100\n",
            "  Batch size = 8\n",
            "100% 13/13 [00:20<00:00,  1.58s/it]>>>>>>>>> bleu\n",
            "08/31/2021 00:50:21 - INFO - __main__ -   ***** test metrics *****\n",
            "08/31/2021 00:50:21 - INFO - __main__ -     init_mem_cpu_alloc_delta  =                       366MB\n",
            "08/31/2021 00:50:21 - INFO - __main__ -     init_mem_cpu_peaked_delta =                         0MB\n",
            "08/31/2021 00:50:21 - INFO - __main__ -     init_mem_gpu_alloc_delta  =                      1081MB\n",
            "08/31/2021 00:50:21 - INFO - __main__ -     init_mem_gpu_peaked_delta =                         0MB\n",
            "08/31/2021 00:50:21 - INFO - __main__ -     test_bleu                 =                     25.8441\n",
            "08/31/2021 00:50:21 - INFO - __main__ -     test_check_point          = AraT5-base-title-generation\n",
            "08/31/2021 00:50:21 - INFO - __main__ -     test_gen_len              =                       12.94\n",
            "08/31/2021 00:50:21 - INFO - __main__ -     test_loss                 =                      2.2615\n",
            "08/31/2021 00:50:21 - INFO - __main__ -     test_mem_cpu_alloc_delta  =                        14MB\n",
            "08/31/2021 00:50:21 - INFO - __main__ -     test_mem_cpu_peaked_delta =                         0MB\n",
            "08/31/2021 00:50:21 - INFO - __main__ -     test_mem_gpu_alloc_delta  =                         0MB\n",
            "08/31/2021 00:50:21 - INFO - __main__ -     test_mem_gpu_peaked_delta =                       316MB\n",
            "08/31/2021 00:50:21 - INFO - __main__ -     test_runtime              =                  0:00:22.92\n",
            "08/31/2021 00:50:21 - INFO - __main__ -     test_samples              =                         100\n",
            "08/31/2021 00:50:21 - INFO - __main__ -     test_samples_per_second   =                       4.363\n",
            "100% 13/13 [00:21<00:00,  1.63s/it]\n"
          ]
        }
      ],
      "source": [
        "\n",
        "!python run_trainier_seq2seq_huggingface.py \\\n",
        "        --learning_rate 5e-5 \\\n",
        "        --max_target_length 128 --max_source_length 128 \\\n",
        "        --per_device_train_batch_size 8 --per_device_eval_batch_size 8 \\\n",
        "        --model_name_or_path \"UBC-NLP/AraT5-base-title-generation\" \\\n",
        "        --output_dir \"/content/AraT5-base-title-generation_output\" \\\n",
        "        --train_file \"/content/ARGEn_title_genration_sample_train.tsv\" \\\n",
        "        --validation_file \"/content/ARGEn_title_genration_sample_valid.tsv\" \\\n",
        "        --test_file \"/content/ARGEn_title_genration_sample_valid.tsv\" \\\n",
        "        --metric_for_best_model \"eval_bleu\" \\\n",
        "        --task \"title_generation\" --text_column \"document\" --summary_column \"title\" \\\n",
        "        --predict_with_generate --do_predict"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 63,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XpzjjugzrnVe",
        "outputId": "20ab5681-a698-471b-c0b7-40e8537498d4"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "وزارة النقل: تنفيذ 3 آلاف كلم من الطرق والجسور بالولايات الثلاث\n",
            "البنوك السودانية تتوقع آثار إيجابية على الاقتصاد والمواطن بعد قرار البنك المركزي\n",
            "الغرفة النقابية الوطنية للباعثين العقاريين: ارتفاع أسعار العقارات وتوقعات بارتفاع في الأسعار\n",
            "أسعارالعملات الاجنبية مقابل الجنيه السوداني ليوم الاثنين الموافق24سبتمبر2018\n",
            "الإسكان: تخصيص 300 ألف شقة لمحدودى الدخل\n",
            "الذهب يهبط لأدنى مستوى في 3 أشهر ونصف\n",
            "بسبب خسائر فادحة تكبدتهاهم خلال أسبوع فقط مصالح الجمارك التونسية تشرع في حملة لمحاربة التهريب على الحدود الجزائرية\n",
            "الهيئة الملكية بالجبيل توقع مذكرات تفاهم مع شركات عالمية\n",
            "البرلمان يوافق على تصدير إناث الثروة الحيوانية\n",
            "وزير المالية: التمديد 3 سنوات في منظومة الامتيازات لن يخلق سابقة\n"
          ]
        }
      ],
      "source": [
        "#top 10 titles genrated from our full finetuned model (UBC-NLP/AraT5-base-title-generation)\n",
        "!head -10 /content/AraT5-base-title-generation_output/test_preds_seq2seq.txt"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "XF0ZUWnrgN0G"
      },
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [],
      "name": "Fine-tuning AraT5.ipynb",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}