notebook/finetune-to-livedoor-corpus.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Finetuning of the pretrained Japanese BERT model\n",
    "\n",
    "Finetune the pretrained model to solve multi-class classification problems.  \n",
    "This notebook requires the following objects:\n",
    "- trained sentencepiece model (model and vocab files)\n",
    "- pretraiend Japanese BERT model\n",
    "\n",
    "Dataset is livedoor ニュースコーパス in https://www.rondhuit.com/download.html.  \n",
    "We make test:dev:train = 2:2:6 datasets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Results:\n",
    "\n",
    "- Full training data\n",
    "  - BERT with SentencePiece\n",
    "    ```\n",
    "                    precision    recall  f1-score   support\n",
    "\n",
    "    dokujo-tsushin       0.98      0.94      0.96       178\n",
    "      it-life-hack       0.96      0.97      0.96       172\n",
    "     kaden-channel       0.99      0.98      0.99       176\n",
    "    livedoor-homme       0.98      0.88      0.93        95\n",
    "       movie-enter       0.96      0.99      0.98       158\n",
    "            peachy       0.94      0.98      0.96       174\n",
    "              smax       0.98      0.99      0.99       167\n",
    "      sports-watch       0.98      1.00      0.99       190\n",
    "        topic-news       0.99      0.98      0.98       163\n",
    "\n",
    "         micro avg       0.97      0.97      0.97      1473\n",
    "         macro avg       0.97      0.97      0.97      1473\n",
    "      weighted avg       0.97      0.97      0.97      1473\n",
    "    ```\n",
    "  - sklearn GradientBoostingClassifier with MeCab\n",
    "    ```\n",
    "                      precision    recall  f1-score   support\n",
    "\n",
    "    dokujo-tsushin       0.89      0.86      0.88       178\n",
    "      it-life-hack       0.91      0.90      0.91       172\n",
    "     kaden-channel       0.90      0.94      0.92       176\n",
    "    livedoor-homme       0.79      0.74      0.76        95\n",
    "       movie-enter       0.93      0.96      0.95       158\n",
    "            peachy       0.87      0.92      0.89       174\n",
    "              smax       0.99      1.00      1.00       167\n",
    "      sports-watch       0.93      0.98      0.96       190\n",
    "        topic-news       0.96      0.86      0.91       163\n",
    "\n",
    "         micro avg       0.92      0.92      0.92      1473\n",
    "         macro avg       0.91      0.91      0.91      1473\n",
    "      weighted avg       0.92      0.92      0.91      1473\n",
    "    ```\n",
    "\n",
    "- Small training data (1/5 of full training data)\n",
    "  - BERT with SentencePiece\n",
    "    ```\n",
    "                    precision    recall  f1-score   support\n",
    "\n",
    "    dokujo-tsushin       0.97      0.87      0.92       178\n",
    "      it-life-hack       0.86      0.86      0.86       172\n",
    "     kaden-channel       0.95      0.94      0.95       176\n",
    "    livedoor-homme       0.82      0.82      0.82        95\n",
    "       movie-enter       0.97      0.99      0.98       158\n",
    "            peachy       0.89      0.95      0.92       174\n",
    "              smax       0.94      0.96      0.95       167\n",
    "      sports-watch       0.97      0.97      0.97       190\n",
    "        topic-news       0.94      0.94      0.94       163\n",
    "\n",
    "         micro avg       0.93      0.93      0.93      1473\n",
    "         macro avg       0.92      0.92      0.92      1473\n",
    "      weighted avg       0.93      0.93      0.93      1473\n",
    "    ```\n",
    "  - sklearn GradientBoostingClassifier with MeCab\n",
    "    ```\n",
    "                    precision    recall  f1-score   support\n",
    "\n",
    "    dokujo-tsushin       0.82      0.71      0.76       178\n",
    "      it-life-hack       0.86      0.88      0.87       172\n",
    "     kaden-channel       0.91      0.87      0.89       176\n",
    "    livedoor-homme       0.67      0.63      0.65        95\n",
    "       movie-enter       0.87      0.95      0.91       158\n",
    "            peachy       0.70      0.78      0.73       174\n",
    "              smax       1.00      1.00      1.00       167\n",
    "      sports-watch       0.87      0.95      0.91       190\n",
    "        topic-news       0.92      0.82      0.87       163\n",
    "\n",
    "         micro avg       0.85      0.85      0.85      1473\n",
    "         macro avg       0.85      0.84      0.84      1473\n",
    "      weighted avg       0.86      0.85      0.85      1473\n",
    "    ```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import configparser\n",
    "import glob\n",
    "import os\n",
    "import pandas as pd\n",
    "import subprocess\n",
    "import sys\n",
    "import tarfile \n",
    "from urllib.request import urlretrieve\n",
    "\n",
    "CURDIR = os.getcwd()\n",
    "CONFIGPATH = os.path.join(CURDIR, os.pardir, 'config.ini')\n",
    "config = configparser.ConfigParser()\n",
    "config.read(CONFIGPATH)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data preparing\n",
    "\n",
    "You need execute the following cells just once."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "FILEURL = config['FINETUNING-DATA']['FILEURL']\n",
    "FILEPATH = config['FINETUNING-DATA']['FILEPATH']\n",
    "EXTRACTDIR = config['FINETUNING-DATA']['TEXTDIR']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download and unzip data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "urlretrieve(FILEURL, FILEPATH)\n",
    "\n",
    "mode = \"r:gz\"\n",
    "tar = tarfile.open(FILEPATH, mode) \n",
    "tar.extractall(EXTRACTDIR) \n",
    "tar.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data preprocessing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def extract_txt(filename):\n",
    "    with open(filename) as text_file:\n",
    "        # 0: URL, 1: timestamp\n",
    "        text = text_file.readlines()[2:]\n",
    "        text = [sentence.strip() for sentence in text]\n",
    "        text = list(filter(lambda line: line != '', text))\n",
    "        return ''.join(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "categories = [ \n",
    "    name for name \n",
    "    in os.listdir( os.path.join(EXTRACTDIR, \"text\") ) \n",
    "    if os.path.isdir( os.path.join(EXTRACTDIR, \"text\", name) ) ]\n",
    "\n",
    "categories = sorted(categories)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "categories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "table = str.maketrans({\n",
    "    '\\n': '',\n",
    "    '\\t': '　',\n",
    "    '\\r': '',\n",
    "})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "all_text = []\n",
    "all_label = []\n",
    "\n",
    "for cat in categories:\n",
    "    files = glob.glob(os.path.join(EXTRACTDIR, \"text\", cat, \"{}*.txt\".format(cat)))\n",
    "    files = sorted(files)\n",
    "    body = [ extract_txt(elem).translate(table) for elem in files ]\n",
    "    label = [cat] * len(body)\n",
    "    \n",
    "    all_text.extend(body)\n",
    "    all_label.extend(label)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame({'text' : all_text, 'label' : all_label})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df.sample(frac=1, random_state=23).reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save data as tsv files.  \n",
    "test:dev:train = 2:2:6. To check the usability of finetuning, we also prepare sampled training data (1/5 of full training data)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[:len(df) // 5].to_csv( os.path.join(EXTRACTDIR, \"test.tsv\"), sep='\\t', index=False)\n",
    "df[len(df) // 5:len(df)*2 // 5].to_csv( os.path.join(EXTRACTDIR, \"dev.tsv\"), sep='\\t', index=False)\n",
    "df[len(df)*2 // 5:].to_csv( os.path.join(EXTRACTDIR, \"train.tsv\"), sep='\\t', index=False)\n",
    "\n",
    "### 1/5 of full training data.\n",
    "# df[:len(df) // 5].to_csv( os.path.join(EXTRACTDIR, \"test.tsv\"), sep='\\t', index=False)\n",
    "# df[len(df) // 5:len(df)*2 // 5].to_csv( os.path.join(EXTRACTDIR, \"dev.tsv\"), sep='\\t', index=False)\n",
    "# df[len(df)*2 // 5:].sample(frac=0.2, random_state=23).to_csv( os.path.join(EXTRACTDIR, \"train.tsv\"), sep='\\t', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Finetune pre-trained model\n",
    "\n",
    "It will take a lot of hours to execute the following cells on CPU environment.  \n",
    "You can also use colab to recieve the power of TPU. You need to uplode the created data onto your GCS bucket.\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1zZH2GWe0U-7GjJ2w2duodFfEUptvHjcx)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PRETRAINED_MODEL_PATH = '../model/model.ckpt-1400000'\n",
    "FINETUNE_OUTPUT_DIR = '../model/livedoor_output'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "# It will take many hours on CPU environment.\n",
    "\n",
    "!python3 ../src/run_classifier.py \\\n",
    "  --task_name=livedoor \\\n",
    "  --do_train=true \\\n",
    "  --do_eval=true \\\n",
    "  --data_dir=../data/livedoor \\\n",
    "  --model_file=../model/wiki-ja.model \\\n",
    "  --vocab_file=../model/wiki-ja.vocab \\\n",
    "  --init_checkpoint={PRETRAINED_MODEL_PATH} \\\n",
    "  --max_seq_length=512 \\\n",
    "  --train_batch_size=4 \\\n",
    "  --learning_rate=2e-5 \\\n",
    "  --num_train_epochs=10 \\\n",
    "  --output_dir={FINETUNE_OUTPUT_DIR}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Predict using the finetuned model\n",
    "\n",
    "Let's predict test data using the finetuned model.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "# It will take many hours on CPU environment.\n",
    "\n",
    "!python3 ../src/run_classifier.py \\\n",
    "  --task_name=livedoor \\\n",
    "  --do_predict=true \\\n",
    "  --data_dir=../data/livedoor \\\n",
    "  --model_file=../model/wiki-ja.model \\\n",
    "  --vocab_file=../model/wiki-ja.vocab \\\n",
    "  --output_dir={FINETUNE_OUTPUT_DIR}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.append(\"../src\")\n",
    "\n",
    "from run_classifier import LivedoorProcessor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "processor = LivedoorProcessor()\n",
    "label_list = processor.get_labels()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result = pd.read_csv(FINETUNE_OUTPUT_DIR+\"/test_results.tsv\", sep='\\t', header=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read test data set and add prediction results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_df = pd.read_csv(\"../data/livedoor/test.tsv\", sep='\\t')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_df['predict'] = [ label_list[idx] for idx in result.idxmax(axis=1) ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sum( test_df['label'] == test_df['predict'] ) / len(test_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A littel more detailed check using `sklearn.metrics`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install scikit-learn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import classification_report\n",
    "from sklearn.metrics import confusion_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(classification_report(test_df['label'], test_df['predict']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(confusion_matrix(test_df['label'], test_df['predict']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Simple baseline model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from sklearn.metrics import classification_report\n",
    "from sklearn.metrics import confusion_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df = pd.read_csv(\"../data/livedoor/train.tsv\", sep='\\t')\n",
    "dev_df = pd.read_csv(\"../data/livedoor/dev.tsv\", sep='\\t')\n",
    "test_df = pd.read_csv(\"../data/livedoor/test.tsv\", sep='\\t')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!apt-get install -q -y mecab libmecab-dev mecab-ipadic mecab-ipadic-utf8"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install mecab-python3==0.7"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.ensemble import GradientBoostingClassifier\n",
    "import MeCab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m = MeCab.Tagger(\"-Owakati\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_dev_df = pd.concat([train_df, dev_df])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_dev_xs = train_dev_df['text'].apply(lambda x: m.parse(x))\n",
    "train_dev_ys = train_dev_df['label']\n",
    "\n",
    "test_xs = test_df['text'].apply(lambda x: m.parse(x))\n",
    "test_ys = test_df['label']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vectorizer = TfidfVectorizer(max_features=750)\n",
    "train_dev_xs_ = vectorizer.fit_transform(train_dev_xs)\n",
    "test_xs_ = vectorizer.transform(test_xs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following set up is not exactly identical to that of BERT because inside Classifier it uses `train_test_split` with shuffle.  \n",
    "In addition, parameters are not well tuned, however, we think it's enough to check the power of BERT."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "model = GradientBoostingClassifier(n_estimators=200,\n",
    "                                   validation_fraction=len(dev_df)/len(train_df),\n",
    "                                   n_iter_no_change=5,\n",
    "                                   tol=0.01,\n",
    "                                   random_state=23)\n",
    "\n",
    "### 1/5 of full training data.\n",
    "# model = GradientBoostingClassifier(n_estimators=200,\n",
    "#                                    validation_fraction=len(dev_df)/len(train_df),\n",
    "#                                    n_iter_no_change=5,\n",
    "#                                    tol=0.01,\n",
    "#                                    random_state=23)\n",
    "\n",
    "model.fit(train_dev_xs_, train_dev_ys)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(classification_report(test_ys, model.predict(test_xs_)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(confusion_matrix(test_ys, model.predict(test_xs_)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}