02-pipeline/pipeline-finetuning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "13a556a4-c49c-497b-aa04-984872017906",
   "metadata": {},
   "source": [
    "# Pipeline for Fine-Tuning Your Own Custom Text Classification Model\n",
    "\n",
    "This notebook contains the code and instructions to train an LLM-based text classifyer.\n",
    "\n",
    "To run the pipeline, ensure that there are two sub-folders in the same folder as this notebook file:\n",
    "\n",
    "* (1) An `src` folder containing the two files, `finetuning.py` and `models.py`. These files contain the main codebase for our text classfication pipeline and are called by the script below during execution. \n",
    "* (2) A `data` folder that contains further sub-folder(s) with the name(s) of your dataset(s). Upload your data into a sub-folder in the following format:\n",
    "    * `all-x-labeled.csv` — the text data for which you have corresponding annotations / class labels (one column, no headers)\n",
    "    * `all-y-labeled.csv` — the labels that correspond to the previous `all-x-labeled.csv` file (one column, no headers, class labels need to be integer values, and the ordering needs to align with the text data, i.e., the first label belongs to the first row of text data, and so on)  \n",
    "    * `all-x-unlabeled.csv` — the text data for which you have NO LABELS and that you want to predict/auto-label using the fine-tuned classification model produced in the following (this should be the majority of your corpus)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "056ec174",
   "metadata": {},
   "source": [
    "# A: Install and import necessary packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "67c0b7c0-53dc-48d1-8bad-7175c4d52a3f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Install required packages (only required if not already installed)\n",
    "# !pip install sentencepiece\n",
    "# !pip install pandas\n",
    "# !pip install numpy\n",
    "# !pip install wandb\n",
    "# !pip install scikit-learn\n",
    "# !pip install torch\n",
    "# !pip install torchmetrics\n",
    "# !pip install transformers\n",
    "# !pip install tqdm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d41e398a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Configure GPU workspace \n",
    "%env CUBLAS_WORKSPACE_CONFIG=:4096:8\n",
    "%env TOKENIZERS_PARALLELISM=false\n",
    "\n",
    "# Note: \n",
    "# This only works with NVIDIA GPUs. If your computer does not have such a GPU, consider setting this code up on Google Colab. \n",
    "# Also see our note on Google Colab in the accompanying `README` file for an option to speed up execution. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b02ed085-bde3-489d-b27b-61a1fce416d5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import standard Python packages\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import pickle\n",
    "import shutil\n",
    "import glob\n",
    "import time\n",
    "import os\n",
    "import gc\n",
    "\n",
    "# Import deep learning packages\n",
    "import torch\n",
    "torch.backends.cuda.matmul.allow_tf32 = True\n",
    "import wandb\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn.metrics import precision_score, accuracy_score, recall_score, f1_score\n",
    "\n",
    "# Import pipeline code\n",
    "from src.finetuning import train_and_predict_test, init_model, predict_y_from_trained_model, read_x_from_csv, init_misc, compute_and_print_metrics_for_dataset_b, set_seeds"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b5ab78e",
   "metadata": {},
   "source": [
    "# B: Set Necessary Hyperparameters"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1854cdf",
   "metadata": {},
   "source": [
    "Choose a name for your project (`PROJECT_NAME`). Model outputs will be named based on a concatonation of your `PROJECT_NAME`, `DATASET` name, text language (`LANGUAGE_FOR_MODEL`), and the choosen `LANGUAGE_MODEL` as defined in the following cell.   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "94880065",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Choose a project name:\n",
    "\n",
    "PROJECT_NAME = \"a-name-for-your-project\"  \n",
    "\n",
    "# **************************************************************************************************************************\n",
    "# Provide the name of the subfolder with your data. This subfolder must be inside 'data' folder. \n",
    "# Create separate subfolders for each dataset.\n",
    "# Prepare your data by labeling the required training and validation data (Figure 3 - Steps 1 and 2).\n",
    "\n",
    "DATASET = \"your-dataset-folder-name\"\n",
    "#DATASET = \"01-nyt-sentiment\"\n",
    "#DATASET = \"02-twitter-stance\"\n",
    "#DATASET = \"03-emotion-angry\"\n",
    "#DATASET = \"04-brexit-stance\"\n",
    "\n",
    "# **************************************************************************************************************************\n",
    "# Define the language of your text data. Your selected language model will then be laoded in the corresponding language. \n",
    "# Currently, English and German are pre-implemented (to choose custom models for other languages, use the CUSTOM_MODEL_NAME \n",
    "# option below. This allows free choice of any language model that is available via Huggingface).\n",
    "#\n",
    "# Supported values: [\"en\", \"de\"]\n",
    "\n",
    "LANGUAGE_FOR_MODEL = \"en\"\n",
    "\n",
    "# **************************************************************************************************************************\n",
    "# Choose a pretrained large language model. RoBERTA tends to show strong results across different tasks and datasets \n",
    "# and is a good initial choice (Figure 3 - Step 3).\n",
    "# \n",
    "# Recommended: \"ROB-LRG\"\n",
    "\n",
    "# LANGUAGE_MODEL = \"ROB-BASE\"\n",
    "# https://huggingface.co/roberta-base\n",
    "\n",
    "LANGUAGE_MODEL = \"ROB-LRG\"\n",
    "# https://huggingface.co/roberta-large\n",
    "\n",
    "# LANGUAGE_MODEL = \"DEB-V3\"\n",
    "# https://huggingface.co/microsoft/deberta-v3-large\n",
    "\n",
    "# LANGUAGE_MODEL = \"ELE-LRG\"\n",
    "# https://huggingface.co/google/electra-large-discriminator\n",
    "\n",
    "# LANGUAGE_MODEL = \"XLNET-LRG\"\n",
    "# https://huggingface.co/xlnet-large-cased\n",
    "\n",
    "# LANGUAGE_MODEL = \"ELE-BS-GER\"\n",
    "# To use the electra base model in german,\n",
    "# set LANGUAGE_FOR_MODEL=\"de\" above.\n",
    "# https://huggingface.co/german-nlp-group/electra-base-german-uncased\n",
    "\n",
    "# For BART and ChatGPT, see separate notebooks\n",
    "\n",
    "# Instead of selecting a model from the list above,\n",
    "# it is possible to choose another custom model provided\n",
    "# by the huggingface library. To use a custom model from Huggingface, \n",
    "# set the model ID with the following variable:\n",
    "\n",
    "CUSTOM_MODEL_NAME = None\n",
    "\n",
    "# Examples of available models are:\n",
    "# CUSTOM_MODEL_NAME = \"bert-base-cased\"\n",
    "# CUSTOM_MODEL_NAME = \"bert-base-german-cased\"\n",
    "# CUSTOM_MODEL_NAME = \"xlm-roberta-large-finetuned-conll03-german\"\n",
    "# CUSTOM_MODEL_NAME = \"distilbert-base-german-cased\"\n",
    "# CUSTOM_MODEL_NAME = \"distilbert-base-cased\"\n",
    "# CUSTOM_MODEL_NAME = \"stefan-it/albert-large-german-cased\"\n",
    "# CUSTOM_MODEL_NAME = \"albert-large-v2\"\n",
    "\n",
    "# For more model choices see the Huggingface model repository under:\n",
    "# https://huggingface.co/models?pipeline_tag=text-classification&sort=likes\n",
    "\n",
    "# **************************************************************************************************************************\n",
    "# Create a RUN_ID based on the above choices \n",
    "\n",
    "if CUSTOM_MODEL_NAME is None: \n",
    "    RUN_ID = PROJECT_NAME + \"-\" + DATASET + \"-\" + LANGUAGE_MODEL + \"-\" + LANGUAGE_FOR_MODEL\n",
    "else:\n",
    "    RUN_ID = PROJECT_NAME + \"-\" + DATASET + \"-\" + CUSTOM_MODEL_NAME + \"-\" + LANGUAGE_FOR_MODEL"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "275351c1",
   "metadata": {},
   "source": [
    "# C: (Optional) Set Advanced Hyperparameters"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f6e112d",
   "metadata": {},
   "source": [
    "The following cell provides the option to change our recommended default hyperparameters. This option also allows for systematic hyperparater optimization via grid search should this be wished (in this case, we recommend combining such an approach in combination with the external logging option (Step E) to keep track of the choosen parameters and the respective model results. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f830b124",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Starting with default hyperparameters \n",
    "# (Figure 3 - Step 3).\n",
    "\n",
    "# **************************************************************************************************************************\n",
    "# Choose a random seed. This is an arbitrary number that influences the optimization procedure. Set the seed for \n",
    "# reproducibility: The same seed should yield the same results during training.\n",
    "\n",
    "RAND_SEED = 1234\n",
    "\n",
    "# **************************************************************************************************************************\n",
    "# Choose the number of epochs for training. One epoch is one full loop through the training and validation datasets. \n",
    "# The more epochs, the longer the training will take and the stronger the model may overfit on the training dataset. \n",
    "# A low number of epochs may not lead to the full performance potential of the model but reduces training time.\n",
    "# \n",
    "# Recommended: between 5 and 20 epochs\n",
    "\n",
    "N_EPOCHS = 10\n",
    "\n",
    "# **************************************************************************************************************************\n",
    "# Choose how many samples are together in one optimization step. Higher numbers require more VRAM on the GPU or RAM \n",
    "# on the CPU. Using multiples of 4 is not required, but common practice. If you run our of memory, consider reducing to 2.\n",
    "# \n",
    "# Recommended: 4, 8, 16, 32\n",
    "\n",
    "BATCH_SIZE = 4\n",
    "\n",
    "# **************************************************************************************************************************\n",
    "# Choose the number of gradient accumulation steps. This determines after how many steps backpropagation occurs. \n",
    "# It can be used as an approximate virtual batch size: Virtual batch size ~ batch size * accumulation steps.\n",
    "# With a batch size of 4 and an accumulation step size of 8, we get roughly a batch size of 32.                        \n",
    "\n",
    "GRADIENT_ACC_STEPS = 8\n",
    "\n",
    "# **************************************************************************************************************************\n",
    "# Chooese the dropout rate for the classification head (NOT the transformer backbone model). A higher dropout rate may \n",
    "# reduce overfitting on a small training set. The dropout rate needs to be in the range [0,1]. \n",
    "# Higher value mean more dropout is applied, i.e. more information is lost during a forward pass.\n",
    "# \n",
    "# Recommended values: 0.1-0.4\n",
    "\n",
    "DROPOUT_RATE = 0.1\n",
    "\n",
    "# **************************************************************************************************************************\n",
    "# Choose the learning rate (LR) for the optimizer. A higher LR means bigger steps are taken during training and training \n",
    "# completes faster. Setting the LR too high may lead to reduced performance or overfitting. For transfer learning, \n",
    "# LRs around 1e-5 usually work best.\n",
    "#\n",
    "# Recommended values: [1e-5, 2e-5, 5e-5]\n",
    "\n",
    "LEARNING_RATE = 1e-5\n",
    "\n",
    "# **************************************************************************************************************************\n",
    "# Choose strategy for dealing with class imbalance. Correctly dealing with class imbalance can be key for model performance. \n",
    "# For very large datasets, undersampling may work well. \n",
    "# For small datasets, choose either upsamling or loss_weight.\n",
    "#\n",
    "# Options: [\"upsampling\", \"undersampling\", \"loss_weight\"]\n",
    "# \n",
    "# Recommended: \"loss_weight\"\n",
    "\n",
    "IMBALANCE_STRATEGY = 'loss_weight'\n",
    "\n",
    "# **************************************************************************************************************************\n",
    "# Enable detailed print statements for the entire pipeline including training.\n",
    "#\n",
    "# Options: [ True, False ]\n",
    "\n",
    "IS_DEBUG_ENABLED = True\n",
    "\n",
    "# **************************************************************************************************************************\n",
    "# Ensure that this flag is \"True\" for the final run of your model to use the all available training data for optimization.\n",
    "# \n",
    "# Set this flag to \"False\" to avoid overfitting on the training dataset. The validation split is a smaller subsplit from \n",
    "# the training dataset. If a model performs well on this 'unseen' data, it will likely also perform well on the unlabeled \n",
    "# data.\n",
    "# \n",
    "# (Figure 3 - Step 6 bottom).\n",
    "# \n",
    "# Options: [ True, False ]\n",
    "\n",
    "DO_VALIDATION_SET = False"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5f85779",
   "metadata": {},
   "source": [
    "# D: Load Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "26744810",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run this cell, no choices required.\n",
    "\n",
    "dataset_sentences = f\"./data/{DATASET}/all-x-labeled.csv\"\n",
    "dataset_labels = f\"./data/{DATASET}/all-y-labeled.csv\"\n",
    "\n",
    "all_x = np.squeeze(np.array(pd.read_csv(dataset_sentences, header=None, sep='\\t\\t', engine='python')))\n",
    "all_y = np.squeeze(np.array(pd.read_csv(dataset_labels, dtype=np.float32, header=None)))\n",
    "\n",
    "os.makedirs(f'./data/{DATASET}/{RUN_ID}', exist_ok=True)\n",
    "\n",
    "print(all_x.shape, all_y.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28adc956",
   "metadata": {},
   "source": [
    "# E: (Optional) Enable/Disable logging via Weights & Biases"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc6fb450",
   "metadata": {},
   "source": [
    "Note: you need to sign up with `Weigths & Biases` to use this option (https://wandb.ai/site)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "bce31685",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Option 1: no external logging of the training metrics               \n",
    "IS_LOGGING_ENABLED = False\n",
    "wandb_config = None\n",
    "\n",
    "# Option 2: external logging of the training metrics (for finetuned analysis and optimization of hyperparameters)\n",
    "# IS_LOGGING_ENABLED = True\n",
    "# wandb_config = { \"project\": \"ipz-nlp\", \"entity\": \"mnbucher\" }"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1648ab6",
   "metadata": {},
   "source": [
    "# F: Start Fine-tuning"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3785e50f",
   "metadata": {},
   "source": [
    "The following cell initiates the model training process. Depending on your computer and GPU availability, this may take a while. In case of excessive run times, consider the Google Colab option mentioned previously (see: Step A)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "855ea53c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run this cell and inspect the results. Depending on the results, decide how you want to procede (Figure 3 - Steps 4 to 6).\n",
    "\n",
    "# Set seed\n",
    "set_seeds(RAND_SEED)\n",
    "\n",
    "# Randomly shuffle loaded dataset\n",
    "idxs_shuffle = np.arange(all_x.shape[0])\n",
    "np.random.shuffle(idxs_shuffle)\n",
    "all_x = all_x[idxs_shuffle]\n",
    "all_y = all_y[idxs_shuffle]\n",
    "\n",
    "# Prepare training\n",
    "init_misc(RAND_SEED, RUN_ID, IS_DEBUG_ENABLED)\n",
    "\n",
    "# Train and evaluate model on train/val splits\n",
    "train_and_predict_test(all_x, all_y, RUN_ID, N_EPOCHS, IMBALANCE_STRATEGY, dataset_B_unlabelled_x=None, learning_rate=LEARNING_RATE, dropout_rate=DROPOUT_RATE, batch_size=BATCH_SIZE, gradient_accumulation_steps=GRADIENT_ACC_STEPS, rand_seed=RAND_SEED, language_model=LANGUAGE_MODEL, language_for_model=LANGUAGE_FOR_MODEL, custom_model_name=CUSTOM_MODEL_NAME, do_validation_set=DO_VALIDATION_SET, log_with_wandb=IS_LOGGING_ENABLED, is_debug=IS_DEBUG_ENABLED, wandb_config=wandb_config)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5049b279",
   "metadata": {},
   "source": [
    "# G: Make Predictions on Unlabeled Data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d1ad9ae",
   "metadata": {},
   "source": [
    "Predict labels for hitherto unlabeled data (`all-x-unlabeled.csv`) and save the predicted labels (`predictions-x-unlabeled.csv`) in the `data` folder under the `RUN_ID` name defined above. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "44493f8e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Once model training is completed to your satisfaction, use the fine-tuned model to auto-label the unlabeled part of your data.\n",
    "# (Figure 3 - Step 7).\n",
    "\n",
    "dataset_B_unlabelled_x = read_x_from_csv(f\"./data/{DATASET}/all-x-unlabeled.csv\")\n",
    "\n",
    "# **************************************************************************************************************************\n",
    "\n",
    "n_classes = len(list(np.unique(all_y)))\n",
    "\n",
    "dvc = init_misc(RAND_SEED, RUN_ID, IS_DEBUG_ENABLED, remove_log_files=False)\n",
    "model, _, _ = init_model(LANGUAGE_MODEL, LANGUAGE_FOR_MODEL, CUSTOM_MODEL_NAME, LEARNING_RATE, DROPOUT_RATE, n_classes, dvc, N_EPOCHS, GRADIENT_ACC_STEPS, None)\n",
    "max_seq_length = 512\n",
    "\n",
    "dataset_B_unlabelled_y_pred = predict_y_from_trained_model(RUN_ID, LANGUAGE_MODEL, LANGUAGE_FOR_MODEL, CUSTOM_MODEL_NAME, dataset_B_unlabelled_x, model, BATCH_SIZE, RAND_SEED, max_seq_length, dvc, IS_DEBUG_ENABLED)\n",
    "\n",
    "np.savetxt(\"./output/predictions-x-unlabeled.csv\", dataset_B_unlabelled_y_pred, fmt='%f', encoding=\"utf-8\") ### NEW\n",
    "\n",
    "print(\"finished!\")\n",
    "\n",
    "# **************************************************************************************************************************\n",
    "\n",
    "# Clean up\n",
    "files = [ f for f in glob.glob('./output/*.csv') ] \n",
    "files\n",
    "files_dest = [ f.replace(\"/output/\", f\"/data/{DATASET}/{RUN_ID}/\") for f in glob.glob('./output/*.csv') ]\n",
    "files_dest\n",
    "for f in range(len(files)):\n",
    "    shutil.move(files[f], files_dest[f])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}