Session4_SentimentAnalysis_29_3_23.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "64f9c1be",
   "metadata": {},
   "source": [
    "##  Natural Language Processing using Python\n",
    "#### This  4 hour workshop will introduce the audience to the field of Natural Language Processing,  we will cover the fundamental tasks in NLP and walk through the Python code for some of the important steps in an NLP project.\n",
    "\n",
    "#### Author: Dr. Nimrita Koul, Associate Professor, Machine Learning\n",
    "#### Who this talk is suitable for: Data analysts, students and anyone interested in using Natural Language Processing\n",
    "#### Prerequisites: Python Programming, basic familiarity with concepts in data science and machine learning\n",
    "#### Duration: 4 Sessions of 1 hour each\n",
    "#### Time:  Starting 9th March 2023, Every Thursday,  8.00 pm to 9.00 pm Indian Standard Time\n",
    "#### Github URLs: \n",
    "\n",
    "Session1: \n",
    "https://github.com/NimritaKoul/NLP_WWC2023/blob/main/Session1_Introduction%20to%20Natural%20Language%20Processing.ipynb\n",
    "\n",
    "Session 2:\n",
    "https://github.com/NimritaKoul/NLP_WWC2023/blob/main/Session2_TextPreprocessing_Representation.ipynb\n",
    "\n",
    "Session 3:\n",
    "https://github.com/NimritaKoul/NLP_WWC2023/blob/main/Session3-TextClassification_Transformers.ipynb\n",
    "\n",
    "Session 4: \n",
    "\n",
    "https://github.com/NimritaKoul/NLP_WWC2023/blob/main/Session4_SentimentAnalysis_29_3_23.ipynb\n",
    "\n",
    "https://github.com/NimritaKoul/NLP_WWC2023/blob/main/Session4_Chatbot29_3_2023.ipynb\n",
    "\n",
    "\n",
    "### Contents: \n",
    "\n",
    "#### Session 1:\n",
    "1. Introduction to Natural Language Processing, components, task, applications. \n",
    "2. Python Tools and Libraries for NLP - with focus on NLTK and SpaCy \n",
    "\n",
    "#### Session 2:\n",
    "1. Text Preprocessing - Cleaning, normalization - tokenization, stopword removal, case conversion, stemming, lemmatization, POS tagging, Named Entity Recognition\n",
    "2. Text Representation - Bag of word models, TF-IDF, Word Embeddings\n",
    "\n",
    "#### Session 3:\n",
    "1. Text Classification - Text classification models Naive Bayes, Decision Tree, RNN<br>\n",
    "2. Transformers\n",
    "\n",
    "\n",
    "#### Session 4 :  Case Study\n",
    "<span style=\"color:Blue\"> \n",
    "1. Sentiment Analysis<br>\n",
    "2. Chatbot \n",
    "</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6fc6580",
   "metadata": {},
   "source": [
    "## Sentiment Analysis\n",
    "\n",
    "<img src = 'fine-grained.png' width = 800 height = 600>\n",
    "<center> Source: https://getthematic.com/sentiment-analysis/</center>\n",
    "\n",
    "#### Sentiment analysis is the use of natural language processing to identify, extract, quantify the study emotional states and subjective information in the data. \n",
    "\n",
    "##### Sentiment analysis generally follows these steps:\n",
    "\n",
    "1. Data Collection : using web scrapping or publicly available data or surveys etc. \n",
    "2. Data Cleaning: Pre-processing data to normalize it, remove noise, stop words, words with specific POS tags which are irrelevant to sentiment of the text, e.g., contractions, punctuation, URLs, etc. \n",
    "3. Feature Extraction: Using techniques like Bag of words or word embeddings. \n",
    "4. Train an ML model: E.g., a deep learning model like BERT. \n",
    "5. Deploy the model for Sentiment classification: A trained model can be used to analyse a new piece of text and score the sentiment in it. \n",
    "\n",
    "##### Types of sentiment analysis\n",
    "\n",
    "1. Fine-grained sentiment analysis: This type of sentiment analysis identifies more number of categories of sentiment e.g., extremely satisfied, very satisfied, just ok, very unsatisfied, extremely unsatisfied. \n",
    "\n",
    "2. Emotion detection analysis:  This type of sentiment analysis identifies emotions like happiness, frustration, shock, anger and sadness.\n",
    "\n",
    "3. Intent-based analysis: Identifies the intention or motivations behind a text. E.g., intention of a customer who have given an unhappy review about a product may be that the customer service support was delayed.\n",
    "\n",
    "4. Aspect-based analysis : This focuses on specific component of a text. E.g., a customer might review a product saying the battery life was too short. Aspect based sentiment analysis system will note that the negative sentiment isn't about the product as a whole but about the battery life.\n",
    "\n",
    "\n",
    "##### Some Applications \n",
    "\n",
    "- Keeping track of customer sentiments, or sentiment of citizens through social media monitoring,\n",
    "- Monitoring brand awareness, reputation and receptivity in businesses.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7368cc68",
   "metadata": {},
   "source": [
    "## BERT Model\n",
    "\n",
    "(https://arxiv.org/abs/1810.04805)\n",
    "\n",
    "- Developed by a team at Google, <b>Bidirectional Encoder Representations from Transformers (BERT)</b> is a family of <b>masked-language models (MLM)</b> trained on unlabeled plain text corpora from Toronta BookCorpus and English Wikipedia. \n",
    "\n",
    "- BERT is a <b>language representation model</b> pre-trained on deep birectional representations from unlabeled text. It is conditioned on both left and right context in all layers and can be fine tuned on small data sets for NLP tasks.\n",
    "\n",
    "-  Contextual pre-trained representations can be unidirectional or bidirectional. Context-free models such as word2vec or GloVe generate a single word embedding representation for each word in the vocabulary. For example, the word “bank” would have the same context-free representation in “bank account” and “bank of the river.” Contextual models generate a representation of each word that is based on the other words in the sentence. For example, in the sentence “I accessed the bank account,” a unidirectional contextual model would represent “bank” based on “I accessed the” but not “account.” However, BERT represents “bank” using both its previous and next context — “I accessed the ... account” — starting from the very bottom of a deep neural network, making it deeply bidirectional.\n",
    "\n",
    "<img src = 'bertarch.png'>\n",
    "<center> Source: https://ai.googleblog.com/2018/11/open-sourcing-bert-state-of-art-pre.html</center>\n",
    "\n",
    "\n",
    "#####  BERT is pre-trained for two tasks - languge modelling and next sentence prediction.\n",
    "#####  BERT framework has 2 steps: pretraining and fine tuning.\n",
    "\n",
    "-  During pretraining, model is trained on unlabeled data over different pre-training tasks.  \n",
    "- For fine tuning, BERT is first initialized with the pretrained parameters and all parameters are fine tuned using labeled data from downstream tasks. Each downstream task has separate fine-tuned models, even though they are initialized with same pre-trained parameters.\n",
    "- BERT can be fine-tuned with just one additional output layer on smaller datasets for NLP tasks like language inference, paraphrasing, text classification, question answering, conversational AI. \n",
    "\n",
    "\n",
    "### BERT Architecture\n",
    "\n",
    "<img src = 'bert3.png' width = 700 height = 700>\n",
    "<center> Source: https://arxiv.org/abs/1810.04805 </center>\n",
    "\n",
    "#### BERT architecture is a multilayer bidirectional transformer encoder.\n",
    "\n",
    "##### Input/Output Representations: Input representation is able to represent a single sentence as well as a pair of sentences (e.g., (question,answer)) in one token sequence. \n",
    "- It uses WordPiece embeddings with 30,000 token vocabulary.\n",
    "\n",
    "##### Tokenization in BERT    \n",
    "\n",
    "- First token of every sequence is a special classification token [CLS]. Final hidden state corresponding to this token is used as the aggrerate sequence representation for classification tasks. Sentence pairs are packed together into a single sequence.\n",
    "\n",
    "- The sentences are differentiated in two ways- first, they are separated with a special token [SEP], second, a learned embedding is added to every token indicating whether it belongs to sentence A or B. \n",
    "\n",
    "- For a given token, its input representation is constructed by <b>summing the corresponding token, segment and position embeddings</b>.\n",
    "\n",
    "- Token Embeddings is a word vector, with the first word as the CLS flag, which can be used for classification tasks.\n",
    "\n",
    "- Segment Embeddings is used to distinguish between two sentences. \n",
    "\n",
    "- BERT learns a unique position embedding for the input sequence, and this position-specific information can flow through the model to the key and query vectors. \n",
    "\n",
    "- BERT position embedding is different fromt the general Transformer positional embedding.\n",
    "\n",
    "<img src = 'bert2.png'>\n",
    "<center> Source: https://arxiv.org/abs/1810.04805 </center>\n",
    "\n",
    "##### Tokenization in BERT    \n",
    "\n",
    "<img src = 'tokenization.png' width = 700 height = 700>\n",
    "<center> Source: https://colab.research.google.com/github/jalammar/jalammar.github.io/blob/master/notebooks/bert/A_Visual_Notebook_to_Using_BERT_for_the_First_Time.ipynb </center>\n",
    "\n",
    "#### Pretraining Phase of BERT\n",
    "\n",
    "###### Pretraining Data: \n",
    " - BERT used the BookCorpus with 800M words and words of English Wikipedia (2500M words). It used document level corpus rather than shuffled sentence-level corpus to extract long contiguous sequences.\n",
    "\n",
    "##### BERT is pretrained using two unsupervised tasks:\n",
    "\n",
    "##### Task 1: Masked LM (MLM)\n",
    "- 15% of the tokens of all WordPiece tokens in each input sequence are masked at random and then model attempts to predict these masked tokens from rest of the context tokens. Final hidden vectors corresponding to the mask tokens are fed into an output softmax over the vocabulary. \n",
    "- Though this creates a bidirectional pre-trained model, this creates a mismatch between pretraining and finetuning as the [MASK] token is not present during fine tuning. To mitigate this, masked words are not always replaced with [MASK] token. \n",
    "- Training data generator chooses 15% tokens at random for prediction. If the ith token is chosen, model replaces the ith token with: (1)the [MASK] token 80% of the time (2) a random token 10% of the time (3) the unchanged ith token 10% of the time. Then, Ti is used to predict the original token with cross entropy loss.\n",
    "\n",
    "\n",
    "<img src = 'mask.png'>\n",
    "<center> Source: https://ai.googleblog.com/2018/11/open-sourcing-bert-state-of-art-pre.html</center>\n",
    "\n",
    "- This needs a classification layer on top of encoder input, multiplication of output vectors by the embedding matrix transforming them into vocaculary dimension, calculating the probability of each word in the vocaculary with softmax.\n",
    "\n",
    "<img src = 'bert1.png'>\n",
    "<center> Source: https://arxiv.org/abs/1810.04805</center>\n",
    "\n",
    "##### Task 2: Next Sentence Prediction (NSP)\n",
    "- BERT is pretrained on binarized next sentence predictiont ask. For this task, sentence pairs are used as training data. For sentence pairs A and B, 50% of the time B is actual next sentence that follows A and 50% of the time it is a random sentence from corpus. \n",
    "\n",
    "<img src = 'next.png'>\n",
    "<center> Source: https://ai.googleblog.com/2018/11/open-sourcing-bert-state-of-art-pre.html</center>\n",
    "\n",
    "#### Fine Tuning Phase of BERT\n",
    "\n",
    "- <b>Self-attention mechinism</b> in the Transformer allows BERT to model many downstream tasks which may involve single text or pairs of text by swapping out appropriate inputs and outputs. \n",
    "\n",
    "- For applications involving text pairs, a common pattern is to independently encode text pairs and then apply bidirectional cross attention, but BERT does both these in <b>single step by encoding a concatenated text pair with self-attention as it effectively includes bidirectional cross attention between two sentences.</b> \n",
    "\n",
    "- For each task, we just need to plug in the task specific inputs and outputs into BERT and fine tune all parameters end-to-end. For example, during fine tuning, at the input side, sentence A and sentence B from pre-training are analogous to sentence pairs in paraphrasing, or hypothesis-premise pairs in entailment, or question-passage pairs in question answering, or a  text-label pair in text classification or sequence tagging. At the output side, the token representations are fed into an output layer for token-level tasks, such as sequence tagging or question answering, and the [CLS] representation is fed into an output layer for classification, such as entailment or sentiment analysis. \n",
    "\n",
    "- When training the BERT model, Masked LM and Next Sentence Prediction are trained together, with the goal of minimizing the combined loss function of the two strategies.\n",
    "\n",
    "#### BERT Model Sizes \n",
    "\n",
    "<img src = 'BERT-size-and-architecture.png' width = 700 height = 700>\n",
    "<center> Source: https://huggingface.co/blog/bert-101</center>\n",
    "\n",
    "- Original BERT was for English language in two model sizes : \n",
    "      1. BERT Base with 12 encoders and 12 bidirectional self-attention heads, 110 million parameters\n",
    "      2. BERT Large with 24 encoders and 16 bidirectional self-attention heads, 340 million parameters. \n",
    "\n",
    "- But now we have following BERT pre-trained models:\n",
    "\n",
    "    - BERT-Base, Uncased: 12-layer, 768-hidden, 12-heads, 110M parameters\n",
    "    - BERT-Large, Uncased: 24-layer, 1024-hidden, 16-heads, 340M parameters\n",
    "    - BERT-Base, Cased: 12-layer, 768-hidden, 12-heads , 110M parameters\n",
    "    - BERT-Large, Cased: 24-layer, 1024-hidden, 16-heads, 340M parameters\n",
    "    - BERT-Base, Multilingual Cased (New, recommended): 104 languages, 12-layer, 768-hidden, 12-heads, 110M parameters\n",
    "    - BERT-Base, Chinese: Chinese Simplified and Traditional, 12-layer, 768-hidden, 12-heads, 110M parameters\n",
    "\n",
    "- BERT is also available for languages other than English:\n",
    "    - CamemBERT(French)\n",
    "    - AraBERT(Arabic)\n",
    "    - mBERT(Multilingual)\n",
    "\n",
    "- BERT Variants:\n",
    "    - RoBERTa\n",
    "    - DistilBERT\n",
    "    - AlBERT\n",
    "    \n",
    "#### BERT can be used on a wide variety of language tasks:\n",
    "\n",
    "- Sentiment Analysis\n",
    "- Conversational AI and Question answering\n",
    "- Text prediction\n",
    "- Text generation\n",
    "- Text Summarization\n",
    "- Polysemy Resolution: Can differentiate words that have multiple meanings (like ‘bank’) based on the surrounding text. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae66e7df",
   "metadata": {},
   "source": [
    "#### References:\n",
    "\n",
    "1. Dataset: SMILE Twitter dataset. Wang, Bo; Tsakalidis, Adam; Liakata, Maria; Zubiaga, Arkaitz; Procter, Rob; Jensen, Eric (2016): SMILE Twitter Emotion dataset. figshare. Dataset. https://doi.org/10.6084/m9.figshare.3187909.v2\n",
    "\n",
    "2. https://huggingface.co/blog/bert-101\n",
    "\n",
    "3. http://jalammar.github.io/illustrated-bert/\n",
    "\n",
    "4. https://www.coursera.org/projects/sentiment-analysis-bert\n",
    "\n",
    "5. https://huggingface.co/transformers/v2.2.0/index.html\n",
    "\n",
    "6. https://huggingface.co/transformers/v2.2.0/model_doc/bert.html#bertforsequenceclassification\n",
    "\n",
    "7. https://colab.research.google.com/github/jalammar/jalammar.github.io/blob/master/notebooks/bert/A_Visual_Notebook_to_Using_BERT_for_the_First_Time.ipynb\n",
    "\n",
    "8. https://colab.research.google.com/drive/1kEg0SnYNtw_IJwu_kl5y3qRVs-BKBmNO\n",
    "\n",
    "9. https://www.techtarget.com/searchbusinessanalytics/definition/opinion-mining-sentiment-mining\n",
    "\n",
    "10. https://en.wikipedia.org/wiki/Sentiment_analysis\n",
    "\n",
    "11. http://nlp.seas.harvard.edu/2018/04/03/attention.html\n",
    "\n",
    "12. http://nlp.seas.harvard.edu/2018/04/03/attention.html"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2d334c9",
   "metadata": {},
   "source": [
    "#### Code: Sentiment Analysis using BERT"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "de194b25",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Required imports\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "import torch\n",
    "import pandas as pd\n",
    "from tqdm.notebook import tqdm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "b9b3b673",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>sentiment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>@aandraous @britishmuseum @AndrewsAntonio Merc...</td>\n",
       "      <td>nocode</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Dorian Gray with Rainbow Scarf #LoveWins (from...</td>\n",
       "      <td>happy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>@SelectShowcase @Tate_StIves ... Replace with ...</td>\n",
       "      <td>happy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>@Sofabsports thank you for following me back. ...</td>\n",
       "      <td>happy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>@britishmuseum @TudorHistory What a beautiful ...</td>\n",
       "      <td>happy</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                text sentiment\n",
       "0  @aandraous @britishmuseum @AndrewsAntonio Merc...    nocode\n",
       "1  Dorian Gray with Rainbow Scarf #LoveWins (from...     happy\n",
       "2  @SelectShowcase @Tate_StIves ... Replace with ...     happy\n",
       "3  @Sofabsports thank you for following me back. ...     happy\n",
       "4  @britishmuseum @TudorHistory What a beautiful ...     happy"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Reading dataset\n",
    "df = pd.read_csv('smile-annotations-final.csv', header = None)\n",
    "df.columns = ['id', 'text', 'sentiment']\n",
    "df = df.drop(['id'], axis = 1)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "dcfafebd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "nocode               1572\n",
       "happy                1137\n",
       "not-relevant          214\n",
       "angry                  57\n",
       "surprise               35\n",
       "sad                    32\n",
       "happy|surprise         11\n",
       "happy|sad               9\n",
       "disgust|angry           7\n",
       "disgust                 6\n",
       "sad|disgust             2\n",
       "sad|angry               2\n",
       "sad|disgust|angry       1\n",
       "Name: sentiment, dtype: int64"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Seeing all the sentiment labels and their frequency\n",
    "df.sentiment.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "7d04d6aa",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>sentiment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Dorian Gray with Rainbow Scarf #LoveWins (from...</td>\n",
       "      <td>happy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>@SelectShowcase @Tate_StIves ... Replace with ...</td>\n",
       "      <td>happy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>@Sofabsports thank you for following me back. ...</td>\n",
       "      <td>happy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>@britishmuseum @TudorHistory What a beautiful ...</td>\n",
       "      <td>happy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>@NationalGallery @ThePoldarkian I have always ...</td>\n",
       "      <td>happy</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                text sentiment\n",
       "1  Dorian Gray with Rainbow Scarf #LoveWins (from...     happy\n",
       "2  @SelectShowcase @Tate_StIves ... Replace with ...     happy\n",
       "3  @Sofabsports thank you for following me back. ...     happy\n",
       "4  @britishmuseum @TudorHistory What a beautiful ...     happy\n",
       "5  @NationalGallery @ThePoldarkian I have always ...     happy"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# we will drop the rows with labels that are ambiguous or where the frequency is very less\n",
    "df = df[~df.sentiment.str.contains('\\|')]\n",
    "df = df[df.sentiment.isin(['happy', 'not-relevant', 'angry', 'surprise', 'sad'])]\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "09e126ca",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Since sentiment labels are text, we will convert them to numbers using sklearn's LabelEncoder\n",
    "from sklearn import preprocessing\n",
    "le = preprocessing.LabelEncoder()\n",
    "le.fit(df['sentiment'])\n",
    "df['sentiment'] = le.transform(df['sentiment'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "d3ce15bd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>sentiment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Dorian Gray with Rainbow Scarf #LoveWins (from...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>@SelectShowcase @Tate_StIves ... Replace with ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>@Sofabsports thank you for following me back. ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>@britishmuseum @TudorHistory What a beautiful ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>@NationalGallery @ThePoldarkian I have always ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                text  sentiment\n",
       "1  Dorian Gray with Rainbow Scarf #LoveWins (from...          1\n",
       "2  @SelectShowcase @Tate_StIves ... Replace with ...          1\n",
       "3  @Sofabsports thank you for following me back. ...          1\n",
       "4  @britishmuseum @TudorHistory What a beautiful ...          1\n",
       "5  @NationalGallery @ThePoldarkian I have always ...          1"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "4b3a7fa4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is a dictionary that maps sentiment label to a number\n",
    "label_dict = dict(zip(le.classes_, range(len(le.classes_))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "3b7a204d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'angry': 0, 'happy': 1, 'not-relevant': 2, 'sad': 3, 'surprise': 4}"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# All sentiment labels and their numeric values\n",
    "label_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "8214572c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1       dorian gray rainbow scarf lovewins britishmuse...\n",
       "2       selectshowcase tatestives replace wish artist ...\n",
       "3       sofabsports thank following back great hear di...\n",
       "4       britishmuseum tudorhistory beautiful jewel por...\n",
       "5       nationalgallery thepoldarkian always loved pai...\n",
       "                              ...                        \n",
       "3078    thewhitechapel campaignforwool slowtextiles an...\n",
       "3079    britishmuseum thanks ranking u tripadvisors th...\n",
       "3080    mt alihaggett looking forward public engagemen...\n",
       "3082                mrstuchbery britishmuseum mesmerising\n",
       "3083    nationalgallery nd genocide biafrans promised ...\n",
       "Name: text, Length: 1475, dtype: object"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Text preprocessing\n",
    "import numpy\n",
    "import re\n",
    "import string\n",
    "import unicodedata\n",
    "from nltk.corpus import stopwords\n",
    "from nltk.stem import WordNetLemmatizer\n",
    "stop_words = stopwords.words('english')\n",
    "lemmatizer = WordNetLemmatizer()\n",
    "\n",
    "def process_text(text):\n",
    "    text = unicodedata.normalize('NFKD', text).encode('ascii', 'ignore').decode('utf-8', 'ignore')\n",
    "    text = re.sub(r'[^a-zA-Z\\s]', '', text)\n",
    "    text = text.translate(str.maketrans('', '', string.punctuation))\n",
    "    text = text.lower()\n",
    "    text = \" \".join([word for word in str(text).split() if word not in stop_words])\n",
    "    text = \" \".join([lemmatizer.lemmatize(word) for word in text.split()])\n",
    "    return text\n",
    "df['text'] = df['text'].apply(process_text)\n",
    "df['text']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1188b7b2",
   "metadata": {},
   "source": [
    "#### Train - Test Split\n",
    "<img src = 'traintestsplit_distillbert.png' width = 600 height = 400>\n",
    "<center> Source: https://colab.research.google.com/github/jalammar/jalammar.github.io/blob/master/notebooks/bert/A_Visual_Notebook_to_Using_BERT_for_the_First_Time.ipynb</center>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "c9661f94",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Splitting training and test portions of data\n",
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_val, y_train, y_val = train_test_split(df['text'], df['sentiment'], test_size=0.25, random_state=17, stratify=df['sentiment'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "61d8631a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1959    britishmuseum one favourite napoleon comet hat...\n",
       "1417    studio work done thewhitechapel joy httptcoevw...\n",
       "282     tateliverpool richard great idea thank liverpo...\n",
       "1188    britishmuseum perinsarosh best camden family l...\n",
       "1004    psframes nationalgallery love national gallery...\n",
       "Name: text, dtype: object"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "fd71d44b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1959    1\n",
       "1417    1\n",
       "282     1\n",
       "1188    1\n",
       "1004    1\n",
       "Name: sentiment, dtype: int32"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "0451bb51",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.series.Series"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de8378fc",
   "metadata": {},
   "source": [
    "#### We need to have installed transformers library before running the code in following cells. You can install transformers using the pip command from Jupyter Notebook. Uncomment the below cell to run it and install transformers.\n",
    "\n",
    "- Transformers library from HuggingFace (https://huggingface.co/docs/transformers/index) provides thousands of pretrained models to perform tasks on different modalities such as text, vision, and audio.\n",
    "- Transformers provides APIs to quickly download and use those pretrained models on a given text, fine-tune them on your own datasets and then share them with the community through HuggingFace Hub."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "94e52dc1",
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install transformers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "a033700c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# required imports\n",
    "from transformers import BertTokenizer # for tokenizing our input\n",
    "from torch.utils.data import TensorDataset # for converting input vectors to Tensors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "a2734126",
   "metadata": {},
   "outputs": [],
   "source": [
    "#https://huggingface.co/docs/transformers/model_doc/bert#transformers.BertTokenizer.build_inputs_with_special_tokens\n",
    "# Object of BertTokenizer for Uncased Base BERT model\n",
    "tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2070b91",
   "metadata": {},
   "source": [
    "#### Now, we have loaded the BertTokenizer, next we will encode our data using this object.\n",
    "\n",
    "- The method batch_encode_plus() of BertTokenizer converts text to standard represenation of BERT Transformer\n",
    "- It builds model inputs from a sequence or a pair of sequences for sequence-classification tasks by concatenating and adding special tokens to sequences. Special tokens are [CLS], [SEP] etc. for indicating Beginning of a sequence and a separator respectively. \n",
    "\n",
    "- A BERT sequence has the following format:\n",
    "     - single sequence: [CLS] X [SEP]\n",
    "     - pair of sequences: [CLS] A [SEP] B [SEP]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "6f722ce7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Encode input text and labels using pretrained encodings in the BERTTokenizer\n",
    "# Parameters to BertTokenizer: input data, add\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "\n",
    "en_traindata = tokenizer.batch_encode_plus(X_train, add_special_tokens=True, return_attention_mask=True, \n",
    "    pad_to_max_length=True, truncation=True, max_length=256,   return_tensors='pt')\n",
    "\n",
    "en_valdata = tokenizer.batch_encode_plus(X_val, add_special_tokens=True, return_attention_mask=True, \n",
    "    pad_to_max_length=True, truncation=True, max_length=256,  return_tensors='pt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "4ce83130",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'input_ids': tensor([[  101,  2329,  7606,  ...,     0,     0,     0],\n",
       "        [  101,  2996,  2147,  ...,     0,     0,     0],\n",
       "        [  101,  9902,  3669,  ...,     0,     0,     0],\n",
       "        ...,\n",
       "        [  101, 16047,  2583,  ...,     0,     0,     0],\n",
       "        [  101,  2047,  3358,  ...,     0,     0,     0],\n",
       "        [  101,  8395,  2329,  ...,     0,     0,     0]]), 'token_type_ids': tensor([[0, 0, 0,  ..., 0, 0, 0],\n",
       "        [0, 0, 0,  ..., 0, 0, 0],\n",
       "        [0, 0, 0,  ..., 0, 0, 0],\n",
       "        ...,\n",
       "        [0, 0, 0,  ..., 0, 0, 0],\n",
       "        [0, 0, 0,  ..., 0, 0, 0],\n",
       "        [0, 0, 0,  ..., 0, 0, 0]]), 'attention_mask': tensor([[1, 1, 1,  ..., 0, 0, 0],\n",
       "        [1, 1, 1,  ..., 0, 0, 0],\n",
       "        [1, 1, 1,  ..., 0, 0, 0],\n",
       "        ...,\n",
       "        [1, 1, 1,  ..., 0, 0, 0],\n",
       "        [1, 1, 1,  ..., 0, 0, 0],\n",
       "        [1, 1, 1,  ..., 0, 0, 0]])}"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let us see what our data encodings look like now\n",
    "en_traindata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "cfdee114",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "transformers.tokenization_utils_base.BatchEncoding"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(en_traindata)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "ee31db7d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Converting training and validation data to Torch TensorDataset Format\n",
    "\n",
    "#Training Data\n",
    "input_ids_traindata = en_traindata['input_ids']\n",
    "attention_masks_traindata = en_traindata['attention_mask']\n",
    "\n",
    "# Converting training labels to LongTensors\n",
    "labels_traindata = torch.tensor(y_train.values)\n",
    "labels_traindata = labels_traindata.type(torch.LongTensor)\n",
    "\n",
    "\n",
    "tensor_traindata = TensorDataset(input_ids_traindata, attention_masks_traindata, labels_traindata)\n",
    "\n",
    "\n",
    "# Validation Data\n",
    "input_ids_valdata = en_valdata['input_ids']\n",
    "attention_masks_valdata = en_valdata['attention_mask']\n",
    "\n",
    "# Converting validation labels to LongTensors\n",
    "labels_valdata = torch.tensor(y_val.values)\n",
    "labels_valdata = labels_valdata.type(torch.LongTensor)\n",
    "\n",
    "tensor_valdata = TensorDataset(input_ids_valdata, attention_masks_valdata, labels_valdata)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "2e45df44",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1106"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(tensor_traindata)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "d3bb71e8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "369"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(tensor_valdata)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "bc5c60ce",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.utils.data.dataset.TensorDataset"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(tensor_traindata)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1f91c64",
   "metadata": {},
   "source": [
    "#### Importing pre-trained Base Uncased BERT model for Sequene Classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "bc357214",
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "from transformers import logging\n",
    "logging.set_verbosity_error()\n",
    "\n",
    "\n",
    "#Setting up BERT Pretrained Model\n",
    "from transformers import BertForSequenceClassification\n",
    "\n",
    "num_of_unique_labels = df['sentiment'].nunique()\n",
    "\n",
    "bert_model = BertForSequenceClassification.from_pretrained(\"bert-base-uncased\", num_labels=num_of_unique_labels,\n",
    "                                                      output_attentions=False, output_hidden_states=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c671b8e3",
   "metadata": {},
   "source": [
    "### Creating Data Loaders\n",
    "###### https://pytorch.org/tutorials/beginner/basics/data_tutorial.html#:~:text=PyTorch%20provides%20two%20data%20primitives,easy%20access%20to%20the%20samples.\n",
    "\n",
    "- DataLoader batches the training samples into mini-batches, reshuffle the data at every epoch to reduce model overfitting, and use Python’s multiprocessing to speed up data retrieval."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "a9c450dd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating dataloaders before training a model\n",
    "from torch.utils.data import DataLoader, RandomSampler, SequentialSampler\n",
    "batch_size = 64\n",
    "\n",
    "loader_traindata = DataLoader(tensor_traindata, sampler=RandomSampler(tensor_traindata),  batch_size=batch_size)\n",
    "loader_valdata = DataLoader(tensor_valdata, sampler=SequentialSampler(tensor_valdata), batch_size=batch_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "25d652b8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.utils.data.dataloader.DataLoader"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(loader_traindata)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40f183a0",
   "metadata": {},
   "source": [
    "### Optimizer and Learning Rate Scheduler for BERT Sequence Classification Model\n",
    "##### https://huggingface.co/transformers/v3.0.2/main_classes/optimizer_schedules.html\n",
    "\n",
    "- An optimization algorithm or optimizer like Gradient Descent, Stochastic GD, Adam etc. is used to update the weights during model training. It can use a fixed learning rate or a varying learning rate. \n",
    "\n",
    "###### The optimizer we use in this code is AdamW and the scheduler is get_linear_schedule_with_warmup()\n",
    "\n",
    "- PyTorch Tabular uses <b>Adam optimizer with weight decay</b> with a learning rate of 1e-3 by default. This is mainly because of a rule of thumb which provides a good starting point. \n",
    "\n",
    "- Scheduler - <b>get_linear_schedule_with_warmup() Creates a schedule with a learning rate that decreases linearly from the initial lr set in the optimizer to 0, after a warmup period during which it increases linearly from 0 to the initial lr set in the optimizer.</b>\n",
    "- scheduler parameters:\n",
    "    -1. optimizer (Optimizer) – The optimizer for which to schedule the learning rate.\n",
    "    -2. num_warmup_steps (int) – The number of steps for the warmup phase.\n",
    "    -3. num_training_steps (int) – The totale number of training steps.\n",
    "    -4. last_epoch (int, optional, defaults to -1) – The index of the last epoch when resuming training.\n",
    "    \n",
    "- Warm up steps is a parameter which is used to lower the learning rate in order to reduce the impact of deviating the model from learning on sudden new data set exposure. By default, number of warm up steps is 0.  Then you make bigger steps, because you are probably not near the minima. But as you are approaching the minima, you make smaller steps to converge to it.\n",
    "\n",
    "- Number of training steps is number of batches * number of epochs.\n",
    "\n",
    "- To update the learning rate, we need to call scheduler.step() for every batch of data after calling optimizer.step()."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "861be9f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Optimiser and Learning Rate Scheduler\n",
    "from transformers import AdamW, get_linear_schedule_with_warmup\n",
    "#AdamW : Implements Adam algorithm with weight decay fix\n",
    "adam_optimizer = AdamW(bert_model.parameters(), lr=1e-5,   eps=1e-8)\n",
    "epochs = 3\n",
    "\n",
    "scheduler = get_linear_schedule_with_warmup(adam_optimizer, num_warmup_steps=0, \n",
    "                                            num_training_steps=len(loader_traindata)*epochs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "9c856b72",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.optim.lr_scheduler.LambdaLR"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(scheduler)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f03f3b84",
   "metadata": {},
   "source": [
    "#### Training Loop\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "fc0db501",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cpu\n"
     ]
    }
   ],
   "source": [
    "# Setting seeds for reproducibility of results\n",
    "import numpy as np\n",
    "import random\n",
    "seed_val = 17\n",
    "random.seed(seed_val)\n",
    "np.random.seed(seed_val)\n",
    "torch.manual_seed(seed_val)\n",
    "torch.cuda.manual_seed_all(seed_val)\n",
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "bert_model.to(device)\n",
    "print(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f6e18ca",
   "metadata": {},
   "source": [
    "#### Evaluation of pre-trained model\n",
    "##### during evaluation we dont want weights to be updated or any gradients to be calculated"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "da2e25b6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function to evaluate the pre-trained model \n",
    "# during evaluation we dont want weights to be updated or any gradients to be calculated\n",
    "def evaluate(loader_valdata):\n",
    "\n",
    "    # Turning on the evaluation model of BERT model, no weight will be updated during this\n",
    "    bert_model.eval()\n",
    "    \n",
    "    loss_val_total = 0\n",
    "    predictions, true_vals = [], []\n",
    "    \n",
    "    for batch in loader_valdata:\n",
    "        batch = tuple(b.to(device) for b in batch)\n",
    "        \n",
    "        inputs = {'input_ids':      batch[0],\n",
    "                  'attention_mask': batch[1],\n",
    "                  'labels':         batch[2],\n",
    "                 }\n",
    "        # with torch.no_grad() we disable gradient calculation.\n",
    "        with torch.no_grad():        \n",
    "            outputs = bert_model(**inputs)\n",
    "            \n",
    "        loss = outputs[0]\n",
    "        logits = outputs[1]\n",
    "        loss_val_total += loss.item()\n",
    "\n",
    "        logits = logits.detach().cpu().numpy()\n",
    "        label_ids = inputs['labels'].cpu().numpy()\n",
    "        predictions.append(logits)\n",
    "        true_vals.append(label_ids)\n",
    "    \n",
    "    loss_val_avg = loss_val_total/len(loader_valdata) \n",
    "    \n",
    "    predictions = np.concatenate(predictions, axis=0)\n",
    "    true_vals = np.concatenate(true_vals, axis=0)\n",
    "            \n",
    "    return loss_val_avg, predictions, true_vals"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "450f5db5",
   "metadata": {},
   "source": [
    "### Next we will Fine Tune (Train) the pre-trained BERT model with our dataset\n",
    "#### This may take a long time depending on the configuration of your computer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "ba11477d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e0143a40080040988a32e48df7c99c66",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/3 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Epoch 1:   0%|          | 0/18 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Epoch 1\n",
      "Training loss: 1.0122020079029932\n",
      "Validation loss: 0.8863703906536102\n",
      "F1 Score (Weighted): 0.6694638462464257\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Epoch 2:   0%|          | 0/18 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Epoch 2\n",
      "Training loss: 0.8649637136194441\n",
      "Validation loss: 0.825147807598114\n",
      "F1 Score (Weighted): 0.6694638462464257\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Epoch 3:   0%|          | 0/18 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Epoch 3\n",
      "Training loss: 0.7995994885762533\n",
      "Validation loss: 0.7879895369211832\n",
      "F1 Score (Weighted): 0.6994331002108932\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "from sklearn.metrics import f1_score\n",
    "from sklearn.metrics import classification_report\n",
    "\n",
    "# BERT Model Fine Tuning\n",
    "# During this step, we are in trainig mode which is set by train() function and gradient update is enabled \n",
    "for epoch in tqdm(range(1, epochs+1)):\n",
    "    #Enable training by learning new weights\n",
    "    bert_model.train()\n",
    "    \n",
    "    loss_train_total = 0\n",
    "\n",
    "    progress_bar = tqdm(loader_traindata, desc='Epoch {:1d}'.format(epoch), leave=False, disable=False)\n",
    "    for batch in progress_bar:\n",
    "        #Sets the gradients of all optimized torch.Tensors to zero\n",
    "        bert_model.zero_grad()\n",
    "        \n",
    "        batch = tuple(b.to(device) for b in batch)\n",
    "        \n",
    "        inputs = {'input_ids':      batch[0],\n",
    "                  'attention_mask': batch[1],\n",
    "                  'labels':         batch[2],\n",
    "                 }       \n",
    "\n",
    "        outputs = bert_model(**inputs)\n",
    "        \n",
    "        loss = outputs[0]\n",
    "        loss_train_total += loss.item()\n",
    "        loss.backward()\n",
    "        # Clips gradient norm of parameters. The norm is computed over all gradients together, \n",
    "        # as if they were concatenated into a single vector. Gradients are modified in-place \n",
    "        torch.nn.utils.clip_grad_norm_(bert_model.parameters(), 1.0)\n",
    "\n",
    "        adam_optimizer.step()\n",
    "        scheduler.step()\n",
    "        \n",
    "        progress_bar.set_postfix({'training_loss': '{:.3f}'.format(loss.item()/len(batch))})\n",
    "         \n",
    "        \n",
    "    torch.save(bert_model.state_dict(), f'finetuned_BERT_epoch_{epoch}.model')\n",
    "        \n",
    "    tqdm.write(f'\\nEpoch {epoch}')\n",
    "    \n",
    "    loss_train_avg = loss_train_total/len(loader_traindata)            \n",
    "    tqdm.write(f'Training loss: {loss_train_avg}')\n",
    "    \n",
    "    # Evaluating fine tuned model after each epoch of training\n",
    "    val_loss, predictions, true_vals = evaluate(loader_valdata)\n",
    "    val_f1 = f1_score(true_vals.flatten(), np.argmax(predictions, axis = 1).flatten(), average = 'weighted')\n",
    "    \n",
    "    \n",
    "    tqdm.write(f'Validation loss: {val_loss}')\n",
    "    tqdm.write(f'F1 Score (Weighted): {val_f1}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a28529c0",
   "metadata": {},
   "source": [
    "### Load finetuned BERT model and evaluate it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "3a3a02c3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BertForSequenceClassification(\n",
       "  (bert): BertModel(\n",
       "    (embeddings): BertEmbeddings(\n",
       "      (word_embeddings): Embedding(30522, 768, padding_idx=0)\n",
       "      (position_embeddings): Embedding(512, 768)\n",
       "      (token_type_embeddings): Embedding(2, 768)\n",
       "      (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "      (dropout): Dropout(p=0.1, inplace=False)\n",
       "    )\n",
       "    (encoder): BertEncoder(\n",
       "      (layer): ModuleList(\n",
       "        (0-11): 12 x BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "            (intermediate_act_fn): GELUActivation()\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "      )\n",
       "    )\n",
       "    (pooler): BertPooler(\n",
       "      (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "      (activation): Tanh()\n",
       "    )\n",
       "  )\n",
       "  (dropout): Dropout(p=0.1, inplace=False)\n",
       "  (classifier): Linear(in_features=768, out_features=5, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bert_model = BertForSequenceClassification.from_pretrained(\"bert-base-uncased\",\n",
    "                                                      num_labels=df['sentiment'].nunique(),\n",
    "                                                      output_attentions=False,\n",
    "                                                      output_hidden_states=False)\n",
    "\n",
    "bert_model.to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "c910e23a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<All keys matched successfully>"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bert_model.load_state_dict(torch.load('finetuned_BERT_epoch_3.model', map_location=torch.device('cpu')))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "af3e5e36",
   "metadata": {},
   "outputs": [],
   "source": [
    "_, predictions, true_vals = evaluate(loader_valdata)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "7670288e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.00      0.00      0.00         0\n",
      "           1       1.00      0.78      0.88       364\n",
      "           2       0.09      1.00      0.17         5\n",
      "           3       0.00      0.00      0.00         0\n",
      "           4       0.00      0.00      0.00         0\n",
      "\n",
      "    accuracy                           0.78       369\n",
      "   macro avg       0.22      0.36      0.21       369\n",
      "weighted avg       0.99      0.78      0.87       369\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaIAAAEmCAYAAAAgKpShAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAjJUlEQVR4nO3deZwU5Z3H8c9vGBC8JTIDkVFRUCOoGNGYsImCuqCgghqDGsMrUUddiZqwMSobNSaY7HplE4OKYtR4ongQQaIiiBIPwANQNGJkYRQGDzy5nJnf/tEFNuMc3UN3P13V37evetFd3VX1e6bG/s5TVf2UuTsiIiKhlIUuQERESpuCSEREglIQiYhIUAoiEREJSkEkIiJBlYcuoDlr6yjZy/nWrK8PXUIwnTq0C12CSEF1LMdyub5O+4/K6rNzzUvX5XT7bVG0QSQiIm1g8TvQpSASEUkSC97ByZqCSEQkSdQjEhGRoNQjEhGRoNQjEhGRoNQjEhGRoNQjEhGRoNQjEhGRoNQjEhGRoNQjEhGRoNQjEhGRoNQjEhGRoNQjEhGRoBREIiISVJkOzYmISEjqEYmISFC6WEFERIJSj0hERIJSj0hERIKKYY8ofhXn2eynZ3HMkEEMHXwEE24aH7qcvPrtZWM4cuC/cfIJx3zltTtvv4WD99+bj1atClBZ4ZXSfm9MbU9Y282ym4qAgihNfX09V4y9nHE33MyDk6cwbeojvLV4ceiy8mbI0cO59s9f/Z+vdsVyXnjuWbp27RagqsIrtf2eTm1PYNutLLupCBRHFUVi4YL5VFXtQveqKtp36MDgo4Ywc8b00GXlzf4H9GPb7bb7yvw/XPXfjDpvdNH8tZRvpbbf06ntCWx7DHtEeTtHZGZ7AccCOwEOvAtMdvdF+drm5lpZW0vXbl03Pq+orGTB/PkBKyq8WTOfpEtFBb323Ct0KQVTyvtdbU9g24ukl5ONvFRsZr8E7gEMeAGYEz2+28wubGG5ajOba2ZzQxyvdbypmgpeRyhr16zh1gk3Un32T0OXUlClvN/V9k0lou3qEW10GtDb3b9In2lm1wCvAr9vaiF3Hw+MB1hb18RvSZ5VVnZlxfIVG5+vrK2loqKi0GUEU1OzjOXvvMMPfzAcgPdW1jLy5OO55a/38rUduwSuLn9Keb+r7Qlsu3pEGzUAX29ifrfotaLUu88+LF26hJqaZXyxfj3Tpk7hkAEDQ5dVMD177cGjTz7DQ1Of4KGpT9ClopLb7pqU6BCC0t7vansC257jixXMrMrMZpjZIjN71czOi+ZfZmbvmNnL0XRU2jIXmdliM3vDzAa1to189YjOB6ab2ZvAsmjezkBPYFSetrnZysvLuWjMJZxdfToNDfUMG348PXv2Cl1W3vzqwv/kxXkv8NFHH3H0oAGccdYojhl+fOiyCq7U9ns6tT2Bbc/94bY6YLS7v2hm2wDzzOzx6LVr3f2qTTdvewMjgN6kOiRPmNke7l7fbMnu+TkCZmZlwEGkLlYwoAaY01Ix6UIcmisWa9Zn9CNKpE4d2oUuQaSgOpaT0+TodOyNWX12rnn4zKy2b2YPA9cB/YHPmgiiiwDc/XfR878Dl7n7s82tM29Xzbl7A/BcvtYvIiJNyLJHZGbVQHXarPHR+fqm3rsrsD/wPKkgGmVmPwLmkuo1rSLV+Uj/7K+J5jUrfme1RESkeVmeI3L38e7eL21qLoS2BiYB57v7J8D1wO5AX2A5cPWGtzaxeIu9NI01JyKSJHm4JNvM2pMKoTvd/QEAd69Ne/0m4JHoaQ1QlbZ4d1LfI22WekQiIgliZllNGazPgAnAIne/Jm1++hhgw4GF0ePJwAgz28LMegC9SH2ftFnqEYmIJIjl/lbh/YFTgQVm9nI072LgJDPrS+qw2xLgTAB3f9XMJgKvkbri7pzWLlJTEImIJEiuR4dw92do+rzP1BaWGQuMzXQbCiIRkQSJ4zBFCiIRkQRREImISFAKIhERCSt+OaQgEhFJEvWIREQkKAWRiIgEpSASEZGgFEQiIhJW/HJIQSQikiTqEYmISFAKIhERCUpBJCIiYcUvhxREIiJJoh6R5MTX+58XuoRgVs25LnQJIrGmIBIRkaAURCIiEpSCSEREwopfDimIRESSRD0iEREJSkEkIiJBKYhERCSs+OWQgkhEJEnUIxIRkaAURCIiEpSCSEREglIQiYhIWPHLIQWRiEiSqEckIiJBKYhERCSoGOaQgkhEJEnUIxIRkaBimEMKIhGRJFGPSEREgophDimIRESSpKwsfklUFrqAYjP76VkcM2QQQwcfwYSbxocuJ+e6V27PtPHn8tKk/2Le/WM456RDAdh3j5146rbRPHfPhTxz5wX0673LJstVdd2B92ZfzfmnHhag6vxL+n5vidqerLabZTe1vj6rMrMZZrbIzF41s/Oi+Z3N7HEzezP6d4e0ZS4ys8Vm9oaZDWptGwqiNPX19Vwx9nLG3XAzD06ewrSpj/DW4sWhy8qpuvoGLrzmAfY//rcc8qOrOPMH32Ov3boy9vxhjB3/KAeP+D2/uf4Rxp4/bJPl/uc/j+ex2a+GKTrPSmG/N0dtT17by8osqykDdcBod/8GcDBwjpntDVwITHf3XsD06DnRayOA3sBgYJyZtWux5ja3NoEWLphPVdUudK+qon2HDgw+aggzZ0wPXVZOrXj/E15+vQaAz1av4/W3V/D1LtvjDttu1RGA7bbuxPL3Pt64zNGH7svbNe/z2lsrgtScb6Ww35ujtiev7WaW1dQad1/u7i9Gjz8FFgE7AccCt0Vvuw0YFj0+FrjH3de5+9vAYuCglrahIEqzsraWrt26bnxeUVlJbW1twIrya+dunem7Z3fmLFzCL666nyvOH8abj/6G3/1sOJf86WEAtuzYgdE/PoKxN04NXG3+lNp+T6e2J6/t2QaRmVWb2dy0qbqFde8K7A88D1S6+3JIhRVQEb1tJ2BZ2mI10bxmFTyIzOzHLby28QcS4nit403VVPA6CmGrTh24+6rT+cVVk/j087VUf/+7XHD1A/Q68ldccNUkrr/0FAB+dfYQ/nTHk3y+Zn3givOnlPZ7Y2r7ppLQ9mzPEbn7eHfvlzY1+eFrZlsDk4Dz3f2TlkpoYt5Xf9hpQlw192vgL029EP0AxgOsrWu58HyorOzKiuVfHn5aWVtLRUVFC0vEU3l5GXdfdQb3PjqXh598BYBThn6L0f9zPwCTHn+JcZecDMCBfXZh+OF9GXv+MLbbphMNDc7a9V9ww72zgtWfa6Wy35uitiev7fkIUzNrTyqE7nT3B6LZtWbWzd2Xm1k3YGU0vwaoSlu8O/BuS+vPS4/IzOY3My0AKvOxzVzo3Wcfli5dQk3NMr5Yv55pU6dwyICBocvKuRsuPYU33l7BH+94cuO85e99zHcP6AXAoQftweKl7wFw+Gl/YK8hl7LXkEu57s6ZXDnhsUSFEJTOfm+K2p68tufhqjkDJgCL3P2atJcmAyOjxyOBh9PmjzCzLcysB9ALeKGlbeSrR1QJDAJWNZpvwD/ytM3NVl5ezkVjLuHs6tNpaKhn2PDj6dmzV+iycuo7fXfjlKHfYsE/3+G5ey4E4NLrJnPOb+7iyl+cQHl5GevW1THqt3cHrrRwSmG/N0dtT17b89Aj6g+cCiwws5ejeRcDvwcmmtlpwFLg+wDu/qqZTQReI3XF3TnuXt9ize65PwJmZhOAv7j7M028dpe7n9zaOkIcmisWOxw4KnQJwayac13oEkQKqmN5bm9l1++3M7L67Jz7XwOCnxjLS4/I3U9r4bVWQ0hERNomjhdcaIgfEZEEiWEOKYhERJJEPSIREQkqhjmkIBIRSRL1iEREJKgY5pCCSEQkSdQjEhGRoGKYQwoiEZEkUY9IRESCUhCJiEhQMcwhBZGISJKoRyQiIkHFMIcURCIiSaIekYiIBBXDHFIQiYgkSVkMk0hBJCKSIDHMIQWRiEiS6ByRiIgEVRa/HFIQiYgkiXpEkhPzpvx36BKCqW/w0CUE0y6Of8pK0YlhDimIRESSxIhfEimIREQSJI4d62aDyMz+BDR7nMTdz81LRSIi0mZJO0c0t2BViIhITsQwh5oPIne/Lf25mW3l7p/nvyQREWmrOI6sUNbaG8zs22b2GrAoer6fmY3Le2UiIpI1s+ymYtBqEAF/AAYBHwC4+yvA9/JYk4iItJGZZTUVg4yumnP3ZY0Krs9POSIisjmKJFuykkkQLTOz7wBuZh2Ac4kO04mISHGJ4zmiTILoLOB/gZ2Ad4C/A+fksygREWmbRAaRu78PnFKAWkREZDPF8QutmVw1t5uZ/c3M3jOzlWb2sJntVojiREQkO3G8WCGTq+buAiYC3YCvA/cBd+ezKBERaZukXr5t7v5Xd6+LpjtoYegfEREJJ1E9IjPrbGadgRlmdqGZ7Wpmu5jZBcCUwpUoIiKZKrPsptaY2S3RaZmFafMuM7N3zOzlaDoq7bWLzGyxmb1hZoMyqbmlixXmker5bCj1zLTXHPhNJhsQEZHCyUMv51bgOuD2RvOvdferGm17b2AE0JvUqZwnzGwPd2/xu6ctjTXXoy0Vi4hIOLmOIXefZWa7Zvj2Y4F73H0d8LaZLQYOAp5taaGMRlYwsz7A3kDHtOIap6OIiASW7feIzKwaqE6bNd7dx2ew6Cgz+xGpOzWMdvdVpL5v+lzae2qieS1qNYjM7FLgUFJBNBU4EniGr3bTREQksGyPzEWhk0nwpLue1OmZDadprgZ+QtMdslYvbsvkqrkTgMOAFe7+Y2A/YItMqxURkcIpxFVz7l7r7vXu3gDcROrwG6R6QFVpb+0OvNva+jIJojXRxurMbFtgJZDYL7TOfnoWxwwZxNDBRzDhpmz/SIin+vp6fn7GSfz2otRNd99e/E9+ec5IzvvJiYy9+DxWf/5Z4Arzb8iggZw4/GhGnDCMU35wfOhyCqoUf+c3SGLbC/E9IjPrlvZ0OLDhirrJwAgz28LMegC9gBdaW18m54jmmtn2pFJvHvBZJiuOo/r6eq4Yezk33vQXKisrOfkHJ3DogIHs3rNn6NLy6pFJd9N95x6sXp0KnHFXXc7Is35Gn74H8MTUh3jo3ts5+Sf/EbjK/LvxltvZYYcdQpdRUKX6Ow/JbXuux5ozs7tJnZ7Z0cxqgEuBQ82sL6nDbkuIrqp291fNbCLwGlAHnNPaFXOQQY/I3f/D3T9y9xuAI4CR0SG6xFm4YD5VVbvQvaqK9h06MPioIcycMT10WXn1/nu1zHvuaQ4fMmzjvHeW/R+99/smAH37Hcyzs5L9Myhlpfg7v0FS257rHpG7n+Tu3dy9vbt3d/cJ7n6qu+/j7vu6+zHuvjzt/WPdfXd339PdH82k5pa+0PrNxhPQGSiPHrfyw7C9zOwwM9u60fzBmRQWwsraWrp267rxeUVlJbW1tQEryr9brruKkWeeR1nZl78KO/fYnRdmPwXA7JlP8P7KZP8MIHVc/ZwzT+PkE49j0n33hi6nYErxd36DpLY9jiMrtHRo7uoWXnNgYHMvmtm5pG4VsQiYYGbnufvD0ctXANOaWW7jZYTXjbuR086obupteeNNXNxRLDsqH+Y8O4vttu/M7nvuzcKX526cP+qCS7n5T1cy8fbxHNT/EMrbtw9YZWH85fa76FJRyYcffMDZ1T9h1x67cUC/A0OXlXel9jufLqltz+TEf7Fp6QutAzZjvWcAB7j7Z9EXoe43s13d/X9p4ftW6ZcRrq0r/Hh2lZVdWbF8xcbnK2trqaioKHQZBfP6wleY84+nmPf8M3yxfj2rV3/OtWPH8LMxY7nsynFA6jDd3OeeCVxp/nWpqASg89e+xoDDDufVhfNLIohK7Xc+XVLbHscwzVd4tnP3zwDcfQmpE11Hmtk15P6LvznTu88+LF26hJqaZXyxfj3Tpk7hkAHNdvxi79QzfsrN901j/D1TGH3J79hn/378bMxYPlr1IQANDQ3c/9ebGXR0sq8iW7N6NZ9HVwauWb2a5/4xm9177hG4qsIotd/5dElte67HmiuEjEZWaIMVZtbX3V8GiHpGQ4FbgH3ytM3NVl5ezkVjLuHs6tNpaKhn2PDj6dmzV+iyCu7p6dN49OGJABz83YEcduSxgSvKrw8++IDR548CUldSDT5qKP3/7buBqyqMUv6dT2rbiyVcsmHuuT8CZmbdgTp3X9HEa/3dfXZr6whxaK5Y/Gvl56FLCGaXHbcMXUIw7eL4CSKbrWN5bo8Sjf7bG1l9dl599J7Bf/EyGeLHSN0qfDd3v9zMdga6unuz3yVy95oWXms1hEREpG3i+PdMJueIxgHfBk6Knn8K/DlvFYmISJvF8Q6tmZwj+pa7f9PMXgJw91Vm1iHPdYmISBvkemSFQsgkiL4ws3ZEI6iaWRegIa9ViYhIm8Txe0SZ1PxH4EGgwszGkroFxBV5rUpERNokkYfm3P1OM5tH6lYQBgxz90V5r0xERLKWyENz0VVyq4G/pc9z96X5LExERLIXwxzK6BzRFFLnh4zUrcJ7AG8AvfNYl4iItEEcL9/O5NDcJiMhRCNvn5m3ikREpM0SeWiuMXd/0cySPxqkiEgMxTCHMjpH9PO0p2XAN4H38laRiIi0WSIPzQHbpD2uI3XOaFJ+yhERkc1hxXuDg2a1GETRF1m3dvdfFKgeERHZDInqEZlZubvXZXJbcBERKQ6JCiLgBVLng142s8nAfcDG+xO4+wN5rk1ERLIUxzu0ZnKOqDPwATCQL79P5ICCSESkyCStR1QRXTG3kC8DaIOSvWmdiEgxi2GHqMUgagdsDU1egqEgEhEpQkn7Qutyd7+8YJWIiMhmaxfD+0C0FETxi1URkRJXFsOP7paC6LCCVSGbqOrcKXQJwbSL45lWkSISwyNzzQeRu39YyEJERGTzxfFvuawHPRURkeKVtIsVREQkZmKYQwoiEZEkUY9IRESCimEOKYhERJIkhl8jUhCJiCRJUgc9FRGRmIhfDMWzFyciIs0oM8tqao2Z3WJmK81sYdq8zmb2uJm9Gf27Q9prF5nZYjN7w8wGZVRzm1oqIiJFybKcMnArMLjRvAuB6e7eC5gePcfM9gZGAL2jZcZFd/pukYJIRCRBzLKbWuPus4DGI+0cC9wWPb4NGJY2/x53X+fubwOLgYNa24aCSEQkQcws26nazOamTdUZbKbS3ZcDRP9WRPN3Apalva8mmtciXawgIpIg2fYu3H08MD5Hm2/T/esURCIiCVKgy7drzaybuy83s27Aymh+DVCV9r7uwLutrUyH5kREEiQPFys0ZTIwMno8Eng4bf4IM9vCzHoAvYAXWluZekQiIgmS6x6Rmd0NHArsaGY1wKXA74GJZnYasBT4PoC7v2pmE4HXgDrgHHevb3Ub7q0evgtibV3rxxWT6ou6htAlBNO+XJ10KS0dy3P7HdQHXlme1Wfncft1C/4dWPWIREQSREP8iIhIUPGLIQWRiEiixLBDpKvmGpv99CyOGTKIoYOPYMJNubq0Ph7u/OutnDh8KCcedzQX/3I069atC11SwZTyflfbk9X2MiyrqRgoiNLU19dzxdjLGXfDzTw4eQrTpj7CW4sXhy6rIFbW1nLvXXdw+933M/GBv9HQ0MBj06aGLqsgSnm/q+3Ja3uuh/gpBAVRmoUL5lNVtQvdq6po36EDg48awswZ00OXVTD19fWsW7eWuro61q5ZQ5cuFa0vlAClvN/V9uS13bL8rxjkLYjM7CAzOzB6vLeZ/dzMjsrX9nJhZW0tXbt13fi8orKS2tragBUVTkVlJT8c+WOGDjqMwYd/j6232YaDv9M/dFkFUcr7XW1PXtvVI4qY2aXAH4Hrzex3wHXA1sCFZjamheU2Dr4X4nitN/HVpTheCtkWn3zyMU/NeJLJUx9n2uNPsWbNGqY+Mjl0WQVRyvtdbd9UEtoex3NE+bpq7gSgL7AFsALo7u6fmNmVwPPA2KYWSh98L8QXWisru7Ji+YqNz1fW1lJRURqHp1547lm+vtNO7NC5MwADDjuc+a+8xFFDjwlcWf6V8n5X25PX9jhmab4OzdW5e727rwbecvdPANx9DVC0wwb07rMPS5cuoaZmGV+sX8+0qVM4ZMDA0GUVRNeu3Vg4/xXWrlmDuzPn+efYtcfuocsqiFLe72p78toex0Nz+eoRrTezLaMgOmDDTDPbjiIOovLyci4acwlnV59OQ0M9w4YfT8+evUKXVRB99t2Pw44YxCkjjqddu3bsudc3OO6EE0OXVRClvN/V9uS1vVguQMhGXsaaM7Mt3P0rX0Ixsx2Bbu6+oLV1aKy50qSx5qTU5Hqsuemvv5/VZ+dhe+0YPLny0iNqKoSi+e8D7+djmyIiEs8ekYb4ERFJkGI575MNBZGISIKoRyQiIkGVxS+HFEQiIkmiHpGIiASlc0QiIhJUuxgmkYJIRCRB4hdDCiIRkWSJYRIpiEREEkQXK4iISFAxPEWkIBIRSZIY5pCCSEQkUWKYRAoiEZEE0TkiEREJSueIREQkqBjmkIJIRCRRYphECiIRkQTROSIREQlK54hERCSoGOaQgqgYeegCRCS+YphECiIRkQTROSIREQlK54hERCSofOSQmS0BPgXqgTp372dmnYF7gV2BJcCJ7r6qLesvy02ZIiJSFCzLKXMD3L2vu/eLnl8ITHf3XsD06HmbKIhERBLEsvxvMxwL3BY9vg0Y1tYVKYhERBLELNvJqs1sbtpU3cRqHXjMzOalvV7p7ssBon8r2lqzzhGJiCRItn0cdx8PjG/lbf3d/V0zqwAeN7PX21Zd09QjEhFJkjycI3L3d6N/VwIPAgcBtWbWDSD6d2VbS1YQiYgkSK7PEZnZVma2zYbHwL8DC4HJwMjobSOBh9tasw7NiYgkSB6+R1QJPGipFZcDd7n7NDObA0w0s9OApcD327oBBZGISILkOofc/V/Afk3M/wA4LBfbUBCJiCSJRlYQEZGQNNaciIgEpbHmREQkqBjmkIJIRCRRYphECiIRkQTROSIREQlK54hERCSoGOaQgkhEJFFimEQaa66R2U/P4pghgxg6+Agm3NTagLTJcvedt/OD447mxOFDueuO21pfIEFKeb+r7clqewHvR5QzCqI09fX1XDH2csbdcDMPTp7CtKmP8NbixaHLKojFb/6Thybdx213TuSu+x7imVkzWfp/S0KXVRClvN/V9uS1Pdv7ERUDBVGahQvmU1W1C92rqmjfoQODjxrCzBnTQ5dVEEve/hf77LsfHTt1ory8nG8ecCAzn3widFkFUcr7XW1PXtvLLLupGBQsiMzs9kJtq61W1tbStVvXjc8rKiupra0NWFHh7N6zFy/Nm8tHH61i7Zo1/OOZWdSuWBG6rIIo5f2utiex7Xm4IVGe5eViBTOb3HgWMMDMtgdw92OaWa4aqAa4btyNnHZGU3eszR/Hm6qpoDWE0mO33fnRj09n1JmnseWWW9Jrj71oV94udFkFUcr7XW3fVBLaHscm5Ouque7Aa8DNpO51bkA/4OqWFkq/Ze3auiZ+S/KssrIrK5Z/2QtYWVtLRUWbb8MeO8cedwLHHncCAH/+47VUVFYGrqgwSnm/q+3Ja3sMcyhvh+b6AfOAMcDH7j4TWOPuT7n7U3na5mbr3Wcfli5dQk3NMr5Yv55pU6dwyICBocsqmA8/+ACAFcvfZcb0xxl05JDAFRVGKe93tT15bY/jxQp56RG5ewNwrZndF/1bm69t5VJ5eTkXjbmEs6tPp6GhnmHDj6dnz16hyyqYX44+j48//ojy8nIuuPhXbLvtdqFLKohS3u9qe/LaXiyXZGfD3PN/BMzMhgD93f3iTJcJcWiuWKyvawhdQjAdynUhp5SWjuW5TY4Vn3yR1Wdn123bB0+uggRRWyiISpOCSEpNroOoNssgqiyCICr6w2UiIpK5Yjnvkw0FkYhIgsTxHJGCSEQkSeKXQwoiEZEkiWEOKYhERJJE54hERCQonSMSEZGg4tgj0pc2REQkKPWIREQSJI49IgWRiEiC6ByRiIgEpR6RiIgEFcMcUhCJiCRKDJNIQSQikiA6RyQiIkHpHJGIiAQVwxzSF1pFRBLFspwyWaXZYDN7w8wWm9mFuS5ZQSQikiCW5X+trs+sHfBn4Ehgb+AkM9s7lzUriEREEsQsuykDBwGL3f1f7r4euAc4Npc1F+05olzfxz1bZlbt7uNDbLtjedi/D0K2PTS1vfTanrR2Z/vZaWbVQHXarPGNfh47AcvSntcA32p7hV+lHlHzqlt/S2Kp7aWpVNtequ0GwN3Hu3u/tKlxKDcVbJ7LGhREIiLSkhqgKu15d+DdXG5AQSQiIi2ZA/Qysx5m1gEYAUzO5QaK9hxREUjMMeM2UNtLU6m2vVTbnRF3rzOzUcDfgXbALe7+ai63Ye45PdQnIiKSFR2aExGRoBREIiISlIKokXwPZVHMzOwWM1tpZgtD11JIZlZlZjPMbJGZvWpm54WuqVDMrKOZvWBmr0Rt/3XomgrNzNqZ2Utm9kjoWkqVgihNIYayKHK3AoNDFxFAHTDa3b8BHAycU0L7fR0w0N33A/oCg83s4LAlFdx5wKLQRZQyBdGm8j6URTFz91nAh6HrKDR3X+7uL0aPPyX1obRT2KoKw1M+i562j6aSuYLJzLoDQ4CbQ9dSyhREm2pqKIuS+ECSFDPbFdgfeD5wKQUTHZp6GVgJPO7uJdN24A/ABUBD4DpKmoJoU3kfykKKl5ltDUwCznf3T0LXUyjuXu/ufUl9Y/4gM+sTuKSCMLOhwEp3nxe6llKnINpU3oeykOJkZu1JhdCd7v5A6HpCcPePgJmUznnC/sAxZraE1GH4gWZ2R9iSSpOCaFN5H8pCio+ZGTABWOTu14Sup5DMrIuZbR897gQcDrwetKgCcfeL3L27u+9K6v/1J939h4HLKkkKojTuXgdsGMpiETAx10NZFDMzuxt4FtjTzGrM7LTQNRVIf+BUUn8RvxxNR4UuqkC6ATPMbD6pP8Qed3ddxiwFpSF+REQkKPWIREQkKAWRiIgEpSASEZGgFEQiIhKUgkhERIJSEEkwZlYfXSq90MzuM7MtN2Ndt5rZCdHjm1satNTMDjWz77RhG0vMbMdM5zd6z2ctvd7E+y8zs//MtkaROFIQSUhr3L2vu/cB1gNnpb8YjYaeNXc/3d1fa+EthwJZB5GI5IeCSIrF00DPqLcyw8zuAhZEA3JeaWZzzGy+mZ0JqdEQzOw6M3vNzKYAFRtWZGYzzaxf9Hiwmb0Y3W9nejSo6VnAz6Le2Hej0QUmRduYY2b9o2W/ZmaPRfequZGmxyLchJk9ZGbzonv7VDd67eqolulm1iWat7uZTYuWedrM9srJT1MkRspDFyBiZuWk7gE1LZp1ENDH3d+OPsw/dvcDzWwLYLaZPUZqhOw9gX2ASuA14JZG6+0C3AR8L1pXZ3f/0MxuAD5z96ui990FXOvuz5jZzqRG1vgGcCnwjLtfbmZDgE2CpRk/ibbRCZhjZpPc/QNgK+BFdx9tZpdE6x4FjAfOcvc3zexbwDhgYBt+jCKxpSCSkDpFtx+AVI9oAqlDZi+4+9vR/H8H9t1w/gfYDugFfA+4293rgXfN7Mkm1n8wMGvDuty9uXstHQ7snRpyDoBtzWybaBvHRctOMbNVGbTpXDMbHj2uimr9gNRtBu6N5t8BPBCN9v0d4L60bW+RwTZEEkVBJCGtiW4/sFH0gfx5+izgp+7+90bvO4rWb9FhGbwHUoeov+3ua5qoJeMxsMzsUFKh9m13X21mM4GOzbzdo+1+1PhnIFJqdI5Iit3fgbOj2zRgZnuY2VbALGBEdA6pGzCgiWWfBQ4xsx7Rsp2j+Z8C26S97zFSh8mI3tc3ejgLOCWadySwQyu1bgesikJoL1I9sg3KgA29upNJHfL7BHjbzL4fbcPMbL9WtiGSOAoiKXY3kzr/86KZLQRuJNWTfxB4E1gAXA881XhBd3+P1HmdB8zsFb48NPY3YPiGixWAc4F+0cUQr/Hl1Xu/Br5nZi+SOkS4tJVapwHl0UjWvwGeS3vtc6C3mc0jdQ7o8mj+KcBpUX2vUkK3phfZQKNvi4hIUOoRiYhIUAoiEREJSkEkIiJBKYhERCQoBZGIiASlIBIRkaAURCIiEtT/A2nuYi2won3tAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#import tensorflow as tf\n",
    "import numpy as np\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn.metrics import classification_report\n",
    "\n",
    "predictions_flat = np.argmax(predictions, axis=1).flatten()\n",
    "true_vals_flat = true_vals.flatten()\n",
    "\n",
    "cm = confusion_matrix(true_vals_flat, predictions_flat)\n",
    "#cm = tf.math.confusion_matrix(true_vals_flat, predictions_flat).numpy()\n",
    "con_mat_df = pd.DataFrame(cm)\n",
    "    \n",
    "print(classification_report(predictions_flat, true_vals_flat))\n",
    "\n",
    "sns.heatmap(con_mat_df, annot=True, fmt='g', cmap=plt.cm.Blues)\n",
    "plt.tight_layout()\n",
    "plt.ylabel('True label')\n",
    "plt.xlabel('Predicted label')\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}