Session 2 - embeddings.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Session 2 - embeddings.ipynb",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.8"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "heading_collapsed": true,
        "id": "0XJc38cLugS-"
      },
      "source": [
        "# TF-IDF"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "KjXq4o9SugS_"
      },
      "source": [
        "# !pip install spacy\n",
        "# !pip install numpy\n",
        "# !pip install pandas\n",
        "# !pip install spacy\n",
        "# !pip install sklearn\n",
        "# !pip install matplotlib"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "3oRtjzFQYbMv"
      },
      "source": [
        "import spacy\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "from spacy.lang.fr import French\n",
        "from sklearn.manifold import MDS\n",
        "from sklearn.preprocessing import MinMaxScaler\n",
        "from sklearn.feature_extraction.text import TfidfVectorizer"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "vWzzZG9JYiwB"
      },
      "source": [
        "texteNLP = \"\"\"Le traitement automatique du langage naturel (abr. TALN), ou traitement automatique de la langue naturelle, \n",
        " ou encore traitement automatique des langues (abr. TAL) est un domaine multidisciplinaire impliquant la linguistique,\n",
        " l'informatique et l'intelligence artificielle, qui vise à créer des outils de traitement de la langue naturelle pour diverses applications.\n",
        " Il ne doit pas être confondu avec la linguistique informatique, qui vise à comprendre les langues au moyen d'outils informatiques.\n",
        " Le TALN est sorti des laboratoires de recherche pour être progressivement mis en œuvre dans des applications informatiques nécessitant\n",
        " l'intégration du langage humain à la machine. Aussi le TALN est-il parfois appelé ingénierie linguistique.\n",
        " En France, le traitement automatique de la langue naturelle a sa revue, Traitement automatique des langues,\n",
        " publiée par l’Association pour le traitement automatique des langues (ATALA).\"\"\"\n",
        "\n",
        "texteDataScience = \"\"\"En termes généraux, la science des données est l'extraction de connaissance d'ensembles de données.\n",
        " Elle emploie des techniques et des théories tirées de plusieurs autres domaines plus larges des mathématiques, analyse,\n",
        " optimisation et statistique principalement, la théorie de l'information et la technologie de l'information, notamment le traitement de signal,\n",
        " des modèles probabilistes, l'apprentissage automatique, l'apprentissage statistique, la programmation informatique, l'ingénierie de données,\n",
        " la reconnaissance de formes et l'apprentissage, la visualisation, l'analytique prophétique, la modélisation d'incertitude, le stockage de données,\n",
        " la géo-visualisation, la compression de données et le calcul à haute performance.\n",
        " Les méthodes qui s'adaptent aux données de masse sont particulièrement intéressantes dans la science des données,\n",
        " bien que la discipline ne soit généralement pas considérée comme limitée à ces données.\n",
        "\n",
        "La science des données (en anglais data science) est une discipline qui s'appuie sur des outils mathématiques, de statistiques,\n",
        " d'informatique (cette science est principalement une « science des données numériques ») et de visualisation des données.\n",
        " Elle est en plein développement, dans le monde universitaire ainsi que dans le secteur privé et le secteur public.\n",
        " Moore en 1991 a défini la statistique comme la science des données6 (définition reprise par d'autres dont James T. McClave et al. en 1997)\n",
        " et U. Beck en 2001 oppose la science des données à la science de l'expérience, voyant une dissociation croissante entre ces deux types de science,\n",
        " que tendrait selon lui à encourager une société de la gestion du risque au sein d'une « civilisation du danger ».\"\"\"\n",
        "\n",
        "texteEconometrie = \"\"\"L'économétrie est une branche de la science économique qui a pour objectif d'estimer et de tester les modèles économiques.\n",
        " L'économétrie en tant que discipline naît dans les années 1930 avec la création de la société d'économétrie par Irving Fisher et Ragnar Frisch (1930)\n",
        " et la création de la revue Econometrica (1933). Depuis lors, l'économétrie n'a cessé de se développer et de prendre une importance croissante\n",
        " au sein de la science économique.\n",
        "\n",
        "L'économétrie théorique se focalise essentiellement sur deux questions, l'identification et l'estimation statistique. L'économétrie appliquée\n",
        " utilise les méthodes économétriques pour comprendre des domaines de l'économie comme l'analyse du marché du travail,\n",
        " l'économie de l'éducation ou encore tester la pertinence empirique des modèles de croissance.\n",
        "\n",
        "L'économétrie appliquée utilise aussi bien des données issues d'un protocole expérimental, que ce soit une expérience de laboratoire\n",
        " ou une expérience de terrain, que des données issues directement de l'observation du réel sans manipulation du chercheur.\n",
        " Lorsque l'économètre utilise des données issues directement de l'observation du réel, il est fréquent d'identifier des expériences naturelles\n",
        " pour retrouver une situation quasi-expérimentale. On parle parfois de révolution de crédibilité, terme controversé, pour désigner l'essor fulgurant\n",
        " de ces méthodes de recherche dans la discipline, et en économie en général.\"\"\"\n",
        "\n",
        "texteHistoire = \"\"\"L’histoire, souvent écrit avec la première lettre majuscule,\n",
        " est à la fois l’étude et l'écriture des faits et des événements passés quelles que soient leur variété et leur complexité.\n",
        " L'histoire est également une science humaine et sociale. On désigne aussi couramment sous le terme d’histoire (par synecdoque) le passé lui-même,\n",
        " comme dans les leçons de l'histoire. L'histoire est un récit écrit par lequel des hommes et des femmes (les historiens et historiennes)\n",
        " s'efforcent de faire connaître les temps révolus. Ces tentatives ne sont jamais entièrement indépendantes de conditionnements étrangers au domaine\n",
        " telle que la vision du monde de leur auteur ou de sa culture, mais elles sont censées être élaborées à partir de sources plutôt que guidées\n",
        " par la spéculation ou l'idéologie.\n",
        "\n",
        "Au cours des siècles, les historiens ont façonné leurs méthodes ainsi que les champs d'intervention, tout en réévaluant leurs sources,\n",
        " leur origine et leur exploitation. La discipline universitaire d'étude et écriture de l'histoire, y comprise la critique des méthodes,\n",
        " est l'historiographie. Elle s'appuie sur diverses sciences auxiliaires complétant selon les travaux menés la compétence générale de l'historien.\n",
        " Elle reste malgré tout une construction humaine, inévitablement inscrite dans son époque, susceptible d'être utilisée en dehors de son domaine,\n",
        " notamment à des fins d'ordre politique. \n",
        "\"\"\"\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "UH2LKmshYkDM"
      },
      "source": [
        "nlp = French()\n",
        "\n",
        "documentNLP = nlp(texteNLP)\n",
        "documentDataScience = nlp(texteDataScience)\n",
        "documentEconometrie = nlp(texteEconometrie)\n",
        "documentHistoire = nlp(texteHistoire)\n",
        "\n",
        "bagOfWordsNLP = []\n",
        "bagOfWordsDataScience = []\n",
        "bagOfWordsEconometrie = []\n",
        "bagOfWordsHistoire = []\n",
        "\n",
        "for token in documentNLP:\n",
        "    bagOfWordsNLP.append(token.text)\n",
        "\n",
        "for token in documentDataScience:\n",
        "    bagOfWordsDataScience.append(token.text)\n",
        "\n",
        "for token in documentEconometrie:\n",
        "    bagOfWordsEconometrie.append(token.text)\n",
        "\n",
        "for token in documentHistoire:\n",
        "    bagOfWordsHistoire.append(token.text)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "DUmZrE2lYk42"
      },
      "source": [
        "uniqueWords = set(bagOfWordsNLP).union(set(bagOfWordsDataScience)).union(set(bagOfWordsEconometrie)).union(set(bagOfWordsHistoire))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "U2k-oq2MYlFy"
      },
      "source": [
        "numOfWordsNLP = dict.fromkeys(uniqueWords, 0)\n",
        "for word in bagOfWordsNLP:\n",
        "    numOfWordsNLP[word] += 1\n",
        "\n",
        "numOfWordsDataScience = dict.fromkeys(uniqueWords, 0)\n",
        "for word in bagOfWordsDataScience:\n",
        "    numOfWordsDataScience[word] += 1\n",
        "\n",
        "numOfWordsEconometrie = dict.fromkeys(uniqueWords, 0)\n",
        "for word in bagOfWordsEconometrie:\n",
        "    numOfWordsEconometrie[word] += 1\n",
        "\n",
        "numOfWordsHistoire = dict.fromkeys(uniqueWords, 0)\n",
        "for word in bagOfWordsHistoire:\n",
        "    numOfWordsHistoire[word] += 1"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "U4UZsBqBYlac"
      },
      "source": [
        "def computeTF(wordDict, bagOfWords):\n",
        "    tfDict = {}\n",
        "    bagOfWordsCount = len(bagOfWords)\n",
        "    for word, count in wordDict.items():\n",
        "        tfDict[word] = count / float(bagOfWordsCount)\n",
        "    return tfDict"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "1eGebIApYlje"
      },
      "source": [
        "tfNLP = computeTF(numOfWordsNLP, bagOfWordsNLP)\n",
        "tfDataScience = computeTF(numOfWordsDataScience, bagOfWordsDataScience)\n",
        "tfEconometrie = computeTF(numOfWordsEconometrie, bagOfWordsEconometrie)\n",
        "tfHistoire = computeTF(numOfWordsHistoire, bagOfWordsHistoire)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "yw240CWgYlsr"
      },
      "source": [
        "def computeIDF(documents):\n",
        "    import math\n",
        "    N = len(documents)\n",
        "    \n",
        "    idfDict = dict.fromkeys(documents[0].keys(), 0)\n",
        "    for document in documents:\n",
        "        for word, val in document.items():\n",
        "            if val > 0:\n",
        "                idfDict[word] += 1\n",
        "    \n",
        "    for word, val in idfDict.items():\n",
        "        idfDict[word] = math.log(N / float(val))\n",
        "    return idfDict"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "C7_ejnuOZWez"
      },
      "source": [
        "idfs = computeIDF([numOfWordsNLP, numOfWordsDataScience, numOfWordsEconometrie, numOfWordsHistoire])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "vW2xwamlZWmq"
      },
      "source": [
        "def computeTFIDF(tfBagOfWords, idfs):\n",
        "    tfidf = {}\n",
        "    for word, val in tfBagOfWords.items():\n",
        "        tfidf[word] = val * idfs[word]\n",
        "    return tfidf"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "TB2O-vKAZWtm"
      },
      "source": [
        "tfidfNLP = computeTFIDF(tfNLP, idfs)\n",
        "tfidfDataScience = computeTFIDF(tfDataScience, idfs)\n",
        "tfidfEconometrie = computeTFIDF(tfEconometrie, idfs)\n",
        "tfidfHistoire = computeTFIDF(tfHistoire, idfs)\n",
        "df = pd.DataFrame([tfidfNLP, tfidfDataScience, tfidfEconometrie, tfidfHistoire])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "4KiDyXbubBF1"
      },
      "source": [
        "print(df)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "oLiRFXtzZWzE"
      },
      "source": [
        "vectorizer = TfidfVectorizer()\n",
        "vectors = vectorizer.fit_transform([texteNLP, texteDataScience, texteEconometrie, texteHistoire])\n",
        "feature_names = vectorizer.get_feature_names()\n",
        "dense = vectors.todense()\n",
        "denselist = dense.tolist()\n",
        "df = pd.DataFrame(denselist, columns=feature_names)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "w6LHAKsYZW37"
      },
      "source": [
        "print(df)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "rmYJ_oGYZW8Y"
      },
      "source": [
        "df.loc[0]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "3o4KAtR6ZXAs"
      },
      "source": [
        "dist_NLP_DataScience = np.linalg.norm(df.loc[0] - df.loc[1])\n",
        "dist_NLP_Econometrie = np.linalg.norm(df.loc[0] - df.loc[2])\n",
        "dist_NLP_Histoire = np.linalg.norm(df.loc[0] - df.loc[3])\n",
        "dist_DataScience_Econometrie = np.linalg.norm(df.loc[1] - df.loc[2])\n",
        "dist_DataScience_Histoire = np.linalg.norm(df.loc[1] - df.loc[3])\n",
        "dist_Econometrie_Histoire = np.linalg.norm(df.loc[2] - df.loc[3])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "qVQzDI2EZXE3",
        "scrolled": true
      },
      "source": [
        "print(dist_NLP_DataScience)\n",
        "print(dist_NLP_Econometrie)\n",
        "print(dist_NLP_Histoire)\n",
        "print(dist_DataScience_Econometrie)\n",
        "print(dist_DataScience_Histoire)\n",
        "print(dist_Econometrie_Histoire)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "czg_laqsugUU"
      },
      "source": [
        "data = df.values\n",
        "X = data.data\n",
        "scaler = MinMaxScaler()\n",
        "X_scaled = scaler.fit_transform(X)\n",
        "mds = MDS(2,random_state=0)\n",
        "X_2d = mds.fit_transform(X_scaled)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "L8p1DFP9ugUa"
      },
      "source": [
        "x = [row[0] for row in X_2d]\n",
        "y = [row[1] for row in X_2d]\n",
        "\n",
        "group = np.array(['NLP', 'DataScience', 'Econometrie', 'Histoire'])\n",
        "cdict = {0: 'red', 1: 'green', 2: 'blue', 3: 'orange'}\n",
        "\n",
        "fig, ax = plt.subplots()\n",
        "\n",
        "for i in range(0, 4):\n",
        "    ax.scatter(x[i], y[i], c = cdict[i], label = group[i])\n",
        "ax.legend()\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "heading_collapsed": true,
        "id": "bafsUlFOugUe"
      },
      "source": [
        "# Embeddings de mots"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "heading_collapsed": true,
        "hidden": true,
        "id": "WMGgjIn2k1Ig"
      },
      "source": [
        "## Word2Vec"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "nAftgmbGaCey"
      },
      "source": [
        "%reset -f"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "YEu7vZHiugUg"
      },
      "source": [
        "# !pip install gensim"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "dww0XB9-ZXXm"
      },
      "source": [
        "import gensim \n",
        "from gensim.models import Word2Vec "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "xYaB2OBaZXbG"
      },
      "source": [
        "texte1 = \"\"\"L’exploration de données, connue aussi sous l'expression de fouille de données, forage de données, prospection de données, data mining,\n",
        " ou encore extraction de connaissances à partir de données, a pour objet l’extraction d'un savoir ou d'une connaissance à partir de grandes quantités\n",
        " de données, par des méthodes automatiques ou semi-automatiques.\n",
        "Elle se propose d'utiliser un ensemble d'algorithmes issus de disciplines scientifiques diverses telles que les statistiques,\n",
        " l'intelligence artificielle ou l'informatique, pour construire des modèles à partir des données,\n",
        " c'est-à-dire trouver des structures intéressantes ou des motifs selon des critères fixés au préalable,\n",
        " et d'en extraire un maximum de connaissances.\n",
        "L'utilisation industrielle ou opérationnelle de ce savoir dans le monde professionnel permet de résoudre des problèmes très divers,\n",
        " allant de la gestion de la relation client à la maintenance préventive, en passant par la détection de fraudes ou encore l'optimisation de sites web.\n",
        "C'est aussi le mode de travail du journalisme de données.\n",
        "L'exploration de données fait suite, dans l'escalade de l'exploitation des données de l'entreprise, à l'informatique décisionnelle.\n",
        "Celle-ci permet de constater un fait, tel que le chiffre d'affaires, et de l'expliquer comme le chiffre d'affaires décliné par produits,\n",
        " tandis que l'exploration de données permet de classer les faits et de les prévoir dans une certaine mesure ou encore de les éclairer en révélant\n",
        " par exemple les variables ou paramètres qui pourraient faire comprendre pourquoi le chiffre d'affaires de tel point de vente est supérieur\n",
        " à celui de tel autre. \"\"\"\n",
        "\n",
        "texte2 = \"\"\"En statistique, les analyses multivariées ont pour caractéristique de s'intéresser à des lois de probabilité à plusieurs variables.\n",
        "Les analyses bivariées sont des cas particuliers à deux variables.\n",
        "Les analyses multivariées sont très diverses selon l'objectif recherché, la nature des variables et la mise en œuvre formelle.\n",
        "On peut identifier deux grandes familles : celle des méthodes descriptives (visant à structurer et résumer l'information)\n",
        " et celle des méthodes explicatives visant à expliquer une ou des variables dites « dépendantes » (variables à expliquer) par un ensemble de variables\n",
        " dites « indépendantes » (variables explicatives).\n",
        "Les méthodes appelées en français analyse des données en sont un sous-ensemble. \"\"\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "js32mArVugUp"
      },
      "source": [
        "texte = nlp(texte1 + texte2)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "Xd-JukqaZXeg"
      },
      "source": [
        "data = []\n",
        "\n",
        "# iterate through each sentence in the file \n",
        "for sent in texte: \n",
        "    temp = []\n",
        "      \n",
        "    # tokenize the sentence into words \n",
        "    for token in texte: \n",
        "        temp.append(token.text.lower()) \n",
        "  \n",
        "    data.append(temp)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "hH25xm_jZXh7"
      },
      "source": [
        "print(data)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "heading_collapsed": true,
        "hidden": true,
        "id": "3QFXtLWm6wch"
      },
      "source": [
        "### Continuous bag of words (CBOW)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "hidden": true,
        "id": "b4tlfQB561cS"
      },
      "source": [
        "Le modèle CBOW prédit le mot courant étant donné les mots de contexte dans une fenêtre autour du mot courant."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "A5jZxGw0ZXlm"
      },
      "source": [
        "# Create CBOW model \n",
        "model1 = gensim.models.Word2Vec(data, min_count = 1,  \n",
        "                              size = 100, window = 5,\n",
        "                              sg = 0)\n",
        "\n",
        "# Print results\n",
        "print(\"Cosine similarity between 'données' \" + \n",
        "               \"and 'connaissance' - CBOW : \", \n",
        "    model1.wv.similarity('données', 'connaissance'))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "heading_collapsed": true,
        "hidden": true,
        "id": "NSBSotcP7gna"
      },
      "source": [
        "### Skip gram"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "hidden": true,
        "id": "OM3PLzyv7hQb"
      },
      "source": [
        "La méthode \"skip gram\" fait le contraire de ce que fait la méthode \"cbow\" : elle prédit les mots de contexte d'un mot donné."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "6A0xo1WJZXpV"
      },
      "source": [
        "# Create Skip Gram model \n",
        "model2 = gensim.models.Word2Vec(data, min_count = 1,\n",
        "                                size = 100, window = 5,\n",
        "                                sg = 1) \n",
        "\n",
        "# Print results \n",
        "print(\"Cosine similarity between 'données' \" +\n",
        "          \"and 'connaissance' - Skip Gram : \", \n",
        "    model2.wv.similarity('données', 'connaissance')) "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "SAEF7KdfZXwJ"
      },
      "source": [
        "word_vectors = model1.wv"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "J740GCysZXtC"
      },
      "source": [
        "print(word_vectors['données'])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "VgLh1oxr-0vv"
      },
      "source": [
        "import gensim.downloader as api\n",
        "\n",
        "# api.info(\"glove-wiki-gigaword-100\")\n",
        "word_vectors = api.load(\"glove-wiki-gigaword-100\")  # load pre-trained word-vectors from gensim-data"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "j_FlOX5e-19a"
      },
      "source": [
        "result = word_vectors.most_similar(positive=['woman', 'king'], negative=['man'])\n",
        "print(\"{}: {:.4f}\".format(*result[0]))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "sszMfjlFjVnp"
      },
      "source": [
        "result = word_vectors.most_similar(positive=['female', 'lion'], negative=['male'])\n",
        "print(\"{}: {:.4f}\".format(*result[0]))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "08wJCNXF-2Xy"
      },
      "source": [
        "print(word_vectors.doesnt_match(\"breakfast cereal dinner lunch\".split()))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "8uWeTuU2-2g0"
      },
      "source": [
        "result = word_vectors.similar_by_word(\"cat\")\n",
        "print(\"{}: {:.4f}\".format(*result[0]))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "heading_collapsed": true,
        "hidden": true,
        "id": "z0DVkHDjsS4z"
      },
      "source": [
        "### FastText"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "SMRAVtne-2ny"
      },
      "source": [
        "from gensim.models import FastText\n",
        "\n",
        "# from gensim.test.utils import common_texts\n",
        "# print(common_texts[0])\n",
        "# ['human', 'interface', 'computer']\n",
        "\n",
        "model = FastText(data, size=100, window=5, min_count=5, workers=4, sg=1)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "oPPHel-8-2ts"
      },
      "source": [
        "model.wv.most_similar(\"données\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "2WRTrD6Sv1nz"
      },
      "source": [
        "!pip install fasttext"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "LHBYQpJJ-2zC"
      },
      "source": [
        "import fasttext.util\n",
        "\n",
        "fasttext.util.download_model('en', if_exists='ignore')\n",
        "ft = fasttext.load_model('cc.en.300.bin')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "PFt0bOct-27l"
      },
      "source": [
        "ft.get_dimension()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "6RRvkNjD-2_n"
      },
      "source": [
        "# fasttext.util.reduce_model(ft, 100)\n",
        "# ft.get_dimension()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "FtvT4wQV-3DT"
      },
      "source": [
        "ft.get_word_vector('hello').shape"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "YFUESWkK-3HX"
      },
      "source": [
        "ft.get_nearest_neighbors('hello')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "heading_collapsed": true,
        "hidden": true,
        "id": "ZgdS6lYaugWD"
      },
      "source": [
        "## Embeddings keras"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vlvSzMHKwreB",
        "outputId": "f4a1510e-cc08-4328-a16e-18fe117621fd",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        }
      },
      "source": [
        "!python -m spacy download fr_core_news_md"
      ],
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\u001b[38;5;2m✔ Download and installation successful\u001b[0m\n",
            "You can now load the model via spacy.load('fr_core_news_md')\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RGRNSy1_wh6r"
      },
      "source": [
        "import spacy\n",
        "import fr_core_news_md\n",
        "\n",
        "# nlp = spacy.load('fr_core_news_md')\n",
        "nlp = fr_core_news_md.load()"
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "y_JRBXhCLX4g"
      },
      "source": [
        "import keras\n",
        "from keras.preprocessing.text import one_hot#,Tokenizer\n",
        "from keras.preprocessing.sequence import pad_sequences\n",
        "from keras.layers import Embedding, Input, Flatten#, Dense\n",
        "from keras.models import Model"
      ],
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "hidden": true,
        "id": "P6SIjdm6ugWH"
      },
      "source": [
        "Création d'un corpus d'exemple"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "jdWm6_gpugWL"
      },
      "source": [
        "sample_text_1=\"Le déménagement à Atrium est prévu pour le 14.\"\n",
        "sample_text_2=\"Mais bon, la dernière fois qu'on a prévu un déménagement...\"\n",
        "sample_text_3=\"Allez un peu d'optimisme, cette fois ci c'est la bonne.\"\n",
        "\n",
        "corp=[sample_text_1,sample_text_2,sample_text_3]\n",
        "no_docs=len(corp)"
      ],
      "execution_count": 27,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "hidden": true,
        "id": "rkSjCyn2ugWO"
      },
      "source": [
        "Encodage du corpus en one-hot à l'aide de la fonction keras."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "uQQWfWYBugWQ"
      },
      "source": [
        "vocab_size=50 \n",
        "encod_corp=[]\n",
        "for i,doc in enumerate(corp):\n",
        "    # taille de vocab 50 pour être sur que chaque mot est encodé sur un entier unique.\n",
        "    encod_corp.append(one_hot(doc,50))\n",
        "    print(\"The encoding for document\",i+1,\" is : \",one_hot(doc,50))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "hidden": true,
        "id": "n4Db_qIbugWV"
      },
      "source": [
        "Padding des documents : la couche d'embedding de keras nécessite des entrées de la même longueur."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "fSru2I-jugWg"
      },
      "source": [
        "# length of maximum document. will be nedded whenever create embeddings for the words\n",
        "maxlen = -1\n",
        "for doc in corp:\n",
        "    tokens = nlp(doc)\n",
        "    if(maxlen < len(tokens)):\n",
        "        maxlen = len(tokens)\n",
        "print(\"The maximum number of words in any document is :\", maxlen)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "ug4-iq48ugWn"
      },
      "source": [
        "# now to create embeddings all of our docs need to be of same length. hence we can pad the docs with zeros.\n",
        "pad_corp=pad_sequences(encod_corp, maxlen=maxlen, padding='post', value=0.0)\n",
        "print(\"No of padded documents: \", len(pad_corp))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "eoISsYrGugWq"
      },
      "source": [
        "for i,doc in enumerate(pad_corp):\n",
        "     print(\"The padded encoding for document\",i+1,\" is : \",doc)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "hidden": true,
        "id": "yiIvcDDYugWy"
      },
      "source": [
        "Création d'une couche d'embedding de dimension 8"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "GdOdRysAugWy"
      },
      "source": [
        "# specifying the input shape\n",
        "input=Input(shape=(no_docs,maxlen),dtype='float64')"
      ],
      "execution_count": 32,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "M4hjleycugW3"
      },
      "source": [
        "'''\n",
        "shape of input. \n",
        "each document has 12 element or words which is the value of our maxlen variable.\n",
        "\n",
        "'''\n",
        "word_input=Input(shape=(maxlen,),dtype='float64')  \n",
        "\n",
        "# creating the embedding\n",
        "word_embedding=Embedding(input_dim=vocab_size,output_dim=8,input_length=maxlen)(word_input)\n",
        "\n",
        "word_vec=Flatten()(word_embedding) # flatten\n",
        "embed_model =Model([word_input],word_vec) # combining all into a Keras model"
      ],
      "execution_count": 33,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "K5PoLHYCugW7"
      },
      "source": [
        "embed_model.compile(optimizer=keras.optimizers.Adam(lr=1e-3),loss='binary_crossentropy',metrics=['acc']) "
      ],
      "execution_count": 34,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "GR1xJNaOugW-"
      },
      "source": [
        "print(type(word_embedding))\n",
        "print(word_embedding)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "2eOGhpZWugXA"
      },
      "source": [
        "print(embed_model.summary()) # summary of the model"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "oozzCv4hugXI"
      },
      "source": [
        "embeddings=embed_model.predict(pad_corp) # finally getting the embeddings."
      ],
      "execution_count": 37,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "hvMcmP1uugXK"
      },
      "source": [
        "print(\"Shape of embeddings : \",embeddings.shape)\n",
        "print(embeddings)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "Pv_NYOc5ugXN"
      },
      "source": [
        "embeddings=embeddings.reshape(-1,maxlen,8)\n",
        "print(\"Shape of embeddings : \",embeddings.shape) \n",
        "print(embeddings)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "HfvmBJJWugXP"
      },
      "source": [
        "for i,doc in enumerate(embeddings):\n",
        "    for j,word in enumerate(doc):\n",
        "        print(\"L'embedding du \",j+1,\"ème mot\",\"dans le\",i+1,\"ème document est : \\n\\n\",word)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "heading_collapsed": true,
        "hidden": true,
        "id": "fa_jPXzmugXU"
      },
      "source": [
        "## Transformers"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "cbByX7YNs8up"
      },
      "source": [
        "import numpy as np\n",
        "import torch\n",
        "from torch.optim import Adam\n",
        "from torch.utils.data import TensorDataset, DataLoader, RandomSampler, SequentialSampler\n",
        "from keras.preprocessing.sequence import pad_sequences\n",
        "from sklearn.model_selection import train_test_split\n",
        "from transformers import CamembertTokenizer, CamembertConfig\n",
        "from transformers import CamembertForTokenClassification\n",
        "from tqdm import tqdm, trange\n",
        "import tensorflow as tf"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "8Jg51kgBs8ur"
      },
      "source": [
        "sentences, labels = [], []\n",
        "with open(\"frwikinews-20130110-pages-articles.txt.tok.stanford-pos\", \"r\", encoding=\"utf-8\") as f:\n",
        "    for line in f.readlines():\n",
        "        sentence = []\n",
        "        sent_tag = []\n",
        "        tokens = line.replace(\"\\n\", \"\").split(\" \")\n",
        "        for token in tokens:\n",
        "            splits = token.split(\"_\")\n",
        "            if len(splits) != 2: continue\n",
        "            word, tag = splits\n",
        "            sentence.append(word)\n",
        "            sent_tag.append(tag)\n",
        "        sentences.append(\" \".join(sentence))\n",
        "        labels.append(sent_tag)\n",
        "    f.close()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "tUu3j8t1awJ8"
      },
      "source": [
        "# import urllib.request\n",
        "\n",
        "# webUrl  = urllib.request.urlopen(\n",
        "#     'https://github.com/QuentinFresnel/FormationNLP/raw/dev/frwikinews-20130110-pages-articles.txt.tok.stanford-pos'\n",
        "#     )\n",
        "\n",
        "# #get the result code and print it\n",
        "# # print (\"result code: \" + str(webUrl.getcode()))\n",
        "\n",
        "# sentences, labels = [], []\n",
        "# for line in webUrl:\n",
        "#     sentence = []\n",
        "#     sent_tag = []\n",
        "#     tokens = line.decode().replace(\"\\n\", \"\").split(\" \")\n",
        "#     for token in tokens:\n",
        "#         splits = token.split(\"_\")\n",
        "#         if len(splits) != 2: continue\n",
        "#         word, tag = splits\n",
        "#         sentence.append(word)\n",
        "#         sent_tag.append(tag)\n",
        "#     sentences.append(\" \".join(sentence))\n",
        "#     labels.append(sent_tag)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "QY1SZebjs8uw"
      },
      "source": [
        "tags_val = list(set().union(*labels))\n",
        "tag2idx = {t:i for i,t in enumerate(tags_val)}\n",
        "# tag2idx[\"<PAD>\"] = len(tag2idx)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "JXb5tjcVs8uy",
        "outputId": "8792dd14-a777-46c1-ceff-9d37f546a39c"
      },
      "source": [
        "lens = np.array(list(map(len, sentences)))\n",
        "lens.min(), lens.max(), lens.mean()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(21, 1733, 156.07599958838796)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "mYAPiLfEs8u4"
      },
      "source": [
        "MAX_LEN = 150\n",
        "batch_size = 64"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "YNWUGfkFs8u6",
        "outputId": "c8ab581b-9d52-4503-a361-8de65a667186"
      },
      "source": [
        "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
        "n_gpu = torch.cuda.device_count()\n",
        "device"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "device(type='cpu')"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 74
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "ZAoXVXfWs8vA"
      },
      "source": [
        "tokenizer = CamembertTokenizer.from_pretrained('camembert-base/', do_lower_case=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "yeWnY3fRs8vC",
        "outputId": "ea8efba5-d671-49a0-e46f-7731ffdc8cfa"
      },
      "source": [
        "tokenized_texts = [tokenizer.tokenize(sent) for sent in sentences]\n",
        "print(tokenized_texts[0])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['▁à', '▁la', '▁suite', '▁de', '▁la', '▁parution', '▁le', '▁matin', '▁même', '▁d', \"'\", '▁un', '▁article', '▁2', '▁=', '▁le', '▁concernant', '▁dans', '▁le', '▁quotidien', '▁libération', '▁', ',', '▁christ', 'oph', 'e', '▁h', 'onde', 'la', 'tte', '▁décide', '▁de', '▁ne', '▁pas', '▁présenter', '▁le', '▁journal', '▁de', '▁13', '▁h', '▁00', '▁de', '▁france', '▁2', '▁', '.']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "R3qpBjijs8vF"
      },
      "source": [
        "input_ids = pad_sequences([tokenizer.convert_tokens_to_ids(txt) for txt in tokenized_texts],\n",
        "                          maxlen=MAX_LEN, dtype=\"long\", truncating=\"post\", padding=\"post\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "RcugdjFbs8vJ"
      },
      "source": [
        "tags = pad_sequences([[tag2idx.get(l) for l in lab] for lab in labels],\n",
        "                     maxlen=MAX_LEN, value=tag2idx[\"PONCT\"], padding=\"post\",\n",
        "                     dtype=\"long\", truncating=\"post\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "JjAFGgL1s8vR"
      },
      "source": [
        "attention_masks = [[float(i>0) for i in ii] for ii in input_ids]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "9NodaF6Ds8vV"
      },
      "source": [
        "tr_inputs, val_inputs, tr_tags, val_tags = train_test_split(input_ids, tags, \n",
        "                                                            random_state=2018, test_size=0.1)\n",
        "tr_masks, val_masks, _, _ = train_test_split(attention_masks, input_ids,\n",
        "                                             random_state=2018, test_size=0.1)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "Jfj1AhHks8vZ"
      },
      "source": [
        "tr_inputs = torch.tensor(tr_inputs)\n",
        "val_inputs = torch.tensor(val_inputs)\n",
        "tr_tags = torch.tensor(tr_tags)\n",
        "val_tags = torch.tensor(val_tags)\n",
        "tr_masks = torch.tensor(tr_masks)\n",
        "val_masks = torch.tensor(val_masks)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "CM5FamoMs8vc"
      },
      "source": [
        "train_data = TensorDataset(tr_inputs, tr_masks, tr_tags)\n",
        "train_sampler = RandomSampler(train_data)\n",
        "train_dataloader = DataLoader(train_data, sampler=train_sampler, batch_size=batch_size)\n",
        "\n",
        "valid_data = TensorDataset(val_inputs, val_masks, val_tags)\n",
        "valid_sampler = SequentialSampler(valid_data)\n",
        "valid_dataloader = DataLoader(valid_data, sampler=valid_sampler, batch_size=batch_size)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "dMJfprRws8vg",
        "outputId": "b1c9d660-7c83-4c32-9278-8f94594547c3"
      },
      "source": [
        "model = CamembertForTokenClassification.from_pretrained(\"camembert-base/\", num_labels=len(tag2idx))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Some weights of the model checkpoint at camembert-base/ were not used when initializing CamembertForTokenClassification: ['lm_head.bias', 'lm_head.dense.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight', 'lm_head.layer_norm.bias', 'lm_head.decoder.weight']\n",
            "- This IS expected if you are initializing CamembertForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).\n",
            "- This IS NOT expected if you are initializing CamembertForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of CamembertForTokenClassification were not initialized from the model checkpoint at camembert-base/ and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
            "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "FEw884sas8vi"
      },
      "source": [
        "FULL_FINETUNING = True #True\n",
        "if FULL_FINETUNING:\n",
        "    param_optimizer = list(model.named_parameters())\n",
        "    no_decay = ['bias', 'gamma', 'beta']\n",
        "    optimizer_grouped_parameters = [\n",
        "        {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)],\n",
        "         'weight_decay_rate': 0.01},\n",
        "        {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)],\n",
        "         'weight_decay_rate': 0.0}\n",
        "    ]\n",
        "else:\n",
        "    param_optimizer = list(model.classifier.named_parameters()) \n",
        "    optimizer_grouped_parameters = [{\"params\": [p for n, p in param_optimizer]}]\n",
        "optimizer = Adam(optimizer_grouped_parameters, lr=3e-5)\n",
        "\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "srDgrdwXs8vn"
      },
      "source": [
        "from seqeval.metrics import f1_score\n",
        "\n",
        "def flat_accuracy(preds, labels):\n",
        "    pred_flat = np.argmax(preds, axis=2).flatten()\n",
        "    labels_flat = labels.flatten()\n",
        "    return np.sum(pred_flat == labels_flat) / len(labels_flat)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "BJCjJDJVs8vq"
      },
      "source": [
        "import time\n",
        "import datetime\n",
        "\n",
        "def format_time(elapsed):\n",
        "    '''\n",
        "    Takes a time in seconds and returns a string hh:mm:ss\n",
        "    '''\n",
        "    # Round to the nearest second.\n",
        "    elapsed_rounded = int(round((elapsed)))\n",
        "    \n",
        "    # Format as hh:mm:ss\n",
        "    return str(datetime.timedelta(seconds=elapsed_rounded))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "zNbKgUPPs8vs",
        "outputId": "5529238d-8db5-4d53-fb65-646ab657b361"
      },
      "source": [
        "epochs = 10\n",
        "max_grad_norm = 1.0\n",
        "total_t0 = time.time()\n",
        "\n",
        "for _ in trange(epochs, desc=\"Epoch\"):\n",
        "    t0 = time.time()\n",
        "    # TRAIN loop\n",
        "    model.train()\n",
        "    tr_loss = 0\n",
        "    nb_tr_examples, nb_tr_steps = 0, 0\n",
        "    for step, batch in enumerate(train_dataloader):\n",
        "        \n",
        "        if step % 10 == 0 and not step == 0:\n",
        "            # Calculate elapsed time in minutes.\n",
        "            elapsed = format_time(time.time() - t0)\n",
        "            \n",
        "            # Report progress.\n",
        "            print('  Batch {:>5,}  of  {:>5,}.    Elapsed: {:}.'.format(step, len(train_dataloader), elapsed))\n",
        "            \n",
        "        # add batch to gpu\n",
        "        batch = tuple(t.to(device) for t in batch)\n",
        "        b_input_ids, b_input_mask, b_labels = batch\n",
        "        # forward pass\n",
        "        loss, something_else = model(b_input_ids, token_type_ids=None,\n",
        "                     attention_mask=b_input_mask, labels=b_labels)\n",
        "        # backward pass\n",
        "        loss.backward()\n",
        "        # track train loss\n",
        "        tr_loss += loss.item()\n",
        "        nb_tr_examples += b_input_ids.size(0)\n",
        "        nb_tr_steps += 1\n",
        "        # gradient clipping\n",
        "        torch.nn.utils.clip_grad_norm_(parameters=model.parameters(), max_norm=max_grad_norm)\n",
        "        # update parameters\n",
        "        optimizer.step()\n",
        "        model.zero_grad()\n",
        "    # print train loss per epoch\n",
        "    print(\"Train loss: {}\".format(tr_loss/nb_tr_steps))\n",
        "    # VALIDATION on validation set\n",
        "    model.eval()\n",
        "    eval_loss, eval_accuracy = 0, 0\n",
        "    nb_eval_steps, nb_eval_examples = 0, 0\n",
        "    predictions , true_labels = [], []\n",
        "    for batch in valid_dataloader:\n",
        "        batch = tuple(t.to(device) for t in batch)\n",
        "        b_input_ids, b_input_mask, b_labels = batch\n",
        "        \n",
        "        with torch.no_grad():\n",
        "            tmp_eval_loss = model(b_input_ids, token_type_ids=None,\n",
        "                                  attention_mask=b_input_mask, labels=b_labels)[0]\n",
        "            logits = model(b_input_ids, token_type_ids=None,\n",
        "                           attention_mask=b_input_mask)[0]\n",
        "        logits = logits.detach().cpu().numpy()\n",
        "        label_ids = b_labels.to('cpu').numpy()\n",
        "        predictions.extend([list(p) for p in np.argmax(logits, axis=2)])\n",
        "        true_labels.append(label_ids)\n",
        "        \n",
        "        tmp_eval_accuracy = flat_accuracy(logits, label_ids)\n",
        "        \n",
        "        eval_loss += tmp_eval_loss.mean().item()\n",
        "        eval_accuracy += tmp_eval_accuracy\n",
        "        \n",
        "        nb_eval_examples += b_input_ids.size(0)\n",
        "        nb_eval_steps += 1\n",
        "    eval_loss = eval_loss/nb_eval_steps\n",
        "    print(\"Validation loss: {}\".format(eval_loss))\n",
        "    print(\"Validation Accuracy: {}\".format(eval_accuracy/nb_eval_steps))\n",
        "    pred_tags = [tags_val[p_i] for p in predictions for p_i in p]\n",
        "    valid_tags = [tags_val[l_ii] for l in true_labels for l_i in l for l_ii in l_i]\n",
        "    print(\"F1-Score: {}\".format(f1_score(pred_tags, valid_tags)))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch:   0%|          | 0/10 [00:00<?, ?it/s]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "  Batch    10  of  1,230.    Elapsed: 0:02:31.\n",
            "  Batch    20  of  1,230.    Elapsed: 0:04:46.\n",
            "  Batch    30  of  1,230.    Elapsed: 0:07:00.\n",
            "  Batch    50  of  1,230.    Elapsed: 0:11:32.\n",
            "  Batch    60  of  1,230.    Elapsed: 0:13:45.\n",
            "  Batch    70  of  1,230.    Elapsed: 0:15:57.\n",
            "  Batch    80  of  1,230.    Elapsed: 0:18:15.\n",
            "  Batch    90  of  1,230.    Elapsed: 0:20:32.\n",
            "  Batch   100  of  1,230.    Elapsed: 0:22:43.\n",
            "  Batch   110  of  1,230.    Elapsed: 0:24:55.\n",
            "  Batch   120  of  1,230.    Elapsed: 0:27:06.\n",
            "  Batch   130  of  1,230.    Elapsed: 0:29:17.\n",
            "  Batch   140  of  1,230.    Elapsed: 0:31:28.\n",
            "  Batch   150  of  1,230.    Elapsed: 0:33:39.\n",
            "  Batch   160  of  1,230.    Elapsed: 0:35:50.\n",
            "  Batch   170  of  1,230.    Elapsed: 0:38:02.\n",
            "  Batch   180  of  1,230.    Elapsed: 0:40:14.\n",
            "  Batch   190  of  1,230.    Elapsed: 0:42:25.\n",
            "  Batch   200  of  1,230.    Elapsed: 0:44:36.\n",
            "  Batch   210  of  1,230.    Elapsed: 0:46:47.\n",
            "  Batch   220  of  1,230.    Elapsed: 0:48:59.\n",
            "  Batch   230  of  1,230.    Elapsed: 0:51:09.\n",
            "  Batch   240  of  1,230.    Elapsed: 0:53:20.\n",
            "  Batch   250  of  1,230.    Elapsed: 0:55:30.\n",
            "  Batch   260  of  1,230.    Elapsed: 0:57:42.\n",
            "  Batch   270  of  1,230.    Elapsed: 0:59:54.\n",
            "  Batch   280  of  1,230.    Elapsed: 1:02:06.\n",
            "  Batch   290  of  1,230.    Elapsed: 1:04:18.\n",
            "  Batch   300  of  1,230.    Elapsed: 1:06:29.\n",
            "  Batch   310  of  1,230.    Elapsed: 1:08:40.\n",
            "  Batch   320  of  1,230.    Elapsed: 1:10:51.\n",
            "  Batch   330  of  1,230.    Elapsed: 1:13:02.\n",
            "  Batch   340  of  1,230.    Elapsed: 1:15:13.\n",
            "  Batch   350  of  1,230.    Elapsed: 1:17:24.\n",
            "  Batch   360  of  1,230.    Elapsed: 1:19:37.\n",
            "  Batch   370  of  1,230.    Elapsed: 1:21:48.\n",
            "  Batch   380  of  1,230.    Elapsed: 1:23:59.\n",
            "  Batch   390  of  1,230.    Elapsed: 1:26:09.\n",
            "  Batch   400  of  1,230.    Elapsed: 1:28:19.\n",
            "  Batch   410  of  1,230.    Elapsed: 1:30:30.\n",
            "  Batch   420  of  1,230.    Elapsed: 1:32:41.\n",
            "  Batch   430  of  1,230.    Elapsed: 1:34:52.\n",
            "  Batch   440  of  1,230.    Elapsed: 1:37:04.\n",
            "  Batch   450  of  1,230.    Elapsed: 1:39:16.\n",
            "  Batch   460  of  1,230.    Elapsed: 1:41:28.\n",
            "  Batch   470  of  1,230.    Elapsed: 1:43:39.\n",
            "  Batch   480  of  1,230.    Elapsed: 1:45:52.\n",
            "  Batch   490  of  1,230.    Elapsed: 1:48:04.\n",
            "  Batch   500  of  1,230.    Elapsed: 1:50:16.\n",
            "  Batch   510  of  1,230.    Elapsed: 1:52:28.\n",
            "  Batch   520  of  1,230.    Elapsed: 1:54:42.\n",
            "  Batch   530  of  1,230.    Elapsed: 1:56:54.\n",
            "  Batch   540  of  1,230.    Elapsed: 1:59:06.\n",
            "  Batch   550  of  1,230.    Elapsed: 2:01:18.\n",
            "  Batch   560  of  1,230.    Elapsed: 2:03:30.\n",
            "  Batch   570  of  1,230.    Elapsed: 2:05:42.\n",
            "  Batch   580  of  1,230.    Elapsed: 2:07:55.\n",
            "  Batch   590  of  1,230.    Elapsed: 2:10:08.\n",
            "  Batch   600  of  1,230.    Elapsed: 2:12:21.\n",
            "  Batch   610  of  1,230.    Elapsed: 2:14:34.\n",
            "  Batch   620  of  1,230.    Elapsed: 2:16:46.\n",
            "  Batch   630  of  1,230.    Elapsed: 2:18:58.\n",
            "  Batch   640  of  1,230.    Elapsed: 2:21:10.\n",
            "  Batch   650  of  1,230.    Elapsed: 2:23:22.\n",
            "  Batch   660  of  1,230.    Elapsed: 2:25:34.\n",
            "  Batch   670  of  1,230.    Elapsed: 2:27:46.\n",
            "  Batch   680  of  1,230.    Elapsed: 2:29:58.\n",
            "  Batch   690  of  1,230.    Elapsed: 2:32:10.\n",
            "  Batch   700  of  1,230.    Elapsed: 2:34:22.\n",
            "  Batch   710  of  1,230.    Elapsed: 2:36:34.\n",
            "  Batch   720  of  1,230.    Elapsed: 2:38:44.\n",
            "  Batch   730  of  1,230.    Elapsed: 2:40:56.\n",
            "  Batch   740  of  1,230.    Elapsed: 2:43:09.\n",
            "  Batch   750  of  1,230.    Elapsed: 2:45:21.\n",
            "  Batch   760  of  1,230.    Elapsed: 2:47:32.\n",
            "  Batch   770  of  1,230.    Elapsed: 2:49:44.\n",
            "  Batch   780  of  1,230.    Elapsed: 2:51:57.\n",
            "  Batch   790  of  1,230.    Elapsed: 2:54:09.\n",
            "  Batch   800  of  1,230.    Elapsed: 2:56:21.\n",
            "  Batch   810  of  1,230.    Elapsed: 2:58:33.\n",
            "  Batch   820  of  1,230.    Elapsed: 3:00:45.\n",
            "  Batch   830  of  1,230.    Elapsed: 3:02:57.\n",
            "  Batch   840  of  1,230.    Elapsed: 3:05:09.\n",
            "  Batch   850  of  1,230.    Elapsed: 3:07:22.\n",
            "  Batch   860  of  1,230.    Elapsed: 3:09:34.\n",
            "  Batch   870  of  1,230.    Elapsed: 3:11:46.\n",
            "  Batch   880  of  1,230.    Elapsed: 3:13:59.\n",
            "  Batch   890  of  1,230.    Elapsed: 3:16:11.\n",
            "  Batch   900  of  1,230.    Elapsed: 3:18:22.\n",
            "  Batch   910  of  1,230.    Elapsed: 3:20:34.\n",
            "  Batch   920  of  1,230.    Elapsed: 3:22:46.\n",
            "  Batch   930  of  1,230.    Elapsed: 3:24:58.\n",
            "  Batch   940  of  1,230.    Elapsed: 3:27:10.\n",
            "  Batch   950  of  1,230.    Elapsed: 3:29:22.\n",
            "  Batch   960  of  1,230.    Elapsed: 3:31:34.\n",
            "  Batch   970  of  1,230.    Elapsed: 3:33:46.\n",
            "  Batch   980  of  1,230.    Elapsed: 3:35:58.\n",
            "  Batch   990  of  1,230.    Elapsed: 3:38:10.\n",
            "  Batch 1,000  of  1,230.    Elapsed: 3:40:23.\n",
            "  Batch 1,010  of  1,230.    Elapsed: 3:42:35.\n",
            "  Batch 1,020  of  1,230.    Elapsed: 3:44:47.\n",
            "  Batch 1,030  of  1,230.    Elapsed: 3:47:00.\n",
            "  Batch 1,040  of  1,230.    Elapsed: 3:49:12.\n",
            "  Batch 1,050  of  1,230.    Elapsed: 3:51:24.\n",
            "  Batch 1,060  of  1,230.    Elapsed: 3:53:36.\n",
            "  Batch 1,070  of  1,230.    Elapsed: 3:55:49.\n",
            "  Batch 1,080  of  1,230.    Elapsed: 3:58:01.\n",
            "  Batch 1,090  of  1,230.    Elapsed: 4:00:13.\n",
            "  Batch 1,100  of  1,230.    Elapsed: 4:02:26.\n",
            "  Batch 1,110  of  1,230.    Elapsed: 4:04:39.\n",
            "  Batch 1,120  of  1,230.    Elapsed: 4:06:52.\n",
            "  Batch 1,130  of  1,230.    Elapsed: 4:09:04.\n",
            "  Batch 1,140  of  1,230.    Elapsed: 4:11:17.\n",
            "  Batch 1,150  of  1,230.    Elapsed: 4:13:29.\n",
            "  Batch 1,160  of  1,230.    Elapsed: 4:15:44.\n",
            "  Batch 1,170  of  1,230.    Elapsed: 4:17:57.\n",
            "  Batch 1,180  of  1,230.    Elapsed: 4:20:09.\n",
            "  Batch 1,190  of  1,230.    Elapsed: 4:22:22.\n",
            "  Batch 1,200  of  1,230.    Elapsed: 4:24:34.\n",
            "  Batch 1,210  of  1,230.    Elapsed: 4:26:47.\n",
            "  Batch 1,220  of  1,230.    Elapsed: 4:28:59.\n",
            "Train loss: 1.5070774912834168\n",
            "Validation loss: 0.7504648927354465\n",
            "Validation Accuracy: 0.20958368698013924\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "Epoch:  10%|█         | 1/10 [4:46:16<42:56:31, 17176.87s/it]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "F1-Score: 0.6678528399311532\n",
            "  Batch    10  of  1,230.    Elapsed: 0:02:13.\n",
            "  Batch    20  of  1,230.    Elapsed: 0:04:26.\n",
            "  Batch    30  of  1,230.    Elapsed: 0:06:39.\n",
            "  Batch    40  of  1,230.    Elapsed: 0:08:51.\n",
            "  Batch    50  of  1,230.    Elapsed: 0:11:03.\n",
            "  Batch    60  of  1,230.    Elapsed: 0:13:16.\n",
            "  Batch    70  of  1,230.    Elapsed: 0:15:26.\n",
            "  Batch    80  of  1,230.    Elapsed: 0:17:39.\n",
            "  Batch    90  of  1,230.    Elapsed: 0:19:50.\n",
            "  Batch   100  of  1,230.    Elapsed: 0:22:02.\n",
            "  Batch   110  of  1,230.    Elapsed: 0:24:14.\n",
            "  Batch   120  of  1,230.    Elapsed: 0:26:26.\n",
            "  Batch   130  of  1,230.    Elapsed: 0:28:38.\n",
            "  Batch   140  of  1,230.    Elapsed: 0:30:50.\n",
            "  Batch   150  of  1,230.    Elapsed: 0:33:01.\n",
            "  Batch   160  of  1,230.    Elapsed: 0:35:13.\n",
            "  Batch   170  of  1,230.    Elapsed: 0:37:25.\n",
            "  Batch   180  of  1,230.    Elapsed: 0:39:37.\n",
            "  Batch   190  of  1,230.    Elapsed: 0:41:50.\n",
            "  Batch   200  of  1,230.    Elapsed: 0:44:02.\n",
            "  Batch   210  of  1,230.    Elapsed: 0:46:15.\n",
            "  Batch   220  of  1,230.    Elapsed: 0:48:26.\n",
            "  Batch   230  of  1,230.    Elapsed: 0:50:38.\n",
            "  Batch   240  of  1,230.    Elapsed: 0:52:50.\n",
            "  Batch   250  of  1,230.    Elapsed: 0:55:02.\n",
            "  Batch   260  of  1,230.    Elapsed: 0:57:15.\n",
            "  Batch   270  of  1,230.    Elapsed: 0:59:27.\n",
            "  Batch   280  of  1,230.    Elapsed: 1:01:40.\n",
            "  Batch   290  of  1,230.    Elapsed: 1:03:51.\n",
            "  Batch   300  of  1,230.    Elapsed: 1:06:02.\n",
            "  Batch   310  of  1,230.    Elapsed: 1:08:14.\n",
            "  Batch   320  of  1,230.    Elapsed: 1:10:26.\n",
            "  Batch   330  of  1,230.    Elapsed: 1:12:38.\n",
            "  Batch   340  of  1,230.    Elapsed: 1:14:49.\n",
            "  Batch   350  of  1,230.    Elapsed: 1:17:02.\n",
            "  Batch   360  of  1,230.    Elapsed: 1:19:14.\n",
            "  Batch   370  of  1,230.    Elapsed: 1:21:26.\n",
            "  Batch   380  of  1,230.    Elapsed: 1:23:38.\n",
            "  Batch   390  of  1,230.    Elapsed: 1:25:50.\n",
            "  Batch   400  of  1,230.    Elapsed: 1:28:02.\n",
            "  Batch   410  of  1,230.    Elapsed: 1:30:14.\n",
            "  Batch   420  of  1,230.    Elapsed: 1:32:26.\n",
            "  Batch   430  of  1,230.    Elapsed: 1:34:38.\n",
            "  Batch   440  of  1,230.    Elapsed: 1:36:50.\n",
            "  Batch   450  of  1,230.    Elapsed: 1:39:02.\n",
            "  Batch   460  of  1,230.    Elapsed: 1:41:13.\n",
            "  Batch   470  of  1,230.    Elapsed: 1:43:25.\n",
            "  Batch   480  of  1,230.    Elapsed: 1:45:36.\n",
            "  Batch   490  of  1,230.    Elapsed: 1:47:52.\n",
            "  Batch   500  of  1,230.    Elapsed: 1:50:03.\n",
            "  Batch   510  of  1,230.    Elapsed: 1:52:14.\n",
            "  Batch   520  of  1,230.    Elapsed: 1:54:25.\n",
            "  Batch   530  of  1,230.    Elapsed: 1:56:37.\n",
            "  Batch   540  of  1,230.    Elapsed: 1:58:49.\n",
            "  Batch   550  of  1,230.    Elapsed: 2:01:00.\n",
            "  Batch   560  of  1,230.    Elapsed: 2:03:12.\n",
            "  Batch   570  of  1,230.    Elapsed: 2:05:25.\n",
            "  Batch   580  of  1,230.    Elapsed: 2:07:36.\n",
            "  Batch   590  of  1,230.    Elapsed: 2:09:49.\n",
            "  Batch   600  of  1,230.    Elapsed: 2:12:00.\n",
            "  Batch   610  of  1,230.    Elapsed: 2:14:13.\n",
            "  Batch   620  of  1,230.    Elapsed: 2:16:23.\n",
            "  Batch   630  of  1,230.    Elapsed: 2:18:35.\n",
            "  Batch   640  of  1,230.    Elapsed: 2:20:48.\n",
            "  Batch   650  of  1,230.    Elapsed: 2:23:00.\n",
            "  Batch   660  of  1,230.    Elapsed: 2:25:12.\n",
            "  Batch   670  of  1,230.    Elapsed: 2:27:23.\n",
            "  Batch   680  of  1,230.    Elapsed: 2:29:36.\n",
            "  Batch   690  of  1,230.    Elapsed: 2:31:49.\n",
            "  Batch   700  of  1,230.    Elapsed: 2:34:01.\n",
            "  Batch   710  of  1,230.    Elapsed: 2:36:11.\n",
            "  Batch   720  of  1,230.    Elapsed: 2:38:23.\n",
            "  Batch   730  of  1,230.    Elapsed: 2:40:35.\n",
            "  Batch   740  of  1,230.    Elapsed: 2:42:46.\n",
            "  Batch   750  of  1,230.    Elapsed: 2:44:56.\n",
            "  Batch   760  of  1,230.    Elapsed: 2:47:07.\n",
            "  Batch   770  of  1,230.    Elapsed: 2:49:19.\n",
            "  Batch   780  of  1,230.    Elapsed: 2:51:30.\n",
            "  Batch   790  of  1,230.    Elapsed: 2:53:41.\n",
            "  Batch   800  of  1,230.    Elapsed: 2:55:53.\n",
            "  Batch   810  of  1,230.    Elapsed: 2:58:04.\n",
            "  Batch   820  of  1,230.    Elapsed: 3:00:16.\n",
            "  Batch   830  of  1,230.    Elapsed: 3:02:28.\n",
            "  Batch   840  of  1,230.    Elapsed: 3:04:39.\n",
            "  Batch   850  of  1,230.    Elapsed: 3:06:51.\n",
            "  Batch   860  of  1,230.    Elapsed: 3:09:04.\n",
            "  Batch   870  of  1,230.    Elapsed: 3:11:15.\n",
            "  Batch   880  of  1,230.    Elapsed: 3:13:25.\n",
            "  Batch   890  of  1,230.    Elapsed: 3:15:37.\n",
            "  Batch   900  of  1,230.    Elapsed: 3:17:48.\n",
            "  Batch   910  of  1,230.    Elapsed: 3:19:59.\n",
            "  Batch   920  of  1,230.    Elapsed: 3:22:11.\n",
            "  Batch   930  of  1,230.    Elapsed: 3:24:23.\n",
            "  Batch   940  of  1,230.    Elapsed: 3:26:34.\n",
            "  Batch   950  of  1,230.    Elapsed: 3:28:44.\n",
            "  Batch   960  of  1,230.    Elapsed: 3:30:56.\n",
            "  Batch   970  of  1,230.    Elapsed: 3:33:08.\n",
            "  Batch   980  of  1,230.    Elapsed: 3:35:19.\n",
            "  Batch   990  of  1,230.    Elapsed: 3:37:31.\n",
            "  Batch 1,000  of  1,230.    Elapsed: 3:39:43.\n",
            "  Batch 1,010  of  1,230.    Elapsed: 3:41:55.\n",
            "  Batch 1,020  of  1,230.    Elapsed: 3:44:06.\n",
            "  Batch 1,030  of  1,230.    Elapsed: 3:46:18.\n",
            "  Batch 1,040  of  1,230.    Elapsed: 3:48:29.\n",
            "  Batch 1,050  of  1,230.    Elapsed: 3:50:40.\n",
            "  Batch 1,060  of  1,230.    Elapsed: 3:52:50.\n",
            "  Batch 1,070  of  1,230.    Elapsed: 3:55:01.\n",
            "  Batch 1,080  of  1,230.    Elapsed: 3:57:13.\n",
            "  Batch 1,090  of  1,230.    Elapsed: 3:59:26.\n",
            "  Batch 1,100  of  1,230.    Elapsed: 4:01:38.\n",
            "  Batch 1,110  of  1,230.    Elapsed: 4:03:51.\n",
            "  Batch 1,120  of  1,230.    Elapsed: 4:06:03.\n",
            "  Batch 1,130  of  1,230.    Elapsed: 4:08:16.\n",
            "  Batch 1,140  of  1,230.    Elapsed: 4:10:30.\n",
            "  Batch 1,150  of  1,230.    Elapsed: 4:12:44.\n",
            "  Batch 1,160  of  1,230.    Elapsed: 4:14:57.\n",
            "  Batch 1,170  of  1,230.    Elapsed: 4:17:10.\n",
            "  Batch 1,180  of  1,230.    Elapsed: 4:19:23.\n",
            "  Batch 1,190  of  1,230.    Elapsed: 4:21:35.\n",
            "  Batch 1,200  of  1,230.    Elapsed: 4:23:49.\n",
            "  Batch 1,210  of  1,230.    Elapsed: 4:26:02.\n",
            "  Batch 1,220  of  1,230.    Elapsed: 4:28:15.\n",
            "Train loss: 0.6372126102689805\n",
            "Validation loss: 0.7680652433068212\n",
            "Validation Accuracy: 0.19712785746619133\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "Epoch:  20%|██        | 2/10 [9:32:01<38:08:58, 17167.32s/it]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "F1-Score: 0.6456441531748572\n",
            "  Batch    10  of  1,230.    Elapsed: 0:02:14.\n",
            "  Batch    20  of  1,230.    Elapsed: 0:04:27.\n",
            "  Batch    30  of  1,230.    Elapsed: 0:06:40.\n",
            "  Batch    40  of  1,230.    Elapsed: 0:08:54.\n",
            "  Batch    50  of  1,230.    Elapsed: 0:11:07.\n",
            "  Batch    60  of  1,230.    Elapsed: 0:13:22.\n",
            "  Batch    70  of  1,230.    Elapsed: 0:15:37.\n",
            "  Batch    80  of  1,230.    Elapsed: 0:17:51.\n",
            "  Batch    90  of  1,230.    Elapsed: 0:20:06.\n",
            "  Batch   100  of  1,230.    Elapsed: 0:22:21.\n",
            "  Batch   110  of  1,230.    Elapsed: 0:24:37.\n",
            "  Batch   120  of  1,230.    Elapsed: 0:26:50.\n",
            "  Batch   130  of  1,230.    Elapsed: 0:29:03.\n",
            "  Batch   140  of  1,230.    Elapsed: 0:31:15.\n",
            "  Batch   150  of  1,230.    Elapsed: 0:33:28.\n",
            "  Batch   160  of  1,230.    Elapsed: 0:35:41.\n",
            "  Batch   170  of  1,230.    Elapsed: 0:37:53.\n",
            "  Batch   180  of  1,230.    Elapsed: 0:40:07.\n",
            "  Batch   190  of  1,230.    Elapsed: 0:42:21.\n",
            "  Batch   200  of  1,230.    Elapsed: 0:44:35.\n",
            "  Batch   210  of  1,230.    Elapsed: 0:46:50.\n",
            "  Batch   220  of  1,230.    Elapsed: 0:49:04.\n",
            "  Batch   230  of  1,230.    Elapsed: 0:51:18.\n",
            "  Batch   240  of  1,230.    Elapsed: 0:53:31.\n",
            "  Batch   250  of  1,230.    Elapsed: 0:55:44.\n",
            "  Batch   260  of  1,230.    Elapsed: 0:57:58.\n",
            "  Batch   270  of  1,230.    Elapsed: 1:00:12.\n",
            "  Batch   280  of  1,230.    Elapsed: 1:02:25.\n",
            "  Batch   290  of  1,230.    Elapsed: 1:04:39.\n",
            "  Batch   300  of  1,230.    Elapsed: 1:06:53.\n",
            "  Batch   310  of  1,230.    Elapsed: 1:09:07.\n",
            "  Batch   320  of  1,230.    Elapsed: 1:11:21.\n",
            "  Batch   330  of  1,230.    Elapsed: 1:13:35.\n",
            "  Batch   340  of  1,230.    Elapsed: 1:15:49.\n",
            "  Batch   350  of  1,230.    Elapsed: 1:18:03.\n",
            "  Batch   360  of  1,230.    Elapsed: 1:20:17.\n",
            "  Batch   370  of  1,230.    Elapsed: 1:22:31.\n",
            "  Batch   380  of  1,230.    Elapsed: 1:24:45.\n",
            "  Batch   390  of  1,230.    Elapsed: 1:26:59.\n",
            "  Batch   400  of  1,230.    Elapsed: 1:29:13.\n",
            "  Batch   410  of  1,230.    Elapsed: 1:31:28.\n",
            "  Batch   420  of  1,230.    Elapsed: 1:33:43.\n",
            "  Batch   430  of  1,230.    Elapsed: 1:35:56.\n",
            "  Batch   440  of  1,230.    Elapsed: 1:38:11.\n",
            "  Batch   450  of  1,230.    Elapsed: 1:40:25.\n",
            "  Batch   460  of  1,230.    Elapsed: 1:42:40.\n",
            "  Batch   470  of  1,230.    Elapsed: 1:44:52.\n",
            "  Batch   480  of  1,230.    Elapsed: 1:47:07.\n",
            "  Batch   490  of  1,230.    Elapsed: 1:49:21.\n",
            "  Batch   500  of  1,230.    Elapsed: 1:51:34.\n",
            "  Batch   510  of  1,230.    Elapsed: 1:53:50.\n",
            "  Batch   520  of  1,230.    Elapsed: 1:56:04.\n",
            "  Batch   530  of  1,230.    Elapsed: 1:58:17.\n",
            "  Batch   540  of  1,230.    Elapsed: 2:00:31.\n",
            "  Batch   550  of  1,230.    Elapsed: 2:02:45.\n",
            "  Batch   560  of  1,230.    Elapsed: 2:04:58.\n",
            "  Batch   570  of  1,230.    Elapsed: 2:07:13.\n",
            "  Batch   580  of  1,230.    Elapsed: 2:09:29.\n",
            "  Batch   590  of  1,230.    Elapsed: 2:11:43.\n",
            "  Batch   600  of  1,230.    Elapsed: 2:13:57.\n",
            "  Batch   610  of  1,230.    Elapsed: 2:16:12.\n",
            "  Batch   620  of  1,230.    Elapsed: 2:18:26.\n",
            "  Batch   630  of  1,230.    Elapsed: 2:20:39.\n",
            "  Batch   640  of  1,230.    Elapsed: 2:22:54.\n",
            "  Batch   650  of  1,230.    Elapsed: 2:25:07.\n",
            "  Batch   660  of  1,230.    Elapsed: 2:27:21.\n",
            "  Batch   670  of  1,230.    Elapsed: 2:29:35.\n",
            "  Batch   680  of  1,230.    Elapsed: 2:31:48.\n",
            "  Batch   690  of  1,230.    Elapsed: 2:34:04.\n",
            "  Batch   700  of  1,230.    Elapsed: 2:36:19.\n",
            "  Batch   710  of  1,230.    Elapsed: 2:38:31.\n",
            "  Batch   720  of  1,230.    Elapsed: 2:40:45.\n",
            "  Batch   730  of  1,230.    Elapsed: 2:42:58.\n",
            "  Batch   740  of  1,230.    Elapsed: 2:45:12.\n",
            "  Batch   750  of  1,230.    Elapsed: 2:47:26.\n",
            "  Batch   760  of  1,230.    Elapsed: 2:49:42.\n",
            "  Batch   770  of  1,230.    Elapsed: 2:51:56.\n",
            "  Batch   780  of  1,230.    Elapsed: 2:54:09.\n",
            "  Batch   790  of  1,230.    Elapsed: 2:56:23.\n",
            "  Batch   800  of  1,230.    Elapsed: 2:58:36.\n",
            "  Batch   810  of  1,230.    Elapsed: 3:00:50.\n",
            "  Batch   820  of  1,230.    Elapsed: 3:03:04.\n",
            "  Batch   830  of  1,230.    Elapsed: 3:05:17.\n",
            "  Batch   840  of  1,230.    Elapsed: 3:07:31.\n",
            "  Batch   850  of  1,230.    Elapsed: 3:09:44.\n",
            "  Batch   860  of  1,230.    Elapsed: 3:11:59.\n",
            "  Batch   870  of  1,230.    Elapsed: 3:14:13.\n",
            "  Batch   880  of  1,230.    Elapsed: 3:16:27.\n",
            "  Batch   890  of  1,230.    Elapsed: 3:18:42.\n",
            "  Batch   900  of  1,230.    Elapsed: 3:20:56.\n",
            "  Batch   910  of  1,230.    Elapsed: 3:23:11.\n",
            "  Batch   920  of  1,230.    Elapsed: 3:25:25.\n",
            "  Batch   930  of  1,230.    Elapsed: 3:27:38.\n",
            "  Batch   940  of  1,230.    Elapsed: 3:29:52.\n",
            "  Batch   950  of  1,230.    Elapsed: 3:32:05.\n",
            "  Batch   960  of  1,230.    Elapsed: 3:34:18.\n",
            "  Batch   970  of  1,230.    Elapsed: 3:36:31.\n",
            "  Batch   980  of  1,230.    Elapsed: 3:38:45.\n",
            "  Batch   990  of  1,230.    Elapsed: 3:41:00.\n",
            "  Batch 1,000  of  1,230.    Elapsed: 3:43:14.\n",
            "  Batch 1,010  of  1,230.    Elapsed: 3:45:27.\n",
            "  Batch 1,020  of  1,230.    Elapsed: 3:47:41.\n",
            "  Batch 1,030  of  1,230.    Elapsed: 3:49:54.\n",
            "  Batch 1,040  of  1,230.    Elapsed: 3:52:09.\n",
            "  Batch 1,050  of  1,230.    Elapsed: 3:54:25.\n",
            "  Batch 1,060  of  1,230.    Elapsed: 3:56:41.\n",
            "  Batch 1,070  of  1,230.    Elapsed: 3:58:57.\n",
            "  Batch 1,080  of  1,230.    Elapsed: 4:01:12.\n",
            "  Batch 1,090  of  1,230.    Elapsed: 4:03:27.\n",
            "  Batch 1,100  of  1,230.    Elapsed: 4:05:41.\n",
            "  Batch 1,110  of  1,230.    Elapsed: 4:07:56.\n",
            "  Batch 1,120  of  1,230.    Elapsed: 4:10:12.\n",
            "  Batch 1,130  of  1,230.    Elapsed: 4:12:25.\n",
            "  Batch 1,140  of  1,230.    Elapsed: 4:14:41.\n",
            "  Batch 1,150  of  1,230.    Elapsed: 4:16:54.\n",
            "  Batch 1,160  of  1,230.    Elapsed: 4:19:10.\n",
            "  Batch 1,170  of  1,230.    Elapsed: 4:21:25.\n",
            "  Batch 1,180  of  1,230.    Elapsed: 4:23:40.\n",
            "  Batch 1,190  of  1,230.    Elapsed: 4:25:53.\n",
            "  Batch 1,200  of  1,230.    Elapsed: 4:28:08.\n",
            "  Batch 1,210  of  1,230.    Elapsed: 4:30:23.\n",
            "  Batch 1,220  of  1,230.    Elapsed: 4:32:37.\n",
            "Train loss: 0.3921777993077185\n",
            "Validation loss: 0.24186624412554025\n",
            "Validation Accuracy: 0.2336084882306343\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "Epoch:  30%|███       | 3/10 [14:22:30<33:32:00, 17245.81s/it]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "F1-Score: 0.8512631076728391\n",
            "  Batch    10  of  1,230.    Elapsed: 0:02:13.\n",
            "  Batch    20  of  1,230.    Elapsed: 0:04:25.\n",
            "  Batch    30  of  1,230.    Elapsed: 0:06:39.\n",
            "  Batch    40  of  1,230.    Elapsed: 0:08:53.\n",
            "  Batch    50  of  1,230.    Elapsed: 0:11:06.\n",
            "  Batch    60  of  1,230.    Elapsed: 0:13:20.\n",
            "  Batch    70  of  1,230.    Elapsed: 0:15:36.\n",
            "  Batch    80  of  1,230.    Elapsed: 0:17:50.\n",
            "  Batch    90  of  1,230.    Elapsed: 0:20:04.\n",
            "  Batch   100  of  1,230.    Elapsed: 0:22:18.\n",
            "  Batch   110  of  1,230.    Elapsed: 0:24:32.\n",
            "  Batch   120  of  1,230.    Elapsed: 0:26:46.\n",
            "  Batch   130  of  1,230.    Elapsed: 0:29:02.\n",
            "  Batch   140  of  1,230.    Elapsed: 0:31:15.\n",
            "  Batch   150  of  1,230.    Elapsed: 0:33:31.\n",
            "  Batch   160  of  1,230.    Elapsed: 0:35:45.\n",
            "  Batch   170  of  1,230.    Elapsed: 0:37:58.\n",
            "  Batch   180  of  1,230.    Elapsed: 0:40:12.\n",
            "  Batch   190  of  1,230.    Elapsed: 0:42:26.\n",
            "  Batch   200  of  1,230.    Elapsed: 0:44:39.\n",
            "  Batch   210  of  1,230.    Elapsed: 0:46:53.\n",
            "  Batch   220  of  1,230.    Elapsed: 0:49:07.\n",
            "  Batch   230  of  1,230.    Elapsed: 0:51:22.\n",
            "  Batch   240  of  1,230.    Elapsed: 0:53:36.\n",
            "  Batch   250  of  1,230.    Elapsed: 0:55:50.\n",
            "  Batch   260  of  1,230.    Elapsed: 0:58:06.\n",
            "  Batch   270  of  1,230.    Elapsed: 1:00:20.\n",
            "  Batch   280  of  1,230.    Elapsed: 1:02:34.\n",
            "  Batch   290  of  1,230.    Elapsed: 1:04:48.\n",
            "  Batch   300  of  1,230.    Elapsed: 1:07:02.\n",
            "  Batch   310  of  1,230.    Elapsed: 1:09:16.\n",
            "  Batch   320  of  1,230.    Elapsed: 1:11:30.\n",
            "  Batch   330  of  1,230.    Elapsed: 1:13:45.\n",
            "  Batch   340  of  1,230.    Elapsed: 1:15:57.\n",
            "  Batch   350  of  1,230.    Elapsed: 1:18:12.\n",
            "  Batch   360  of  1,230.    Elapsed: 1:20:27.\n",
            "  Batch   370  of  1,230.    Elapsed: 1:22:43.\n",
            "  Batch   380  of  1,230.    Elapsed: 1:24:58.\n",
            "  Batch   390  of  1,230.    Elapsed: 1:27:13.\n",
            "  Batch   400  of  1,230.    Elapsed: 1:29:29.\n",
            "  Batch   410  of  1,230.    Elapsed: 1:31:44.\n",
            "  Batch   420  of  1,230.    Elapsed: 1:33:59.\n",
            "  Batch   430  of  1,230.    Elapsed: 1:36:12.\n",
            "  Batch   440  of  1,230.    Elapsed: 1:38:28.\n",
            "  Batch   450  of  1,230.    Elapsed: 1:40:43.\n",
            "  Batch   460  of  1,230.    Elapsed: 1:42:57.\n",
            "  Batch   470  of  1,230.    Elapsed: 1:45:12.\n",
            "  Batch   480  of  1,230.    Elapsed: 1:47:26.\n",
            "  Batch   490  of  1,230.    Elapsed: 1:49:40.\n",
            "  Batch   500  of  1,230.    Elapsed: 1:51:54.\n",
            "  Batch   510  of  1,230.    Elapsed: 1:54:08.\n",
            "  Batch   520  of  1,230.    Elapsed: 1:56:22.\n",
            "  Batch   530  of  1,230.    Elapsed: 1:58:37.\n",
            "  Batch   540  of  1,230.    Elapsed: 2:00:51.\n",
            "  Batch   550  of  1,230.    Elapsed: 2:03:06.\n",
            "  Batch   560  of  1,230.    Elapsed: 2:05:20.\n",
            "  Batch   570  of  1,230.    Elapsed: 2:07:35.\n",
            "  Batch   580  of  1,230.    Elapsed: 2:09:50.\n",
            "  Batch   590  of  1,230.    Elapsed: 2:12:04.\n",
            "  Batch   600  of  1,230.    Elapsed: 2:14:19.\n",
            "  Batch   610  of  1,230.    Elapsed: 2:16:32.\n",
            "  Batch   620  of  1,230.    Elapsed: 2:18:46.\n",
            "  Batch   630  of  1,230.    Elapsed: 2:20:59.\n",
            "  Batch   640  of  1,230.    Elapsed: 2:23:13.\n",
            "  Batch   650  of  1,230.    Elapsed: 2:25:27.\n",
            "  Batch   660  of  1,230.    Elapsed: 2:27:42.\n",
            "  Batch   670  of  1,230.    Elapsed: 2:29:57.\n",
            "  Batch   680  of  1,230.    Elapsed: 2:32:12.\n",
            "  Batch   690  of  1,230.    Elapsed: 2:34:26.\n",
            "  Batch   700  of  1,230.    Elapsed: 2:36:41.\n",
            "  Batch   710  of  1,230.    Elapsed: 2:38:55.\n",
            "  Batch   720  of  1,230.    Elapsed: 2:41:10.\n",
            "  Batch   730  of  1,230.    Elapsed: 2:43:23.\n",
            "  Batch   740  of  1,230.    Elapsed: 2:45:38.\n",
            "  Batch   750  of  1,230.    Elapsed: 2:47:53.\n",
            "  Batch   760  of  1,230.    Elapsed: 2:50:07.\n",
            "  Batch   770  of  1,230.    Elapsed: 2:52:21.\n",
            "  Batch   780  of  1,230.    Elapsed: 2:54:36.\n",
            "  Batch   790  of  1,230.    Elapsed: 2:56:50.\n",
            "  Batch   800  of  1,230.    Elapsed: 2:59:04.\n",
            "  Batch   810  of  1,230.    Elapsed: 3:01:18.\n",
            "  Batch   820  of  1,230.    Elapsed: 3:03:31.\n",
            "  Batch   830  of  1,230.    Elapsed: 3:05:44.\n",
            "  Batch   840  of  1,230.    Elapsed: 3:07:58.\n",
            "  Batch   850  of  1,230.    Elapsed: 3:10:11.\n",
            "  Batch   860  of  1,230.    Elapsed: 3:12:25.\n",
            "  Batch   870  of  1,230.    Elapsed: 3:14:38.\n",
            "  Batch   880  of  1,230.    Elapsed: 3:16:52.\n",
            "  Batch   890  of  1,230.    Elapsed: 3:19:07.\n",
            "  Batch   900  of  1,230.    Elapsed: 3:21:20.\n",
            "  Batch   910  of  1,230.    Elapsed: 3:23:35.\n",
            "  Batch   920  of  1,230.    Elapsed: 3:25:49.\n",
            "  Batch   930  of  1,230.    Elapsed: 3:28:04.\n",
            "  Batch   940  of  1,230.    Elapsed: 3:30:18.\n",
            "  Batch   950  of  1,230.    Elapsed: 3:32:32.\n",
            "  Batch   960  of  1,230.    Elapsed: 3:34:45.\n",
            "  Batch   970  of  1,230.    Elapsed: 3:36:59.\n",
            "  Batch   980  of  1,230.    Elapsed: 3:39:13.\n",
            "  Batch   990  of  1,230.    Elapsed: 3:41:28.\n",
            "  Batch 1,000  of  1,230.    Elapsed: 3:43:43.\n",
            "  Batch 1,010  of  1,230.    Elapsed: 3:45:57.\n",
            "  Batch 1,020  of  1,230.    Elapsed: 3:48:10.\n",
            "  Batch 1,030  of  1,230.    Elapsed: 3:50:25.\n",
            "  Batch 1,040  of  1,230.    Elapsed: 3:52:39.\n",
            "  Batch 1,050  of  1,230.    Elapsed: 3:54:53.\n",
            "  Batch 1,060  of  1,230.    Elapsed: 3:57:07.\n",
            "  Batch 1,070  of  1,230.    Elapsed: 3:59:21.\n",
            "  Batch 1,080  of  1,230.    Elapsed: 4:01:35.\n",
            "  Batch 1,090  of  1,230.    Elapsed: 4:03:49.\n",
            "  Batch 1,100  of  1,230.    Elapsed: 4:06:03.\n",
            "  Batch 1,110  of  1,230.    Elapsed: 4:08:16.\n",
            "  Batch 1,120  of  1,230.    Elapsed: 4:10:30.\n",
            "  Batch 1,130  of  1,230.    Elapsed: 4:12:45.\n",
            "  Batch 1,140  of  1,230.    Elapsed: 4:14:59.\n",
            "  Batch 1,150  of  1,230.    Elapsed: 4:17:13.\n",
            "  Batch 1,160  of  1,230.    Elapsed: 4:19:27.\n",
            "  Batch 1,170  of  1,230.    Elapsed: 4:21:41.\n",
            "  Batch 1,180  of  1,230.    Elapsed: 4:23:56.\n",
            "  Batch 1,190  of  1,230.    Elapsed: 4:26:09.\n",
            "  Batch 1,200  of  1,230.    Elapsed: 4:28:23.\n",
            "  Batch 1,210  of  1,230.    Elapsed: 4:30:38.\n",
            "  Batch 1,220  of  1,230.    Elapsed: 4:32:52.\n",
            "Train loss: 0.2821777369191007\n",
            "Validation loss: 0.21323060010471484\n",
            "Validation Accuracy: 0.23461974127199678\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "Epoch:  40%|████      | 4/10 [19:13:06<28:50:16, 17302.71s/it]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "F1-Score: 0.8555478903830289\n",
            "  Batch    10  of  1,230.    Elapsed: 0:02:14.\n",
            "  Batch    20  of  1,230.    Elapsed: 0:04:29.\n",
            "  Batch    30  of  1,230.    Elapsed: 0:06:44.\n",
            "  Batch    40  of  1,230.    Elapsed: 0:08:58.\n",
            "  Batch    50  of  1,230.    Elapsed: 0:11:13.\n",
            "  Batch    60  of  1,230.    Elapsed: 0:13:25.\n",
            "  Batch    70  of  1,230.    Elapsed: 0:15:39.\n",
            "  Batch    80  of  1,230.    Elapsed: 0:17:55.\n",
            "  Batch    90  of  1,230.    Elapsed: 0:20:09.\n",
            "  Batch   100  of  1,230.    Elapsed: 0:22:24.\n",
            "  Batch   110  of  1,230.    Elapsed: 0:24:39.\n",
            "  Batch   120  of  1,230.    Elapsed: 0:26:55.\n",
            "  Batch   130  of  1,230.    Elapsed: 0:29:09.\n",
            "  Batch   140  of  1,230.    Elapsed: 0:31:23.\n",
            "  Batch   150  of  1,230.    Elapsed: 0:33:36.\n",
            "  Batch   160  of  1,230.    Elapsed: 0:35:50.\n",
            "  Batch   170  of  1,230.    Elapsed: 0:38:05.\n",
            "  Batch   180  of  1,230.    Elapsed: 0:40:20.\n",
            "  Batch   190  of  1,230.    Elapsed: 0:42:34.\n",
            "  Batch   200  of  1,230.    Elapsed: 0:44:48.\n",
            "  Batch   210  of  1,230.    Elapsed: 0:47:02.\n",
            "  Batch   220  of  1,230.    Elapsed: 0:49:17.\n",
            "  Batch   230  of  1,230.    Elapsed: 0:51:32.\n",
            "  Batch   240  of  1,230.    Elapsed: 0:53:45.\n",
            "  Batch   250  of  1,230.    Elapsed: 0:55:59.\n",
            "  Batch   260  of  1,230.    Elapsed: 0:58:14.\n",
            "  Batch   270  of  1,230.    Elapsed: 1:00:29.\n",
            "  Batch   280  of  1,230.    Elapsed: 1:02:42.\n",
            "  Batch   290  of  1,230.    Elapsed: 1:04:56.\n",
            "  Batch   300  of  1,230.    Elapsed: 1:07:11.\n",
            "  Batch   310  of  1,230.    Elapsed: 1:09:26.\n",
            "  Batch   320  of  1,230.    Elapsed: 1:11:41.\n",
            "  Batch   330  of  1,230.    Elapsed: 1:13:54.\n",
            "  Batch   340  of  1,230.    Elapsed: 1:16:08.\n",
            "  Batch   350  of  1,230.    Elapsed: 1:18:23.\n",
            "  Batch   360  of  1,230.    Elapsed: 1:20:37.\n",
            "  Batch   370  of  1,230.    Elapsed: 1:22:53.\n",
            "  Batch   380  of  1,230.    Elapsed: 1:25:08.\n",
            "  Batch   390  of  1,230.    Elapsed: 1:27:22.\n",
            "  Batch   400  of  1,230.    Elapsed: 1:29:37.\n",
            "  Batch   410  of  1,230.    Elapsed: 1:31:52.\n",
            "  Batch   420  of  1,230.    Elapsed: 1:34:06.\n",
            "  Batch   430  of  1,230.    Elapsed: 1:36:20.\n",
            "  Batch   440  of  1,230.    Elapsed: 1:38:35.\n",
            "  Batch   450  of  1,230.    Elapsed: 1:40:49.\n",
            "  Batch   460  of  1,230.    Elapsed: 1:43:04.\n",
            "  Batch   470  of  1,230.    Elapsed: 1:45:19.\n",
            "  Batch   480  of  1,230.    Elapsed: 1:47:32.\n",
            "  Batch   490  of  1,230.    Elapsed: 1:49:46.\n",
            "  Batch   500  of  1,230.    Elapsed: 1:51:59.\n",
            "  Batch   510  of  1,230.    Elapsed: 1:54:14.\n",
            "  Batch   520  of  1,230.    Elapsed: 1:56:28.\n",
            "  Batch   530  of  1,230.    Elapsed: 1:58:43.\n",
            "  Batch   540  of  1,230.    Elapsed: 2:00:58.\n",
            "  Batch   550  of  1,230.    Elapsed: 2:03:12.\n",
            "  Batch   560  of  1,230.    Elapsed: 2:05:27.\n",
            "  Batch   570  of  1,230.    Elapsed: 2:07:41.\n",
            "  Batch   580  of  1,230.    Elapsed: 2:09:55.\n",
            "  Batch   590  of  1,230.    Elapsed: 2:12:10.\n",
            "  Batch   600  of  1,230.    Elapsed: 2:14:24.\n",
            "  Batch   610  of  1,230.    Elapsed: 2:16:38.\n",
            "  Batch   620  of  1,230.    Elapsed: 2:18:52.\n",
            "  Batch   630  of  1,230.    Elapsed: 2:21:07.\n",
            "  Batch   640  of  1,230.    Elapsed: 2:23:18.\n",
            "  Batch   650  of  1,230.    Elapsed: 2:25:32.\n",
            "  Batch   660  of  1,230.    Elapsed: 2:27:45.\n",
            "  Batch   670  of  1,230.    Elapsed: 2:29:58.\n",
            "  Batch   680  of  1,230.    Elapsed: 2:32:13.\n",
            "  Batch   690  of  1,230.    Elapsed: 2:34:27.\n",
            "  Batch   700  of  1,230.    Elapsed: 2:36:41.\n",
            "  Batch   710  of  1,230.    Elapsed: 2:38:55.\n",
            "  Batch   720  of  1,230.    Elapsed: 2:41:10.\n",
            "  Batch   730  of  1,230.    Elapsed: 2:43:24.\n",
            "  Batch   740  of  1,230.    Elapsed: 2:45:38.\n",
            "  Batch   750  of  1,230.    Elapsed: 2:47:52.\n",
            "  Batch   760  of  1,230.    Elapsed: 2:50:06.\n",
            "  Batch   770  of  1,230.    Elapsed: 2:52:20.\n",
            "  Batch   780  of  1,230.    Elapsed: 2:54:35.\n",
            "  Batch   790  of  1,230.    Elapsed: 2:56:48.\n",
            "  Batch   800  of  1,230.    Elapsed: 2:59:02.\n",
            "  Batch   810  of  1,230.    Elapsed: 3:01:16.\n",
            "  Batch   820  of  1,230.    Elapsed: 3:03:31.\n",
            "  Batch   830  of  1,230.    Elapsed: 3:05:45.\n",
            "  Batch   840  of  1,230.    Elapsed: 3:07:58.\n",
            "  Batch   850  of  1,230.    Elapsed: 3:10:12.\n",
            "  Batch   860  of  1,230.    Elapsed: 3:12:26.\n",
            "  Batch   870  of  1,230.    Elapsed: 3:14:39.\n",
            "  Batch   880  of  1,230.    Elapsed: 3:16:53.\n",
            "  Batch   890  of  1,230.    Elapsed: 3:19:09.\n",
            "  Batch   900  of  1,230.    Elapsed: 3:21:23.\n",
            "  Batch   910  of  1,230.    Elapsed: 3:23:38.\n",
            "  Batch   920  of  1,230.    Elapsed: 3:25:51.\n",
            "  Batch   930  of  1,230.    Elapsed: 3:28:06.\n",
            "  Batch   940  of  1,230.    Elapsed: 3:30:21.\n",
            "  Batch   950  of  1,230.    Elapsed: 3:32:36.\n",
            "  Batch   960  of  1,230.    Elapsed: 3:34:51.\n",
            "  Batch   970  of  1,230.    Elapsed: 3:37:07.\n",
            "  Batch   980  of  1,230.    Elapsed: 3:39:21.\n",
            "  Batch   990  of  1,230.    Elapsed: 3:41:34.\n",
            "  Batch 1,000  of  1,230.    Elapsed: 3:43:49.\n",
            "  Batch 1,010  of  1,230.    Elapsed: 3:46:03.\n",
            "  Batch 1,020  of  1,230.    Elapsed: 3:48:18.\n",
            "  Batch 1,030  of  1,230.    Elapsed: 3:50:34.\n",
            "  Batch 1,040  of  1,230.    Elapsed: 3:52:49.\n",
            "  Batch 1,050  of  1,230.    Elapsed: 3:55:05.\n",
            "  Batch 1,060  of  1,230.    Elapsed: 3:57:20.\n",
            "  Batch 1,070  of  1,230.    Elapsed: 3:59:35.\n",
            "  Batch 1,080  of  1,230.    Elapsed: 4:01:50.\n",
            "  Batch 1,090  of  1,230.    Elapsed: 4:04:05.\n",
            "  Batch 1,100  of  1,230.    Elapsed: 4:06:21.\n",
            "  Batch 1,110  of  1,230.    Elapsed: 4:08:37.\n",
            "  Batch 1,120  of  1,230.    Elapsed: 4:10:51.\n",
            "  Batch 1,130  of  1,230.    Elapsed: 4:13:04.\n",
            "  Batch 1,140  of  1,230.    Elapsed: 4:15:18.\n",
            "  Batch 1,150  of  1,230.    Elapsed: 4:17:32.\n",
            "  Batch 1,160  of  1,230.    Elapsed: 4:19:46.\n",
            "  Batch 1,170  of  1,230.    Elapsed: 4:22:01.\n",
            "  Batch 1,180  of  1,230.    Elapsed: 4:24:16.\n",
            "  Batch 1,190  of  1,230.    Elapsed: 4:26:32.\n",
            "  Batch 1,200  of  1,230.    Elapsed: 4:28:46.\n",
            "  Batch 1,210  of  1,230.    Elapsed: 4:31:00.\n",
            "  Batch 1,220  of  1,230.    Elapsed: 4:33:14.\n",
            "Train loss: 0.21357444705880754\n",
            "Validation loss: 0.19522546144732594\n",
            "Validation Accuracy: 0.23548037613874262\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "Epoch:  50%|█████     | 5/10 [24:04:10<24:05:56, 17351.27s/it]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "F1-Score: 0.8583599313546852\n",
            "  Batch    10  of  1,230.    Elapsed: 0:02:15.\n",
            "  Batch    20  of  1,230.    Elapsed: 0:04:28.\n",
            "  Batch    30  of  1,230.    Elapsed: 0:06:41.\n",
            "  Batch    40  of  1,230.    Elapsed: 0:08:54.\n",
            "  Batch    50  of  1,230.    Elapsed: 0:11:08.\n",
            "  Batch    60  of  1,230.    Elapsed: 0:13:21.\n",
            "  Batch    70  of  1,230.    Elapsed: 0:15:35.\n",
            "  Batch    80  of  1,230.    Elapsed: 0:17:51.\n",
            "  Batch    90  of  1,230.    Elapsed: 0:20:10.\n",
            "  Batch   100  of  1,230.    Elapsed: 0:22:26.\n",
            "  Batch   110  of  1,230.    Elapsed: 0:24:42.\n",
            "  Batch   120  of  1,230.    Elapsed: 0:26:58.\n",
            "  Batch   130  of  1,230.    Elapsed: 0:29:13.\n",
            "  Batch   140  of  1,230.    Elapsed: 0:31:28.\n",
            "  Batch   150  of  1,230.    Elapsed: 0:33:45.\n",
            "  Batch   160  of  1,230.    Elapsed: 0:36:00.\n",
            "  Batch   170  of  1,230.    Elapsed: 0:38:16.\n",
            "  Batch   180  of  1,230.    Elapsed: 0:40:31.\n",
            "  Batch   190  of  1,230.    Elapsed: 0:42:46.\n",
            "  Batch   200  of  1,230.    Elapsed: 0:45:02.\n",
            "  Batch   210  of  1,230.    Elapsed: 0:47:18.\n",
            "  Batch   220  of  1,230.    Elapsed: 0:49:34.\n",
            "  Batch   230  of  1,230.    Elapsed: 0:51:49.\n",
            "  Batch   240  of  1,230.    Elapsed: 0:54:04.\n",
            "  Batch   250  of  1,230.    Elapsed: 0:56:18.\n",
            "  Batch   260  of  1,230.    Elapsed: 0:58:32.\n",
            "  Batch   270  of  1,230.    Elapsed: 1:00:47.\n",
            "  Batch   280  of  1,230.    Elapsed: 1:03:03.\n",
            "  Batch   290  of  1,230.    Elapsed: 1:05:18.\n",
            "  Batch   300  of  1,230.    Elapsed: 1:07:34.\n",
            "  Batch   310  of  1,230.    Elapsed: 1:09:48.\n",
            "  Batch   320  of  1,230.    Elapsed: 1:12:01.\n",
            "  Batch   330  of  1,230.    Elapsed: 1:14:15.\n",
            "  Batch   340  of  1,230.    Elapsed: 1:16:30.\n",
            "  Batch   350  of  1,230.    Elapsed: 1:18:45.\n",
            "  Batch   360  of  1,230.    Elapsed: 1:20:59.\n",
            "  Batch   370  of  1,230.    Elapsed: 1:23:14.\n",
            "  Batch   380  of  1,230.    Elapsed: 1:25:29.\n",
            "  Batch   390  of  1,230.    Elapsed: 1:27:45.\n",
            "  Batch   400  of  1,230.    Elapsed: 1:29:59.\n",
            "  Batch   410  of  1,230.    Elapsed: 1:32:13.\n",
            "  Batch   420  of  1,230.    Elapsed: 1:34:27.\n",
            "  Batch   430  of  1,230.    Elapsed: 1:36:40.\n",
            "  Batch   440  of  1,230.    Elapsed: 1:38:53.\n",
            "  Batch   450  of  1,230.    Elapsed: 1:41:08.\n",
            "  Batch   460  of  1,230.    Elapsed: 1:43:23.\n",
            "  Batch   470  of  1,230.    Elapsed: 1:45:36.\n",
            "  Batch   480  of  1,230.    Elapsed: 1:47:51.\n",
            "  Batch   490  of  1,230.    Elapsed: 1:50:05.\n",
            "  Batch   500  of  1,230.    Elapsed: 1:52:19.\n",
            "  Batch   510  of  1,230.    Elapsed: 1:54:34.\n",
            "  Batch   520  of  1,230.    Elapsed: 1:56:48.\n",
            "  Batch   530  of  1,230.    Elapsed: 1:59:02.\n",
            "  Batch   540  of  1,230.    Elapsed: 2:01:16.\n",
            "  Batch   550  of  1,230.    Elapsed: 2:03:30.\n",
            "  Batch   560  of  1,230.    Elapsed: 2:05:45.\n",
            "  Batch   570  of  1,230.    Elapsed: 2:07:59.\n",
            "  Batch   580  of  1,230.    Elapsed: 2:10:15.\n",
            "  Batch   590  of  1,230.    Elapsed: 2:12:29.\n",
            "  Batch   600  of  1,230.    Elapsed: 2:14:43.\n",
            "  Batch   610  of  1,230.    Elapsed: 2:16:57.\n",
            "  Batch   620  of  1,230.    Elapsed: 2:19:11.\n",
            "  Batch   630  of  1,230.    Elapsed: 2:21:25.\n",
            "  Batch   640  of  1,230.    Elapsed: 2:23:39.\n",
            "  Batch   650  of  1,230.    Elapsed: 2:25:54.\n",
            "  Batch   660  of  1,230.    Elapsed: 2:28:10.\n",
            "  Batch   670  of  1,230.    Elapsed: 2:30:25.\n",
            "  Batch   680  of  1,230.    Elapsed: 2:32:41.\n",
            "  Batch   690  of  1,230.    Elapsed: 2:34:56.\n",
            "  Batch   700  of  1,230.    Elapsed: 2:37:09.\n",
            "  Batch   710  of  1,230.    Elapsed: 2:39:24.\n",
            "  Batch   720  of  1,230.    Elapsed: 2:41:39.\n",
            "  Batch   730  of  1,230.    Elapsed: 2:43:52.\n",
            "  Batch   740  of  1,230.    Elapsed: 2:46:06.\n",
            "  Batch   750  of  1,230.    Elapsed: 2:48:21.\n",
            "  Batch   760  of  1,230.    Elapsed: 2:50:35.\n",
            "  Batch   770  of  1,230.    Elapsed: 2:52:50.\n",
            "  Batch   780  of  1,230.    Elapsed: 2:55:04.\n",
            "  Batch   790  of  1,230.    Elapsed: 2:57:17.\n",
            "  Batch   800  of  1,230.    Elapsed: 2:59:31.\n",
            "  Batch   810  of  1,230.    Elapsed: 3:01:44.\n",
            "  Batch   820  of  1,230.    Elapsed: 3:03:59.\n",
            "  Batch   830  of  1,230.    Elapsed: 3:06:14.\n",
            "  Batch   840  of  1,230.    Elapsed: 3:08:28.\n",
            "  Batch   850  of  1,230.    Elapsed: 3:10:43.\n",
            "  Batch   860  of  1,230.    Elapsed: 3:12:57.\n",
            "  Batch   870  of  1,230.    Elapsed: 3:15:11.\n",
            "  Batch   880  of  1,230.    Elapsed: 3:17:26.\n",
            "  Batch   890  of  1,230.    Elapsed: 3:19:42.\n",
            "  Batch   900  of  1,230.    Elapsed: 3:21:57.\n",
            "  Batch   910  of  1,230.    Elapsed: 3:24:11.\n",
            "  Batch   920  of  1,230.    Elapsed: 3:26:26.\n",
            "  Batch   930  of  1,230.    Elapsed: 3:28:40.\n",
            "  Batch   940  of  1,230.    Elapsed: 3:30:53.\n",
            "  Batch   950  of  1,230.    Elapsed: 3:33:07.\n",
            "  Batch   960  of  1,230.    Elapsed: 3:35:22.\n",
            "  Batch   970  of  1,230.    Elapsed: 3:37:35.\n",
            "  Batch   980  of  1,230.    Elapsed: 3:39:49.\n",
            "  Batch   990  of  1,230.    Elapsed: 3:42:04.\n",
            "  Batch 1,000  of  1,230.    Elapsed: 3:44:18.\n",
            "  Batch 1,010  of  1,230.    Elapsed: 3:46:32.\n",
            "  Batch 1,020  of  1,230.    Elapsed: 3:48:46.\n",
            "  Batch 1,030  of  1,230.    Elapsed: 3:51:01.\n",
            "  Batch 1,040  of  1,230.    Elapsed: 3:53:15.\n",
            "  Batch 1,050  of  1,230.    Elapsed: 3:55:29.\n",
            "  Batch 1,060  of  1,230.    Elapsed: 3:57:43.\n",
            "  Batch 1,070  of  1,230.    Elapsed: 3:59:57.\n",
            "  Batch 1,080  of  1,230.    Elapsed: 4:02:12.\n",
            "  Batch 1,090  of  1,230.    Elapsed: 4:04:27.\n",
            "  Batch 1,100  of  1,230.    Elapsed: 4:06:41.\n",
            "  Batch 1,110  of  1,230.    Elapsed: 4:08:55.\n",
            "  Batch 1,120  of  1,230.    Elapsed: 4:11:09.\n",
            "  Batch 1,130  of  1,230.    Elapsed: 4:13:23.\n",
            "  Batch 1,140  of  1,230.    Elapsed: 4:15:37.\n",
            "  Batch 1,150  of  1,230.    Elapsed: 4:17:51.\n",
            "  Batch 1,160  of  1,230.    Elapsed: 4:20:04.\n",
            "  Batch 1,170  of  1,230.    Elapsed: 4:22:17.\n",
            "  Batch 1,180  of  1,230.    Elapsed: 4:24:31.\n",
            "  Batch 1,190  of  1,230.    Elapsed: 4:26:45.\n",
            "  Batch 1,200  of  1,230.    Elapsed: 4:28:59.\n",
            "  Batch 1,210  of  1,230.    Elapsed: 4:31:13.\n",
            "  Batch 1,220  of  1,230.    Elapsed: 4:33:27.\n",
            "Train loss: 0.1818620647053893\n",
            "Validation loss: 0.1211707437016668\n",
            "Validation Accuracy: 0.24341099417190065\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "Epoch:  60%|██████    | 6/10 [28:55:13<19:18:59, 17384.76s/it]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "F1-Score: 0.8905476457013995\n",
            "  Batch    10  of  1,230.    Elapsed: 0:02:14.\n",
            "  Batch    20  of  1,230.    Elapsed: 0:04:28.\n",
            "  Batch    30  of  1,230.    Elapsed: 0:06:42.\n",
            "  Batch    40  of  1,230.    Elapsed: 0:08:58.\n",
            "  Batch    50  of  1,230.    Elapsed: 0:11:13.\n",
            "  Batch    60  of  1,230.    Elapsed: 0:13:28.\n",
            "  Batch    70  of  1,230.    Elapsed: 0:15:43.\n",
            "  Batch    80  of  1,230.    Elapsed: 0:17:57.\n",
            "  Batch    90  of  1,230.    Elapsed: 0:20:11.\n",
            "  Batch   100  of  1,230.    Elapsed: 0:22:25.\n",
            "  Batch   110  of  1,230.    Elapsed: 0:24:38.\n",
            "  Batch   120  of  1,230.    Elapsed: 0:26:54.\n",
            "  Batch   130  of  1,230.    Elapsed: 0:29:09.\n",
            "  Batch   140  of  1,230.    Elapsed: 0:31:23.\n",
            "  Batch   150  of  1,230.    Elapsed: 0:33:36.\n",
            "  Batch   160  of  1,230.    Elapsed: 0:35:50.\n",
            "  Batch   170  of  1,230.    Elapsed: 0:38:05.\n",
            "  Batch   180  of  1,230.    Elapsed: 0:40:19.\n",
            "  Batch   190  of  1,230.    Elapsed: 0:42:32.\n",
            "  Batch   200  of  1,230.    Elapsed: 0:44:46.\n",
            "  Batch   210  of  1,230.    Elapsed: 0:47:00.\n",
            "  Batch   220  of  1,230.    Elapsed: 0:49:15.\n",
            "  Batch   230  of  1,230.    Elapsed: 0:51:28.\n",
            "  Batch   240  of  1,230.    Elapsed: 0:53:43.\n",
            "  Batch   250  of  1,230.    Elapsed: 0:55:58.\n",
            "  Batch   260  of  1,230.    Elapsed: 0:58:12.\n",
            "  Batch   270  of  1,230.    Elapsed: 1:00:27.\n",
            "  Batch   280  of  1,230.    Elapsed: 1:02:42.\n",
            "  Batch   290  of  1,230.    Elapsed: 1:04:56.\n",
            "  Batch   300  of  1,230.    Elapsed: 1:07:11.\n",
            "  Batch   310  of  1,230.    Elapsed: 1:09:25.\n",
            "  Batch   320  of  1,230.    Elapsed: 1:11:39.\n",
            "  Batch   330  of  1,230.    Elapsed: 1:13:53.\n",
            "  Batch   340  of  1,230.    Elapsed: 1:16:08.\n",
            "  Batch   350  of  1,230.    Elapsed: 1:18:23.\n",
            "  Batch   360  of  1,230.    Elapsed: 1:20:36.\n",
            "  Batch   370  of  1,230.    Elapsed: 1:22:50.\n",
            "  Batch   380  of  1,230.    Elapsed: 1:25:04.\n",
            "  Batch   390  of  1,230.    Elapsed: 1:27:20.\n",
            "  Batch   400  of  1,230.    Elapsed: 1:29:34.\n",
            "  Batch   410  of  1,230.    Elapsed: 1:31:49.\n",
            "  Batch   420  of  1,230.    Elapsed: 1:34:03.\n",
            "  Batch   430  of  1,230.    Elapsed: 1:36:18.\n",
            "  Batch   440  of  1,230.    Elapsed: 1:38:34.\n",
            "  Batch   450  of  1,230.    Elapsed: 1:40:48.\n",
            "  Batch   460  of  1,230.    Elapsed: 1:43:03.\n",
            "  Batch   470  of  1,230.    Elapsed: 1:45:18.\n",
            "  Batch   480  of  1,230.    Elapsed: 1:47:32.\n",
            "  Batch   490  of  1,230.    Elapsed: 1:49:47.\n",
            "  Batch   500  of  1,230.    Elapsed: 1:52:02.\n",
            "  Batch   510  of  1,230.    Elapsed: 1:54:17.\n",
            "  Batch   520  of  1,230.    Elapsed: 1:56:32.\n",
            "  Batch   530  of  1,230.    Elapsed: 1:58:46.\n",
            "  Batch   540  of  1,230.    Elapsed: 2:01:00.\n",
            "  Batch   550  of  1,230.    Elapsed: 2:03:15.\n",
            "  Batch   560  of  1,230.    Elapsed: 2:05:28.\n",
            "  Batch   570  of  1,230.    Elapsed: 2:07:43.\n",
            "  Batch   580  of  1,230.    Elapsed: 2:09:57.\n",
            "  Batch   590  of  1,230.    Elapsed: 2:12:11.\n",
            "  Batch   600  of  1,230.    Elapsed: 2:14:25.\n",
            "  Batch   610  of  1,230.    Elapsed: 2:16:39.\n",
            "  Batch   620  of  1,230.    Elapsed: 2:18:55.\n",
            "  Batch   630  of  1,230.    Elapsed: 2:21:09.\n",
            "  Batch   640  of  1,230.    Elapsed: 2:23:23.\n",
            "  Batch   650  of  1,230.    Elapsed: 2:25:37.\n",
            "  Batch   660  of  1,230.    Elapsed: 2:27:53.\n",
            "  Batch   670  of  1,230.    Elapsed: 2:30:07.\n",
            "  Batch   680  of  1,230.    Elapsed: 2:32:22.\n",
            "  Batch   690  of  1,230.    Elapsed: 2:34:36.\n",
            "  Batch   700  of  1,230.    Elapsed: 2:36:49.\n",
            "  Batch   710  of  1,230.    Elapsed: 2:39:02.\n",
            "  Batch   720  of  1,230.    Elapsed: 2:41:17.\n",
            "  Batch   730  of  1,230.    Elapsed: 2:43:31.\n",
            "  Batch   740  of  1,230.    Elapsed: 2:45:47.\n",
            "  Batch   750  of  1,230.    Elapsed: 2:48:01.\n",
            "  Batch   760  of  1,230.    Elapsed: 2:50:15.\n",
            "  Batch   770  of  1,230.    Elapsed: 2:52:29.\n",
            "  Batch   780  of  1,230.    Elapsed: 2:54:44.\n",
            "  Batch   790  of  1,230.    Elapsed: 2:56:58.\n",
            "  Batch   800  of  1,230.    Elapsed: 2:59:11.\n",
            "  Batch   810  of  1,230.    Elapsed: 3:01:26.\n",
            "  Batch   820  of  1,230.    Elapsed: 3:03:42.\n",
            "  Batch   830  of  1,230.    Elapsed: 3:05:56.\n",
            "  Batch   840  of  1,230.    Elapsed: 3:08:10.\n",
            "  Batch   850  of  1,230.    Elapsed: 3:10:24.\n",
            "  Batch   860  of  1,230.    Elapsed: 3:12:38.\n",
            "  Batch   870  of  1,230.    Elapsed: 3:14:53.\n",
            "  Batch   880  of  1,230.    Elapsed: 3:17:08.\n",
            "  Batch   890  of  1,230.    Elapsed: 3:19:23.\n",
            "  Batch   900  of  1,230.    Elapsed: 3:21:37.\n",
            "  Batch   910  of  1,230.    Elapsed: 3:23:51.\n",
            "  Batch   920  of  1,230.    Elapsed: 3:26:05.\n",
            "  Batch   930  of  1,230.    Elapsed: 3:28:19.\n",
            "  Batch   940  of  1,230.    Elapsed: 3:30:34.\n",
            "  Batch   950  of  1,230.    Elapsed: 3:32:48.\n",
            "  Batch   960  of  1,230.    Elapsed: 3:35:02.\n",
            "  Batch   970  of  1,230.    Elapsed: 3:37:17.\n",
            "  Batch   980  of  1,230.    Elapsed: 3:39:31.\n",
            "  Batch   990  of  1,230.    Elapsed: 3:41:46.\n",
            "  Batch 1,000  of  1,230.    Elapsed: 3:44:00.\n",
            "  Batch 1,010  of  1,230.    Elapsed: 3:46:14.\n",
            "  Batch 1,020  of  1,230.    Elapsed: 3:48:28.\n",
            "  Batch 1,030  of  1,230.    Elapsed: 3:50:42.\n",
            "  Batch 1,040  of  1,230.    Elapsed: 3:52:56.\n",
            "  Batch 1,050  of  1,230.    Elapsed: 3:55:10.\n",
            "  Batch 1,060  of  1,230.    Elapsed: 3:57:24.\n",
            "  Batch 1,070  of  1,230.    Elapsed: 3:59:36.\n",
            "  Batch 1,080  of  1,230.    Elapsed: 4:01:50.\n",
            "  Batch 1,090  of  1,230.    Elapsed: 4:04:05.\n",
            "  Batch 1,100  of  1,230.    Elapsed: 4:06:21.\n",
            "  Batch 1,110  of  1,230.    Elapsed: 4:08:35.\n",
            "  Batch 1,120  of  1,230.    Elapsed: 4:10:50.\n",
            "  Batch 1,130  of  1,230.    Elapsed: 4:13:06.\n",
            "  Batch 1,140  of  1,230.    Elapsed: 4:15:20.\n",
            "  Batch 1,150  of  1,230.    Elapsed: 4:17:34.\n",
            "  Batch 1,160  of  1,230.    Elapsed: 4:19:49.\n",
            "  Batch 1,170  of  1,230.    Elapsed: 4:22:03.\n",
            "  Batch 1,180  of  1,230.    Elapsed: 4:24:17.\n",
            "  Batch 1,190  of  1,230.    Elapsed: 4:26:29.\n",
            "  Batch 1,200  of  1,230.    Elapsed: 4:28:43.\n",
            "  Batch 1,210  of  1,230.    Elapsed: 4:30:57.\n",
            "  Batch 1,220  of  1,230.    Elapsed: 4:33:11.\n",
            "Train loss: 0.15604332405013766\n",
            "Validation loss: 0.10744619274335186\n",
            "Validation Accuracy: 0.2435288116052736\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "Epoch:  70%|███████   | 7/10 [33:46:10<14:30:19, 17406.42s/it]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "F1-Score: 0.8964830655648985\n",
            "  Batch    10  of  1,230.    Elapsed: 0:02:13.\n",
            "  Batch    20  of  1,230.    Elapsed: 0:04:26.\n",
            "  Batch    30  of  1,230.    Elapsed: 0:06:41.\n",
            "  Batch    40  of  1,230.    Elapsed: 0:08:57.\n",
            "  Batch    50  of  1,230.    Elapsed: 0:11:11.\n",
            "  Batch    60  of  1,230.    Elapsed: 0:13:25.\n",
            "  Batch    70  of  1,230.    Elapsed: 0:15:39.\n",
            "  Batch    80  of  1,230.    Elapsed: 0:17:52.\n",
            "  Batch    90  of  1,230.    Elapsed: 0:20:07.\n",
            "  Batch   100  of  1,230.    Elapsed: 0:22:21.\n",
            "  Batch   110  of  1,230.    Elapsed: 0:24:36.\n",
            "  Batch   120  of  1,230.    Elapsed: 0:26:51.\n",
            "  Batch   130  of  1,230.    Elapsed: 0:29:05.\n",
            "  Batch   140  of  1,230.    Elapsed: 0:31:18.\n",
            "  Batch   150  of  1,230.    Elapsed: 0:33:33.\n",
            "  Batch   160  of  1,230.    Elapsed: 0:35:47.\n",
            "  Batch   170  of  1,230.    Elapsed: 0:38:01.\n",
            "  Batch   180  of  1,230.    Elapsed: 0:40:15.\n",
            "  Batch   190  of  1,230.    Elapsed: 0:42:29.\n",
            "  Batch   200  of  1,230.    Elapsed: 0:44:44.\n",
            "  Batch   210  of  1,230.    Elapsed: 0:46:59.\n",
            "  Batch   220  of  1,230.    Elapsed: 0:49:13.\n",
            "  Batch   230  of  1,230.    Elapsed: 0:51:26.\n",
            "  Batch   240  of  1,230.    Elapsed: 0:53:41.\n",
            "  Batch   250  of  1,230.    Elapsed: 0:55:55.\n",
            "  Batch   260  of  1,230.    Elapsed: 0:58:09.\n",
            "  Batch   270  of  1,230.    Elapsed: 1:00:22.\n",
            "  Batch   280  of  1,230.    Elapsed: 1:02:37.\n",
            "  Batch   290  of  1,230.    Elapsed: 1:04:50.\n",
            "  Batch   300  of  1,230.    Elapsed: 1:07:04.\n",
            "  Batch   310  of  1,230.    Elapsed: 1:09:18.\n",
            "  Batch   320  of  1,230.    Elapsed: 1:11:33.\n",
            "  Batch   330  of  1,230.    Elapsed: 1:13:47.\n",
            "  Batch   340  of  1,230.    Elapsed: 1:16:01.\n",
            "  Batch   350  of  1,230.    Elapsed: 1:18:17.\n",
            "  Batch   360  of  1,230.    Elapsed: 1:20:31.\n",
            "  Batch   370  of  1,230.    Elapsed: 1:22:45.\n",
            "  Batch   380  of  1,230.    Elapsed: 1:24:59.\n",
            "  Batch   390  of  1,230.    Elapsed: 1:27:14.\n",
            "  Batch   400  of  1,230.    Elapsed: 1:29:27.\n",
            "  Batch   410  of  1,230.    Elapsed: 1:31:41.\n",
            "  Batch   420  of  1,230.    Elapsed: 1:33:57.\n",
            "  Batch   430  of  1,230.    Elapsed: 1:36:12.\n",
            "  Batch   440  of  1,230.    Elapsed: 1:38:25.\n",
            "  Batch   450  of  1,230.    Elapsed: 1:40:39.\n",
            "  Batch   460  of  1,230.    Elapsed: 1:42:54.\n",
            "  Batch   470  of  1,230.    Elapsed: 1:45:07.\n",
            "  Batch   480  of  1,230.    Elapsed: 1:47:21.\n",
            "  Batch   490  of  1,230.    Elapsed: 1:49:35.\n",
            "  Batch   500  of  1,230.    Elapsed: 1:51:47.\n",
            "  Batch   510  of  1,230.    Elapsed: 1:54:02.\n",
            "  Batch   520  of  1,230.    Elapsed: 1:56:16.\n",
            "  Batch   530  of  1,230.    Elapsed: 1:58:30.\n",
            "  Batch   540  of  1,230.    Elapsed: 2:00:44.\n",
            "  Batch   550  of  1,230.    Elapsed: 2:02:59.\n",
            "  Batch   560  of  1,230.    Elapsed: 2:05:14.\n",
            "  Batch   570  of  1,230.    Elapsed: 2:07:28.\n",
            "  Batch   580  of  1,230.    Elapsed: 2:09:44.\n",
            "  Batch   590  of  1,230.    Elapsed: 2:11:58.\n",
            "  Batch   600  of  1,230.    Elapsed: 2:14:13.\n",
            "  Batch   610  of  1,230.    Elapsed: 2:16:28.\n",
            "  Batch   620  of  1,230.    Elapsed: 2:18:43.\n",
            "  Batch   630  of  1,230.    Elapsed: 2:20:56.\n",
            "  Batch   640  of  1,230.    Elapsed: 2:23:11.\n",
            "  Batch   650  of  1,230.    Elapsed: 2:25:25.\n",
            "  Batch   660  of  1,230.    Elapsed: 2:27:39.\n",
            "  Batch   670  of  1,230.    Elapsed: 2:29:53.\n",
            "  Batch   680  of  1,230.    Elapsed: 2:32:07.\n",
            "  Batch   690  of  1,230.    Elapsed: 2:34:22.\n",
            "  Batch   700  of  1,230.    Elapsed: 2:36:37.\n",
            "  Batch   710  of  1,230.    Elapsed: 2:38:52.\n",
            "  Batch   720  of  1,230.    Elapsed: 2:41:07.\n",
            "  Batch   730  of  1,230.    Elapsed: 2:43:21.\n",
            "  Batch   740  of  1,230.    Elapsed: 2:45:36.\n",
            "  Batch   750  of  1,230.    Elapsed: 2:47:50.\n",
            "  Batch   760  of  1,230.    Elapsed: 2:50:03.\n",
            "  Batch   770  of  1,230.    Elapsed: 2:52:17.\n",
            "  Batch   780  of  1,230.    Elapsed: 2:54:32.\n",
            "  Batch   790  of  1,230.    Elapsed: 2:56:45.\n",
            "  Batch   800  of  1,230.    Elapsed: 2:59:00.\n",
            "  Batch   810  of  1,230.    Elapsed: 3:01:14.\n",
            "  Batch   820  of  1,230.    Elapsed: 3:03:29.\n",
            "  Batch   830  of  1,230.    Elapsed: 3:05:42.\n",
            "  Batch   840  of  1,230.    Elapsed: 3:07:56.\n",
            "  Batch   850  of  1,230.    Elapsed: 3:10:11.\n",
            "  Batch   860  of  1,230.    Elapsed: 3:12:24.\n",
            "  Batch   870  of  1,230.    Elapsed: 3:14:39.\n",
            "  Batch   880  of  1,230.    Elapsed: 3:16:54.\n",
            "  Batch   890  of  1,230.    Elapsed: 3:19:08.\n",
            "  Batch   900  of  1,230.    Elapsed: 3:21:22.\n",
            "  Batch   910  of  1,230.    Elapsed: 3:23:36.\n",
            "  Batch   920  of  1,230.    Elapsed: 3:25:51.\n",
            "  Batch   930  of  1,230.    Elapsed: 3:28:06.\n",
            "  Batch   940  of  1,230.    Elapsed: 3:30:21.\n",
            "  Batch   950  of  1,230.    Elapsed: 3:32:36.\n",
            "  Batch   960  of  1,230.    Elapsed: 3:34:51.\n",
            "  Batch   970  of  1,230.    Elapsed: 3:37:05.\n",
            "  Batch   980  of  1,230.    Elapsed: 3:39:19.\n",
            "  Batch   990  of  1,230.    Elapsed: 3:41:33.\n",
            "  Batch 1,000  of  1,230.    Elapsed: 3:43:47.\n",
            "  Batch 1,010  of  1,230.    Elapsed: 3:46:01.\n",
            "  Batch 1,020  of  1,230.    Elapsed: 3:48:15.\n",
            "  Batch 1,030  of  1,230.    Elapsed: 3:50:29.\n",
            "  Batch 1,040  of  1,230.    Elapsed: 3:52:44.\n",
            "  Batch 1,050  of  1,230.    Elapsed: 3:54:58.\n",
            "  Batch 1,060  of  1,230.    Elapsed: 3:57:11.\n",
            "  Batch 1,070  of  1,230.    Elapsed: 3:59:25.\n",
            "  Batch 1,080  of  1,230.    Elapsed: 4:01:40.\n",
            "  Batch 1,090  of  1,230.    Elapsed: 4:03:55.\n",
            "  Batch 1,100  of  1,230.    Elapsed: 4:06:08.\n",
            "  Batch 1,110  of  1,230.    Elapsed: 4:08:23.\n",
            "  Batch 1,120  of  1,230.    Elapsed: 4:10:38.\n",
            "  Batch 1,130  of  1,230.    Elapsed: 4:12:53.\n",
            "  Batch 1,140  of  1,230.    Elapsed: 4:15:08.\n",
            "  Batch 1,150  of  1,230.    Elapsed: 4:17:22.\n",
            "  Batch 1,160  of  1,230.    Elapsed: 4:19:37.\n",
            "  Batch 1,170  of  1,230.    Elapsed: 4:21:52.\n",
            "  Batch 1,180  of  1,230.    Elapsed: 4:24:06.\n",
            "  Batch 1,190  of  1,230.    Elapsed: 4:26:21.\n",
            "  Batch 1,200  of  1,230.    Elapsed: 4:28:34.\n",
            "  Batch 1,210  of  1,230.    Elapsed: 4:30:48.\n",
            "  Batch 1,220  of  1,230.    Elapsed: 4:33:02.\n",
            "Train loss: 0.1379555811000064\n",
            "Validation loss: 0.1477762524881502\n",
            "Validation Accuracy: 0.23960669170486057\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "Epoch:  80%|████████  | 8/10 [38:37:05<9:40:41, 17420.93s/it] "
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "F1-Score: 0.8772330836818727\n",
            "  Batch    10  of  1,230.    Elapsed: 0:02:16.\n",
            "  Batch    20  of  1,230.    Elapsed: 0:04:30.\n",
            "  Batch    30  of  1,230.    Elapsed: 0:06:44.\n",
            "  Batch    40  of  1,230.    Elapsed: 0:08:58.\n",
            "  Batch    50  of  1,230.    Elapsed: 0:11:13.\n",
            "  Batch    60  of  1,230.    Elapsed: 0:13:27.\n",
            "  Batch    70  of  1,230.    Elapsed: 0:15:42.\n",
            "  Batch    80  of  1,230.    Elapsed: 0:17:56.\n",
            "  Batch    90  of  1,230.    Elapsed: 0:20:10.\n",
            "  Batch   100  of  1,230.    Elapsed: 0:22:24.\n",
            "  Batch   110  of  1,230.    Elapsed: 0:24:40.\n",
            "  Batch   120  of  1,230.    Elapsed: 0:26:55.\n",
            "  Batch   130  of  1,230.    Elapsed: 0:29:10.\n",
            "  Batch   140  of  1,230.    Elapsed: 0:31:24.\n",
            "  Batch   150  of  1,230.    Elapsed: 0:33:40.\n",
            "  Batch   160  of  1,230.    Elapsed: 0:35:55.\n",
            "  Batch   170  of  1,230.    Elapsed: 0:38:11.\n",
            "  Batch   180  of  1,230.    Elapsed: 0:40:26.\n",
            "  Batch   190  of  1,230.    Elapsed: 0:42:40.\n",
            "  Batch   200  of  1,230.    Elapsed: 0:44:55.\n",
            "  Batch   210  of  1,230.    Elapsed: 0:47:11.\n",
            "  Batch   220  of  1,230.    Elapsed: 0:49:26.\n",
            "  Batch   230  of  1,230.    Elapsed: 0:51:41.\n",
            "  Batch   240  of  1,230.    Elapsed: 0:53:56.\n",
            "  Batch   250  of  1,230.    Elapsed: 0:56:12.\n",
            "  Batch   260  of  1,230.    Elapsed: 0:58:28.\n",
            "  Batch   270  of  1,230.    Elapsed: 1:00:42.\n",
            "  Batch   280  of  1,230.    Elapsed: 1:02:55.\n",
            "  Batch   290  of  1,230.    Elapsed: 1:05:10.\n",
            "  Batch   300  of  1,230.    Elapsed: 1:07:26.\n",
            "  Batch   310  of  1,230.    Elapsed: 1:09:40.\n",
            "  Batch   320  of  1,230.    Elapsed: 1:11:54.\n",
            "  Batch   330  of  1,230.    Elapsed: 1:14:08.\n",
            "  Batch   340  of  1,230.    Elapsed: 1:16:22.\n",
            "  Batch   350  of  1,230.    Elapsed: 1:18:38.\n",
            "  Batch   360  of  1,230.    Elapsed: 1:20:52.\n",
            "  Batch   370  of  1,230.    Elapsed: 1:23:06.\n",
            "  Batch   380  of  1,230.    Elapsed: 1:25:20.\n",
            "  Batch   390  of  1,230.    Elapsed: 1:27:34.\n",
            "  Batch   400  of  1,230.    Elapsed: 1:29:49.\n",
            "  Batch   410  of  1,230.    Elapsed: 1:32:03.\n",
            "  Batch   420  of  1,230.    Elapsed: 1:34:17.\n",
            "  Batch   430  of  1,230.    Elapsed: 1:36:30.\n",
            "  Batch   440  of  1,230.    Elapsed: 1:38:45.\n",
            "  Batch   450  of  1,230.    Elapsed: 1:40:59.\n",
            "  Batch   460  of  1,230.    Elapsed: 1:43:13.\n",
            "  Batch   470  of  1,230.    Elapsed: 1:45:28.\n",
            "  Batch   480  of  1,230.    Elapsed: 1:47:43.\n",
            "  Batch   490  of  1,230.    Elapsed: 1:49:56.\n",
            "  Batch   500  of  1,230.    Elapsed: 1:52:11.\n",
            "  Batch   510  of  1,230.    Elapsed: 1:54:25.\n",
            "  Batch   520  of  1,230.    Elapsed: 1:56:39.\n",
            "  Batch   530  of  1,230.    Elapsed: 1:58:53.\n",
            "  Batch   540  of  1,230.    Elapsed: 2:01:08.\n",
            "  Batch   550  of  1,230.    Elapsed: 2:03:23.\n",
            "  Batch   560  of  1,230.    Elapsed: 2:05:38.\n",
            "  Batch   570  of  1,230.    Elapsed: 2:07:53.\n",
            "  Batch   580  of  1,230.    Elapsed: 2:10:08.\n",
            "  Batch   590  of  1,230.    Elapsed: 2:12:22.\n",
            "  Batch   600  of  1,230.    Elapsed: 2:14:36.\n",
            "  Batch   610  of  1,230.    Elapsed: 2:16:50.\n",
            "  Batch   620  of  1,230.    Elapsed: 2:19:04.\n",
            "  Batch   630  of  1,230.    Elapsed: 2:21:18.\n",
            "  Batch   640  of  1,230.    Elapsed: 2:23:32.\n",
            "  Batch   650  of  1,230.    Elapsed: 2:25:46.\n",
            "  Batch   660  of  1,230.    Elapsed: 2:28:02.\n",
            "  Batch   670  of  1,230.    Elapsed: 2:30:16.\n",
            "  Batch   680  of  1,230.    Elapsed: 2:32:30.\n",
            "  Batch   690  of  1,230.    Elapsed: 2:34:44.\n",
            "  Batch   700  of  1,230.    Elapsed: 2:36:57.\n",
            "  Batch   710  of  1,230.    Elapsed: 2:39:12.\n",
            "  Batch   720  of  1,230.    Elapsed: 2:41:25.\n",
            "  Batch   730  of  1,230.    Elapsed: 2:43:40.\n",
            "  Batch   740  of  1,230.    Elapsed: 2:45:55.\n",
            "  Batch   750  of  1,230.    Elapsed: 2:48:08.\n",
            "  Batch   760  of  1,230.    Elapsed: 2:50:22.\n",
            "  Batch   770  of  1,230.    Elapsed: 2:52:37.\n",
            "  Batch   780  of  1,230.    Elapsed: 2:54:51.\n",
            "  Batch   790  of  1,230.    Elapsed: 2:57:06.\n",
            "  Batch   800  of  1,230.    Elapsed: 2:59:21.\n",
            "  Batch   810  of  1,230.    Elapsed: 3:01:34.\n",
            "  Batch   820  of  1,230.    Elapsed: 3:03:48.\n",
            "  Batch   830  of  1,230.    Elapsed: 3:06:03.\n",
            "  Batch   840  of  1,230.    Elapsed: 3:08:18.\n",
            "  Batch   850  of  1,230.    Elapsed: 3:10:33.\n",
            "  Batch   860  of  1,230.    Elapsed: 3:12:46.\n",
            "  Batch   870  of  1,230.    Elapsed: 3:15:01.\n",
            "  Batch   880  of  1,230.    Elapsed: 3:17:14.\n",
            "  Batch   890  of  1,230.    Elapsed: 3:19:28.\n",
            "  Batch   900  of  1,230.    Elapsed: 3:21:42.\n",
            "  Batch   910  of  1,230.    Elapsed: 3:23:57.\n",
            "  Batch   920  of  1,230.    Elapsed: 3:26:12.\n",
            "  Batch   930  of  1,230.    Elapsed: 3:28:26.\n",
            "  Batch   940  of  1,230.    Elapsed: 3:30:41.\n",
            "  Batch   950  of  1,230.    Elapsed: 3:32:56.\n",
            "  Batch   960  of  1,230.    Elapsed: 3:35:10.\n",
            "  Batch   970  of  1,230.    Elapsed: 3:37:24.\n",
            "  Batch   980  of  1,230.    Elapsed: 3:39:38.\n",
            "  Batch   990  of  1,230.    Elapsed: 3:41:53.\n",
            "  Batch 1,000  of  1,230.    Elapsed: 3:44:07.\n",
            "  Batch 1,010  of  1,230.    Elapsed: 3:46:21.\n",
            "  Batch 1,020  of  1,230.    Elapsed: 3:48:37.\n",
            "  Batch 1,030  of  1,230.    Elapsed: 3:50:51.\n",
            "  Batch 1,040  of  1,230.    Elapsed: 3:53:06.\n",
            "  Batch 1,050  of  1,230.    Elapsed: 3:55:20.\n",
            "  Batch 1,060  of  1,230.    Elapsed: 3:57:35.\n",
            "  Batch 1,070  of  1,230.    Elapsed: 3:59:49.\n",
            "  Batch 1,080  of  1,230.    Elapsed: 4:02:03.\n",
            "  Batch 1,090  of  1,230.    Elapsed: 4:04:16.\n",
            "  Batch 1,100  of  1,230.    Elapsed: 4:06:30.\n",
            "  Batch 1,110  of  1,230.    Elapsed: 4:08:44.\n",
            "  Batch 1,120  of  1,230.    Elapsed: 4:10:59.\n",
            "  Batch 1,130  of  1,230.    Elapsed: 4:13:14.\n",
            "  Batch 1,140  of  1,230.    Elapsed: 4:15:28.\n",
            "  Batch 1,150  of  1,230.    Elapsed: 4:17:42.\n",
            "  Batch 1,160  of  1,230.    Elapsed: 4:19:56.\n",
            "  Batch 1,170  of  1,230.    Elapsed: 4:22:11.\n",
            "  Batch 1,180  of  1,230.    Elapsed: 4:24:24.\n",
            "  Batch 1,190  of  1,230.    Elapsed: 4:26:38.\n",
            "  Batch 1,200  of  1,230.    Elapsed: 4:28:51.\n",
            "  Batch 1,210  of  1,230.    Elapsed: 4:31:06.\n",
            "  Batch 1,220  of  1,230.    Elapsed: 4:33:19.\n",
            "Train loss: 0.12237776792570343\n",
            "Validation loss: 0.1056310157325581\n",
            "Validation Accuracy: 0.24297984566853392\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "Epoch:  90%|█████████ | 9/10 [43:28:13<4:50:34, 17434.96s/it]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "F1-Score: 0.8982955602092547\n",
            "  Batch    10  of  1,230.    Elapsed: 0:02:14.\n",
            "  Batch    20  of  1,230.    Elapsed: 0:04:27.\n",
            "  Batch    30  of  1,230.    Elapsed: 0:06:42.\n",
            "  Batch    40  of  1,230.    Elapsed: 0:08:55.\n",
            "  Batch    50  of  1,230.    Elapsed: 0:11:10.\n",
            "  Batch    60  of  1,230.    Elapsed: 0:13:26.\n",
            "  Batch    70  of  1,230.    Elapsed: 0:15:40.\n",
            "  Batch    80  of  1,230.    Elapsed: 0:17:54.\n",
            "  Batch    90  of  1,230.    Elapsed: 0:20:09.\n",
            "  Batch   100  of  1,230.    Elapsed: 0:22:23.\n",
            "  Batch   110  of  1,230.    Elapsed: 0:24:39.\n",
            "  Batch   120  of  1,230.    Elapsed: 0:26:53.\n",
            "  Batch   130  of  1,230.    Elapsed: 0:29:07.\n",
            "  Batch   140  of  1,230.    Elapsed: 0:31:21.\n",
            "  Batch   150  of  1,230.    Elapsed: 0:33:35.\n",
            "  Batch   160  of  1,230.    Elapsed: 0:35:50.\n",
            "  Batch   170  of  1,230.    Elapsed: 0:38:04.\n",
            "  Batch   180  of  1,230.    Elapsed: 0:40:18.\n",
            "  Batch   190  of  1,230.    Elapsed: 0:42:33.\n",
            "  Batch   200  of  1,230.    Elapsed: 0:44:46.\n",
            "  Batch   210  of  1,230.    Elapsed: 0:47:00.\n",
            "  Batch   220  of  1,230.    Elapsed: 0:49:14.\n",
            "  Batch   230  of  1,230.    Elapsed: 0:51:28.\n",
            "  Batch   240  of  1,230.    Elapsed: 0:53:42.\n",
            "  Batch   250  of  1,230.    Elapsed: 0:55:58.\n",
            "  Batch   260  of  1,230.    Elapsed: 0:58:14.\n",
            "  Batch   270  of  1,230.    Elapsed: 1:00:28.\n",
            "  Batch   280  of  1,230.    Elapsed: 1:02:44.\n",
            "  Batch   290  of  1,230.    Elapsed: 1:04:58.\n",
            "  Batch   300  of  1,230.    Elapsed: 1:07:14.\n",
            "  Batch   310  of  1,230.    Elapsed: 1:09:28.\n",
            "  Batch   320  of  1,230.    Elapsed: 1:11:43.\n",
            "  Batch   330  of  1,230.    Elapsed: 1:13:58.\n",
            "  Batch   340  of  1,230.    Elapsed: 1:16:13.\n",
            "  Batch   350  of  1,230.    Elapsed: 1:18:28.\n",
            "  Batch   360  of  1,230.    Elapsed: 1:20:42.\n",
            "  Batch   370  of  1,230.    Elapsed: 1:22:57.\n",
            "  Batch   380  of  1,230.    Elapsed: 1:25:12.\n",
            "  Batch   390  of  1,230.    Elapsed: 1:27:27.\n",
            "  Batch   400  of  1,230.    Elapsed: 1:29:42.\n",
            "  Batch   410  of  1,230.    Elapsed: 1:31:57.\n",
            "  Batch   420  of  1,230.    Elapsed: 1:34:11.\n",
            "  Batch   430  of  1,230.    Elapsed: 1:36:25.\n",
            "  Batch   440  of  1,230.    Elapsed: 1:38:40.\n",
            "  Batch   450  of  1,230.    Elapsed: 1:40:55.\n",
            "  Batch   460  of  1,230.    Elapsed: 1:43:10.\n",
            "  Batch   470  of  1,230.    Elapsed: 1:45:26.\n",
            "  Batch   480  of  1,230.    Elapsed: 1:47:41.\n",
            "  Batch   490  of  1,230.    Elapsed: 1:49:56.\n",
            "  Batch   500  of  1,230.    Elapsed: 1:52:10.\n",
            "  Batch   510  of  1,230.    Elapsed: 1:54:24.\n",
            "  Batch   520  of  1,230.    Elapsed: 1:56:38.\n",
            "  Batch   530  of  1,230.    Elapsed: 1:58:53.\n",
            "  Batch   540  of  1,230.    Elapsed: 2:01:07.\n",
            "  Batch   550  of  1,230.    Elapsed: 2:03:22.\n",
            "  Batch   560  of  1,230.    Elapsed: 2:05:37.\n",
            "  Batch   570  of  1,230.    Elapsed: 2:07:51.\n",
            "  Batch   580  of  1,230.    Elapsed: 2:10:06.\n",
            "  Batch   590  of  1,230.    Elapsed: 2:12:22.\n",
            "  Batch   600  of  1,230.    Elapsed: 2:14:37.\n",
            "  Batch   610  of  1,230.    Elapsed: 2:16:53.\n",
            "  Batch   620  of  1,230.    Elapsed: 2:19:09.\n",
            "  Batch   630  of  1,230.    Elapsed: 2:21:24.\n",
            "  Batch   640  of  1,230.    Elapsed: 2:23:39.\n",
            "  Batch   650  of  1,230.    Elapsed: 2:25:53.\n",
            "  Batch   660  of  1,230.    Elapsed: 2:28:08.\n",
            "  Batch   670  of  1,230.    Elapsed: 2:30:24.\n",
            "  Batch   680  of  1,230.    Elapsed: 2:32:40.\n",
            "  Batch   690  of  1,230.    Elapsed: 2:34:55.\n",
            "  Batch   700  of  1,230.    Elapsed: 2:37:11.\n",
            "  Batch   710  of  1,230.    Elapsed: 2:39:26.\n",
            "  Batch   720  of  1,230.    Elapsed: 2:41:41.\n",
            "  Batch   730  of  1,230.    Elapsed: 2:43:57.\n",
            "  Batch   740  of  1,230.    Elapsed: 2:46:10.\n",
            "  Batch   750  of  1,230.    Elapsed: 2:48:25.\n",
            "  Batch   760  of  1,230.    Elapsed: 2:50:39.\n",
            "  Batch   770  of  1,230.    Elapsed: 2:52:53.\n",
            "  Batch   780  of  1,230.    Elapsed: 2:55:08.\n",
            "  Batch   790  of  1,230.    Elapsed: 2:57:22.\n",
            "  Batch   800  of  1,230.    Elapsed: 2:59:38.\n",
            "  Batch   810  of  1,230.    Elapsed: 3:01:53.\n",
            "  Batch   820  of  1,230.    Elapsed: 3:04:08.\n",
            "  Batch   830  of  1,230.    Elapsed: 3:06:21.\n",
            "  Batch   840  of  1,230.    Elapsed: 3:08:36.\n",
            "  Batch   850  of  1,230.    Elapsed: 3:10:51.\n",
            "  Batch   860  of  1,230.    Elapsed: 3:13:07.\n",
            "  Batch   870  of  1,230.    Elapsed: 3:15:24.\n",
            "  Batch   880  of  1,230.    Elapsed: 3:17:39.\n",
            "  Batch   890  of  1,230.    Elapsed: 3:19:56.\n",
            "  Batch   900  of  1,230.    Elapsed: 3:22:11.\n",
            "  Batch   910  of  1,230.    Elapsed: 3:24:27.\n",
            "  Batch   920  of  1,230.    Elapsed: 3:26:42.\n",
            "  Batch   930  of  1,230.    Elapsed: 3:28:57.\n",
            "  Batch   940  of  1,230.    Elapsed: 3:31:13.\n",
            "  Batch   950  of  1,230.    Elapsed: 3:33:27.\n",
            "  Batch   960  of  1,230.    Elapsed: 3:35:43.\n",
            "  Batch   970  of  1,230.    Elapsed: 3:37:59.\n",
            "  Batch   980  of  1,230.    Elapsed: 3:40:16.\n",
            "  Batch   990  of  1,230.    Elapsed: 3:42:32.\n",
            "  Batch 1,000  of  1,230.    Elapsed: 3:44:49.\n",
            "  Batch 1,010  of  1,230.    Elapsed: 3:47:05.\n",
            "  Batch 1,020  of  1,230.    Elapsed: 3:49:20.\n",
            "  Batch 1,030  of  1,230.    Elapsed: 3:51:36.\n",
            "  Batch 1,040  of  1,230.    Elapsed: 3:53:53.\n",
            "  Batch 1,050  of  1,230.    Elapsed: 3:56:09.\n",
            "  Batch 1,060  of  1,230.    Elapsed: 3:58:24.\n",
            "  Batch 1,070  of  1,230.    Elapsed: 4:00:40.\n",
            "  Batch 1,080  of  1,230.    Elapsed: 4:02:55.\n",
            "  Batch 1,090  of  1,230.    Elapsed: 4:05:11.\n",
            "  Batch 1,100  of  1,230.    Elapsed: 4:07:27.\n",
            "  Batch 1,110  of  1,230.    Elapsed: 4:09:42.\n",
            "  Batch 1,120  of  1,230.    Elapsed: 4:11:57.\n",
            "  Batch 1,130  of  1,230.    Elapsed: 4:14:13.\n",
            "  Batch 1,140  of  1,230.    Elapsed: 4:16:29.\n",
            "  Batch 1,150  of  1,230.    Elapsed: 4:18:44.\n",
            "  Batch 1,160  of  1,230.    Elapsed: 4:20:58.\n",
            "  Batch 1,170  of  1,230.    Elapsed: 4:23:12.\n",
            "  Batch 1,180  of  1,230.    Elapsed: 4:25:26.\n",
            "  Batch 1,190  of  1,230.    Elapsed: 4:27:39.\n",
            "  Batch 1,200  of  1,230.    Elapsed: 4:29:54.\n",
            "  Batch 1,210  of  1,230.    Elapsed: 4:32:08.\n",
            "  Batch 1,220  of  1,230.    Elapsed: 4:34:23.\n",
            "Train loss: 0.1120520415496293\n",
            "Validation loss: 0.09837092554373462\n",
            "Validation Accuracy: 0.24350299538844558\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "Epoch: 100%|██████████| 10/10 [48:20:27<00:00, 17402.76s/it] "
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "F1-Score: 0.899932572383748\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "1sTXvalhs8vx"
      },
      "source": [
        "model.eval()\n",
        "predictions = []\n",
        "true_labels = []\n",
        "eval_loss, eval_accuracy = 0, 0\n",
        "nb_eval_steps, nb_eval_examples = 0, 0\n",
        "for batch in valid_dataloader:\n",
        "    batch = tuple(t.to(device) for t in batch)\n",
        "    b_input_ids, b_input_mask, b_labels = batch\n",
        "\n",
        "    with torch.no_grad():\n",
        "        tmp_eval_loss = model(b_input_ids, token_type_ids=None,\n",
        "                              attention_mask=b_input_mask, labels=b_labels)[0]\n",
        "        logits = model(b_input_ids, token_type_ids=None,\n",
        "                       attention_mask=b_input_mask)[0]\n",
        "        \n",
        "    logits = logits.detach().cpu().numpy()\n",
        "    predictions.extend([list(p) for p in np.argmax(logits, axis=2)])\n",
        "    label_ids = b_labels.to('cpu').numpy()\n",
        "    true_labels.append(label_ids)\n",
        "#     tmp_eval_accuracy = flat_accuracy(logits, label_ids)\n",
        "\n",
        "    eval_loss += tmp_eval_loss.mean().item()\n",
        "#     eval_accuracy += tmp_eval_accuracy\n",
        "\n",
        "    nb_eval_examples += b_input_ids.size(0)\n",
        "    nb_eval_steps += 1\n",
        "\n",
        "pred_tags = [[tags_val[p_i] for p_i in p] for p in predictions]\n",
        "valid_tags = [[tags_val[l_ii] for l_ii in l_i] for l in true_labels for l_i in l ]\n",
        "print(\"Validation loss: {}\".format(eval_loss/nb_eval_steps))\n",
        "# print(\"Validation Accuracy: {}\".format(eval_accuracy/nb_eval_steps))\n",
        "print(\"Validation F1-Score: {}\".format(f1_score(pred_tags, valid_tags)))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "r6JNS6UYs8vz"
      },
      "source": [
        "import pickle\n",
        "pickle.dump(model, open('CamemBERT_POS', 'wb'))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "heading_collapsed": true,
        "id": "PYikfpbLugYS"
      },
      "source": [
        "# Embeddings de doc"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "heading_collapsed": true,
        "hidden": true,
        "id": "BjCIw55xugYU"
      },
      "source": [
        "## Doc2Vec"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "hidden": true,
        "id": "_YKBbdbB8V3B"
      },
      "source": [
        "Utilisation du modèle Doc2Vec de Gensim  \n",
        "\n",
        "Demonstration sur le corpus lee Background, 314 documents obtenus à partir d'un service de résumé d'article par mail australien sur une grande variété de sujets.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "CRKcnSLzLKs5"
      },
      "source": [
        "import logging\n",
        "logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "0izPSUqCLKtE"
      },
      "source": [
        "import os\n",
        "import gensim\n",
        "# Set file names for train and test data\n",
        "test_data_dir = os.path.join(gensim.__path__[0], 'test', 'test_data')\n",
        "lee_train_file = os.path.join(test_data_dir, 'lee_background.cor')\n",
        "lee_test_file = os.path.join(test_data_dir, 'lee.cor')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "Ydji5yVtLKtM"
      },
      "source": [
        "import smart_open\n",
        "\n",
        "def read_corpus(fname, tokens_only=False):\n",
        "    with smart_open.open(fname, encoding=\"iso-8859-1\") as f:\n",
        "        for i, line in enumerate(f):\n",
        "            tokens = gensim.utils.simple_preprocess(line)\n",
        "            if tokens_only:\n",
        "                yield tokens\n",
        "            else:\n",
        "                # For training data, add tags\n",
        "                yield gensim.models.doc2vec.TaggedDocument(tokens, [i])\n",
        "\n",
        "train_corpus = list(read_corpus(lee_train_file))\n",
        "test_corpus = list(read_corpus(lee_test_file, tokens_only=True))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "hidden": true,
        "id": "Z8gIsgByLKtT"
      },
      "source": [
        "\n",
        "\n",
        "\n",
        "Aperçu du jeu de train\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "6hvPFckPLKtU"
      },
      "source": [
        "print(train_corpus[:2])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "hidden": true,
        "id": "mJweA-XfLKtb"
      },
      "source": [
        "Aperçu du jeu de test\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "oMrYDqBhLKtd"
      },
      "source": [
        "print(test_corpus[:2])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "hidden": true,
        "id": "0fFw9-GNLKtk"
      },
      "source": [
        "### Entrainement\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "1hMFvbhJLKtl"
      },
      "source": [
        "model = gensim.models.doc2vec.Doc2Vec(vector_size=50, min_count=2, epochs=40)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "hidden": true,
        "id": "AawhBdwXLKtp"
      },
      "source": [
        "Construction du vocabulaire\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "IsYsdsARLKtr"
      },
      "source": [
        "model.build_vocab(train_corpus)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "hidden": true,
        "id": "ZMTrLw3LCSjT"
      },
      "source": [
        "Entrainement"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "NwQWJNdGLKt0"
      },
      "source": [
        "model.train(train_corpus, total_examples=model.corpus_count, epochs=model.epochs)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "hidden": true,
        "id": "yZRrhB2nLKt5"
      },
      "source": [
        "la fonction ``model.infer_vector`` nous permet maintenant de générer l'embeddings.\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "TO9U7nxCLKt7"
      },
      "source": [
        "vector = model.infer_vector(['only', 'you', 'can', 'prevent', 'forest', 'fires', 'really', 'da'])\n",
        "print(vector, len(vector))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "hidden": true,
        "id": "g424ZE1WLKuC"
      },
      "source": [
        "### Essai sur nos données d'entrainement\n",
        "-------------------\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "y4tC54xDLKuE"
      },
      "source": [
        "ranks = []\n",
        "second_ranks = []\n",
        "for doc_id in range(len(train_corpus)):\n",
        "    inferred_vector = model.infer_vector(train_corpus[doc_id].words)\n",
        "    sims = model.docvecs.most_similar([inferred_vector], topn=len(model.docvecs))\n",
        "    rank = [docid for docid, sim in sims].index(doc_id)\n",
        "    ranks.append(rank)\n",
        "\n",
        "    second_ranks.append(sims[1])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "P9_hmTRKEh2u"
      },
      "source": [
        "second_ranks[-5:]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "ineM8KXNB3G3"
      },
      "source": [
        "print(sims[:5], '\\n', sims[-5:])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "4uI9ofS6LKuO"
      },
      "source": [
        "display('Document ({}): «{}»'.format(doc_id, ' '.join(train_corpus[doc_id].words)))\n",
        "print()\n",
        "display(u'SIMILAR/DISSIMILAR DOCS PER MODEL %s:' % model)\n",
        "print()\n",
        "for label, index in [('MOST', 0), ('SECOND-MOST', 1), ('MEDIAN', len(sims)//2), ('LEAST', len(sims) - 1)]:\n",
        "    display(u'%s %s: «%s»' % (label, sims[index], ' '.join(train_corpus[sims[index][0]].words)))\n",
        "    print()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "hidden": true,
        "id": "nywYZwvkLKuT"
      },
      "source": [
        "Le document le plus proche est le même ... Cohérent, mais il est pertinent de regarder le second plus proche, pour juger de la pertinence de l'algorithme, ainsi que le plus éloigné.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "uV5KV7T7LKuT"
      },
      "source": [
        "# Pick a random document from the corpus and infer a vector from the model\n",
        "import random\n",
        "doc_id = random.randint(0, len(train_corpus) - 1)\n",
        "# Compare and print the second-most-similar document\n",
        "display('Train Document ({}): «{}»'.format(doc_id, ' '.join(train_corpus[doc_id].words)))\n",
        "sim_id = second_ranks[doc_id]\n",
        "print()\n",
        "display('Similar Document {}: «{}»'.format(sim_id, ' '.join(train_corpus[sim_id[0]].words)))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "hidden": true,
        "id": "i9Sppy4SLKuW"
      },
      "source": [
        "\n",
        "### Test\n",
        "-----------------\n",
        "\n",
        "On va créer l'embedding à partir d'un vecteur du jeu de test et trouver celui du dataset d'entrainement le plus proche\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "wrnlmeNwLKuX"
      },
      "source": [
        "# Pick a random document from the test corpus and infer a vector from the model\n",
        "doc_id = random.randint(0, len(test_corpus) - 1)\n",
        "inferred_vector = model.infer_vector(test_corpus[doc_id])\n",
        "sims = model.docvecs.most_similar([inferred_vector], topn=len(model.docvecs))\n",
        "\n",
        "# Compare and print the most/median/least similar documents from the train corpus\n",
        "display('Test Document ({}): «{}»'.format(doc_id, ' '.join(test_corpus[doc_id])))\n",
        "print()\n",
        "display(u'SIMILAR/DISSIMILAR DOCS PER MODEL %s:' % model)\n",
        "print()\n",
        "for label, index in [('MOST', 0), ('MEDIAN', len(sims)//2), ('LEAST', len(sims) - 1)]:\n",
        "    display(u'%s %s: «%s»' % (label, sims[index], ' '.join(train_corpus[sims[index][0]].words)))\n",
        "    print()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "heading_collapsed": true,
        "hidden": true,
        "id": "TGp0qeXvtFeW"
      },
      "source": [
        "## LDA Model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "hidden": true,
        "id": "4C1DJL5ms8wu"
      },
      "source": [
        "Assessing the Model\n",
        "-------------------\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "hidden": true,
        "id": "ulPfI3kv8V3y"
      },
      "source": [
        "Modele LDA de Gensim, avec le corpus NIPS"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "ToR80j6jYeGC"
      },
      "source": [
        "%reset -f"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "w2Hcagg7tFeY"
      },
      "source": [
        "import logging\n",
        "logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "eHfA8Ss9tFel"
      },
      "source": [
        "import io\n",
        "import os.path\n",
        "import re\n",
        "import tarfile\n",
        "import nltk\n",
        "\n",
        "import smart_open\n",
        "\n",
        "def extract_documents(url='https://cs.nyu.edu/~roweis/data/nips12raw_str602.tgz'):\n",
        "    fname = url.split('/')[-1]\n",
        "    \n",
        "    # Download the file to local storage first.\n",
        "    # We can't read it on the fly because of \n",
        "    # https://github.com/RaRe-Technologies/smart_open/issues/331\n",
        "    if not os.path.isfile(fname):\n",
        "        with smart_open.open(url, \"rb\") as fin:\n",
        "            with smart_open.open(fname, 'wb') as fout:\n",
        "                while True:\n",
        "                    buf = fin.read(io.DEFAULT_BUFFER_SIZE)\n",
        "                    if not buf:\n",
        "                        break\n",
        "                    fout.write(buf)\n",
        "                         \n",
        "    with tarfile.open(fname, mode='r:gz') as tar:\n",
        "        # Ignore directory entries, as well as files like README, etc.\n",
        "        files = [\n",
        "            m for m in tar.getmembers()\n",
        "            if m.isfile() and re.search(r'nipstxt/nips\\d+/\\d+\\.txt', m.name)\n",
        "        ]\n",
        "        for member in sorted(files, key=lambda x: x.name):\n",
        "            member_bytes = tar.extractfile(member).read()\n",
        "            yield member_bytes.decode('utf-8', errors='replace')\n",
        "\n",
        "docs = list(extract_documents())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "1wUTd2bctFe1"
      },
      "source": [
        "print(len(docs))\n",
        "print(docs[0][:500])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "Lr2-gpuHtFe8"
      },
      "source": [
        "# Tokenize the documents.\n",
        "from nltk.tokenize import RegexpTokenizer\n",
        "\n",
        "# Split the documents into tokens.\n",
        "tokenizer = RegexpTokenizer(r'\\w+')\n",
        "for idx in range(len(docs)):\n",
        "    docs[idx] = docs[idx].lower()  # Convert to lowercase.\n",
        "    docs[idx] = tokenizer.tokenize(docs[idx])  # Split into words.\n",
        "\n",
        "# Remove numbers, but not words that contain numbers.\n",
        "docs = [[token for token in doc if not token.isnumeric()] for doc in docs]\n",
        "\n",
        "# Remove words that are only one character.\n",
        "docs = [[token for token in doc if len(token) > 1] for doc in docs]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "p2D2GTM-tFfE"
      },
      "source": [
        "# Lemmatize the documents.\n",
        "nltk.download('wordnet')\n",
        "from nltk.stem.wordnet import WordNetLemmatizer\n",
        "\n",
        "lemmatizer = WordNetLemmatizer()\n",
        "docs = [[lemmatizer.lemmatize(token) for token in doc] for doc in docs]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "_bwLiLmptFfL"
      },
      "source": [
        "# Compute bigrams.\n",
        "from gensim.models import Phrases\n",
        "\n",
        "# Add bigrams and trigrams to docs (only ones that appear 20 times or more).\n",
        "bigram = Phrases(docs, min_count=20)\n",
        "for idx in range(len(docs)):\n",
        "    for token in bigram[docs[idx]]:\n",
        "        if '_' in token:\n",
        "            # Token is a bigram, add to document.\n",
        "            docs[idx].append(token)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "CRNGpD0EtFfP"
      },
      "source": [
        "# Remove rare and common tokens.\n",
        "from gensim.corpora import Dictionary\n",
        "\n",
        "# Create a dictionary representation of the documents.\n",
        "dictionary = Dictionary(docs)\n",
        "\n",
        "# Filter out words that occur less than 20 documents, or more than 50% of the documents.\n",
        "dictionary.filter_extremes(no_below=20, no_above=0.5)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "n_hZjiu3tFfV"
      },
      "source": [
        "# Bag-of-words representation of the documents.\n",
        "corpus = [dictionary.doc2bow(doc) for doc in docs]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "mJf9QhAatFfc"
      },
      "source": [
        "print('Number of unique tokens: %d' % len(dictionary))\n",
        "print('Number of documents: %d' % len(corpus))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "qKqG2x8ytFfj",
        "scrolled": true
      },
      "source": [
        "# Train LDA model.\n",
        "from gensim.models import LdaModel\n",
        "\n",
        "# Set training parameters.\n",
        "num_topics = 10\n",
        "chunksize = 2000\n",
        "passes = 20\n",
        "iterations = 400\n",
        "eval_every = None  # Don't evaluate model perplexity, takes too much time.\n",
        "\n",
        "# Make a index to word dictionary.\n",
        "temp = dictionary[0]  # This is only to \"load\" the dictionary.\n",
        "id2word = dictionary.id2token\n",
        "\n",
        "model = LdaModel(\n",
        "    corpus=corpus,\n",
        "    id2word=id2word,\n",
        "    chunksize=chunksize,\n",
        "    alpha='auto',\n",
        "    eta='auto',\n",
        "    iterations=iterations,\n",
        "    num_topics=num_topics,\n",
        "    passes=passes,\n",
        "    eval_every=eval_every\n",
        ")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "AZr6b66itFfr",
        "scrolled": true
      },
      "source": [
        "top_topics = model.top_topics(corpus) #, num_words=20)\n",
        "\n",
        "# Average topic coherence is the sum of topic coherences of all topics, divided by the number of topics.\n",
        "avg_topic_coherence = sum([t[1] for t in top_topics]) / num_topics\n",
        "print('Average topic coherence: %.4f.' % avg_topic_coherence)\n",
        "\n",
        "from pprint import pprint\n",
        "pprint(top_topics)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "heading_collapsed": true,
        "hidden": true,
        "id": "VOgn7Wl9swsB"
      },
      "source": [
        "## Word Movers' Distance"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "hidden": true,
        "id": "clYphaxP8V4K"
      },
      "source": [
        "Modèle WMD de Gensim"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "Y8ZMoUZSYk3r"
      },
      "source": [
        "%reset -f"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "RTN0tDhpswsI"
      },
      "source": [
        "# Initialize logging.\n",
        "import logging\n",
        "logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)\n",
        "\n",
        "sentence_obama = 'Obama speaks to the media in Illinois'\n",
        "sentence_president = 'The president greets the press in Chicago'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "hidden": true,
        "id": "TjIOnZ1CswsP"
      },
      "source": [
        "These sentences have very similar content, and as such the WMD should be low.\n",
        "Before we compute the WMD, we want to remove stopwords (\"the\", \"to\", etc.),\n",
        "as these do not contribute a lot to the information in the sentences.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "loN6LYHbI295"
      },
      "source": [
        "!python -m spacy download en_core_web_sm"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "jrC0uLn2KDiQ"
      },
      "source": [
        "import spacy\n",
        "nlp = spacy.load('en_core_web_sm')\n",
        "from spacy.lang.en.stop_words import STOP_WORDS\n",
        "print(list(STOP_WORDS)[:10])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "7aFAmyu_Kaze"
      },
      "source": [
        "def preprocess(sentence):\n",
        "    sentence=nlp(sentence)\n",
        "    return [token.text.lower() for token in sentence if not token.text.lower() in STOP_WORDS and not token.is_punct]\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "kaVrbptAKwY8"
      },
      "source": [
        "sentence_obama = preprocess(sentence_obama)\n",
        "sentence_president = preprocess(sentence_president)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "hidden": true,
        "id": "dDg7-4KkswsX"
      },
      "source": [
        "On va maintenant utiliser les poids pré entrainés d'un model Word2Vec de Gensim\n",
        "\n",
        "Attention cet embedding prend beaucoup de mémoire.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "ZW-VlCMHswsZ"
      },
      "source": [
        "import gensim.downloader as api\n",
        "model = api.load('word2vec-google-news-300')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "hidden": true,
        "id": "eSeoEAz3swsf"
      },
      "source": [
        "On va maintenant calculer la distance avec la méthod ``wmdistance`` .\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "QQpDAHYhswsh"
      },
      "source": [
        "distance = model.wmdistance(sentence_obama, sentence_president)\n",
        "print('distance = %.4f' % distance)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "hidden": true,
        "id": "QqVqTgTXswsn"
      },
      "source": [
        "Si on essaye avec des phrases dont le sens est différent\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "LVe5tkUkswso"
      },
      "source": [
        "sentence_orange = preprocess('Oranges are my favorite fruit')\n",
        "distance = model.wmdistance(sentence_obama, sentence_orange)\n",
        "print('distance = %.4f' % distance)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "8Fm3r_8rswsu"
      },
      "source": [
        "model.init_sims(replace=True)  # Normalizes the vectors in the word2vec class.\n",
        "\n",
        "distance = model.wmdistance(sentence_obama, sentence_president)  # Compute WMD as normal.\n",
        "print('distance: %r' % distance)\n",
        "\n",
        "distance = model.wmdistance(sentence_obama, sentence_orange)\n",
        "print('distance = %.4f' % distance)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "heading_collapsed": true,
        "hidden": true,
        "id": "izA3-6kffbdT"
      },
      "source": [
        "## DistillBERT sentiment analysis"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "uShDVUAFZUwz"
      },
      "source": [
        "%reset -f"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "To9ENLU90WGl"
      },
      "source": [
        "!pip install transformers"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "fvFvBLJV0Dkv"
      },
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.linear_model import LogisticRegression\n",
        "from sklearn.model_selection import GridSearchCV\n",
        "from sklearn.model_selection import cross_val_score\n",
        "import torch\n",
        "import transformers as ppb\n",
        "import urllib.request\n",
        "import os\n",
        "import zipfile\n",
        "import warnings\n",
        "warnings.filterwarnings('ignore')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "KDW7swH8-Py5"
      },
      "source": [
        "print(torch.__version__)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "PkHh_CVP-BAE"
      },
      "source": [
        "df = pd.read_csv('https://github.com/clairett/pytorch-sentiment-classification/raw/master/data/SST2/train.tsv', delimiter='\\t', header=None)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "gTM3hOHW4hUY"
      },
      "source": [
        "batch_1 = df"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "6t-svuwwGo3K"
      },
      "source": [
        "pd.set_option('display.max_colwidth',None)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "RUHzX36pSg3v"
      },
      "source": [
        "batch_1.head()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "jGvcfcCP5xpZ"
      },
      "source": [
        "batch_1[1].value_counts()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "q1InADgf5xm2"
      },
      "source": [
        "# DistilBERT:\n",
        "model_class, tokenizer_class, pretrained_weights = (ppb.DistilBertModel, ppb.DistilBertTokenizer, 'distilbert-base-uncased')\n",
        "\n",
        "## Bert\n",
        "#model_class, tokenizer_class, pretrained_weights = (ppb.BertModel, ppb.BertTokenizer, 'bert-base-uncased')\n",
        "\n",
        "# chargement modèle et tokenizer\n",
        "tokenizer = tokenizer_class.from_pretrained(pretrained_weights)\n",
        "model = model_class.from_pretrained(pretrained_weights)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "Dg82ndBA5xlN"
      },
      "source": [
        "# Tokenization de nos phrases pour distillBERT\n",
        "tokenized = batch_1[0].apply((lambda x: tokenizer.encode(x, add_special_tokens=True)))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "URn-DWJt5xhP"
      },
      "source": [
        "# Padding afin que chaque phrase fasse la même taille\n",
        "\n",
        "max_len = 0\n",
        "for i in tokenized.values:\n",
        "    if len(i) > max_len:\n",
        "        max_len = len(i)\n",
        "\n",
        "padded = np.array([i + [0]*(max_len-len(i)) for i in tokenized.values])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "jdi7uXo95xeq"
      },
      "source": [
        "np.array(padded).shape"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "4K_iGRNa_Ozc"
      },
      "source": [
        "# On cache ce padding\n",
        "attention_mask = np.where(padded != 0, 1, 0)\n",
        "attention_mask.shape"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "39UVjAV56PJz"
      },
      "source": [
        "# On applique le modèle sur nos token avec le masque\n",
        "model.eval()\n",
        "input_ids = torch.tensor(padded)  \n",
        "attention_mask = torch.tensor(attention_mask)\n",
        "\n",
        "with torch.no_grad():\n",
        "    last_hidden_states = model(input_ids, attention_mask=attention_mask)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "su7oNNFXce0f"
      },
      "source": [
        "np.shape(last_hidden_states[0])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "C9t60At16PVs"
      },
      "source": [
        "# Chaque token obtient un vecteur, ici, seul le token special 'cls' en position 1 nous intéresse\n",
        "features = last_hidden_states[0][:,0,:].numpy()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "JD3fX2yh6PTx"
      },
      "source": [
        "labels = batch_1[1]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "hidden": true,
        "id": "0IKS9JAJugaX"
      },
      "source": [
        "### Entrainnement d'une reg log"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "ddAqbkoU6PP9"
      },
      "source": [
        "train_features, test_features, train_labels, test_labels = train_test_split(features, labels, random_state=11)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "gG-EVWx4CzBc"
      },
      "source": [
        "lr_clf = LogisticRegression(C=1)\n",
        "lr_clf.fit(train_features, train_labels)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "hidden": true,
        "id": "iCoyxRJ7ECTA"
      },
      "source": [
        "lr_clf.score(test_features, test_labels)"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}