sample/nlp-character_embedding.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Character Embedding](https://cdn-images-1.medium.com/max/1000/0*AgFPi9E5iqGju--j.jpg)\n",
    "\n",
    "Photo Credit: https://cdn.pixabay.com/photo/2016/11/30/12/16/question-mark-1872665_960_720.jpg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Character Embedding\n",
    "In 2013, Tomas Mikolov introduced word embedding to learn a better quality of word. At that time word2vec is state of the art on dealing with text. Later on, doc2vec is introduced as well. What if we think in another angel? Instead of aggregate from word to document, is it possible to aggregate from character to word. \n",
    "\n",
    "In this article, you will go through what, why, when and how on Character Embedding\n",
    "\n",
    "**What?**\n",
    "\n",
    "Text Understanding from Scratch introduced using character CNN. In Char CNN paper, a list of character are defined 70 characters which including 26 English letters, 10 digits, 33 special characters and new line character.\n",
    "\n",
    "On the other hand, Google Brain team introduced Exploring the Limits of Language Modeling and released the lm_1b model which includes 256 vectors (including 52 characters, special characters) and the dimension is just 16. By comparing to word embedding, the dimension can increase up to 300 while the number of vectors is huge.\n",
    "\n",
    "**Why?**\n",
    "\n",
    "In english, all words are formed by 26 (or 52 if including both upper and lower case character, or even more if including special characters). Having the character embedding, every single word's vector can be formed even it is out-of-vocabulary words (optional). On the other than, word2vec embedding can only handle those seen words.\n",
    "\n",
    "Another benefit is that it good fits for misspelling words, emoticons, new words (e.g. In 2018, Oxford English Dictionary introduced new word which is boba tea 波霸奶茶. Before that we do not have any pre-trained word embedding for that).\n",
    "It handles infrequent words better than word2vec embedding as later one suffers from lack of enough training opportunity for those rare words.\n",
    "\n",
    "Third reason is that as there are only small amount of vector, it reduces model complexity and improving the performance (in terms of speed)\n",
    "\n",
    "**When?**\n",
    "\n",
    "In NLP, we can apply character embedding on:\n",
    "1. [Text Classification](https://arxiv.org/pdf/1509.01626.pdf)\n",
    "2. [Language Model](https://arxiv.org/pdf/1602.02410.pdf)\n",
    "3. [Named Entity Recognition](https://www.aclweb.org/anthology/Q16-1026)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import sklearn\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.metrics import accuracy_score, confusion_matrix\n",
    "\n",
    "# Preprocessing\n",
    "import nltk\n",
    "from nltk.tokenize import sent_tokenize\n",
    "\n",
    "# Modeling\n",
    "import keras\n",
    "from keras.models import Model, load_model\n",
    "from keras.layers import Dense, Input, Dropout, MaxPooling1D, Conv1D, GlobalMaxPool1D, Bidirectional\n",
    "from keras.layers import LSTM, Lambda, Bidirectional, concatenate, BatchNormalization, Embedding\n",
    "from keras.layers import TimeDistributed\n",
    "from keras.optimizers import Adam\n",
    "import tensorflow as tf\n",
    "import keras.backend as K\n",
    "\n",
    "import IPython\n",
    "from IPython.display import SVG\n",
    "from keras.utils.vis_utils import model_to_dot\n",
    "\n",
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Ingestion"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_raw_data():\n",
    "    # Get file from http://archive.ics.uci.edu/ml/datasets/News+Aggregator\n",
    "    df = pd.read_csv(\n",
    "        \"./../data/demo_data/NewsAggregatorDataset/newsCorpora.csv\", \n",
    "        sep='\\t', \n",
    "        names=['id', 'headline', 'url', 'publisher', 'category', 'story', 'hostname', 'timestamp']\n",
    "    )\n",
    "    \n",
    "    # Category: b = business, t = science and technology, e = entertainment, m = health\n",
    "    return df[['category', 'headline']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>category</th>\n",
       "      <th>headline</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>b</td>\n",
       "      <td>Fed official says weak data caused by weather,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>b</td>\n",
       "      <td>Fed's Charles Plosser sees high bar for change...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>b</td>\n",
       "      <td>US open: Stocks fall after Fed official hints ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>b</td>\n",
       "      <td>Fed risks falling 'behind the curve', Charles ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>b</td>\n",
       "      <td>Fed's Plosser: Nasty Weather Has Curbed Job Gr...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  category                                           headline\n",
       "0        b  Fed official says weak data caused by weather,...\n",
       "1        b  Fed's Charles Plosser sees high bar for change...\n",
       "2        b  US open: Stocks fall after Fed official hints ...\n",
       "3        b  Fed risks falling 'behind the curve', Charles ...\n",
       "4        b  Fed's Plosser: Nasty Weather Has Curbed Job Gr..."
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = get_raw_data()\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of news: 422419\n"
     ]
    }
   ],
   "source": [
    "print('Number of news: %d' % (len(df)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_df, test_df = train_test_split(df, test_size=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Modeling\n",
    "\n",
    "**_Character Embedding_**\n",
    "1. Define a list a characters (i.e. m). For example, can use alphanumeric and some special characters. For my example, English characters (52), number (10), special characters (20) and one unknown character, UNK. Total 83 characters.\n",
    "2. Transfer characters as 1-hot encoding and got a sequence for vectors. For unknown characters and blank characters, use all-zero vector to replace it. If exceeding pre-defined maximum length of characters (i.e. l), ignoring it. The output is 16 dimensions vector per every single character.\n",
    "3. Using 3 1D CNN layers (configurable) to learn the sequence\n",
    "\n",
    "**_Sentence Embedding_**\n",
    "1. Bi-directional LSTM followed CNN layers\n",
    "2. Some dropout layers are added after LSTM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class CharCNN:\n",
    "    __author__ = \"Edward Ma\"\n",
    "    __copyright__ = \"Copyright 2018, Edward Ma\"\n",
    "    __credits__ = [\"Edward Ma\"]\n",
    "    __license__ = \"Apache\"\n",
    "    __version__ = \"2.0\"\n",
    "    __maintainer__ = \"Edward Ma\"\n",
    "    __email__ = \"makcedward@gmail.com\"\n",
    "    \n",
    "    CHAR_DICT = 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789 .!?:,\\'%-\\(\\)/$|&;[]\"'\n",
    "    \n",
    "    def __init__(self, max_len_of_sentence, max_num_of_setnence, verbose=10):\n",
    "        self.max_len_of_sentence = max_len_of_sentence\n",
    "        self.max_num_of_setnence = max_num_of_setnence\n",
    "        self.verbose = verbose\n",
    "        \n",
    "        self.num_of_char = 0\n",
    "        self.num_of_label = 0\n",
    "        self.unknown_label = ''\n",
    "        \n",
    "    def build_char_dictionary(self, char_dict=None, unknown_label='UNK'):\n",
    "        \"\"\"\n",
    "            Define possbile char set. Using \"UNK\" if character does not exist in this set\n",
    "        \"\"\" \n",
    "        \n",
    "        if char_dict is None:\n",
    "            char_dict = self.CHAR_DICT\n",
    "            \n",
    "        self.unknown_label = unknown_label\n",
    "\n",
    "        chars = []\n",
    "\n",
    "        for c in char_dict:\n",
    "            chars.append(c)\n",
    "\n",
    "        chars = list(set(chars))\n",
    "        \n",
    "        chars.insert(0, unknown_label)\n",
    "\n",
    "        self.num_of_char = len(chars)\n",
    "        self.char_indices = dict((c, i) for i, c in enumerate(chars))\n",
    "        self.indices_char = dict((i, c) for i, c in enumerate(chars))\n",
    "        \n",
    "        if self.verbose > 5:\n",
    "            print('Totoal number of chars:', self.num_of_char)\n",
    "\n",
    "            print('First 3 char_indices sample:', {k: self.char_indices[k] for k in list(self.char_indices)[:3]})\n",
    "            print('First 3 indices_char sample:', {k: self.indices_char[k] for k in list(self.indices_char)[:3]})\n",
    "            \n",
    "\n",
    "        return self.char_indices, self.indices_char, self.num_of_char\n",
    "    \n",
    "    def convert_labels(self, labels):\n",
    "        \"\"\"\n",
    "            Convert label to numeric\n",
    "        \"\"\"\n",
    "        self.label2indexes = dict((l, i) for i, l in enumerate(labels))\n",
    "        self.index2labels = dict((i, l) for i, l in enumerate(labels))\n",
    "\n",
    "        if self.verbose > 5:\n",
    "            print('Label to Index: ', self.label2indexes)\n",
    "            print('Index to Label: ', self.index2labels)\n",
    "            \n",
    "        self.num_of_label = len(self.label2indexes)\n",
    "\n",
    "        return self.label2indexes, self.index2labels\n",
    "    \n",
    "    def _transform_raw_data(self, df, x_col, y_col, label2indexes=None, sample_size=None):\n",
    "        \"\"\"\n",
    "            ##### Transform raw data to list\n",
    "        \"\"\"\n",
    "        \n",
    "        x = []\n",
    "        y = []\n",
    "\n",
    "        actual_max_sentence = 0\n",
    "        \n",
    "        if sample_size is None:\n",
    "            sample_size = len(df)\n",
    "\n",
    "        for i, row in df.head(sample_size).iterrows():\n",
    "            x_data = row[x_col]\n",
    "            y_data = row[y_col]\n",
    "\n",
    "            sentences = sent_tokenize(x_data)\n",
    "            x.append(sentences)\n",
    "\n",
    "            if len(sentences) > actual_max_sentence:\n",
    "                actual_max_sentence = len(sentences)\n",
    "\n",
    "            y.append(label2indexes[y_data])\n",
    "\n",
    "        if self.verbose > 5:\n",
    "            print('Number of news: %d' % (len(x)))\n",
    "            print('Actual max sentence: %d' % actual_max_sentence)\n",
    "\n",
    "        return x, y\n",
    "    \n",
    "    def _transform_training_data(self, x_raw, y_raw, max_len_of_sentence=None, max_num_of_setnence=None):\n",
    "        \"\"\"\n",
    "            ##### Transform preorcessed data to numpy\n",
    "        \"\"\"\n",
    "        unknown_value = self.char_indices[self.unknown_label]\n",
    "        \n",
    "        x = np.ones((len(x_raw), max_num_of_setnence, max_len_of_sentence), dtype=np.int64) * unknown_value\n",
    "        y = np.array(y_raw)\n",
    "        \n",
    "        if max_len_of_sentence is None:\n",
    "            max_len_of_sentence = self.max_len_of_sentence\n",
    "        if max_num_of_setnence is None:\n",
    "            max_num_of_setnence = self.max_num_of_setnence\n",
    "\n",
    "        for i, doc in enumerate(x_raw):\n",
    "            for j, sentence in enumerate(doc):\n",
    "                if j < max_num_of_setnence:\n",
    "                    for t, char in enumerate(sentence[-max_len_of_sentence:]):\n",
    "                        if char not in self.char_indices:\n",
    "                            x[i, j, (max_len_of_sentence-1-t)] = self.char_indices['UNK']\n",
    "                        else:\n",
    "                            x[i, j, (max_len_of_sentence-1-t)] = self.char_indices[char]\n",
    "\n",
    "        return x, y\n",
    "\n",
    "    def _build_character_block(self, block, dropout=0.3, filters=[64, 100], kernel_size=[3, 3], \n",
    "                         pool_size=[2, 2], padding='valid', activation='relu', \n",
    "                         kernel_initializer='glorot_normal'):\n",
    "        \n",
    "        for i in range(len(filters)):\n",
    "            block = Conv1D(\n",
    "                filters=filters[i], kernel_size=kernel_size[i],\n",
    "                padding=padding, activation=activation, kernel_initializer=kernel_initializer)(block)\n",
    "\n",
    "        block = Dropout(dropout)(block)\n",
    "        block = MaxPooling1D(pool_size=pool_size[i])(block)\n",
    "\n",
    "        block = GlobalMaxPool1D()(block)\n",
    "        block = Dense(128, activation='relu')(block)\n",
    "        return block\n",
    "    \n",
    "    def _build_sentence_block(self, max_len_of_sentence, max_num_of_setnence, \n",
    "                              char_dimension=16,\n",
    "                              filters=[[3, 5, 7], [200, 300, 300], [300, 400, 400]], \n",
    "#                               filters=[[100, 200, 200], [200, 300, 300], [300, 400, 400]], \n",
    "                              kernel_sizes=[[4, 3, 3], [5, 3, 3], [6, 3, 3]], \n",
    "                              pool_sizes=[[2, 2, 2], [2, 2, 2], [2, 2, 2]],\n",
    "                              dropout=0.4):\n",
    "        \n",
    "        sent_input = Input(shape=(max_len_of_sentence, ), dtype='int64')\n",
    "        embedded = Embedding(self.num_of_char, char_dimension, input_length=max_len_of_sentence)(sent_input)\n",
    "        \n",
    "        blocks = []\n",
    "        for i, filter_layers in enumerate(filters):\n",
    "            blocks.append(\n",
    "                self._build_character_block(\n",
    "                    block=embedded, filters=filters[i], kernel_size=kernel_sizes[i], pool_size=pool_sizes[i])\n",
    "            )\n",
    "\n",
    "        sent_output = concatenate(blocks, axis=-1)\n",
    "        sent_output = Dropout(dropout)(sent_output)\n",
    "        sent_encoder = Model(inputs=sent_input, outputs=sent_output)\n",
    "\n",
    "        return sent_encoder\n",
    "    \n",
    "    def _build_document_block(self, sent_encoder, max_len_of_sentence, max_num_of_setnence, \n",
    "                             num_of_label, dropout=0.3, \n",
    "                             loss='sparse_categorical_crossentropy', optimizer='rmsprop', metrics=['accuracy']):\n",
    "        doc_input = Input(shape=(max_num_of_setnence, max_len_of_sentence), dtype='int64')\n",
    "        doc_output = TimeDistributed(sent_encoder)(doc_input)\n",
    "\n",
    "        doc_output = Bidirectional(LSTM(128, return_sequences=False, dropout=dropout))(doc_output)\n",
    "\n",
    "        doc_output = Dropout(dropout)(doc_output)\n",
    "        doc_output = Dense(128, activation='relu')(doc_output)\n",
    "        doc_output = Dropout(dropout)(doc_output)\n",
    "        doc_output = Dense(num_of_label, activation='sigmoid')(doc_output)\n",
    "\n",
    "        doc_encoder = Model(inputs=doc_input, outputs=doc_output)\n",
    "        doc_encoder.compile(loss=loss, optimizer=optimizer, metrics=metrics)\n",
    "        return doc_encoder\n",
    "    \n",
    "    def preporcess(self, labels, char_dict=None, unknown_label='UNK'):\n",
    "        if self.verbose > 3:\n",
    "            print('-----> Stage: preprocess')\n",
    "            \n",
    "        self.build_char_dictionary(char_dict, unknown_label)\n",
    "        self.convert_labels(labels)\n",
    "    \n",
    "    def process(self, df, x_col, y_col, \n",
    "                max_len_of_sentence=None, max_num_of_setnence=None, label2indexes=None, sample_size=None):\n",
    "        if self.verbose > 3:\n",
    "            print('-----> Stage: process')\n",
    "            \n",
    "        if sample_size is None:\n",
    "            sample_size = 1000\n",
    "        if label2indexes is None:\n",
    "            if self.label2indexes is None:\n",
    "                raise Exception('Does not initalize label2indexes. Please invoke preprocess step first')\n",
    "            label2indexes = self.label2indexes\n",
    "        if max_len_of_sentence is None:\n",
    "            max_len_of_sentence = self.max_len_of_sentence\n",
    "        if max_num_of_setnence is None:\n",
    "            max_num_of_setnence = self.max_num_of_setnence\n",
    "\n",
    "        x_preprocess, y_preprocess = self._transform_raw_data(\n",
    "            df=df, x_col=x_col, y_col=y_col, label2indexes=label2indexes)\n",
    "        \n",
    "        x_preprocess, y_preprocess = self._transform_training_data(\n",
    "            x_raw=x_preprocess, y_raw=y_preprocess,\n",
    "            max_len_of_sentence=max_len_of_sentence, max_num_of_setnence=max_num_of_setnence)\n",
    "        \n",
    "        if self.verbose > 5:\n",
    "            print('Shape: ', x_preprocess.shape, y_preprocess.shape)\n",
    "\n",
    "        return x_preprocess, y_preprocess\n",
    "    \n",
    "    def build_model(self, char_dimension=16, display_summary=False, display_architecture=False, \n",
    "                    loss='sparse_categorical_crossentropy', optimizer='rmsprop', metrics=['accuracy']):\n",
    "        if self.verbose > 3:\n",
    "            print('-----> Stage: build model')\n",
    "            \n",
    "        sent_encoder = self._build_sentence_block(\n",
    "            char_dimension=char_dimension,\n",
    "            max_len_of_sentence=self.max_len_of_sentence, max_num_of_setnence=self.max_num_of_setnence)\n",
    "                \n",
    "        doc_encoder = self._build_document_block(\n",
    "            sent_encoder=sent_encoder, num_of_label=self.num_of_label,\n",
    "            max_len_of_sentence=self.max_len_of_sentence, max_num_of_setnence=self.max_num_of_setnence, \n",
    "            loss=loss, optimizer=optimizer, metrics=metrics)\n",
    "        \n",
    "        if display_architecture:\n",
    "            print('Sentence Architecture')\n",
    "            IPython.display.display(SVG(model_to_dot(sent_encoder).create(prog='dot', format='svg')))\n",
    "            print()\n",
    "            print('Document Architecture')\n",
    "            IPython.display.display(SVG(model_to_dot(doc_encoder).create(prog='dot', format='svg')))\n",
    "        \n",
    "        if display_summary:\n",
    "            print(doc_encoder.summary())\n",
    "            \n",
    "        \n",
    "        self.model = {\n",
    "            'sent_encoder': sent_encoder,\n",
    "            'doc_encoder': doc_encoder\n",
    "        }\n",
    "        \n",
    "        return doc_encoder\n",
    "    \n",
    "    def train(self, x_train, y_train, x_test, y_test, batch_size=128, epochs=1, shuffle=True):\n",
    "        if self.verbose > 3:\n",
    "            print('-----> Stage: train model')\n",
    "            \n",
    "        self.get_model().fit(\n",
    "            x_train, y_train, validation_data=(x_test, y_test), \n",
    "            batch_size=batch_size, epochs=epochs, shuffle=shuffle)\n",
    "        \n",
    "#         return self.model['doc_encoder']\n",
    "\n",
    "    def predict(self, x, return_prob=False):\n",
    "        if self.verbose > 3:\n",
    "            print('-----> Stage: predict')\n",
    "            \n",
    "        if return_prob:\n",
    "            return self.get_model().predict(x_test)\n",
    "        \n",
    "        return self.get_model().predict(x_test).argmax(axis=-1)\n",
    "    \n",
    "    def get_model(self):\n",
    "        return self.model['doc_encoder']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-----> Stage: preprocess\n",
      "Totoal number of chars: 83\n",
      "First 3 char_indices sample: {'u': 1, ';': 2, 'h': 3}\n",
      "First 3 indices_char sample: {0: 'UNK', 1: 'u', 2: ';'}\n",
      "Label to Index:  {'m': 3, 'e': 2, 'b': 0, 't': 1}\n",
      "Index to Label:  {0: 'b', 1: 't', 2: 'e', 3: 'm'}\n",
      "-----> Stage: process\n",
      "Number of news: 337935\n",
      "Actual max sentence: 15\n",
      "unknown_value: 0\n",
      "Shape:  (337935, 5, 256) (337935,)\n",
      "-----> Stage: process\n",
      "Number of news: 84484\n",
      "Actual max sentence: 12\n",
      "unknown_value: 0\n",
      "Shape:  (84484, 5, 256) (84484,)\n",
      "-----> Stage: build model\n",
      "-----> Stage: train model\n",
      "Train on 337935 samples, validate on 84484 samples\n",
      "Epoch 1/10\n",
      "337935/337935 [==============================] - 11080s 33ms/step - loss: 1.2129 - acc: 0.4373 - val_loss: 1.1089 - val_acc: 0.4961\n",
      "Epoch 2/10\n",
      "337935/337935 [==============================] - 11040s 33ms/step - loss: 1.0237 - acc: 0.5594 - val_loss: 0.9715 - val_acc: 0.5940\n",
      "Epoch 3/10\n",
      "337935/337935 [==============================] - 11017s 33ms/step - loss: 0.7820 - acc: 0.6824 - val_loss: 0.7290 - val_acc: 0.7149\n",
      "Epoch 4/10\n",
      "337935/337935 [==============================] - 11025s 33ms/step - loss: 0.5948 - acc: 0.7741 - val_loss: 0.5955 - val_acc: 0.7756\n",
      "Epoch 5/10\n",
      "337935/337935 [==============================] - 11002s 33ms/step - loss: 0.4538 - acc: 0.8378 - val_loss: 0.4922 - val_acc: 0.8052\n",
      "Epoch 6/10\n",
      "337935/337935 [==============================] - 11050s 33ms/step - loss: 0.3666 - acc: 0.8715 - val_loss: 0.4280 - val_acc: 0.8546\n",
      "Epoch 7/10\n",
      "337935/337935 [==============================] - 11017s 33ms/step - loss: 0.3070 - acc: 0.8940 - val_loss: 0.4630 - val_acc: 0.8226\n",
      "Epoch 8/10\n",
      "337935/337935 [==============================] - 11012s 33ms/step - loss: 0.2666 - acc: 0.9083 - val_loss: 0.3481 - val_acc: 0.8803\n",
      "Epoch 9/10\n",
      "337935/337935 [==============================] - 11034s 33ms/step - loss: 0.2373 - acc: 0.9188 - val_loss: 0.2728 - val_acc: 0.9069\n",
      "Epoch 10/10\n",
      "337935/337935 [==============================] - 11014s 33ms/step - loss: 0.2136 - acc: 0.9272 - val_loss: 0.2499 - val_acc: 0.9110\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "    Maximum number of characters per sentence is 256.\n",
    "    Maximum number of sentence is 5\n",
    "\"\"\"\n",
    "\n",
    "char_cnn = CharCNN(max_len_of_sentence=256, max_num_of_setnence=5)\n",
    "\n",
    "\"\"\"\n",
    "    First of all, we need to prepare meta information including character dictionary \n",
    "    and converting label from text to numeric (as keras support numeric input only).\n",
    "\"\"\"\n",
    "char_cnn.preporcess(labels=df['category'].unique())\n",
    "\n",
    "\"\"\"\n",
    "    We have to transform raw input training data and testing to numpy format for keras input\n",
    "\"\"\"\n",
    "x_train, y_train = char_cnn.process(\n",
    "    df=train_df, x_col='headline', y_col='category')\n",
    "x_test, y_test = char_cnn.process(\n",
    "    df=test_df, x_col='headline', y_col='category')\n",
    "\n",
    "char_cnn.build_model()\n",
    "char_cnn.train(x_train, y_train, x_test, y_test, batch_size=2048, epochs=10)\n",
    "\n",
    "char_cnn.get_model().save('./char_cnn_model.h5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[  9.91751850e-01,   2.97714826e-02,   1.48440502e-03,\n",
       "          6.78197597e-04],\n",
       "       [  4.47248220e-01,   9.31121230e-01,   6.35791337e-03,\n",
       "          2.18354538e-03],\n",
       "       [  9.97535825e-01,   3.43346526e-03,   5.12215891e-04,\n",
       "          1.37654960e-03],\n",
       "       ..., \n",
       "       [  1.38067767e-01,   9.78998482e-01,   1.27230678e-02,\n",
       "          3.33724148e-03],\n",
       "       [  1.94361171e-04,   2.12071711e-04,   9.99878764e-01,\n",
       "          9.05996567e-05],\n",
       "       [  8.41215439e-03,   1.44472681e-02,   9.66172934e-01,\n",
       "          1.35373905e-01]], dtype=float32)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "char_cnn_model_loaded = load_model('./char_cnn_model.h5')\n",
    "char_cnn_model_loaded.predict(x_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusion\n",
    "Character Embedding is a brilliant design for solving lots of text classification. It resolved some word embedding. FAIR did a further step. They introduced to use subword embedding to build fastText. \n",
    "\n",
    "This is some comment on Character Embedding as it does not include any word meaning but just using characters. We may include both Character Embedding and Word Embedding together to solve our NLP problem."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}