ML Pipeline Preparation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ML Pipeline Preparation\n",
    "Follow the instructions below to help you create your ML pipeline.\n",
    "### 1. Import libraries and load data from database.\n",
    "- Import Python libraries\n",
    "- Load dataset from database with [`read_sql_table`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_sql_table.html)\n",
    "- Define feature and target variables X and Y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package wordnet to\n",
      "[nltk_data]     C:\\Users\\ahmed\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package wordnet is already up-to-date!\n",
      "[nltk_data] Downloading package punkt to\n",
      "[nltk_data]     C:\\Users\\ahmed\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n",
      "[nltk_data] Downloading package stopwords to\n",
      "[nltk_data]     C:\\Users\\ahmed\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# import libraries\n",
    "import re\n",
    "import pickle\n",
    "import pandas as pd \n",
    "from sqlalchemy import create_engine\n",
    "import nltk \n",
    "from nltk.tokenize import word_tokenize \n",
    "from nltk.corpus import stopwords \n",
    "from nltk.stem import WordNetLemmatizer\n",
    "from sklearn.pipeline import Pipeline \n",
    "from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer\n",
    "from sklearn.multioutput import MultiOutputClassifier\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn.ensemble import RandomForestClassifier, AdaBoostClassifier\n",
    "from sklearn.model_selection import train_test_split,  GridSearchCV \n",
    "from sklearn.metrics import classification_report\n",
    "\n",
    "\n",
    "nltk.download(['wordnet', 'punkt', 'stopwords'])\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load data from database\n",
    "\n",
    "engine = create_engine('sqlite:///data//InsertDatabaseName.db')\n",
    "df = pd.read_sql_table('InsertTableName',engine)\n",
    "\n",
    "X = df['message']  # Message Column\n",
    "Y = df.iloc[:, 4:] # Classification label"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Write a tokenization function to process your text data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tokenize(text):\n",
    "    \"\"\"\n",
    "    Function: split text into words and return the root form of the words\n",
    "    Args:\n",
    "      text(str): the message\n",
    "    Return:\n",
    "      lemm(list of str): a list of the root form of the message words\n",
    "    \"\"\"\n",
    "    # Normalize text\n",
    "    text = re.sub(r\"[^a-zA-Z0-9]\", \" \", text.lower())\n",
    "    \n",
    "    # Tokenize text\n",
    "    words = word_tokenize(text)\n",
    "    \n",
    "    # Remove stop words\n",
    "    stop = stopwords.words(\"english\")\n",
    "    words = [t for t in words if t not in stop]\n",
    "    \n",
    "    # Lemmatization\n",
    "    lemm = [WordNetLemmatizer().lemmatize(w) for w in words]\n",
    "    return lemm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Build a machine learning pipeline\n",
    "This machine pipeline should take in the `message` column as input and output classification results on the other 36 categories in the dataset. You may find the [MultiOutputClassifier](http://scikit-learn.org/stable/modules/generated/sklearn.multioutput.MultiOutputClassifier.html) helpful for predicting multiple target variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Pipleine 1: Random Forest Classifier\n",
    "\n",
    "pipeline_rfc = Pipeline([\n",
    "        ('vect', CountVectorizer(tokenizer = tokenize)),\n",
    "        ('tfidf', TfidfTransformer()),\n",
    "        ('clf',  MultiOutputClassifier(RandomForestClassifier()))\n",
    "    ])\n",
    "\n",
    "\n",
    "# Pipeline 2: Naive Bayes classifier\n",
    "\n",
    "pipeline_nbc = Pipeline([\n",
    "        ('vect', CountVectorizer(tokenizer = tokenize)),\n",
    "        ('tfidf', TfidfTransformer()),\n",
    "        ('clf',  MultiOutputClassifier(MultinomialNB()))\n",
    "    ])\n",
    "\n",
    "\n",
    "# Pipleine 3: Adaboost Classifier \n",
    "\n",
    "pipeline_ada = Pipeline([\n",
    "        ('vect', CountVectorizer(tokenizer = tokenize)),\n",
    "        ('tfidf', TfidfTransformer()),\n",
    "        ('clf',  MultiOutputClassifier(AdaBoostClassifier()))\n",
    "    ])\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Train pipeline\n",
    "- Split data into train and test sets\n",
    "- Train pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Spiliting data\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, Y)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\ahmed\\Anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:246: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\ahmed\\Anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:246: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\ahmed\\Anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:246: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\ahmed\\Anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:246: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\ahmed\\Anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:246: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\ahmed\\Anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:246: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\ahmed\\Anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:246: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\ahmed\\Anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:246: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\ahmed\\Anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:246: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\ahmed\\Anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:246: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\ahmed\\Anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:246: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\ahmed\\Anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:246: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\ahmed\\Anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:246: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\ahmed\\Anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:246: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\ahmed\\Anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:246: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\ahmed\\Anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:246: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\ahmed\\Anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:246: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\ahmed\\Anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:246: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\ahmed\\Anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:246: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\ahmed\\Anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:246: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\ahmed\\Anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:246: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\ahmed\\Anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:246: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\ahmed\\Anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:246: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\ahmed\\Anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:246: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\ahmed\\Anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:246: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\ahmed\\Anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:246: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\ahmed\\Anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:246: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\ahmed\\Anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:246: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\ahmed\\Anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:246: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\ahmed\\Anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:246: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\ahmed\\Anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:246: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\ahmed\\Anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:246: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\ahmed\\Anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:246: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\ahmed\\Anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:246: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\ahmed\\Anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:246: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\ahmed\\Anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:246: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "     steps=[('vect', CountVectorizer(analyzer='word', binary=False, decode_error='strict',\n",
       "        dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',\n",
       "        lowercase=True, max_df=1.0, max_features=None, min_df=1,\n",
       "        ngram_range=(1, 1), preprocessor=None, stop_words=None,\n",
       "        strip..._score=False, random_state=None, verbose=0,\n",
       "            warm_start=False),\n",
       "           n_jobs=None))])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Fit the Random Forest Classifier \n",
    "pipeline_rfc.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\ahmed\\Anaconda3\\lib\\site-packages\\sklearn\\naive_bayes.py:465: RuntimeWarning: divide by zero encountered in log\n",
      "  self.class_log_prior_ = (np.log(self.class_count_) -\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "     steps=[('vect', CountVectorizer(analyzer='word', binary=False, decode_error='strict',\n",
       "        dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',\n",
       "        lowercase=True, max_df=1.0, max_features=None, min_df=1,\n",
       "        ngram_range=(1, 1), preprocessor=None, stop_words=None,\n",
       "        strip...fier(estimator=MultinomialNB(alpha=1.0, class_prior=None, fit_prior=True),\n",
       "           n_jobs=None))])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Fit the Naive Bayes classifier \n",
    "pipeline_nbc.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "     steps=[('vect', CountVectorizer(analyzer='word', binary=False, decode_error='strict',\n",
       "        dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',\n",
       "        lowercase=True, max_df=1.0, max_features=None, min_df=1,\n",
       "        ngram_range=(1, 1), preprocessor=None, stop_words=None,\n",
       "        strip...or=None,\n",
       "          learning_rate=1.0, n_estimators=50, random_state=None),\n",
       "           n_jobs=None))])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Fit the Adaboost Classifier \n",
    "pipeline_ada.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Test your model\n",
    "Report the f1 score, precision and recall for each output category of the dataset. You can do this by iterating through the columns and calling sklearn's `classification_report` on each."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_scores(y_test, y_pred):\n",
    "    #Testing the model\n",
    "    # Printing the classification report for each label\n",
    "    i = 0\n",
    "    for col in y_test:\n",
    "        print('Feature {}: {}'.format(i+1, col))\n",
    "        print(classification_report(y_test[col], y_pred[:, i]))\n",
    "        i = i + 1\n",
    "    accuracy = (y_pred == y_test.values).mean()\n",
    "    print('The model accuracy is {:.3f}'.format(accuracy))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Feature 1: related\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.62      0.45      0.52      1555\n",
      "           1       0.84      0.90      0.87      4939\n",
      "           2       0.25      0.42      0.31        60\n",
      "\n",
      "   micro avg       0.79      0.79      0.79      6554\n",
      "   macro avg       0.57      0.59      0.57      6554\n",
      "weighted avg       0.78      0.79      0.78      6554\n",
      "\n",
      "Feature 2: request\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.89      0.98      0.94      5463\n",
      "           1       0.82      0.42      0.55      1091\n",
      "\n",
      "   micro avg       0.89      0.89      0.89      6554\n",
      "   macro avg       0.85      0.70      0.74      6554\n",
      "weighted avg       0.88      0.89      0.87      6554\n",
      "\n",
      "Feature 3: offer\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00      6527\n",
      "           1       0.00      0.00      0.00        27\n",
      "\n",
      "   micro avg       1.00      1.00      1.00      6554\n",
      "   macro avg       0.50      0.50      0.50      6554\n",
      "weighted avg       0.99      1.00      0.99      6554\n",
      "\n",
      "Feature 4: aid_related\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.76      0.86      0.80      3881\n",
      "           1       0.74      0.60      0.66      2673\n",
      "\n",
      "   micro avg       0.75      0.75      0.75      6554\n",
      "   macro avg       0.75      0.73      0.73      6554\n",
      "weighted avg       0.75      0.75      0.75      6554\n",
      "\n",
      "Feature 5: medical_help\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.93      0.99      0.96      6084\n",
      "           1       0.53      0.11      0.18       470\n",
      "\n",
      "   micro avg       0.93      0.93      0.93      6554\n",
      "   macro avg       0.73      0.55      0.57      6554\n",
      "weighted avg       0.91      0.93      0.91      6554\n",
      "\n",
      "Feature 6: medical_products\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      1.00      0.98      6253\n",
      "           1       0.68      0.10      0.17       301\n",
      "\n",
      "   micro avg       0.96      0.96      0.96      6554\n",
      "   macro avg       0.82      0.55      0.58      6554\n",
      "weighted avg       0.95      0.96      0.94      6554\n",
      "\n",
      "Feature 7: search_and_rescue\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.97      1.00      0.99      6372\n",
      "           1       0.50      0.04      0.07       182\n",
      "\n",
      "   micro avg       0.97      0.97      0.97      6554\n",
      "   macro avg       0.74      0.52      0.53      6554\n",
      "weighted avg       0.96      0.97      0.96      6554\n",
      "\n",
      "Feature 8: security\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      1.00      0.99      6441\n",
      "           1       0.00      0.00      0.00       113\n",
      "\n",
      "   micro avg       0.98      0.98      0.98      6554\n",
      "   macro avg       0.49      0.50      0.50      6554\n",
      "weighted avg       0.97      0.98      0.97      6554\n",
      "\n",
      "Feature 9: military\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.97      1.00      0.98      6326\n",
      "           1       0.68      0.06      0.11       228\n",
      "\n",
      "   micro avg       0.97      0.97      0.97      6554\n",
      "   macro avg       0.83      0.53      0.54      6554\n",
      "weighted avg       0.96      0.97      0.95      6554\n",
      "\n",
      "Feature 10: child_alone\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00      6554\n",
      "\n",
      "   micro avg       1.00      1.00      1.00      6554\n",
      "   macro avg       1.00      1.00      1.00      6554\n",
      "weighted avg       1.00      1.00      1.00      6554\n",
      "\n",
      "Feature 11: water\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      0.99      0.98      6153\n",
      "           1       0.82      0.42      0.55       401\n",
      "\n",
      "   micro avg       0.96      0.96      0.96      6554\n",
      "   macro avg       0.89      0.71      0.77      6554\n",
      "weighted avg       0.95      0.96      0.95      6554\n",
      "\n",
      "Feature 12: food\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.92      0.99      0.96      5813\n",
      "           1       0.85      0.35      0.49       741\n",
      "\n",
      "   micro avg       0.92      0.92      0.92      6554\n",
      "   macro avg       0.89      0.67      0.73      6554\n",
      "weighted avg       0.91      0.92      0.90      6554\n",
      "\n",
      "Feature 13: shelter\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.94      0.99      0.97      6006\n",
      "           1       0.83      0.31      0.45       548\n",
      "\n",
      "   micro avg       0.94      0.94      0.94      6554\n",
      "   macro avg       0.88      0.65      0.71      6554\n",
      "weighted avg       0.93      0.94      0.92      6554\n",
      "\n",
      "Feature 14: clothing\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      0.99      6453\n",
      "           1       0.62      0.08      0.14       101\n",
      "\n",
      "   micro avg       0.99      0.99      0.99      6554\n",
      "   macro avg       0.80      0.54      0.57      6554\n",
      "weighted avg       0.98      0.99      0.98      6554\n",
      "\n",
      "Feature 15: money\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      1.00      0.99      6393\n",
      "           1       0.85      0.07      0.13       161\n",
      "\n",
      "   micro avg       0.98      0.98      0.98      6554\n",
      "   macro avg       0.91      0.53      0.56      6554\n",
      "weighted avg       0.97      0.98      0.97      6554\n",
      "\n",
      "Feature 16: missing_people\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      0.99      6483\n",
      "           1       0.50      0.03      0.05        71\n",
      "\n",
      "   micro avg       0.99      0.99      0.99      6554\n",
      "   macro avg       0.74      0.51      0.52      6554\n",
      "weighted avg       0.98      0.99      0.98      6554\n",
      "\n",
      "Feature 17: refugees\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.97      1.00      0.98      6344\n",
      "           1       0.53      0.04      0.08       210\n",
      "\n",
      "   micro avg       0.97      0.97      0.97      6554\n",
      "   macro avg       0.75      0.52      0.53      6554\n",
      "weighted avg       0.96      0.97      0.95      6554\n",
      "\n",
      "Feature 18: death\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.97      1.00      0.98      6283\n",
      "           1       0.65      0.17      0.26       271\n",
      "\n",
      "   micro avg       0.96      0.96      0.96      6554\n",
      "   macro avg       0.81      0.58      0.62      6554\n",
      "weighted avg       0.95      0.96      0.95      6554\n",
      "\n",
      "Feature 19: other_aid\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.88      0.99      0.93      5713\n",
      "           1       0.52      0.07      0.12       841\n",
      "\n",
      "   micro avg       0.87      0.87      0.87      6554\n",
      "   macro avg       0.70      0.53      0.52      6554\n",
      "weighted avg       0.83      0.87      0.83      6554\n",
      "\n",
      "Feature 20: infrastructure_related\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.94      1.00      0.97      6137\n",
      "           1       0.17      0.00      0.01       417\n",
      "\n",
      "   micro avg       0.94      0.94      0.94      6554\n",
      "   macro avg       0.55      0.50      0.49      6554\n",
      "weighted avg       0.89      0.94      0.91      6554\n",
      "\n",
      "Feature 21: transport\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      1.00      0.98      6258\n",
      "           1       0.56      0.10      0.18       296\n",
      "\n",
      "   micro avg       0.96      0.96      0.96      6554\n",
      "   macro avg       0.76      0.55      0.58      6554\n",
      "weighted avg       0.94      0.96      0.94      6554\n",
      "\n",
      "Feature 22: buildings\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      1.00      0.98      6233\n",
      "           1       0.64      0.14      0.23       321\n",
      "\n",
      "   micro avg       0.95      0.95      0.95      6554\n",
      "   macro avg       0.80      0.57      0.60      6554\n",
      "weighted avg       0.94      0.95      0.94      6554\n",
      "\n",
      "Feature 23: electricity\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      1.00      0.99      6420\n",
      "           1       1.00      0.06      0.11       134\n",
      "\n",
      "   micro avg       0.98      0.98      0.98      6554\n",
      "   macro avg       0.99      0.53      0.55      6554\n",
      "weighted avg       0.98      0.98      0.97      6554\n",
      "\n",
      "Feature 24: tools\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00      6524\n",
      "           1       0.00      0.00      0.00        30\n",
      "\n",
      "   micro avg       1.00      1.00      1.00      6554\n",
      "   macro avg       0.50      0.50      0.50      6554\n",
      "weighted avg       0.99      1.00      0.99      6554\n",
      "\n",
      "Feature 25: hospitals\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      1.00      6496\n",
      "           1       0.00      0.00      0.00        58\n",
      "\n",
      "   micro avg       0.99      0.99      0.99      6554\n",
      "   macro avg       0.50      0.50      0.50      6554\n",
      "weighted avg       0.98      0.99      0.99      6554\n",
      "\n",
      "Feature 26: shops\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00      6523\n",
      "           1       0.00      0.00      0.00        31\n",
      "\n",
      "   micro avg       1.00      1.00      1.00      6554\n",
      "   macro avg       0.50      0.50      0.50      6554\n",
      "weighted avg       0.99      1.00      0.99      6554\n",
      "\n",
      "Feature 27: aid_centers\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      0.99      6483\n",
      "           1       0.00      0.00      0.00        71\n",
      "\n",
      "   micro avg       0.99      0.99      0.99      6554\n",
      "   macro avg       0.49      0.50      0.50      6554\n",
      "weighted avg       0.98      0.99      0.98      6554\n",
      "\n",
      "Feature 28: other_infrastructure\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      1.00      0.98      6254\n",
      "           1       0.17      0.00      0.01       300\n",
      "\n",
      "   micro avg       0.95      0.95      0.95      6554\n",
      "   macro avg       0.56      0.50      0.49      6554\n",
      "weighted avg       0.92      0.95      0.93      6554\n",
      "\n",
      "Feature 29: weather_related\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.88      0.96      0.91      4773\n",
      "           1       0.84      0.63      0.72      1781\n",
      "\n",
      "   micro avg       0.87      0.87      0.87      6554\n",
      "   macro avg       0.86      0.80      0.82      6554\n",
      "weighted avg       0.87      0.87      0.86      6554\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\ahmed\\Anaconda3\\lib\\site-packages\\sklearn\\metrics\\classification.py:1143: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Feature 30: floods\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      1.00      0.97      6010\n",
      "           1       0.88      0.36      0.51       544\n",
      "\n",
      "   micro avg       0.94      0.94      0.94      6554\n",
      "   macro avg       0.91      0.68      0.74      6554\n",
      "weighted avg       0.94      0.94      0.93      6554\n",
      "\n",
      "Feature 31: storm\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.94      0.98      0.96      5946\n",
      "           1       0.70      0.39      0.50       608\n",
      "\n",
      "   micro avg       0.93      0.93      0.93      6554\n",
      "   macro avg       0.82      0.69      0.73      6554\n",
      "weighted avg       0.92      0.93      0.92      6554\n",
      "\n",
      "Feature 32: fire\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      1.00      6496\n",
      "           1       0.00      0.00      0.00        58\n",
      "\n",
      "   micro avg       0.99      0.99      0.99      6554\n",
      "   macro avg       0.50      0.50      0.50      6554\n",
      "weighted avg       0.98      0.99      0.99      6554\n",
      "\n",
      "Feature 33: earthquake\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.97      0.99      0.98      5975\n",
      "           1       0.91      0.69      0.79       579\n",
      "\n",
      "   micro avg       0.97      0.97      0.97      6554\n",
      "   macro avg       0.94      0.84      0.88      6554\n",
      "weighted avg       0.97      0.97      0.96      6554\n",
      "\n",
      "Feature 34: cold\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      1.00      0.99      6439\n",
      "           1       0.64      0.06      0.11       115\n",
      "\n",
      "   micro avg       0.98      0.98      0.98      6554\n",
      "   macro avg       0.81      0.53      0.55      6554\n",
      "weighted avg       0.98      0.98      0.98      6554\n",
      "\n",
      "Feature 35: other_weather\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      1.00      0.97      6202\n",
      "           1       0.52      0.03      0.06       352\n",
      "\n",
      "   micro avg       0.95      0.95      0.95      6554\n",
      "   macro avg       0.73      0.52      0.52      6554\n",
      "weighted avg       0.93      0.95      0.92      6554\n",
      "\n",
      "Feature 36: direct_report\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.87      0.97      0.92      5324\n",
      "           1       0.75      0.35      0.48      1230\n",
      "\n",
      "   micro avg       0.86      0.86      0.86      6554\n",
      "   macro avg       0.81      0.66      0.70      6554\n",
      "weighted avg       0.84      0.86      0.84      6554\n",
      "\n",
      "The model accuracy is 0.945\n"
     ]
    }
   ],
   "source": [
    "# Prediction: the Random Forest Classifier  \n",
    "y_pred = pipeline_rfc.predict(X_test)\n",
    "plot_scores(y_test, y_pred)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Feature 1: related\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.76      0.09      0.17      1555\n",
      "           1       0.77      0.99      0.87      4939\n",
      "           2       0.00      0.00      0.00        60\n",
      "\n",
      "   micro avg       0.77      0.77      0.77      6554\n",
      "   macro avg       0.51      0.36      0.34      6554\n",
      "weighted avg       0.76      0.77      0.69      6554\n",
      "\n",
      "Feature 2: request\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.87      0.99      0.92      5463\n",
      "           1       0.84      0.23      0.36      1091\n",
      "\n",
      "   micro avg       0.86      0.86      0.86      6554\n",
      "   macro avg       0.85      0.61      0.64      6554\n",
      "weighted avg       0.86      0.86      0.83      6554\n",
      "\n",
      "Feature 3: offer\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00      6527\n",
      "           1       0.00      0.00      0.00        27\n",
      "\n",
      "   micro avg       1.00      1.00      1.00      6554\n",
      "   macro avg       0.50      0.50      0.50      6554\n",
      "weighted avg       0.99      1.00      0.99      6554\n",
      "\n",
      "Feature 4: aid_related\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.77      0.84      0.80      3881\n",
      "           1       0.73      0.64      0.68      2673\n",
      "\n",
      "   micro avg       0.76      0.76      0.76      6554\n",
      "   macro avg       0.75      0.74      0.74      6554\n",
      "weighted avg       0.75      0.76      0.75      6554\n",
      "\n",
      "Feature 5: medical_help\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.93      1.00      0.96      6084\n",
      "           1       0.00      0.00      0.00       470\n",
      "\n",
      "   micro avg       0.93      0.93      0.93      6554\n",
      "   macro avg       0.46      0.50      0.48      6554\n",
      "weighted avg       0.86      0.93      0.89      6554\n",
      "\n",
      "Feature 6: medical_products\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      1.00      0.98      6253\n",
      "           1       0.00      0.00      0.00       301\n",
      "\n",
      "   micro avg       0.95      0.95      0.95      6554\n",
      "   macro avg       0.48      0.50      0.49      6554\n",
      "weighted avg       0.91      0.95      0.93      6554\n",
      "\n",
      "Feature 7: search_and_rescue\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.97      1.00      0.99      6372\n",
      "           1       0.00      0.00      0.00       182\n",
      "\n",
      "   micro avg       0.97      0.97      0.97      6554\n",
      "   macro avg       0.49      0.50      0.49      6554\n",
      "weighted avg       0.95      0.97      0.96      6554\n",
      "\n",
      "Feature 8: security\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      1.00      0.99      6441\n",
      "           1       0.00      0.00      0.00       113\n",
      "\n",
      "   micro avg       0.98      0.98      0.98      6554\n",
      "   macro avg       0.49      0.50      0.50      6554\n",
      "weighted avg       0.97      0.98      0.97      6554\n",
      "\n",
      "Feature 9: military\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.97      1.00      0.98      6326\n",
      "           1       0.00      0.00      0.00       228\n",
      "\n",
      "   micro avg       0.97      0.97      0.97      6554\n",
      "   macro avg       0.48      0.50      0.49      6554\n",
      "weighted avg       0.93      0.97      0.95      6554\n",
      "\n",
      "Feature 10: child_alone\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00      6554\n",
      "\n",
      "   micro avg       1.00      1.00      1.00      6554\n",
      "   macro avg       1.00      1.00      1.00      6554\n",
      "weighted avg       1.00      1.00      1.00      6554\n",
      "\n",
      "Feature 11: water\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.94      1.00      0.97      6153\n",
      "           1       0.00      0.00      0.00       401\n",
      "\n",
      "   micro avg       0.94      0.94      0.94      6554\n",
      "   macro avg       0.47      0.50      0.48      6554\n",
      "weighted avg       0.88      0.94      0.91      6554\n",
      "\n",
      "Feature 12: food\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.89      1.00      0.94      5813\n",
      "           1       0.62      0.02      0.03       741\n",
      "\n",
      "   micro avg       0.89      0.89      0.89      6554\n",
      "   macro avg       0.75      0.51      0.49      6554\n",
      "weighted avg       0.86      0.89      0.84      6554\n",
      "\n",
      "Feature 13: shelter\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.92      1.00      0.96      6006\n",
      "           1       0.33      0.00      0.00       548\n",
      "\n",
      "   micro avg       0.92      0.92      0.92      6554\n",
      "   macro avg       0.62      0.50      0.48      6554\n",
      "weighted avg       0.87      0.92      0.88      6554\n",
      "\n",
      "Feature 14: clothing\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      1.00      0.99      6453\n",
      "           1       0.00      0.00      0.00       101\n",
      "\n",
      "   micro avg       0.98      0.98      0.98      6554\n",
      "   macro avg       0.49      0.50      0.50      6554\n",
      "weighted avg       0.97      0.98      0.98      6554\n",
      "\n",
      "Feature 15: money\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      1.00      0.99      6393\n",
      "           1       0.00      0.00      0.00       161\n",
      "\n",
      "   micro avg       0.98      0.98      0.98      6554\n",
      "   macro avg       0.49      0.50      0.49      6554\n",
      "weighted avg       0.95      0.98      0.96      6554\n",
      "\n",
      "Feature 16: missing_people\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      0.99      6483\n",
      "           1       0.00      0.00      0.00        71\n",
      "\n",
      "   micro avg       0.99      0.99      0.99      6554\n",
      "   macro avg       0.49      0.50      0.50      6554\n",
      "weighted avg       0.98      0.99      0.98      6554\n",
      "\n",
      "Feature 17: refugees\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.97      1.00      0.98      6344\n",
      "           1       0.00      0.00      0.00       210\n",
      "\n",
      "   micro avg       0.97      0.97      0.97      6554\n",
      "   macro avg       0.48      0.50      0.49      6554\n",
      "weighted avg       0.94      0.97      0.95      6554\n",
      "\n",
      "Feature 18: death\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      1.00      0.98      6283\n",
      "           1       0.00      0.00      0.00       271\n",
      "\n",
      "   micro avg       0.96      0.96      0.96      6554\n",
      "   macro avg       0.48      0.50      0.49      6554\n",
      "weighted avg       0.92      0.96      0.94      6554\n",
      "\n",
      "Feature 19: other_aid\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.87      1.00      0.93      5713\n",
      "           1       0.00      0.00      0.00       841\n",
      "\n",
      "   micro avg       0.87      0.87      0.87      6554\n",
      "   macro avg       0.44      0.50      0.47      6554\n",
      "weighted avg       0.76      0.87      0.81      6554\n",
      "\n",
      "Feature 20: infrastructure_related\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.94      1.00      0.97      6137\n",
      "           1       0.00      0.00      0.00       417\n",
      "\n",
      "   micro avg       0.94      0.94      0.94      6554\n",
      "   macro avg       0.47      0.50      0.48      6554\n",
      "weighted avg       0.88      0.94      0.91      6554\n",
      "\n",
      "Feature 21: transport\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      1.00      0.98      6258\n",
      "           1       0.00      0.00      0.00       296\n",
      "\n",
      "   micro avg       0.95      0.95      0.95      6554\n",
      "   macro avg       0.48      0.50      0.49      6554\n",
      "weighted avg       0.91      0.95      0.93      6554\n",
      "\n",
      "Feature 22: buildings\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      1.00      0.97      6233\n",
      "           1       0.00      0.00      0.00       321\n",
      "\n",
      "   micro avg       0.95      0.95      0.95      6554\n",
      "   macro avg       0.48      0.50      0.49      6554\n",
      "weighted avg       0.90      0.95      0.93      6554\n",
      "\n",
      "Feature 23: electricity\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      1.00      0.99      6420\n",
      "           1       0.00      0.00      0.00       134\n",
      "\n",
      "   micro avg       0.98      0.98      0.98      6554\n",
      "   macro avg       0.49      0.50      0.49      6554\n",
      "weighted avg       0.96      0.98      0.97      6554\n",
      "\n",
      "Feature 24: tools\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00      6524\n",
      "           1       0.00      0.00      0.00        30\n",
      "\n",
      "   micro avg       1.00      1.00      1.00      6554\n",
      "   macro avg       0.50      0.50      0.50      6554\n",
      "weighted avg       0.99      1.00      0.99      6554\n",
      "\n",
      "Feature 25: hospitals\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      1.00      6496\n",
      "           1       0.00      0.00      0.00        58\n",
      "\n",
      "   micro avg       0.99      0.99      0.99      6554\n",
      "   macro avg       0.50      0.50      0.50      6554\n",
      "weighted avg       0.98      0.99      0.99      6554\n",
      "\n",
      "Feature 26: shops\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00      6523\n",
      "           1       0.00      0.00      0.00        31\n",
      "\n",
      "   micro avg       1.00      1.00      1.00      6554\n",
      "   macro avg       0.50      0.50      0.50      6554\n",
      "weighted avg       0.99      1.00      0.99      6554\n",
      "\n",
      "Feature 27: aid_centers\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      0.99      6483\n",
      "           1       0.00      0.00      0.00        71\n",
      "\n",
      "   micro avg       0.99      0.99      0.99      6554\n",
      "   macro avg       0.49      0.50      0.50      6554\n",
      "weighted avg       0.98      0.99      0.98      6554\n",
      "\n",
      "Feature 28: other_infrastructure\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      1.00      0.98      6254\n",
      "           1       0.00      0.00      0.00       300\n",
      "\n",
      "   micro avg       0.95      0.95      0.95      6554\n",
      "   macro avg       0.48      0.50      0.49      6554\n",
      "weighted avg       0.91      0.95      0.93      6554\n",
      "\n",
      "Feature 29: weather_related\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.82      0.97      0.89      4773\n",
      "           1       0.83      0.42      0.56      1781\n",
      "\n",
      "   micro avg       0.82      0.82      0.82      6554\n",
      "   macro avg       0.82      0.70      0.72      6554\n",
      "weighted avg       0.82      0.82      0.80      6554\n",
      "\n",
      "Feature 30: floods\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.92      1.00      0.96      6010\n",
      "           1       0.00      0.00      0.00       544\n",
      "\n",
      "   micro avg       0.92      0.92      0.92      6554\n",
      "   macro avg       0.46      0.50      0.48      6554\n",
      "weighted avg       0.84      0.92      0.88      6554\n",
      "\n",
      "Feature 31: storm\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.91      1.00      0.95      5946\n",
      "           1       0.73      0.02      0.04       608\n",
      "\n",
      "   micro avg       0.91      0.91      0.91      6554\n",
      "   macro avg       0.82      0.51      0.49      6554\n",
      "weighted avg       0.89      0.91      0.87      6554\n",
      "\n",
      "Feature 32: fire\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      1.00      6496\n",
      "           1       0.00      0.00      0.00        58\n",
      "\n",
      "   micro avg       0.99      0.99      0.99      6554\n",
      "   macro avg       0.50      0.50      0.50      6554\n",
      "weighted avg       0.98      0.99      0.99      6554\n",
      "\n",
      "Feature 33: earthquake\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.92      1.00      0.96      5975\n",
      "           1       0.89      0.12      0.21       579\n",
      "\n",
      "   micro avg       0.92      0.92      0.92      6554\n",
      "   macro avg       0.91      0.56      0.58      6554\n",
      "weighted avg       0.92      0.92      0.89      6554\n",
      "\n",
      "Feature 34: cold\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      1.00      0.99      6439\n",
      "           1       0.00      0.00      0.00       115\n",
      "\n",
      "   micro avg       0.98      0.98      0.98      6554\n",
      "   macro avg       0.49      0.50      0.50      6554\n",
      "weighted avg       0.97      0.98      0.97      6554\n",
      "\n",
      "Feature 35: other_weather\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      1.00      0.97      6202\n",
      "           1       0.00      0.00      0.00       352\n",
      "\n",
      "   micro avg       0.95      0.95      0.95      6554\n",
      "   macro avg       0.47      0.50      0.49      6554\n",
      "weighted avg       0.90      0.95      0.92      6554\n",
      "\n",
      "Feature 36: direct_report\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.84      0.99      0.91      5324\n",
      "           1       0.78      0.16      0.27      1230\n",
      "\n",
      "   micro avg       0.83      0.83      0.83      6554\n",
      "   macro avg       0.81      0.58      0.59      6554\n",
      "weighted avg       0.82      0.83      0.79      6554\n",
      "\n",
      "The model accuracy is 0.937\n"
     ]
    }
   ],
   "source": [
    "# Prediction: the Naive Bayes classifier \n",
    "y_pred = pipeline_nbc.predict(X_test)\n",
    "plot_scores(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Feature 1: related\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.59      0.08      0.15      1555\n",
      "           1       0.77      0.98      0.86      4939\n",
      "           2       0.40      0.17      0.24        60\n",
      "\n",
      "   micro avg       0.76      0.76      0.76      6554\n",
      "   macro avg       0.59      0.41      0.41      6554\n",
      "weighted avg       0.72      0.76      0.69      6554\n",
      "\n",
      "Feature 2: request\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.91      0.97      0.94      5463\n",
      "           1       0.77      0.52      0.62      1091\n",
      "\n",
      "   micro avg       0.89      0.89      0.89      6554\n",
      "   macro avg       0.84      0.75      0.78      6554\n",
      "weighted avg       0.89      0.89      0.89      6554\n",
      "\n",
      "Feature 3: offer\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00      6527\n",
      "           1       0.00      0.00      0.00        27\n",
      "\n",
      "   micro avg       0.99      0.99      0.99      6554\n",
      "   macro avg       0.50      0.50      0.50      6554\n",
      "weighted avg       0.99      0.99      0.99      6554\n",
      "\n",
      "Feature 4: aid_related\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.77      0.87      0.82      3881\n",
      "           1       0.77      0.62      0.68      2673\n",
      "\n",
      "   micro avg       0.77      0.77      0.77      6554\n",
      "   macro avg       0.77      0.74      0.75      6554\n",
      "weighted avg       0.77      0.77      0.76      6554\n",
      "\n",
      "Feature 5: medical_help\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      0.98      0.96      6084\n",
      "           1       0.57      0.27      0.37       470\n",
      "\n",
      "   micro avg       0.93      0.93      0.93      6554\n",
      "   macro avg       0.76      0.63      0.67      6554\n",
      "weighted avg       0.92      0.93      0.92      6554\n",
      "\n",
      "Feature 6: medical_products\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.97      0.99      0.98      6253\n",
      "           1       0.58      0.33      0.42       301\n",
      "\n",
      "   micro avg       0.96      0.96      0.96      6554\n",
      "   macro avg       0.77      0.66      0.70      6554\n",
      "weighted avg       0.95      0.96      0.95      6554\n",
      "\n",
      "Feature 7: search_and_rescue\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      1.00      0.99      6372\n",
      "           1       0.55      0.18      0.27       182\n",
      "\n",
      "   micro avg       0.97      0.97      0.97      6554\n",
      "   macro avg       0.76      0.59      0.63      6554\n",
      "weighted avg       0.97      0.97      0.97      6554\n",
      "\n",
      "Feature 8: security\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      1.00      0.99      6441\n",
      "           1       0.16      0.04      0.06       113\n",
      "\n",
      "   micro avg       0.98      0.98      0.98      6554\n",
      "   macro avg       0.57      0.52      0.52      6554\n",
      "weighted avg       0.97      0.98      0.97      6554\n",
      "\n",
      "Feature 9: military\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      0.99      0.99      6326\n",
      "           1       0.66      0.35      0.46       228\n",
      "\n",
      "   micro avg       0.97      0.97      0.97      6554\n",
      "   macro avg       0.82      0.67      0.72      6554\n",
      "weighted avg       0.97      0.97      0.97      6554\n",
      "\n",
      "Feature 10: child_alone\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00      6554\n",
      "\n",
      "   micro avg       1.00      1.00      1.00      6554\n",
      "   macro avg       1.00      1.00      1.00      6554\n",
      "weighted avg       1.00      1.00      1.00      6554\n",
      "\n",
      "Feature 11: water\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      0.98      0.98      6153\n",
      "           1       0.72      0.65      0.68       401\n",
      "\n",
      "   micro avg       0.96      0.96      0.96      6554\n",
      "   macro avg       0.85      0.82      0.83      6554\n",
      "weighted avg       0.96      0.96      0.96      6554\n",
      "\n",
      "Feature 12: food\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      0.98      0.97      5813\n",
      "           1       0.82      0.64      0.72       741\n",
      "\n",
      "   micro avg       0.94      0.94      0.94      6554\n",
      "   macro avg       0.89      0.81      0.84      6554\n",
      "weighted avg       0.94      0.94      0.94      6554\n",
      "\n",
      "Feature 13: shelter\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      0.99      0.97      6006\n",
      "           1       0.80      0.58      0.67       548\n",
      "\n",
      "   micro avg       0.95      0.95      0.95      6554\n",
      "   macro avg       0.88      0.78      0.82      6554\n",
      "weighted avg       0.95      0.95      0.95      6554\n",
      "\n",
      "Feature 14: clothing\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      0.99      6453\n",
      "           1       0.66      0.33      0.44       101\n",
      "\n",
      "   micro avg       0.99      0.99      0.99      6554\n",
      "   macro avg       0.82      0.66      0.72      6554\n",
      "weighted avg       0.98      0.99      0.98      6554\n",
      "\n",
      "Feature 15: money\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      0.99      0.99      6393\n",
      "           1       0.55      0.29      0.38       161\n",
      "\n",
      "   micro avg       0.98      0.98      0.98      6554\n",
      "   macro avg       0.77      0.64      0.68      6554\n",
      "weighted avg       0.97      0.98      0.97      6554\n",
      "\n",
      "Feature 16: missing_people\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      0.99      6483\n",
      "           1       0.46      0.15      0.23        71\n",
      "\n",
      "   micro avg       0.99      0.99      0.99      6554\n",
      "   macro avg       0.72      0.58      0.61      6554\n",
      "weighted avg       0.99      0.99      0.99      6554\n",
      "\n",
      "Feature 17: refugees\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      0.99      0.98      6344\n",
      "           1       0.52      0.26      0.35       210\n",
      "\n",
      "   micro avg       0.97      0.97      0.97      6554\n",
      "   macro avg       0.75      0.63      0.67      6554\n",
      "weighted avg       0.96      0.97      0.96      6554\n",
      "\n",
      "Feature 18: death\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      0.99      0.99      6283\n",
      "           1       0.73      0.51      0.60       271\n",
      "\n",
      "   micro avg       0.97      0.97      0.97      6554\n",
      "   macro avg       0.86      0.75      0.79      6554\n",
      "weighted avg       0.97      0.97      0.97      6554\n",
      "\n",
      "Feature 19: other_aid\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.89      0.98      0.93      5713\n",
      "           1       0.47      0.14      0.22       841\n",
      "\n",
      "   micro avg       0.87      0.87      0.87      6554\n",
      "   macro avg       0.68      0.56      0.57      6554\n",
      "weighted avg       0.83      0.87      0.84      6554\n",
      "\n",
      "Feature 20: infrastructure_related\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.94      0.99      0.96      6137\n",
      "           1       0.38      0.10      0.15       417\n",
      "\n",
      "   micro avg       0.93      0.93      0.93      6554\n",
      "   macro avg       0.66      0.54      0.56      6554\n",
      "weighted avg       0.91      0.93      0.91      6554\n",
      "\n",
      "Feature 21: transport\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.97      0.99      0.98      6258\n",
      "           1       0.69      0.25      0.37       296\n",
      "\n",
      "   micro avg       0.96      0.96      0.96      6554\n",
      "   macro avg       0.83      0.62      0.67      6554\n",
      "weighted avg       0.95      0.96      0.95      6554\n",
      "\n",
      "Feature 22: buildings\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.97      0.99      0.98      6233\n",
      "           1       0.62      0.40      0.49       321\n",
      "\n",
      "   micro avg       0.96      0.96      0.96      6554\n",
      "   macro avg       0.80      0.69      0.73      6554\n",
      "weighted avg       0.95      0.96      0.95      6554\n",
      "\n",
      "Feature 23: electricity\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      0.99      0.99      6420\n",
      "           1       0.56      0.34      0.42       134\n",
      "\n",
      "   micro avg       0.98      0.98      0.98      6554\n",
      "   macro avg       0.77      0.67      0.71      6554\n",
      "weighted avg       0.98      0.98      0.98      6554\n",
      "\n",
      "Feature 24: tools\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00      6524\n",
      "           1       0.00      0.00      0.00        30\n",
      "\n",
      "   micro avg       0.99      0.99      0.99      6554\n",
      "   macro avg       0.50      0.50      0.50      6554\n",
      "weighted avg       0.99      0.99      0.99      6554\n",
      "\n",
      "Feature 25: hospitals\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      0.99      6496\n",
      "           1       0.22      0.07      0.11        58\n",
      "\n",
      "   micro avg       0.99      0.99      0.99      6554\n",
      "   macro avg       0.61      0.53      0.55      6554\n",
      "weighted avg       0.98      0.99      0.99      6554\n",
      "\n",
      "Feature 26: shops\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00      6523\n",
      "           1       0.17      0.03      0.05        31\n",
      "\n",
      "   micro avg       0.99      0.99      0.99      6554\n",
      "   macro avg       0.58      0.52      0.53      6554\n",
      "weighted avg       0.99      0.99      0.99      6554\n",
      "\n",
      "Feature 27: aid_centers\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      0.99      6483\n",
      "           1       0.29      0.08      0.13        71\n",
      "\n",
      "   micro avg       0.99      0.99      0.99      6554\n",
      "   macro avg       0.64      0.54      0.56      6554\n",
      "weighted avg       0.98      0.99      0.98      6554\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Feature 28: other_infrastructure\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      0.99      0.97      6254\n",
      "           1       0.35      0.08      0.13       300\n",
      "\n",
      "   micro avg       0.95      0.95      0.95      6554\n",
      "   macro avg       0.66      0.54      0.55      6554\n",
      "weighted avg       0.93      0.95      0.94      6554\n",
      "\n",
      "Feature 29: weather_related\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.89      0.96      0.92      4773\n",
      "           1       0.86      0.68      0.76      1781\n",
      "\n",
      "   micro avg       0.88      0.88      0.88      6554\n",
      "   macro avg       0.87      0.82      0.84      6554\n",
      "weighted avg       0.88      0.88      0.88      6554\n",
      "\n",
      "Feature 30: floods\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      0.99      0.98      6010\n",
      "           1       0.88      0.56      0.68       544\n",
      "\n",
      "   micro avg       0.96      0.96      0.96      6554\n",
      "   macro avg       0.92      0.78      0.83      6554\n",
      "weighted avg       0.95      0.96      0.95      6554\n",
      "\n",
      "Feature 31: storm\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      0.98      0.97      5946\n",
      "           1       0.75      0.52      0.61       608\n",
      "\n",
      "   micro avg       0.94      0.94      0.94      6554\n",
      "   macro avg       0.85      0.75      0.79      6554\n",
      "weighted avg       0.93      0.94      0.93      6554\n",
      "\n",
      "Feature 32: fire\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      1.00      6496\n",
      "           1       0.42      0.09      0.14        58\n",
      "\n",
      "   micro avg       0.99      0.99      0.99      6554\n",
      "   macro avg       0.70      0.54      0.57      6554\n",
      "weighted avg       0.99      0.99      0.99      6554\n",
      "\n",
      "Feature 33: earthquake\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      0.99      0.99      5975\n",
      "           1       0.88      0.80      0.84       579\n",
      "\n",
      "   micro avg       0.97      0.97      0.97      6554\n",
      "   macro avg       0.93      0.90      0.91      6554\n",
      "weighted avg       0.97      0.97      0.97      6554\n",
      "\n",
      "Feature 34: cold\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      0.99      6439\n",
      "           1       0.58      0.28      0.38       115\n",
      "\n",
      "   micro avg       0.98      0.98      0.98      6554\n",
      "   macro avg       0.78      0.64      0.68      6554\n",
      "weighted avg       0.98      0.98      0.98      6554\n",
      "\n",
      "Feature 35: other_weather\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      0.99      0.97      6202\n",
      "           1       0.51      0.14      0.22       352\n",
      "\n",
      "   micro avg       0.95      0.95      0.95      6554\n",
      "   macro avg       0.73      0.57      0.60      6554\n",
      "weighted avg       0.93      0.95      0.93      6554\n",
      "\n",
      "Feature 36: direct_report\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.87      0.96      0.91      5324\n",
      "           1       0.68      0.40      0.51      1230\n",
      "\n",
      "   micro avg       0.85      0.85      0.85      6554\n",
      "   macro avg       0.78      0.68      0.71      6554\n",
      "weighted avg       0.84      0.85      0.84      6554\n",
      "\n",
      "The model accuracy is 0.948\n"
     ]
    }
   ],
   "source": [
    "# Prediction: the Adaboost Classifier \n",
    "y_pred = pipeline_ada.predict(X_test)\n",
    "plot_scores(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. Improve your model\n",
    "Use grid search to find better parameters. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'memory': None,\n",
       " 'steps': [('vect',\n",
       "   CountVectorizer(analyzer='word', binary=False, decode_error='strict',\n",
       "           dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',\n",
       "           lowercase=True, max_df=1.0, max_features=None, min_df=1,\n",
       "           ngram_range=(1, 1), preprocessor=None, stop_words=None,\n",
       "           strip_accents=None, token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
       "           tokenizer=<function tokenize at 0x0000028F37B789D8>,\n",
       "           vocabulary=None)),\n",
       "  ('tfidf',\n",
       "   TfidfTransformer(norm='l2', smooth_idf=True, sublinear_tf=False, use_idf=True)),\n",
       "  ('clf',\n",
       "   MultiOutputClassifier(estimator=RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
       "               max_depth=None, max_features='auto', max_leaf_nodes=None,\n",
       "               min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "               min_samples_leaf=1, min_samples_split=2,\n",
       "               min_weight_fraction_leaf=0.0, n_estimators='warn', n_jobs=None,\n",
       "               oob_score=False, random_state=None, verbose=0,\n",
       "               warm_start=False),\n",
       "              n_jobs=None))],\n",
       " 'vect': CountVectorizer(analyzer='word', binary=False, decode_error='strict',\n",
       "         dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',\n",
       "         lowercase=True, max_df=1.0, max_features=None, min_df=1,\n",
       "         ngram_range=(1, 1), preprocessor=None, stop_words=None,\n",
       "         strip_accents=None, token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
       "         tokenizer=<function tokenize at 0x0000028F37B789D8>,\n",
       "         vocabulary=None),\n",
       " 'tfidf': TfidfTransformer(norm='l2', smooth_idf=True, sublinear_tf=False, use_idf=True),\n",
       " 'clf': MultiOutputClassifier(estimator=RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
       "             max_depth=None, max_features='auto', max_leaf_nodes=None,\n",
       "             min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "             min_samples_leaf=1, min_samples_split=2,\n",
       "             min_weight_fraction_leaf=0.0, n_estimators='warn', n_jobs=None,\n",
       "             oob_score=False, random_state=None, verbose=0,\n",
       "             warm_start=False),\n",
       "            n_jobs=None),\n",
       " 'vect__analyzer': 'word',\n",
       " 'vect__binary': False,\n",
       " 'vect__decode_error': 'strict',\n",
       " 'vect__dtype': numpy.int64,\n",
       " 'vect__encoding': 'utf-8',\n",
       " 'vect__input': 'content',\n",
       " 'vect__lowercase': True,\n",
       " 'vect__max_df': 1.0,\n",
       " 'vect__max_features': None,\n",
       " 'vect__min_df': 1,\n",
       " 'vect__ngram_range': (1, 1),\n",
       " 'vect__preprocessor': None,\n",
       " 'vect__stop_words': None,\n",
       " 'vect__strip_accents': None,\n",
       " 'vect__token_pattern': '(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
       " 'vect__tokenizer': <function __main__.tokenize(text)>,\n",
       " 'vect__vocabulary': None,\n",
       " 'tfidf__norm': 'l2',\n",
       " 'tfidf__smooth_idf': True,\n",
       " 'tfidf__sublinear_tf': False,\n",
       " 'tfidf__use_idf': True,\n",
       " 'clf__estimator__bootstrap': True,\n",
       " 'clf__estimator__class_weight': None,\n",
       " 'clf__estimator__criterion': 'gini',\n",
       " 'clf__estimator__max_depth': None,\n",
       " 'clf__estimator__max_features': 'auto',\n",
       " 'clf__estimator__max_leaf_nodes': None,\n",
       " 'clf__estimator__min_impurity_decrease': 0.0,\n",
       " 'clf__estimator__min_impurity_split': None,\n",
       " 'clf__estimator__min_samples_leaf': 1,\n",
       " 'clf__estimator__min_samples_split': 2,\n",
       " 'clf__estimator__min_weight_fraction_leaf': 0.0,\n",
       " 'clf__estimator__n_estimators': 'warn',\n",
       " 'clf__estimator__n_jobs': None,\n",
       " 'clf__estimator__oob_score': False,\n",
       " 'clf__estimator__random_state': None,\n",
       " 'clf__estimator__verbose': 0,\n",
       " 'clf__estimator__warm_start': False,\n",
       " 'clf__estimator': RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
       "             max_depth=None, max_features='auto', max_leaf_nodes=None,\n",
       "             min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "             min_samples_leaf=1, min_samples_split=2,\n",
       "             min_weight_fraction_leaf=0.0, n_estimators='warn', n_jobs=None,\n",
       "             oob_score=False, random_state=None, verbose=0,\n",
       "             warm_start=False),\n",
       " 'clf__n_jobs': None}"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Show parameters for the pipline\n",
    "pipeline_rfc.get_params()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'memory': None,\n",
       " 'steps': [('vect',\n",
       "   CountVectorizer(analyzer='word', binary=False, decode_error='strict',\n",
       "           dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',\n",
       "           lowercase=True, max_df=1.0, max_features=None, min_df=1,\n",
       "           ngram_range=(1, 1), preprocessor=None, stop_words=None,\n",
       "           strip_accents=None, token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
       "           tokenizer=<function tokenize at 0x0000028F37B789D8>,\n",
       "           vocabulary=None)),\n",
       "  ('tfidf',\n",
       "   TfidfTransformer(norm='l2', smooth_idf=True, sublinear_tf=False, use_idf=True)),\n",
       "  ('clf',\n",
       "   MultiOutputClassifier(estimator=AdaBoostClassifier(algorithm='SAMME.R', base_estimator=None,\n",
       "             learning_rate=1.0, n_estimators=50, random_state=None),\n",
       "              n_jobs=None))],\n",
       " 'vect': CountVectorizer(analyzer='word', binary=False, decode_error='strict',\n",
       "         dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',\n",
       "         lowercase=True, max_df=1.0, max_features=None, min_df=1,\n",
       "         ngram_range=(1, 1), preprocessor=None, stop_words=None,\n",
       "         strip_accents=None, token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
       "         tokenizer=<function tokenize at 0x0000028F37B789D8>,\n",
       "         vocabulary=None),\n",
       " 'tfidf': TfidfTransformer(norm='l2', smooth_idf=True, sublinear_tf=False, use_idf=True),\n",
       " 'clf': MultiOutputClassifier(estimator=AdaBoostClassifier(algorithm='SAMME.R', base_estimator=None,\n",
       "           learning_rate=1.0, n_estimators=50, random_state=None),\n",
       "            n_jobs=None),\n",
       " 'vect__analyzer': 'word',\n",
       " 'vect__binary': False,\n",
       " 'vect__decode_error': 'strict',\n",
       " 'vect__dtype': numpy.int64,\n",
       " 'vect__encoding': 'utf-8',\n",
       " 'vect__input': 'content',\n",
       " 'vect__lowercase': True,\n",
       " 'vect__max_df': 1.0,\n",
       " 'vect__max_features': None,\n",
       " 'vect__min_df': 1,\n",
       " 'vect__ngram_range': (1, 1),\n",
       " 'vect__preprocessor': None,\n",
       " 'vect__stop_words': None,\n",
       " 'vect__strip_accents': None,\n",
       " 'vect__token_pattern': '(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
       " 'vect__tokenizer': <function __main__.tokenize(text)>,\n",
       " 'vect__vocabulary': None,\n",
       " 'tfidf__norm': 'l2',\n",
       " 'tfidf__smooth_idf': True,\n",
       " 'tfidf__sublinear_tf': False,\n",
       " 'tfidf__use_idf': True,\n",
       " 'clf__estimator__algorithm': 'SAMME.R',\n",
       " 'clf__estimator__base_estimator': None,\n",
       " 'clf__estimator__learning_rate': 1.0,\n",
       " 'clf__estimator__n_estimators': 50,\n",
       " 'clf__estimator__random_state': None,\n",
       " 'clf__estimator': AdaBoostClassifier(algorithm='SAMME.R', base_estimator=None,\n",
       "           learning_rate=1.0, n_estimators=50, random_state=None),\n",
       " 'clf__n_jobs': None}"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Show parameters for the pipline\n",
    "pipeline_ada.get_params()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv='warn', error_score='raise-deprecating',\n",
       "       estimator=Pipeline(memory=None,\n",
       "     steps=[('vect', CountVectorizer(analyzer='word', binary=False, decode_error='strict',\n",
       "        dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',\n",
       "        lowercase=True, max_df=1.0, max_features=None, min_df=1,\n",
       "        ngram_range=(1, 1), preprocessor=None, stop_words=None,\n",
       "        strip..._score=False, random_state=None, verbose=0,\n",
       "            warm_start=False),\n",
       "           n_jobs=None))]),\n",
       "       fit_params=None, iid='warn', n_jobs=None,\n",
       "       param_grid={'tfidf__use_idf': (True, False), 'clf__estimator__n_estimators': [10, 20]},\n",
       "       pre_dispatch='2*n_jobs', refit=True, return_train_score='warn',\n",
       "       scoring=None, verbose=0)"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Using grid search\n",
    "# Create Grid search parameters for Random Forest Classifier   \n",
    "parameters_rfc = {\n",
    "        'tfidf__use_idf': (True, False),\n",
    "        'clf__estimator__n_estimators': [10, 20]\n",
    "}\n",
    "\n",
    "cv_rfc = GridSearchCV(pipeline_rfc, param_grid = parameters_rfc)\n",
    "cv_rfc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv='warn', error_score='raise-deprecating',\n",
       "       estimator=Pipeline(memory=None,\n",
       "     steps=[('vect', CountVectorizer(analyzer='word', binary=False, decode_error='strict',\n",
       "        dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',\n",
       "        lowercase=True, max_df=1.0, max_features=None, min_df=1,\n",
       "        ngram_range=(1, 1), preprocessor=None, stop_words=None,\n",
       "        strip...or=None,\n",
       "          learning_rate=1.0, n_estimators=50, random_state=None),\n",
       "           n_jobs=None))]),\n",
       "       fit_params=None, iid='warn', n_jobs=None,\n",
       "       param_grid={'tfidf__use_idf': (True, False), 'clf__estimator__n_estimators': [50, 60, 70]},\n",
       "       pre_dispatch='2*n_jobs', refit=True, return_train_score='warn',\n",
       "       scoring=None, verbose=0)"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Using grid search\n",
    "# Create Grid search parameters\n",
    "parameters_ada = {\n",
    "        'tfidf__use_idf': (True, False),\n",
    "        'clf__estimator__n_estimators': [50, 60, 70]\n",
    "}\n",
    "\n",
    "cv_ada = GridSearchCV(pipeline_ada, param_grid = parameters_ada)\n",
    "cv_ada"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7. Test your model\n",
    "Show the accuracy, precision, and recall of the tuned model.  \n",
    "\n",
    "Since this project focuses on code quality, process, and  pipelines, there is no minimum performance metric needed to pass. However, make sure to fine tune your models for accuracy, precision and recall to make your project stand out - especially for your portfolio!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\ahmed\\Anaconda3\\lib\\site-packages\\sklearn\\model_selection\\_split.py:2053: FutureWarning: You should specify a value for 'cv' instead of relying on the default value. The default value will change from 3 to 5 in version 0.22.\n",
      "  warnings.warn(CV_WARNING, FutureWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv='warn', error_score='raise-deprecating',\n",
       "       estimator=Pipeline(memory=None,\n",
       "     steps=[('vect', CountVectorizer(analyzer='word', binary=False, decode_error='strict',\n",
       "        dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',\n",
       "        lowercase=True, max_df=1.0, max_features=None, min_df=1,\n",
       "        ngram_range=(1, 1), preprocessor=None, stop_words=None,\n",
       "        strip..._score=False, random_state=None, verbose=0,\n",
       "            warm_start=False),\n",
       "           n_jobs=None))]),\n",
       "       fit_params=None, iid='warn', n_jobs=None,\n",
       "       param_grid={'tfidf__use_idf': (True, False), 'clf__estimator__n_estimators': [10, 20]},\n",
       "       pre_dispatch='2*n_jobs', refit=True, return_train_score='warn',\n",
       "       scoring=None, verbose=0)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Fit the first tuned model\n",
    "cv_rfc.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv='warn', error_score='raise-deprecating',\n",
       "       estimator=Pipeline(memory=None,\n",
       "     steps=[('vect', CountVectorizer(analyzer='word', binary=False, decode_error='strict',\n",
       "        dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',\n",
       "        lowercase=True, max_df=1.0, max_features=None, min_df=1,\n",
       "        ngram_range=(1, 1), preprocessor=None, stop_words=None,\n",
       "        strip...or=None,\n",
       "          learning_rate=1.0, n_estimators=50, random_state=None),\n",
       "           n_jobs=None))]),\n",
       "       fit_params=None, iid='warn', n_jobs=None,\n",
       "       param_grid={'tfidf__use_idf': (True, False), 'clf__estimator__n_estimators': [50, 60, 70]},\n",
       "       pre_dispatch='2*n_jobs', refit=True, return_train_score='warn',\n",
       "       scoring=None, verbose=0)"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Fit the second tuned model\n",
    "cv_ada.fit(X_train, y_train)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Feature 1: related\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.66      0.41      0.51      1555\n",
      "           1       0.83      0.93      0.88      4939\n",
      "           2       0.25      0.37      0.30        60\n",
      "\n",
      "   micro avg       0.80      0.80      0.80      6554\n",
      "   macro avg       0.58      0.57      0.56      6554\n",
      "weighted avg       0.79      0.80      0.79      6554\n",
      "\n",
      "Feature 2: request\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.91      0.98      0.94      5463\n",
      "           1       0.82      0.49      0.61      1091\n",
      "\n",
      "   micro avg       0.90      0.90      0.90      6554\n",
      "   macro avg       0.86      0.73      0.78      6554\n",
      "weighted avg       0.89      0.90      0.89      6554\n",
      "\n",
      "Feature 3: offer\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00      6527\n",
      "           1       0.00      0.00      0.00        27\n",
      "\n",
      "   micro avg       1.00      1.00      1.00      6554\n",
      "   macro avg       0.50      0.50      0.50      6554\n",
      "weighted avg       0.99      1.00      0.99      6554\n",
      "\n",
      "Feature 4: aid_related\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.78      0.85      0.82      3881\n",
      "           1       0.75      0.66      0.70      2673\n",
      "\n",
      "   micro avg       0.77      0.77      0.77      6554\n",
      "   macro avg       0.77      0.75      0.76      6554\n",
      "weighted avg       0.77      0.77      0.77      6554\n",
      "\n",
      "Feature 5: medical_help\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.93      0.99      0.96      6084\n",
      "           1       0.59      0.10      0.17       470\n",
      "\n",
      "   micro avg       0.93      0.93      0.93      6554\n",
      "   macro avg       0.76      0.55      0.57      6554\n",
      "weighted avg       0.91      0.93      0.91      6554\n",
      "\n",
      "Feature 6: medical_products\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      1.00      0.98      6253\n",
      "           1       0.69      0.07      0.13       301\n",
      "\n",
      "   micro avg       0.96      0.96      0.96      6554\n",
      "   macro avg       0.82      0.54      0.55      6554\n",
      "weighted avg       0.94      0.96      0.94      6554\n",
      "\n",
      "Feature 7: search_and_rescue\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.97      1.00      0.99      6372\n",
      "           1       0.57      0.07      0.12       182\n",
      "\n",
      "   micro avg       0.97      0.97      0.97      6554\n",
      "   macro avg       0.77      0.53      0.55      6554\n",
      "weighted avg       0.96      0.97      0.96      6554\n",
      "\n",
      "Feature 8: security\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      1.00      0.99      6441\n",
      "           1       0.00      0.00      0.00       113\n",
      "\n",
      "   micro avg       0.98      0.98      0.98      6554\n",
      "   macro avg       0.49      0.50      0.50      6554\n",
      "weighted avg       0.97      0.98      0.97      6554\n",
      "\n",
      "Feature 9: military\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.97      1.00      0.98      6326\n",
      "           1       0.59      0.04      0.08       228\n",
      "\n",
      "   micro avg       0.97      0.97      0.97      6554\n",
      "   macro avg       0.78      0.52      0.53      6554\n",
      "weighted avg       0.95      0.97      0.95      6554\n",
      "\n",
      "Feature 10: child_alone\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00      6554\n",
      "\n",
      "   micro avg       1.00      1.00      1.00      6554\n",
      "   macro avg       1.00      1.00      1.00      6554\n",
      "weighted avg       1.00      1.00      1.00      6554\n",
      "\n",
      "Feature 11: water\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      1.00      0.98      6153\n",
      "           1       0.85      0.36      0.51       401\n",
      "\n",
      "   micro avg       0.96      0.96      0.96      6554\n",
      "   macro avg       0.91      0.68      0.74      6554\n",
      "weighted avg       0.95      0.96      0.95      6554\n",
      "\n",
      "Feature 12: food\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      0.99      0.97      5813\n",
      "           1       0.84      0.60      0.70       741\n",
      "\n",
      "   micro avg       0.94      0.94      0.94      6554\n",
      "   macro avg       0.90      0.79      0.84      6554\n",
      "weighted avg       0.94      0.94      0.94      6554\n",
      "\n",
      "Feature 13: shelter\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.94      0.99      0.97      6006\n",
      "           1       0.86      0.36      0.51       548\n",
      "\n",
      "   micro avg       0.94      0.94      0.94      6554\n",
      "   macro avg       0.90      0.68      0.74      6554\n",
      "weighted avg       0.94      0.94      0.93      6554\n",
      "\n",
      "Feature 14: clothing\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      0.99      6453\n",
      "           1       0.69      0.09      0.16       101\n",
      "\n",
      "   micro avg       0.99      0.99      0.99      6554\n",
      "   macro avg       0.84      0.54      0.58      6554\n",
      "weighted avg       0.98      0.99      0.98      6554\n",
      "\n",
      "Feature 15: money\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      1.00      0.99      6393\n",
      "           1       0.75      0.04      0.07       161\n",
      "\n",
      "   micro avg       0.98      0.98      0.98      6554\n",
      "   macro avg       0.86      0.52      0.53      6554\n",
      "weighted avg       0.97      0.98      0.97      6554\n",
      "\n",
      "Feature 16: missing_people\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      0.99      6483\n",
      "           1       0.67      0.03      0.05        71\n",
      "\n",
      "   micro avg       0.99      0.99      0.99      6554\n",
      "   macro avg       0.83      0.51      0.52      6554\n",
      "weighted avg       0.99      0.99      0.98      6554\n",
      "\n",
      "Feature 17: refugees\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.97      1.00      0.98      6344\n",
      "           1       0.58      0.05      0.10       210\n",
      "\n",
      "   micro avg       0.97      0.97      0.97      6554\n",
      "   macro avg       0.77      0.53      0.54      6554\n",
      "weighted avg       0.96      0.97      0.96      6554\n",
      "\n",
      "Feature 18: death\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.97      1.00      0.98      6283\n",
      "           1       0.75      0.22      0.34       271\n",
      "\n",
      "   micro avg       0.96      0.96      0.96      6554\n",
      "   macro avg       0.86      0.61      0.66      6554\n",
      "weighted avg       0.96      0.96      0.96      6554\n",
      "\n",
      "Feature 19: other_aid\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.88      0.99      0.93      5713\n",
      "           1       0.52      0.04      0.07       841\n",
      "\n",
      "   micro avg       0.87      0.87      0.87      6554\n",
      "   macro avg       0.70      0.52      0.50      6554\n",
      "weighted avg       0.83      0.87      0.82      6554\n",
      "\n",
      "Feature 20: infrastructure_related\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.94      1.00      0.97      6137\n",
      "           1       0.20      0.00      0.00       417\n",
      "\n",
      "   micro avg       0.94      0.94      0.94      6554\n",
      "   macro avg       0.57      0.50      0.49      6554\n",
      "weighted avg       0.89      0.94      0.91      6554\n",
      "\n",
      "Feature 21: transport\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      1.00      0.98      6258\n",
      "           1       0.58      0.04      0.07       296\n",
      "\n",
      "   micro avg       0.96      0.96      0.96      6554\n",
      "   macro avg       0.77      0.52      0.52      6554\n",
      "weighted avg       0.94      0.96      0.94      6554\n",
      "\n",
      "Feature 22: buildings\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      1.00      0.98      6233\n",
      "           1       0.66      0.14      0.23       321\n",
      "\n",
      "   micro avg       0.95      0.95      0.95      6554\n",
      "   macro avg       0.81      0.57      0.60      6554\n",
      "weighted avg       0.94      0.95      0.94      6554\n",
      "\n",
      "Feature 23: electricity\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      1.00      0.99      6420\n",
      "           1       0.60      0.02      0.04       134\n",
      "\n",
      "   micro avg       0.98      0.98      0.98      6554\n",
      "   macro avg       0.79      0.51      0.52      6554\n",
      "weighted avg       0.97      0.98      0.97      6554\n",
      "\n",
      "Feature 24: tools\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00      6524\n",
      "           1       0.00      0.00      0.00        30\n",
      "\n",
      "   micro avg       1.00      1.00      1.00      6554\n",
      "   macro avg       0.50      0.50      0.50      6554\n",
      "weighted avg       0.99      1.00      0.99      6554\n",
      "\n",
      "Feature 25: hospitals\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      1.00      6496\n",
      "           1       0.00      0.00      0.00        58\n",
      "\n",
      "   micro avg       0.99      0.99      0.99      6554\n",
      "   macro avg       0.50      0.50      0.50      6554\n",
      "weighted avg       0.98      0.99      0.99      6554\n",
      "\n",
      "Feature 26: shops\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00      6523\n",
      "           1       0.00      0.00      0.00        31\n",
      "\n",
      "   micro avg       1.00      1.00      1.00      6554\n",
      "   macro avg       0.50      0.50      0.50      6554\n",
      "weighted avg       0.99      1.00      0.99      6554\n",
      "\n",
      "Feature 27: aid_centers\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      0.99      6483\n",
      "           1       0.00      0.00      0.00        71\n",
      "\n",
      "   micro avg       0.99      0.99      0.99      6554\n",
      "   macro avg       0.49      0.50      0.50      6554\n",
      "weighted avg       0.98      0.99      0.98      6554\n",
      "\n",
      "Feature 28: other_infrastructure\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      1.00      0.98      6254\n",
      "           1       0.50      0.00      0.01       300\n",
      "\n",
      "   micro avg       0.95      0.95      0.95      6554\n",
      "   macro avg       0.73      0.50      0.49      6554\n",
      "weighted avg       0.93      0.95      0.93      6554\n",
      "\n",
      "Feature 29: weather_related\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.88      0.96      0.92      4773\n",
      "           1       0.85      0.67      0.75      1781\n",
      "\n",
      "   micro avg       0.88      0.88      0.88      6554\n",
      "   macro avg       0.87      0.81      0.83      6554\n",
      "weighted avg       0.87      0.88      0.87      6554\n",
      "\n",
      "Feature 30: floods\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      1.00      0.97      6010\n",
      "           1       0.88      0.38      0.53       544\n",
      "\n",
      "   micro avg       0.94      0.94      0.94      6554\n",
      "   macro avg       0.91      0.69      0.75      6554\n",
      "weighted avg       0.94      0.94      0.93      6554\n",
      "\n",
      "Feature 31: storm\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.94      0.99      0.97      5946\n",
      "           1       0.77      0.43      0.55       608\n",
      "\n",
      "   micro avg       0.94      0.94      0.94      6554\n",
      "   macro avg       0.86      0.71      0.76      6554\n",
      "weighted avg       0.93      0.94      0.93      6554\n",
      "\n",
      "Feature 32: fire\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      1.00      6496\n",
      "           1       0.50      0.02      0.03        58\n",
      "\n",
      "   micro avg       0.99      0.99      0.99      6554\n",
      "   macro avg       0.75      0.51      0.51      6554\n",
      "weighted avg       0.99      0.99      0.99      6554\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\ahmed\\Anaconda3\\lib\\site-packages\\sklearn\\metrics\\classification.py:1143: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Feature 33: earthquake\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      0.99      0.98      5975\n",
      "           1       0.89      0.76      0.82       579\n",
      "\n",
      "   micro avg       0.97      0.97      0.97      6554\n",
      "   macro avg       0.93      0.88      0.90      6554\n",
      "weighted avg       0.97      0.97      0.97      6554\n",
      "\n",
      "Feature 34: cold\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      1.00      0.99      6439\n",
      "           1       0.71      0.09      0.16       115\n",
      "\n",
      "   micro avg       0.98      0.98      0.98      6554\n",
      "   macro avg       0.85      0.54      0.57      6554\n",
      "weighted avg       0.98      0.98      0.98      6554\n",
      "\n",
      "Feature 35: other_weather\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      1.00      0.97      6202\n",
      "           1       0.48      0.04      0.07       352\n",
      "\n",
      "   micro avg       0.95      0.95      0.95      6554\n",
      "   macro avg       0.71      0.52      0.52      6554\n",
      "weighted avg       0.92      0.95      0.92      6554\n",
      "\n",
      "Feature 36: direct_report\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.87      0.98      0.92      5324\n",
      "           1       0.78      0.36      0.49      1230\n",
      "\n",
      "   micro avg       0.86      0.86      0.86      6554\n",
      "   macro avg       0.82      0.67      0.71      6554\n",
      "weighted avg       0.85      0.86      0.84      6554\n",
      "\n",
      "The model accuracy is 0.948\n"
     ]
    }
   ],
   "source": [
    "# Predicting using the first tuned model \n",
    "y_pred = cv_rfc.predict(X_test)\n",
    "plot_scores(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Feature 1: related\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.60      0.12      0.20      1555\n",
      "           1       0.77      0.97      0.86      4939\n",
      "           2       0.40      0.17      0.24        60\n",
      "\n",
      "   micro avg       0.76      0.76      0.76      6554\n",
      "   macro avg       0.59      0.42      0.43      6554\n",
      "weighted avg       0.73      0.76      0.70      6554\n",
      "\n",
      "Feature 2: request\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.92      0.97      0.94      5463\n",
      "           1       0.76      0.55      0.64      1091\n",
      "\n",
      "   micro avg       0.90      0.90      0.90      6554\n",
      "   macro avg       0.84      0.76      0.79      6554\n",
      "weighted avg       0.89      0.90      0.89      6554\n",
      "\n",
      "Feature 3: offer\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00      6527\n",
      "           1       0.00      0.00      0.00        27\n",
      "\n",
      "   micro avg       0.99      0.99      0.99      6554\n",
      "   macro avg       0.50      0.50      0.50      6554\n",
      "weighted avg       0.99      0.99      0.99      6554\n",
      "\n",
      "Feature 4: aid_related\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.77      0.87      0.82      3881\n",
      "           1       0.77      0.62      0.68      2673\n",
      "\n",
      "   micro avg       0.77      0.77      0.77      6554\n",
      "   macro avg       0.77      0.74      0.75      6554\n",
      "weighted avg       0.77      0.77      0.76      6554\n",
      "\n",
      "Feature 5: medical_help\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.94      0.98      0.96      6084\n",
      "           1       0.54      0.26      0.35       470\n",
      "\n",
      "   micro avg       0.93      0.93      0.93      6554\n",
      "   macro avg       0.74      0.62      0.66      6554\n",
      "weighted avg       0.92      0.93      0.92      6554\n",
      "\n",
      "Feature 6: medical_products\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.97      0.99      0.98      6253\n",
      "           1       0.53      0.31      0.39       301\n",
      "\n",
      "   micro avg       0.96      0.96      0.96      6554\n",
      "   macro avg       0.75      0.65      0.68      6554\n",
      "weighted avg       0.95      0.96      0.95      6554\n",
      "\n",
      "Feature 7: search_and_rescue\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      1.00      0.99      6372\n",
      "           1       0.55      0.19      0.28       182\n",
      "\n",
      "   micro avg       0.97      0.97      0.97      6554\n",
      "   macro avg       0.76      0.59      0.64      6554\n",
      "weighted avg       0.97      0.97      0.97      6554\n",
      "\n",
      "Feature 8: security\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      1.00      0.99      6441\n",
      "           1       0.14      0.04      0.06       113\n",
      "\n",
      "   micro avg       0.98      0.98      0.98      6554\n",
      "   macro avg       0.56      0.52      0.52      6554\n",
      "weighted avg       0.97      0.98      0.97      6554\n",
      "\n",
      "Feature 9: military\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      0.99      0.98      6326\n",
      "           1       0.60      0.30      0.40       228\n",
      "\n",
      "   micro avg       0.97      0.97      0.97      6554\n",
      "   macro avg       0.79      0.65      0.69      6554\n",
      "weighted avg       0.96      0.97      0.96      6554\n",
      "\n",
      "Feature 10: child_alone\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00      6554\n",
      "\n",
      "   micro avg       1.00      1.00      1.00      6554\n",
      "   macro avg       1.00      1.00      1.00      6554\n",
      "weighted avg       1.00      1.00      1.00      6554\n",
      "\n",
      "Feature 11: water\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      0.98      0.98      6153\n",
      "           1       0.72      0.62      0.67       401\n",
      "\n",
      "   micro avg       0.96      0.96      0.96      6554\n",
      "   macro avg       0.85      0.80      0.82      6554\n",
      "weighted avg       0.96      0.96      0.96      6554\n",
      "\n",
      "Feature 12: food\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      0.98      0.97      5813\n",
      "           1       0.80      0.64      0.71       741\n",
      "\n",
      "   micro avg       0.94      0.94      0.94      6554\n",
      "   macro avg       0.88      0.81      0.84      6554\n",
      "weighted avg       0.94      0.94      0.94      6554\n",
      "\n",
      "Feature 13: shelter\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      0.99      0.97      6006\n",
      "           1       0.78      0.57      0.66       548\n",
      "\n",
      "   micro avg       0.95      0.95      0.95      6554\n",
      "   macro avg       0.87      0.78      0.82      6554\n",
      "weighted avg       0.95      0.95      0.95      6554\n",
      "\n",
      "Feature 14: clothing\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      0.99      6453\n",
      "           1       0.52      0.23      0.32       101\n",
      "\n",
      "   micro avg       0.98      0.98      0.98      6554\n",
      "   macro avg       0.76      0.61      0.65      6554\n",
      "weighted avg       0.98      0.98      0.98      6554\n",
      "\n",
      "Feature 15: money\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      0.99      0.99      6393\n",
      "           1       0.52      0.29      0.37       161\n",
      "\n",
      "   micro avg       0.98      0.98      0.98      6554\n",
      "   macro avg       0.75      0.64      0.68      6554\n",
      "weighted avg       0.97      0.98      0.97      6554\n",
      "\n",
      "Feature 16: missing_people\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      0.99      6483\n",
      "           1       0.46      0.18      0.26        71\n",
      "\n",
      "   micro avg       0.99      0.99      0.99      6554\n",
      "   macro avg       0.73      0.59      0.63      6554\n",
      "weighted avg       0.99      0.99      0.99      6554\n",
      "\n",
      "Feature 17: refugees\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      0.99      0.98      6344\n",
      "           1       0.49      0.26      0.34       210\n",
      "\n",
      "   micro avg       0.97      0.97      0.97      6554\n",
      "   macro avg       0.73      0.63      0.66      6554\n",
      "weighted avg       0.96      0.97      0.96      6554\n",
      "\n",
      "Feature 18: death\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      0.99      0.98      6283\n",
      "           1       0.69      0.50      0.58       271\n",
      "\n",
      "   micro avg       0.97      0.97      0.97      6554\n",
      "   macro avg       0.84      0.74      0.78      6554\n",
      "weighted avg       0.97      0.97      0.97      6554\n",
      "\n",
      "Feature 19: other_aid\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.89      0.97      0.93      5713\n",
      "           1       0.46      0.15      0.23       841\n",
      "\n",
      "   micro avg       0.87      0.87      0.87      6554\n",
      "   macro avg       0.67      0.56      0.58      6554\n",
      "weighted avg       0.83      0.87      0.84      6554\n",
      "\n",
      "Feature 20: infrastructure_related\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.94      0.99      0.97      6137\n",
      "           1       0.43      0.10      0.17       417\n",
      "\n",
      "   micro avg       0.93      0.93      0.93      6554\n",
      "   macro avg       0.69      0.55      0.57      6554\n",
      "weighted avg       0.91      0.93      0.91      6554\n",
      "\n",
      "Feature 21: transport\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.97      0.99      0.98      6258\n",
      "           1       0.63      0.25      0.36       296\n",
      "\n",
      "   micro avg       0.96      0.96      0.96      6554\n",
      "   macro avg       0.80      0.62      0.67      6554\n",
      "weighted avg       0.95      0.96      0.95      6554\n",
      "\n",
      "Feature 22: buildings\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.97      0.99      0.98      6233\n",
      "           1       0.63      0.41      0.50       321\n",
      "\n",
      "   micro avg       0.96      0.96      0.96      6554\n",
      "   macro avg       0.80      0.70      0.74      6554\n",
      "weighted avg       0.95      0.96      0.96      6554\n",
      "\n",
      "Feature 23: electricity\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      0.99      6420\n",
      "           1       0.58      0.34      0.43       134\n",
      "\n",
      "   micro avg       0.98      0.98      0.98      6554\n",
      "   macro avg       0.79      0.67      0.71      6554\n",
      "weighted avg       0.98      0.98      0.98      6554\n",
      "\n",
      "Feature 24: tools\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00      6524\n",
      "           1       0.00      0.00      0.00        30\n",
      "\n",
      "   micro avg       0.99      0.99      0.99      6554\n",
      "   macro avg       0.50      0.50      0.50      6554\n",
      "weighted avg       0.99      0.99      0.99      6554\n",
      "\n",
      "Feature 25: hospitals\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      0.99      6496\n",
      "           1       0.21      0.07      0.10        58\n",
      "\n",
      "   micro avg       0.99      0.99      0.99      6554\n",
      "   macro avg       0.60      0.53      0.55      6554\n",
      "weighted avg       0.98      0.99      0.99      6554\n",
      "\n",
      "Feature 26: shops\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00      6523\n",
      "           1       0.12      0.03      0.05        31\n",
      "\n",
      "   micro avg       0.99      0.99      0.99      6554\n",
      "   macro avg       0.56      0.52      0.52      6554\n",
      "weighted avg       0.99      0.99      0.99      6554\n",
      "\n",
      "Feature 27: aid_centers\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      0.99      6483\n",
      "           1       0.25      0.10      0.14        71\n",
      "\n",
      "   micro avg       0.99      0.99      0.99      6554\n",
      "   macro avg       0.62      0.55      0.57      6554\n",
      "weighted avg       0.98      0.99      0.98      6554\n",
      "\n",
      "Feature 28: other_infrastructure\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      0.99      0.97      6254\n",
      "           1       0.32      0.08      0.13       300\n",
      "\n",
      "   micro avg       0.95      0.95      0.95      6554\n",
      "   macro avg       0.64      0.54      0.55      6554\n",
      "weighted avg       0.93      0.95      0.94      6554\n",
      "\n",
      "Feature 29: weather_related\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.89      0.95      0.92      4773\n",
      "           1       0.85      0.69      0.76      1781\n",
      "\n",
      "   micro avg       0.88      0.88      0.88      6554\n",
      "   macro avg       0.87      0.82      0.84      6554\n",
      "weighted avg       0.88      0.88      0.88      6554\n",
      "\n",
      "Feature 30: floods\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      0.99      0.98      6010\n",
      "           1       0.86      0.57      0.69       544\n",
      "\n",
      "   micro avg       0.96      0.96      0.96      6554\n",
      "   macro avg       0.91      0.78      0.83      6554\n",
      "weighted avg       0.95      0.96      0.95      6554\n",
      "\n",
      "Feature 31: storm\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      0.98      0.97      5946\n",
      "           1       0.75      0.53      0.62       608\n",
      "\n",
      "   micro avg       0.94      0.94      0.94      6554\n",
      "   macro avg       0.85      0.76      0.80      6554\n",
      "weighted avg       0.93      0.94      0.94      6554\n",
      "\n",
      "Feature 32: fire\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      1.00      6496\n",
      "           1       0.39      0.12      0.18        58\n",
      "\n",
      "   micro avg       0.99      0.99      0.99      6554\n",
      "   macro avg       0.69      0.56      0.59      6554\n",
      "weighted avg       0.99      0.99      0.99      6554\n",
      "\n",
      "Feature 33: earthquake\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      0.99      0.99      5975\n",
      "           1       0.87      0.81      0.84       579\n",
      "\n",
      "   micro avg       0.97      0.97      0.97      6554\n",
      "   macro avg       0.93      0.90      0.91      6554\n",
      "weighted avg       0.97      0.97      0.97      6554\n",
      "\n",
      "Feature 34: cold\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      0.99      6439\n",
      "           1       0.60      0.26      0.36       115\n",
      "\n",
      "   micro avg       0.98      0.98      0.98      6554\n",
      "   macro avg       0.79      0.63      0.68      6554\n",
      "weighted avg       0.98      0.98      0.98      6554\n",
      "\n",
      "Feature 35: other_weather\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      0.99      0.97      6202\n",
      "           1       0.47      0.14      0.22       352\n",
      "\n",
      "   micro avg       0.95      0.95      0.95      6554\n",
      "   macro avg       0.71      0.57      0.60      6554\n",
      "weighted avg       0.93      0.95      0.93      6554\n",
      "\n",
      "Feature 36: direct_report\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.88      0.95      0.91      5324\n",
      "           1       0.68      0.42      0.52      1230\n",
      "\n",
      "   micro avg       0.85      0.85      0.85      6554\n",
      "   macro avg       0.78      0.69      0.72      6554\n",
      "weighted avg       0.84      0.85      0.84      6554\n",
      "\n",
      "The model accuracy is 0.948\n"
     ]
    }
   ],
   "source": [
    "# Predicting using the second tuned model \n",
    "y_pred = cv_ada.predict(X_test)\n",
    "plot_scores(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8. Try improving your model further. Here are a few ideas:\n",
    "* try other machine learning algorithms\n",
    "* add other features besides the TF-IDF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9. Export your model as a pickle file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a pickle file for the model\n",
    "file_name = 'classifier.pkl'\n",
    "with open (file_name, 'wb') as f:\n",
    "    pickle.dump(cv_ada, f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 10. Use this notebook to complete `train.py`\n",
    "Use the template file attached in the Resources folder to write a script that runs the steps above to create a database and export a model based on a new dataset specified by the user."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}