09_Random_Forest.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "09 Random Forest.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyMcpo8YTLdr769fmsN5bvH3",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/sandipanpaul21/Tree-Based-Models-in-Python/blob/master/09_Random_Forest.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vKn3iXwy6EQF"
      },
      "source": [
        "#### **Random forests — An ensemble of decision trees**\n",
        "\n",
        "First, we discuss some of the drawbacks of the Decision Tree algorithm. This will motivate you to use Random Forests.\n",
        "\n",
        "1. Small changes to training data can result in a significantly different tree structure.\n",
        "2. It may have the problem of overfitting (the model fits the training data very well but it fails to generalize for new input data) unless you tune the model hyperparameter of max_depth.\n",
        "\n",
        "So, instead of training a single decision tree, it is better to train a group of decision trees which together make a random forest.\n",
        "\n",
        "#### **How random forests work behind the scenes**\n",
        "\n",
        "The main two concepts behind random forests are:\n",
        "1. **The wisdom of the crowd —** a large group of people are collectively smarter than individual experts\n",
        "2. **Diversification —** a set of uncorrelated tress\n",
        "\n",
        "A random forest consists of a group (an ensemble) of individual decision trees. Therefore, the technique is called **Ensemble Learning**. \n",
        "\n",
        "A large group of uncorrelated decision trees can produce more accurate and stable results than any of individual decision trees. When you train a random forest for a classification task, you actually train a group of decision trees. Then you obtain the predictions of all the individual trees and predict the class that gets the most votes. Although some individual trees produce wrong predictions, many can produce accurate predictions. As a group, they can move towards accurate predictions. This is called the wisdom of the crowd. \n",
        "\n",
        "To maintain a low correlation (high diversification) between individual trees, the algorithm automatically considers the following things.\n",
        "\n",
        "1. Feature randomness\n",
        "\n",
        "2. Bagging (bootstrap aggregating)\n",
        "\n",
        "**Feature randomness**\n",
        "\n",
        "- In a normal decision tree, the algorithm searches very best feature out of all the features when it wants to split a node. \n",
        "- In contrast, each tree in a random forest searches very best feature out of a random subset of features. \n",
        "- This creates extra randomness when growing the tress inside a random forest. Because of feature randomness, the decision trees in a random forest are uncorrelated.\n",
        "\n",
        "![Feature Randomness](https://miro.medium.com/max/1400/1*h59804V5gNRmQFKdzmkGGw.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bavrN8l86hPw"
      },
      "source": [
        "**Bagging (bootstrap aggregating)**\n",
        "\n",
        "- In a random forest, each decision tree is trained on a different random sample of the training set. \n",
        "- When sampling is done with replacement, the method is called *bagging (bootstrap aggregating)*. In statistics, resampling with replacement is called bootstrapping. The bootstrap method reduces the correlation between decision trees. \n",
        "- In a decision tree, small changes to training data can result in a significantly different tree structure. The bootstrap method takes the advantage of this to produce uncorrelated trees. \n",
        "- We can demonstrate the bootstrap method with the following simple example. The same thing applies in the random forest.\n",
        "  1. Imagine that we have a training set of 10 observations which are numbered from 1–10. Out of these observations, we perform sampling using the bootstrap method. \n",
        "  2. We want to consider: \n",
        "    \n",
        "    i. Sample size — In machine learning, it is common to use a sample size that is the same as the training set. In this example, the sample size is 10.\n",
        "\n",
        "    ii. The number of samples — This is equal to the number of decision trees in the random forest.\n",
        "    \n",
        "    iii. To create the first sample, we randomly choose an observation from the training set. Let’s say it is the 5th observation. This observation is returned to the training dataset and we repeat the process until we make the entire sample. After the entire process, imagine that we make the first sample with the following observations.\n",
        "    Sample_1 = [5, 4, 6, 6, 5, 1, 3, 2, 10, 9]\n",
        "    \n",
        "    iv. Then we train a decision tree with this sample. Because of the replacement, some observations may appear more times in the sample. \n",
        "    \n",
        "    v. Also, note that some observations don’t appear at least 1 time in the sample. Those observations are called out-of-bag (oob) observations. The oob observations for the first sample are:\n",
        "    \n",
        "    oob_1 = [7, 8]\n",
        "\n",
        "    vi. The decision tree corresponding to sample 1 never sees those oob observations during the training process. So, this set of oob observations can be used as a validation set for that decision tree. \n",
        "\n",
        "    vii. We can evaluate the entire ensemble by averaging out the oob evaluations of each decision tree. This is called the **out-of-bag evaluation** which is an alternative to cross-validation.\n",
        "  \n",
        "  3. Let’s create another sample.\n",
        "\n",
        "    Sample_2 = [5, 4, 4, 5, 5, 1, 3, 2, 10, 9]\n",
        "  \n",
        "    oob_2 = [6, 7, 8]\n",
        "\n",
        "- Likewise, we create a number of samples that is equal to the number of decision trees in the random forest.\n",
        "\n",
        "![](https://miro.medium.com/max/1400/1*-BtU2gT8Og5RE-BybDA9iQ.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zmsSwpNptcjc"
      },
      "source": [
        "#### **Feature importance in a random forest**\n",
        "\n",
        "- Another great advantage of a random forest is that it allows you to get an idea about the relative importance of each feature based on a score computed during the training phase. \n",
        "\n",
        "- For this, he Scikit-learn RandomForestClassifier provides an attribute called feature_importances_. \n",
        "  - This returns an array of values which sum to 1. \n",
        "  - The higher the score, the more important the feature. \n",
        "  - The score is calculated based on the Gini impurity which measures the quality of a split (the lower the Gini, the better the split). \n",
        "  - Features with splits that have a greater mean decrease in Gini impurity are considered more important.\n",
        "\n",
        "- By looking at the feature importance, you can decide which features to drop because they don’t contribute enough for making the model. This is important because of the following reasons.\n",
        "  - Removing the least important features will increase the accuracy of the model. \n",
        "  - This is because we remove the noise by removing unnecessary features\n",
        "  - By removing the unnecessary features, you will avoid the problem of overfitting.\n",
        "  - A lesser amount of features also reduces training time."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ycrWx04vAxJR",
        "outputId": "d419109c-12ea-4528-f398-699a7c69fcc1"
      },
      "source": [
        "# Libraries\n",
        "from sklearn import datasets\n",
        "import pandas as pd\n",
        "import warnings\n",
        "warnings.filterwarnings(\"ignore\")\n",
        "\n",
        "# Loading Dataset\n",
        "breast = datasets.load_breast_cancer()\n",
        "df = pd.DataFrame(breast.data)\n",
        "df.columns = breast.feature_names\n",
        "df[\"WINE_TYPE\"] = breast.target\n",
        "X = df.drop('WINE_TYPE',axis = 1)\n",
        "y = df['WINE_TYPE']\n",
        "\n",
        "# summarize the dataset\n",
        "print(\"Size of X and y :\")\n",
        "print(X.shape, y.shape)\n",
        "print(\"\\n\")\n",
        "print(\"Dataset Sample (BREAST CANCER DATASET)\")\n",
        "print(df.head())"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Size of X and y :\n",
            "(569, 30) (569,)\n",
            "\n",
            "\n",
            "Dataset Sample (BREAST CANCER DATASET)\n",
            "   mean radius  mean texture  ...  worst fractal dimension  WINE_TYPE\n",
            "0        17.99         10.38  ...                  0.11890          0\n",
            "1        20.57         17.77  ...                  0.08902          0\n",
            "2        19.69         21.25  ...                  0.08758          0\n",
            "3        11.42         20.38  ...                  0.17300          0\n",
            "4        20.29         14.34  ...                  0.07678          0\n",
            "\n",
            "[5 rows x 31 columns]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LvJdA1gEwvPN",
        "outputId": "31b86f6a-3c06-4a79-de0c-4c7eedcf1702"
      },
      "source": [
        "# We will evaluate the model using repeated stratified k-fold cross-validation with three repeats and 10 folds. \n",
        "# We will report the mean and standard deviation of the accuracy of the model across all repeats and folds.\n",
        "\n",
        "# evaluate random forest algorithm for classification\n",
        "from numpy import mean\n",
        "from numpy import std\n",
        "from sklearn.datasets import make_classification\n",
        "from sklearn.model_selection import cross_val_score\n",
        "from sklearn.model_selection import RepeatedStratifiedKFold\n",
        "from sklearn.ensemble import RandomForestClassifier\n",
        "\n",
        "# define the model\n",
        "model = RandomForestClassifier()\n",
        "print(model)\n",
        "\n",
        "# evaluate the model\n",
        "cv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)\n",
        "print(cv)\n",
        "n_scores = cross_val_score(model, X, y, scoring='accuracy', cv=cv, n_jobs=-1, error_score='raise')\n",
        "print(\"\\n\")\n",
        "# report performance\n",
        "print('Accuracy: %.3f (%.3f)' % (mean(n_scores), std(n_scores)))\n",
        "\n",
        "# Inference : \n",
        "# In this case, we can see the random forest ensemble with default hyperparameters \n",
        "# It achieves a classification accuracy of about 96 percent."
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "RandomForestClassifier()\n",
            "RepeatedStratifiedKFold(n_repeats=3, n_splits=10, random_state=1)\n",
            "\n",
            "\n",
            "Accuracy: 0.960 (0.032)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TPjRrptKy1PH"
      },
      "source": [
        "### **Random Forest Hyperparameters**\n",
        "\n",
        "**Explore Number of Samples**\n",
        "\n",
        "- Each decision tree in the ensemble is fit on a bootstrap sample drawn from the training dataset.\n",
        "- This can be turned off by setting the “bootstrap” argument to False, if you desire. In that case, the whole training dataset will be used to train each decision tree. This is not recommended.\n",
        "- The “max_samples” argument can be set to a float between 0 and 1 to control the percentage of the size of the training dataset to make the bootstrap sample used to train each decision tree.\n",
        "- For example, if the training dataset has 100 rows, the max_samples argument could be set to 0.5 and each decision tree will be fit on a bootstrap sample with (100 * 0.5) or 50 rows of data.\n",
        "- A smaller sample size will make trees more different, and a larger sample size will make the trees more similar. Setting max_samples to “None” will make the sample size the same size as the training dataset and this is the default.\n",
        "\n",
        "The example below demonstrates the effect of different bootstrap sample sizes from 10 percent to 100 percent on the random forest algorithm."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 481
        },
        "id": "2obPTo93xg9x",
        "outputId": "19fef32f-8c9c-4002-ce51-e99e5e851796"
      },
      "source": [
        "# explore random forest bootstrap sample size on performance\n",
        "from numpy import mean\n",
        "from numpy import std\n",
        "from numpy import arange\n",
        "from sklearn.datasets import make_classification\n",
        "from sklearn.model_selection import cross_val_score\n",
        "from sklearn.model_selection import RepeatedStratifiedKFold\n",
        "from sklearn.ensemble import RandomForestClassifier\n",
        "from matplotlib import pyplot\n",
        "\n",
        "print(\"% of Sample Size vs Accuracy of the Model\")  \n",
        "# get a list of models to evaluate\n",
        "def get_models():\n",
        "\tmodels = dict()\n",
        "\t# explore ratios from 10% to 100% in 10% increments\n",
        "\tfor i in arange(0.1, 1.1, 0.1):\n",
        "\t\tkey = '%.1f' % i\n",
        "\t\t# set max_samples=None to use 100%\n",
        "\t\tif i == 1.0:\n",
        "\t\t\ti = None\n",
        "\t\tmodels[key] = RandomForestClassifier(max_samples=i)\n",
        "\treturn models\n",
        " \n",
        "# evaluate a given model using cross-validation\n",
        "def evaluate_model(model, X, y):\n",
        "\t# define the evaluation procedure\n",
        "\tcv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)\n",
        "\t# evaluate the model and collect the results\n",
        "\tscores = cross_val_score(model, X, y, scoring='accuracy', cv=cv, n_jobs=-1)\n",
        "\treturn scores\n",
        " \n",
        "# get the models to evaluate\n",
        "models = get_models()\n",
        "\n",
        "# evaluate the models and store results\n",
        "results, names = list(), list()\n",
        "for name, model in models.items():\n",
        "\t# evaluate the model\n",
        "\tscores = evaluate_model(model, X, y)\n",
        "\t# store the results\n",
        "\tresults.append(scores)\n",
        "\tnames.append(name)\n",
        "\t# summarize the performance along the way\n",
        "\tprint('>%s %.3f (%.3f)' % (name, mean(scores), std(scores)))\n",
        "\n",
        "print(\"BOX PLOT OF RANDOM FOREST BOOTSTRAP SAMPLE SIZE vs CLASSIFICATION ACCURACY\")\n",
        "# plot model performance for comparison\n",
        "pyplot.boxplot(results, labels=names, showmeans=True)\n",
        "pyplot.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "% of Sample Size vs Accuracy of the Model\n",
            ">0.1 0.947 (0.032)\n",
            ">0.2 0.952 (0.028)\n",
            ">0.3 0.954 (0.030)\n",
            ">0.4 0.955 (0.030)\n",
            ">0.5 0.956 (0.028)\n",
            ">0.6 0.960 (0.028)\n",
            ">0.7 0.956 (0.032)\n",
            ">0.8 0.958 (0.028)\n",
            ">0.9 0.960 (0.025)\n",
            ">1.0 0.963 (0.025)\n",
            "BOX PLOT OF RANDOM FOREST BOOTSTRAP SAMPLE SIZE vs CLASSIFICATION ACCURACY\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAa60lEQVR4nO3df5Ac5X3n8fdHKzEb8cNIaI9yECAlhxPt6rhgNkpyV7Y41xURpA6CcByUcx3K+eA2ifauSuASCKrgRCFCjHyXYO6mwFFsXCURTMU2qdgQjpWKk8u6YzFIWFASMucYCZdZWyIuSyckpO/9MS1ptOyPlqZn5tmez6tqqnqe7un+bPfsd599uqdHEYGZmZXXtHYHMDOz5nKhNzMrORd6M7OSc6E3Mys5F3ozs5Kb3u4Ao82ZMyfmzZvX7hhmZlPKSy+99JOI6BlrXnKFft68eQwPD7c7hpnZlCLpH8ab56EbM7OSc6E3Mys5F3ozs5JzoTczKzkXejOzkpu00EtaL+kdSd8bZ74k/YWk3ZK2S/po3bybJb2RPW4uMnjKBgcH6e7uRhLd3d0MDg62O5KZdbA8PfovAUsmmH8NcFn2uBX4HwCSZgP3AL8BLALukTSrkbBTweDgINVqlbVr13LgwAHWrl1LtVp1sTeztpm00EfEC8C+CRa5Hng8arYC50v6MPDbwHMRsS8i9gPPMfEfjFJ47LHHePDBB1m5ciUzZ85k5cqVPPjggzz22GPtjmZmHaqIMfqLgLfqnu/J2sZr/wBJt0oaljQ8MjJy2gEkTfpolffee4+BgYFT2gYGBnjvvfdasv0U9kWeDKnkSCFDKjlSyJBKjhQyFJkjiZOxEfFoRPRHRH9Pz5if4J3s9ac8xmtrhUqlQrVaPaWtWq1SqVRasv0U9sXo7aWSI4UMqeRIIUMqOVLI0OwcRdwCYS9wcd3zuVnbXuCqUe2bC9he0m655RZWrVoF1Hry1WqVVatWfaCXb2bWKkUU+qeBFZKeoHbi9R8j4keSngXW1p2AvRq4s4DtJe3hhx8GYPXq1dx2221UKhUGBgZOtJuZtZom+/dA0kZqPfM5wI+pXUkzAyAiqqoNJH2B2onWg8AfRsRw9tp/D6zOVnV/RPzVZIH6+/uj0ZuaSWrpcE3KUtkXKeRIIUMqOVLIkEqOFDIUkUPSSxHRP9a8SXv0EbFskvkB/Mk489YD6/OENDOz5kjiZKyZmTWPC72ZWcm50JuZlZwLvZlZybnQm5mVnAu9mVnJudCbmZWcC72ZWcm50JuZlZwLvZlZybnQm5mVnAu9mVnJudCbmZWcC72ZWcm50JuZlZwLvZlZybnQm5mVnAu9mVnJudCbmZWcC72ZWcm50JuZlZwLvZlZybnQF0RSrkfRZs+ePen2Jss0e/bswnN1qsmOR55jUsTxaPR9Uab3RAr7ot3vi+l5FpK0BPhzoAv4YkT86aj5lwLrgR5gH/DpiNiTzfsz4Heo/VF5DvjPERFnnDhRY/1IksZsL9L+/fsb3kajf4Bmz57N/v37G9rOrFmz2LdvX0M5UpDC8SgiRzM6Je2Swr5o9/ti0h69pC7gEeAaoBdYJql31GIPAY9HxOXAGuCB7LX/AviXwOXAQuDXgcVnnNaSdPxN3Mgjzx8KMzszeYZuFgG7I+LNiDgMPAFcP2qZXmAom95UNz+AbuAsoALMAH7caGgzM8svT6G/CHir7vmerK3eNmBpNn0DcK6kCyLiO9QK/4+yx7MR8froDUi6VdKwpOGRkZHT/RnMzGwCRZ2MvR1YLOllakMze4Gjkv4psACYS+2PwyckfWz0iyPi0Yjoj4j+np6egiKZmRnkOxm7F7i47vncrO2EiHibrEcv6Rzgxoh4V9ItwNaI+Hk271vAbwH/q4DsZmaWQ54e/YvAZZLmSzoLuAl4un4BSXMkHV/XndSuwAH4IbWe/nRJM6j19j8wdGNmZs0zaaGPiPeBFcCz1Ir0kxGxQ9IaSddli10F7JS0C7gQuD9rfwr4PvAqtXH8bRHxt8X+CGY1KVwvbSe1+9pxO0mpXdLe398fw8PDDa2jFdevp5KjiG00uo4UMhSxjhQypLKOFDKkso4UMuRZh6SXIqJ/rHn+ZKyZWcm50JuZlZwLvZlZybnQm5mVnAu9mVnJudCbmZWcC72ZWcm50JuZlZwLvZlZybnQm5mVnAu9mVnJudCbmZWcC72ZWcm50JuZlZwLvZlZybnQm5mVnAu9mVnJudCbmZWcC72ZWclNye+MnT17Nvv3729oO7NmzWLfvn1n/PoUMkAa32eZQoYi1lHId/ze+6HGXn9iPf/Y/hwNZkjhmAJJ7ItWvC8m+s7YKVnoU3gDpZAhlXWkkAFI4hc6lX2RwjFN5Y9eCvtisnWMHBzhsy98locWP8ScX5hzRuuYqNBPP828ZsnSf/lZ47/Q9xaXp9M1ejygc45JdXuV7/74u1S3Vbn7N+8ufP0eozcza6ORgyN8Y/c3CIKv7/46P/l/Pyl8G7kKvaQlknZK2i3pjjHmXyrpeUnbJW2WNLdu3iWS/l7S65JekzSvuPhmZlNbdXuVY3EMgGNxjOq2auHbmLTQS+oCHgGuAXqBZZJ6Ry32EPB4RFwOrAEeqJv3OPC5iFgALALeKSK42VQ0cnCE5c8sb0qvzU5fu4/H8d78kWNHADhy7EhTevV5evSLgN0R8WZEHAaeAK4ftUwvMJRNbzo+P/uDMD0ingOIiJ9HxMFCkptNQfVjsdZ+7T4e9b3545rRq89T6C8C3qp7vidrq7cNWJpN3wCcK+kC4CPAu5L+RtLLkj6X/Ydg1nFaMRZr+aVwPLa9s+1Eb/64I8eO8Mo7rxS6naKuurkd+IKk5cALwF7gaLb+jwFXAD8E/hpYDvxl/Ysl3QrcCnDJJZcUFMksLWONxTbjCoupIs8lhc2UwvF46rqnWrKdPD36vcDFdc/nZm0nRMTbEbE0Iq4A7sra3qXW+38lG/Z5H/g68NHRG4iIRyOiPyL6e3p6zvBHMUtXq8Zip5J2Dpt02vHIU+hfBC6TNF/SWcBNwNP1C0iaI+n4uu4E1te99nxJx6v3J4DXGo9tNrW0aix2qmj3sEmnHY9JC33WE18BPAu8DjwZETskrZF0XbbYVcBOSbuAC4H7s9cepTas87ykVwEBjxX+U5glrlVjsXm0+0oTaM0lhRNJ6Xi0gm+B0KbXl2kdKWSYbB1FfMS80QyprOO+rffx1Z1f5VO/8qlxx6SbmWHk4AjX/M01vHf0vRNtla4Kz9z4zAeOTQr7M4UMedYx0S0Q/MlYa7pUepC+rLH9QybQecMmKfC9bqxhcc95E97AqnrBLL577jlUv9jP3T8d+46fcc95zYr3geI28M8H2nKVRwpSuNKk04ZNUuChmza9vhXryHv5WjP3Rf2/6eP9e15EhonWcd/W+/jaG1/jyLEjzJg2g6WXLR2zuE2FY9rIOqbakEkq60ghQ551eOimQ6UwXNHuk26ddhndRDxk0rlc6EsqhbHYFIqsi9tJHjLpXB6jbxJ/6m/iItuqLC5uJ7XqU5iWHhf6Jmn2FwlMZLyedKtPQqZQZF3czFzom6LdV3mk0JMGF1mzVHiMvgnafQIyhZ60maXDPfqCpTBs4p60mdVzj75gvsrDzFLjQl8wD5uYWWo8dFMwD5uYWWpK16NP4QZaZmYpKV2hT+Fj/2ZmKZmSQzfj3S1xpGsa35j7i8S0aXz99Y0MPLeOOUePjbGGxu+WONkdG3Ovo0Gp5JDU0OtnzZrVcIZGc6SQIZUcKWRIJUcKGRrNUaq7V+a9S+FE6yhSK7YxFTKkkiOFDKnkSCFDKjlSyFBEjo64e2UKN9AyM0tRaQq9r183MxtbaQq9r183MxvblDwZOxZfv25mNrbS9OjNzGxsLvRmZiXnQm9mVnK5Cr2kJZJ2Stot6Y4x5l8q6XlJ2yVtljR31PzzJO2R9IWigpuZWT6TFnpJXcAjwDVAL7BMUu+oxR4CHo+Iy4E1wAOj5t8HvNB4XDMzO115evSLgN0R8WZEHAaeAK4ftUwvMJRNb6qfL+lK4ELg7xuPa2ZmpytPob8IeKvu+Z6srd42YGk2fQNwrqQLJE0D1gG3T7QBSbdKGpY0PDIyki+5mZnlUtTJ2NuBxZJeBhYDe4GjwB8D34yIPRO9OCIejYj+iOjv6ekpKJKZmUG+D0ztBS6uez43azshIt4m69FLOge4MSLelfRbwMck/TFwDnCWpJ9HxAdO6JqZWXPkKfQvApdJmk+twN8E/EH9ApLmAPsi4hhwJ7AeICL+bd0yy4F+F3kzs9aadOgmIt4HVgDPAq8DT0bEDklrJF2XLXYVsFPSLmonXu9vUl4zMztNpbof/enw/eg7L0cKGVLJkUKGVHKkkKGIHB1xP3ozMxubC72ZWcm50JuZlZwLvZlZybnQm5mVnAu9mVnJudCbmZWcC72ZWcm50JuZlZwLvZlZyeW5qVmSJDX0+lmzZhWUxEYb79iMbk/hY+dmnWBKFvrJCkQq967oVN73ZmmZkoU+Re7FmlmqXOgL4gJuZqnyyVgzs5JzoTczKzkXejOzknOhNzMrORd6M7OSc6E3Mys5F3ozs5JzoTczKzkXejOzkstV6CUtkbRT0m5Jd4wx/1JJz0vaLmmzpLlZ+69J+o6kHdm83y/6BzAzs4lNWugldQGPANcAvcAySb2jFnsIeDwiLgfWAA9k7QeBfxcRfcAS4L9JOr+o8GZmNrk8PfpFwO6IeDMiDgNPANePWqYXGMqmNx2fHxG7IuKNbPpt4B2gp4jgZmaWT55CfxHwVt3zPVlbvW3A0mz6BuBcSRfULyBpEXAW8P3RG5B0q6RhScMjIyN5s5uZWQ5FnYy9HVgs6WVgMbAXOHp8pqQPA18B/jAijo1+cUQ8GhH9EdHf0+MOv5lZkfLcpngvcHHd87lZ2wnZsMxSAEnnADdGxLvZ8/OAvwPuioitRYQ2M7P88vToXwQukzRf0lnATcDT9QtImiPp+LruBNZn7WcBX6N2ovap4mKbmVlekxb6iHgfWAE8C7wOPBkROyStkXRdtthVwE5Ju4ALgfuz9k8BHweWS3ole/xa0T+EmZmNT6l9M1J/f38MDw83tA5/Z+xJ3hcnpbIvUsiRQoZUcqSQoYgckl6KiP6x5vmTsWZmJedCb2ZWci70ZmYl50LfBIODg3R3dyOJ7u5uBgcH2x3JzDqYC33BBgcHqVarrF27lgMHDrB27Vqq1aqLvZm1ja+6KVh3dzdr165l5cqVJ9o+//nPs3r1ag4dOtTyPKlcUZCCVPZFCjlSyJBKjhQyFJFjoqtuXOgLJokDBw4wc+bME20HDx7k7LPPbkumVN7E7SBp0mU69Zi0I0Oe4wGtPyYpHI8icvjyyhaqVCpUq9VT2qrVKpVKpU2JOldETPqw1slzPHxMmiPPvW7sNNxyyy2sWrUKgIGBAarVKqtWrWJgYKDNycysU7nQF+zhhx8GYPXq1dx2221UKhUGBgZOtJuZtZrH6EvO+yI9KRyTFDKkIpV94TF6MzM7Yy70ZmYl50JvZlZyLvRmZiXnQm9mVnIu9GZmJedCb2ZWci70ZmYl50JvZlZyLvRmZiXnQm9mVnIu9GZmJZer0EtaImmnpN2S7hhj/qWSnpe0XdJmSXPr5t0s6Y3scXOR4c3MbHKTFnpJXcAjwDVAL7BMUu+oxR4CHo+Iy4E1wAPZa2cD9wC/ASwC7pE0q7j4ZmY2mTw9+kXA7oh4MyIOA08A149aphcYyqY31c3/beC5iNgXEfuB54Aljcc2M7O88hT6i4C36p7vydrqbQOWZtM3AOdKuiDna5F0q6RhScMjIyN5s5uZWQ5FnYy9HVgs6WVgMbAXOJr3xRHxaET0R0R/T09PQZHMzAzyfZXgXuDiuudzs7YTIuJtsh69pHOAGyPiXUl7gatGvXZzA3nNzOw05enRvwhcJmm+pLOAm4Cn6xeQNEfS8XXdCazPpp8FrpY0KzsJe3XWZmZmLTJpoY+I94EV1Ar068CTEbFD0hpJ12WLXQXslLQLuBC4P3vtPuA+an8sXgTWZG1mZtYi/nLwkvO+SE8KxySFDKlIZV/4y8GnmI0bN7Jw4UK6urpYuHAhGzdubHektklhX6SQwU7lY9JiEZHU48orr4xG1X6s9tiwYUPMnz8/hoaG4vDhwzE0NBTz58+PDRs2tCVPp++LFDKM1s5jkkKG1I5JCscjovEcwHCMU1fbXthHP6Z6oe/r64uhoaFT2oaGhqKvr68teTp9X6SQYbQUCkunvy/qpXA8Ippb6EsxRi9p0mVa9XN2dXVx6NAhZsyYcaLtyJEjdHd3c/Ro7o8WnDHvi7Qy5Dke0Pxj4vfFSSnsi2a8L0o/Rj/eX7H6R6ssWLCALVu2nNK2ZcsWFixY0JLte1+klSHP8WjFMUkhw3FT4ZikkKHQHHk32KpHEUM37ZTa+GM7pbAvUshgp/IxaQ46bYy+3TZs2BB9fX0xbdq06Ovr6+g3cAr7IoUMdiofk+JNVOhLMUZvZtbpSj9Gb2Zm43OhNzMrORd6M7OSc6E3Mys5F3ozs5JzoTczKzkXejOzknOhNzMrORd6M7OSc6E3Mys5F3ozs5JzoTczKzkXejOzknOhNzMrORd6M7OSy1XoJS2RtFPSbkl3jDH/EkmbJL0sabuka7P2GZK+LOlVSa9LurPoH8DMzCY2aaGX1AU8AlwD9ALLJPWOWuxu4MmIuAK4CfjvWfvvAZWI+GfAlcB/lDSvmOhmZpZHnh79ImB3RLwZEYeBJ4DrRy0TwHnZ9IeAt+vaz5Y0HfgF4DDws4ZTm5lZbnkK/UXAW3XP92Rt9e4FPi1pD/BNYDBrfwo4APwI+CHwUETsG70BSbdKGpY0PDIycno/gZmZTaiok7HLgC9FxFzgWuArkqZR+2/gKPCLwHzgNkm/NPrFEfFoRPRHRH9PT09BkczMDPIV+r3AxXXP52Zt9T4DPAkQEd8BuoE5wB8Az0TEkYh4B/g2MOaX1xZhcHCQ7u5uJNHd3c3g4ODkL2qCjRs3snDhQrq6uli4cCEbN27syAyQxjFJZV+kkCOFDCnl6BgRMeEDmA68Sa1HfhawDegbtcy3gOXZ9AJqY/QCVgF/lbWfDbwGXD7R9q688so4EytWrIjp06fHunXr4sCBA7Fu3bqYPn16rFix4ozWd6Y2bNgQ8+fPj6GhoTh8+HAMDQ3F/PnzY8OGDR2VISKNY5LKvkghRwoZUspRNsBwjFfHx5sRpxbya4FdwPeBu7K2NcB12XQvtd76NuAV4Oqs/Rzgq8COrMh/drJtnWmhr1QqsW7dulPa1q1bF5VK5YzWd6b6+vpiaGjolLahoaHo6+vrqAwRaRyTVPZFCjlSyJBSjrKZqNCrNj8d/f39MTw8fNqvk8SBAweYOXPmibaDBw9y9tln08qfsauri0OHDjFjxowTbUeOHKG7u5ujR492TAZI45iksi9SyJFChpRylI2klyJizKHx0nwytlKpUK1WT2mrVqtUKpWW5liwYAFbtmw5pW3Lli0sWLCgozJAGscklX2RQo4UMqSUo6OM19Vv18Nj9OXIEJHGMUllX6SQI4UMKeUoGxodo2/l40wLfUStsFQqlQCiUqm0vMgft2HDhujr64tp06ZFX19fW97AKWSISOOYpLIvUsiRQoaUcpTJRIW+NGP0ZmadrCPG6M3MbGwu9GZmJedCb2ZWci70ZmYl50JvZlZyyV11I2kE+IcGVzMH+EkBcRqVQo4UMkAaOVLIAGnkSCEDpJEjhQzQeI5LI2LM2/8mV+iLIGl4vMuMOi1HChlSyZFChlRypJAhlRwpZGh2Dg/dmJmVnAu9mVnJlbXQP9ruAJkUcqSQAdLIkUIGSCNHChkgjRwpZIAm5ijlGL2ZmZ1U1h69mZllXOjNzEpuShd6SUsk7ZS0W9IdY8z/uKTvSnpf0ifblGGlpNckbZf0vKRL25RjQNKrkl6RtEVSb6sz1C13o6SQ1JRLyXLsi+WSRrJ98Yqk/9DqDNkyn8reGzskbSg6Q54ckv5r3X7YJendNmS4RNImSS9nvyfXFp0hZ45Ls9/R7ZI2S5rbhAzrJb0j6XvjzJekv8gybpf00UI2PN79i1N/AF3UvsP2lzj5peW9o5aZB1wOPA58sk0Z/hUwM5v+I+Cv25TjvLrp64BnWp0hW+5c4AVgK9Dfpn2xHPhCm9+blwEvA7Oy5/+kHTlGLT8IrG/DvngU+KNsuhf4QZuOyVeBm7PpTwBfaUKOjwMfBb43zvxrgW8BAn4T+N9FbHcq9+gXAbsj4s2IOAw8AVxfv0BE/CAitgPH2phhU0QczJ5uBQrvJeTM8bO6p2cDRZ+FnzRD5j7gQeBQwds/3RzNlCfDLcAjEbEfICLeaVOOesuAjW3IEMB52fSHgLcLzpA3Ry8wlE1vGmN+wyLiBWDfBItcDzweNVuB8yV9uNHtTuVCfxHwVt3zPVlbyhk+Q+2vdVtySPoTSd8H/gz4T63OkP0benFE/F3B2z6tHJkbs3+Nn5J0cRsyfAT4iKRvS9oqaUnBGfLmAGrDFsB8Tha6Vma4F/i0pD3AN6n9Z1G0PDm2AUuz6RuAcyVd0IQsE2lKXZvKhX5KkfRpoB/4XLsyRMQjEfHLwCrg7lZuW9I04PPAba3c7jj+FpgXEZcDzwFfbkOG6dSGb66i1pN+TNL5bchx3E3AUxFxtA3bXgZ8KSLmUhu6+Er2fmm124HFkl4GFgN7gXbsj8JN5UK/F6jvic3N2pLLIOlfA3cB10XEe+3KUecJ4HdbnOFcYCGwWdIPqI0/Pt2EE7KT7ouI+GndcfgicGWrM1DrqT0dEUci4v8Cu6gV/lbnOO4mih+2yZvhM8CTABHxHaCb2g2+WpojIt6OiKURcQW131ciovCT05NoTl0r+mRDqx7UekRvUvt38/jJlb5xlv0SzTkZO2kG4ApqJ4Eua+e+qN8+8G+Y4IuEm308suU305yTsXn2xYfrpm8AtrYhwxLgy9n0HGr/rl/QjmMC/CrwA7IPULZhX3wLWJ5NL6A2Rl9olpw55gDTsun7gTVF749s3fMY/2Ts73Dqydj/U8g2m/GDtOpB7d+8XVkhvStrW0Ot5wzw69R6TgeAnwI72pDhfwI/Bl7JHk+3aV/8ObAjy7BpoiLcrAyjlt1MEwp9zn3xQLYvtmX74lfbkEHUhrJeA14FbmrHvsie3wv8aTO2n3Nf9ALfzo7HK8DVbcrxSeCNbJkvApUmZNgI/Ag4ktWmzwADwEDd++KRLOOrRf2O+BYIZmYlN5XH6M3MLAcXejOzknOhNzMrORd6M7OSc6E3Mys5F3ozs5JzoTczK7n/D1wgZNYVpfKgAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SGqjNhhyyZRw"
      },
      "source": [
        "- In this case, the results suggest that using a bootstrap sample size that is equal to the size of the training dataset achieves the best results on this dataset.\n",
        "- A box and whisker plot is created for the distribution of accuracy scores for each bootstrap sample size.\n",
        "- In this case, we can see a general trend that the larger the sample, the better the performance of the model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "evdVvGFvzT9P"
      },
      "source": [
        "**Explore Number of Features**\n",
        "\n",
        "- The number of features that is randomly sampled for each split point is perhaps the most important feature to configure for random forest.\n",
        "\n",
        "- It is set via the max_features argument and defaults to the square root of the number of input features. In this case, for our test dataset, this would be sqrt(30) or about ~5 features (will take all 30 features for now).\n",
        "\n",
        "The example below explores the effect of the number of features randomly selected at each split point on model accuracy. We will try values from 1 to 30 and would expect a small value, around four, to perform well based on the heuristic."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 823
        },
        "id": "A-iEkP8_yI1e",
        "outputId": "3b02f073-4ae3-4a31-fe76-687bdeb3a859"
      },
      "source": [
        "# explore random forest number of features effect on performance\n",
        "from numpy import mean\n",
        "from numpy import std\n",
        "from sklearn.datasets import make_classification\n",
        "from sklearn.model_selection import cross_val_score\n",
        "from sklearn.model_selection import RepeatedStratifiedKFold\n",
        "from sklearn.ensemble import RandomForestClassifier\n",
        "from matplotlib import pyplot\n",
        "\n",
        "print(\"Number of Feature vs Accuracy\")\n",
        "# get a list of models to evaluate\n",
        "def get_models():\n",
        "\tmodels = dict()\n",
        "\t# explore number of features from 1 to 30\n",
        "\tfor i in range(1,30):\n",
        "\t\tmodels[str(i)] = RandomForestClassifier(max_features=i)\n",
        "\treturn models\n",
        " \n",
        "# evaluate a given model using cross-validation\n",
        "def evaluate_model(model, X, y):\n",
        "\t# define the evaluation procedure\n",
        "\tcv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)\n",
        "\t# evaluate the model and collect the results\n",
        "\tscores = cross_val_score(model, X, y, scoring='accuracy', cv=cv, n_jobs=-1)\n",
        "\treturn scores\n",
        " \n",
        "# get the models to evaluate\n",
        "models = get_models()\n",
        "# evaluate the models and store results\n",
        "results, names = list(), list()\n",
        "for name, model in models.items():\n",
        "\t# evaluate the model\n",
        "\tscores = evaluate_model(model, X, y)\n",
        "\t# store the results\n",
        "\tresults.append(scores)\n",
        "\tnames.append(name)\n",
        "\t# summarize the performance along the way\n",
        "\tprint('>%s %.3f (%.3f)' % (name, mean(scores), std(scores)))\n",
        "\n",
        "print(\"BOX PLOT of RANDOM FOREST FEATURE SET vs CLASSIFICATION ACCURACY\")\n",
        "# plot model performance for comparison\n",
        "pyplot.boxplot(results, labels=names, showmeans=True)\n",
        "pyplot.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Number of Feature vs Accuracy\n",
            ">1 0.961 (0.022)\n",
            ">2 0.960 (0.027)\n",
            ">3 0.960 (0.026)\n",
            ">4 0.961 (0.029)\n",
            ">5 0.961 (0.024)\n",
            ">6 0.958 (0.031)\n",
            ">7 0.964 (0.027)\n",
            ">8 0.962 (0.027)\n",
            ">9 0.961 (0.027)\n",
            ">10 0.963 (0.028)\n",
            ">11 0.964 (0.027)\n",
            ">12 0.961 (0.029)\n",
            ">13 0.962 (0.028)\n",
            ">14 0.964 (0.027)\n",
            ">15 0.962 (0.027)\n",
            ">16 0.962 (0.027)\n",
            ">17 0.960 (0.025)\n",
            ">18 0.960 (0.027)\n",
            ">19 0.959 (0.027)\n",
            ">20 0.959 (0.027)\n",
            ">21 0.962 (0.026)\n",
            ">22 0.957 (0.024)\n",
            ">23 0.958 (0.026)\n",
            ">24 0.960 (0.028)\n",
            ">25 0.959 (0.028)\n",
            ">26 0.958 (0.027)\n",
            ">27 0.960 (0.028)\n",
            ">28 0.958 (0.027)\n",
            ">29 0.957 (0.025)\n",
            "BOX PLOT of RANDOM FOREST FEATURE SET vs CLASSIFICATION ACCURACY\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXsAAAD4CAYAAAANbUbJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3df5RU9Znn8fdDA90i/mikYxRUmBOd5ecksZckuxqMmTjg7IEVJ1nJZKMzbBg2A5tZNasGxx941JjAbmYcJ300MImZI47DToTZMSqxcRz2mIxNlEbCoOAa+aGhDRhHiIDw7B/3NhbVVfdHdd3uW9zP65w6fevep7713Pu996lb31tdZe6OiIic2IYMdgIiIpI9FXsRkQJQsRcRKQAVexGRAlCxFxEpgKGDnUC50aNH+7hx4wY7DRGRhrJhw4Y33b2t2vLcFftx48bR1dU12GmIiDQUM/t51HIN44iIFICKvYhIAajYi4gUgIq9iEgBqNiLiBRAbLE3sxVmtsfMXqyy3Mzsz81sm5l1m9lHS5ZdbWYvh7er65m4iDSWRYsW0dLSgpnR0tLCokWLBjulQklyZv9dYEbE8pnA+eFtPvBtADMbBdwKfAyYBtxqZq39SVZEGtOiRYvo6OjgrrvuYv/+/dx11110dHSo4A+g2GLv7s8AeyNCZgMPeuDHwOlmdhbwO8Bad9/r7vuAtUS/aIjICeqBBx7gnnvu4dprr2XEiBFce+213HPPPTzwwAODnVph1GPMfgywo+T+znBetfl9mNl8M+sys66enp5Ky4+7RUkTm6e2y+Mbte0TtX8ate3y+Czbjoo/ePAgCxYsOG7eggULOHjwYF1yadT+ybrvS+XiAq273+/u7e7e3tbW9799e39gxd2J+7GV0pgkP8ySl7bL4we77Ubchmo7Pj5JbBa5NDc309HRcdy8jo4Ompub+912lnk3ctvl6vF1CbuAc0rujw3n7QIuKZv/dB2eT0QazJe+9CVuuOEGIDij7+jo4IYbbuhzti/ZqUexXwMsNLOHCS7G/srdXzezJ4C7Si7KXgbcVIfnE5EGc++99wLwta99jeuuu47m5mYWLFhwbL5kL7bYm9lKgjP00Wa2k+ATNsMA3L0DeAy4HNgGHAD+IFy218zuAJ4Lm1ri7lEXekXkBHbvvfequA+i2GLv7nNjljvwx1WWrQBW1JaaiIjUSy4u0IqISLZU7EVECkDFXkSkAFTsRUQKQMVeRKQAVOxFRApAxV5EpABU7EVECkDFXkSkAFTsRUQKQMVeRKQAVOxFRApAxV5EpABU7EVECkDFXkSkAFTsRUQKQMVeRKQAVOxFRApAxV5EpABU7EVECkDFXkSkAFTsRUQKINfFftSoUZgZZgZwbHrUqFH9js+y7dL4Rm27ND7Ltgejf4rS91lKm3dpTOnj6tF2nvo+jSzbrsTcPT7IbAbwZ0AT8B13/3rZ8vOAFUAbsBf4grvvDJd9A/hdgheWtcBXPOJJ29vbvaurq7ddKoXWY36WbQ/GczZq21qf7NoeNWoU+/bt6zO/tbWVvXv3JoqvFps27zQxjdoPMDDbMOJxG9y9vdrjhlZt8f0GmoD7gM8AO4HnzGyNu/+sJGwp8KC7f8/MLgXuBv6zmf074N8DU8O49cB04Om45xWR/tm3b1/VYpI0PuoMXPrK8zZMMowzDdjm7q+4+yHgYWB2WcxEoDOcXley3IEWYDjQDAwDftHfpEVEJJ0kxX4MsKPk/s5wXqmNwJxw+grgFDM7w92fJSj+r4e3J9x9S/kTmNl8M+sys66enp606yAiIjHqdYH2emC6mT1PMEyzCzhiZh8CJgBjCV4gLjWzi8sf7O73u3u7u7e3tbXVKSUREekVO2ZPULjPKbk/Npx3jLvvJjyzN7ORwJXu/paZfQn4sbu/Ey77IfAJ4J/qkLuIiCSU5Mz+OeB8MxtvZsOBq4A1pQFmNtrMetu6ieCTOQCvEZzxDzWzYQRn/X2GcUREJFuxxd7d3wMWAk8QFOpH3H2zmS0xs1lh2CXAVjN7CTgTuDOcvwrYDmwiGNff6O5/X99VEBE58fX3fyySDOPg7o8Bj5XNu6VkehVBYS9/3BHgjxJnIyIiFfX3Y525/g9aERGpDxV7EZECULEXESkAFXsRkQJQsRcRKQAVexGRAlCxFxEpABV7EZECULEXESkAFXsRkQJQsRcRKQAVexGRAlCxFxEpABV7EZECULEXESkAFXsRkQJQsRcRKQAV+wbVc6CHax6/hjd//eZgp5JaI+cu0qhU7BtUR3cHP/3FT+nY2DHYqaSWl9z1oiNFYuW/aTjY2tvbvaurCwh+X7FSftXmc9tp1Ru+7Ve1x9Yzvg5t99zRysyxZ3NwyBCajx7l8Z27GX3kaO7z5rbT6Gkakiz3lG2n2lfCtu84o5W/PWUkn/vXd7j5l/si806TS5ptWLd9fIDbTrtNRo0axb59+/rMb21tZe/evf1quyjHfaV+K51nZhvcvb1akydUsU8zP8u2s37Os794Nh/8zAc5fPQww4YMY875c/jTT/xpZBs9B3r46jNfZen0pYw+aXRd807T9pJnl/CDl38Qm3ut2ypJLmbGnv17mPl3Mzl45CDNTc08fuXjtI1oG7Rt2J/5jdB2msLWCOuTq20Ybr+4Yt8QwziN+nY7i7x7DvTQenErh48eBuDw0cM8uu1Rhp42NPJxWQ6dJG176GlDWb1tdercs8ilo7uDo34UgKN+NNF2SbMNG3WfTSvpetrtbwdFqexmt789QJk2vkrbMM32a4hin5cxXkh3EGeRd0d3B9jx8476UdpmtVV9TM+BHlZvW43jPLrt0US5J13PNG23zWo7VmDT5J50eyfNpZYXnbTbME/7bJaKsp4ngkTF3sxmmNlWM9tmZjdWWH6emT1lZt1m9rSZjS1Zdq6ZPWlmW8zsZ2Y2Lk2CtRSqtO2nOQNLexZb77w37tnIkGHHd9vho4cZ8aERkTlndRabpu0RHxpxrMCmyT1pMUmaSy0vOmnWM6u+75WXdw1ZH5u15JOH7ZJXscXezJqA+4CZwERgrplNLAtbCjzo7lOBJcDdJcseBL7p7hOAacCeNAnWUqiyOvuu9Sw2ad5JrJq1iheveZFNV2867rb91u0V4+txFlstNm3b22/d3ifvqNzTbO80uaR90Um7nrX0/WC/Y6xFLcdm1vnkYbvk9UUnyZn9NGCbu7/i7oeAh4HZZTETgc5wel3v8vBFYai7rwVw93fc/UDS5God403a6bW8NU+yc9c6rp6FepzFVoutpe000hSTNLmkfdFJ03bW11RqedeQtvgkiR+I6y9ppD2WsyzIeXnRKZek2I8BdpTc3xnOK7URmBNOXwGcYmZnABcAb5nZ35nZ82b2zfCdQiK1jvEm7fRa3pon2blrGVfvzb3eO2A9zmJbL26tmFMtwzJJpS0mWeaSpu2sr6nU8q4hbfFJEp/1Cz2kf7eTZrtkVZCzHsLrj3q9DF8P/IWZXQM8A+wCjoTtXwx8BHgN+BvgGmB56YPNbD4wH+Dcc889Nr/WMd4knd6ft+a9qu3ctYyr9+beuwPe/PGbI2OT2n7r9qof7+KWvvGV1hOjYk5p204jbTHJMpc0bWd5TaWWdw3lLyQLfmtB1dhK8YPx4tor6fGQ9lhOu016H1P6sdtqah3CS9J2fyUp9ruAc0rujw3nHePuuwnP7M1sJHClu79lZjuBF9z9lXDZo8DHKSv27n4/cD8En7PvnZ/2AE7T6WmLSZqde9WsVVU/b1ut8NSyA2ah0noOGTaEF/a8MOh51LuYZCFt32f9jrGWM94kQ3hZvrhCuuMh7bHc3w8sVHvhqccQXtxJXn9eGJIU++eA881sPEGRvwr4fGmAmY0G9rr7UeAmYEXJY083szZ37wEuBbpSZZhCmk5PW0yy3rnzcrGr0nqaGZt806Dn0ZtLPbZ3XmT5jjHtGW/UEF69zjiTFqusPuVVjw8sVHvhqccQXtxJXn/e/ccWe3d/z8wWAk8ATcAKd99sZkuALndfA1wC3G1mTjCM88fhY4+Y2fXAU2ZmwAbggVQZppCm0/NUTPJ2sUsGTpbvGNOe8aYZwqtVkmJVy6e8kh7LWX7sNuuPRZdfD0j77j9RNXH3x4DHyubdUjK9ClhV5bFrgampsqpRngp4GgNxsUvyKct9Nu2716yH8JKexWZ5PGT5sdssh/Cg/x/n1qljDjTq+LTkW9oXkqyH8JKexWZ5PNTjAwv1euEZiI/0lsp1sfdbT6345T9+66mDkE12GvUdSZ406r6SZd552iZpzmLzdDw08kd6y+lbLzNoezCes1Hb1voUo+3Sb2rtNWzIMN5Y+wa7H9zdcOsz0M/5e2t+j637tvaJ/fXPf822W7b1Pi7yWy9zfWYvIicGDVX2Ty0f5y6nYi8imcvT0ExRNcRXHIuISP+o2IuIFICKvYhIAajYi4gUgIq9iEgBqNiLiBSAir2ISAGo2IuIFICKvYhIAajYi4gUQO6/LiH4zZPjtba21iU+y7Yrxavt/PSP+n7w246LTyPLvu9vLnlpO9fFvvS7NKp9Q1yt8Vm2XRrfqG0niVfb0fGN2naS+Ky3YZqCnGXeaaXpn964cnHrWWveuS72IlI8WRfkvBjo9dSYvYhIAajYi4gUgIq9iEgBqNiLiBSAir2ISAGo2IuIFECiYm9mM8xsq5ltM7MbKyw/z8yeMrNuM3vazMaWLT/VzHaa2V/UK3EREUkuttibWRNwHzATmAjMNbOJZWFLgQfdfSqwBLi7bPkdwDP9T1dERGqR5Mx+GrDN3V9x90PAw8DsspiJQGc4va50uZldCJwJPNn/dEVEpBZJiv0YYEfJ/Z3hvFIbgTnh9BXAKWZ2hpkNAZYB10c9gZnNN7MuM+vq6elJlrmIiCRWrwu01wPTzex5YDqwCzgCfBl4zN13Rj3Y3e9393Z3b29ra6tTSiIi0ivJd+PsAs4puT82nHeMu+8mPLM3s5HAle7+lpl9ArjYzL4MjASGm9k77t7nIq+IiGQnSbF/DjjfzMYTFPmrgM+XBpjZaGCvux8FbgJWALj775fEXAO0q9CLiAy82GEcd38PWAg8AWwBHnH3zWa2xMxmhWGXAFvN7CWCi7F3ZpSviIjUwPL29aHt7e3e1dXVZ37arwBNE6+2852L2h7YtvOUy2C3XS0m7rGDkbeZbXD39mqP03/QiogUgIq9iEgBqNiLiBSAir2ISAGo2IuIFICKvYhIAajYi4gUgIq9iEgBqNiLiBSAir2ISAEk+SI0kROSmR33N29fHXKi6d3OvdONsr1L8+7V2to6CJn0j4q9FFajFJsTRSNu79KcG+kFqhIN44iIFICKvYhIAajYi4gUgIq9iEgBqNiLiBSAir2ISAGo2IuIFICKvYhIAajYi4gUgIq9iEgBqNiLiBRAomJvZjPMbKuZbTOzGyssP8/MnjKzbjN72szGhvM/bGbPmtnmcNl/qvcKiIhIvNhib2ZNwH3ATGAiMNfMJpaFLQUedPepwBLg7nD+AeCL7j4JmAF8y8xOr1fyIiKSTJIz+2nANnd/xd0PAQ8Ds8tiJgKd4fS63uXu/pK7vxxO7wb2AG31SFxERJJLUuzHADtK7u8M55XaCMwJp68ATjGzM0oDzGwaMBzYXv4EZjbfzLrMrKunpydp7iIiklC9LtBeD0w3s+eB6cAu4EjvQjM7C/g+8AfufrT8we5+v7u3u3t7W5tO/EVE6i3Jj5fsAs4puT82nHdMOEQzB8DMRgJXuvtb4f1TgX8AFrv7j+uRtIiIpJPkzP454HwzG29mw4GrgDWlAWY22sx627oJWBHOHw78gODi7ar6pS0iImnEFnt3fw9YCDwBbAEecffNZrbEzGaFYZcAW83sJeBM4M5w/ueATwLXmNkL4e3D9V4JERGJZnn7TcX29nbv6urqMz/t7z+miVfb+c4l6/VMIy95N3Lfp5GnvPPetpltcPf2ao/Tf9CKiBSAir2ISAGo2IuIFICKvYgMiKlTp2Jmx25Tp04d7JQKRcVeRDI3depUNm3axKxZs+jp6WHWrFls2rRJBX8AqdiLSOZ6C/3q1asZPXo0q1evPlbwZWA0RLE3s2N/e6cbQdq8S+OzzCXLthupf4oiTd+X9mHafTYufvny5ZH3+yNt3lm2neXx0J+2G6LYu/txt0aRNu8s13Gg2m6k/imKWvfBesfPmzcv8n5/ZLkPZrlNss6lVEMUexFpbFOmTGHNmjXMnj2bN998k9mzZ7NmzRqmTJky2KkVRpIvQhMR6Zfu7m6mTp3KmjVr6P1m2ylTptDd3T3ImRWHir2IDAgV9sGlYRwRkQJQsRcRKQAVexGRAlCxFxEpABV7EZECULEXESkAFXsRkQJQsRcRKQAVexGRAlCxFxEpABV7EZECULEXESmARMXezGaY2VYz22ZmN1ZYfp6ZPWVm3Wb2tJmNLVl2tZm9HN6urmfyIiKSTGyxN7Mm4D5gJjARmGtmE8vClgIPuvtUYAlwd/jYUcCtwMeAacCtZtZav/RFRCSJJGf204Bt7v6Kux8CHgZml8VMBDrD6XUly38HWOvue919H7AWmNH/tEVEJI0kxX4MsKPk/s5wXqmNwJxw+grgFDM7I+FjMbP5ZtZlZl09PT1JcxcRkYTqdYH2emC6mT0PTAd2AUeSPtjd73f3dndv7/0VGxERqZ8kv1S1Czin5P7YcN4x7r6b8MzezEYCV7r7W2a2C7ik7LFP9yNfERGpQZIz++eA881svJkNB64C1pQGmNloM+tt6yZgRTj9BHCZmbWGF2YvC+eJiMgAii327v4esJCgSG8BHnH3zWa2xMxmhWGXAFvN7CXgTODO8LF7gTsIXjCeA5aE80REZACZuw92Dsdpb2/3rq6ufrdjZiRdtzSxtcRnJes8smy/UfsnL3nnZR/MWl76Mmv1yMXMNrh7e7Xl+g9aETlm5cqVTJ48maamJiZPnszKlSsHO6WGkuftl+QCrYgUwMqVK1m8eDHLly/noosuYv369cybNw+AuXPnDnJ2+Zf77efuubpdeOGFXg/BqtU/tpb4rGSdR5btN2r/5CXvLNZx0qRJ3tnZedy8zs5OnzRpUt2fK6m89GUS/dl+9cgF6PKI2nrCjdmb2XH349YvTXzatrOUZS55ajsv/ZOnvLNaz6amJt59912GDRt2bN7hw4dpaWnhyJHE/zZTN3nqzyRq2X71zKNwY/blr2b1jE/bdpayzCVPbeelf/KUd1brOWHCBNavX3/cvPXr1zNhwoS6PUcaeerPJGrZfgNZU064Yi8itVm8eDHz5s1j3bp1HD58mHXr1jFv3jwWL1482Kk1hLxvP12gFRHg/YuIixYtYsuWLUyYMIE777wzHxcXG0Det98JN2YvIlJEhRuzFxGRvlTsRUQKQMVeRKQAVOxFRApAxV5EpABU7EVECkDFXkSkAFTsRUQKQMVeRKQAVOxFRApAxV5EpABU7EVECkDFXkSkAFTsRUQKQMVeRKQAEhV7M5thZlvNbJuZ3Vhh+blmts7MnjezbjO7PJw/zMy+Z2abzGyLmd1U7xUQEZF4scXezJqA+4CZwERgrplNLAu7GXjE3T8CXAX8ZTj/s0Czu08BLgT+yMzG1Sd1ERFJKsmZ/TRgm7u/4u6HgIeB2WUxDpwaTp8G7C6Zf7KZDQVOAg4Bb/c7axERSSVJsR8D7Ci5vzOcV+o24AtmthN4DFgUzl8F7AdeB14Dlrr73vInMLP5ZtZlZl09PT3p1kBERGLV6wLtXOC77j4WuBz4vpkNIXhXcAQ4GxgPXGdmv1H+YHe/393b3b29ra2tTimJiEivJMV+F3BOyf2x4bxS84BHANz9WaAFGA18Hnjc3Q+7+x7g/wJVfxB3oK1cuZLJkyfT1NTE5MmTWblyZV3j01i0aBEtLS2YGS0tLSxatCj+QQllmbfaljxQfybg7pE3YCjwCsGZ+XBgIzCpLOaHwDXh9ASCMXsDbgD+Kpx/MvAzYGrU81144YU+EB566CEfP368d3Z2+qFDh7yzs9PHjx/vDz30UF3i01i4cKEPHTrUly1b5vv37/dly5b50KFDfeHChf1uO8u81bbkgfozAHR5VC2PWujvF/PLgZeA7cDicN4SYFY4PZHgrH0j8AJwWTh/JPC3wOaw0H817rkGqthPmjTJOzs7j5vX2dnpkyZNqkt8Gs3Nzb5s2bLj5i1btsybm5v73XaWeattyQP1ZyCu2FsQkx/t7e3e1dWV+fM0NTXx7rvvMmzYsGPzDh8+TEtLC0eOHOl3fBpmxv79+xkxYsSxeQcOHODkk0+mv/2TZd5qW/JA/Rkwsw3uXnWYvLD/QTthwgTWr19/3Lz169czYcKEusSn0dzcTEdHx3HzOjo6aG5u7nfbWeattiUP1J8JRZ32D8ZNY/Yas2/ktmXgqT8D1GPMfiBvA1Xs3YOdZNKkST5kyBCfNGlS7M6RNj6NhQsXenNzswPe3Nxcl0LfK8u81bbkgfpTY/YiIoWgMXsREVGxFxEpAhV7EZECULEXESkAFXsRkQLI3adxzKwH+HmFRaOBN1M0lSZebec7F7U9sG3nKRe1nTz2PHev/rXBUZ/LzNONmM+Q9idebec7F7Wtvlfbtbfde9MwjohIAajYi4gUQCMV+/szjFfb/Y9X2ydO22nj1Xa+2wZyeIFWRETqr5HO7EVEpEYq9iIiRVDLR3gG8gasAPYALyaIPQdYR/ATiJuBr8TEtwD/TPBzipuB2xM8RxPwPPB/EsS+Cmwi+KnG2I9LAacDq4B/AbYAn6gS95thm723t4E/iWn7v4fr+CKwEmiJiP1KGLe5UruV+gQYBawFXg7/tkbEfjZs+yjQnqDtb4bbpBv4AXB6TPwdYewLwJPA2XH7EnAd4MDomLZvA3aVbPvLo9oGFoW5bwa+EdP235S0+yrwQkTsh4Ef9+5bwLSYtn8LeDbcH/8eODXqmKnUnxGxFfszIr5if0bE9+nParHV+jOi7T79GdV2pf6MaLtPf0bEVuzPiPiK/RlZA+ICBvsGfBL4KMmK/VnAR8PpUwh+N3diRLwBI8PpYcBPgI/HPMe1wEMkL/aj4+JK4r8H/JdwejglRS3iMU3AGwT/UFEtZgzw/4CTwvuPEP5AfIXYyQSFfgTBj83/CPhQXJ8A3wBuDKdvBO6JiJ1A8IL1NH2LfaX4y4Ch4fQ9vW1HxJ9aMv3fgI6ofSk8oJ4g+Ge+0TFt3wZcn2Q/BT4Vbr/m8P4Hku7XwDLgloi2nwRmhtOXA0/H5PIcMD2c/kPgjqhjplJ/RsRW7M+I+Ir9GRHfpz+rxVbrz4i2+/RnRGzF/ozKpbw/I9qu2J8R8RX7M+qW+2Ecd38G2Jsw9nV3/2k4/a8EZ8djIuLd3d8J7w4Lb1WvWJvZWOB3ge8kyz45MzuN4CBdHuZ2yN3fSvDQTwPb3b3Sfx2XGgqcZGZDCQr57ipxE4CfuPsBd38P+EdgTmlAlT6ZTfBiRfj3P1aLdfct7r610pNXiX8yzAWCs5+xMfFvl9w9mbBPI/al/wX8D8r6PuW+Vyn2vwJfd/eDYcyeJG2bmQGfI3gHVi3WgVPD6dMo6c8q8RcAz4TTa4Erw9hqx0yf/qwWW60/I+Ir9mdEfJ/+jDnW+/RnmtoQEVuxP+PaLu3PiNiK/RkRX7E/o+S+2NfKzMYBHyE4W4+KazKzFwje9q5196j4bxHsREcTpuHAk2a2wczmx8SOB3qAvzKz583sO2Z2coLnuIqwKFRNwn0XsBR4DXgd+JW7P1kl/EXgYjM7w8xG8P7b2jhnuvvr4fQbwJkJHlOLPwR+GBdkZnea2Q7g9wnOqKrFzQZ2ufvGFDksNLNuM1thZq0RcRcQbMufmNk/mtm/Tdj+xcAv3P3liJg/Ab4ZruNS4KaYNjcTFHAIhl369GnZMRPZn0mPrwTxFfuzPD6qP0tjk/RnhVyq9mdZbGx/VlnPiv1ZFhvbn2Xxsf3ZR9ypfx5uwDgSDOOUxI8ENgBzUjzmdIKxsclVlv8H4C/D6UtINowzJvz7AYLrAp+MiG0H3gM+Ft7/M2LemhEM9bxJcGBGxbUCnUAbwbuXR4EvRMTPC7ffM8C3gW/F9QnwVtnyfXH9R4VhnJj4xQRjvJZ0/yA4aG6vFEvwDucnwGnh/VcpG3arsJ5nEgydDQHuBFZExL4I3EswXDiNYCjNEqznt4HrYvL4c+DKcPpzwI9i4v8NwVDBBuBW4JdRx0xMf1Y8viL6s1p8tf6sevxW6M9jsQn7s3w9o/qzPDauP6utZ6X+LG87rj/L4yP7s+KxEBeQh1u1g6JK7DCC8bpra3ieW6gwHhsuuxvYGe5AbwAHgL9O0fZt1doOl38QeLXk/sXAP8S0ORt4MsFzfxZYXnL/i4QvXAkeexfw5bg+AbYCZ4XTZwFb4/qPFMUeuIbggtSINPsHcG5ZnsdigSkE7+heDW/vEbz7+WDCtsu3Qfn9x4FPldzfDrTFrOdQ4BfA2Jjn+hXv/5+MAW+n2CYXAP8cdcxU689KsVH9WS2+Wn9GtV/en+Wxcf2ZoO1x1dqO68+I9ezTn1XartqfCfI+rj+r3U6oYZxwbGw5sMXd/2eC+DYzOz2cPgn4DMGV9j7c/SZ3H+vu4wiGTjrd/QsRbZ9sZqf0ThNclHqxWry7vwHsMLPfDGd9muAKfJS5xAzhhF4DPm5mI8Jt9GmCsb9quX8g/HsuwRnTQwmeYw1wdTh9NbA6wWMSMbMZBMNns9z9QIL480vuzqZ6n25y9w+4+7iwX3cSXAx7I6Lts0ruXkFEnxK8g/pU+LgLeP+dWJTfBv7F3XfGxO0GpofTlxJ8aqaqkj4dAtxMcJEz6pjp0581HF8V46v1Z0R8n/6sFBvVnxFt9+nPiPWs2J8x2+W4/oyIrdifEXlX7M9Ica8Gg30jKGavA4fDzpsXEXsRwTh578e0jn00rkr8VIKPUXYTHLS3JMzpEmKGcYDfIBi66f1Y5+IE7X6Y4GNX3eGO1RoRezLwS8K3rAnavp2g6L0IfJ/wEwVVYv+J4IVmI/DpJH0CnAE8Fe6kPwJGRcReEU4fJDjreSKm7W3AjrAAGzAAAAC3SURBVJI+7YiJ/9/henYTfCxtTLXYsvV6leM/jVOp7e8TfNytm6AgnhUROxz46zCXnwKXxu3XwHeBBQm290UEb+E3EgxdXBgT/xWCT3K8BHyd988iKx4zlfozIrZif0bEV+zPiPg+/Vkttlp/RrTdpz8jYiv2Z1Qu5f0Z0XbF/oyIr9ifUTd9XYKISAGcUMM4IiJSmYq9iEgBqNiLiBSAir2ISAGo2IuIFICKvYhIAajYi4gUwP8Hd8w/wM4ani8AAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QsYzTr-UzmFM"
      },
      "source": [
        "- In this case, the results suggest that a value of four on this dataset. \n",
        "- A box and whisker plot is created for the distribution of accuracy scores for each feature set size.\n",
        "- We can see a trend in performance rising and peaking with value 7."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xAMwnNms37aL"
      },
      "source": [
        "**Explore Number of Trees**\n",
        "- The number of trees is another key hyperparameter to configure for the random forest.\n",
        "- Typically, the number of trees is increased until the model performance stabilizes. Intuition might suggest that more trees will lead to overfitting, although this is not the case. \n",
        "- Both bagging and random forest algorithms appear to be somewhat immune to overfitting the training dataset given the stochastic nature of the learning algorithm.\n",
        "- The number of trees can be set via the “n_estimators” argument and defaults to 100.\n",
        "\n",
        "The example below explores the effect of the number of trees with values between 10 to 1,000."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 391
        },
        "id": "_LvdImLOzlFR",
        "outputId": "5c3ee2db-8b53-47ac-ceb8-259b98ecbb47"
      },
      "source": [
        "# explore random forest number of trees effect on performance\n",
        "from numpy import mean\n",
        "from numpy import std\n",
        "from sklearn.datasets import make_classification\n",
        "from sklearn.model_selection import cross_val_score\n",
        "from sklearn.model_selection import RepeatedStratifiedKFold\n",
        "from sklearn.ensemble import RandomForestClassifier\n",
        "from matplotlib import pyplot\n",
        "\n",
        "print(\"Number of Trees vs Model Accuracy\")\n",
        "# get a list of models to evaluate\n",
        "def get_models():\n",
        "\tmodels = dict()\n",
        "\t# define number of trees to consider\n",
        "\tn_trees = [10, 50, 100, 500, 1000]\n",
        "\tfor n in n_trees:\n",
        "\t\tmodels[str(n)] = RandomForestClassifier(n_estimators=n)\n",
        "\treturn models\n",
        "\n",
        "# evaluate a given model using cross-validation\n",
        "def evaluate_model(model, X, y):\n",
        "\t# define the evaluation procedure\n",
        "\tcv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)\n",
        "\t# evaluate the model and collect the results\n",
        "\tscores = cross_val_score(model, X, y, scoring='accuracy', cv=cv, n_jobs=-1)\n",
        "\treturn scores\n",
        "\n",
        "# get the models to evaluate\n",
        "models = get_models()\n",
        "# evaluate the models and store results\n",
        "results, names = list(), list()\n",
        "for name, model in models.items():\n",
        "\t# evaluate the model\n",
        "\tscores = evaluate_model(model, X, y)\n",
        "\t# store the results\n",
        "\tresults.append(scores)\n",
        "\tnames.append(name)\n",
        "\t# summarize the performance along the way\n",
        "\tprint('>%s %.3f (%.3f)' % (name, mean(scores), std(scores)))\n",
        "\n",
        "print(\"BOX PLOT OF RANDOM FOREST ENSEMBLE SIZE vs CLASSIFICATION ACCURACY\")\n",
        "# plot model performance for comparison\n",
        "pyplot.boxplot(results, labels=names, showmeans=True)\n",
        "pyplot.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Number of Trees vs Model Accuracy\n",
            ">10 0.950 (0.027)\n",
            ">50 0.965 (0.029)\n",
            ">100 0.962 (0.028)\n",
            ">500 0.960 (0.028)\n",
            ">1000 0.960 (0.027)\n",
            "BOX PLOT OF RANDOM FOREST ENSEMBLE SIZE vs CLASSIFICATION ACCURACY\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAT3klEQVR4nO3dcYxd5Znf8e/PBkIVSGJjF7EYsKNSCZuidJl1sm1T0/2DtbMSCFB38XYVSFd1pV2q/lGqghKJ1Cii6lKpXZUu67YuZSVALGoSVpsNoRiUUiUS4wWTEGpw0O5iQ8NEhqxSmtrYT/+YY+5l8Mxce67n3nnn+5GufO/7njvznIfDb84999xzU1VIktq1YtQFSJLOLINekhpn0EtS4wx6SWqcQS9JjTtr1AXMtGbNmlq/fv2oy5CkJWXv3r0/rqq1J5sbu6Bfv349k5OToy5DkpaUJH8+25yHbiSpcQa9JDXOoJekxhn0ktQ4g16SGjdv0CfZneStJN+fZT5JfjfJgSQvJvn5vrlbkrza3W4ZZuGSpMEMskf/ALB1jvltwOXdbQfwewBJVgN3AZ8GNgN3JVm1kGIlSadu3qCvqm8Dh+dY5HrgwZr2XeATSS4Cfhl4sqoOV9XbwJPM/QdDknQGDOMDUxcDr/c9PtiNzTb+IUl2MP1qgEsvvXQIJc0vyYJ/RivX8rcX04bRB7AX/exFzyh7MRafjK2qXcAugImJiUXpxnxNT9LERjoIezFtkHW0Fz32omfcezGMs24OAZf0PV7Xjc02LklaRMMI+seBz3dn33wG+ElVvQk8AVybZFX3Juy13ZgkaRHNe+gmycPANcCaJAeZPpPmbICquh/4BvA54ADwLvCFbu5wkruB57oftbOq5npTV5J0Bswb9FW1fZ75An57lrndwO7TK02SNAx+MlaSGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoG/c6tWrSbKgG7Dgn7F69eoRd8Je9LMXPcuhF2cNslCSrcC/A1YC/6mq/tWM+cuA3cBa4DDwG1V1sJv718CvMP1H5Ungn1ZVDW0NNKe3336bcWj3if8ZRsle9NiLnuXQi3n36JOsBO4DtgEbge1JNs5Y7F7gwaq6CtgJ3NM9928Bfxu4CrgS+AVgy9CqlyTNa5BDN5uBA1X1WlUdAR4Brp+xzEZgT3f/6b75As4FzgE+ApwN/GihRUuSBjdI0F8MvN73+GA31m8fcGN3/wbg/CQXVNV3mA7+N7vbE1X18sxfkGRHkskkk1NTU6e6DpKkOQzrzdjbgS1Jnmf60Mwh4FiSvwZcAaxj+o/DLyX57MwnV9Wuqpqoqom1a9cOqSRJEgz2Zuwh4JK+x+u6sfdV1Rt0e/RJzgNuqqp3kvwj4LtV9dNu7k+AXwT+xxBqlyQNYJA9+ueAy5NsSHIOcDPweP8CSdYkOfGz7mT6DByAv2B6T/+sJGczvbf/oUM3kqQzZ96gr6r3gNuAJ5gO6Uer6qUkO5Nc1y12DbA/ySvAhcBXuvHHgB8C32P6OP6+qvqj4a6CJGkuGYfzR/tNTEzU5OTkqMsgyVicW7tQ47Ie41DHONQwLnWMQw3jUsc41DCMOpLsraqJk835yVhJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BrTlPvTnHrN2/lx//3x6MuRdJpMug1p/tfvJ8//dGfcv+++0ddiqTTZNBrVlPvTvH1A1+nKL524Gvu1UtLlEGvWd3/4v0cr+MAHK/j7tVLS1ST3xm7evVq3n777SFVdPpWrVrF4cOHR1vElz9+Wk+bWrmCbet+jv+3orcv8JHjx/nmwTdYc+z4adbyk9N73rCcZi/OCHvRYy96FtCLub4ztsmgb+XLfkdZw93fvZuvvvpVjh4/+v7Y2SvO5sbLb+RLn/nSotUxTONQw7jUMQ41jEsd41DDMOrwy8F1yva9te8DIQ9w9PhRXnjrhRFVNHqegaSl6qxRF6Dx9Nh1j426hLHTfwbS6byqkUZloD36JFuT7E9yIMkdJ5m/LMlTSV5M8kySdX1zlyb5VpKXk/wgyfrhlS8tDs9A+iBf3fQshV7MG/RJVgL3AduAjcD2JBtnLHYv8GBVXQXsBO7pm3sQ+J2qugLYDLw1jMKlxeQZSB/k5yt6lkIvBtmj3wwcqKrXquoI8Ahw/YxlNgJ7uvtPn5jv/iCcVVVPAlTVT6vq3aFULi2SE3vzJ96zOHr86LLeq/fVTc9S6cUgQX8x8Hrf44PdWL99wI3d/RuA85NcAPx14J0k/y3J80l+p3uFIC0Z/XvzJyznvXpf3fQslV4M66yb24EtSZ4HtgCHgGNMv9n72W7+F4BPArfOfHKSHUkmk0xOTU0NqSRpODwDqcdXNz1LqReDnHVzCLik7/G6bux9VfUG3R59kvOAm6rqnSQHgReq6rVu7mvAZ4D/POP5u4BdMH0e/emtinRmeAZSz1yvbpbbmUhLqReD7NE/B1yeZEOSc4Cbgcf7F0iyJsmJn3UnsLvvuZ9IsrZ7/EvADxZetqRR8NVNz1Lqxbx79FX1XpLbgCeAlcDuqnopyU5gsqoeB64B7klSwLeB3+6eeyzJ7cBTSQLsBf7jmVkVSWear256llIvvATCGTQOdYxDDeNSxzjUMC51jEMN41LHONQwjDq8BIIkLWMGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQX8SS+H60pI0KIP+JJbC9aUlaVAG/QxL5frSkjQog36GpXJ9aUkalEHfZyldX1qSBmXQ9/GbhCS1yKDvs5SuLy1JgxrkG6aWjaV0fWlJGlSTQV93fQy+/PFRlzFdxxiY/s6X0Vq1atWoSwDsRT970dN6L5oM+vzLvxyfLxL48mhrGEYfxuWLGRbKXvTYi57l0AuP0UtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1LiBgj7J1iT7kxxIcsdJ5i9L8lSSF5M8k2TdjPmPJTmY5N8Pq3BJ0mDmDfokK4H7gG3ARmB7ko0zFrsXeLCqrgJ2AvfMmL8b+PbCy5UknapB9ug3Aweq6rWqOgI8Alw/Y5mNwJ7u/tP980muBi4EvrXwciVJp2qQoL8YeL3v8cFurN8+4Mbu/g3A+UkuSLIC+DfA7XP9giQ7kkwmmZyamhqscknSQIb1ZuztwJYkzwNbgEPAMeC3gG9U1cG5nlxVu6pqoqom1q5dO6SSJEkw2EXNDgGX9D1e1429r6reoNujT3IecFNVvZPkF4HPJvkt4DzgnCQ/raoPvaErSTozBgn654DLk2xgOuBvBn69f4Eka4DDVXUcuBPYDVBV/6BvmVuBCUNekhbXvIduquo94DbgCeBl4NGqeinJziTXdYtdA+xP8grTb7x+5QzVK0k6RRm3ayhPTEzU5OTkgn7GuFwbelzqWKhW1mMY7EWPvegZh14k2VtVEyeb85OxktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekho3yEXNlqQkoy6BVatWjboESWoz6IdxzYlxuHaFJA2Dh24kqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGDRT0SbYm2Z/kQJI7TjJ/WZKnkryY5Jkk67rxTyX5TpKXurlfG/YKSJLmNm/QJ1kJ3AdsAzYC25NsnLHYvcCDVXUVsBO4pxt/F/h8VW0CtgL/NsknhlW8JGl+g+zRbwYOVNVrVXUEeAS4fsYyG4E93f2nT8xX1StV9Wp3/w3gLWDtMAqXJA1mkKC/GHi97/HBbqzfPuDG7v4NwPlJLuhfIMlm4BzghzN/QZIdSSaTTE5NTQ1auyRpAMN6M/Z2YEuS54EtwCHg2InJJBcBfwB8oaqOz3xyVe2qqomqmli71h1+SRqmQb545BBwSd/jdd3Y+7rDMjcCJDkPuKmq3ukefwz4Y+CLVfXdYRQtSRrcIHv0zwGXJ9mQ5BzgZuDx/gWSrEly4mfdCezuxs8Bvsr0G7WPDa9sSdKg5g36qnoPuA14AngZeLSqXkqyM8l13WLXAPuTvAJcCHylG/9V4O8CtyZ5obt9atgrIUmaXcbte1EnJiZqcnJy1GX4nbF97EWPveixFz3j0Iske6tq4mRzfjJWkhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNW6QT8Y2KcmClxn16VTDYi+mDdKHQZZroRdqy7INev9n7LEX0+yDWuWhG0lqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekho3UNAn2Zpkf5IDSe44yfxlSZ5K8mKSZ5Ks65u7Jcmr3e2WYRYvSZrfvEGfZCVwH7AN2AhsT7JxxmL3Ag9W1VXATuCe7rmrgbuATwObgbuSrBpe+ZKk+QyyR78ZOFBVr1XVEeAR4PoZy2wE9nT3n+6b/2Xgyao6XFVvA08CWxdetiRpUIME/cXA632PD3Zj/fYBN3b3bwDOT3LBgM8lyY4kk0kmp6amBq1dkjSAYb0ZezuwJcnzwBbgEHBs0CdX1a6qmqiqibVr1w6pJEkSDPbl4IeAS/oer+vG3ldVb9Dt0Sc5D7ipqt5Jcgi4ZsZzn1lAvZKkUzTIHv1zwOVJNiQ5B7gZeLx/gSRrkpz4WXcCu7v7TwDXJlnVvQl7bTcmSVok8wZ9Vb0H3MZ0QL8MPFpVLyXZmeS6brFrgP1JXgEuBL7SPfcwcDfTfyyeA3Z2Y5KkRZKqGnUNHzAxMVGTk5OjLkPSPJIwbvkxKuPQiyR7q2riZHN+MlaSGmfQS1LjDHpJatwgp1dKWmaSDGW5UR+3HoYWemHQS/qQFgJ6WFrohYduJKlxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1bqCgT7I1yf4kB5LccZL5S5M8neT5JC8m+Vw3fnaS/5rke0leTnLnsFdAkjS3eYM+yUrgPmAbsBHYnmTjjMW+BDxaVX8TuBn4D9343wc+UlV/A7ga+MdJ1g+ndEnSIAbZo98MHKiq16rqCPAIcP2MZQr4WHf/48AbfeMfTXIW8FeAI8BfLrhqSdLABgn6i4HX+x4f7Mb6fRn4jSQHgW8A/6Qbfwz4P8CbwF8A91bV4Zm/IMmOJJNJJqempk5tDSRJcxrWm7HbgQeqah3wOeAPkqxg+tXAMeDngA3AP0vyyZlPrqpdVTVRVRNr164dUkmSJBgs6A8Bl/Q9XteN9ftN4FGAqvoOcC6wBvh14JtVdbSq3gL+JzCx0KLPpIcffpgrr7ySlStXcuWVV/Lwww+PuqSRsRc99qLHXvQsmV5U1Zw34CzgNab3yM8B9gGbZizzJ8Ct3f0rmD5GH+BfAP+lG/8o8APgqrl+39VXX12j8tBDD9WGDRtqz549deTIkdqzZ09t2LChHnrooZHVNCr2osde9NiLnnHrBTBZs+X4bBP1wSD/HPAK8EPgi93YTuC67v5GpvfW9wEvANd24+cBfwi81IX8P5/vd40y6Ddt2lR79uz5wNiePXtq06ZNI6podOxFj73osRc949aLuYI+0/PjY2JioiYnJ0fyu1euXMnPfvYzzj777PfHjh49yrnnnsuxY8dGUtOo2Isee9FjL3rGrRdJ9lbVSQ+N+8nYPldccQXPPvvsB8aeffZZrrjiihFVNDr2osde9NiLniXVi9l29Ud18xj9eLAXPfaix170jFsvWOgx+sW8jTLoq6b/423atKlWrFhRmzZtWpYb8An2osde9NiLnnHqxVxB7zF6SWqAx+glaRkz6CWpcQa9JDXOoJekxhn0ktS4sTvrJskU8OejroPpi7L9eNRFjAl70WMveuxFzzj04rKqOunlf8cu6MdFksnZTlVabuxFj73osRc9494LD91IUuMMeklqnEE/u12jLmCM2Isee9FjL3rGuhceo5ekxrlHL0mNM+glqXEGPZBkd5K3kny/b2x1kieTvNr9u2qUNS6WJH+W5HtJXkgy2Y0tm16cyraQab+b5ECSF5P8/OgqH75T2RZa68WwtoMkt3TLv5rkllGsCxj0JzwAbJ0xdgfwVFVdDjzVPV4u/l5VfarvvODl1IsHGHxb2AZc3t12AL+3SDUupkG3hdZ68QAL3A6SrAbuAj4NbAbuGtlO0mwXql9uN2A98P2+x/uBi7r7FwH7R13jIvXhz4A1M8aWVS8G3RaA3we2n2y5Fm6nsi202IuFbgfAduD3+8Y/sNxi3tyjn92FVfVmd/9/AxeOsphFVMC3kuxNsqMbW669OGG29b8YeL1vuYPdWCtOZVtovRdw6us+Nj05axS/dKmpqkqyXM5D/TtVdSjJXwWeTPK/+ieXWS8+ZJmtv9vCLJbaurtHP7sfJbkIoPv3rRHXsyiq6lD371vAV5k+trgse9FntvU/BFzSt9y6bqwJp7gtNN2Lzqmu+9j0xKCf3ePAiXfJbwG+PsJaFkWSjyY5/8R94Frg+yzDXsww2/o/Dny+O+viM8BP+l7aL2mnsS0024s+p7ruTwDXJlnVvQl7bTe2+Eb9hsc43ICHgTeBo0wfR/tN4AKm31l/FfjvwOpR17kIffgksK+7vQR8sRtfNr04lW0BCHAf8EPge8DEqOsf1bbQWi+GtR0A/xA40N2+MKr18RIIktQ4D91IUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktS4/w/QR9J1gkzLEgAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7H7EThKB4bum"
      },
      "source": [
        "In this case, we can see that performance rises and stays flat after about 100 trees. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4MkDzc-q4oGY"
      },
      "source": [
        "**Explore Tree Depth**\n",
        "- A final interesting hyperparameter is the maximum depth of decision trees used in the ensemble.\n",
        "- By default, trees are constructed to an arbitrary depth and are not pruned. This is a sensible default, although we can also explore fitting trees with different fixed depths.\n",
        "- The maximum tree depth can be specified via the max_depth argument and is set to None (no maximum depth) by default.\n",
        "\n",
        "The example below explores the effect of random forest maximum tree depth on model performance."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 445
        },
        "id": "NLYGHLIc4Uhq",
        "outputId": "673decd3-237b-4aa1-a12e-4a9d288f9fb7"
      },
      "source": [
        "# explore random forest tree depth effect on performance\n",
        "from numpy import mean\n",
        "from numpy import std\n",
        "from sklearn.datasets import make_classification\n",
        "from sklearn.model_selection import cross_val_score\n",
        "from sklearn.model_selection import RepeatedStratifiedKFold\n",
        "from sklearn.ensemble import RandomForestClassifier\n",
        "from matplotlib import pyplot\n",
        "\n",
        "print(\"Tree Depth vs Model Accuracy\")\n",
        "# get a list of models to evaluate\n",
        "def get_models():\n",
        "\tmodels = dict()\n",
        "\t# consider tree depths from 1 to 7 and None=full\n",
        "\tdepths = [i for i in range(1,8)] + [None]\n",
        "\tfor n in depths:\n",
        "\t\tmodels[str(n)] = RandomForestClassifier(max_depth=n)\n",
        "\treturn models\n",
        "\n",
        "# evaluate a given model using cross-validation\n",
        "def evaluate_model(model, X, y):\n",
        "\t# define the evaluation procedure\n",
        "\tcv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)\n",
        "\t# evaluate the model and collect the results\n",
        "\tscores = cross_val_score(model, X, y, scoring='accuracy', cv=cv, n_jobs=-1)\n",
        "\treturn scores\n",
        "\n",
        "# get the models to evaluate\n",
        "models = get_models()\n",
        "# evaluate the models and store results\n",
        "results, names = list(), list()\n",
        "for name, model in models.items():\n",
        "\t# evaluate the model\n",
        "\tscores = evaluate_model(model, X, y)\n",
        "\t# store the results\n",
        "\tresults.append(scores)\n",
        "\tnames.append(name)\n",
        "\t# summarize the performance along the way\n",
        "\tprint('>%s %.3f (%.3f)' % (name, mean(scores), std(scores)))\n",
        " \n",
        "print(\" BOX PLOT OF RANDOM FOREST MAXIMUM TREE DEPTH vs CLASSIFICATION ACCURACY\")\n",
        "# plot model performance for comparison\n",
        "pyplot.boxplot(results, labels=names, showmeans=True)\n",
        "pyplot.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Tree Depth vs Model Accuracy\n",
            ">1 0.924 (0.032)\n",
            ">2 0.947 (0.031)\n",
            ">3 0.955 (0.026)\n",
            ">4 0.956 (0.030)\n",
            ">5 0.959 (0.027)\n",
            ">6 0.961 (0.026)\n",
            ">7 0.961 (0.027)\n",
            ">None 0.961 (0.026)\n",
            " BOX PLOT OF RANDOM FOREST MAXIMUM TREE DEPTH vs CLASSIFICATION ACCURACY\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAakUlEQVR4nO3df5RX9X3n8eeLAWfiz4BMPYljhLYmHSDG6CyNGxP8sVpIe/SI26w0cUPKhuW0zLZVN2rIiQYPWjfSk03qZo4WNKYBj6FR2Y0VPYIn5VS3DCJYoCjxtBHMytcF6woVRua9f3zv0C/DzHwvmTtzv/fyepzzPdzv597v576/X+A1n+/n3rlXEYGZmZXXmLwLMDOzkeWgNzMrOQe9mVnJOejNzErOQW9mVnJj8y6gv4kTJ8akSZPyLsPMrFA2btz4VkS0DrSu4YJ+0qRJdHd3512GmVmhSPqnwdZ56sbMrOQc9GZmJeegNzMrOQe9mVnJOejNzEqubtBLWi5pj6S/H2S9JH1H0k5JWyRdWLPuS5JeTR5fyrJwG32dnZ20tLQgiZaWFjo7O/MuycxSSDOifwiYOcT6WcB5yWM+8D0ASROA24HfBKYDt0saP5xiLT+dnZ10dXVx1113sX//fu666y66uroc9mYFUDfoI+KnwN4hNrkGeDiqXgA+KOlDwG8Bz0TE3ojYBzzD0D8wrIE98MAD3HPPPdx4442cfPLJ3Hjjjdxzzz088MADeZdmZnVkMUd/NvB6zfNdSdtg7ceQNF9St6TuSqWSQUmNT1LqRyM4ePAgCxYsOKptwYIFHDx4MKeKjlaUz7MIdR5Pja6zGHU2xMHYiLg/IjoioqO1dcDf4C2diDjmMVR73pqbm+nq6jqqrauri+bm5pwqOlpRPs8i1DlQLUWps179J2qdWVwCYTdwTs3ztqRtN3Bpv/bnMtif5eArX/kKt9xyC1AdyXd1dXHLLbccM8o3s8aTRdCvBhZKeoTqgdd/johfSFoD3FVzAPYq4LYM9mc5+O53vwvA1772NW666Saam5tZsGDBkXYza1yq91VB0kqqI/OJwJtUz6QZBxARXapOKv051QOtB4AvR0R38trfB76WdLUkIh6sV1BHR0ecqBc1k5T71EKZFOXzdJ3ZKUKNMDJ1StoYER0Dras7oo+IOXXWB/CHg6xbDixPU6SZmY2MhjgYa2ZmI8dBb2ZWcg56M7OSc9CbmZWcg97MrOQc9GZmJeegNzMrOQe9mVnJOejNzErOQW9mVnIOejOzknPQm5mVnIPezKzkHPRmZiXnoDczKzkHvZlZyTnozcxKzkFvZlZyDnozs5JLFfSSZkraIWmnpFsHWH+upGclbZH0nKS2mnX/TdJWSdslfSe5mfiIkXRcDxtao32WEyZMOK560mw3YcIE11mSOtPWeKLVWffm4JKagPuAK4FdwAZJqyNiW81m9wIPR8T3JV0O3A3cIOnfAp8Gzk+2Ww/MAJ4bduWDGOjO6irIneEbUaN9nvv27ct83yPxQ8p1Zst1Dk+aEf10YGdEvBYRh4BHgGv6bTMFWJssr6tZH0ALcBLQDIwD3hxu0WZmll6aoD8beL3m+a6krdZmYHayfC1wmqQzI+J5qsH/i+SxJiK299+BpPmSuiV1VyqV430PZmY2hKwOxt4MzJC0ierUzG7gsKRfB9qBNqo/HC6X9Jn+L46I+yOiIyI6WltbMyrJzMwgxRw91dA+p+Z5W9J2RES8QTKil3QqcF1EvC3pK8ALEfFusu6vgYuBv8mgdjMzSyHNiH4DcJ6kyZJOAq4HVtduIGmipL6+bgOWJ8s/pzrSHytpHNXR/jFTN2XXqEfizezEUDfoI+J9YCGwhmpIPxoRWyUtlnR1stmlwA5JrwBnAUuS9lXAz4CXqc7jb46I/5ntW2h8fUfis3zs27cv77dlZgWhRjvtsKOjI7q7uzPtM+/TK0di/3m+p7Lt2326zzL0KWljRHQMtM6/GWtmVnIOejOzknPQm5mVnIPezKzkHPRmZiXnoDczKzkHvZlZyTnozcxKzkFvZlZyDnozs5Jz0JuZlZyD3sys5Bz0ZmYl56A3Mys5B72ZWck56M3MSs43HhkFRbnBwYQJEzK/c9X48ePZu3dvZv0V5bPkjjOy7e9Iv/+ccX+uM9v+8qtzqBuPOOhHQVHCqQh9FqFG9+k+8+jTd5gys2GpHKgw96m5vPUvb+VdyqCKUCPkU2eqoJc0U9IOSTsl3TrA+nMlPStpi6TnJLXVrPuIpKclbZe0TdKk7Mo3s9HQtaWLF998ka7NXXmXMqgi1Aj51Fk36CU1AfcBs4ApwBxJU/ptdi/wcEScDywG7q5Z9zDwrYhoB6YDe7Io3MxGR+VAhSd2PkEQPL7z8YYcMRehRsivzjQj+unAzoh4LSIOAY8A1/TbZgqwNlle17c++YEwNiKeAYiIdyPiQCaVm5VAEaYburZ00Ru9APRGb0OOmItQI+RXZ5qgPxt4veb5rqSt1mZgdrJ8LXCapDOBjwJvS/qxpE2SvpV8QzAzGn+6oW8E2tPbA0BPb0/DjZiLUCPkW2dWB2NvBmZI2gTMAHYDh4GxwGeS9f8G+FVgbv8XS5ovqVtSd6VSyagks8ZWhOmG2hFon0YbMRehRsi3zjRBvxs4p+Z5W9J2RES8ERGzI+KTwKKk7W2qo/+Xkmmf94HHgQv77yAi7o+IjojoaG1t/SXfilmxFGG6YfOezUdGoH16ent4ac9LOVV0rCLUCPnWWfc8ekljgVeAK6gG/Abg9yJia802E4G9EdEraQlwOCK+kUzTvAj8u4ioSHoQ6I6I+wbbn8+jd5+j2V9efVYOVJj141kcPHzwSFtzUzNPXfcUEz8wsWHqdJ/F6XNY59EnI/GFwBpgO/BoRGyVtFjS1clmlwI7JL0CnAUsSV57mOq0zbOSXgYEPFC3YrNhavSDnEWZbrBySDVHHxFPRsRHI+LXIqIvxL8REauT5VURcV6yzX+KiIM1r30mIs6PiI9HxNzkzB0rqEYP0D6NfpCzKNMNVg6+BMIoKNNXxDtfuJMf7fgRn//Y5/n6p76eSZ/HJcW1RCpNY5jV9mEOjhlDc28vT+16g4mHe4d+UcbXPCnT37n7LEafQ03djM20Iiu1/meJLPjEgkHnk0eKvvlO3X/0XS/cSe+rj0FvD71jm+m68qYhfyhJIu7IuFCzBuJr3VhqRThLpCjnVJuNJge9pVKUAPVBTrNjOegtlaIEqA9ymh3Lc/SWSlECdNXVq/IuwazhOOgtFQeoWXF56sbMrOQc9GZmJeegNzMrOQd9gyjKpQXMrHgc9A2i0a/NYmbFVeizbiZMmMC+fftSbSsp1Xbjx49n7969wynrGHH76UNeo6XSNIYn2j5MjBnD49tXsuCZpXWvzRK3n55pjUf6THEtmePuM2Np/y7TGj9+fKb99XGd2XKdv7xCX9SsLBcluvOFO3ns1cfo6e1h3JhxzD5vdt0LhuV5oba8LxKXluvMVhHqLEKNMGI588tfj95GVlEuLWBmxeWgz1lRLi1gZsXloM9ZUS4tYGbFVeiDsWXgSwuY2UjziN7MrOQc9GZmJZcq6CXNlLRD0k5Jtw6w/lxJz0raIuk5SW391p8uaZekP8+qcDMzS6du0EtqAu4DZgFTgDmSpvTb7F7g4Yg4H1gM3N1v/Z3AT4dfrpmZHa80I/rpwM6IeC0iDgGPANf022YKsDZZXle7XtJFwFnA08Mv18zMjleaoD8beL3m+a6krdZmYHayfC1wmqQzJY0BlgI3D7UDSfMldUvqrlQq6So3M7NUsjoYezMwQ9ImYAawGzgM/AHwZETsGurFEXF/RHREREdra2tGJZmZGaQ7j343cE7N87ak7YiIeINkRC/pVOC6iHhb0sXAZyT9AXAqcJKkdyPimAO6ZmY2MtIE/QbgPEmTqQb89cDv1W4gaSKwNyJ6gduA5QAR8YWabeYCHQ55M7PRVXfqJiLeBxYCa4DtwKMRsVXSYklXJ5tdCuyQ9ArVA69LRqheMzM7Tr5Msfss1L6Ph+vMVhHqLEKN4MsUm5lZxhz0ZmYl56A3Mys5B72ZWck56M3MSs5Bb2ZWcg56M7OSc9CbmZVcoe8ZG7efDneckX2fI0BSpv2NHz8+0/4GM1jdA7UX4RdVzE5EhQ56ffOdkfmN0zsy7TJ1jY34W32NVo+ZHT9P3ZiZlZyD3sys5Bz0ZmYlV/qgrxyoMPepubz1L2/lXYqZWS5KH/RdW7p48c0X6drclXcpZma5KHXQVw5UeGLnEwTB4zsf96jezE5IpQ76ri1d9EYvAL3R61G9mZ2QShv0faP5nt4eAHp6ezyqN7MTUmmDvnY038ejejM7EaUKekkzJe2QtFPSrQOsP1fSs5K2SHpOUlvSfoGk5yVtTdb9h6zfwGA279l8ZDTfp6e3h5f2vDRaJZiZNYS6l0CQ1ATcB1wJ7AI2SFodEdtqNrsXeDgivi/pcuBu4AbgAPAfI+JVSR8GNkpaExFvZ/5O+ll19aqR3oWZWSGkGdFPB3ZGxGsRcQh4BLim3zZTgLXJ8rq+9RHxSkS8miy/AewBWrMo3MzM0kkT9GcDr9c835W01doMzE6WrwVOk3Rm7QaSpgMnAT/rvwNJ8yV1S+quVCppazczsxSyOhh7MzBD0iZgBrAbONy3UtKHgB8AX47od4QUiIj7I6IjIjpaWz3gNzPLUprLFO8Gzql53pa0HZFMy8wGkHQqcF3fPLyk04GfAIsi4oUsijYzs/TSjOg3AOdJmizpJOB6YHXtBpImSurr6zZgedJ+EvAY1QO1PjpqZpaDukEfEe8DC4E1wHbg0YjYKmmxpKuTzS4Fdkh6BTgLWJK0fx74LDBX0kvJ44Ks34SZmQ1OjXYHoY6Ojuju7k617UjckSnPuzw14h2miqwon6frzE4RaoQRy66NEdEx0LrS/masmZlVOejNzErOQW9mVnIOejOzknPQm5mVXJpfmGpokjLtb/z48Zn2Z2aWt0IHfdrTk4pyypWZ2Ujw1I2ZWck56M3MSs5Bb2ZWcg56M7OSc9CbmZWcg97MrOQc9GZmJeegNzMrOQe9mVnJOejNzErOQW9mVnIOejOzkksV9JJmStohaaekWwdYf66kZyVtkfScpLaadV+S9Gry+FKWxZuZWX11g15SE3AfMAuYAsyRNKXfZvcCD0fE+cBi4O7ktROA24HfBKYDt0vydYDNzEZRmhH9dGBnRLwWEYeAR4Br+m0zBVibLK+rWf9bwDMRsTci9gHPADOHX7aZmaWVJujPBl6veb4raau1GZidLF8LnCbpzJSvRdJ8Sd2SuiuVStrazcwshawOxt4MzJC0CZgB7AYOp31xRNwfER0R0dHa2ppRSWZmBunuMLUbOKfmeVvSdkREvEEyopd0KnBdRLwtaTdwab/XPjeMes3M7DilGdFvAM6TNFnSScD1wOraDSRNlNTX123A8mR5DXCVpPHJQdirkjYzMxsldYM+It4HFlIN6O3AoxGxVdJiSVcnm10K7JD0CnAWsCR57V7gTqo/LDYAi5M2K6CVK1cybdo0mpqamDZtGitXrsy7JDNLIdXNwSPiSeDJfm3fqFleBawa5LXL+dcRvhXUypUrWbRoEcuWLeOSSy5h/fr1zJs3D4A5c+bkXJ2ZDcW/GWupLFmyhGXLlnHZZZcxbtw4LrvsMpYtW8aSJUvyLs3M6lBE5F3DUTo6OqK7uzvTPiXRaO9zII1cZ1NTE++99x7jxo070tbT00NLSwuHD6c+wWrESEq9bZ6fcRHqPJ4awXXWM1p1StoYER0DrfOI3lJpb29n/fr1R7WtX7+e9vb2nCo6WkSkfrjO7Gp0ncWo00FvqSxatIh58+axbt06enp6WLduHfPmzWPRokV5l2ZmdaQ6GGvWd8C1s7OT7du3097ezpIlS3wg1qwAPEffQIpSp5k1Hs/Rm5mdwBz0ZmYl56A3Mys5B72ZWck56M3MSs5Bb2ZWcg56M7OSc9CbmZWcg97MrOQc9GZmJeegNzMrOQe9mVnJOejNzEouVdBLmilph6Sdkm4dYP1HJK2TtEnSFkmfS9rHSfq+pJclbZd0W9ZvwMzMhlY36CU1AfcBs4ApwBxJU/pt9nXg0Yj4JHA98D+S9t8FmiPi48BFwH+WNCmb0s3MLI00I/rpwM6IeC0iDgGPANf02yaA05PlM4A3atpPkTQW+ABwCHhn2FWbmVlqaYL+bOD1mue7krZadwBflLQLeBLoTNpXAfuBXwA/B+6NiL39dyBpvqRuSd2VSuX43oGZmQ0pq4Oxc4CHIqIN+BzwA0ljqH4bOAx8GJgM3CTpV/u/OCLuj4iOiOhobW3NqCTL2sqVK5k2bRpNTU1MmzaNlStX5l3SgDo7O2lpaUESLS0tdHZ21n9RDoryeVoJpLgj+cXAmprntwG39dtmK3BOzfPXgF+hOrd/Q037cuDzQ+3voosuiqxV32bja+Q6V6xYEZMnT461a9fGoUOHYu3atTF58uRYsWJF3qUdZeHChTF27NhYunRp7N+/P5YuXRpjx46NhQsX5l3aUYryeVpxAN0xWI4PtiL+NZzHJsE9GTgJ2AxM7bfNXwNzk+V2qnP0Am4BHkzaTwG2AecPtT8HfWOaOnVqrF279qi2tWvXxtSpU3OqaGDNzc2xdOnSo9qWLl0azc3NOVU0sKJ8nlYcQwV9qpuDJ6dLfhtoApZHxBJJi5OOVydn4TwAnEr1AOxXI+JpSacCD1I9W0dJ6H9rqH355uCNWWdTUxPvvfce48aNO9LW09NDS0sLhw8fzrGyo0li//79nHzyyUfaDhw4wCmnnNJQn21RPk8rjmHfHDwinoyIj0bEr0XEkqTtGxGxOlneFhGfjohPRMQFEfF00v5uRPxuREyNiCn1Qt4aV3t7O+vXrz+qbf369bS3t+dU0cCam5vp6uo6qq2rq4vm5uacKhpYUT5PK4nBhvp5PTx105iKMqfsOXo7UTGcOfrRfjjoG9eKFSti6tSpMWbMmJg6dWrDhtLChQujubk5gGhubm64kO9TlM/TimGooE81Rz+aPEff+HWaWeMZ9hy9mZkVl4PezKzkHPRmZiXnoDczKzkHvZlZyTnozcxKzkFvZlZyDnozs5Jz0JuZlZyD3sys5Bz0ZmYl56A3Mys5B72ZWck56M3MSs5Bb2ZWcg56M7OSSxX0kmZK2iFpp6RbB1j/EUnrJG2StCW5mXjfuvMlPS9pq6SXJbVk+QbMzGxoY+ttIKkJuA+4EtgFbJC0OiK21Wz2deDRiPiepCnAk8AkSWOBvwRuiIjNks4EejJ/F2ZmNqg0I/rpwM6IeC0iDgGPANf02yaA05PlM4A3kuWrgC0RsRkgIv5vRBweftlmZpZWmqA/G3i95vmupK3WHcAXJe2iOprvTNo/CoSkNZJelPTVgXYgab6kbkndlUrluN6AmZkNLauDsXOAhyKiDfgc8ANJY6hODV0CfCH581pJV/R/cUTcHxEdEdHR2to6rEIkHfMYrL1vXR6Op04zs+GoO0cP7AbOqXnelrTVmgfMBIiI55MDrhOpjv5/GhFvAUh6ErgQeHaYdQ8qIkaq60wVpU4zK740I/oNwHmSJks6CbgeWN1vm58DVwBIagdagAqwBvi4pJOTA7MzgG2YmdmoqTuij4j3JS2kGtpNwPKI2CppMdAdEauBm4AHJP0J1QOzc6M6ZN0n6c+o/rAI4MmI+MlIvRkzMzuWGm0KoaOjI7q7u/Muw8ysUCRtjIiOgdb5N2PNzErOQW9mVnIOejOzknPQm5mVnIPezKzkGu6sG0kV4J8y7nYi8FbGfY4E15kt15mtItRZhBphZOo8NyIGvLRAwwX9SJDUPdhpR43EdWbLdWarCHUWoUYY/To9dWNmVnIOejOzkjtRgv7+vAtIyXVmy3Vmqwh1FqFGGOU6T4g5ejOzE9mJMqI3MzthOejNzEqu1EEvabmkPZL+Pu9ahiLpHEnrJG2TtFXSH+Vd00AktUj6O0mbkzq/mXdNg5HUJGmTpP+Vdy2DkfSPkl6W9JKkhr1kq6QPSlol6R8kbZd0cd419SfpY8nn2Pd4R9Ifj+L+Q9LSmuc3S7pjtPZfT6mDHniI5M5XDe594KaImAJ8CvhDSVNyrmkgB4HLI+ITwAXATEmfyrmmwfwRsD3vIlK4LCIuaPBzv/878FRE/AbwCRrwc42IHcnneAFwEXAAeGwUSzgIzJY0cRT3mVqpgz4ifgrszbuOeiLiFxHxYrL8/6j+R+p/A/bcRdW7ydNxyaPhjuZLagN+G/iLvGspOklnAJ8FlgFExKGIeDvfquq6AvhZRGT9G/ZDeZ/qmTR/0n+FpEmS1kraIulZSR9J2h+S9B1JfyvpNUn/vuY1/1XShuQ1w/7mXOqgLyJJk4BPAv8730oGlkyJvATsAZ6JiEas89vAV4HevAupI4CnJW2UND/vYgYxmeptQR9MpsL+QtIpeRdVx/XAyhz2ex/wheSHY63vAt+PiPOBHwLfqVn3IeAS4HeAPwWQdBVwHjCd6jfniyR9djiFOegbiKRTgb8C/jgi3sm7noFExOHk63EbMF3StLxrqiXpd4A9EbEx71pSuCQiLgRmUZ2uG9Z/5hEyFrgQ+F5EfBLYD9yab0mDS+5rfTXwo9Hed/J/9mHgv/RbdTGwIln+AdVg7/N4RPRGxDbgrKTtquSxCXgR+A2qwf9Lc9A3CEnjqIb8DyPix3nXU0/y9X0djXcM5NPA1ZL+EXgEuFzSX+Zb0sAiYnfy5x6q88nT861oQLuAXTXf3FZRDf5GNQt4MSLezGn/3wbmAWm/9RysWVbNn3f3HXOIiF+PiGXDKcpB3wAkieoc6PaI+LO86xmMpFZJH0yWPwBcCfxDvlUdLSJui4i2iJhE9Sv82oj4Ys5lHUPSKZJO61umOoJruLPDIuL/AK9L+ljSdAWwLceS6plDPtM2AETEXuBRqmHf52+p/lsE+ALwN3W6WQP8fvINH0lnS/qV4dRV6qCXtBJ4HviYpF2S5tV7TU4+DdxAdfTZd3rY5/IuagAfAtZJ2gJsoDpH37CnLza4s4D1kjYDfwf8JCKeyrmmwXQCP0z+3i8A7sq5ngElPzCvBPL+RryU6mWI+3QCX04+vxuonhE2qIh4mupUz/OSXqb6Leq04RTkSyCYmZVcqUf0ZmbmoDczKz0HvZlZyTnozcxKzkFvZlZyDnozs5Jz0JuZldz/B2CnZVVDlAsBAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SxpNkzBK5g_0"
      },
      "source": [
        "- In this case, we can see that larger depth results in better model performance, with the default of no maximum depth achieving the best performance on this dataset.\n",
        "- A box and whisker plot is created for the distribution of accuracy scores for each configured maximum tree depth.\n",
        "- In this case, we can see a trend of improved performance with increase in tree depth, supporting the default of no maximum depth.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K_RgKyAm86iZ"
      },
      "source": [
        "**Important Hyperparameters**\n",
        "\n",
        "Hyperparameters are used in random forests to either enhance the performance and predictive power of models or to make the model faster.\n",
        "\n",
        "**Following hyperparameters increases the predictive power:**\n",
        "\n",
        "1. **n_estimators–** number of trees the algorithm builds before averaging the predictions.\n",
        "\n",
        "2. **max_features–** maximum number of features random forest considers splitting a node.\n",
        "\n",
        "3. **mini_sample_leaf–** determines the minimum number of leaves required to split an internal node.\n",
        "\n",
        "**Following hyperparameters increases the speed:**\n",
        "\n",
        "1. **n_jobs–** it tells the engine how many processors it is allowed to use. If the value is 1, it can use only one processor but if the value is -1 there is no limit.\n",
        "\n",
        "2. **random_state–** controls randomness of the sample. The model will always produce the same results if it has a definite value of random state and if it has been given the same hyperparameters and the same training data.\n",
        "\n",
        "3. **oob_score –** OOB means out of the bag. It is a random forest cross-validation method. In this one-third of the sample is not used to train the data instead used to evaluate its performance. These samples are called out of bag samples."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 841
        },
        "id": "4YAyRd658ZoP",
        "outputId": "71ea947f-0ff7-4c4a-8c9a-0d51a42c8c61"
      },
      "source": [
        "# Feature Importance\n",
        "\n",
        "print(\"Feature vs Importance\")\n",
        "# define the model\n",
        "model = RandomForestRegressor()\n",
        "# fit the model\n",
        "model.fit(X, y)\n",
        "# get importance\n",
        "importance = model.feature_importances_\n",
        "# summarize feature importance\n",
        "for i,v in enumerate(importance):\n",
        "\tprint('Feature: %0d, Score: %.5f' % (i,v))\n",
        "\n",
        "print(\"Plot Feature Importance\")\n",
        "pyplot.bar([x for x in range(len(importance))], importance)\n",
        "pyplot.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Feature vs Importance\n",
            "Feature: 0, Score: 0.00155\n",
            "Feature: 1, Score: 0.01837\n",
            "Feature: 2, Score: 0.00140\n",
            "Feature: 3, Score: 0.00505\n",
            "Feature: 4, Score: 0.00384\n",
            "Feature: 5, Score: 0.00152\n",
            "Feature: 6, Score: 0.00539\n",
            "Feature: 7, Score: 0.06155\n",
            "Feature: 8, Score: 0.00203\n",
            "Feature: 9, Score: 0.00225\n",
            "Feature: 10, Score: 0.00312\n",
            "Feature: 11, Score: 0.00265\n",
            "Feature: 12, Score: 0.00212\n",
            "Feature: 13, Score: 0.01419\n",
            "Feature: 14, Score: 0.00216\n",
            "Feature: 15, Score: 0.00267\n",
            "Feature: 16, Score: 0.00331\n",
            "Feature: 17, Score: 0.00332\n",
            "Feature: 18, Score: 0.00242\n",
            "Feature: 19, Score: 0.00487\n",
            "Feature: 20, Score: 0.09551\n",
            "Feature: 21, Score: 0.02931\n",
            "Feature: 22, Score: 0.29336\n",
            "Feature: 23, Score: 0.11481\n",
            "Feature: 24, Score: 0.00939\n",
            "Feature: 25, Score: 0.00274\n",
            "Feature: 26, Score: 0.00783\n",
            "Feature: 27, Score: 0.29573\n",
            "Feature: 28, Score: 0.00322\n",
            "Feature: 29, Score: 0.00430\n",
            "Plot Feature Importance\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAARMUlEQVR4nO3dcaxedX3H8fdnRXDRTWHcGG2BVq2bOA1s15JF54xDrCOhLEEtxgUTls7FZi5kiXUu4GpcEDfjkjFHN5qoG6sobrsJNYwpbjMO7UUQLaTzUqu0cVItzhEdWPjuj+egj3e3vef2Pre3z6/vV3Jzz/md3zn9/jjt5zn5nfMcUlVIktr1U8tdgCRpaRn0ktQ4g16SGmfQS1LjDHpJatwpy13AbGeeeWatXr16ucuQpLFy1113fbuqJubadsIF/erVq5menl7uMiRprCT5+pG2OXUjSY0z6CWpcQa9JDWuV9AnWZ9kT5KZJFvm2P6WJF9Ock+SzyY5d2jbO7r99iR5zSiLlyTNb96gT7ICuB54LXAucPlwkHduqqoXV9V5wHXA+7t9zwU2Ai8C1gN/2R1PknSc9LmiXwfMVNXeqnoM2AFsGO5QVd8bWn0a8OSb0jYAO6rq0ar6GjDTHU+SdJz0ebxyJfDg0Pp+4ILZnZK8FbgKOBV41dC+d87ad+Uc+24CNgGcffbZfeqWJPU0spuxVXV9VT0PeDvwRwvcd1tVTVbV5MTEnM/7S5KOUZ+gPwCcNbS+qms7kh3Apce4ryRpxPpM3ewC1iZZwyCkNwJvHO6QZG1VfbVbvRh4cnkKuCnJ+4HnAGuBL4yicElaSqu33Dpvn33XXnwcKlm8eYO+qg4n2QzcBqwAtlfV7iRbgemqmgI2J7kQ+CHwMHBFt+/uJDcD9wGHgbdW1eNLNBZJy6ylcGxJr3fdVNVOYOestquHlt92lH3fA7znWAuUJC2O34yVpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUuF5Bn2R9kj1JZpJsmWP7VUnuS3Jvkk8lOWdo2+NJ7ul+pkZZvCRpfqfM1yHJCuB64NXAfmBXkqmqum+o293AZFV9P8nvAtcBb+i2/aCqzhtx3ZKknvpc0a8DZqpqb1U9BuwANgx3qKo7qur73eqdwKrRlilJOlZ9gn4l8ODQ+v6u7UiuBD45tP7UJNNJ7kxy6Vw7JNnU9Zk+ePBgj5IkSX3NO3WzEEneBEwCvzbUfE5VHUjyXODTSb5cVQ8M71dV24BtAJOTkzXKmiTpZNfniv4AcNbQ+qqu7SckuRB4J3BJVT36ZHtVHeh+7wU+A5y/iHolSQvUJ+h3AWuTrElyKrAR+ImnZ5KcD9zAIOQfGmo/Pclp3fKZwMuA4Zu4kqQlNu/UTVUdTrIZuA1YAWyvqt1JtgLTVTUFvA94OvCxJADfqKpLgBcCNyR5gsGHyrWzntaRJC2xXnP0VbUT2Dmr7eqh5QuPsN/ngBcvpkBJ0uL4zVhJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mN6xX0SdYn2ZNkJsmWObZfleS+JPcm+VSSc4a2XZHkq93PFaMsXpI0v3mDPskK4HrgtcC5wOVJzp3V7W5gsqpeAnwcuK7b9wzgGuACYB1wTZLTR1e+JGk+fa7o1wEzVbW3qh4DdgAbhjtU1R1V9f1u9U5gVbf8GuD2qjpUVQ8DtwPrR1O6JKmPPkG/EnhwaH1/13YkVwKfXMi+STYlmU4yffDgwR4lSZL6GunN2CRvAiaB9y1kv6raVlWTVTU5MTExypIk6aTXJ+gPAGcNra/q2n5CkguBdwKXVNWjC9lXkrR0+gT9LmBtkjVJTgU2AlPDHZKcD9zAIOQfGtp0G3BRktO7m7AXdW2SpOPklPk6VNXhJJsZBPQKYHtV7U6yFZiuqikGUzVPBz6WBOAbVXVJVR1K8m4GHxYAW6vq0JKMRJI0p3mDHqCqdgI7Z7VdPbR84VH23Q5sP9YCJUmL4zdjJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNa5X0CdZn2RPkpkkW+bY/ookX0xyOMlls7Y9nuSe7mdqVIVLkvo5Zb4OSVYA1wOvBvYDu5JMVdV9Q92+AbwZ+IM5DvGDqjpvBLVKko7BvEEPrANmqmovQJIdwAbgR0FfVfu6bU8sQY2SpEXoM3WzEnhwaH1/19bXU5NMJ7kzyaVzdUiyqeszffDgwQUcWpI0n+NxM/acqpoE3gh8IMnzZneoqm1VNVlVkxMTE8ehJEk6efQJ+gPAWUPrq7q2XqrqQPd7L/AZ4PwF1CdJWqQ+Qb8LWJtkTZJTgY1Ar6dnkpye5LRu+UzgZQzN7UuSlt68QV9Vh4HNwG3A/cDNVbU7ydYklwAkeWmS/cDrgBuS7O52fyEwneRLwB3AtbOe1pEkLbE+T91QVTuBnbParh5a3sVgSmf2fp8DXrzIGiVJi+A3YyWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNa7X/0pQkkZt9ZZbj7p937UXH6dK2ucVvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjesV9EnWJ9mTZCbJljm2vyLJF5McTnLZrG1XJPlq93PFqAqXJPUzb9AnWQFcD7wWOBe4PMm5s7p9A3gzcNOsfc8ArgEuANYB1yQ5ffFlS5L66nNFvw6Yqaq9VfUYsAPYMNyhqvZV1b3AE7P2fQ1we1UdqqqHgduB9SOoW5LUU5+gXwk8OLS+v2vro9e+STYlmU4yffDgwZ6HliT1cULcjK2qbVU1WVWTExMTy12OJDWlT9AfAM4aWl/VtfWxmH0lSSPQ5+2Vu4C1SdYwCOmNwBt7Hv824E+GbsBeBLxjwVVKWhK+QfLkMO8VfVUdBjYzCO37gZuraneSrUkuAUjy0iT7gdcBNyTZ3e17CHg3gw+LXcDWrk2SdJz0eh99Ve0Eds5qu3poeReDaZm59t0ObF9EjZKkRTghbsZKkpaOQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMadstwF6PhZveXWo27fd+3Fx6kSSceTV/SS1DiDXpIaZ9BLUuMMeklqXK+gT7I+yZ4kM0m2zLH9tCQf7bZ/Psnqrn11kh8kuaf7+avRli9Jms+8T90kWQFcD7wa2A/sSjJVVfcNdbsSeLiqnp9kI/Be4A3dtgeq6rwR1y1J6qnPFf06YKaq9lbVY8AOYMOsPhuAD3XLHwd+PUlGV6Yk6Vj1CfqVwIND6/u7tjn7VNVh4L+Bn+u2rUlyd5J/TfKrc/0BSTYlmU4yffDgwQUNQJJ0dEt9M/abwNlVdT5wFXBTkp+d3amqtlXVZFVNTkxMLHFJknRy6RP0B4CzhtZXdW1z9klyCvAM4DtV9WhVfQegqu4CHgBesNiiJUn99Qn6XcDaJGuSnApsBKZm9ZkCruiWLwM+XVWVZKK7mUuS5wJrgb2jKV2S1Me8T91U1eEkm4HbgBXA9qranWQrMF1VU8CNwEeSzACHGHwYALwC2Jrkh8ATwFuq6tBSDESSNLdeLzWrqp3AzlltVw8t/y/wujn2uwW4ZZE1SpIWwW/GSlLjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhrX6wtTkk5uq7fcetTt+669+DhVomPhFb0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY07aZ+j97lgSScLr+glqXEGvSQ17qSdutHJYb4pOnCaTu3zil6SGmfQS1LjnLqRdELzCbnFM+glnVROxg8Op24kqXFe0Y/Icl4lnIxXKDo6/05oWK+gT7Ie+HNgBfA3VXXtrO2nAR8Gfhn4DvCGqtrXbXsHcCXwOPB7VXXbyKofU/4jPDn0Pc/+fTh5LNe5njfok6wArgdeDewHdiWZqqr7hrpdCTxcVc9PshF4L/CGJOcCG4EXAc8B/iXJC6rq8VEPRKOzkGfPRx1myxmO4xDMfigcmeflyPpc0a8DZqpqL0CSHcAGYDjoNwDv6pY/DvxFknTtO6rqUeBrSWa64/3HaMr//0b9H9y/PJLGXarq6B2Sy4D1VfXb3fpvARdU1eahPl/p+uzv1h8ALmAQ/ndW1d927TcCn6yqj8/6MzYBm7rVnwf2LH5oP3Im8O0RHm85OZYTk2M5MZ1sYzmnqibm2nBC3Iytqm3AtqU4dpLpqppcimMfb47lxORYTkyO5cf6PF55ADhraH1V1zZnnySnAM9gcFO2z76SpCXUJ+h3AWuTrElyKoObq1Oz+kwBV3TLlwGfrsGc0BSwMclpSdYAa4EvjKZ0SVIf807dVNXhJJuB2xg8Xrm9qnYn2QpMV9UUcCPwke5m6yEGHwZ0/W5mcOP2MPDWZXjiZkmmhJaJYzkxOZYTk2PpzHszVpI03nwFgiQ1zqCXpMY1G/RJ1ifZk2QmyZblrmcxkuxL8uUk9ySZXu56FirJ9iQPdd+3eLLtjCS3J/lq9/v05ayxryOM5V1JDnTn554kv7GcNfaR5KwkdyS5L8nuJG/r2sfuvBxlLGN3XgCSPDXJF5J8qRvPH3fta5J8vsu0j3YPx/Q7Zotz9N1rG/6Todc2AJfPem3D2EiyD5isqrH88keSVwCPAB+uql/s2q4DDlXVtd0H8elV9fblrLOPI4zlXcAjVfWny1nbQiR5NvDsqvpikp8B7gIuBd7MmJ2Xo4zl9YzZeQHo3irwtKp6JMlTgM8CbwOuAj5RVTuS/BXwpar6YJ9jtnpF/6PXNlTVY8CTr23QMqiqf2PwNNawDcCHuuUPMfiHecI7wljGTlV9s6q+2C3/D3A/sJIxPC9HGctYqoFHutWndD8FvIrBK2Zggeem1aBfCTw4tL6fMT7xDE7yPye5q3tdRAueVVXf7Jb/C3jWchYzApuT3NtN7Zzw0x3DkqwGzgc+z5ifl1ljgTE9L0lWJLkHeAi4HXgA+G5VHe66LCjTWg361ry8qn4JeC3w1m76oBndl+vGeQ7xg8DzgPOAbwJ/trzl9Jfk6cAtwO9X1feGt43beZljLGN7Xqrq8ao6j8HbBNYBv7CY47Ua9E29eqGqDnS/HwL+gcGJH3ff6uZWn5xjfWiZ6zlmVfWt7h/mE8BfMybnp5v/vQX4u6r6RNc8ludlrrGM63kZVlXfBe4AfgV4ZveKGVhgprUa9H1e2zAWkjytu8FEkqcBFwFfOfpeY2H4tRlXAP+0jLUsypPB2PlNxuD8dDf8bgTur6r3D20au/NypLGM43kBSDKR5Jnd8k8zeKjkfgaBf1nXbUHnpsmnbgC6R6k+wI9f2/CeZS7pmCR5LoOreBi8suKmcRtLkr8HXsngVavfAq4B/hG4GTgb+Drw+qo64W9yHmEsr2QwPVDAPuB3hua5T0hJXg78O/Bl4Imu+Q8ZzG2P1Xk5ylguZ8zOC0CSlzC42bqCwcX4zVW1tcuCHcAZwN3Am7r/18f8x2w16CVJA61O3UiSOga9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJatz/AanJgeJoVypeAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Brsd2UxDGwOS"
      },
      "source": [
        "So from the above plot we can say 6 important variavles are there (6 high bars)"
      ]
    }
  ]
}