10_Grid_Search_in_RF.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "10 Grid Search in RF.ipynb",
      "provenance": [],
      "authorship_tag": "ABX9TyO3H0bzNKjV+2OwMW6kFQa3",
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/sandipanpaul21/Tree-Based-Models-in-Python/blob/master/10_Grid_Search_in_RF.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JZNE-NRwY6Hy"
      },
      "source": [
        "### **Hyperparameter Tuning the Random Forest in Python**\n",
        "\n",
        "  - The best way to think about hyperparameters is like the settings of an algorithm that can be adjusted to optimize performance, just as we might *turn the knobs of an AM radio to get a clear signal* (or your parents might have!). \n",
        "  - While \n",
        "    1. **model parameters** are learned during training — such as the slope and intercept in a linear regression\n",
        "    2. **hyperparameters** must be set by the programmer before training.\n",
        "      \n",
        "      i. In the case of a random forest, hyperparameters include the number of decision trees in the forest and the number of features considered by each tree when splitting a node. \n",
        "\n",
        "      ii. The parameters of a random forest are the variables and thresholds used to split each node learned during training. \n",
        "\n",
        "  - Hyperparameter tuning relies more on experimental results than theory, and thus the best method to determine the optimal settings is to try many different combinations evaluate the performance of each model.\n",
        "  - However, evaluating each model only on the training set can lead to one of the most fundamental problems in machine learning: **overfitting.**\n",
        "  - Overfitting is if we optimize the model for the training data, then our model will score very well on the training set, but will not be able to generalize to new data, such as in a test set. When a model performs highly on the training set but poorly on the test set, this is known as overfitting, or essentially creating a model that knows the training set very well but cannot be applied to new problems. \n",
        "  - It’s like a student who has memorized the simple problems in the textbook but has no idea how to apply concepts in the messy real world.\n",
        "  - An overfit model may look impressive on the training set, but will be useless in a real application. Therefore, the standard procedure for hyperparameter optimization accounts for overfitting through cross validation."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7eOb1itIag30"
      },
      "source": [
        "### **Cross Validation**\n",
        "\n",
        "- When we approach a machine learning problem, we make sure to split our data into a training and a testing set. \n",
        "- In K-Fold CV, we further split our training set into K number of subsets, called folds. We then iteratively fit the model K times, each time training the data on K-1 of the folds and evaluating on the Kth fold (called the validation data). \n",
        "- As an example, \n",
        "\n",
        "  i. consider fitting a model with K = 5. The first iteration we train on the first four folds and evaluate on the fifth.\n",
        "  \n",
        "  ii. The second time we train on the first, second, third, and fifth fold and evaluate on the fourth. We repeat this procedure 3 more times, each time evaluating on a different fold. \n",
        "  \n",
        "  iii. At the very end of training, we average the performance on each of the folds to come up with final validation metrics for the model.\n",
        "\n",
        "![K Cross Validation](https://miro.medium.com/max/1400/0*KH3dnbGNcmyV_ODL.png)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lnMRKd1zc_zE"
      },
      "source": [
        "- For hyperparameter tuning, we perform many iterations of the entire K-Fold CV process, each time using different model settings.\n",
        "- We then compare all of the models, select the best one, train it on the full training set, and then evaluate on the testing set.\n",
        "- Each time we want to assess a different set of hyperparameters, we have to split our training data into K fold and train and evaluate K times. \n",
        "- If we have 10 sets of hyperparameters and are using 5-Fold CV, that represents 50 training loops."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WiiCarb3cgVF"
      },
      "source": [
        "# Libraries\n",
        "from sklearn import datasets\n",
        "import pandas as pd\n",
        "import warnings\n",
        "warnings.filterwarnings(\"ignore\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "id": "js1zxNsjcqkt",
        "outputId": "3e35f3fd-a27b-4536-8a4c-ed34a24e9808"
      },
      "source": [
        "# Lets Take another Dataset : IRIS Dataset\n",
        "# Loading the Dataset\n",
        "\n",
        "boston = datasets.load_boston()\n",
        "boston_data = pd.DataFrame(boston.data)\n",
        "boston_data.columns = boston.feature_names\n",
        "boston_data['Type']=boston.target\n",
        "boston_data.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>CRIM</th>\n",
              "      <th>ZN</th>\n",
              "      <th>INDUS</th>\n",
              "      <th>CHAS</th>\n",
              "      <th>NOX</th>\n",
              "      <th>RM</th>\n",
              "      <th>AGE</th>\n",
              "      <th>DIS</th>\n",
              "      <th>RAD</th>\n",
              "      <th>TAX</th>\n",
              "      <th>PTRATIO</th>\n",
              "      <th>B</th>\n",
              "      <th>LSTAT</th>\n",
              "      <th>Type</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0.00632</td>\n",
              "      <td>18.0</td>\n",
              "      <td>2.31</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.538</td>\n",
              "      <td>6.575</td>\n",
              "      <td>65.2</td>\n",
              "      <td>4.0900</td>\n",
              "      <td>1.0</td>\n",
              "      <td>296.0</td>\n",
              "      <td>15.3</td>\n",
              "      <td>396.90</td>\n",
              "      <td>4.98</td>\n",
              "      <td>24.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0.02731</td>\n",
              "      <td>0.0</td>\n",
              "      <td>7.07</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.469</td>\n",
              "      <td>6.421</td>\n",
              "      <td>78.9</td>\n",
              "      <td>4.9671</td>\n",
              "      <td>2.0</td>\n",
              "      <td>242.0</td>\n",
              "      <td>17.8</td>\n",
              "      <td>396.90</td>\n",
              "      <td>9.14</td>\n",
              "      <td>21.6</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0.02729</td>\n",
              "      <td>0.0</td>\n",
              "      <td>7.07</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.469</td>\n",
              "      <td>7.185</td>\n",
              "      <td>61.1</td>\n",
              "      <td>4.9671</td>\n",
              "      <td>2.0</td>\n",
              "      <td>242.0</td>\n",
              "      <td>17.8</td>\n",
              "      <td>392.83</td>\n",
              "      <td>4.03</td>\n",
              "      <td>34.7</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>0.03237</td>\n",
              "      <td>0.0</td>\n",
              "      <td>2.18</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.458</td>\n",
              "      <td>6.998</td>\n",
              "      <td>45.8</td>\n",
              "      <td>6.0622</td>\n",
              "      <td>3.0</td>\n",
              "      <td>222.0</td>\n",
              "      <td>18.7</td>\n",
              "      <td>394.63</td>\n",
              "      <td>2.94</td>\n",
              "      <td>33.4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0.06905</td>\n",
              "      <td>0.0</td>\n",
              "      <td>2.18</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.458</td>\n",
              "      <td>7.147</td>\n",
              "      <td>54.2</td>\n",
              "      <td>6.0622</td>\n",
              "      <td>3.0</td>\n",
              "      <td>222.0</td>\n",
              "      <td>18.7</td>\n",
              "      <td>396.90</td>\n",
              "      <td>5.33</td>\n",
              "      <td>36.2</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "      CRIM    ZN  INDUS  CHAS    NOX  ...    TAX  PTRATIO       B  LSTAT  Type\n",
              "0  0.00632  18.0   2.31   0.0  0.538  ...  296.0     15.3  396.90   4.98  24.0\n",
              "1  0.02731   0.0   7.07   0.0  0.469  ...  242.0     17.8  396.90   9.14  21.6\n",
              "2  0.02729   0.0   7.07   0.0  0.469  ...  242.0     17.8  392.83   4.03  34.7\n",
              "3  0.03237   0.0   2.18   0.0  0.458  ...  222.0     18.7  394.63   2.94  33.4\n",
              "4  0.06905   0.0   2.18   0.0  0.458  ...  222.0     18.7  396.90   5.33  36.2\n",
              "\n",
              "[5 rows x 14 columns]"
            ]
          },
          "metadata": {},
          "execution_count": 2
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 222
        },
        "id": "gYkM3vQTdzOo",
        "outputId": "2cde83df-8b1c-4a11-88e4-7f548bbd19e0"
      },
      "source": [
        "print('Feature Columns are ')\n",
        "features = boston_data.iloc[:,:13]\n",
        "features.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Feature Columns are \n"
          ]
        },
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>CRIM</th>\n",
              "      <th>ZN</th>\n",
              "      <th>INDUS</th>\n",
              "      <th>CHAS</th>\n",
              "      <th>NOX</th>\n",
              "      <th>RM</th>\n",
              "      <th>AGE</th>\n",
              "      <th>DIS</th>\n",
              "      <th>RAD</th>\n",
              "      <th>TAX</th>\n",
              "      <th>PTRATIO</th>\n",
              "      <th>B</th>\n",
              "      <th>LSTAT</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0.00632</td>\n",
              "      <td>18.0</td>\n",
              "      <td>2.31</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.538</td>\n",
              "      <td>6.575</td>\n",
              "      <td>65.2</td>\n",
              "      <td>4.0900</td>\n",
              "      <td>1.0</td>\n",
              "      <td>296.0</td>\n",
              "      <td>15.3</td>\n",
              "      <td>396.90</td>\n",
              "      <td>4.98</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0.02731</td>\n",
              "      <td>0.0</td>\n",
              "      <td>7.07</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.469</td>\n",
              "      <td>6.421</td>\n",
              "      <td>78.9</td>\n",
              "      <td>4.9671</td>\n",
              "      <td>2.0</td>\n",
              "      <td>242.0</td>\n",
              "      <td>17.8</td>\n",
              "      <td>396.90</td>\n",
              "      <td>9.14</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0.02729</td>\n",
              "      <td>0.0</td>\n",
              "      <td>7.07</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.469</td>\n",
              "      <td>7.185</td>\n",
              "      <td>61.1</td>\n",
              "      <td>4.9671</td>\n",
              "      <td>2.0</td>\n",
              "      <td>242.0</td>\n",
              "      <td>17.8</td>\n",
              "      <td>392.83</td>\n",
              "      <td>4.03</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>0.03237</td>\n",
              "      <td>0.0</td>\n",
              "      <td>2.18</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.458</td>\n",
              "      <td>6.998</td>\n",
              "      <td>45.8</td>\n",
              "      <td>6.0622</td>\n",
              "      <td>3.0</td>\n",
              "      <td>222.0</td>\n",
              "      <td>18.7</td>\n",
              "      <td>394.63</td>\n",
              "      <td>2.94</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0.06905</td>\n",
              "      <td>0.0</td>\n",
              "      <td>2.18</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.458</td>\n",
              "      <td>7.147</td>\n",
              "      <td>54.2</td>\n",
              "      <td>6.0622</td>\n",
              "      <td>3.0</td>\n",
              "      <td>222.0</td>\n",
              "      <td>18.7</td>\n",
              "      <td>396.90</td>\n",
              "      <td>5.33</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "      CRIM    ZN  INDUS  CHAS    NOX  ...  RAD    TAX  PTRATIO       B  LSTAT\n",
              "0  0.00632  18.0   2.31   0.0  0.538  ...  1.0  296.0     15.3  396.90   4.98\n",
              "1  0.02731   0.0   7.07   0.0  0.469  ...  2.0  242.0     17.8  396.90   9.14\n",
              "2  0.02729   0.0   7.07   0.0  0.469  ...  2.0  242.0     17.8  392.83   4.03\n",
              "3  0.03237   0.0   2.18   0.0  0.458  ...  3.0  222.0     18.7  394.63   2.94\n",
              "4  0.06905   0.0   2.18   0.0  0.458  ...  3.0  222.0     18.7  396.90   5.33\n",
              "\n",
              "[5 rows x 13 columns]"
            ]
          },
          "execution_count": 33,
          "metadata": {},
          "output_type": "execute_result"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3gdO3b6UeSrX",
        "outputId": "b978c5ba-9bce-4678-b6df-2171722a9b47"
      },
      "source": [
        "print(\"Target Variable is\")\n",
        "labels = boston_data['Type']\n",
        "labels.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Target Variable is\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "0    24.0\n",
              "1    21.6\n",
              "2    34.7\n",
              "3    33.4\n",
              "4    36.2\n",
              "Name: Type, dtype: float64"
            ]
          },
          "execution_count": 34,
          "metadata": {},
          "output_type": "execute_result"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TaN4fw3-em9C",
        "outputId": "36aa4dcf-2347-44e5-857d-111cd72af127"
      },
      "source": [
        "# Convert to numpy arrays\n",
        "import numpy as np\n",
        "\n",
        "features = np.array(features)\n",
        "labels = np.array(labels)\n",
        "\n",
        "# Training and Testing Sets\n",
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "train_features, test_features, train_labels, test_labels = train_test_split(features, labels, \n",
        "                                                                            test_size = 0.25, \n",
        "                                                                            random_state = 42)\n",
        "print('Training Features Shape:', train_features.shape)\n",
        "print('Training Labels Shape:', train_labels.shape)\n",
        "print('Testing Features Shape:', test_features.shape)\n",
        "print('Testing Labels Shape:', test_labels.shape)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Training Features Shape: (379, 13)\n",
            "Training Labels Shape: (379,)\n",
            "Testing Features Shape: (127, 13)\n",
            "Testing Labels Shape: (127,)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "epWIluE4fi0T",
        "outputId": "dfd66624-c280-4d25-a205-44e5a02b43cb"
      },
      "source": [
        "# Examine the Default Random Forest to Determine Parameters\n",
        "\n",
        "from sklearn.ensemble import RandomForestRegressor\n",
        "rf = RandomForestRegressor(random_state = 42)\n",
        "\n",
        "from pprint import pprint\n",
        "\n",
        "# Look at parameters used by our current forest\n",
        "print('Parameters currently in use:\\n')\n",
        "pprint(rf.get_params())"
      ],
      "execution_count": null,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Parameters currently in use:\n",
            "\n",
            "{'bootstrap': True,\n",
            " 'ccp_alpha': 0.0,\n",
            " 'criterion': 'squared_error',\n",
            " 'max_depth': None,\n",
            " 'max_features': 'auto',\n",
            " 'max_leaf_nodes': None,\n",
            " 'max_samples': None,\n",
            " 'min_impurity_decrease': 0.0,\n",
            " 'min_samples_leaf': 1,\n",
            " 'min_samples_split': 2,\n",
            " 'min_weight_fraction_leaf': 0.0,\n",
            " 'n_estimators': 100,\n",
            " 'n_jobs': None,\n",
            " 'oob_score': False,\n",
            " 'random_state': 42,\n",
            " 'verbose': 0,\n",
            " 'warm_start': False}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sM08ieB-f6gW"
      },
      "source": [
        "- **n_estimators =** number of trees in the foreset\n",
        "- **max_features =** max number of features considered for splitting a node\n",
        "- **max_depth =** max number of levels in each decision tree\n",
        "- **min_samples_split =** min number of data points placed in a node before the node is split\n",
        "- **min_samples_leaf =** min number of data points allowed in a leaf node\n",
        "- **bootstrap =** method for sampling data points (with or without replacement)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9BheLhklgKCg",
        "outputId": "4fffd600-9708-45ac-8d69-fbe5e9be0348"
      },
      "source": [
        "print(\"Random Search with Cross Validation\")\n",
        "from sklearn.model_selection import RandomizedSearchCV\n",
        "\n",
        "# Number of trees in random forest\n",
        "n_estimators = [int(x) for x in np.linspace(start = 200, stop = 2000, num = 10)]\n",
        "# Number of features to consider at every split\n",
        "max_features = ['auto', 'sqrt']\n",
        "# Maximum number of levels in tree\n",
        "max_depth = [int(x) for x in np.linspace(10, 110, num = 11)]\n",
        "max_depth.append(None)\n",
        "# Minimum number of samples required to split a node\n",
        "min_samples_split = [2, 5, 10]\n",
        "# Minimum number of samples required at each leaf node\n",
        "min_samples_leaf = [1, 2, 4]\n",
        "# Method of selecting samples for training each tree\n",
        "bootstrap = [True, False]\n",
        "\n",
        "# Create the random grid\n",
        "random_grid = {'n_estimators': n_estimators,\n",
        "               'max_features': max_features,\n",
        "               'max_depth': max_depth,\n",
        "               'min_samples_split': min_samples_split,\n",
        "               'min_samples_leaf': min_samples_leaf,\n",
        "               'bootstrap': bootstrap}\n",
        "\n",
        "pprint(random_grid)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Random Search with Cross Validation\n",
            "{'bootstrap': [True, False],\n",
            " 'max_depth': [10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 110, None],\n",
            " 'max_features': ['auto', 'sqrt'],\n",
            " 'min_samples_leaf': [1, 2, 4],\n",
            " 'min_samples_split': [2, 5, 10],\n",
            " 'n_estimators': [200, 400, 600, 800, 1000, 1200, 1400, 1600, 1800, 2000]}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "24qtQ2o5gcTw"
      },
      "source": [
        "On each iteration, the algorithm will choose a difference combination of the features. Altogether, there are 2 * 12 * 2 * 3 * 3 * 10 = 4320 settings! However, the benefit of a random search is that we are not trying every combination, but selecting at random to sample a wide range of values."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "background_save": true,
          "base_uri": "https://localhost:8080/"
        },
        "id": "cuyr4rFHgY-5",
        "outputId": "016bff6f-9b75-4458-c376-15e45a506f81"
      },
      "source": [
        "# Use the random grid to search for best hyperparameters\n",
        "\n",
        "# First create the base model to tune\n",
        "rf = RandomForestRegressor(random_state = 42)\n",
        "\n",
        "# Random search of parameters, using 3 fold cross validation, \n",
        "# search across 100 different combinations, and use all available cores\n",
        "rf_random = RandomizedSearchCV(estimator=rf, param_distributions=random_grid,\n",
        "                              n_iter = 100, scoring='neg_mean_absolute_error', \n",
        "                              cv = 3, verbose=2, random_state=42, n_jobs=-1,\n",
        "                              return_train_score=True)\n",
        "\n",
        "# Fit the random search model\n",
        "rf_random.fit(train_features, train_labels)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Fitting 3 folds for each of 100 candidates, totalling 300 fits\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "RandomizedSearchCV(cv=3, estimator=RandomForestRegressor(random_state=42),\n",
              "                   n_iter=100, n_jobs=-1,\n",
              "                   param_distributions={'bootstrap': [True, False],\n",
              "                                        'max_depth': [10, 20, 30, 40, 50, 60,\n",
              "                                                      70, 80, 90, 100, 110,\n",
              "                                                      None],\n",
              "                                        'max_features': ['auto', 'sqrt'],\n",
              "                                        'min_samples_leaf': [1, 2, 4],\n",
              "                                        'min_samples_split': [2, 5, 10],\n",
              "                                        'n_estimators': [200, 400, 600, 800,\n",
              "                                                         1000, 1200, 1400, 1600,\n",
              "                                                         1800, 2000]},\n",
              "                   random_state=42, return_train_score=True,\n",
              "                   scoring='neg_mean_absolute_error', verbose=2)"
            ]
          },
          "execution_count": null,
          "metadata": {},
          "output_type": "execute_result"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "background_save": true
        },
        "id": "72tMPUH1iSTo",
        "outputId": "7589fb1a-445d-4791-b4cb-7e6a89dbe4bc"
      },
      "source": [
        "print(\"Best Parameter\")\n",
        "rf_random.best_params_"
      ],
      "execution_count": null,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Best Parameter\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "{'bootstrap': False,\n",
              " 'max_depth': None,\n",
              " 'max_features': 'sqrt',\n",
              " 'min_samples_leaf': 1,\n",
              " 'min_samples_split': 2,\n",
              " 'n_estimators': 400}"
            ]
          },
          "execution_count": null,
          "metadata": {},
          "output_type": "execute_result"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TX0Yc2Xkg-XW"
      },
      "source": [
        "- The most important arguments in RandomizedSearchCV are n_iter, which controls the number of different combinations to try, and cv which is the number of folds to use for cross validation (we use 100 and 3 respectively). \n",
        "- More iterations will cover a wider search space and more cv folds reduces the chances of overfitting, but raising each will increase the run time. Machine learning is a field of trade-offs, and performance vs time is one of the most fundamental.\n",
        "\n",
        "- Evaluate Random Search\n",
        "  \n",
        "  To determine if random search yielded a better model, we compare the base model with the best random search model.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "background_save": true
        },
        "id": "GS2MpC-6gwTl",
        "outputId": "7bbf2513-a01b-435b-8c14-bf037813bcdb"
      },
      "source": [
        "def evaluate(model, test_features, test_labels):\n",
        "  predictions = model.predict(test_features)\n",
        "  errors = abs(predictions - test_labels)\n",
        "  mape = 100 * np.mean(errors / test_labels)\n",
        "  accuracy = 100 - mape\n",
        "  print('Model Performance')\n",
        "  print('Average Error: {:0.4f} degrees.'.format(np.mean(errors)))\n",
        "  print('Accuracy = {:0.2f}%.'.format(accuracy))\n",
        "  return accuracy\n",
        "\n",
        "base_model = RandomForestRegressor(n_estimators = 10, random_state = 42)\n",
        "base_model.fit(train_features, train_labels)\n",
        "base_accuracy = evaluate(base_model, test_features, test_labels)\n",
        "\n",
        "print(\"\\n\")\n",
        "best_random = rf_random.best_estimator_\n",
        "random_accuracy = evaluate(best_random, test_features, test_labels)\n",
        "\n",
        "print(\"\\n\")\n",
        "print('Improvement of {:0.2f}%.'.format( 100 * (random_accuracy - base_accuracy) / base_accuracy))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Model Performance\n",
            "Average Error: 2.2394 degrees.\n",
            "Accuracy = 88.33%.\n",
            "\n",
            "\n",
            "Model Performance\n",
            "Average Error: 1.9170 degrees.\n",
            "Accuracy = 90.41%.\n",
            "\n",
            "\n",
            "Improvement of 2.35%.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Kph_HPLhhvjg"
      },
      "source": [
        "### **Grid Search with Cross Validation**\n",
        "\n",
        "- Random search allowed us to narrow down the range for each hyperparameter. \n",
        "- Now that we know where to concentrate our search, we can explicitly specify every combination of settings to try. \n",
        "- We do this with GridSearchCV, a method that, instead of sampling randomly from a distribution, evaluates all combinations we define. \n",
        "- To use Grid Search, we make another grid based on the best values provided by random search:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "background_save": true
        },
        "id": "Y1im-Pc-hlpJ",
        "outputId": "01506393-ccac-4fd6-ba34-1013b17c4fd4"
      },
      "source": [
        "from sklearn.model_selection import GridSearchCV\n",
        "# Create the parameter grid based on the results of random search \n",
        "param_grid = {\n",
        "    'bootstrap': [True],\n",
        "    'max_depth': [80, 90, 100, 110],\n",
        "    'max_features': [2, 3],\n",
        "    'min_samples_leaf': [3, 4, 5],\n",
        "    'min_samples_split': [8, 10, 12],\n",
        "    'n_estimators': [100, 200, 300, 1000]\n",
        "}\n",
        "# Create a based model\n",
        "rf = RandomForestRegressor()\n",
        "# Instantiate the grid search model\n",
        "grid_search = GridSearchCV(estimator = rf, param_grid = param_grid, \n",
        "                          cv = 3, n_jobs = -1, verbose = 2)\n",
        "grid_search"
      ],
      "execution_count": null,
      "outputs": [
        {
          "data": {
            "text/plain": [
              "GridSearchCV(cv=3, estimator=RandomForestRegressor(), n_jobs=-1,\n",
              "             param_grid={'bootstrap': [True], 'max_depth': [80, 90, 100, 110],\n",
              "                         'max_features': [2, 3], 'min_samples_leaf': [3, 4, 5],\n",
              "                         'min_samples_split': [8, 10, 12],\n",
              "                         'n_estimators': [100, 200, 300, 1000]},\n",
              "             verbose=2)"
            ]
          },
          "execution_count": null,
          "metadata": {},
          "output_type": "execute_result"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uhcahuPniqwZ"
      },
      "source": [
        "This will try out 1 * 4 * 2 * 3 * 3 * 4 = 288 combinations of settings. We can fit the model, display the best hyperparameters, and evaluate performance:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "background_save": true
        },
        "id": "_aJaEoVXh9pa",
        "outputId": "7307705e-210f-4585-a245-762b46d819be"
      },
      "source": [
        "# Fit the grid search to the data\n",
        "grid_search.fit(train_features, train_labels)\n",
        "grid_search.best_params_\n",
        "\n",
        "print(\"\\n\")\n",
        "best_grid = grid_search.best_estimator_\n",
        "grid_accuracy = evaluate(best_grid, test_features, test_labels)\n",
        "\n",
        "print(\"\\n\")\n",
        "print('Improvement of {:0.2f}%.'.format( 100 * (grid_accuracy - base_accuracy) / base_accuracy))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Fitting 3 folds for each of 288 candidates, totalling 864 fits\n",
            "\n",
            "\n",
            "Model Performance\n",
            "Average Error: 2.1252 degrees.\n",
            "Accuracy = 89.03%.\n",
            "\n",
            "\n",
            "Improvement of 0.79%.\n"
          ]
        }
      ]
    }
  ]
}