Fake_news_detection_using_title.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "JisnpFXE-Fkz"
   },
   "source": [
    "# **Fake News Detection using title**\n",
    "\n",
    "This notebook detects the fake news based on the title of the article. The fake news detection is modelled using three algorithms to predict fake and real news and the sections covered in this notebook are:\n",
    "\n",
    "1. Install libraries \n",
    "2. Import necessary libraries and load the data\n",
    "3. Data Cleaning\n",
    "4. Text Preprocessing\n",
    "5. Function for Evaluation Metrics\n",
    "6. Apply the algortihms to predict the fake news: \n",
    "           1. Decision Tree\n",
    "           2. Gradient Boosting Tree Classifier\n",
    "           3. Neural Network using Keras and Elephas\n",
    "7. Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "OZ2Oj2Q-QQdw"
   },
   "source": [
    "**1. Install Libraries**\n",
    "\n",
    "This project is implemented using Apache Spark written in Pyspark. In order to code the project in jupyter notebook, Pyspark module should be installed which is done using the command !pip install pyspark.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 235
    },
    "colab_type": "code",
    "id": "pzgxpA0axXm9",
    "outputId": "470903a1-0acd-4a6e-9e5c-5b8905a477f4"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting pyspark\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/8e/b0/bf9020b56492281b9c9d8aae8f44ff51e1bc91b3ef5a884385cb4e389a40/pyspark-3.0.0.tar.gz (204.7MB)\n",
      "\u001b[K     |████████████████████████████████| 204.7MB 50kB/s \n",
      "\u001b[?25hCollecting py4j==0.10.9\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/9e/b6/6a4fb90cd235dc8e265a6a2067f2a2c99f0d91787f06aca4bcf7c23f3f80/py4j-0.10.9-py2.py3-none-any.whl (198kB)\n",
      "\u001b[K     |████████████████████████████████| 204kB 43.6MB/s \n",
      "\u001b[?25hBuilding wheels for collected packages: pyspark\n",
      "  Building wheel for pyspark (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Created wheel for pyspark: filename=pyspark-3.0.0-py2.py3-none-any.whl size=205044182 sha256=79faf51df169d0254cc0408559af938db8826fa1fb983b5c957fc9f93d169b3f\n",
      "  Stored in directory: /root/.cache/pip/wheels/57/27/4d/ddacf7143f8d5b76c45c61ee2e43d9f8492fc5a8e78ebd7d37\n",
      "Successfully built pyspark\n",
      "Installing collected packages: py4j, pyspark\n",
      "Successfully installed py4j-0.10.9 pyspark-3.0.0\n"
     ]
    }
   ],
   "source": [
    "!pip install pyspark"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "E5rqxKPYL-EH"
   },
   "source": [
    "Elephas is an extension of Keras, which allows you to run distributed deep learning models at scale with Spark. To implement deep learning models, Elephas is installed which supports the following versions of Keras and Tensorflow. \n",
    "\n",
    "*   Keras: 2.2.4\n",
    "*   TensorFlow: 1.14.0\n",
    "\n",
    "Thus the above versions are installed and the runtime is restarted to use the new version."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 994
    },
    "colab_type": "code",
    "id": "8fMM3pZsVsXX",
    "outputId": "ef49c9ff-c6da-482f-e35d-7accd56ccb7d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting q\n",
      "  Downloading https://files.pythonhosted.org/packages/53/bc/51619d89e0bd855567e7652fa16d06f1ed36a85f108a7fe71f6629bf719d/q-2.6-py2.py3-none-any.whl\n",
      "Collecting keras==2.2.4\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/5e/10/aa32dad071ce52b5502266b5c659451cfd6ffcbf14e6c8c4f16c0ff5aaab/Keras-2.2.4-py2.py3-none-any.whl (312kB)\n",
      "\u001b[K     |████████████████████████████████| 317kB 3.5MB/s \n",
      "\u001b[?25hRequirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from keras==2.2.4) (2.10.0)\n",
      "Requirement already satisfied: keras-applications>=1.0.6 in /usr/local/lib/python3.6/dist-packages (from keras==2.2.4) (1.0.8)\n",
      "Requirement already satisfied: scipy>=0.14 in /usr/local/lib/python3.6/dist-packages (from keras==2.2.4) (1.4.1)\n",
      "Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.6/dist-packages (from keras==2.2.4) (1.12.0)\n",
      "Requirement already satisfied: numpy>=1.9.1 in /usr/local/lib/python3.6/dist-packages (from keras==2.2.4) (1.18.5)\n",
      "Requirement already satisfied: keras-preprocessing>=1.0.5 in /usr/local/lib/python3.6/dist-packages (from keras==2.2.4) (1.1.2)\n",
      "Requirement already satisfied: pyyaml in /usr/local/lib/python3.6/dist-packages (from keras==2.2.4) (3.13)\n",
      "Installing collected packages: q, keras\n",
      "  Found existing installation: Keras 2.3.1\n",
      "    Uninstalling Keras-2.3.1:\n",
      "      Successfully uninstalled Keras-2.3.1\n",
      "Successfully installed keras-2.2.4 q-2.6\n",
      "Requirement already satisfied: q in /usr/local/lib/python3.6/dist-packages (2.6)\n",
      "Collecting tensorflow==1.14.0\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/de/f0/96fb2e0412ae9692dbf400e5b04432885f677ad6241c088ccc5fe7724d69/tensorflow-1.14.0-cp36-cp36m-manylinux1_x86_64.whl (109.2MB)\n",
      "\u001b[K     |████████████████████████████████| 109.2MB 83kB/s \n",
      "\u001b[?25hRequirement already satisfied: six>=1.10.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==1.14.0) (1.12.0)\n",
      "Requirement already satisfied: google-pasta>=0.1.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow==1.14.0) (0.2.0)\n",
      "Collecting tensorboard<1.15.0,>=1.14.0\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/91/2d/2ed263449a078cd9c8a9ba50ebd50123adf1f8cfbea1492f9084169b89d9/tensorboard-1.14.0-py3-none-any.whl (3.1MB)\n",
      "\u001b[K     |████████████████████████████████| 3.2MB 9.6MB/s \n",
      "\u001b[?25hRequirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==1.14.0) (1.1.0)\n",
      "Collecting tensorflow-estimator<1.15.0rc0,>=1.14.0rc0\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/3c/d5/21860a5b11caf0678fbc8319341b0ae21a07156911132e0e71bffed0510d/tensorflow_estimator-1.14.0-py2.py3-none-any.whl (488kB)\n",
      "\u001b[K     |████████████████████████████████| 491kB 40.2MB/s \n",
      "\u001b[?25hRequirement already satisfied: keras-applications>=1.0.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow==1.14.0) (1.0.8)\n",
      "Requirement already satisfied: gast>=0.2.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==1.14.0) (0.3.3)\n",
      "Requirement already satisfied: wrapt>=1.11.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow==1.14.0) (1.12.1)\n",
      "Requirement already satisfied: numpy<2.0,>=1.14.5 in /usr/local/lib/python3.6/dist-packages (from tensorflow==1.14.0) (1.18.5)\n",
      "Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.6/dist-packages (from tensorflow==1.14.0) (0.34.2)\n",
      "Requirement already satisfied: keras-preprocessing>=1.0.5 in /usr/local/lib/python3.6/dist-packages (from tensorflow==1.14.0) (1.1.2)\n",
      "Requirement already satisfied: protobuf>=3.6.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow==1.14.0) (3.12.2)\n",
      "Requirement already satisfied: astor>=0.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==1.14.0) (0.8.1)\n",
      "Requirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow==1.14.0) (1.30.0)\n",
      "Requirement already satisfied: absl-py>=0.7.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==1.14.0) (0.9.0)\n",
      "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.6/dist-packages (from tensorboard<1.15.0,>=1.14.0->tensorflow==1.14.0) (3.2.2)\n",
      "Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.6/dist-packages (from tensorboard<1.15.0,>=1.14.0->tensorflow==1.14.0) (49.1.0)\n",
      "Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.6/dist-packages (from tensorboard<1.15.0,>=1.14.0->tensorflow==1.14.0) (1.0.1)\n",
      "Requirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from keras-applications>=1.0.6->tensorflow==1.14.0) (2.10.0)\n",
      "Requirement already satisfied: importlib-metadata; python_version < \"3.8\" in /usr/local/lib/python3.6/dist-packages (from markdown>=2.6.8->tensorboard<1.15.0,>=1.14.0->tensorflow==1.14.0) (1.7.0)\n",
      "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.6/dist-packages (from importlib-metadata; python_version < \"3.8\"->markdown>=2.6.8->tensorboard<1.15.0,>=1.14.0->tensorflow==1.14.0) (3.1.0)\n",
      "Installing collected packages: tensorboard, tensorflow-estimator, tensorflow\n",
      "  Found existing installation: tensorboard 2.2.2\n",
      "    Uninstalling tensorboard-2.2.2:\n",
      "      Successfully uninstalled tensorboard-2.2.2\n",
      "  Found existing installation: tensorflow-estimator 2.2.0\n",
      "    Uninstalling tensorflow-estimator-2.2.0:\n",
      "      Successfully uninstalled tensorflow-estimator-2.2.0\n",
      "  Found existing installation: tensorflow 2.2.0\n",
      "    Uninstalling tensorflow-2.2.0:\n",
      "      Successfully uninstalled tensorflow-2.2.0\n",
      "Successfully installed tensorboard-1.14.0 tensorflow-1.14.0 tensorflow-estimator-1.14.0\n"
     ]
    }
   ],
   "source": [
    "!pip install q keras==2.2.4\n",
    "!pip install q tensorflow==1.14.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 50
    },
    "colab_type": "code",
    "id": "Ef5UUy5BVtlz",
    "outputId": "8972d197-1479-41e8-fe58-65e6ea4f5c39"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.2.4\n",
      "1.14.0\n"
     ]
    }
   ],
   "source": [
    "import keras\n",
    "import tensorflow\n",
    "\n",
    "print(keras.__version__)\n",
    "print(tensorflow.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Lw9uQf_9i3Vn"
   },
   "source": [
    "Install elephas to integrate Keras with Apache Spark using !pip install elephas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "id": "I-AYhjdNV7dY",
    "outputId": "122e1bb4-65df-4023-9121-3085778e05d1"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting elephas\n",
      "  Downloading https://files.pythonhosted.org/packages/d9/f7/4b1542d95ad17fefff5cfbc0df732f8b082c758ed48d1738768df5cec4da/elephas-0.4.3-py3-none-any.whl\n",
      "Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from elephas) (1.12.0)\n",
      "Requirement already satisfied: tensorflow in /usr/local/lib/python3.6/dist-packages (from elephas) (1.14.0)\n",
      "Requirement already satisfied: keras in /usr/local/lib/python3.6/dist-packages (from elephas) (2.2.4)\n",
      "Requirement already satisfied: pyspark in /usr/local/lib/python3.6/dist-packages (from elephas) (3.0.0)\n",
      "Requirement already satisfied: cython in /usr/local/lib/python3.6/dist-packages (from elephas) (0.29.21)\n",
      "Collecting hyperas\n",
      "  Downloading https://files.pythonhosted.org/packages/04/34/87ad6ffb42df9c1fa9c4c906f65813d42ad70d68c66af4ffff048c228cd4/hyperas-0.4.1-py3-none-any.whl\n",
      "Requirement already satisfied: flask in /usr/local/lib/python3.6/dist-packages (from elephas) (1.1.2)\n",
      "Requirement already satisfied: wrapt>=1.11.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow->elephas) (1.12.1)\n",
      "Requirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow->elephas) (1.30.0)\n",
      "Requirement already satisfied: keras-applications>=1.0.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow->elephas) (1.0.8)\n",
      "Requirement already satisfied: protobuf>=3.6.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow->elephas) (3.12.2)\n",
      "Requirement already satisfied: absl-py>=0.7.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow->elephas) (0.9.0)\n",
      "Requirement already satisfied: google-pasta>=0.1.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow->elephas) (0.2.0)\n",
      "Requirement already satisfied: numpy<2.0,>=1.14.5 in /usr/local/lib/python3.6/dist-packages (from tensorflow->elephas) (1.18.5)\n",
      "Requirement already satisfied: tensorboard<1.15.0,>=1.14.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow->elephas) (1.14.0)\n",
      "Requirement already satisfied: astor>=0.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow->elephas) (0.8.1)\n",
      "Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.6/dist-packages (from tensorflow->elephas) (0.34.2)\n",
      "Requirement already satisfied: keras-preprocessing>=1.0.5 in /usr/local/lib/python3.6/dist-packages (from tensorflow->elephas) (1.1.2)\n",
      "Requirement already satisfied: tensorflow-estimator<1.15.0rc0,>=1.14.0rc0 in /usr/local/lib/python3.6/dist-packages (from tensorflow->elephas) (1.14.0)\n",
      "Requirement already satisfied: gast>=0.2.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow->elephas) (0.3.3)\n",
      "Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow->elephas) (1.1.0)\n",
      "Requirement already satisfied: scipy>=0.14 in /usr/local/lib/python3.6/dist-packages (from keras->elephas) (1.4.1)\n",
      "Requirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from keras->elephas) (2.10.0)\n",
      "Requirement already satisfied: pyyaml in /usr/local/lib/python3.6/dist-packages (from keras->elephas) (3.13)\n",
      "Requirement already satisfied: py4j==0.10.9 in /usr/local/lib/python3.6/dist-packages (from pyspark->elephas) (0.10.9)\n",
      "Requirement already satisfied: nbconvert in /usr/local/lib/python3.6/dist-packages (from hyperas->elephas) (5.6.1)\n",
      "Requirement already satisfied: entrypoints in /usr/local/lib/python3.6/dist-packages (from hyperas->elephas) (0.3)\n",
      "Requirement already satisfied: jupyter in /usr/local/lib/python3.6/dist-packages (from hyperas->elephas) (1.0.0)\n",
      "Requirement already satisfied: nbformat in /usr/local/lib/python3.6/dist-packages (from hyperas->elephas) (5.0.7)\n",
      "Requirement already satisfied: hyperopt in /usr/local/lib/python3.6/dist-packages (from hyperas->elephas) (0.1.2)\n",
      "Requirement already satisfied: click>=5.1 in /usr/local/lib/python3.6/dist-packages (from flask->elephas) (7.1.2)\n",
      "Requirement already satisfied: itsdangerous>=0.24 in /usr/local/lib/python3.6/dist-packages (from flask->elephas) (1.1.0)\n",
      "Requirement already satisfied: Jinja2>=2.10.1 in /usr/local/lib/python3.6/dist-packages (from flask->elephas) (2.11.2)\n",
      "Requirement already satisfied: Werkzeug>=0.15 in /usr/local/lib/python3.6/dist-packages (from flask->elephas) (1.0.1)\n",
      "Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from protobuf>=3.6.1->tensorflow->elephas) (49.1.0)\n",
      "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.6/dist-packages (from tensorboard<1.15.0,>=1.14.0->tensorflow->elephas) (3.2.2)\n",
      "Requirement already satisfied: pygments in /usr/local/lib/python3.6/dist-packages (from nbconvert->hyperas->elephas) (2.1.3)\n",
      "Requirement already satisfied: pandocfilters>=1.4.1 in /usr/local/lib/python3.6/dist-packages (from nbconvert->hyperas->elephas) (1.4.2)\n",
      "Requirement already satisfied: defusedxml in /usr/local/lib/python3.6/dist-packages (from nbconvert->hyperas->elephas) (0.6.0)\n",
      "Requirement already satisfied: bleach in /usr/local/lib/python3.6/dist-packages (from nbconvert->hyperas->elephas) (3.1.5)\n",
      "Requirement already satisfied: jupyter-core in /usr/local/lib/python3.6/dist-packages (from nbconvert->hyperas->elephas) (4.6.3)\n",
      "Requirement already satisfied: traitlets>=4.2 in /usr/local/lib/python3.6/dist-packages (from nbconvert->hyperas->elephas) (4.3.3)\n",
      "Requirement already satisfied: testpath in /usr/local/lib/python3.6/dist-packages (from nbconvert->hyperas->elephas) (0.4.4)\n",
      "Requirement already satisfied: mistune<2,>=0.8.1 in /usr/local/lib/python3.6/dist-packages (from nbconvert->hyperas->elephas) (0.8.4)\n",
      "Requirement already satisfied: qtconsole in /usr/local/lib/python3.6/dist-packages (from jupyter->hyperas->elephas) (4.7.5)\n",
      "Requirement already satisfied: notebook in /usr/local/lib/python3.6/dist-packages (from jupyter->hyperas->elephas) (5.2.2)\n",
      "Requirement already satisfied: ipykernel in /usr/local/lib/python3.6/dist-packages (from jupyter->hyperas->elephas) (4.10.1)\n",
      "Requirement already satisfied: ipywidgets in /usr/local/lib/python3.6/dist-packages (from jupyter->hyperas->elephas) (7.5.1)\n",
      "Requirement already satisfied: jupyter-console in /usr/local/lib/python3.6/dist-packages (from jupyter->hyperas->elephas) (5.2.0)\n",
      "Requirement already satisfied: ipython-genutils in /usr/local/lib/python3.6/dist-packages (from nbformat->hyperas->elephas) (0.2.0)\n",
      "Requirement already satisfied: jsonschema!=2.5.0,>=2.4 in /usr/local/lib/python3.6/dist-packages (from nbformat->hyperas->elephas) (2.6.0)\n",
      "Requirement already satisfied: networkx in /usr/local/lib/python3.6/dist-packages (from hyperopt->hyperas->elephas) (2.4)\n",
      "Requirement already satisfied: pymongo in /usr/local/lib/python3.6/dist-packages (from hyperopt->hyperas->elephas) (3.10.1)\n",
      "Requirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from hyperopt->hyperas->elephas) (4.41.1)\n",
      "Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from hyperopt->hyperas->elephas) (0.16.0)\n",
      "Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.6/dist-packages (from Jinja2>=2.10.1->flask->elephas) (1.1.1)\n",
      "Requirement already satisfied: importlib-metadata; python_version < \"3.8\" in /usr/local/lib/python3.6/dist-packages (from markdown>=2.6.8->tensorboard<1.15.0,>=1.14.0->tensorflow->elephas) (1.7.0)\n",
      "Requirement already satisfied: webencodings in /usr/local/lib/python3.6/dist-packages (from bleach->nbconvert->hyperas->elephas) (0.5.1)\n",
      "Requirement already satisfied: packaging in /usr/local/lib/python3.6/dist-packages (from bleach->nbconvert->hyperas->elephas) (20.4)\n",
      "Requirement already satisfied: decorator in /usr/local/lib/python3.6/dist-packages (from traitlets>=4.2->nbconvert->hyperas->elephas) (4.4.2)\n",
      "Requirement already satisfied: jupyter-client>=4.1 in /usr/local/lib/python3.6/dist-packages (from qtconsole->jupyter->hyperas->elephas) (5.3.5)\n",
      "Requirement already satisfied: qtpy in /usr/local/lib/python3.6/dist-packages (from qtconsole->jupyter->hyperas->elephas) (1.9.0)\n",
      "Requirement already satisfied: pyzmq>=17.1 in /usr/local/lib/python3.6/dist-packages (from qtconsole->jupyter->hyperas->elephas) (19.0.1)\n",
      "Requirement already satisfied: terminado>=0.3.3; sys_platform != \"win32\" in /usr/local/lib/python3.6/dist-packages (from notebook->jupyter->hyperas->elephas) (0.8.3)\n",
      "Requirement already satisfied: tornado>=4 in /usr/local/lib/python3.6/dist-packages (from notebook->jupyter->hyperas->elephas) (4.5.3)\n",
      "Requirement already satisfied: ipython>=4.0.0 in /usr/local/lib/python3.6/dist-packages (from ipykernel->jupyter->hyperas->elephas) (5.5.0)\n",
      "Requirement already satisfied: widgetsnbextension~=3.5.0 in /usr/local/lib/python3.6/dist-packages (from ipywidgets->jupyter->hyperas->elephas) (3.5.1)\n",
      "Requirement already satisfied: prompt-toolkit<2.0.0,>=1.0.0 in /usr/local/lib/python3.6/dist-packages (from jupyter-console->jupyter->hyperas->elephas) (1.0.18)\n",
      "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.6/dist-packages (from importlib-metadata; python_version < \"3.8\"->markdown>=2.6.8->tensorboard<1.15.0,>=1.14.0->tensorflow->elephas) (3.1.0)\n",
      "Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.6/dist-packages (from packaging->bleach->nbconvert->hyperas->elephas) (2.4.7)\n",
      "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.6/dist-packages (from jupyter-client>=4.1->qtconsole->jupyter->hyperas->elephas) (2.8.1)\n",
      "Requirement already satisfied: ptyprocess; os_name != \"nt\" in /usr/local/lib/python3.6/dist-packages (from terminado>=0.3.3; sys_platform != \"win32\"->notebook->jupyter->hyperas->elephas) (0.6.0)\n",
      "Requirement already satisfied: simplegeneric>0.8 in /usr/local/lib/python3.6/dist-packages (from ipython>=4.0.0->ipykernel->jupyter->hyperas->elephas) (0.8.1)\n",
      "Requirement already satisfied: pickleshare in /usr/local/lib/python3.6/dist-packages (from ipython>=4.0.0->ipykernel->jupyter->hyperas->elephas) (0.7.5)\n",
      "Requirement already satisfied: pexpect; sys_platform != \"win32\" in /usr/local/lib/python3.6/dist-packages (from ipython>=4.0.0->ipykernel->jupyter->hyperas->elephas) (4.8.0)\n",
      "Requirement already satisfied: wcwidth in /usr/local/lib/python3.6/dist-packages (from prompt-toolkit<2.0.0,>=1.0.0->jupyter-console->jupyter->hyperas->elephas) (0.2.5)\n",
      "Installing collected packages: hyperas, elephas\n",
      "Successfully installed elephas-0.4.3 hyperas-0.4.1\n"
     ]
    }
   ],
   "source": [
    "!pip install elephas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1T5GNiYERbvf"
   },
   "source": [
    "The data for this project is taken from Kaggle: Fake and real news dataset. Steps to follow to download data from kaggle directly into jupyter notebook:\n",
    "\n",
    "1. Install kaggle using pip command\n",
    "2. Navigate to presonal kaggle account and click Create New API Token, a json file is downloaded into the system.\n",
    "3. Upload the file into the notebook using file.upload command\n",
    "4. Make a directory called Kaggle and copy the file into the directory. Change the permissions of the file using the below commands. \n",
    "       !mkdir -p ~/.kaggle\n",
    "       !cp kaggle.json ~/.kaggle/\n",
    "       !chmod 600 ~/.kaggle/kaggle.json\n",
    "5. Navigate to the Kaggle dataset page and click on the 3 dots on the right hand side of the page. Click on Copy API command and paste in the notebook cell to download the data. It looks something as below:\n",
    "        !kaggle datasets download -d clmentbisaillon/fake-and-real-news-dataset\n",
    "6. Using unzip command, the data is unzipped and the files are chekced using !ls command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "B6EjrPIcDF2h"
   },
   "outputs": [],
   "source": [
    "from google.colab import files\n",
    "#upload the kaggle.json file\n",
    "files.upload()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "GTyFwpV5DbFZ"
   },
   "outputs": [],
   "source": [
    "#make a directory and copy the json file\n",
    "!mkdir -p ~/.kaggle\n",
    "!cp kaggle.json ~/.kaggle/\n",
    "\n",
    "#change permission to 600: the owner has full read and write access to the file, while no other user can access the file\n",
    "!chmod 600 ~/.kaggle/kaggle.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 67
    },
    "colab_type": "code",
    "id": "jWxL5vxqDrzA",
    "outputId": "db41e185-5a36-4242-b2c9-a2b86cfe476c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading fake-and-real-news-dataset.zip to /content\n",
      " 81% 33.0M/41.0M [00:00<00:00, 45.0MB/s]\n",
      "100% 41.0M/41.0M [00:00<00:00, 75.9MB/s]\n"
     ]
    }
   ],
   "source": [
    "#https://www.kaggle.com/clmentbisaillon/fake-and-real-news-dataset\n",
    "#download the data using Copy API command from the data page\n",
    "!kaggle datasets download -d clmentbisaillon/fake-and-real-news-dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "b-R4bVzrEJgv",
    "outputId": "cab8b8a4-6064-454e-c56a-2b17ee320282"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "fake-and-real-news-dataset.zip\tkaggle.json  sample_data\n"
     ]
    }
   ],
   "source": [
    "#list the files present\n",
    "!ls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 67
    },
    "colab_type": "code",
    "id": "AO1Dd8CbEPUm",
    "outputId": "ef9d61f1-73f3-405d-9cc5-5836c185a84b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Archive:  fake-and-real-news-dataset.zip\n",
      "  inflating: Fake.csv                \n",
      "  inflating: True.csv                \n"
     ]
    }
   ],
   "source": [
    "#unzip the fake news data\n",
    "!unzip \\*.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "NYFOcGLtES8u",
    "outputId": "6278af70-faf9-4730-e8e4-8f9a31bb95b6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "fake-and-real-news-dataset.zip\tFake.csv  kaggle.json  sample_data  True.csv\n"
     ]
    }
   ],
   "source": [
    "#check for the unzipped files\n",
    "!ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "V9E88M4fTpWV"
   },
   "source": [
    "**2. Import statements and load the data**\n",
    "\n",
    "**2.1 Import the libraries**\n",
    "\n",
    "The necessary libraries and modules are imported. The main import statements are:\n",
    "\n",
    "1. SparkContext: Main entry point for Spark functionality. A SparkContext represents the connection to a Spark cluster, and can be used to create RDD and broadcast variables on that cluster\n",
    "2. SparkConf: For configuring Spark and Used to set various Spark parameters as key-value pairs\n",
    "3. SQLContext: The entry point into all relational functionality in Spark is the SQLContext class. To create a basic SQLContext, all needed is a SparkContext."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "9vU9wKE5x2iL",
    "outputId": "2c7aeb53-3c76-4ed5-a10d-e4531dda8be5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING\n"
     ]
    }
   ],
   "source": [
    "from pyspark import SparkContext, SparkConf\n",
    "from pyspark.sql import SQLContext, SparkSession\n",
    "from pyspark.sql import *\n",
    "from pyspark.sql.functions import isnan, when, count, col, udf,  lower, regexp_replace, rand, round, lit\n",
    "from pyspark.ml.feature import RegexTokenizer, StopWordsRemover, StringIndexer, HashingTF, IDF, IndexToString, Word2Vec\n",
    "from pyspark.sql.types import IntegerType\n",
    "from pyspark.ml import Pipeline\n",
    "from pyspark.ml.tuning import ParamGridBuilder, CrossValidator\n",
    "from pyspark.ml.evaluation import BinaryClassificationEvaluator\n",
    "from pyspark.ml.classification import RandomForestClassifier\n",
    "from pyspark.ml.classification import NaiveBayes\n",
    "from pyspark.ml.classification import GBTClassifier\n",
    "from pyspark.ml.evaluation import MulticlassClassificationEvaluator\n",
    "from pyspark.mllib.evaluation import MulticlassMetrics, BinaryClassificationMetrics\n",
    "from keras.models import Sequential\n",
    "from keras.layers.core import Dense, Dropout, Activation\n",
    "from keras import optimizers, regularizers\n",
    "from keras.optimizers import Adam\n",
    "import keras\n",
    "from elephas.spark_model import SparkModel\n",
    "from elephas.ml_model import ElephasEstimator\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "sfQD9r7UkckQ"
   },
   "outputs": [],
   "source": [
    "#create a SparkConf with the following attributes:\n",
    "#setAppName: Set application name\n",
    "#setMaster: Set master URL to connect to\n",
    "#spark driver memory is a flag that controls the amount of memory to allocate for a driver, which is 1GB by default. It is\n",
    "#set to 15g to accomadate the text processing and training of models\n",
    "conf = SparkConf().setAppName('Fake_news_detection').setMaster('local[4]').set('spark.driver.memory', '15g')\n",
    "\n",
    "#create an entry point using SparkConf: conf\n",
    "sc = SparkContext(conf=conf)\n",
    "\n",
    "#SQLcontext is created using sc for realational functionality\n",
    "sql_context = SQLContext(sc)\n",
    "\n",
    "#spark = SparkSession.builder.getOrCreate()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "DydwmdOylkev"
   },
   "source": [
    "**2.2 Load the data**\n",
    "\n",
    "The dataset is divided into two CSV files: Fake News and Real News. The two files are read individually and joined together to a single dataset. The target feature for Real data is set to True and Fake data is set to false. \n",
    "\n",
    "sql_context.read.csv is used to read the csv files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "kEXUiQVtyZQp"
   },
   "outputs": [],
   "source": [
    "#read the real news data\n",
    "true_data = sql_context.read.csv('True.csv', header = True, inferSchema = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "D6I3D4v9m7r_"
   },
   "source": [
    "New column: target is created using withColumn and lit() funcitons.\n",
    "\n",
    "        withColumn: Returns a new DataFrame by adding a column or replacing the existing column that has the same name. \n",
    "        Parameters:\tcolName: string, name of the new column.\n",
    "                       col: a Column expression for the new column.\n",
    "        lit() function is used for column expression. It is a way for us to interact with column literals in PySpark"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 202
    },
    "colab_type": "code",
    "id": "vny3FbEYnpih",
    "outputId": "51bf185b-8ece-43af-d840-9cc2cbbeb5cb"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+--------------------+------------+------------------+------+\n",
      "|               title|                text|     subject|              date|target|\n",
      "+--------------------+--------------------+------------+------------------+------+\n",
      "|As U.S. budget fi...|WASHINGTON (Reute...|politicsNews|December 31, 2017 |  True|\n",
      "|U.S. military to ...|WASHINGTON (Reute...|politicsNews|December 29, 2017 |  True|\n",
      "|Senior U.S. Repub...|WASHINGTON (Reute...|politicsNews|December 31, 2017 |  True|\n",
      "|FBI Russia probe ...|WASHINGTON (Reute...|politicsNews|December 30, 2017 |  True|\n",
      "|Trump wants Posta...|SEATTLE/WASHINGTO...|politicsNews|December 29, 2017 |  True|\n",
      "+--------------------+--------------------+------------+------------------+------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#create a new column \"target\" with values as True\n",
    "true_data = true_data.withColumn(\"target\", lit(\"True\"))\n",
    "\n",
    "#display the true news data\n",
    "true_data.show(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "lMDrzQu1TRi8",
    "outputId": "842aadc4-c659-4b54-e75b-b2c875665937"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "21417"
      ]
     },
     "execution_count": 16,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#check the length of the true data\n",
    "true_data.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "hgu7BQ5-yg_J"
   },
   "outputs": [],
   "source": [
    "#read the fake news data\n",
    "fake_data = sql_context.read.csv('Fake.csv', header=True, inferSchema=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 202
    },
    "colab_type": "code",
    "id": "KFyGGXqARqHd",
    "outputId": "f9db583a-f48d-41c4-fb63-a60a1f98e0f9"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+--------------------+-------+-----------------+------+\n",
      "|               title|                text|subject|             date|target|\n",
      "+--------------------+--------------------+-------+-----------------+------+\n",
      "| Donald Trump Sen...|Donald Trump just...|   News|December 31, 2017|  Fake|\n",
      "| Drunk Bragging T...|House Intelligenc...|   News|December 31, 2017|  Fake|\n",
      "| Sheriff David Cl...|On Friday, it was...|   News|December 30, 2017|  Fake|\n",
      "| Trump Is So Obse...|On Christmas day,...|   News|December 29, 2017|  Fake|\n",
      "| Pope Francis Jus...|Pope Francis used...|   News|December 25, 2017|  Fake|\n",
      "+--------------------+--------------------+-------+-----------------+------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#create a new column \"target\" with values as Fake\n",
    "fake_data = fake_data.withColumn(\"target\", lit(\"Fake\"))\n",
    "\n",
    "#display the fake news data\n",
    "fake_data.show(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "Yzte3kKwTUcl",
    "outputId": "3d0d0b9c-9794-4f07-9c61-c6292b7143b9"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "23489"
      ]
     },
     "execution_count": 19,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#check the length of fake news \n",
    "fake_data.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "KGzah0bOSq8k"
   },
   "outputs": [],
   "source": [
    "#concatanate fake and real news using union function \n",
    "data = true_data.union(fake_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "xSMJFfbLRhPe"
   },
   "outputs": [],
   "source": [
    "#shuffle the data\n",
    "data = data.orderBy(rand())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 202
    },
    "colab_type": "code",
    "id": "YysbEPq4S63L",
    "outputId": "dea259c8-ffe2-436a-dcaf-fc34a9735a22"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+--------------------+-----------+------------------+------+\n",
      "|               title|                text|    subject|              date|target|\n",
      "+--------------------+--------------------+-----------+------------------+------+\n",
      "|US Delta Force Be...|21st Century Wire...|Middle-east|     March 3, 2016|  Fake|\n",
      "| WATCH: CNN Host ...|One of Trump s bi...|       News| February 13, 2017|  Fake|\n",
      "| Florida Continue...|The Florida Depar...|       News|    March 28, 2016|  Fake|\n",
      "|Moldova sends tro...|CHISINAU (Reuters...|  worldnews|September 7, 2017 |  True|\n",
      "|THE REAL TRUTH AB...|This is the best ...|  left-news|       Feb 3, 2016|  Fake|\n",
      "+--------------------+--------------------+-----------+------------------+------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#dispaly the combined data\n",
    "data.show(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "r72POcaES8Wq",
    "outputId": "f605b555-b934-42d1-f35a-e88df5cffeaa"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The length of the data: 44906\n"
     ]
    }
   ],
   "source": [
    "#check the length of combined data\n",
    "print(\"The length of the data: \" +str(data.count()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ni0bbONToxiG"
   },
   "source": [
    "**3. Data Cleaning**\n",
    "\n",
    "Remove the null and NaN values from all the columns of the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "nuvEAGRFU2U-"
   },
   "outputs": [],
   "source": [
    "#remove title, subject and date columns\n",
    "columns = [\"text\", \"subject\", \"date\"]\n",
    "data = data.drop(*columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "cLBJ_qGAo42N"
   },
   "outputs": [],
   "source": [
    "#drop the null values\n",
    "data = data.na.drop()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 118
    },
    "colab_type": "code",
    "id": "S2h7Yn1qV0_e",
    "outputId": "d6560bf1-bb43-4eee-a9d4-d00ff13517fc"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-----+------+\n",
      "|title|target|\n",
      "+-----+------+\n",
      "|    0|     0|\n",
      "+-----+------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#check for null values\n",
    "data.select([count(when(isnan(c), c)).alias(c) for c in data.columns]).show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 84
    },
    "colab_type": "code",
    "id": "4EUXa2ptzSYc",
    "outputId": "c315a754-9b38-4726-e53c-0ad20a8c7ed5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- title: string (nullable = true)\n",
      " |-- target: string (nullable = false)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#print the schema of the data\n",
    "data.printSchema()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xzgKiKyl0Hdr"
   },
   "source": [
    "This shows all the columns of the data are string type and descriptive features columns can be null but the target feature cannot be null."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "P37IisItsfmV"
   },
   "source": [
    "**4. Text Preprocessing**\n",
    "\n",
    "The text preprocessing involves the following steps:\n",
    "\n",
    "1. Tokenize the text into words using RegexTokenizer which is more advanced tokenization based on regular expression matching\n",
    "2. Removal of Stopwords using StopWordsRemover. Default english stopwords are used to remove unwanted words from the data.\n",
    "3. Converting text to numerical. Word2Vec is an Estimator which takes sequences of words representing documents, computes distributed vector representation of words and trains a Word2VecModel. The model maps each word to a unique fixed-size vector. The Word2VecModel transforms each document into a vector using the average of all words in the document. The advantage of this representations is that similar words are close in the vector space, which makes generalization to novel patterns easier and model estimation more robust.\n",
    "\n",
    "4. Convert target categorical values into intergers using StringIndexer.  StringIndexer encodes a string column of labels to a column of label indices.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "itWOzkU1BB9B"
   },
   "outputs": [],
   "source": [
    "'''Tokenization'''\n",
    "#set the Regextokenizer with pattern = \\\\W\n",
    "tokenizer = RegexTokenizer(inputCol = \"title\", outputCol = \"words\", pattern = \"\\\\W\")\n",
    "\n",
    "'''Stopwords removal'''\n",
    "#load the default english stopwords\n",
    "add_stopwords = StopWordsRemover.loadDefaultStopWords(\"english\")\n",
    "\n",
    "#create a stopwords functions for thw words column from the tokenizer\n",
    "remover = StopWordsRemover(inputCol = \"words\", outputCol = \"filtered\").setStopWords(add_stopwords)\n",
    "#remover.transform(tokenizer).show(truncate=False)\n",
    "\n",
    "'''TF-IDF'''\n",
    "#create HashingTF and IDF for the filtered columns\n",
    "hashingTF = HashingTF(inputCol = \"filtered\", outputCol = \"rawFeatures\", numFeatures = 10000)\n",
    "#idf = IDF(inputCol = \"rawFeatures\", outputCol = \"features\", minDocFreq = 5)\n",
    "idf = IDF(inputCol = \"rawFeatures\", outputCol = \"features\", minDocFreq = 5) #minDocFreq: remove sparse terms\n",
    "\n",
    "'''Word to vector'''\n",
    "w2v = Word2Vec(vectorSize = 50, minCount = 0, inputCol = \"filtered\", outputCol = \"features\")\n",
    "\n",
    "'''Label Indexer'''\n",
    "#Convert the target feature categorical values into intergers\n",
    "label_stringIdx = StringIndexer(inputCol = \"target\", outputCol = \"label\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "UfjVSAF31Yma"
   },
   "source": [
    "Next create a Pipelime for the above functions to be perfomed on the data. The stages of the Pipline are set as: tokenization, removal of stopwords, coverting words to vector using Word2vec and converting categorical values to intergers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 202
    },
    "colab_type": "code",
    "id": "XpYmz8jq1U1b",
    "outputId": "40e3963e-90d5-4a46-f426-74cbf151b6ef"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+------+--------------------+--------------------+--------------------+-----+\n",
      "|               title|target|               words|            filtered|            features|label|\n",
      "+--------------------+------+--------------------+--------------------+--------------------+-----+\n",
      "|US Delta Force Be...|  Fake|[us, delta, force...|[us, delta, force...|[0.05744405344805...|  0.0|\n",
      "| WATCH: CNN Host ...|  Fake|[watch, cnn, host...|[watch, cnn, host...|[-0.1560412953064...|  0.0|\n",
      "| Florida Continue...|  Fake|[florida, continu...|[florida, continu...|[-7.2545789589639...|  0.0|\n",
      "|Moldova sends tro...|  True|[moldova, sends, ...|[moldova, sends, ...|[0.03677053409410...|  1.0|\n",
      "|THE REAL TRUTH AB...|  Fake|[the, real, truth...|[real, truth, oba...|[-0.0849549174308...|  0.0|\n",
      "+--------------------+------+--------------------+--------------------+--------------------+-----+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#pipleline for text pre-processing\n",
    "pipeline = Pipeline(stages = [tokenizer, remover, w2v, label_stringIdx])\n",
    "\n",
    "#fit the dat for the pipeline\n",
    "pipelineFit = pipeline.fit(data)\n",
    "dataset = pipelineFit.transform(data)\n",
    "\n",
    "dataset.show(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "abSMYFSg3zR6"
   },
   "source": [
    "The data is converted to numerical format and below shows the schema of the preprocessed data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 185
    },
    "colab_type": "code",
    "id": "ELiqLTTmHqA3",
    "outputId": "9c77586c-5a68-44dc-8275-f9b829673c8a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- title: string (nullable = true)\n",
      " |-- target: string (nullable = false)\n",
      " |-- words: array (nullable = true)\n",
      " |    |-- element: string (containsNull = true)\n",
      " |-- filtered: array (nullable = true)\n",
      " |    |-- element: string (containsNull = true)\n",
      " |-- features: vector (nullable = true)\n",
      " |-- label: double (nullable = false)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#check the schema for preprocessed data\n",
    "dataset.printSchema()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "bsav6mnz4EDD"
   },
   "source": [
    "**5. Function for Evaluation Metrics**\n",
    "\n",
    "This project is implmented using three machine learning algorithms. The performance of the models are evaluated confusion matrix, F1 score and Area under the curve. The model with less False positives and false negatives, high F1 score and AUC will be considered as a better model for fake news detection. Matplotlib and Numpy is used to plot the confusion matrix. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "4Kuv7a9AhJvO"
   },
   "outputs": [],
   "source": [
    "#function to calculate the evaluation metrics\n",
    "def eval_metrics(predictions):\n",
    "    '''Accuracy'''\n",
    "    #label = predictions.select(['prediction', 'label'])\n",
    "    evaluator = MulticlassClassificationEvaluator(labelCol = \"label\", predictionCol = \"prediction\", metricName = \"accuracy\")\n",
    "    accuracy = evaluator.evaluate(predictions)*100\n",
    "\n",
    "    #select prediction and label columns to calculate metric values\n",
    "    results = predictions.select(['prediction', 'label'])\n",
    "    #convert the results to rdd to get predictions and labels\n",
    "    predictionAndLabels = results.rdd\n",
    "\n",
    "    '''Area under the curve'''\n",
    "    #binary_evaluator = BinaryClassificationEvaluator()\n",
    "    #auc = binary_evaluator.evaluate(predictions, {binary_evaluator.metricName: \"areaUnderROC\"})\n",
    "    binary_evaluator = BinaryClassificationMetrics(predictionAndLabels)\n",
    "    auc = binary_evaluator.areaUnderROC\n",
    "\n",
    "    '''Classification report'''\n",
    "    #create a multiclassmetrics instance\n",
    "    metrics = MulticlassMetrics(predictionAndLabels)\n",
    "\n",
    "    #compute Recall with respect to Real and fake news data\n",
    "    real_recall = metrics.recall(1.0)\n",
    "    fake_recall = metrics.recall(0.0)\n",
    "\n",
    "    #compute Precision with respect to Real and fake news data\n",
    "    real_precision = metrics.precision(1.0)\n",
    "    fake_precision = metrics.precision(0.0)\n",
    "\n",
    "    #compute F1 Score with respect to Real and fake news data\n",
    "    real_f1 = metrics.fMeasure(1.0)\n",
    "    fake_f1 = metrics.fMeasure(0.0)\n",
    "\n",
    "    #create a classificaiton report for the model\n",
    "    real = Row(News = 'Real', recall = real_recall, precision = real_precision, f1_score = real_f1)\n",
    "    fake = Row(News = 'Fake', recall = fake_recall, precision = fake_precision, f1_score = fake_f1) \n",
    "  \n",
    "    classification_report = [real, fake]\n",
    "    classification_report_df = sql_context.createDataFrame(classification_report)\n",
    "\n",
    "    '''Confusion matrix'''\n",
    "    conf_matrix = metrics.confusionMatrix().toArray()\n",
    "\n",
    "    return accuracy, auc, classification_report_df, conf_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "J1E4dDnQLozO"
   },
   "outputs": [],
   "source": [
    "#function for confusion matrix\n",
    "def plot_confusion_matrix(cm, target_names, title='Confusion Matrix', cmap=plt.cm.Blues):\n",
    "    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n",
    "    plt.title(title)\n",
    "    plt.colorbar()\n",
    "    tick_marks = np.arange(len(target_names))\n",
    "    plt.xticks(tick_marks, target_names)\n",
    "    plt.yticks(tick_marks, target_names)\n",
    "    plt.tight_layout()\n",
    "\n",
    "    width, height = cm.shape\n",
    "\n",
    "    for x in range(width):\n",
    "        for y in range(height):\n",
    "            plt.annotate(str(cm[x][y]), xy=(y, x), \n",
    "                        horizontalalignment='center',\n",
    "                        verticalalignment='center')\n",
    "            \n",
    "    plt.ylabel('True Label')\n",
    "    plt.xlabel('Predicted Label')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "SVoZZnmS4AGR"
   },
   "source": [
    "**6. Modelling**\n",
    "\n",
    "This section implements different algorithms to predict the fake news. Initially the data is split into train and test in the ratio 80:20 using randomSplit function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "q5Q2-4N2zuUX"
   },
   "outputs": [],
   "source": [
    "#split the data\n",
    "train, test = dataset.randomSplit([0.8, 0.2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 50
    },
    "colab_type": "code",
    "id": "WgMbCeQZ4mEn",
    "outputId": "0589adb6-d983-4a2e-b7ab-818b206d62d2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Dataset Count: 35873\n",
      "Test Dataset Count: 9033\n"
     ]
    }
   ],
   "source": [
    "#check the train and test data count\n",
    "print(\"Training Dataset Count: \" + str(train.count()))\n",
    "print(\"Test Dataset Count: \" + str(test.count()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9mILXdfVbmNL"
   },
   "source": [
    "**6.1 Decision Tree**\n",
    "\n",
    "A decision tree is a flowchart-like tree structure with internal node representing feature, the branch represents a decision rule, and each leaf node represents the outcome. The topmost node is known as the root node and the bottom nodes are leaf nodes. Decision trees are white box ML algorithms that can handle high dimensional data with good accuracy.\n",
    "\n",
    "The decision trees selects the best feature to split at the root node based on Attribute Selection measures: Gini Index or Entropy. The measured values is calculated at every step before a node split till the required number of nodes are grown. The ASM used here is Gini Index. An attribute with higher Gini Index is considered for the split. Gini Index considers a binary split for each attribute using the below formula. \n",
    "\n",
    "$Gini(D) = 1 - \\sum_{i=1}^m P_i^2$\n",
    "\n",
    "Where, Pi is the probability that a tuple in D belongs to class Ci.\n",
    "\n",
    "The decision parameters:\n",
    "\n",
    "  1. labelCol: Label column name\n",
    "  2. featureCol: features column name\n",
    "  3. impurity: Gini index.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "r7r3loRpbomx"
   },
   "outputs": [],
   "source": [
    "from pyspark.ml.classification import DecisionTreeClassifier\n",
    "#define decision tree\n",
    "dt = DecisionTreeClassifier(labelCol=\"label\", featuresCol = \"features\")\n",
    "\n",
    "#train the model and predict on test data\n",
    "dtModel = dt.fit(train)\n",
    "dt_predictions = dtModel.transform(test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "dJS9mr49cBi6"
   },
   "outputs": [],
   "source": [
    "#calculate the evaluation metrics of naive bayes model\n",
    "dt_accuracy, dt_auc, dt_classification_report_df, dt_conf_matrix = eval_metrics(dt_predictions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 50
    },
    "colab_type": "code",
    "id": "2XILnMIzcObn",
    "outputId": "7f43299c-b493-4263-cbb9-04950162aae0"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Decision Tree accuracy is: 88.07\n",
      "Decision Tree AUC is: 0.88\n"
     ]
    }
   ],
   "source": [
    "#print the auuracy and area under the curve\n",
    "print(\"Decision Tree accuracy is: {:0.2f}\".format(dt_accuracy))\n",
    "print(\"Decision Tree AUC is: {:0.2f}\".format(dt_auc))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 134
    },
    "colab_type": "code",
    "id": "MhTI-fNIch4y",
    "outputId": "babb8a24-e814-4f41-e38c-4cb45291f7ef"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+----+------------------+------------------+------------------+\n",
      "|News|            recall|         precision|          f1_score|\n",
      "+----+------------------+------------------+------------------+\n",
      "|Real|0.8629730962152303|0.8880807132801501|0.8753469010175764|\n",
      "|Fake|0.8973531310522918|0.8740306015510375| 0.885538330855808|\n",
      "+----+------------------+------------------+------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#display the classification report\n",
    "dt_classification_report_df.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 67
    },
    "colab_type": "code",
    "id": "tgxwFuI3cneW",
    "outputId": "9265ec47-80fa-4285-e621-24b4dbe22b70"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion Matrix\n",
      "[[4170.  477.]\n",
      " [ 601. 3785.]]\n"
     ]
    }
   ],
   "source": [
    "#print the confusion matrix\n",
    "print(\"Confusion Matrix\")\n",
    "print(dt_conf_matrix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 311
    },
    "colab_type": "code",
    "id": "hT6YimMzcrDz",
    "outputId": "b92c6c55-395a-49f8-b401-5fb066da7973"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWMAAAEmCAYAAAC3V/E+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deZxdRZ3+8c/TSzYSyB5CAgQxbKIEiITdALIOGlBRECEgDjoC4z7KjD9FEBUVUQZxxAEJiwYQkbAMGFmEOLIkEAIhLBlDICGQPQGy9fL9/XGqwyXpvn27ud13yfPmdV65p06dOnXS4dt169SpUkRgZmalVVPqCpiZmYOxmVlZcDA2MysDDsZmZmXAwdjMrAw4GJuZlQEHY+swSb0l3SFplaRb3kU5p0r6czHrVgqS/kfSxFLXwyqbg3EVk/RpSdMlvSlpUQoaBxeh6E8Aw4BBEXFSZwuJiBsj4qgi1OcdJI2XFJJu2yR9r5T+YIHlXCDphvbyRcSxETGpk9U1AxyMq5akrwI/B35AFjh3AK4EJhSh+B2BFyKisQhldZUlwAGSBuWkTQReKNYFlPH/Q1YcEeGtyjZgG+BN4KQ8eXqSBetX0/ZzoGc6Nh5YAHwNWAwsAs5Mx74HbAAa0jXOAi4AbsgpexQQQF3aPwP4B/AGMA84NSd9Ws55BwKPA6vSnwfmHHsQuAj4Wyrnz8DgNu6tpf7/BZyT0mqBhcB3gAdz8v4CeAVYDcwADknpx2xyn0/l1OPiVI+1wHtT2ufS8V8Bt+aUfwlwH6BS/7vwVt6bf6tXpwOAXsBtefL8B7A/MAbYC9gP+HbO8W3JgvoIsoD7S0kDIuK7ZK3tmyKib0Rcna8ikrYCLgeOjYh+ZAF3Ziv5BgJ3pbyDgJ8Bd23Ssv00cCYwFOgBfD3ftYHrgNPT56OBZ8h+8eR6nOzvYCDwO+AWSb0i4p5N7nOvnHNOA84G+gHzNynva8D7JZ0h6RCyv7uJEeF5BywvB+PqNAhYGvm7EU4FLoyIxRGxhKzFe1rO8YZ0vCEi7iZrHe7ayfo0A3tK6h0RiyJidit5/gl4MSKuj4jGiPg98BzwkZw8v42IFyJiLXAzWRBtU0T8LzBQ0q5kQfm6VvLcEBHL0jUvJfvG0N59XhsRs9M5DZuUt4bs7/FnwA3AeRGxoJ3yzByMq9QyYLCkujx5tuOdrbr5KW1jGZsE8zVA345WJCLeAj4FfAFYJOkuSbsVUJ+WOo3I2X+tE/W5HjgXOIxWvilI+rqkOWlkyEqybwOD2ynzlXwHI+JRsm4Zkf3SMGuXg3F1+juwHjghT55XyR7EtdiBzb/CF+otoE/O/ra5ByPi3og4EhhO1tr9TQH1aanTwk7WqcX1wBeBu1OrdaPUjfBvwCeBARHRn6y/Wi1Vb6PMvF0Oks4ha2G/mso3a5eDcRWKiFVkD6p+KekESX0k1Us6VtKPU7bfA9+WNETS4JS/3WFcbZgJHCppB0nbAOe3HJA0TNKE1He8nqy7o7mVMu4GdknD8eokfQrYA7izk3UCICLmAR8i6yPfVD+gkWzkRZ2k7wBb5xx/HRjVkRETknYBvg98hqy74t8k5e1OMQMH46qV+j+/SvZQbgnZV+tzgT+lLN8HpgOzgKeBJ1JaZ641FbgplTWDdwbQmlSPV4HlZIHxX1opYxlwPNkDsGVkLcrjI2JpZ+q0SdnTIqK1Vv+9wD1kw93mA+t4ZxdEywstyyQ90d51UrfQDcAlEfFURLwI/DtwvaSe7+YerPrJD3nNzErPLWMzszLgYGxmVgYcjM3MyoCDsZlZGcj3UkDFUV3vUI9+pa6GFdHeu+9Q6ipYEc2f/xJLly5V+zkLV7v1jhGNawvKG2uX3BsRxxTz+sVSXcG4Rz967vrJUlfDiuhvj15R6ipYER00bmzRy4zGtQX/f79u5i/be7uyZKoqGJvZlkhQBTOZOhibWWUToKL2fJSEg7GZVT63jM3MSk1QU1vqSrxrDsZmVvncTWFmVmKiKropKv8OzGwLp6xlXMhWaIlSraQnJd2Z9neS9KikuZJuktQjpfdM+3PT8VE5ZZyf0p+XdHR713QwNrPKp5rCtsJ9CZiTs38JcFlEvBdYQba2IenPFSn9spQPSXsAJwPvI1vc9kpJeTu2HYzNrPIVsWUsaSTZmoz/nfYFHA78IWWZxNur6ExI+6TjR6T8E4DJEbE+LXAwl2zR3zY5GJtZhVNHWsaDJU3P2c5upcCfky1u0LIizSBgZc6akAt4e23GEaQFCdLxVSn/xvRWzmmVH+CZWWXr2EsfSyOizXeyJR0PLI6IGZLGF6F2BXMwNrPKV7zRFAcBH5V0HNCLbE3EXwD9JdWl1u9I3l4odyGwPbAgLbu1DdmyYS3pLXLPaZW7KcyswnWomyKviDg/IkZGxCiyB3D3R8SpwAPAJ1K2icDt6fOUtE86fn9ka9lNAU5Ooy12AkYDj+W7tlvGZlbZBNR2+Rt43wQmS/o+8CRwdUq/mmzB2blkC+6eDBARsyXdDDxLtgL5ORHRlO8CDsZmVvm64A28iHgQeDB9/getjIaIiHXASW2cfzFwcaHXczA2swrnKTTNzMqD56YwMysDbhmbmZVYB+edKFcOxmZW+dwyNjMrA24Zm5mVmkdTmJmVB7eMzcxKrEpW+nAwNrMK5wVJzczKg1vGZmZlwH3GZmYlJo+mMDMrD24Zm5mVnhyMzcxKK1sCz8HYzKy0lLYK52BsZhVObhmbmZUDB2MzszJQU+OhbWZmpeU+YzOz0pP7jM3MykM1BOPK72gxsy2epIK2AsrpJekxSU9Jmi3peyn9WknzJM1M25iULkmXS5oraZakfXLKmijpxbRNbO/abhmbWcUrYst4PXB4RLwpqR6YJul/0rFvRMQfNsl/LDA6beOAXwHjJA0EvguMBQKYIWlKRKxo68JuGZtZZVMHtnZE5s20W5+2yHPKBOC6dN4jQH9Jw4GjgakRsTwF4KnAMfmu7WBsZhWvA90UgyVNz9nObqWsWkkzgcVkAfXRdOji1BVxmaSeKW0E8ErO6QtSWlvpbXI3hZlVtA6OplgaEWPzZYiIJmCMpP7AbZL2BM4HXgN6AFcB3wQu7HytN+eWsZlVvGI9wMsVESuBB4BjImJR6opYD/wW2C9lWwhsn3PayJTWVnqbHIzNrLIJVKOCtnaLkoakFjGSegNHAs+lfmCURfQTgGfSKVOA09Ooiv2BVRGxCLgXOErSAEkDgKNSWpvcTWFmFa+IoymGA5Mk1ZI1Vm+OiDsl3S9pCNljwJnAF1L+u4HjgLnAGuBMgIhYLuki4PGU78KIWJ7vwg7GZlbxihWMI2IWsHcr6Ye3kT+Ac9o4dg1wTaHXdjA2s4rm16HNzMpF5cdiB2Mzq3CqjrkpHIzNrOI5GJuZlQEHYzOzclD5sdjBuFQimtnwwi2ofit6vOd4GpfMomnJU8SG1fTc87OorjcAjYufoGn5Cy1nEetWpOO9aFo9n8aF0yCaqR20B3XD9t38Os1NNLz8F2LNYqjrRf2OR1PTc+tuvNMtU1NTEweNG8t2I0bwx9vv5Ijxh/DmG28AsHjJYsZ+cD9uufVP/OzSn3DT724EoLGpkefmzOGVRUsYOHDgO8p7ad48Tjv1ZJYvX8be++zLNddeT48ePbr9vspVNbSMu+wNPElNOXN/zpQ0qo18oyQ909qxata0ZBbqOWDjfs1Ww6nfeQLU93tHvrqh+9Bzt5PpudvJ1A3fn5q+26G6XkQ007jgIerfczw9dvs0TStepHnd5mPKm5Y/i2p70nOP06gbMobGRX/v8nszuOLyX7Dr7rtv3L/vwYd5dMZMHp0xk3H7H8AJJ3wMgK9+7Rsb0y/8/g855NAPbRaIAf7j37/JeV/6CrOfm8uA/gO49pqru+1eyl2hr0KXe8Duyteh10bEmJztpS68VkWJDW/SvPolagftsTGtps+QdlusTStepGbA6KyMNYtRz22o6bkNqqmldsBomlfN2+yc5lXzqB24W3aN/jvT/MYCsnHq1lUWLFjAPf9zF2d+9nObHVu9ejV/feB+PjLhhM2O3XzT7/nkp07ZLD0i+OsD9/Oxj38CgFNPm8gdU/5U/IpXsJqamoK2ctZttZPUV9J9kp6Q9LSkCa3keY+kJyV9UNLOku6RNEPSw5J26666drWGhdOo2+5AOtLRFc0NNL/xMrXb7JztN7yJ6vtuPK76vkTDW5uf1/DWxnxSDartAU3r3t0NWF7f+NqXufiHP271f/47bv8T4w8/gq23fucv3jVr1jD13ns44WMf3+ycZcuWsU3//tTVZb2KI0aO5NVX8845s+Up0nzGpdSVwbh3ThfFbcA64MSI2Ac4DLhUOd8bJO0K3AqcERGPk01Td15E7At8HbiytYtIOrtlbtJoXNuFt1McTateQnW9qekztEPnNa96iZqthqO6Xl1UMyuGu++6k6FDhrLPvpv330Pbrd+77ryDAw48qNUuCmtfNXRTdOUDvLURMaZlJy1h8gNJhwLNZBMtD0uHhwC3Ax+LiGcl9QUOBG7J+QvsSSsi4iqywE1Nn6Fl//27+a1FNK2eR9Ps+RCN0NTAhvlT6bHjkXnPa1r5IrWpiwJaWsJvbtzPWspbbXae6rfKjvXoS0Qz0bQBah3Qu8rf//dv3HnnFO65527Wr1vH6tWrOfP0z/Db625g6dKlTH/8MW76w22bnXfLzZM5qZUgDTBo0CBWrVxJY2MjdXV1LFywgO22yztP+ZalSl766M5OlFPJgu6+KUi/DrREhVXAy8DBOfVauUmf8+6blViB6rc7gF7vO4Ne7zs9G9nQb0S7gTia1tP85qvUbL3TxjT1GUqsX0Xz+tVEc1PWn7z1qM3Ordl6J5qWPwdA88r/o6bfiKr4h1uuLrr4h/zfSwt4fu5LXHfjZMYfdji/ve4GAG679Q8ce9zx9Or1zl+Gq1atYtpDf+UjH92s5w7IAs2h4w/jj7dmy6/deP0kjv9I63m3RAKkwrZy1p3BeBtgcUQ0SDoM2DHn2AbgRLJ5QT8dEauBeZJOgo0rsO7VjXXtdo1LnmLd7Guh4U3WPzeZhpfv33isaeU/qOm3Paqt35gm1VA38hAa/jGFDc/9jtr+76Wm9yAAGhY9SlN6mFc7aHeicR3rn72exiUzqRt+QLfel73tlpsn88mTN2/9TvnTbRxx5FFstdU7v9mc8JHjePXVVwG4+AeXcPnPf8b7dnsvy5Yv44zPntUtda4M1TGaQl31ZF3SmxHRN2d/MHAH0BeYDuxPtrIqwJ0RsWea1HkqcBHwNNlKq8PJFgWcHBF5lzmp6TM0eu76yaLfi5XOisevKHUVrIgOGjeWGTOmFzUq9tp2l9jh9MsLyvviT46d0d6yS6XSZX3GuYE47S8F2mqW7ZnyrAQ+mJOedzVVMzOojj5jv4FnZpWtAvqDC+FgbGYVTUBNAevblTsHYzOreA7GZmal5m4KM7PSy8YZV340djA2swpX/mOIC1He0xiZmRWgWG/gSeol6TFJT0maLel7KX0nSY9KmivpJkk9UnrPtD83HR+VU9b5Kf15SUe3d20HYzOreEV8A289cHhE7AWMAY6RtD9wCXBZRLwXWAG0vAJ5FrAipV+W8iFpD+Bk4H1k70tcKak234UdjM2sshXYKi4kFkemZQau+rQFcDjwh5Q+CWiZkHpC2icdPyLNRjmB7K3h9RExD5gL7Jfv2g7GZlbRWh7gFdgyHtwy5W7azt6sPKlW0kxgMdn0DP9HNnFZY8qygGzWSdKfrwCk46uAQbnprZzTKj/AM7OK14Hnd0vbm5siIpqAMWmunNuAblnYwi1jM6t4XTFrW5or5wGyOXX6S2ppvI4EWpZaWQhsn+pQRzY75bLc9FbOaZWDsZlVNmVv4BWytVuUNCS1iJHUGzgSmEMWlD+Rsk0kWwwDYEraJx2/P7KpMKcAJ6fRFjsBo4HH8l3b3RRmVtFaJpcvkuHApDTyoQa4OSLulPQsMFnS94EngZblua8Grpc0F1hONoKCiJgt6WbgWaAROCd1f7TJwdjMKlzxXvqIiFnA3q2k/4NWRkNExDrgpDbKuhi4uNBrOxibWcWrghfwHIzNrPJVw+vQDsZmVtk8a5uZWel51jYzszLhYGxmVgaqIBY7GJtZ5XPL2Mys1PwAz8ys9ERhrzqXOwdjM6t4NVXQNHYwNrOKVwWx2MHYzCpbtopH5UdjB2Mzq3hV0GXcdjCWtE++EyPiieJXx8ys46q9ZXxpnmMtC/SZmZVcFcTitoNxRBzWnRUxM+sMkQ1vq3TtLrskqY+kb0u6Ku2PlnR811fNzKwwNSpsK2eFrIH3W2ADcGDaXwh8v8tqZGbWEQUuRlru/cqFBOOdI+LHQANARKyBKvhOYGZVQUBtjQraylkhQ9s2pFVSA0DSzsD6Lq2VmVkHlHmjtyCFBOPvAvcA20u6ETgIOKMrK2Vm1hHl3gVRiHaDcURMlfQEsD/ZN4IvRcTSLq+ZmVkBtIXN2vYh4GCyrop64LYuq5GZWQdVw0RBhQxtuxL4AvA08AzweUm/7OqKmZkVSgVu7ZYjbS/pAUnPSpot6Usp/QJJCyXNTNtxOeecL2mupOclHZ2TfkxKmyvpW+1du5CW8eHA7hHR8gBvEjC7gPPMzLpFEfuMG4GvRcQTkvoBMyRNTccui4ifbnLdPYCTgfcB2wF/kbRLOvxL4EhgAfC4pCkR8WxbFy4kGM8FdgDmp/3tU5qZWcmJ4r3QERGLgEXp8xuS5gAj8pwyAZgcEeuBeZLmAvulY3Mj4h8AkianvG0G4za7KSTdIWkK0A+YI+lBSQ8Ac1KamVnpdeylj8GSpudsZ7ddrEYBewOPpqRzJc2SdI2kASltBPBKzmkLUlpb6W3K1zL+aZ5jZmZlowO9FEsjYmz75akvcCvw5YhYLelXwEVkgxguIptI7bOdq23r8k0U9NdiXsjMrCu0vIFXtPKkerJAfGNE/BEgIl7POf4b4M60u5Cs67bFyJRGnvRWFTKaYn9Jj0t6U9IGSU2SVrd3nplZdynW3BTKMl0NzImIn+WkD8/JdiLZyDKAKcDJknpK2gkYDTwGPA6MlrSTpB5kD/mm5Lt2IQ/wrkgF3QKMBU4Hdsl7hplZNyriKOODgNOApyXNTGn/DpwiaQxZN8VLwOcBImK2pJvJHsw1AudERBOApHOBe4Fa4JqIyDsKraCXPiJirqTadJHfSnoSOL9j92hmVnxS8V76iIhptB7b785zzsXAxa2k353vvE0VEozXpGb2TEk/Jhv2Uchsb2Zm3aIKXsArKKielvKdC7xF1in9sa6slJlZR1TDfMaFTBTU8rLHOuB7AJJuAj7VhfUyMytYmcfZghQ6UdCmDihqLczMOkmoKiYK6mwwNjMrD9U+haakfdo6RDaNZtnZa7cdePBvvyh1NayIBkz4z1JXwYpo/dzFXVJuufcHFyJfy/jSPMeeK3ZFzMw6qxqGd+V7Hfqw7qyImVlnFPt16FJxn7GZVbwqiMUOxmZW2bI18Co/GjsYm1nFq4aWcSGztknSZyR9J+3vIGm/9s4zM+suLStEt7eVs0IeQl5J9pLHKWn/DbK1nczMSi5bdkkFbeWskG6KcRGxT5qpjYhYkSYOMjMrC1U9tC1Hg6Rasnk8kTQEaO7SWpmZdUCZN3oLUkgwvhy4DRgq6WLgE8C3u7RWZmYFUgV0QRSikFnbbpQ0AziCrHvmhIiY0+U1MzMrUBXE4vaDsaQdgDXAHblpEfFyV1bMzKwQAuqqYGxbId0Ud5H1FwvoBewEPA+8rwvrZWZWsC2iZRwR78/dT7O5fbHLamRm1hGqjpc+OvwGXkQ8IWlcV1TGzKwzVMz1oUukkD7jr+bs1gD7AK92WY3MzDoge+mj1LV49woZK90vZ+tJ1oc8oSsrZWbWETUqbGuPpO0lPSDpWUmzJX0ppQ+UNFXSi+nPASldki6XNFfSrNxFOSRNTPlflDSxvWvnbRmnlz36RcTX278NM7PSKOKsbY3A11J3bD9ghqSpwBnAfRHxI0nfAr4FfBM4FhidtnHAr4BxkgYC3wXGkg2AmCFpSkSsaOvCbbaMJdVFRBNwUDHu0MysK7R0UxSjZRwRiyLiifT5DWAOMIKsN2BSyjYJOCF9ngBcF5lHgP6ShgNHA1MjYnkKwFOBY/JdO1/L+DGy/uGZkqYAtwBv5VT6j+3fmplZF+uiGdkkjQL2Bh4FhkXEonToNWBY+jwCeCXntAUpra30NhUymqIXsAw4nLfHGwfgYGxmZaEDr0MPljQ9Z/+qiLhq00yS+gK3Al+OiNW53SAREZLi3dS3NfmC8dA0kuIZ3g7CG+tT7IqYmXVGtgZewdmXRsTYvOVJ9WSB+MacHoDXJQ2PiEWpG6JlmeuFwPY5p49MaQuB8ZukP5jvuvluoRbom7Z+OZ9bNjOzMiBqCtzaLSlrAl8NzImIn+UcmgK0jIiYCNyek356GlWxP7AqdWfcCxwlaUAaeXFUSmtTvpbxooi4sN3am5mVkChqn/FBwGnA05JmprR/B34E3CzpLGA+8Ml07G7gOGAu2Rw+ZwJExHJJFwGPp3wXRsTyfBfOF4yrYBi1mVW9Ir4OHRHTaDv2HdFK/gDOaaOsa4BrCr12vmC82YXNzMpRVc9n3F6T2sysHBS5m6JkOjxRkJlZuanqlrGZWaWogljsYGxmlU1sOatDm5mVLxV1oqCScTA2s4pX+aHYwdjMKpyAWreMzcxKrwpisYOxmVU6uc/YzKzUPJrCzKxMuGVsZlYGKj8UOxibWaXzOGMzs9Jzn7GZWZlwy9jMrAxUfih2MDazCuc38MzMykQVxGIHYzOrdEJV0FHhYGxmFc8tYzOzEsuGtlV+NHYwNrPKpupoGVfDWGkz28JJhW3tl6NrJC2W9ExO2gWSFkqambbjco6dL2mupOclHZ2TfkxKmyvpW4Xcg4OxmVU8FfhfAa4Fjmkl/bKIGJO2uwEk7QGcDLwvnXOlpFpJtcAvgWOBPYBTUt683E1hZhVNQE2Ruiki4iFJowrMPgGYHBHrgXmS5gL7pWNzI+IfAJImp7zP5ivMLWMzq3gdaBkPljQ9Zzu7wEucK2lW6sYYkNJGAK/k5FmQ0tpKz8stYzOreDWFP8FbGhFjO1j8r4CLgEh/Xgp8toNltMvBuAysXLmSf/3i2cx5djaSuOK/fsPo0bty5umn8PL8+eyw445ce/1k+g8YwAvPP8c5nz+Lp2Y+yf+74CLO+/LXWi3zpZfmcdbpn2b58uWM2Xsffn31JHr06NHNd7ZliKYGNjzyC2huhGimZtsx1O9yHOv//nNoXJ/l2fAGNf13pMe+/0w0rKXhqeuItSsgmqnd6XDqtt8fgHV3fwn12w4A9R5Aj7GbN9yiqYGGWTcQq16B+q2o3/sMavoM6r4bLjPF7KZoTUS8vvFa0m+AO9PuQmD7nKwjUxp50tvUpd0UkprS08dnJN0hqX8nyzlD0hXFrl+5+NY3vsKHjzyax2fOZtqjT7DLrrtz2aWX8KHxh/PE08/xofGHc9mllwAwYMBALvnpzznvS1/NW+YF3z6fL573ZZ585nn69x/A9dde0x23smWqqaPHuPPoeci36HHwN2leMofmFfPoecCX6XnIN+l5yDep6b8TtcP2AqBp/sOo77ZZ/nHn0fjcn4jmxqys2vqN57QWiAGaFjyC6vrQc/x3qNtpPI3PT+muOy1ThXZSdC5iSxqes3si0DLSYgpwsqSeknYCRgOPAY8DoyXtJKkH2UO+dn9IXd1nvDY9fdwTWA6c08XXqzirVq3if6c9zGlnZN96evToQf/+/bn7zjs45dTTATjl1NO5647sZzlk6FD2GftB6urr2ywzInjorw8w4cSPZ+d/5jTuuvP2Lr6TLZckVNcz24mmbMv52hwNa2le9gI1w97/9kmN64kIomkDqu8DKvx/xebXn6Z2ZPacqGbbMTQvfYGIKMq9VKQCh7UVOLTt98DfgV0lLZB0FvBjSU9LmgUcBnwFICJmAzeTPZi7BzgnIpoiohE4F7gXmAPcnPLm1Z3dFH8HPgAgaWeyoR9DgDXAP0fEc5I+Anwb6AEsA07N/YpQjea/NI/Bgwfzxc+fxTOzZjFm73340U8vY/Hi19l2ePYLedi227J4ceF/DcuXLWObbfpTV5f9eLcbMZJFr77aJfW3TEQzG6b9hFizhNodD6Gm/6iNx5pff5qawbug+t4A1I46lIbpV7H+/v8Hjeuo3/tM1BKMmxtZP+0nUFND3XuOpHbbD2x+rXWrUK/sS6ZqalF9L2h4C3r07fL7LFfF6qWIiFNaSb46T/6LgYtbSb8buLsj1+6W0RRp3N0RvN1Uvwo4LyL2Bb4OXJnSpwH7R8TewGTg3woo++yWJ6PLli4pfuW7WFNjI0/NfJKzPvd5Hn5kOn222orLfnrJO/JI1bEUeTWTarLuhcMvJFbOp/mNt3/5NS2aQe3wfTfuNy+Zg7YeSc/DL6LHwd+kcfYtRMNaAHoedgE9D/4G9WMm0jDnjzS/VXn/prtb1mesgrZy1tXBuLekmcBrwDBgqqS+wIHALenYr4GWPpmRwL2Snga+QTaYOq+IuCoixkbE2EGDh3TJTXSl7UaMZLsRIxm73zgAJpz4MWbNfJKhQ4fx2qJFALy2aBFDhgwtuMyBgwaxatVKGhuzfshXFy5g+HbbFb/ythnV96Fm0Gial8wBIDa8SfPK+dQMffufctOCR6nddi8kUbPVENRnEPHW4uz81OKt6TOYmoHvJVYv2PwavbYh1q3Mym9uIhrWQf1WXX1rZU0FbuWsW/qMgR3J/i7OSddcmfM2y5iI2D3l/0/gioh4P/B5oFcX16/khm27LSNHjuTFF54H4K8P3M+uu+/Osf90PL+/8ToAfn/jdRx3/EcKLlMShxw6nttvuzU7/4brOe6fPlr8yhsAsf4NomFN9rlpA01Ln0dbDQOgadFMaobuiWrf7uNX7wE0LX0+nbua5jcXZwG5YQ3R1JClb3iTWDEP9d12s+vVDN2TpgWPAdD82kxqBo32N6cqiMbd0mccEWsk/SvwJ7IuiXmSToqIW5T9K/pARDqXPLgAAArBSURBVDwFbMPbQ0AmdkfdysEll/6Cfz7zdDY0bGDUqJ248tdX09zczBmnncz1k37L9jvswLXXTwbg9dde47CDx/HGG6tRTQ2/uuJyHnniabbeemtOOuF4Lr/yKoZvtx3f+/4P+ezpn+b73/sOH9hrzMYHhFZ8sX41DbNugAiIoHb4GGqH7QlA06InqNv5w+/IX/feY2iYdQPrH/ohAPW7fRT16Evzin/Q8PRN2ZOmCGp3/jA1/bIvjQ0v3EXNNjtQO+z91G5/AA1PXc/6By+E+j7U731Gt95vOaqG+YzVlU9hJb0ZEX1z9u8ge/o4jWwg9XCgnuyVwgslTQAuA1YA9wMfjIjxks4AxkbEufmut/c+Y+PBvz3aNTdjJbHtJ65sP5NVjPXTfkLzqpeLGjl3f//eMen2BwvKO27n/jM68dJHt+jSlnFuIE77ud+1N5uMIyJuBzYbgxUR15JN4GFmtpnKbxf7DTwzq3CCqugzdzA2s8pWJZPLOxibWcWrgljsYGxmVaAKorGDsZlVuM5PAlROHIzNrOK5z9jMrMQq4OW6gjgYm1nlq4Jo7GBsZhXPfcZmZmXAfcZmZqXmlz7MzMqDuynMzEosm5ui1LV49xyMzaziVUEsdjA2sypQBdHYwdjMKp77jM3MykA19Bl39YKkZmZdrljrkUq6RtJiSc/kpA2UNFXSi+nPASldki6XNFfSLEn75JwzMeV/UVJB63k6GJtZ5Sve6tDXsvmScN8C7ouI0cB9aR/gWGB02s4mW9cTSQOB7wLjgP2A77YE8HwcjM2somVxtrD/2hMRDwHLN0meAExKnycBJ+SkXxeZR4D+koYDRwNTI2J5RKwAptLKmp+bcp+xmVU2QU3hfcaDJU3P2b8qIq5q55xhEbEofX4NGJY+jwBeycm3IKW1lZ6Xg7GZVb7Cg/HSiBjb2ctEREiKzp6fj7spzKzCFdpJ0ekhF6+n7gfSn4tT+kJg+5x8I1NaW+l5ORibWcWTCts6aQrQMiJiInB7TvrpaVTF/sCq1J1xL3CUpAHpwd1RKS0vd1OYWUUr5kofkn4PjCfrW15ANiriR8DNks4C5gOfTNnvBo4D5gJrgDMBImK5pIuAx1O+CyNi04eCm3EwNrPKV6RoHBGntHHoiFbyBnBOG+VcA1zTkWs7GJtZxfPr0GZmZaAaXod2MDazilcFsdjB2MwqnJddMjMrF5UfjR2MzayiiQ69Dl22HIzNrOK5m8LMrAx4aJuZWTmo/FjsYGxmla8KYrGDsZlVtnc5CVDZcDA2s4rnPmMzs3JQ+bHYwdjMKl8VxGIHYzOrfO4zNjMrMSFqqiAae9klM7My4JaxmVW8KmgYOxibWeXz0DYzs1LzSx9mZqVXzNWhS8nB2MwqXxVEY4+mMLOKpwL/K6gs6SVJT0uaKWl6ShsoaaqkF9OfA1K6JF0uaa6kWZL26ew9OBibWcVrmSyova0DDouIMRExNu1/C7gvIkYD96V9gGOB0Wk7G/hVZ+/BwdjMKp4K3N6FCcCk9HkScEJO+nWReQToL2l4Zy7gYGxmFU9SQRswWNL0nO3sVooL4M+SZuQcHxYRi9Ln14Bh6fMI4JWccxektA7zAzwzq2iiQ10QS3O6HtpycEQslDQUmCrpudyDERGSouM1za+qgvHMJ2cs7d+nbn6p69ENBgNLS10JK6ot5We6Y7ELfOKJGff2rtfgArO3+3ccEQvTn4sl3QbsB7wuaXhELErdEItT9oXA9jmnj0xpHVZVwTgihpS6Dt1B0vQCfrtbBfHPtPMi4philSVpK6AmIt5In48CLgSmABOBH6U/b0+nTAHOlTQZGAesyunO6JCqCsZmZu/SMOC21L9cB/wuIu6R9Dhws6SzgPnAJ1P+u4HjgLnAGuDMzl5YEUXv+rAu5lZU9fHP1DyaojJdVeoKWNH5Z7qFc8vYzKwMuGVsZlYGHIzNzMqAg3EZkdSUJidp2Ua1kW+UpGe6t3bWGTk/02ck3SGpfyfLOUPSFcWun5UPB+PysjZNTtKyvVTqCtm71vIz3RNYDpxT6gpZeXIwLmOS+kq6T9ITaUq/Ca3keY+kJyV9UNLOku5J79Q/LGm3UtTb2vR30rwFbf2sJH1E0qPpZ/oXScPylmhVwy99lJfekmamz/OAk4ATI2K1pMHAI5KmtGSWtCswGTgjIp6SdB/whYh4UdI44Erg8G6+B2uFpFrgCODqlHQVrf+spgH7p/kPPgf8G/C1UtTZupeDcXlZGxFjWnYk1QM/kHQo0EzWqmppKQ0heyXzYxHxrKS+wIHALXp71pSe3VZza0vLL9gRwByyiWfy/axGAjel+Q96kP1Sti2Ag3F5O5Us6O4bEQ2SXgJ6pWOrgJeBg4FnybqcVuYGcysLayNijKQ+wL1kfcbX0vbP6j+Bn0XEFEnjgQu6q6JWWu4zLm/bAItTID6Md854tQE4EThd0qcjYjUwT9JJsHE5mL26v8rWmohYA/wrWZfDGtr+WW3D27N+Tez2ilrJOBiXtxuBsZKeBk4HNp1X9S3geOArkj5K1pI+S9JTwGyyVQisTETEk8As4BTa/lldQNZ9MYMtY0pNS/w6tJlZGXDL2MysDDgYm5mVAQdjM7My4GBsZlYGHIzNzMqAg7EBm80udkt6SaGzZV0r6RPp839L2iNP3vGSDuzENV5Kr4gXlN5GGR2eCa0j5Zt1hIOxtcidXWwD8IXcg5I69bZmRHwuIp7Nk2U82avBZls0B2NrzcPAe1Or9eE0OdGzkmol/UTS45JmSfo8bHyD7ApJz0v6CzC0pSBJD0oamz4fk2ageyrNRjeKLOh/JbXKD5E0RNKt6RqPSzoonTtI0p8lzZb034AokKT9JP09zYT2v2mCpRbbpzq+KOm7Oed8RtJjqV6/ThP9mHUZz01h75BawMcC96SkfYA9I2KepLOBVRHxQUk9gb9J+jOwN7ArsAfZREbPAtdsUu4Q4DfAoamsgRGxXNJ/AW9GxE9Tvt8Bl0XENEk7kM3nsDvwXWBaRFwo6Z+AszpwW88Bh0REo6QPAz8APp6O7QfsSfaK8uOS7gLeAj4FHJReRb+S7I256zpwTbMOcTC2FrnTdz5MNtXjgcBjEdEyc9hRwAda+oPJ5lEYDRwK/D4imoBXJd3fSvn7Aw+1lBURy9uox4eBPXJmM9s6zXJ2KPCxdO5dklZ04N62ASZJGg0EUJ9zbGpELAOQ9EeyiZcagX3JgjNAb2BxB65n1mEOxtZi7aaziKVA9FZuEnBeRNy7Sb7jiliPGrL5fNe1UpfOugh4ICJOTF0jD+Yc23Q+gCC7z0kRcf67uahZR7jP2DriXuBf0jzLSNpF0lbAQ8CnUp/ycOCwVs59BDhU0k7p3IEp/Q2gX06+PwPntexIavkF8RDw6ZR2LDCgA/XOnQntjE2OHSlpoKTewAnA34D7gE9IGtpSV0k7YtaFHIytI/6brD/4CWULov6a7NvVbcCL6dh1ZMsLvUNELAHOBv6YZiq7KR26Azix5QEe2TSTY9MDwmd5e1TH98iC+Wyy7oqX89RzlqQFafsZ8GPgh5KeZPNvg48Bt5LNpnZrRExPoz++DfxZ0ixgKjC8wL8js07xrG1mZmXALWMzszLgYGxmVgYcjM3MyoCDsZlZGXAwNjMrAw7GZmZlwMHYzKwM/H/BR+97J2v5dwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light",
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#plot confusion matrix\n",
    "plot_confusion_matrix(dt_conf_matrix,[\"Fake\",\"Real\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "p6kibCd3uR1M"
   },
   "source": [
    "**6.2 Gradient Bossting Tree Classifier**\n",
    "\n",
    "Gradient Boosting classifiers consists of many weak classifiers that are combined together to make a strong model. The training of the models takes places sequentially and decision trees are mostly used as classfiers in Gradient boosting. Thus GBTs train one tree at a time and training is only parallelized at the single tree level.\n",
    "\n",
    "The main aim of GBT is to minimize the loss of the model. The model trains on the wieghted dataset where all the instances in the data are assigned same weight initially and the classification is done for the first tree. For the next tree the wieghts are misclassified instances are increased while the weights of correctly classified instances are decreased. This process is continued till the loss function has reached it minimum. The loss function for text classification would be how well our model can classify the fake news. \n",
    "\n",
    "A 3-Fold cross validation is performed to tune the parameters of the RF classifier that are defined using ParamGridBuilder.The parameters tuned are:\n",
    "\n",
    "  1. maxDepth: Maximum depth of the tree.\n",
    "  2. minInstancesPerNode: Minimum number of instances each child must have after split.\n",
    "  3. maxIter: Max number of iterations\n",
    "  4. subsamplingRate: Fraction of the training data used for learning each decision tree. It is set to 0.75 where 75% percent of train data is used for training and 25% for validation\n",
    "  5. stepSize: Step size (a.k.a. learning rate) in interval (0, 1] for shrinking the contribution of each estimator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 457
    },
    "colab_type": "code",
    "id": "CyA5pb_s85T9",
    "outputId": "a2c86fc1-46d6-4fd7-fe10-1d6de14a413e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cacheNodeIds: If false, the algorithm will pass trees to executors to match instances with nodes. If true, the algorithm will cache node IDs for each instance. Caching can speed up training of deeper trees. Users can set how often should the cache be checkpointed or disable it by setting checkpointInterval. (default: False)\n",
      "checkpointInterval: set checkpoint interval (>= 1) or disable checkpoint (-1). E.g. 10 means that the cache will get checkpointed every 10 iterations. Note: this setting will be ignored if the checkpoint directory is not set in the SparkContext. (default: 10)\n",
      "featureSubsetStrategy: The number of features to consider for splits at each tree node. Supported options: 'auto' (choose automatically for task: If numTrees == 1, set to 'all'. If numTrees > 1 (forest), set to 'sqrt' for classification and to 'onethird' for regression), 'all' (use all features), 'onethird' (use 1/3 of the features), 'sqrt' (use sqrt(number of features)), 'log2' (use log2(number of features)), 'n' (when n is in the range (0, 1.0], use n * number of features. When n is in the range (1, number of features), use n features). default = 'auto' (default: all)\n",
      "featuresCol: features column name. (default: features)\n",
      "impurity: Criterion used for information gain calculation (case-insensitive). Supported options: variance (default: variance)\n",
      "labelCol: label column name. (default: label)\n",
      "leafCol: Leaf indices column name. Predicted leaf index of each instance in each tree by preorder. (default: )\n",
      "lossType: Loss function which GBT tries to minimize (case-insensitive). Supported options: logistic (default: logistic)\n",
      "maxBins: Max number of bins for discretizing continuous features.  Must be >=2 and >= number of categories for any categorical feature. (default: 32)\n",
      "maxDepth: Maximum depth of the tree. (>= 0) E.g., depth 0 means 1 leaf node; depth 1 means 1 internal node + 2 leaf nodes. (default: 5)\n",
      "maxIter: max number of iterations (>= 0). (default: 20)\n",
      "maxMemoryInMB: Maximum memory in MB allocated to histogram aggregation. If too small, then 1 node will be split per iteration, and its aggregates may exceed this size. (default: 256)\n",
      "minInfoGain: Minimum information gain for a split to be considered at a tree node. (default: 0.0)\n",
      "minInstancesPerNode: Minimum number of instances each child must have after split. If a split causes the left or right child to have fewer than minInstancesPerNode, the split will be discarded as invalid. Should be >= 1. (default: 1)\n",
      "minWeightFractionPerNode: Minimum fraction of the weighted sample count that each child must have after split. If a split causes the fraction of the total weight in the left or right child to be less than minWeightFractionPerNode, the split will be discarded as invalid. Should be in interval [0.0, 0.5). (default: 0.0)\n",
      "predictionCol: prediction column name. (default: prediction)\n",
      "probabilityCol: Column name for predicted class conditional probabilities. Note: Not all models output well-calibrated probability estimates! These probabilities should be treated as confidences, not precise probabilities. (default: probability)\n",
      "rawPredictionCol: raw prediction (a.k.a. confidence) column name. (default: rawPrediction)\n",
      "seed: random seed. (default: -5345075898105589168)\n",
      "stepSize: Step size (a.k.a. learning rate) in interval (0, 1] for shrinking the contribution of each estimator. (default: 0.1)\n",
      "subsamplingRate: Fraction of the training data used for learning each decision tree, in range (0, 1]. (default: 1.0)\n",
      "thresholds: Thresholds in multi-class classification to adjust the probability of predicting each class. Array must have length equal to the number of classes, with values > 0, excepting that at most one value may be 0. The class with largest value p/t is predicted, where p is the original probability of that class and t is the class's threshold. (undefined)\n",
      "validationIndicatorCol: name of the column that indicates whether each row is for training or for validation. False indicates training; true indicates validation. (undefined)\n",
      "validationTol: Threshold for stopping early when fit with validation is used. If the error rate on the validation input changes by less than the validationTol, then learning will stop early (before `maxIter`). This parameter is ignored when fit without validation is used. (default: 0.01)\n",
      "weightCol: weight column name. If this is not set or empty, we treat all instance weights as 1.0. (undefined)\n"
     ]
    }
   ],
   "source": [
    "#check the parameters of GBT classifier\n",
    "gbt_classifier = GBTClassifier()\n",
    "print(gbt_classifier.explainParams())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "xFPmf5hf9mLe"
   },
   "outputs": [],
   "source": [
    "#define the GBT classifier\n",
    "gbt_classifier = GBTClassifier(labelCol = \"label\", featuresCol = \"features\")\n",
    "              \n",
    "#define the parameters for the GBT\n",
    "gbt_paramGrid = (ParamGridBuilder()\n",
    "             .addGrid(gbt_classifier.maxDepth, [4, 6])\n",
    "             .addGrid(gbt_classifier.minInstancesPerNode, [2, 4])\n",
    "             .addGrid(gbt_classifier.stepSize, [0.01])\n",
    "             #.addGrid(gbt.maxBins, [20, 60])\n",
    "             .addGrid(gbt_classifier.maxIter, [20,30])\n",
    "             .addGrid(gbt_classifier.subsamplingRate, [0.75])\n",
    "             .build())\n",
    "#evaluator for CV\n",
    "evaluator = BinaryClassificationEvaluator()\n",
    "\n",
    "#perform 3-fold CV\n",
    "gbt_cv = CrossValidator(estimator = gbt_classifier, estimatorParamMaps = gbt_paramGrid, evaluator = evaluator, numFolds = 3)\n",
    "gbt_cvModel = gbt_cv.fit(train)\n",
    "gbt_predictions = gbt_cvModel.transform(test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 67
    },
    "colab_type": "code",
    "id": "ApqWQPSQMZWD",
    "outputId": "5e7310a5-e77a-4c25-d042-4bfb5d30e264"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Max Depth: 6\n",
      "Min Instances Per Node: 4\n",
      "Number of Trees: 30\n"
     ]
    }
   ],
   "source": [
    "#dispaly the best parameters after CV\n",
    "gbt_best_parameters = gbt_cvModel.bestModel\n",
    "print(\"Max Depth: \" + str(gbt_best_parameters._java_obj.getMaxDepth()))\n",
    "print(\"Min Instances Per Node: \" + str(gbt_best_parameters._java_obj.getMinInstancesPerNode()))\n",
    "print(\"Number of Trees: \" + str(gbt_best_parameters._java_obj.getNumTrees()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "lj0Fo5dfNMhQ"
   },
   "outputs": [],
   "source": [
    "#compute the evaluation metrics using eval_metrics functions\n",
    "gbt_classifier = GBTClassifier(labelCol = \"label\", featuresCol = \"features\",maxDepth = 6, minInstancesPerNode = 4, maxIter = 30, subsamplingRate = 0.75)\n",
    "\n",
    "gbtModel = gbt_classifier.fit(train)\n",
    "gbt_predictions = gbtModel.transform(test)\n",
    "\n",
    "gbt_accuracy, gbt_auc, gbt_classification_report_df, gbt_conf_matrix = eval_metrics(gbt_predictions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 50
    },
    "colab_type": "code",
    "id": "hy1bQ77gNPuD",
    "outputId": "3fcb6a07-4aad-43ee-b6fa-465cb5e7ab6b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GBT accuracy is 90.06\n",
      "GBT AUC is 0.90\n"
     ]
    }
   ],
   "source": [
    "#print accuracy and AUC\n",
    "print ('GBT accuracy is {:0.2f}'.format(gbt_accuracy))\n",
    "print ('GBT AUC is {:0.2f}'.format(gbt_auc))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 134
    },
    "colab_type": "code",
    "id": "IQy3vTRhNXsO",
    "outputId": "4dfc5d9e-6fec-4fa4-a77e-b5af91468f9c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+----+------------------+------------------+------------------+\n",
      "|News|            recall|         precision|          f1_score|\n",
      "+----+------------------+------------------+------------------+\n",
      "|Real| 0.893296853625171|0.9011039558417663|0.8971834211128921|\n",
      "|Fake|0.9074671831289004|0.9001067235859125|0.9037719674239176|\n",
      "+----+------------------+------------------+------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#print classification report\n",
    "gbt_classification_report_df.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 67
    },
    "colab_type": "code",
    "id": "nn4TwYziNZ9M",
    "outputId": "1de14eb1-e944-4ce6-b7c7-f8a39be91152"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion Matrix\n",
      "[[4217.  430.]\n",
      " [ 468. 3918.]]\n"
     ]
    }
   ],
   "source": [
    "#print confusion matrix\n",
    "print(\"Confusion Matrix\")\n",
    "print(gbt_conf_matrix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 311
    },
    "colab_type": "code",
    "id": "8urvRaLrNmJ4",
    "outputId": "31b5c206-fbca-4236-cfec-39c01505bc9a"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWMAAAEmCAYAAAC3V/E+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3debxXVb3/8df7DIAIKAgYk2KKs+WAitrgcHNOtJtlmuJ0yVKbbNBuv2tp3m51yzKHrqWJQzlkJo5IJjftqiAOKKBySpFzIAFRQGU653x+f+x18Auc4Xvwe8534P30sR9899prr7U2X/ycddZee21FBGZmVlxVxW6AmZk5GJuZlQQHYzOzEuBgbGZWAhyMzcxKgIOxmVkJcDC2TpO0maR7JC2VdMf7KOcUSQ8Vsm3FIOkBSeOK3Q4rbw7GFUzSyZKekvS2pAUpaHykAEV/Gtga2CoiTtzYQiLilog4vADtWYekgyWFpLvWS/9wSp+SZznfk3RzR/ki4qiImLCRzTUDHIwrlqSvAz8H/pMscG4DXA2MLUDx2wIvR0RjAcrqKouAAyRtlZM2Dni5UBUo4/+HrDAiwluFbcAWwNvAie3k6UkWrOen7edAz3TsYKAeuABYCCwAzkjHvg+sBtakOs4CvgfcnFP2SCCAmrR/OvAPYDnwCnBKTvpjOecdCEwDlqY/D8w5NgW4FPhbKuchYGAb19bS/l8B56a0aqAB+A9gSk7eXwDzgGXAdOCjKf3I9a7zuZx2XJbasQLYIaWdnY5fA9yZU/6PgIcBFfvfhbfS3vxTvTIdAPQC7monz78DY4A9gQ8D+wHfzTn+AbKgPows4F4lqX9EXEzW274tIvpExHXtNUTS5sAVwFER0Zcs4D7bSr4BwH0p71bAz4D71uvZngycAQwGegDfaK9u4EbgtPT5COAFsh88uaaR/R0MAH4H3CGpV0Q8uN51fjjnnFOB8UBfYO565V0A7CHpdEkfJfu7GxcRXnfA2uVgXJm2AhZH+8MIpwCXRMTCiFhE1uM9Nef4mnR8TUTcT9Y73Gkj29MM7C5ps4hYEBEzW8lzDDAnIm6KiMaI+D3wIvDJnDy/jYiXI2IFcDtZEG1TRPwfMEDSTmRB+cZW8twcEW+kOn9K9htDR9d5Q0TMTOesWa+8d8n+Hn8G3AycHxH1HZRn5mBcod4ABkqqaSfPUNbt1c1NaWvLWC+Yvwv06WxDIuId4LPAOcACSfdJ2jmP9rS0aVjO/j83oj03AecBh9DKbwqSviFpdpoZ8hbZbwMDOyhzXnsHI+JJsmEZkf3QMOuQg3FlehxYBRzfTp75ZDfiWmzDhr/C5+sdoHfO/gdyD0bEpIj4BDCErLf76zza09Kmho1sU4ubgC8B96de61ppGOFbwGeA/hGxJdl4tVqa3kaZ7Q45SDqXrIc9P5Vv1iEH4woUEUvJblRdJel4Sb0l1Uo6StKPU7bfA9+VNEjSwJS/w2lcbXgW+JikbSRtAVzUckDS1pLGprHjVWTDHc2tlHE/sGOajlcj6bPArsC9G9kmACLiFeDjZGPk6+sLNJLNvKiR9B9Av5zjrwMjOzNjQtKOwA+Az5MNV3xLUrvDKWbgYFyx0vjn18luyi0i+9X6POBPKcsPgKeAGcDzwNMpbWPqmgzclsqazroBtCq1Yz6whCwwfrGVMt4AjiW7AfYGWY/y2IhYvDFtWq/sxyKitV7/JOBBsuluc4GVrDsE0fJAyxuSnu6onjQsdDPwo4h4LiLmAN8BbpLU8/1cg1U++SavmVnxuWdsZlYCHIzNzEqAg7GZWQlwMDYzKwHtPRRQdlSzWahH32I3wwpor122KXYTrIDmzn2VxYsXq+Oc+avut21E44q88saKRZMi4shC1l8olRWMe/Sl506fKXYzrID+9uSVxW6CFdBB+48ueJnRuCLv/+9XPntVR09XFk1FBWMz2xQJKmAlUwdjMytvAlTQkY+icDA2s/LnnrGZWbEJqqqL3Yj3zcHYzMqfhynMzIpMeJjCzKz45J6xmVlJcM/YzKwEVEDPuPx/nJjZJi499JHPlm+JUrWkZyTdm/a3k/SkpDpJt0nqkdJ7pv26dHxkThkXpfSXJB3RUZ0OxmZW3loe+shny99XgNk5+z8CLo+IHYA3gbNS+lnAmyn98pQPSbsCJwG7AUcCV0tqd/6dg7GZlb8C9owlDQeOAX6T9gUcCvwhZZnAey/7HZv2SccPS/nHArdGxKr0HsY6YL/26nUwNrMy16lhioGSnsrZxrdS4M/J3sHY8uLcrYC3IqIx7dcDw9LnYaT3JqbjS1P+temtnNMq38Azs/ImoDrvJ/AWR0SbS8dJOhZYGBHTJR1cgNblzcHYzMpf4WZTHAQcJ+looBfQD/gFsKWkmtT7HQ40pPwNwAigPr0dfAuyt5u3pLfIPadVHqYwszJXuNkUEXFRRAyPiJFkN+D+EhGnAI8An07ZxgF3p88T0z7p+F8iIlL6SWm2xXbAKGBqe3W7Z2xm5a/r5xl/G7hV0g+AZ4DrUvp1wE2S6oAlZAGciJgp6XZgFtAInBsRTe1V4GBsZuWvC57Ai4gpwJT0+R+0MhsiIlYCJ7Zx/mXAZfnW52BsZuWt83OIS5KDsZmVP69NYWZWAtwzNjMrNr+Q1MysNLhnbGZWZH7Th5lZKfALSc3MSoN7xmZmJcBjxmZmRSbPpjAzKw3uGZuZFZ8cjM3Miit7BZ6DsZlZcSltZc7B2MzKnNwzNjMrBQ7GZmYloKrKU9vMzIrLY8ZmZsUnjxmbmZUGB2MzsxJQCcG4/Ee9zWyTJymvLY9yekmaKuk5STMlfT+l3yDpFUnPpm3PlC5JV0iqkzRD0t45ZY2TNCdt4zqq2z1jMytvhb2Btwo4NCLellQLPCbpgXTsmxHxh/XyHwWMStv+wDXA/pIGABcDo4EApkuaGBFvtlWxe8ZmVvYK1TOOzNtptzZt0c4pY4Eb03lPAFtKGgIcAUyOiCUpAE8GjmyvbgdjMytrLbMpChGMASRVS3oWWEgWUJ9Mhy5LQxGXS+qZ0oYB83JOr09pbaW3ycHYzMpeJ4LxQElP5Wzj1y8rIpoiYk9gOLCfpN2Bi4CdgX2BAcC3C30NHjM2s/ImUFXeg8aLI2J0Phkj4i1JjwBHRsR/p+RVkn4LfCPtNwAjck4bntIagIPXS5/SXn3uGZtZ2SvgbIpBkrZMnzcDPgG8mMaBUVbI8cAL6ZSJwGlpVsUYYGlELAAmAYdL6i+pP3B4SmuTe8ZmVvYKOM94CDBBUjVZZ/X2iLhX0l8kDSKbt/EscE7Kfz9wNFAHvAucARARSyRdCkxL+S6JiCXtVexgbGZlrZCPQ0fEDGCvVtIPbSN/AOe2cex64Pp863YwNrPyV/4P4DkYm1mZU2U8Du1gbGZlz8HYzKwEOBibmZWC8o/FDsbFEtHM6pfvQLWb0+ODx7J67kPEu4tAVVT1HkzNiIORqmle+SZrXnuYWLGImiFjqBmc3ehtXvkma159b9pirF5GzQf2p2bwh9erJ2hseJTmZXOhqpbabQ6jqvegbr3WTVFTUxMH7T+aocOG8ce77+WcfzuLp6c/RUSww4478uvrbqBPnz6sWrWKs844jWeens6AAVtx8+9uY9uRIzco76FJD/KNr3+FpqYmTj/zbL75rQu7/6JKWCX0jLvsoQ9JTTnLzT0raWQb+UZKeqG1Y5WsadEM1LP/2v3q/jvSY+eT6bHTSURzE01vzAZA1T2pHf5RqgevO9umqld/eu58Ej13PokeO30Gqmqo3nK7DeppXj6XWLWUHrt8ntoRB7OmfkqXXpdlrrziF+y0yy5r93/808uZ+vRzTHtmBiNGbMM1V18JwA3XX0f/Lfsz88U6zv/K1/j372z4lG1TUxNf/fK53H3PAzwzYxZ33Pp7Zs+a1W3XUuryfeCj1AN2Vz6BtyIi9szZXu3CuspKrH6b5mWvUr3VrmvTqvuNXPsPpqr3YGJNtnCUantT1Xtr2vuqmpfXo55boB79Njy29BWqB+yUlbv5B6BpNbHmnYJfk72nvr6eBx+4jzPOPHttWr9+2XcTEaxcsWJtYLj3nrs55dRsqdtP/eunmfKXh8mmrr5n2tSpbL/9Dmz3wQ/So0cPTvzsSdx7z93ddDXloaqqKq+tlHVb6yT1kfSwpKclPS9pbCt5PijpGUn7Stpe0oOSpkt6VNLO3dXWrram4TFqhh5IawNdEU00vfkS1X23ybu85rfmUL3lqFaPxZp3UG2ftfuq3dzBuIt984KvctkPf7zB//zjzzqDkcM/wEsvvciXzj0fgPnzGxg+IlvaoKamhn5bbMEbb7yxznnz5zcwfPh7yx8MGzachoaGLr6KMqM8txLWlcF4s5whiruAlcAJEbE3cAjwU+X83iBpJ+BO4PSImAZcC5wfEfuQLcpxdWuVSBrfsgJTNK7owsspjKalr6KazajqPbjV443z/krV5kOp6jM0r/KiuYmmpa9SveUOhWymbaT777uXwYMGs/c++2xw7Nrrfss/XpvPzjvvwh9uv60IratcHqZoX+4wxQlkP5f+U9IM4M9ka3tunfIOAu4GTomI5yT1AQ4E7kjriv4P2TPjG4iIayNidESMVs1mXXg5hdH8zgKalr3Cypk3smbuJJqXN7B67mQAGv85lWhcQc2wj+Rf3vK5VPUehGp7t3o86wm/vXY/6ylv/v4uwtr0+P/9jXvvnchOO4zktFNOYsojf+GM0z6/9nh1dTUnfvYk/nTXnQAMHTqM+nnZsreNjY0sW7qUrbbaap0yhw4dRn39e0vjNjTUM2xYu0vjblrkYNxZp5AF3X3SWqGvA73SsaXAa0BLFKoC3lpvzHmXDUosQ7VDD6DXbqfTa7fTqN32CKr6DqPHtp+g8Y1ZNC2bR+3Iwzv1j6bpzTlUtTFEAVDVbzualrxERND8zj+huoeDcRe69LIf8vdX63mp7lVuvOVWDj7kUK6fcBN/r6sDsjHje++ZyI47ZaNuxxx7HLfcNAGAP975Bz5+yKEbfP+j992Xuro5vPrKK6xevZo7bruVY449rnsvrIQJkPLbSll3Tm3bAlgYEWskHQJsm3NsNXACMEnS2xHxO2Uv/zsxIu5IwxkfiojnurG93apx3hTUoy+rX85esVW95fbUfGBfYs07rHr5DmhaDYjGRc/Rc+eTUXUPomkNzcvnUTvi4HXLWpxNTqkZuDtV/baleflcVs++GapqqN3msG6+MosIzj5zHMuXLSMI9tjjw1xx1TUAnH7mWZx5+qnstvMO9O8/gJtuuRWA+fPn86UvnM2f7rmfmpoaLv/FlXzymCNoampi3OlnsutuuxXzkkpM6fd686H179wWrOAsqPbJ2R8I3AP0AZ4CxpC9zA/g3ojYPa0jOhm4FHie7OV+Q8jeQ3VrRFzSXp1VvQdHz50+U/BrseJ5c9qVxW6CFdBB+49m+vSnCho5e31gx9jmtCvyyjvnJ0dNz3dx+e7WZT3j3ECc9hcDB7SRffeU5y2y15q0aPcFfmZmUBkPffgJPDMrb2UwHpwPB2MzK2sCqvJ/B17JcjA2s7LnYGxmVmwepjAzK75snnH5R2MHYzMrc5Uxz9jB2MzKXgXE4m59HNrMrEsUam0KSb0kTZX0nKSZkr6f0reT9KSkOkm3SeqR0num/bp0fGROWRel9JckHdFR3Q7GZlbe8lyXIs/e8yrg0Ij4MLAncKSkMcCPgMsjYgfgTeCslP8s4M2UfnnKh6RdgZOA3cgeXrtaUnV7FTsYm1lZa7mBV4iecWRaljmsTVsAhwJ/SOkTgOPT57Fpn3T8sLSWzliyJRxWRcQrQB2wX3t1OxibWdnrRM94YMv652kbv2FZqk5L9y4kWyvn72SrSDamLPVkSwCT/pwHkI4vBbbKTW/lnFb5Bp6Zlb1OzKZY3NFCQRHRBOyZFi67C+iWtww5GJtZeVPXPIEXEW9JeoRsgbMtJdWk3u9woOW9Vw3ACKBeUg3ZUsFv5KS3yD2nVR6mMLOyVsjF5SUNSj1iJG0GfAKYDTwCfDplG0f2ZiKAiWmfdPwvka1LPBE4Kc222A4YBUxtr273jM2szBX0oY8hwIQ086EKuD0i7pU0C7hV0g+AZ4DrUv7rgJsk1QFLyGZQEBEzJd0OzAIagXPT8EebHIzNrOwVKhZHxAxgr1bS/0ErsyEiYiVwYhtlXQZclm/dDsZmVvb8OLSZWbF51TYzs+Lzqm1mZiXCwdjMrARUQCx2MDaz8ueesZlZsfkGnplZ8Qn5haRmZqWgqgK6xg7GZlb2KiAWOxibWXnLFgEq/2jsYGxmZa8ChozbDsaS9m7vxIh4uvDNMTPrvErvGf+0nWMt74QyMyu6CojFbQfjiDikOxtiZrYxRDa9rdx1+KYPSb0lfVfStWl/lKRju75pZmb5qVJ+WynL57VLvwVWAwem/QbgB13WIjOzzlD2po98tlKWTzDePiJ+DKwBiIh3oQJ+JzCziiCgukp5baUsn6ltq9OL+QJA0vbAqi5tlZlZJ5R4pzcv+QTji4EHgRGSbgEOAk7vykaZmXVGqQ9B5KPDYBwRkyU9DYwh+43gKxGxuMtbZmaWB21iq7Z9HPgI2VBFLXBXl7XIzKyTKmGhoHymtl0NnAM8D7wAfEHSVV3dMDOzfCnPrcNypBGSHpE0S9JMSV9J6d+T1CDp2bQdnXPORZLqJL0k6Yic9CNTWp2kCzuqO5+e8aHALhHRcgNvAjAzj/PMzLpFAceMG4ELIuJpSX2B6ZImp2OXR8R/r1fvrsBJwG7AUODPknZMh68CPgHUA9MkTYyIWW1VnE8wrgO2Aeam/REpzcys6EThHuiIiAXAgvR5uaTZwLB2ThkL3BoRq4BXJNUB+6VjdRHxDwBJt6a8bQbjNocpJN0jaSLQF5gtaYqkR4DZKc3MrPg699DHQElP5Wzj2y5WI4G9gCdT0nmSZki6XlL/lDYMmJdzWn1Kayu9Te31jP+7nWNmZiWjE6MUiyNidMflqQ9wJ/DViFgm6RrgUrJJDJeSLaR25sa1tnXtLRT0v4WsyMysK7Q8gVew8qRaskB8S0T8ESAiXs85/mvg3rTbQDZ022J4SqOd9FblM5tijKRpkt6WtFpSk6RlHZ1nZtZdCrU2hbJM1wGzI+JnOelDcrKdQDazDGAicJKknpK2A0YBU4FpwChJ20nqQXaTb2J7dedzA+/KVNAdwGjgNGDHds8wM+tGBZxlfBBwKvC8pGdT2neAz0nak2yY4lXgCwARMVPS7WQ35hqBcyOiCUDSecAkoBq4PiLanYWW10MfEVEnqTpV8ltJzwAXde4azcwKTyrcQx8R8Ritx/b72znnMuCyVtLvb++89eUTjN9N3exnJf2YbNpHPqu9mZl1iwp4AC+voHpqynce8A7ZoPSnurJRZmadUQnrGeezUFDLwx4rge8DSLoN+GwXtsvMLG8lHmfzku9CQes7oKCtMDPbSEIVsVDQxgZjM7PSUOlLaErau61DZMtolpw9d9mGxx7/ZbGbYQXU/6gfF7sJVkCr5vyzS8ot9fHgfLTXM/5pO8deLHRDzMw2ViVM72rvcehDurMhZmYbo9CPQxeLx4zNrOxVQCx2MDaz8pa9A6/8o7GDsZmVvUroGeezapskfV7Sf6T9bSTt19F5ZmbdpeUN0R1tpSyfm5BXkz3k8bm0v5zs3U5mZkWXvXZJeW2lLJ9hiv0jYu+0UhsR8WZaOMjMrCRU9NS2HGskVZOt44mkQUBzl7bKzKwTSrzTm5d8gvEVwF3AYEmXAZ8GvtulrTIzy5PKYAgiH/ms2naLpOnAYWTDM8dHxOwub5mZWZ4qIBZ3HIwlbQO8C9yTmxYRr3Vlw8zM8iGgpgLmtuUzTHEf2XixgF7AdsBLwG5d2C4zs7xtEj3jiNgjdz+t5valLmuRmVlnqDIe+uj0E3gR8bSk/buiMWZmG0OFfD90keQzZvz1nN0qYG9gfpe1yMysE7KHPordivcvn7nSfXO2nmRjyGO7slFmZp1Rpfy2jkgaIekRSbMkzZT0lZQ+QNJkSXPSn/1TuiRdIalO0ozcl3JIGpfyz5E0rqO62+0Zp4c9+kbENzq+DDOz4ijgqm2NwAVpOLYvMF3SZOB04OGI+C9JFwIXAt8GjgJGpW1/4Bpgf0kDgIuB0WQTIKZLmhgRb7ZVcZs9Y0k1EdEEHFSIKzQz6wotwxSF6BlHxIKIeDp9Xg7MBoaRjQZMSNkmAMenz2OBGyPzBLClpCHAEcDkiFiSAvBk4Mj26m6vZzyVbHz4WUkTgTuAd3Ia/ceOL83MrIt1bkW2gZKeytm/NiKubbVYaSSwF/AksHVELEiH/glsnT4PA+blnFaf0tpKb1M+syl6AW8Ah/LefOMAHIzNrCR04nHoxRExuqNMkvoAdwJfjYhlucMgERGSYqMa2o72gvHgNJPiBd4LwmvbU+iGmJltjOwdeAUsT6olC8S35IwAvC5pSEQsSMMQC1N6AzAi5/ThKa0BOHi99Cnt1dveJVQDfdLWN+dzy2ZmVgJEVZ5bhyVlXeDrgNkR8bOcQxOBlhkR44C7c9JPS7MqxgBL03DGJOBwSf3TzIvDU1qb2usZL4iISzpsvZlZEYmCPg59EHAq8LykZ1Pad4D/Am6XdBYwF/hMOnY/cDRQR7aGzxkAEbFE0qXAtJTvkohY0l7F7QXjCphGbWYVr4CPQ0fEY7Qd+w5rJX8A57ZR1vXA9fnW3V4w3qBiM7NSVNHrGXfUpTYzKwUFHqYomk4vFGRmVmoqumdsZlYuKiAWOxibWXkTm87boc3MSpcKulBQ0TgYm1nZK/9Q7GBsZmVOQLV7xmZmxVcBsdjB2MzKnTxmbGZWbJ5NYWZWItwzNjMrAeUfih2MzazceZ6xmVnxeczYzKxEuGdsZlYCyj8UOxibWZnzE3hmZiWiAmKxg7GZlTuhChiocDA2s7LnnrGZWZFlU9vKPxpXwvQ8M9uUKesZ57N1WJR0vaSFkl7ISfuepAZJz6bt6JxjF0mqk/SSpCNy0o9MaXWSLsznMhyMzazsFSoYAzcAR7aSfnlE7Jm2+7M6tStwErBbOudqSdWSqoGrgKOAXYHPpbzt8jCFmZW9Qt3Ai4i/ShqZZ/axwK0RsQp4RVIdsF86VhcR/wCQdGvKO6u9wtwzNrOyJqBK+W3AQElP5Wzj86zmPEkz0jBG/5Q2DJiXk6c+pbWV3i4HYzMre8rzP2BxRIzO2a7No/hrgO2BPYEFwE+74ho8TGFmZa+qC+e2RcTrLZ8l/Rq4N+02ACNysg5PabST3iYH4xLQ1NTERw7Yl6FDh3Hnn+4hIvj+xd/lrjv/QHV1NWePP4cvnfdlli5dylmnn8q8ea/R1NjIl792AaeNO2OD8p55ejrjzz6DlStWcMSRR/GTn/2iIhZSKVXRtIbV038FzU0QTVQN3oPa7Q+naUkdjXPug+Ym1G8Ytbt8GlVV0/zOQtbMuoNY1kDNDkdQs+3H15bVOPdRmuZPBYT6fIDaXU9E1bXr1tfcyJqZtxHLGqC2N7V7nEzVZgO6+apLR8swRZeVLw2JiAVp9wSgZabFROB3kn4GDAVGAVNTk0ZJ2o4sCJ8EnNxRPV0ajCU1Ac+nel4BTo2ItzainNOB0RFxXmFbWBqu+uUv2GnnXVi+bBkAN914A/X19Tzz/GyqqqpYuHAhANf+6ip23mUX/nDXRBYtWsRee+zMSZ87hR49eqxT3lfO/xJXXXMt++63PyccdwwPTXqQI448qtuva5NRVUOPvcejmp5EcxOrn7qG5q12ZM3M2+mx979Rtfkg1vz9IZoWTKdm2H6otje1Ox5H06KZ6xQTK5fSNO9v9DjgAlRdy+oZN9P0+nPUDB29Tr6mhmmoZjN6HPQtmv75LI11D9Bjj1O684pLTOGewJP0e+BgsrHleuBi4GBJewIBvAp8ASAiZkq6nezGXCNwbkQ0pXLOAyYB1cD1ETGTDnT1mPGKNBVkd2AJcG4X11d2GurrefCB+zn9jLPWpv3m2l9x0Xf+H1VV2dczePDg7IDE28uXExG88/bb9O8/gJqadX+eLliwgOXLlrHf/mOQxMmfP5V7J/6p265nUyQJ1fTMdiLrHaMqVFVN1eaDAKgeMIrmhVmHSj36ULXFCFD1BmVFNEPzGqK5CZrXoJ79NsjTvGgm1UP2AaBq8B40L6kjIrro6spAAecZR8TnImJIRNRGxPCIuC4iTo2IPSLiQxFxXE4vmYi4LCK2j4idIuKBnPT7I2LHdOyyfC6jO2/gPU66oyhpe0kPSpou6VFJO6f0T0p6UtIzkv4saetubF9RfOsbX+OyH/5obeAFeOUff+fOP9zGRw7Yl+M/eTR1c+YAcM4Xz+Oll15k+5HD2G+fD/GTn/58nfMAFsxvYOiw4Wv3hw0bzvz587vnYjZhEc2seuLnrPrrpVQNGIX6jSCimeZl9QA0LXyeWLm03TLUawtqtv0Yqx77IasevQxqelG91Y4b1rVqGeq1RXZOVTWq6QVr3i38RZUR5bmVsm4JxmkS9GFkYywA1wLnR8Q+wDeAq1P6Y8CYiNgLuBX4Vh5lj2+ZprJ48aLCN74LPXDfvQwaNIi99t5nnfRVq1bRs2cvHnt8GmecdTZf/ELWa/7z5Ens8aEP8/dXG3h86jN8/avnsywNbVhxSVX0HPNVen7kO8SyecQ7r9Nj95NZ8/I9rJr6S6ju2WHXLNa8S/OiWfQ86Nv0/Oi/Q9NqmhY83U1XUL6yMWPltZWyrr6Bt5mkZ8l6xLOByZL6AAcCd+TcVEq/4zEcuE3SEKAH2Thzu9LUlGsB9t5ndFn9rvb443/jvvvuYdKkB1i5ciXLly3jzNNPZdiw4Yw9/lMAHDf2BM75tzMBuGnCDVzwzW8jie132IFtt9uOl196kdH77re2zCFDhzG/oX7tfkNDPUOHDu3eC9uEqXYzqvpvT/MbL1Gz7cfpOfqLADS98TLxbvudheYldWiz/qhHHwCqB+1O89K5VA/Ze906evYjVi5FvbYkmpuIxpVQ27trLqhMlHaYzU+3jBkD25L9fZ2b6nwr59HCPSNil5T/l8CVEbEH2SB5ry5uX1Fd8sf7KmEAAAtJSURBVIMfMucf85j98itMuOn3fPzgQ7n+hps49rix/O//PgLAo3/9X3YYlf2qOmLECKY88jAAr7/+OnNefomR231wnTKHDBlC3379mPrkE0QEv7v5Jo755NjuvbBNTKx+m1izIvvctIamJXNQ78HE6reztOZGml6dQvXwMe2Wo15b0rz0NaJpNRFB05t1qPfgDfJVDdqVpgXTAWhe+DxV/bf3bJkKGKfolqltEfGupC8DfyIbknhF0okRcYeyf0UfiojngC14bz7euO5oWym64JsXcua4z3PlFT+nT58+XPWrXwNw4Xf+H+PPPoN99/4QEcGll/0XAwcOBGDMvnvxxLRnAPj5FVetndp2+BFHeiZFF4tVy1kz83agGSKo3vpDVA/ahTVz7qN50WwgqB4+huoBO6zNv2rqFdC4CiQaX3uMngdcQNUW21A1eA9WP3lFdgOw71Cqh+8PwJq/P0RVv+FUD9qV6qH7smbmbaz624+hdjNqd+9w1lTFq4T1jNWVd2ElvR0RfXL27wFuJxsbvgYYAtSSPd99iaSxwOXAm8BfgH0j4uB8p7btvc/oeOzxaV1zMVYUWx3zk2I3wQpo1ZNX0LysvqCRc5c99ooJd0/JK+/+2285PSJGd5yz+3Vpzzg3EKf9T+bsbrAyUkTcDdzdSvoNZKspmZltoPz7xX4Cz8zKnKAixswdjM2svOW/VnFJczA2s7JXAbHYwdjMKkAFRGMHYzMrc4VbKKiYHIzNrOx5zNjMrMjK4OG6vDgYm1n5q4Bo7GBsZmXPY8ZmZiXAY8ZmZsXmhz7MzEqDhynMzIosW5ui2K14/xyMzazsVUAs7tYXkpqZdY0CvelD0vWSFkp6ISdtgKTJkuakP/undEm6QlKdpBmS9s45Z1zKP0dSXi/KcDA2s7KnPP/Lww1suNb6hcDDETEKeDjtAxwFjErbeLIXZiBpAHAxsD+wH3BxSwBvj4OxmZU9Kb+tIxHxV2DJesljgQnp8wTg+Jz0GyPzBLBlepnyEcDkiFgSEW8Ck2nlZRrr85ixmZW9Lh4z3joiFqTP/wS2Tp+HAfNy8tWntLbS2+VgbGblL/9oPFDSUzn710bEtfmeHBEhqUteHOpgbGZlLbs3l3c0XrwRLyR9XdKQiFiQhiEWpvQGYEROvuEprQE4eL30KR1V4jFjMytvgqo8t400EWiZETGO916aPBE4Lc2qGAMsTcMZk4DDJfVPN+4OT2ntcs/YzMpfgQaNJf2erFc7UFI92ayI/wJul3QWMBf4TMp+P3A0UAe8C5wBEBFLJF0KTEv5LomI9W8KbsDB2MzKXOHe9BERn2vj0GGt5A3g3DbKuR64vjN1OxibWdnz49BmZkXmN32YmZWKCojGDsZmVva8hKaZWQnwmLGZWQmogFjsYGxmZc6vXTIzKxXlH40djM2srIn39ahzyXAwNrOy52EKM7MS4KltZmaloPxjsYOxmZW/CojFDsZmVt7yfb9dqXMwNrOy5zFjM7NSUP6x2MHYzMpfBcRiB2MzK38eMzYzKzIhqiogGvvt0GZmJcA9YzMrexXQMXYwNrPyVwlT2zxMYWblTe89+NHRlldx0quSnpf0rKSnUtoASZMlzUl/9k/pknSFpDpJMyTtvbGX4WBsZmVNndg64ZCI2DMiRqf9C4GHI2IU8HDaBzgKGJW28cA1G3sdDsZmVv66IBqvZywwIX2eAByfk35jZJ4AtpQ0ZGMqcDA2s7KnPP8DBkp6Kmcb30pxATwkaXrO8a0jYkH6/E9g6/R5GDAv59z6lNZpvoFnZmWvE7MpFucMPbTlIxHRIGkwMFnSi7kHIyIkxUY0s13uGZtZ2SvkKEVENKQ/FwJ3AfsBr7cMP6Q/F6bsDcCInNOHp7ROczA2s7InKa8tj3I2l9S35TNwOPACMBEYl7KNA+5OnycCp6VZFWOApTnDGZ3iYQozK2uioA99bA3clQJ3DfC7iHhQ0jTgdklnAXOBz6T89wNHA3XAu8AZG1uxIgo+9FE0khaR/UVVuoHA4mI3wgpqU/lOt42IQYUsUNKDZH9/+VgcEUcWsv5CqahgvKmQ9FQeNyGsjPg7NY8Zm5mVAAdjM7MS4GBcnq4tdgOs4PydbuI8ZmxmVgLcMzYzKwEOxmZmJcDBuIRIakprqLZsI9vIN1LSC93bOtsYOd/pC5LukbTlRpZzuqQrC90+Kx0OxqVlRVpDtWV7tdgNsvet5TvdHVgCnFvsBllpcjAuYZL6SHpY0tPpzQNjW8nzQUnPSNpX0vaSHkxL/z0qaeditNva9DhpecW2vitJn5T0ZPpO/yxp63ZLtIrhtSlKy2aSnk2fXwFOBE6IiGWSBgJPSJrYklnSTsCtwOkR8Zykh4FzImKOpP2Bq4FDu/karBWSqoHDgOtS0rW0/l09BoxJyzSeDXwLuKAYbbbu5WBcWlZExJ4tO5Jqgf+U9DGgmaxX1dJTGkS2ctSnImKWpD7AgcAdOatT9ey2lltbWn7ADgNmk62P2953NRy4LS3T2IPsh7JtAhyMS9spZEF3n4hYI+lVoFc6thR4DfgIMItsyOmt3GBuJWFFROwpqTcwiWzM+Aba/q5+CfwsIiZKOhj4Xnc11IrLY8albQtgYQrEhwDb5hxbDZxAtpbqyRGxDHhF0omw9q21H+7+JltrIuJd4MtkQw7v0vZ3tQXvLU4+boOCrGI5GJe2W4DRkp4HTgPWf/3LO8CxwNckHUfWkz5L0nPATLKXJVqJiIhngBnA52j7u/oe2fDFdDaNJTUt8ePQZmYlwD1jM7MS4GBsZlYCHIzNzEqAg7GZWQlwMDYzKwEOxgZssLrYHekhhY0t6wZJn06ffyNp13byHizpwI2o49X0iHhe6W2U0emV0DpTvllnOBhbi9zVxVYD5+QelLRRT2tGxNkRMaudLAeTPRpstklzMLbWPArskHqtj6bFiWZJqpb0E0nTJM2Q9AVY+wTZlZJekvRnYHBLQZKmSBqdPh+ZVqB7Lq1GN5Is6H8t9co/KmmQpDtTHdMkHZTO3UrSQ5JmSvoNIPIkaT9Jj6eV0P4vLbDUYkRq4xxJF+ec83lJU1O7/ict9GPWZbw2ha0j9YCPAh5MSXsDu0fEK5LGA0sjYl9JPYG/SXoI2AvYCdiVbCGjWcD165U7CPg18LFU1oCIWCLpV8DbEfHfKd/vgMsj4jFJ25Ct57ALcDHwWERcIukY4KxOXNaLwEcjolHSvwD/CfxrOrYfsDvZI8rTJN0HvAN8FjgoPYp+NdkTczd2ok6zTnEwtha5y3c+SrbU44HA1IhoWTnscOBDLePBZOsojAI+Bvw+IpqA+ZL+0kr5Y4C/tpQVEUvaaMe/ALvmrGbWL61y9jHgU+nc+yS92Ylr2wKYIGkUEEBtzrHJEfEGgKQ/ki281AjsQxacATYDFnaiPrNOczC2FivWX0UsBaJ3cpOA8yNi0nr5ji5gO6rI1vNd2UpbNtalwCMRcUIaGpmSc2z99QCC7DonRMRF76dSs87wmLF1xiTgi2mdZSTtKGlz4K/AZ9OY8hDgkFbOfQL4mKTt0rkDUvpyoG9OvoeA81t2JLX8gPgrcHJKOwro34l2566Edvp6xz4haYCkzYDjgb8BDwOfljS4pa2StsWsCzkYW2f8hmw8+GllL0T9H7Lfru4C5qRjN5K9XmgdEbEIGA/8Ma1Udls6dA9wQssNPLJlJkenG4SzeG9Wx/fJgvlMsuGK19pp5wxJ9Wn7GfBj4IeSnmHD3wanAneSraZ2Z0Q8lWZ/fBd4SNIMYDIwJM+/I7ON4lXbzMxKgHvGZmYlwMHYzKwEOBibmZUAB2MzsxLgYGxmVgIcjM3MSoCDsZlZCfj/PFNg+0S0/dEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light",
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#plot confusion matrix\n",
    "plot_confusion_matrix(gbt_conf_matrix, [\"Fake\",\"Real\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "2B50SaEfpONS"
   },
   "source": [
    "**6.3 Neural Network using Keras and Elephas**\n",
    "\n",
    "Elephas is an extension of Keras, which allows you to run distributed deep learning models at scale with Spark. Elephas implements a class of data-parallel algorithms on top of Keras, using Spark's RDDs and data frames. Keras Models are initialized on the driver, then serialized and shipped to workers, alongside with data and broadcasted model parameters. Spark workers deserialize the model, train their chunk of data and send their gradients back to the driver. The \"master\" model on the driver is updated by an optimizer, which takes gradients either synchronously or asynchronously. The working of elephas is as follows:\n",
    "![alt text](https://raw.githubusercontent.com/maxpumperla/elephas/master/elephas.gif) \n",
    "\n",
    "The neural network model is defined using Keras layers and model is trained using the parameter defined in the elephas sparkML estimator. The neural network model consists:\n",
    "  1. Of 1 input layer, 3 hidden layers and 1 output layer\n",
    "  2. All layers are Dense layers with activation functions for input and hidden layers are ReLU and Softmax for outplut layer\n",
    "  3. A dropout layer of 10% is used after each layer \n",
    "  4. Binary Cross entropy loss function and Adam optimizer with 0.01 learning rate and the metrics used is \"accuracy\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 675
    },
    "colab_type": "code",
    "id": "XaWGLxPWibOB",
    "outputId": "042022d5-d10a-4937-fdca-4a8196eb6d9c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/nn_impl.py:180: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.where in 2.0, which has the same broadcast rule as np.where\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_6 (Dense)              (None, 1024)              52224     \n",
      "_________________________________________________________________\n",
      "activation_6 (Activation)    (None, 1024)              0         \n",
      "_________________________________________________________________\n",
      "dropout_5 (Dropout)          (None, 1024)              0         \n",
      "_________________________________________________________________\n",
      "dense_7 (Dense)              (None, 512)               524800    \n",
      "_________________________________________________________________\n",
      "activation_7 (Activation)    (None, 512)               0         \n",
      "_________________________________________________________________\n",
      "dropout_6 (Dropout)          (None, 512)               0         \n",
      "_________________________________________________________________\n",
      "dense_8 (Dense)              (None, 256)               131328    \n",
      "_________________________________________________________________\n",
      "activation_8 (Activation)    (None, 256)               0         \n",
      "_________________________________________________________________\n",
      "dropout_7 (Dropout)          (None, 256)               0         \n",
      "_________________________________________________________________\n",
      "dense_9 (Dense)              (None, 128)               32896     \n",
      "_________________________________________________________________\n",
      "activation_9 (Activation)    (None, 128)               0         \n",
      "_________________________________________________________________\n",
      "dropout_8 (Dropout)          (None, 128)               0         \n",
      "_________________________________________________________________\n",
      "dense_10 (Dense)             (None, 2)                 258       \n",
      "_________________________________________________________________\n",
      "activation_10 (Activation)   (None, 2)                 0         \n",
      "=================================================================\n",
      "Total params: 741,506\n",
      "Trainable params: 741,506\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "#count the number of target labels\n",
    "tar_class = train.select(\"label\").distinct().count()\n",
    "\n",
    "#dimension of input data\n",
    "input_dim = len(train.select(\"features\").first()[0])\n",
    "\n",
    "#define the neural network model\n",
    "model = Sequential()\n",
    "model.add(Dense(1024, input_dim = input_dim, activation = \"relu\"))\n",
    "model.add(Dropout(0.1))\n",
    "model.add(Dense(512, activation = \"relu\"))\n",
    "model.add(Dropout(0.1))\n",
    "model.add(Dense(256, activation = \"relu\"))\n",
    "model.add(Dropout(0.1))\n",
    "model.add(Dense(128, activation = \"relu\"))\n",
    "model.add(Dropout(0.1))\n",
    "model.add(Dense(tar_class, activation = \"softmax\"))\n",
    "\n",
    "model.compile(optimizer = \"adam\",\n",
    "              loss = 'binary_crossentropy',\n",
    "               metrics=['accuracy'])\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "sqA_OopuEMq2"
   },
   "source": [
    "The parameters of the estimator are:\n",
    "\n",
    "  1. set_keras_model_config(model.to_yaml()): Set the neural network model\n",
    "  2. set_categorical_labels(True): Set to True if labels are categorical\n",
    "  3. set_nb_classes(tar_class): Set the number of target classes\n",
    "  4. set_num_workers(1): Set the number of worker nodes\n",
    "  5. set_epochs(10): Set the number of iterations for the training\n",
    "  6. set_batch_size(64): Set the batch size \n",
    "  7. set_verbosity(1): Set the verbosity for the results display\n",
    "  8. set_validation_split(0.10): Set the validation data split from the training data\n",
    "  9. set_optimizer_config(opt_conf): Set the defined optimizer\n",
    "  10. set_mode(\"synchronous\"): Set the mode to synchronous\n",
    "  11. set_loss(\"binary_crossentropy\"): Set the loss for the model same as keras model.compile loss\n",
    "  12. set_metrics([\"acc\"]): Set the metrics for the model evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "yxYHzmoDBcqZ",
    "outputId": "07e49fd6-d2cf-409f-c48a-8bae9388150b"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ElephasEstimator_6ee8884a0ad4"
      ]
     },
     "execution_count": 53,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#define the adam optimizer with lr = 0.01\n",
    "optimizer_conf = optimizers.Adam(lr = 0.01)\n",
    "opt_conf = optimizers.serialize(optimizer_conf)\n",
    "\n",
    "#define elephas estimator\n",
    "estimator = ElephasEstimator()\n",
    "\n",
    "#set the parameters for the estimator\n",
    "estimator.set_keras_model_config(model.to_yaml())\n",
    "estimator.set_categorical_labels(True)\n",
    "estimator.set_nb_classes(tar_class)\n",
    "estimator.set_num_workers(1)\n",
    "estimator.set_epochs(10)\n",
    "estimator.set_batch_size(64)\n",
    "estimator.set_verbosity(1)\n",
    "estimator.set_validation_split(0.10)\n",
    "estimator.set_optimizer_config(opt_conf)\n",
    "estimator.set_mode(\"synchronous\")\n",
    "estimator.set_loss(\"binary_crossentropy\")\n",
    "estimator.set_metrics([\"acc\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 50
    },
    "colab_type": "code",
    "id": "GqKbWspHB1PM",
    "outputId": "140b231d-7c46-4f53-e78c-220a541737c5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ">>> Fit model\n",
      ">>> Synchronous training complete.\n"
     ]
    }
   ],
   "source": [
    "#train the model using the estimator\n",
    "fitted_pipeline = estimator.fit(train)\n",
    "\n",
    "#Predict the test data\n",
    "nn_prediction = fitted_pipeline.transform(test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "z-96mlWvGS_u"
   },
   "outputs": [],
   "source": [
    "#compute the evaluation metrics \n",
    "nn_accuracy, nn_auc, nn_classification_report, nn_conf_matrix = eval_metrics(nn_prediction)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 50
    },
    "colab_type": "code",
    "id": "7d6JDw7F6AT4",
    "outputId": "cad1bd07-ef0b-4bf5-fcfb-592b36695ef3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Neural Network accuracy is 91.41\n",
      "Neural Network AUC is 0.91\n"
     ]
    }
   ],
   "source": [
    "#print accuracy and AUC\n",
    "print ('Neural Network accuracy is {:0.2f}'.format(nn_accuracy))\n",
    "print ('Neural Network AUC is {:0.2f}'.format(nn_auc))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 134
    },
    "colab_type": "code",
    "id": "YtahDi90H8kN",
    "outputId": "6abfae1a-563c-4a1a-9ad6-47f2bcc9beed"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+----+------------------+------------------+------------------+\n",
      "|News|            recall|         precision|          f1_score|\n",
      "+----+------------------+------------------+------------------+\n",
      "|Real|0.9154126766985864|0.9083710407239819|0.9118782648194413|\n",
      "|Fake|0.9128469980632666|0.9195751138088012|0.9161987041036717|\n",
      "+----+------------------+------------------+------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#print classification report\n",
    "nn_classification_report.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 67
    },
    "colab_type": "code",
    "id": "Y827eamxIKLB",
    "outputId": "ed26db3b-9609-43a9-a563-d44ae0014987"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion Matrix\n",
      "[[4242.  405.]\n",
      " [ 371. 4015.]]\n"
     ]
    }
   ],
   "source": [
    "#print confusion matrix\n",
    "print(\"Confusion Matrix\")\n",
    "print(nn_conf_matrix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 311
    },
    "colab_type": "code",
    "id": "D4loc78JIQCw",
    "outputId": "611b5aca-afda-49ca-9335-5fdedb64cd9a"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWMAAAEmCAYAAAC3V/E+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deZxdRZ338c+3lyRAQjYCZgECGCEsY4BAAiqyb8IEGEAEITj4MI7guDFu4zMoiDPOM4IDAg4KEhDZRCTsRBYVB7ISlixAD2HJAtlIAiTp9PJ7/jjV4ZJ0374dbucu+b55nVffU6dOnTrp8EvdOnWqFBGYmVlp1ZS6AmZm5mBsZlYWHIzNzMqAg7GZWRlwMDYzKwMOxmZmZcDB2LpM0laS7pW0UtKdH6KcsyQ9Usy6lYKkByWNL3U9rLI5GFcxSWdKmibpXUmLUtD4ZBGKPhXYARgYEadtaiERcUtEHF2E+nyApEMlhaS7N0j/eEp/osByfiDpN53li4jjImLCJlbXDHAwrlqSvgH8DPgxWeDcCbgGGFeE4ncGXoqI5iKU1V2WAAdJGpiTNh54qVgXUMb/D1lxRIS3KtuAvsC7wGl58vQkC9YL0/YzoGc6digwH/gmsBhYBHwhHfshsA5oStc4D/gB8JucsocDAdSl/XOBV4B3gHnAWTnpT+acdzAwFViZfh6cc+wJ4FLgr6mcR4DtOri3tvr/ArggpdUCC4B/BZ7IyftfwBvAKmA68KmUfuwG9/lsTj0uS/VYA3w0pX0xHb8WuCun/J8AjwIq9d8Lb+W9+V/16nQQ0Au4O0+efwHGAqOAjwMHAt/POf4RsqA+lCzgXi2pf0RcTNbavj0iekfE9fkqImkb4ErguIjoQxZwZ7aTbwBwf8o7ELgcuH+Dlu2ZwBeA7YEewEX5rg3cBJyTPh8DvED2D0+uqWR/BgOA3wJ3SuoVEQ9tcJ8fzznnbOB8oA/w2gblfRPYR9K5kj5F9mc3PiI874Dl5WBcnQYCSyN/N8JZwCURsTgilpC1eM/OOd6UjjdFxANkrcPdN7E+rcDekraKiEURMaudPJ8BXo6ImyOiOSJuBeYCJ+bk+XVEvBQRa4A7yIJohyLif4ABknYnC8o3tZPnNxGxLF3zp2TfGDq7zxsjYlY6p2mD8laT/TleDvwG+EpEzO+kPDMH4yq1DNhOUl2ePEP4YKvutZS2vowNgvlqoHdXKxIR7wGfBb4ELJJ0v6Q9CqhPW52G5uy/uQn1uRm4EDiMdr4pSLpI0pw0MmQF2beB7Top8418ByNiMlm3jMj+0TDrlINxdXoKaAROypNnIdmDuDY7sfFX+EK9B2yds/+R3IMR8XBEHAUMJmvt/rKA+rTVacEm1qnNzcCXgQdSq3W91I3wLeB0oH9E9CPrr1Zb1TsoM2+Xg6QLyFrYC1P5Zp1yMK5CEbGS7EHV1ZJOkrS1pHpJx0n6j5TtVuD7kgZJ2i7l73QYVwdmAodI2klSX+C7bQck7SBpXOo7biTr7mhtp4wHgI+l4Xh1kj4L7Anct4l1AiAi5gGfJusj31AfoJls5EWdpH8Fts05/hYwvCsjJiR9DPgR8Hmy7opvScrbnWIGDsZVK/V/foPsodwSsq/WFwJ/SFl+BEwDngOeB2aktE251iTg9lTWdD4YQGtSPRYCy8kC4z+2U8Yy4ASyB2DLyFqUJ0TE0k2p0wZlPxkR7bX6HwYeIhvu9hqwlg92QbS90LJM0ozOrpO6hX4D/CQino2Il4HvATdL6vlh7sGqn/yQ18ys9NwyNjMrAw7GZmZlwMHYzKwMOBibmZWBfC8FVBzVbRXq0afU1bAiGjVyp1JXwYro9ddeZenSpeo8Z+Fqt905onlNQXljzZKHI+LYYl6/WKorGPfoQ8/dTy91NayInnzqqlJXwYrokwcdUPQyo3lNwf/fr515dWdvV5ZMVQVjM9sSCapgJlMHYzOrbAJU1J6PknAwNrPK55axmVmpCWpqS12JD83B2Mwqn7spzMxKTLibwsys9OSWsZlZWXDL2MysDLhlbGZWatXx0kfl34GZbdnaXvooZCu0SKlW0jOS7kv7u0iaLKlB0u2SeqT0nmm/IR0fnlPGd1P6i5KO6eyaDsZmVvlUU9hWuK8Cc3L2fwJcEREfBd4Gzkvp5wFvp/QrUj4k7QmcAewFHAtcIynvYGgHYzOrcCpqMJY0DPgM8Ku0L+Bw4HcpywTeX3l9XNonHT8i5R8H3BYRjWlR3AbgwHzXdZ+xmVU2AbUFv4G3naRpOfvXRcR1G+T5GdmCuG3z8Q4EVkREc9qfDwxNn4eSFrGNiGZJK1P+ocDTOWXmntMuB2Mzq3yF9wcvjYjRHRejE4DFETFd0qHFqFqhHIzNrMIVdTTFJ4C/lXQ80AvYFvgvoJ+kutQ6HgYsSPkXADsC8yXVAX2BZTnpbXLPaZf7jM2s8hVpNEVEfDcihkXEcLIHcI9FxFnA48CpKdt44J70eWLaJx1/LCIipZ+RRlvsAowApuS7tlvGZlb5un+c8beB2yT9CHgGuD6lXw/cLKkBWE4WwImIWZLuAGYDzcAFEdGS7wIOxmZW2bo4hrhQEfEE8ET6/ArtjIaIiLXAaR2cfxlwWaHXczA2s8pXBW/gORibWeXz3BRmZqVWHXNTOBibWeVzy9jMrMS80oeZWTnwgqRmZuXBLWMzszLgPmMzsxKTR1OYmZUHt4zNzEpPDsZmZqWVLYHnYGxmVlpKW4VzMDazCie3jM3MyoGDsZlZGaip8dA2M7PScp+xmVnpyX3GZmblwcHYzKwMOBibmZWBagjGlf8I0sy2bOrC1llRUi9JUyQ9K2mWpB+m9BslzZM0M22jUrokXSmpQdJzkvbLKWu8pJfTNr6za7tlbGYVr4gt40bg8Ih4V1I98KSkB9Oxf46I322Q/zhgRNrGANcCYyQNAC4GRgMBTJc0MSLe7ujCbhmbWUVrG01RyNaZyLybduvTFnlOGQfclM57GugnaTBwDDApIpanADwJODbftR2MzazidSEYbydpWs52fjtl1UqaCSwmC6iT06HLUlfEFZJ6prShwBs5p89PaR2ld8jdFGZW2QSqKbibYmlEjM6XISJagFGS+gF3S9ob+C7wJtADuA74NnDJpld6Y24Zm1nFK1Y3Ra6IWAE8DhwbEYtSV0Qj8GvgwJRtAbBjzmnDUlpH6R1yMDazilesYCxpUGoRI2kr4ChgbuoHRlkhJwEvpFMmAuekURVjgZURsQh4GDhaUn9J/YGjU1qH3E1hZhWtyK9DDwYmSKola6zeERH3SXpM0iCyAXIzgS+l/A8AxwMNwGrgCwARsVzSpcDUlO+SiFie78IOxmZW+YoUiyPiOWDfdtIP7yB/ABd0cOwG4IZCr+1gbGaVTdXxBp6DsZlVPAdjM7My4GBsZlYOKj8We2hbqUS00vji7ax75T4A1r32CI1zbqFx7q00vf4o2bjz97Wufou1M6+hZUVD2l9C40u/o3Hub2mcexstb7/c/nVaW1j36sM0zr6ZxpfupLVxVffemAHQ0tLCQQfux9+ddCIAr86bx6c/OZZ9Ro7gnLPOYN26dQDcfNON7Dx0e8YesC9jD9iXG2/4VbvlPTNjOgfs9zfsM3IEF339n8ieG1mb7hhnvLl1WzCW1JIzw9FMScM7yDdc0gvtHatmLUueQz37r9+v7f8xeuxxJj12P4NobaFl2Zz1xyJaaV74FDV9csaQ19RTv/OR9NzjTHrsdiJNC54kmhs3vs7y2ai2Jz33PJu6QaNoXvRUt96XZa6+6r/YfY+R6/f/7798hwv/6Ws8P+dl+vXrx4RfX7/+2N+dejpPT32Gp6c+w7l//8V2y/vqV77M1ddex3OzX6KhoYFHHn6o2++hUhQaiLfYYAysiYhROdur3XitihLr3qV11avUDtxzfVrttsPX/4Wp2Xp7ound9cdaljxPTd/doG7r9Wk1vfpR07MfAKrfBtVtRbSs2eharSvnUTtgj+ycfrvR+s58t6q62YL583nowQc49wvnARAR/OmJxzj5lFMBOOvs8dw78Z6Cy1u0aBHvrFrFgWPGIokzP3829038Q7fUvVLV1NQUtJWzzVY7Sb0lPSpphqTnJY1rJ8+ukp6RdICk3SQ9JGm6pL9I2mNz1bW7NS14krohB9NeR1dECy1vv0htn52y/XXv0rLyFWq327vD8lrfewuiFfXou3F5Te+h+t4ASDWotge0rC3OjVi7vnXR17ns336y/n/+ZcuW0bdvP+rqskc0Q4cOY+HC99+M/cMffs+B+3+cs844jflvvLFReYsWLmDI0GHr97PzF3bzXVSYIs1nXErdGYy3yumiuBtYC5wcEfsBhwE/Vc73Bkm7A3cB50bEVLLJOL4SEfsDFwHXtHcRSee3zcAUzRu3DMtNy8pXUd1W1Gy9fbvHm9/4MzXbDKGm9xAgC9z1Qw7q8CtWNL1H0+t/pH6nw8v+a9iW4MH772PQoEHsu9/+BeU//jMnMueleUyZ/iyHH3Ek/+eL53ZvBatUNXRTdOdoijURMaptR9lEzT+WdAjQSjad3A7p8CDgHuCUiJgtqTdwMHBnzh9gT9oREdeRBW5qtt6+7L9/t763iJZV82iZ9RpEM7Q0se61SfTY+Sia35xCNK+hfpfj1uePNYtZ9+oj2U7LGlrfeQ2oobbfrkTLOta9ch91g8dQs81H2r2e6rchmt5FPXoT0Uq0rIPaXpvhTrdMTz31V+6//14efvhB1q5dyzurVvHP3/waK1euoLm5mbq6OhYsmM+QIdlsigMHDlx/7rl//0W+/71vb1Tm4CFDWbhg/vr97Pwh3X8zlaJKXvrYnJ0oZ5EF3f1TkH4LaIsKK4HXgU/m1GvFBn3OIzcqsQLVDzmIXnudS6+9zqF+52Oo6TM0C8TLZtOy6g3qhx/9gb9YPfc8h157ZVtN349SP+yQLBC3ttA07wFq++9Bbb+Pdni9mm13oWX5XABaV/wvNX2GVsVf3HJ1yY/+jZdfeYM5L81jws238ulDD+fXE37DIZ8+jLt/ny0SccvNEzjhxL8Fsv7gNvffN/EDD/3aDB48mD7bbsuUyU8TEfz2NzfzmRM36uXbYgmQCtvK2eYMxn2BxRHRJOkwYOecY+uAk8lmPzozIlYB8ySdBuvXmfr4ZqzrZtf8xhPQvJp1L/2Oxrm30fzm1Lz5W1c00PruIlqWz6Fx7m00zr2N1tVLAGhaNJmWlfMAqB04kmheS+Psm2leMpO6wQd1961YOy697N+56r+uYJ+RI1i+fDnj08O9a6++ktGj9mbM6FFce/VV/Pcvf73+nLEHvD9Fws+uvJovf+n/sM/IEey6664cc+xxG11jy1UdoynUXU/WJb0bEb1z9rcD7gV6A9OAsWTrRwHcFxF7p6nrJgGXAs+TrSc1mGzpk9siIu9kzjVbbx89dz+96PdipbNs8lWlroIV0ScPOoAZ06cVNSr2+sjHYqdzriwo78v/77jpnU0uXyrd1mecG4jT/lKgo2bZ3inPCuCAnPS8a0aZmUF19Bn7dWgzq2wV0B9cCAdjM6toAmoKXwOvbDkYm1nFczA2Mys1d1OYmZVeNs648qOxg7GZVbjyH0NcCAdjM6t4VRCLPbm8mVW+Yr2BJ6mXpCmSnpU0S9IPU/oukiZLapB0u6QeKb1n2m9Ix4fnlPXdlP6ipGM6u7aDsZlVtgLnpSiw9dwIHB4RHwdGAcdKGgv8BLgiIj4KvA2cl/KfB7yd0q9I+ZC0J3AGsBfZy2vXSKrNd2EHYzOraG0P8IrRMo5M28oO9WkL4HDgdyl9AnBS+jwu7ZOOH5GmBh5HNoVDY0TMAxqAA/Nd28HYzCpeF1rG27XNf5628zcuS7WSZgKLyebK+V+yWSSbU5b5ZFMAk36+AZCOrwQG5qa3c067/ADPzCpeF0ZTLO1soqDIVgMelSYuuxvYLKsMORibWWVT97yBFxErJD1ONsFZP0l1qfU7DGhbN2sBsCMwX1Id2VTBy3LS2+Se0y53U5hZRSvm5PKSBqUWMZK2Ao4C5gCPA6embOPJViYCmJj2Sccfi2xe4onAGWm0xS7ACGBKvmu7ZWxmFa6oL30MBiakkQ81wB0RcZ+k2cBtkn4EPANcn/JfD9wsqQFYTjaCgoiYJekOYDbQDFyQuj865GBsZhWvWLE4Ip4D9m0n/RXaGQ0REWuB0zoo6zLgskKv7WBsZhXPr0ObmZWaZ20zMys9z9pmZlYmHIzNzMpAFcRiB2Mzq3xuGZuZlZof4JmZlZ6QFyQ1MysHNVXQNHYwNrOKVwWx2MHYzCpbNglQ5UdjB2Mzq3hV0GXccTCWtF++EyNiRvGrY2bWddXeMv5pnmNta0KZmZVcFcTijoNxRBy2OStiZrYpRDa8rdJ1utKHpK0lfV/SdWl/hKQTur9qZmaFqVFhWzkrZNmlXwPrgIPT/gLgR91WIzOzrlC20kchWzkrJBjvFhH/ATQBRMRqqILvBGZWFQTU1qigrZwVMrRtXVqYLwAk7QY0dmutzMy6oMwbvQUpJBhfDDwE7CjpFuATwLndWSkzs64o9y6IQnQajCNikqQZwFiybwRfjYil3V4zM7MCqEpmbSukzxjg08ARwGHAp7qvOmZmXVcjFbR1RtKOkh6XNFvSLElfTek/kLRA0sy0HZ9zznclNUh6UdIxOenHprQGSd/p7NqdtowlXQN8FLg1Jf2DpCMj4oJO78zMbDMoYsO4GfhmRMyQ1AeYLmlSOnZFRPznB64r7QmcAewFDAH+KOlj6fDVwFHAfGCqpIkRMbujCxfSZ3w4MDIi2h7gTQBmFX5vZmbdq1h9xhGxCFiUPr8jaQ4wNM8p44DbIqIRmCepATgwHWuIiFdS/W5LeTsMxoV0UzQAO+Xs75jSzMxKTnTppY/tJE3L2c7vsFxpOLAvMDklXSjpOUk3SOqf0oYCb+ScNj+ldZTeoXwTBd1LNpytDzBH0pS0PwaYkq9QM7PNpmsvdCyNiNGdF6newF3A1yJilaRrgUvJYuClZHP3/P0m1rhd+bop/jPPMTOzslHM0RSS6skC8S0R8XuAiHgr5/gvgfvS7gKy3oI2w1IaedLblW+ioD8VWnkzs1JpewOvKGVlTezrgTkRcXlO+uDUnwxwMvBC+jwR+K2ky8ke4I0g6zkQMELSLmRB+AzgzHzXLmQ0xVjgKmAk0AOoBd6LiG0LvkMzs25UxJc+PgGcDTwvaWZK+x7wOUmjyLopXgX+ASAiZkm6g+zBXDNwQUS0pDpdCDxMFjNviIi8Ax8KGU3xc7KoficwGjgH+FjeM8zMNqNiheKIeLKD4h7Ic85lwGXtpD+Q77wNFfTSR0Q0ALUR0RIRvwaOLfQCZmbdSSreSx+lVEjLeLWkHsBMSf9BNgav0Df3zMy6XZnH2YIUElTPTvkuBN4je0J4SndWysysK6phPuNCJgp6LX1cC/wQQNLtwGe7sV5mZgUr8zhbkEK6KdpzUFFrYWa2iUT59wcXYlODsZlZeaiSKTTzvQ69X0eHgPruqc6Hs+/Infjr5J+XuhpWRP2PvLTUVbAianxpUeeZNkG59wcXIl/L+Kd5js0tdkXMzDZVNQzvyvc69GGbsyJmZpuimK9Dl5L7jM2s4lVBLHYwNrPKlq2BV/nR2MHYzCpeNbSMO+33Vubzkv417e8k6cDOzjMz21zaVojubCtnhTyEvIbsJY/Ppf13yBbaMzMruWzZpS1joqAxEbGfpGcAIuLtNHGQmVlZqOqhbTmaJNWSTaqMpEFAa7fWysysC8q80VuQQoLxlcDdwPaSLgNOBb7frbUyMyuQKqALohCFzNp2i6TpwBFk3TMnRcScbq+ZmVmBqiAWF7QG3k7AauDe3LSIeL07K2ZmVggBdVUwtq2Qbor7yfqLBfQCdgFeBPbqxnqZmRVsi2gZR8Q+uftpNrcvd1uNzMy6QlvISx8biogZwJhuqIuZ2SZRgf91Wo60o6THJc2WNEvSV1P6AEmTJL2cfvZP6ZJ0paQGSc/lTj0saXzK/7Kk8Z1du5A+42/k7NYA+wELO70rM7PNIHvpo2jFNQPfjIgZkvoA0yVNAs4FHo2If5f0HeA7wLeB44ARaRsDXAuMkTQAuBgYTdbNO13SxIh4u6MLF9Iy7pOz9STrQx63SbdpZtYNalTY1pmIWJS+/RMR7wBzgKFkMW9CyjYBOCl9HgfcFJmngX6SBgPHAJMiYnkKwJOAY/NdO2/LOL3s0SciLur8NszMSqM7Zm2TNBzYF5gM7BARbcuUvAnskD4PBd7IOW1+SusovUP5ll2qi4hmSZ/oQv3NzDarLnZTbCdpWs7+dRFx3UZlSr2Bu4CvRcSq3GAfESEpNr3G7cvXMp5C1j88U9JE4E7gvZwK/b7YlTEz67Kuzci2NCJG5y1OqicLxLfkxLm3JA2OiEWpG2JxSl8A7Jhz+rCUtgA4dIP0J/Jdt5A+417AMuBw4ATgxPTTzKwsFGvWNmVN4OuBORFxec6hiUDbiIjxwD056eekURVjgZWpO+Nh4GhJ/dPIi6NTWofytYy3TyMpXuD9lz7aFL2Jbma2KbI18IpW3CeAs4HnJc1Mad8D/h24Q9J5wGvA6enYA8DxQAPZm8pfAIiI5ZIuBaamfJdExPJ8F84XjGuB3tDu4DwHYzMrE6KmgDHEhYiIJ2k/5kE2P8+G+QO4oIOybgBuKPTa+YLxooi4pNCCzMxKQVT/69BVcHtmVvWq5HXofMF4oya5mVk5qur5jDvrbDYzKwdbQjeFmVlFqOqWsZlZpaiCWOxgbGaVTWw5q0ObmZUvdc9EQZubg7GZVbzKD8UOxmZW4QTUumVsZlZ6VRCLHYzNrNLJfcZmZqXm0RRmZmXCLWMzszJQ+aHYwdjMKp3HGZuZlZ77jM3MyoRbxmZmZaDyQ7GDsZlVOL+BZ2ZWJqogFjsYm1mlE6qCjopqeAhpZls4qbCt83J0g6TFkl7ISfuBpAWSZqbt+Jxj35XUIOlFScfkpB+b0hokfaeQe3AwNrOKlg1tU0FbAW4Ejm0n/YqIGJW2BwAk7QmcAeyVzrlGUq2kWuBq4DhgT+BzKW9e7qYws8pWYKu3EBHxZ0nDC8w+DrgtIhqBeZIagAPTsYaIeAVA0m0p7+x8hbllbGYVrwvdFNtJmpaznV/gJS6U9Fzqxuif0oYCb+TkmZ/SOkrPy8HYzCqeCvwPWBoRo3O26woo/lpgN2AUsAj4aXfcg7spzKyiCajpxsEUEfHW+mtJvwTuS7sLgB1zsg5LaeRJ75BbxmZW8brQMu562dLgnN2TgbaRFhOBMyT1lLQLMAKYAkwFRkjaRVIPsod8Ezu7jlvGZlbxaor0BE/SrcChZH3L84GLgUMljQICeBX4B4CImCXpDrIHc83ABRHRksq5EHgYqAVuiIhZnV3bwbjE1q5dy5GHHcK6xkaaW5o5+ZRT+b8X/5AjDv0U777zDgCLlyxm9AEHcuddf+DFuXM5/4tfYOYzM/jBpZfx9W9c1G65r86bx9lnncHy5cvYd7/9ueHGm+nRo8fmvLUtTkQr62b8AvXYlh77fJ7WNW/TNOcOaFqD+gyhfo9TUE0drStepel/HyTefYv6PU+jdtBe68tY+6eL0TY7AKBefemx91kbX6e1maa5vyfeWQj1W1G/5+nU9Oq/Ub4tRTG7KSLic+0kX58n/2XAZe2kPwA80JVrd2s3haSWNEj6BUn3Suq3ieWcK+nnxa5fOejZsycPTXqMKTOeZfK0mTzy8ENMfvppHn3iL0yePpPJ02cyZuxBnHTSKQD0HzCAn15xJV/rIAi3+ZfvfZuvfPXrzJrbQP9+/bnxhg7/PlmRtMx/Cm09aP1+87xHqBt2MD3HfA3V9aLlzRnZgV59qd/9ZGp22GfjQmrq6Tn6y/Qc/eV2AzFAy6IZqK4XPcd8jbphB9P8yqTuuJ0KUmgnRXm/pdfdfcZr0iDpvYHlwAXdfL2KI4nevXsD0NTURHNT0wemA1y1ahV/evwxThx3EgDbb789ow84gPr6+g7LjAj+9PhjnPJ3pwJw1tnjuXfiH7rxLiwaV9K6/CVqP7J/th9B69vzqBmUjfWv3WEUrUvnAFDTqz81vT/Cps411rpsDrU7jMrKGrQnrW+/QkR8+JuoVAUOayv3+Ss25wO8p0hj7STtJukhSdMl/UXSHin9REmTJT0j6Y+SdtiM9SuZlpYWxuw/ip2GbM/hRx7FgWPGrD927z1/4NDDj2DbbbctuLxly5bRt18/6uqyXqihw4axcGGnD3PtQ2hqeJC6XY95///45tWorhfZy1ignn2Jxnc6L6i1mcbpv6BxxnW0pOC9oWh8B/Xqm5WrWlTXE5pXF+U+KpUK3MrZZgnG6fXAI3j/ieJ1wFciYn/gIuCalP4kMDYi9gVuA75VQNnntw3gXrJ0SfErvxnU1tYyefpMGl6dz7SpU5j1wvrX4rnj9ls5/bPtdWNZuWhZ9iLqsQ01fYZ86LJ6jv0GPff/EvUjT6Wp4UFa1ywvQg2rW9ZnrIK2ctbdD/C2kjSTrEU8B5gkqTdwMHBnztfxnunnMOD2NJSkBzCvswukQdvXAey//+iK/q7Wr18/Pn3oYTzyyEPstffeLF26lGlTp3D77+7uUjkDBw5k5YoVNDc3U1dXx4L58xkypNMXgGwTta58nZalL9Ky7GVobYaWRpoaHiSa1xLRglRLNK5EPft0WpZ6Zt+AarYaQE2/4cS7i2CrARvk6UOsXZm1tqOFaG6Euq275d4qRXmH2cJslj5jYGeyP68L0jVX5Ey6MSoiRqb8VwE/j4h9yIaP9Orm+pXckiVLWLFiBQBr1qzh0T9OYvfd9wDg7rt+x3HHn0CvXl37Y5DEIYcexu/v+h0At9w8gRNOHFfcitt69bseRa+DLqLX2G9Qv+dp1PTbhR4jT6Wm3y60LsmmI2h5ayY1A0fmLSea1hCtzenze8Sq1z/wQLBNzcA9aHlrJgCtS2ZT03+Xqlh26EOpgn6KzdJNERGrgX8CvgmsJptU4zQAZT6esvbl/TdVxm+OupXam4sWceyRh3HAvn/DJw86gCOOPIrjP3MCAHfecRunn/HBLoo335NTTgEAAApxSURBVHyT3YYP48qfXc5Pfvwjdhs+jFWrVgFw0onHs3DhQgAu+/FPuPJnl7PXHh9l2fJlnPv3523eGzPqdj2K5vn/Q+PknxFNa6gdvB8ArasWsPap/6R1ySyaXppI49SrAIjVS1g34xc0Truadc/+mtodP0XNNtsD0DTvUVqWzgWgdvB+RNMaGif/jOb5/0PdLkeV5gbLSDWMplB3PoWV9G5E9M7Zvxe4g6xv+FpgMFBPNvPRJZLGAVcAbwOPAQdExKGSzgVGR8SF+a63//6j46+Tp3XPzVhJ9D/y0lJXwYqocfovaH1nQVGj4sh99o0J9zxRUN4xu/WbHhGji3n9YunWPuPcQJz2T8zZ3WjO0Ii4B7innfQbyeYZNTPbSHm3eQvjN/DMrKIJqqLP3MHYzCpbBbzQUQgHYzOreFUQix2MzawKVEE0djA2swpX/sPWCuFgbGYVz33GZmYlVgEv1xXEwdjMKl8VRGMHYzOreO4zNjMrA+4zNjMrNb/0YWZWHqqhm2JzLrtkZlZ02dwUxVkDT9INkhZLeiEnbYCkSZJeTj/7p3RJulJSg6TnJO2Xc874lP9lSQVNB+xgbGYVr4hzy9/IxjNKfgd4NCJGAI+mfYDjgBFpO59sWmAkDQAuBsYABwIXtwXwfByMzazyFSkaR8SfyVayzzUOmJA+TwBOykm/KTJPA/3SknHHAJMiYnlEvA1Mop0pgzfkPmMzq3hd6DPeTlLuChTXpXU089khIhalz28CbavWDwXeyMk3P6V1lJ6Xg7GZVbwujKZY+mFW+oiIkNQtyyO5m8LMKl43r0f6Vup+IP1cnNIXADvm5BuW0jpKz8vB2MwqX/dG44m8v0DyeN5fGm4icE4aVTEWWJm6Mx4GjpbUPz24Ozql5eVuCjOraFmcLc44Y0m3AoeS9S3PJxsV8e/AHZLOA14DTk/ZHwCOBxrIVr3/AkBELJd0KTA15bskIjZ8KLgRB2Mzq2yCmiK98xERn+vg0BHt5A3ggg7KuQG4oSvXdjA2s8pX+S/gORibWaXzSh9mZmXBEwWZmZWYV/owMysXVRCNHYzNrOK5z9jMrAy4z9jMrAxUQSx2MDazCudll8zMykXlR2MHYzOraKJ4r0OXkoOxmVU8d1OYmZUBD20zMysHlR+LHYzNrPJVQSx2MDazyiYPbTMzKw/uMzYzKweVH4sdjM2s8lVBLHYwNrPK5z5jM7MSE6KmCqJxTakrYGZWTiS9Kul5STMlTUtpAyRNkvRy+tk/pUvSlZIaJD0nab9Nva6DsZlVvLbhbZ1tXXBYRIyKiNFp/zvAoxExAng07QMcB4xI2/nAtZt6Dw7GZlbxVOB/H8I4YEL6PAE4KSf9psg8DfSTNHhTLuBgbGaVrcBWcWoZbydpWs52fjslBvCIpOk5x3eIiEXp85vADunzUOCNnHPnp7Qu8wM8M6toXVwdemlO10NHPhkRCyRtD0ySNDf3YESEpOhyRTvhlrGZVT4VuBUgIhakn4uBu4EDgbfauh/Sz8Up+wJgx5zTh6W0LnMwNrOKV6w+Y0nbSOrT9hk4GngBmAiMT9nGA/ekzxOBc9KoirHAypzujC5xN4WZVbwiDjPeAbhbWYF1wG8j4iFJU4E7JJ0HvAacnvI/ABwPNACrgS9s6oUdjM2s4hUrFkfEK8DH20lfBhzRTnoAFxTj2g7GZlbxVAVv4DkYm1lFE9UxN4WyVnZ1kLSErD+n2m0HLC11JayotpTf6c4RMaiYBUp6iOzPrxBLI+LYYl6/WKoqGG8pJE0rYKykVRD/Ts1D28zMyoCDsZlZGXAwrkzXlboCVnT+nW7h3GdsZlYG3DI2MysDDsZmZmXAwbiMSGpJS720bcM7yDdc0gubt3a2KXJ+py9IuldSv00s51xJPy92/ax8OBiXlzVpqZe27dVSV8g+tLbf6d7Acoo0j4FVHwfjMiapt6RHJc1ICySOayfPrpKekXSApN0kPZRWKPiLpD1KUW/r0FOkVSA6+l1JOlHS5PQ7/aOkHfKWaFXDc1OUl60kzUyf5wGnASdHxCpJ2wFPS5rYllnS7sBtwLkR8aykR4EvRcTLksYA1wCHb+Z7sHZIqiWb9ev6lHQd7f+ungTGptUkvgh8C/hmKepsm5eDcXlZExGj2nYk1QM/lnQI0ErWqmprKQ0im+D6lIiYLak3cDBwZ84MVj03W82tI23/wA4F5pAt45PvdzUMuD2tJtGD7B9l2wI4GJe3s8iC7v4R0STpVaBXOrYSeB34JDCbrMtpRW4wt7KwJiJGSdoaeJisz/hGOv5dXQVcHhETJR0K/GBzVdRKy33G5a0vsDgF4sOAnXOOrQNOJlvy5cyIWAXMk3QaQFoGZqNJsq00ImI18E9kXQ6r6fh31Zf311Abv1FBVrUcjMvbLcBoSc8D5wAbrlL7HnAC8HVJf0vWkj5P0rPALGCjB35WOhHxDPAc8Dk6/l39gKz7YjpbxpSalvh1aDOzMuCWsZlZGXAwNjMrAw7GZmZlwMHYzKwMOBibmZUBB2MDNppd7M70ksKmlnWjpFPT519J2jNP3kMlHbwJ13g1vSJeUHoHZXR5JrSulG/WFQ7G1iZ3drF1wJdyD0rapLc1I+KLETE7T5ZDyV4NNtuiORhbe/4CfDS1Wv+SJieaLalW0v+TNFXSc5L+Ada/QfZzSS9K+iOwfVtBkp6QNDp9PjbNQPdsmo1uOFnQ/3pqlX9K0iBJd6VrTJX0iXTuQEmPSJol6VeAKJCkAyU9lWZC+580wVKbHVMdX5Z0cc45n5c0JdXrv9NEP2bdxnNT2AekFvBxwEMpaT9g74iYJ+l8YGVEHCCpJ/BXSY8A+wK7A3uSTWQ0G7hhg3IHAb8EDkllDYiI5ZJ+AbwbEf+Z8v0WuCIinpS0E9l8DiOBi4EnI+ISSZ8BzuvCbc0FPhURzZKOBH4M/F06diCwN9krylMl3Q+8B3wW+ER6Ff0asjfmburCNc26xMHY2uRO3/kXsqkeDwamRETbzGFHA3/T1h9MNo/CCOAQ4NaIaAEWSnqsnfLHAn9uKysilndQjyOBPXNmM9s2zXJ2CHBKOvd+SW934d76AhMkjQACqM85NikilgFI+j3ZxEvNwP5kwRlgK2BxF65n1mUOxtZmzYaziKVA9F5uEvCViHh4g3zHF7EeNWTz+a5tpy6b6lLg8Yg4OXWNPJFzbMP5AILsPidExHc/zEXNusJ9xtYVDwP/mOZZRtLHJG0D/Bn4bOpTHgwc1s65TwOHSNolnTsgpb8D9MnJ9wjwlbYdSW3/QPwZODOlHQf070K9c2dCO3eDY0dJGiBpK+Ak4K/Ao8CpkrZvq6uknTHrRg7G1hW/IusPnqFsQdT/Jvt2dTfwcjp2E9nyQh8QEUuA84Hfp5nKbk+H7gVObnuARzbN5Oj0gHA274/q+CFZMJ9F1l3xep56PidpftouB/4D+DdJz7Dxt8EpwF1ks6ndFRHT0uiP7wOPSHoOmAQMLvDPyGyTeNY2M7My4JaxmVkZcDA2MysDDsZmZmXAwdjMrAw4GJuZlQEHYzOzMuBgbGZWBv4/yJvzIGkXamYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light",
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#plot confusion matrix\n",
    "plot_confusion_matrix(nn_conf_matrix, [\"Fake\",\"Real\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**7. Conclusion**\n",
    "\n",
    "The dataset was trained using three algorithms and the model performance is studied using F1 score and confusion matrix. Below shows the table of the models performance:\n",
    "\n",
    "| Model           | AUC | F1 score               | Accuracy |\n",
    "|-----------------|-----|------------------------|----------|\n",
    "|Decision Tree    |0.88 |Real:0.88<br>Fake: 0.88 |88.07%    |\n",
    "|Gradient Boosting|0.90 |Real: 0.89<br>Fake: 0.90|90.6%     |\n",
    "|Neural Network   |0.91 |Real: 0.91<br>Fake: 0.91|91.41%    |\n",
    "\n",
    "The best model is Neural Network with and accuracy of 91% and less false positives and false negatives compared to Decision tree and Gradient Boosting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "tapNZXsqJLf3"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "Fake_news_detection_using_title.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}