Phebe_Joshua_ML_Assignment_1.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Phebe_Joshua_ML_Assignment_1",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/PhebeJ/ML-Assignment/blob/main/Phebe_Joshua_ML_Assignment_1.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qqCjhZ2SaXod"
      },
      "source": [
        "# **MACHINE LEARNING ASSIGNMENT-1**\n",
        "\n",
        "---\n",
        " Submitted By,\n",
        "\n",
        " *Phebe Joshua*\n",
        "\n",
        " *M2 IS*\n",
        " \n",
        " *CET, TVM*\n",
        " \n",
        "---\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "G6OCsakprWpW"
      },
      "source": [
        "# Google colab removes the data directory. If the files are not present download it from external sources\n",
        "\n",
        "import urllib.request\n",
        "import os.path\n",
        "\n",
        "dataFiles = {\n",
        "    \"p1.pdf\" : \"https://temp-jrj.s3.ap-south-1.amazonaws.com/ml-assignment/p1.pdf\",\n",
        "    \"p2.pdf\" : \"https://temp-jrj.s3.ap-south-1.amazonaws.com/ml-assignment/p2.pdf\",\n",
        "    \"p3.pdf\" : \"https://temp-jrj.s3.ap-south-1.amazonaws.com/ml-assignment/p3.pdf\",\n",
        "    \"p4.pdf\" : \"https://temp-jrj.s3.ap-south-1.amazonaws.com/ml-assignment/p4.pdf\",\n",
        "    \"car_dataN.csv\" :\"https://temp-jrj.s3.ap-south-1.amazonaws.com/ml-assignment/car_dataN.csv\",\n",
        "    \"car_data.csv\" : \"https://temp-jrj.s3.ap-south-1.amazonaws.com/ml-assignment/car_data.csv\"\n",
        "}\n",
        "\n",
        "for file_name in dataFiles.keys():\n",
        "\n",
        "    fname = file_name\n",
        "\n",
        "    if os.path.isfile(fname):\n",
        "        continue\n",
        "\n",
        "    print(\"Downloading\" , fname , \"form\", dataFiles[file_name])\n",
        "    urllib.request.urlretrieve(dataFiles[file_name], fname)\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T1cqwhOHA5Gw"
      },
      "source": [
        "# 1. Develop a program to find dissimilarity between data objects given in ‘car_dataN.csv’. \n",
        "\n",
        "In the dataset, ‘make’ is an ordinal type variable with ranking as follows: \n",
        "\n",
        "1. audi \n",
        "2. bmw \n",
        "3. alfa-romero \n",
        "4. chevrolet \n",
        "5. dodge. \n",
        "\n",
        "Other varables are nominal and numeric types. Identify the most dissimilar tuples."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aWDkMG2xSOmm"
      },
      "source": [
        "import pandas as pd\n",
        "import numpy as np"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DN9Mi-J_CGeq"
      },
      "source": [
        "Import data from `car_dataN.csv`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 511
        },
        "id": "l8Bi91zTB-0V",
        "outputId": "0763e07f-396a-42c4-ff41-a550edba47e6"
      },
      "source": [
        "df = pd.read_csv('car_dataN.csv')\n",
        "df"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Obj Id</th>\n",
              "      <th>make</th>\n",
              "      <th>num_of_doors</th>\n",
              "      <th>body_style</th>\n",
              "      <th>wheel_base</th>\n",
              "      <th>curb_weight</th>\n",
              "      <th>num_of_cylinders</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>alfa-romero</td>\n",
              "      <td>two</td>\n",
              "      <td>convertible</td>\n",
              "      <td>88.6</td>\n",
              "      <td>2548</td>\n",
              "      <td>four</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>alfa-romero</td>\n",
              "      <td>two</td>\n",
              "      <td>convertible</td>\n",
              "      <td>88.6</td>\n",
              "      <td>2548</td>\n",
              "      <td>four</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>audi</td>\n",
              "      <td>two</td>\n",
              "      <td>sedan</td>\n",
              "      <td>99.8</td>\n",
              "      <td>2507</td>\n",
              "      <td>five</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>4</td>\n",
              "      <td>audi</td>\n",
              "      <td>four</td>\n",
              "      <td>sedan</td>\n",
              "      <td>105.8</td>\n",
              "      <td>2844</td>\n",
              "      <td>five</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>5</td>\n",
              "      <td>audi</td>\n",
              "      <td>four</td>\n",
              "      <td>sedan</td>\n",
              "      <td>105.8</td>\n",
              "      <td>3086</td>\n",
              "      <td>five</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>6</td>\n",
              "      <td>audi</td>\n",
              "      <td>two</td>\n",
              "      <td>hatchback</td>\n",
              "      <td>99.5</td>\n",
              "      <td>3053</td>\n",
              "      <td>five</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>7</td>\n",
              "      <td>bmw</td>\n",
              "      <td>two</td>\n",
              "      <td>sedan</td>\n",
              "      <td>101.2</td>\n",
              "      <td>2395</td>\n",
              "      <td>four</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>8</td>\n",
              "      <td>bmw</td>\n",
              "      <td>four</td>\n",
              "      <td>sedan</td>\n",
              "      <td>101.2</td>\n",
              "      <td>2395</td>\n",
              "      <td>four</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>9</td>\n",
              "      <td>bmw</td>\n",
              "      <td>four</td>\n",
              "      <td>sedan</td>\n",
              "      <td>110.0</td>\n",
              "      <td>3505</td>\n",
              "      <td>six</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>10</td>\n",
              "      <td>chevrolet</td>\n",
              "      <td>two</td>\n",
              "      <td>hatchback</td>\n",
              "      <td>88.4</td>\n",
              "      <td>1488</td>\n",
              "      <td>three</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>11</td>\n",
              "      <td>chevrolet</td>\n",
              "      <td>two</td>\n",
              "      <td>hatchback</td>\n",
              "      <td>94.5</td>\n",
              "      <td>1874</td>\n",
              "      <td>four</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>12</td>\n",
              "      <td>chevrolet</td>\n",
              "      <td>four</td>\n",
              "      <td>sedan</td>\n",
              "      <td>94.5</td>\n",
              "      <td>1909</td>\n",
              "      <td>four</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>13</td>\n",
              "      <td>dodge</td>\n",
              "      <td>two</td>\n",
              "      <td>hatchback</td>\n",
              "      <td>93.7</td>\n",
              "      <td>1876</td>\n",
              "      <td>four</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>14</td>\n",
              "      <td>dodge</td>\n",
              "      <td>two</td>\n",
              "      <td>hatchback</td>\n",
              "      <td>93.7</td>\n",
              "      <td>1876</td>\n",
              "      <td>four</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>15</td>\n",
              "      <td>dodge</td>\n",
              "      <td>two</td>\n",
              "      <td>hatchback</td>\n",
              "      <td>93.7</td>\n",
              "      <td>2128</td>\n",
              "      <td>four</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "    Obj Id         make num_of_doors  ... wheel_base  curb_weight  num_of_cylinders\n",
              "0        1  alfa-romero          two  ...       88.6         2548              four\n",
              "1        2  alfa-romero          two  ...       88.6         2548              four\n",
              "2        3         audi          two  ...       99.8         2507              five\n",
              "3        4         audi         four  ...      105.8         2844              five\n",
              "4        5         audi         four  ...      105.8         3086              five\n",
              "5        6         audi          two  ...       99.5         3053              five\n",
              "6        7          bmw          two  ...      101.2         2395              four\n",
              "7        8          bmw         four  ...      101.2         2395              four\n",
              "8        9          bmw         four  ...      110.0         3505               six\n",
              "9       10    chevrolet          two  ...       88.4         1488             three\n",
              "10      11    chevrolet          two  ...       94.5         1874              four\n",
              "11      12    chevrolet         four  ...       94.5         1909              four\n",
              "12      13        dodge          two  ...       93.7         1876              four\n",
              "13      14        dodge          two  ...       93.7         1876              four\n",
              "14      15        dodge          two  ...       93.7         2128              four\n",
              "\n",
              "[15 rows x 7 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jqP25AJrCN33"
      },
      "source": [
        "Pre process the data. Replace `make` with ordinal values "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 511
        },
        "id": "XmJ0Wz9JcuyF",
        "outputId": "39735f16-8e9f-49f7-fdba-9960cdaa1584"
      },
      "source": [
        "df['make'] = df['make'].replace('audi', 1)\n",
        "df['make'] = df['make'].replace('bmw', 2)\n",
        "df['make'] = df['make'].replace('alfa-romero', 3)\n",
        "df['make'] = df['make'].replace('chevrolet', 4)\n",
        "df['make'] = df['make'].replace('dodge', 5)\n",
        "\n",
        "df"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Obj Id</th>\n",
              "      <th>make</th>\n",
              "      <th>num_of_doors</th>\n",
              "      <th>body_style</th>\n",
              "      <th>wheel_base</th>\n",
              "      <th>curb_weight</th>\n",
              "      <th>num_of_cylinders</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>3</td>\n",
              "      <td>two</td>\n",
              "      <td>convertible</td>\n",
              "      <td>88.6</td>\n",
              "      <td>2548</td>\n",
              "      <td>four</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>3</td>\n",
              "      <td>two</td>\n",
              "      <td>convertible</td>\n",
              "      <td>88.6</td>\n",
              "      <td>2548</td>\n",
              "      <td>four</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>1</td>\n",
              "      <td>two</td>\n",
              "      <td>sedan</td>\n",
              "      <td>99.8</td>\n",
              "      <td>2507</td>\n",
              "      <td>five</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>4</td>\n",
              "      <td>1</td>\n",
              "      <td>four</td>\n",
              "      <td>sedan</td>\n",
              "      <td>105.8</td>\n",
              "      <td>2844</td>\n",
              "      <td>five</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>5</td>\n",
              "      <td>1</td>\n",
              "      <td>four</td>\n",
              "      <td>sedan</td>\n",
              "      <td>105.8</td>\n",
              "      <td>3086</td>\n",
              "      <td>five</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>6</td>\n",
              "      <td>1</td>\n",
              "      <td>two</td>\n",
              "      <td>hatchback</td>\n",
              "      <td>99.5</td>\n",
              "      <td>3053</td>\n",
              "      <td>five</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>7</td>\n",
              "      <td>2</td>\n",
              "      <td>two</td>\n",
              "      <td>sedan</td>\n",
              "      <td>101.2</td>\n",
              "      <td>2395</td>\n",
              "      <td>four</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>8</td>\n",
              "      <td>2</td>\n",
              "      <td>four</td>\n",
              "      <td>sedan</td>\n",
              "      <td>101.2</td>\n",
              "      <td>2395</td>\n",
              "      <td>four</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>9</td>\n",
              "      <td>2</td>\n",
              "      <td>four</td>\n",
              "      <td>sedan</td>\n",
              "      <td>110.0</td>\n",
              "      <td>3505</td>\n",
              "      <td>six</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>10</td>\n",
              "      <td>4</td>\n",
              "      <td>two</td>\n",
              "      <td>hatchback</td>\n",
              "      <td>88.4</td>\n",
              "      <td>1488</td>\n",
              "      <td>three</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>11</td>\n",
              "      <td>4</td>\n",
              "      <td>two</td>\n",
              "      <td>hatchback</td>\n",
              "      <td>94.5</td>\n",
              "      <td>1874</td>\n",
              "      <td>four</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>12</td>\n",
              "      <td>4</td>\n",
              "      <td>four</td>\n",
              "      <td>sedan</td>\n",
              "      <td>94.5</td>\n",
              "      <td>1909</td>\n",
              "      <td>four</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>13</td>\n",
              "      <td>5</td>\n",
              "      <td>two</td>\n",
              "      <td>hatchback</td>\n",
              "      <td>93.7</td>\n",
              "      <td>1876</td>\n",
              "      <td>four</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>14</td>\n",
              "      <td>5</td>\n",
              "      <td>two</td>\n",
              "      <td>hatchback</td>\n",
              "      <td>93.7</td>\n",
              "      <td>1876</td>\n",
              "      <td>four</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>15</td>\n",
              "      <td>5</td>\n",
              "      <td>two</td>\n",
              "      <td>hatchback</td>\n",
              "      <td>93.7</td>\n",
              "      <td>2128</td>\n",
              "      <td>four</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "    Obj Id  make num_of_doors  ... wheel_base  curb_weight  num_of_cylinders\n",
              "0        1     3          two  ...       88.6         2548              four\n",
              "1        2     3          two  ...       88.6         2548              four\n",
              "2        3     1          two  ...       99.8         2507              five\n",
              "3        4     1         four  ...      105.8         2844              five\n",
              "4        5     1         four  ...      105.8         3086              five\n",
              "5        6     1          two  ...       99.5         3053              five\n",
              "6        7     2          two  ...      101.2         2395              four\n",
              "7        8     2         four  ...      101.2         2395              four\n",
              "8        9     2         four  ...      110.0         3505               six\n",
              "9       10     4          two  ...       88.4         1488             three\n",
              "10      11     4          two  ...       94.5         1874              four\n",
              "11      12     4         four  ...       94.5         1909              four\n",
              "12      13     5          two  ...       93.7         1876              four\n",
              "13      14     5          two  ...       93.7         1876              four\n",
              "14      15     5          two  ...       93.7         2128              four\n",
              "\n",
              "[15 rows x 7 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oueUp05uCfDg"
      },
      "source": [
        "Function for calculating the difference of the two rows"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LAj2hYllci7-"
      },
      "source": [
        "def calcDiffNominal(it1, it2):\n",
        "    if it1 == it2 :\n",
        "        return 0\n",
        "    return 1\n",
        "\n",
        "def calcDiffOrdinal(it1, it2, range):\n",
        "    return abs(it1 - it2) / (range - 1)\n",
        "\n",
        "def calcDiffNumeric(it1, it2):\n",
        "    return abs(it1 - it2)\n",
        "\n",
        "def calculateDifference(row1,row2):\n",
        "\n",
        "    wheelDissm = calcDiffNominal(row1[\"num_of_doors\"], row2[\"num_of_doors\"])\n",
        "    bodyStyleDissm = calcDiffNominal(row1[\"body_style\"], row2[\"body_style\"])\n",
        "    cylinDissm = calcDiffNominal(row1[\"num_of_cylinders\"], row2[\"num_of_cylinders\"])\n",
        "\n",
        "    makeDissm = calcDiffOrdinal(row1[\"make\"], row2[\"make\"], 5)\n",
        "\n",
        "    wheelBaseDissm = calcDiffNumeric(row1[\"wheel_base\"], row2[\"wheel_base\"])\n",
        "    curbweightDissm = calcDiffNumeric(row1[\"curb_weight\"], row2[\"curb_weight\"])\n",
        "\n",
        "    return wheelDissm + bodyStyleDissm + cylinDissm + makeDissm + wheelBaseDissm + curbweightDissm"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YK35vuH7CozI"
      },
      "source": [
        "Iterate over each row and find difference against each rows and find the row with maximum difference."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pdGkaLLRg_Oj"
      },
      "source": [
        "maxRow1 = df.head(1)\n",
        "maxRow2 = df.head(1)\n",
        "maxDistance = 0;\n",
        "for index1, row1 in df.iterrows():\n",
        "    for index2, row2 in df.iterrows():\n",
        "        if(index2 >= index1):\n",
        "            break\n",
        "        currentDistance = calculateDifference(row1,row2)\n",
        "        if(maxDistance < currentDistance):\n",
        "            maxDistance = currentDistance\n",
        "            maxRow1 = row1\n",
        "            maxRow2 = row2\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "raTJkCeVC0lv"
      },
      "source": [
        "Finally print the rows"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 110
        },
        "id": "MzzMzYtwBcnb",
        "outputId": "4a99c645-1f1a-4a70-fdeb-6299fff4d864"
      },
      "source": [
        "pd.DataFrame([maxRow1, maxRow2])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Obj Id</th>\n",
              "      <th>make</th>\n",
              "      <th>num_of_doors</th>\n",
              "      <th>body_style</th>\n",
              "      <th>wheel_base</th>\n",
              "      <th>curb_weight</th>\n",
              "      <th>num_of_cylinders</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>10</td>\n",
              "      <td>4</td>\n",
              "      <td>two</td>\n",
              "      <td>hatchback</td>\n",
              "      <td>88.4</td>\n",
              "      <td>1488</td>\n",
              "      <td>three</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>9</td>\n",
              "      <td>2</td>\n",
              "      <td>four</td>\n",
              "      <td>sedan</td>\n",
              "      <td>110.0</td>\n",
              "      <td>3505</td>\n",
              "      <td>six</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   Obj Id  make num_of_doors  ... wheel_base  curb_weight  num_of_cylinders\n",
              "9      10     4          two  ...       88.4         1488             three\n",
              "8       9     2         four  ...      110.0         3505               six\n",
              "\n",
              "[2 rows x 7 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IIWp6WR6YFE6"
      },
      "source": [
        "\n",
        "\n",
        "---\n",
        "\n",
        "\n",
        "\n",
        "---\n",
        "\n",
        "\n",
        "\n",
        "---\n",
        "\n",
        "\n",
        "\n",
        "---\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4db5dqT_QXuq"
      },
      "source": [
        "# 3. Develop a program to extract all stem words from a given set of documents (paper1, paper2, paper2, paper4). \n",
        "\n",
        "Use cosine similarity to find how similar the documents. (You may python libraries to extract stem\n",
        "words)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "w3T8AVWRQ_9X",
        "outputId": "2ea2bf95-3c7e-46d0-c486-4fa495a1f03f"
      },
      "source": [
        "!pip install PyPDF2"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: PyPDF2 in /usr/local/lib/python3.7/dist-packages (1.26.0)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xFtCb4pTQGaJ"
      },
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "import PyPDF2\n",
        "from nltk.stem import PorterStemmer"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ohd4P1wCkNm-"
      },
      "source": [
        "Function to extract words from a pdf file"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ec1zGEzdRGhu"
      },
      "source": [
        "def readPdf(fileName):\n",
        "    finalText = \"\";\n",
        "    pdfFileObj = open(fileName, 'rb')\n",
        "    pdfReader = PyPDF2.PdfFileReader(pdfFileObj) \n",
        "    for i in range(0, pdfReader.numPages):\n",
        "        finalText += pdfReader.getPage(i).extractText()\n",
        "    pdfFileObj.close()\n",
        "    return finalText"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JqRTmMHgkn9r"
      },
      "source": [
        "Function to stem words and calculate the frequency of each word in the document\n",
        "This function will\n",
        " * Remove all special characters. Use only alphabets so that only valid words are cinsidered\n",
        " * Skips words with length less than 4. This eliminates common words like `the`, `is`, `was`, etc\n",
        " * Skips words with length > 10, this will reduce the chances of concatenated words, URLs etc.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ba2Wsx0HVUfB"
      },
      "source": [
        "def calcFreq(text):\n",
        "\n",
        "    ps = PorterStemmer()\n",
        "\n",
        "    words = text.split()\n",
        "    frequency = {}\n",
        "    for word in words:\n",
        "        word = ''.join(e for e in word if e.isalpha())\n",
        "        word = ps.stem(word)\n",
        "\n",
        "        if len(word) <= 3 or len(word) > 10:\n",
        "            continue\n",
        "\n",
        "        if word in frequency:\n",
        "            frequency[word] += 1\n",
        "        else:\n",
        "            frequency[word] = 1\n",
        "    \n",
        "    return frequency"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k9KDWJO1lU3C"
      },
      "source": [
        "Create a word frequency array `wfCount` by steming words extracted form each file"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vEL4gABXa9BB"
      },
      "source": [
        "files = {\n",
        "    \"p1.pdf\" : \"A High-Throughput Radioactivity-Based Assay for Screening SARS-CoV-2 nsp10-nsp16 Complex\", \n",
        "    \"p2.pdf\": \"COVID-19 vaccination in patients with multiple sclerosis: What we have learnt by February 2021\", \n",
        "    \"p3.pdf\": \"Impact of COVID-19 Pandemic on Consumer Economy: Countermeasures Analysis\", \n",
        "    \"p4.pdf\": \"COVID-19 Impact on the Future of Hospitality and Travel\"\n",
        "}\n",
        "filenames = files.keys()\n",
        "\n",
        "wfCount = {}\n",
        "\n",
        "for fileName in filenames:\n",
        "    text = readPdf(fileName)\n",
        "    wordFreqCount = calcFreq(text)\n",
        "    wfCount[fileName] = wordFreqCount;"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S1Uvjnril5xn"
      },
      "source": [
        "Convert the array to pandas data frame\n",
        "Fill the non numeric values with 0"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 417
        },
        "id": "FQZYlG-9lRW6",
        "outputId": "f60dd59b-c975-4007-dcc4-77fe6062e96d"
      },
      "source": [
        "df = pd.DataFrame(wfCount)\n",
        "\n",
        "for fileName in filenames:\n",
        "    df[fileName] = df[fileName].fillna(0)\n",
        "\n",
        "df"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>p1.pdf</th>\n",
              "      <th>p2.pdf</th>\n",
              "      <th>p3.pdf</th>\n",
              "      <th>p4.pdf</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>discoveri</th>\n",
              "      <td>6.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>author</th>\n",
              "      <td>3.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>2.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>recent</th>\n",
              "      <td>3.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>outbreak</th>\n",
              "      <td>2.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>19.0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>sever</th>\n",
              "      <td>12.0</td>\n",
              "      <td>6.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>1.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>bachelor</th>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>michigan</th>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>2.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attend</th>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>serv</th>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>assist</th>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>2296 rows × 4 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "           p1.pdf  p2.pdf  p3.pdf  p4.pdf\n",
              "discoveri     6.0     0.0     0.0     0.0\n",
              "author        3.0     5.0     4.0     2.0\n",
              "recent        3.0     4.0     1.0     0.0\n",
              "outbreak      2.0     0.0    19.0     0.0\n",
              "sever        12.0     6.0     5.0     1.0\n",
              "...           ...     ...     ...     ...\n",
              "bachelor      0.0     0.0     0.0     1.0\n",
              "michigan      0.0     0.0     0.0     2.0\n",
              "attend        0.0     0.0     0.0     1.0\n",
              "serv          0.0     0.0     0.0     1.0\n",
              "assist        0.0     0.0     0.0     1.0\n",
              "\n",
              "[2296 rows x 4 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SEDeIi8rmNlW"
      },
      "source": [
        "Function to find the cosine simlarity"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "y4oU1LaIh8Mj"
      },
      "source": [
        "def findCosim(i1, i2):\n",
        "    dp = i1.dot(i2)\n",
        "    len = i1.shape[0] * i2.shape[0]\n",
        "    return dp/len"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KxWd4ZMUmQp9"
      },
      "source": [
        "Iterate through all files and print the cosine simlarity"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MXx1ri7ng9Ga",
        "outputId": "4139c137-b8aa-43f3-a516-b08f3d4442d6"
      },
      "source": [
        "for fileName1 in filenames:\n",
        "    for fileName2 in filenames:\n",
        "        if(fileName1 == fileName2):\n",
        "            break\n",
        "        cSim = findCosim(df[fileName1], df[fileName2])\n",
        "        print(\"Cosine simlarity between\", fileName2 , \"and\", fileName1, \"is\", cSim)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Cosine simlarity between p1.pdf and p2.pdf is 0.001396156320945987\n",
            "Cosine simlarity between p1.pdf and p3.pdf is 0.0013373508237322293\n",
            "Cosine simlarity between p2.pdf and p3.pdf is 0.0016774742318105113\n",
            "Cosine simlarity between p1.pdf and p4.pdf is 0.00029345840061188066\n",
            "Cosine simlarity between p2.pdf and p4.pdf is 0.0004194159817407034\n",
            "Cosine simlarity between p3.pdf and p4.pdf is 0.000949613932426034\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}