numeric_features.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Screencast Code\n",
    "\n",
    "The follow code is the same used in the \"Numeric Features\" screencast. Run each code cell to see how "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.sql import SparkSession\n",
    "from pyspark.ml.feature import RegexTokenizer, VectorAssembler, Normalizer, StandardScaler\n",
    "from pyspark.sql.functions import udf\n",
    "from pyspark.sql.types import IntegerType\n",
    "\n",
    "import re"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a SparkSession: note this step was left out of the screencast\n",
    "spark = SparkSession.builder \\\n",
    "    .master(\"local\") \\\n",
    "    .appName(\"Word Count\") \\\n",
    "    .getOrCreate()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Read in the Data Set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "stack_overflow_data = 'Train_onetag_small.json'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = spark.read.json(stack_overflow_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Row(Body=\"<p>I'd like to check if an uploaded file is an image file (e.g png, jpg, jpeg, gif, bmp) or another file. The problem is that I'm using Uploadify to upload the files, which changes the mime type and gives a 'text/octal' or something as the mime type, no matter which file type you upload.</p>\\n\\n<p>Is there a way to check if the uploaded file is an image apart from checking the file extension using PHP?</p>\\n\", Id=1, Tags='php image-processing file-upload upload mime-types', Title='How to check if an uploaded file is an image without mime type?', oneTag='php')"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tokenization\n",
    "\n",
    "Tokenization splits strings into separate words. Spark has a [Tokenizer](https://spark.apache.org/docs/latest/ml-features.html#tokenizer) class as well as RegexTokenizer, which allows for more control over the tokenization process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Row(Body=\"<p>I'd like to check if an uploaded file is an image file (e.g png, jpg, jpeg, gif, bmp) or another file. The problem is that I'm using Uploadify to upload the files, which changes the mime type and gives a 'text/octal' or something as the mime type, no matter which file type you upload.</p>\\n\\n<p>Is there a way to check if the uploaded file is an image apart from checking the file extension using PHP?</p>\\n\", Id=1, Tags='php image-processing file-upload upload mime-types', Title='How to check if an uploaded file is an image without mime type?', oneTag='php', words=['p', 'i', 'd', 'like', 'to', 'check', 'if', 'an', 'uploaded', 'file', 'is', 'an', 'image', 'file', 'e', 'g', 'png', 'jpg', 'jpeg', 'gif', 'bmp', 'or', 'another', 'file', 'the', 'problem', 'is', 'that', 'i', 'm', 'using', 'uploadify', 'to', 'upload', 'the', 'files', 'which', 'changes', 'the', 'mime', 'type', 'and', 'gives', 'a', 'text', 'octal', 'or', 'something', 'as', 'the', 'mime', 'type', 'no', 'matter', 'which', 'file', 'type', 'you', 'upload', 'p', 'p', 'is', 'there', 'a', 'way', 'to', 'check', 'if', 'the', 'uploaded', 'file', 'is', 'an', 'image', 'apart', 'from', 'checking', 'the', 'file', 'extension', 'using', 'php', 'p'])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# split the body text into separate words\n",
    "\n",
    "regexTokenizer = RegexTokenizer(inputCol=\"Body\", outputCol=\"words\", pattern=\"\\\\W\")\n",
    "df = regexTokenizer.transform(df)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# count the number of words in each body tag\n",
    "\n",
    "body_length = udf(lambda x: len(x), IntegerType())\n",
    "df = df.withColumn(\"BodyLength\", body_length(df.words))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# count the number of paragraphs and links in each body tag\n",
    "\n",
    "number_of_paragraphs = udf(lambda x: len(re.findall(\"</p>\", x)), IntegerType())\n",
    "number_of_links = udf(lambda x: len(re.findall(\"</a>\", x)), IntegerType())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df.withColumn(\"NumParagraphs\", number_of_paragraphs(df.Body))\n",
    "df = df.withColumn(\"NumLinks\", number_of_links(df.Body))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Row(Body=\"<p>I'd like to check if an uploaded file is an image file (e.g png, jpg, jpeg, gif, bmp) or another file. The problem is that I'm using Uploadify to upload the files, which changes the mime type and gives a 'text/octal' or something as the mime type, no matter which file type you upload.</p>\\n\\n<p>Is there a way to check if the uploaded file is an image apart from checking the file extension using PHP?</p>\\n\", Id=1, Tags='php image-processing file-upload upload mime-types', Title='How to check if an uploaded file is an image without mime type?', oneTag='php', words=['p', 'i', 'd', 'like', 'to', 'check', 'if', 'an', 'uploaded', 'file', 'is', 'an', 'image', 'file', 'e', 'g', 'png', 'jpg', 'jpeg', 'gif', 'bmp', 'or', 'another', 'file', 'the', 'problem', 'is', 'that', 'i', 'm', 'using', 'uploadify', 'to', 'upload', 'the', 'files', 'which', 'changes', 'the', 'mime', 'type', 'and', 'gives', 'a', 'text', 'octal', 'or', 'something', 'as', 'the', 'mime', 'type', 'no', 'matter', 'which', 'file', 'type', 'you', 'upload', 'p', 'p', 'is', 'there', 'a', 'way', 'to', 'check', 'if', 'the', 'uploaded', 'file', 'is', 'an', 'image', 'apart', 'from', 'checking', 'the', 'file', 'extension', 'using', 'php', 'p'], BodyLength=83, NumParagraphs=2, NumLinks=0),\n",
       " Row(Body='<p>In my favorite editor (vim), I regularly use ctrl-w to execute a certain action. Now, it quite often happens to me that firefox is the active window (on windows) while I still look at vim (thinking vim is the active window) and press ctrl-w which closes firefox. This is not what I want. Is there a way to stop ctrl-w from closing firefox?</p>\\n\\n<p>Rene</p>\\n', Id=2, Tags='firefox', Title='How can I prevent firefox from closing when I press ctrl-w', oneTag='firefox', words=['p', 'in', 'my', 'favorite', 'editor', 'vim', 'i', 'regularly', 'use', 'ctrl', 'w', 'to', 'execute', 'a', 'certain', 'action', 'now', 'it', 'quite', 'often', 'happens', 'to', 'me', 'that', 'firefox', 'is', 'the', 'active', 'window', 'on', 'windows', 'while', 'i', 'still', 'look', 'at', 'vim', 'thinking', 'vim', 'is', 'the', 'active', 'window', 'and', 'press', 'ctrl', 'w', 'which', 'closes', 'firefox', 'this', 'is', 'not', 'what', 'i', 'want', 'is', 'there', 'a', 'way', 'to', 'stop', 'ctrl', 'w', 'from', 'closing', 'firefox', 'p', 'p', 'rene', 'p'], BodyLength=71, NumParagraphs=2, NumLinks=0)]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# VectorAssembler\n",
    "\n",
    "Combine the body length, number of paragraphs, and number of links columns into a vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "assembler = VectorAssembler(inputCols=[\"BodyLength\", \"NumParagraphs\", \"NumLinks\"], outputCol=\"NumFeatures\")\n",
    "df = assembler.transform(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Row(Body=\"<p>I'd like to check if an uploaded file is an image file (e.g png, jpg, jpeg, gif, bmp) or another file. The problem is that I'm using Uploadify to upload the files, which changes the mime type and gives a 'text/octal' or something as the mime type, no matter which file type you upload.</p>\\n\\n<p>Is there a way to check if the uploaded file is an image apart from checking the file extension using PHP?</p>\\n\", Id=1, Tags='php image-processing file-upload upload mime-types', Title='How to check if an uploaded file is an image without mime type?', oneTag='php', words=['p', 'i', 'd', 'like', 'to', 'check', 'if', 'an', 'uploaded', 'file', 'is', 'an', 'image', 'file', 'e', 'g', 'png', 'jpg', 'jpeg', 'gif', 'bmp', 'or', 'another', 'file', 'the', 'problem', 'is', 'that', 'i', 'm', 'using', 'uploadify', 'to', 'upload', 'the', 'files', 'which', 'changes', 'the', 'mime', 'type', 'and', 'gives', 'a', 'text', 'octal', 'or', 'something', 'as', 'the', 'mime', 'type', 'no', 'matter', 'which', 'file', 'type', 'you', 'upload', 'p', 'p', 'is', 'there', 'a', 'way', 'to', 'check', 'if', 'the', 'uploaded', 'file', 'is', 'an', 'image', 'apart', 'from', 'checking', 'the', 'file', 'extension', 'using', 'php', 'p'], BodyLength=83, NumParagraphs=2, NumLinks=0, NumFeatures=DenseVector([83.0, 2.0, 0.0]))"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Normalize the Vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "scaler = Normalizer(inputCol=\"NumFeatures\", outputCol=\"ScaledNumFeatures\")\n",
    "df = scaler.transform(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Row(Body=\"<p>I'd like to check if an uploaded file is an image file (e.g png, jpg, jpeg, gif, bmp) or another file. The problem is that I'm using Uploadify to upload the files, which changes the mime type and gives a 'text/octal' or something as the mime type, no matter which file type you upload.</p>\\n\\n<p>Is there a way to check if the uploaded file is an image apart from checking the file extension using PHP?</p>\\n\", Id=1, Tags='php image-processing file-upload upload mime-types', Title='How to check if an uploaded file is an image without mime type?', oneTag='php', words=['p', 'i', 'd', 'like', 'to', 'check', 'if', 'an', 'uploaded', 'file', 'is', 'an', 'image', 'file', 'e', 'g', 'png', 'jpg', 'jpeg', 'gif', 'bmp', 'or', 'another', 'file', 'the', 'problem', 'is', 'that', 'i', 'm', 'using', 'uploadify', 'to', 'upload', 'the', 'files', 'which', 'changes', 'the', 'mime', 'type', 'and', 'gives', 'a', 'text', 'octal', 'or', 'something', 'as', 'the', 'mime', 'type', 'no', 'matter', 'which', 'file', 'type', 'you', 'upload', 'p', 'p', 'is', 'there', 'a', 'way', 'to', 'check', 'if', 'the', 'uploaded', 'file', 'is', 'an', 'image', 'apart', 'from', 'checking', 'the', 'file', 'extension', 'using', 'php', 'p'], BodyLength=83, NumParagraphs=2, NumLinks=0, NumFeatures=DenseVector([83.0, 2.0, 0.0]), ScaledNumFeatures=DenseVector([0.9997, 0.0241, 0.0])),\n",
       " Row(Body='<p>In my favorite editor (vim), I regularly use ctrl-w to execute a certain action. Now, it quite often happens to me that firefox is the active window (on windows) while I still look at vim (thinking vim is the active window) and press ctrl-w which closes firefox. This is not what I want. Is there a way to stop ctrl-w from closing firefox?</p>\\n\\n<p>Rene</p>\\n', Id=2, Tags='firefox', Title='How can I prevent firefox from closing when I press ctrl-w', oneTag='firefox', words=['p', 'in', 'my', 'favorite', 'editor', 'vim', 'i', 'regularly', 'use', 'ctrl', 'w', 'to', 'execute', 'a', 'certain', 'action', 'now', 'it', 'quite', 'often', 'happens', 'to', 'me', 'that', 'firefox', 'is', 'the', 'active', 'window', 'on', 'windows', 'while', 'i', 'still', 'look', 'at', 'vim', 'thinking', 'vim', 'is', 'the', 'active', 'window', 'and', 'press', 'ctrl', 'w', 'which', 'closes', 'firefox', 'this', 'is', 'not', 'what', 'i', 'want', 'is', 'there', 'a', 'way', 'to', 'stop', 'ctrl', 'w', 'from', 'closing', 'firefox', 'p', 'p', 'rene', 'p'], BodyLength=71, NumParagraphs=2, NumLinks=0, NumFeatures=DenseVector([71.0, 2.0, 0.0]), ScaledNumFeatures=DenseVector([0.9996, 0.0282, 0.0]))]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Scale the Vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "scaler2 = StandardScaler(inputCol=\"NumFeatures\", outputCol=\"ScaledNumFeatures2\", withStd=True)\n",
    "scalerModel = scaler2.fit(df)\n",
    "df = scalerModel.transform(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Row(Body=\"<p>I'd like to check if an uploaded file is an image file (e.g png, jpg, jpeg, gif, bmp) or another file. The problem is that I'm using Uploadify to upload the files, which changes the mime type and gives a 'text/octal' or something as the mime type, no matter which file type you upload.</p>\\n\\n<p>Is there a way to check if the uploaded file is an image apart from checking the file extension using PHP?</p>\\n\", Id=1, Tags='php image-processing file-upload upload mime-types', Title='How to check if an uploaded file is an image without mime type?', oneTag='php', words=['p', 'i', 'd', 'like', 'to', 'check', 'if', 'an', 'uploaded', 'file', 'is', 'an', 'image', 'file', 'e', 'g', 'png', 'jpg', 'jpeg', 'gif', 'bmp', 'or', 'another', 'file', 'the', 'problem', 'is', 'that', 'i', 'm', 'using', 'uploadify', 'to', 'upload', 'the', 'files', 'which', 'changes', 'the', 'mime', 'type', 'and', 'gives', 'a', 'text', 'octal', 'or', 'something', 'as', 'the', 'mime', 'type', 'no', 'matter', 'which', 'file', 'type', 'you', 'upload', 'p', 'p', 'is', 'there', 'a', 'way', 'to', 'check', 'if', 'the', 'uploaded', 'file', 'is', 'an', 'image', 'apart', 'from', 'checking', 'the', 'file', 'extension', 'using', 'php', 'p'], BodyLength=83, NumParagraphs=2, NumLinks=0, NumFeatures=DenseVector([83.0, 2.0, 0.0]), ScaledNumFeatures=DenseVector([0.9997, 0.0241, 0.0]), ScaledNumFeatures2=DenseVector([0.4325, 0.7037, 0.0])),\n",
       " Row(Body='<p>In my favorite editor (vim), I regularly use ctrl-w to execute a certain action. Now, it quite often happens to me that firefox is the active window (on windows) while I still look at vim (thinking vim is the active window) and press ctrl-w which closes firefox. This is not what I want. Is there a way to stop ctrl-w from closing firefox?</p>\\n\\n<p>Rene</p>\\n', Id=2, Tags='firefox', Title='How can I prevent firefox from closing when I press ctrl-w', oneTag='firefox', words=['p', 'in', 'my', 'favorite', 'editor', 'vim', 'i', 'regularly', 'use', 'ctrl', 'w', 'to', 'execute', 'a', 'certain', 'action', 'now', 'it', 'quite', 'often', 'happens', 'to', 'me', 'that', 'firefox', 'is', 'the', 'active', 'window', 'on', 'windows', 'while', 'i', 'still', 'look', 'at', 'vim', 'thinking', 'vim', 'is', 'the', 'active', 'window', 'and', 'press', 'ctrl', 'w', 'which', 'closes', 'firefox', 'this', 'is', 'not', 'what', 'i', 'want', 'is', 'there', 'a', 'way', 'to', 'stop', 'ctrl', 'w', 'from', 'closing', 'firefox', 'p', 'p', 'rene', 'p'], BodyLength=71, NumParagraphs=2, NumLinks=0, NumFeatures=DenseVector([71.0, 2.0, 0.0]), ScaledNumFeatures=DenseVector([0.9996, 0.0282, 0.0]), ScaledNumFeatures2=DenseVector([0.3699, 0.7037, 0.0]))]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}