Lab 2 - A function for validating Elections Nova Scotia's records.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab 2 - A function for validating Elections Nova Scotia's records\n",
    "*© 2021 Colin Conrad*\n",
    "\n",
    "Welcome to Week 2 of INFO 6270, Introduction to Data Science! Last week we explored basic concepts in Python which we will draw from throughout this course. This week will explore a few more basic concepts, but this time, we will do so with a concrete goal. By the end of this week, you will know how to create a function for validating data. This is an essential skill that will help us greatly when exploring more advanced concepts in data science. This week, we will start our first case-based example using [Nova Scotia Public Housing for Seniors data](https://data.novascotia.ca/Community-Services/Public-Housing-Units-Nova-Scotia-Seniors/2d4m-9e6x). \n",
    "\n",
    "As before, much of the material that we will explored in the first 5 weeks has been covered by [Sweigart (2020)](https://automatetheboringstuff.com/), in this case Chapters 2 and 3. If you have not done the readings, you should certainly do so, because this week's challenges will build off of those concepts. \n",
    "\n",
    "**This week, we will achieve the following objectives:**\n",
    "- Specify conditional controls\n",
    "- Loop across string characters\n",
    "- Create a function for performing basic math\n",
    "- Create a function with flow controls\n",
    "- Create a function for validating string data\n",
    "\n",
    "Weekly reading: Sweigart (2020) Ch. 2 and 3. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Case: Elections and Nova Scotia Public Housing for Seniors\n",
    "Elections Officers in Nova Scotian municipalities often need to determine whether a building requires an accessible poll, which is decision made based on whether there are sufficient voters with accessibility needs living in the building. When below a 40 units, it often makes financial sense to provide transportation accommodations for people with accessibility requirements to help travel to the relevant poll, rather than pay elections officers to work at a poll for so few people. However, when over a certain threshold, it is  more affordable to provide a poll in a building or city block.\n",
    "\n",
    "Sometimes it may also make sense to move the district's voting poll to the vicinity of the seniors' residence. In order to run a public poll, the location should have public amenities, such as public water and public sewer, as well as over 100 units in the building. It is thus desirable to label buildings based on whether they require an accessible poll and whether they should be considered for a full public polling location. \n",
    "\n",
    "This is a task which could be done manually by opening the `ns_housing.csv` file and observing the relevant fields. Consider opening the file in Excel. You will quickly realize that manually investigating these cells would be frustrating and potentially very time consuming. Rather than manually reading the number for hundreds of values, we could  use Python to automate this task. Though we do not yet know how to work with CSV files in Python, we will lay the foundations of tools to automate this data entry task."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Objective 1 - Specify conditional controls\n",
    "In Week 1, we discussed one of the fundamental distinguishing factors of programming languages: the fact that they are designed to process logic. Python is thus designed to work similarly to everyday language. For those of you who took INFO 5590 and did the **_required_ readings by [Sweigart (2020)](https://automatetheboringstuff.com/chapter2/)**, you would find something oddly familiar in the text. He opens with a flow diagram expressing the logic behind whether to bring an umbrella. This is fundamentally similar to the business process model notation (BPMN) explored in that class! Figure 1 (below) outlines one of the examples that we explored.\n",
    "\n",
    "![alt text](img/lyle-clock.PNG \"BPMN example given by Lyle et al. (2018).\")\n",
    "**Figure 1 - BPMN example given by Lyle et al. (2018)**\n",
    "\n",
    "In this example, we expressed the logic behind a wake up routine. This logic is fundamentally the same logic that is used in Python. In fact, this type of logic forms the foundation of everyday language and was first outlined by [George Boole](https://en.wikipedia.org/wiki/George_Boole). Using this logic, we could construct elements of this example. Try running the code below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "alarm = \"inactive\" # a variable which specifies whether the alarm is active or inactive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if alarm == \"active\": # a condition which could be met\n",
    "    print(\"I am late!!\")\n",
    "else: # if anything else happens\n",
    "    print(\"Still sleeping...\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is an example of a conditional statement. It is conditional, in the sense that there are different expressions which are triggered depending on whether particular conditions are met. In this case, the `if alarm == \"active\"` condition triggers a different condition than when anything `else` is met. Try changing the alarm variable to `alarm = \"active\"` and see how the output changes.\n",
    "\n",
    "It is not difficult to see how conditionals can come in handy. Sweigart outlines a variety of logical conditions, which we will not get into for this course. However, just know that these adhere to the basic boolean operators: *and, or, not, if, if and only if*. These are the fundamental building blocks of Boole's logic.\n",
    "\n",
    "Similarly, if we consider today's case, we could use this type of logic to sort through some of the record data. For example, let's specify some of the data from the first cell as variables. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "name = \"Riverton Heights I\" # corresponds to NAME value\n",
    "num_residents = 20 # Residential Units\n",
    "water = True # Public WATER\n",
    "sewer = True # Public SEWER"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With this data we could create logic to identify whether these values are appropriate for a public polling location."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if num_residents > 100 and water == True and sewer == True: # tests whether the three conditions are met for a public poll\n",
    "    print(\"Appropriate for public\")\n",
    "else: # triggers if at least one of the conditions are not met\n",
    "    print(\"Not appropriate for a public polling location\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Consider changing the values of `water` to `False` and see what happens. When `water = False`, the conditions are no longer met, so the `else` conditions are triggered, which in this case prints `Not appropriate for a public polling location`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### *Challenge Question 1 (1 point)*\n",
    "Modify the code below to include a second condition which checks whether the entry would require an accessible poll. You can learn more about conditionals and `elif` control flow in this week's readings. Read the case to infer what the condition for `elif` should be. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# prints whether the poll is \"Appropriate for public\", \"Requires accessible poll\", or \"Not appropriate for a poll\".\n",
    "\n",
    "if num_residents > 100 and water == True and sewer == True: # tests whether the three conditions are met\n",
    "    print(\"Appropriate for public\")\n",
    "elif #[insert code here] # tests whether the condition is met for an accessible poll\n",
    "    print(\"Requires accessible poll\")\n",
    "else: # triggers if at least one of the conditions are not met\n",
    "    print(\"Not appropriate for a poll\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Objective 2 - Loop across string characters\n",
    "Though conditional statements are powerful expressions of logic, they are quite limited. Consider Figure 2 below. This is a business process which improves on Lyle's example (IMHO).\n",
    "\n",
    "![alt text](img/conrad-clock.PNG \"BPMN example given by Lyle et al. (2018).\")\n",
    "\n",
    "**Figure 2 - Prof. Conrad's improved smartwatch clock process**\n",
    "\n",
    "Some smart watches can detect when someone is sleeping and could theoretically be used to ensure that someone always receives an adequate night's sleep. In this example, my clock checks whether I have slept at least 8 hours and only wakes me up when necessary. With only `if` statements, we would have to write something like the following code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# this is bad code; don't program like this!\n",
    "\n",
    "hours = -1 # change this depending on the hour!\n",
    "\n",
    "if hours == 0:\n",
    "    print(\"Let Colin sleep.\")\n",
    "    hours += 1\n",
    "elif hours == 1:\n",
    "    print(\"Let Colin sleep.\")\n",
    "    hours += 1\n",
    "elif hours == 2:\n",
    "    print(\"Let Colin sleep.\")\n",
    "    hours += 1\n",
    "elif hours == 3:\n",
    "    print(\"Let Colin sleep.\")\n",
    "    hours += 1\n",
    "elif hours == 4:\n",
    "    print(\"Let Colin sleep.\")\n",
    "    hours += 1\n",
    "elif hours == 5:\n",
    "    print(\"Let Colin sleep.\")\n",
    "    hours += 1\n",
    "elif hours == 6:\n",
    "    print(\"Let Colin sleep.\")\n",
    "    hours += 1\n",
    "elif hours == 7:\n",
    "    print(\"Let Colin sleep.\")\n",
    "    hours += 1\n",
    "elif hours == 8:\n",
    "    print(\"Wake up! It's time to start the day!\")\n",
    "else:\n",
    "    print(\"You overslept!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is highly redundant and if we had to do things this way, it would not take long before you became frustrated and (likely) dropped this course. ***Fortunately*** we don't have to do this because we have *loops*. With loops, we can iterate over a condition until it changes. Consider the following `while` loop, which continues to check my alarm until the `hours == 8` condition is met."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hours = 0\n",
    "\n",
    "while hours < 8: # iterates until hours == 8\n",
    "    hours += 1\n",
    "    print(\"Let Colin sleep.\")\n",
    "    \n",
    "print(\"Wake up! It's time to start the day!\") #print only when hours == 8 or more"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loops are among the most commonly used Python control and Sweigart (2020) discusses loops quite thoroughly in Chapter 2. The most important things to remember for this course is that there are two main types: `while` loops (such as that above) which run as long as a condition is met, and `for` loops which execute for all of the items in a sequence. \n",
    "\n",
    "If we wanted to express similar output as above using a `for` loop, we would need to specify the range of hours (i.e. 0-7) for which Python would print `\"Let Colin sleep\"`. We could thus write a similar for loop using the following code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for h in range(0,8): # check for values 0,1,2,3,4,5,6, and 7\n",
    "    print(\"Let Colin Sleep.\")\n",
    "print(\"Wake up! It's time to start the day!\") # print only when all items have been cycled through"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Though both types of loops are handy, `for` loops are particularly useful when iterating across data. For example, as you likely recall from last week, one of the nice features of Python is that you can pick out characters from a string using its array sequence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "civic_address = \"43 Acadia (101-208)\"\n",
    "print(civic_address[0]) # prints the first character in the string\n",
    "print(civic_address[0:9]) #prints the first 9 characters in a string"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well, you can see where this is going. We can use loops to loop through strings, or other arrays as we will discuss next week! We can use this to create increasingly sophisticated expressions that can be used for data preparation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "civic_address = \"43 Acadia (101-208)\"\n",
    "\n",
    "for i in civic_address: # prints each subsequent character in the civic address\n",
    "    print(i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### *Challenge Question 2 (1 point)*\n",
    "In addition to looping through string characters, we can include `if` statements inside of the loop to check to see whether there are appropriate characters. It might be desirable to exclude characters such as brackets when working through a spreadsheet. The following code is parts of a `for` loop which iterates through each character in `civic_address`. Modify the lines that say `[insert code here]` such that your loop prints a `cleaned_address` that is the original string without brackets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "civic_address = \"43 Acadia (101-208)\"\n",
    "cleaned_address = \"\"\n",
    "\n",
    "for [insert code here] # iterates through each character in civic_address\n",
    "    if i != \"(\" and i != \")\": # checks to see if the character is either an opening bracket or closing bracket\n",
    "        [insert code here] # adds the character to the cleaned_address string\n",
    "\n",
    "print(cleaned_address)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Objective 3 - Create a function for performing basic math\n",
    "The programs that we write are becoming increasingly sophisticated and it is becoming increasingly tempting to repeat code over and over. Fortunately, we have functions to help with this task. As Sweigart summarizes, a function works sort of like a mini program within a program and they greatly simplify our lives. \n",
    "\n",
    "Sweigart does a great job explaining the finer details of functions, and especially does a great job explaining their three most important features: input, logic, and output. When working with data, we can take input (such as strings or integers), perform some sort of logic to them, and `return` their value. For example, if we wanted to create a function that checks whether the alarm should trigger, we could write something like the following. **Please execute the code below**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def checkAlarm(hours): # checks whether the alarm should sound\n",
    "    if hours < 8:\n",
    "        return(\"Let Colin sleep.\")\n",
    "    else: \n",
    "        return(\"Wake up! It's time to start the day!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With a function, we can simplify much of our code and make it easier to read. For example, with the `checkAlarm()` function prepared, we can now run each value a check with any value using a single line of code. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "checkAlarm(12) # change this value to whichever you would like"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When combined with loops, it opens a lot of possibilities. Consider the code below, which executes our cycle with only two lines. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(0,9):\n",
    "    print(checkAlarm(i)) # prints the result of check alarm for each i in range(0,9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When applied to our data, we can use functions to simplify our life by running the calculation for each item in the set, rather than manually calculating it each time. For instance, if we wanted to divide two cells in a csv file for each row in that file, we could create a function which is triggered for each row in that file (more on this in a few weeks)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### *Challenge Question 3 (1 point)*\n",
    "In the cell below, create a function called `unitsPerFloor` which takes in the number of units in a building and the number of floors and returns the number of units on each floor. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def unitsPerFloor(units, floors):\n",
    "    # [insert code here]\n",
    "    return() # make it return the calculated value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Objective 4 - Create a function with flow controls\n",
    "Now it's time to start bringing it all together. The next two questions are a bit more complicated and combine the concepts of flow controls, loops and functions. The following code demonstrates a function which combines flow controls and can be used to check whether the user should wake up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def checkTime(time, percent_rested):\n",
    "    if time == \"12:00\" and percent_rested >= 50: #checks whether two conditions are met\n",
    "        return(\"Wake up!\")\n",
    "    elif percent_rested == 100: # a second conditional statement\n",
    "        return(\"Wake up!\")\n",
    "    else: # triggers for all other conditions\n",
    "        return(\"Stay sleeping\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Try playing with the code below to get different outcomes from the function. Also consider modifying the function and checking differences in the result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "time = \"12:00\" # change these values to see their impact on the results!\n",
    "percent_rested = 60 # change these values to see their impact on the results!\n",
    "\n",
    "checkTime(time, percent_rested)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### *Challenge Question 4 (1 point)*\n",
    "Create a function called checkResidents which performs a check similar to what you wrote in step 1, though is an actual function. Do not copy the code from question 1 however; your function should do the following:\n",
    "- Take input from three variables: the number of residential units, whether the unit has public water, whether the unit has public sewer\n",
    "- Return (*not* print) \"Appropriate for public\" if the number of residential units exceeds 80 and the unit has public water and the unit has public sewer\n",
    "- Return \"Requires accessible poll\" if the number of residential units exceeds 50 though the conditions for \"Appropriate for public\" are not met\n",
    "- Return \"Not appropriate for a poll\" if neither of the above conditions are met\n",
    "- You should demonstrate whether you code works using the cell at the bottom of this question."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def checkResidents(res, wat, sew):\n",
    "    # [insert code here]\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Test your code here!\n",
    "Run the code below to test whether your `checkResidents()` function works. If you aren't able to get it working the first time, keep trying to change your function! Also consider trying conditions which would produce the three outcomes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "residential_units = 135 # do not change this.\n",
    "public_water = True # do not change this.\n",
    "public_sewer = False # do not change this.\n",
    "\n",
    "checkResidents(residential_units, public_water, public_sewer) # calls your function. Do not change this.\n",
    "# should return \"Requires accessible poll\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Objective 5 - Create a function for validating string data\n",
    "Good work! Let's close by bringing it all together. We already know that we can use a loop to iterate through a series of characters in a string and we now know that we can write a function that can take a string as an input. If we wanted to, we could write a loop within a function to iterate through the characters from input strings. In this last challenge question, bring these concepts together to validate the characters in the input strings, in addition to the values. With this, you have completed your first function for string validation, a skill which is widely applicable which we will come back to time and again!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### *Challenge Question 5 (1 point)*\n",
    "Create a new function called `residenceValidation()` which performs a check similar to what you wrote in step 4, though takes different input and also checks for valid characters. Your function should do the following:\n",
    "- Take input from three variables: the name of the unit, the municipality, the number of units\n",
    "- Return (*not* print!) \"Invalid Residence\" if the residence string contains any of the following characters: `(`, `)`, `_`, `&`, `@`\n",
    "- Return \"Urban Residence Poll\" if the county equals `Halifax Regional Municipality` and number of residential units exceeds 50\n",
    "- Return \"Urban Residence Without Poll\" if the county equals `Halifax Regional Municipality` and number of residential units is less than 50\n",
    "- Return \"Rural Residence Poll\" if the county is any value other than `Halifax Regional Municipality` and the number of residential unites exceeds 50\n",
    "- Return \"Rural Residence Without Poll\" if the county is any value other than `Halifax Regional Municipality` and number of residential units is less than 50\n",
    "- You should demonstrate whether you code works using the cell at the bottom of this question."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## [write your function here]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Test your code here!\n",
    "There are a few test samples below. Call your `residenceValidation()` function in the cells below to test whether it works. If you aren't able to get it working the first time, keep trying to change your function! Also consider trying to write your own tests to see if it works properly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Sample Test 1 - Should return \"Urban Residence Poll\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "name = \"Alderney Manor\"\n",
    "municipality = \"Halifax Regional Municipality\"\n",
    "residential_units = 199 \n",
    "\n",
    "# call your function here!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Sample Test 2 - Should return \"Invalid String\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "name = \"Eastwood Manor (GR)\"\n",
    "municipality = \"Halifax Regional Municipality\"\n",
    "residential_units = 137\n",
    "\n",
    "# call your function here!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Sample Test 3 - Should return \"Rural Residence Without Poll\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "name = \"Sunnybrook Manor\"\n",
    "municipality = \"East Hants\"\n",
    "residential_units = 20\n",
    "\n",
    "# call your function here!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References\n",
    "Nova Scotia Department of Community Services (15 February 2019). Public Hosing Units - Nova Scotia Seniors. Retrieved from: https://data.novascotia.ca/Community-Services/Public-Housing-Units-Nova-Scotia-Seniors/2d4m-9e6x"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}