understanding-lists-manipulating-lines.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Understanding lists and manipulating lines\n",
    "\n",
    "By [Allison Parrish](http://www.decontextualize.com/)\n",
    "\n",
    "In this tutorial, I explain how the list data structure works in Python. After going over the basics, I'll show you how to use list comprehensions as a powerful and succinct method to poetically manipulate lines and words from a text.\n",
    "\n",
    "## Lists: the basics\n",
    "\n",
    "A list is a type of value in Python that represents a sequence of values. The list is a very common and versatile data structure in Python and is used frequently to represent (among other things) tabular data and words in a text. Here's how you write one out in Python:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[5, 10, 15, 20, 25, 30]"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[5, 10, 15, 20, 25, 30]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That is: a left square bracket, followed by a series of comma-separated expressions, followed by a right square bracket. Items in a list don't have to be values; they can be more complex expressions as well. Python will evaluate those expressions and put them in the list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[5, 10, 15, 20, 25, 30]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[5, 2*5, 3*5, 4*5, 5*5, 6*5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lists can have an arbitrary number of values. Here's a list with only one value in it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[5]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And here's a list with no values in it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's what happens when we ask Python what type of value a list is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "list"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type([1, 2, 3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's a value of type `list`.\n",
    "\n",
    "Like any other kind of Python value, you can assign a list to a variable:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "my_numbers = [5, 10, 15, 20, 25, 30]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Getting values out of lists\n",
    "\n",
    "Once we have a list, we might want to get values *out* of the list. You can write a Python expression that evaluates to a particular value in a list using square brackets to the right of your list, with a number representing which value you want, numbered from the beginning (the left-hand side) of the list. Here's an example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "15"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[5, 10, 15, 20][2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we were to say this expression out loud, it might read, \"I have a list of four things: 5, 10, 15, 20. Give me back the second item in the list.\" Python evaluates that expression to `15`, the second item in the list.\n",
    "\n",
    "Here's what it looks like to use this indexing notation on a list stored in a variable:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "15"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_numbers[2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The second item? Am I seeing things. 15 is clearly the third item in the list.\n",
    "\n",
    "You're right---good catch. But for reasons too complicated to go into here, Python (along with many other programming languages!) starts list indexes at 0, instead of 1. So what looks like the third element of the list to human eyes is actually the second element to Python. The first element of the list is accessed using index 0, like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[5, 10, 15, 20][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The way I like to conceptualize this is to think of list indexes not as specifying the number of the item you want, but instead specifying how \"far away\" from the beginning of the list to look for that value.\n",
    "\n",
    "If you attempt to use a value for the index of a list that is beyond the end of the list (i.e., the value you use is higher than the last index in the list), Python gives you an error:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "ename": "IndexError",
     "evalue": "list index out of range",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mIndexError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-10-72aecccf47fe>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mmy_numbers\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m47\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mIndexError\u001b[0m: list index out of range"
     ]
    }
   ],
   "source": [
    "my_numbers[47]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that while the type of a list is `list`, the type of an expression using index brackets to get an item out of the list is the type of whatever was in the list to begin with. To illustrate:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "list"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(my_numbers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "int"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(my_numbers[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Indexes can be expressions too\n",
    "\n",
    "The thing that goes inside of the index brackets doesn't have to be a number that you've just typed in there. Any Python expression that evaluates to an integer can go in there."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "25"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_numbers[2 * 2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "20"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = 3\n",
    "[5, 10, 15, 20][x]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Other operations on lists\n",
    "\n",
    "Because lists are so central to Python programming, Python includes a number of built-in functions that allow us to write expressions that evaluate to interesting facts about lists. For example, try putting a list between the parentheses of the `len()` function. It will evaluate to the number of items in the list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(my_numbers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len([20])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len([])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `in` operator checks to see if the value on the left-hand side is in the list on the right-hand side."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "3 in my_numbers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "15 in my_numbers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `max()` function will evaluate to the highest value in the list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "42"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "readings = [9, 8, 42, 3, -17, 2]\n",
    "max(readings)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "... and the `min()` function will evaluate to the lowest value in the list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-17"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "min(readings)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `sum()` function evaluates to the sum of all values in the list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "100"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum([2, 4, 6, 8, 80])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, the `sorted()` function evaluates to a copy of the list, sorted from smallest value to largest value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[-17, 2, 3, 8, 9, 42]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sorted(readings)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Negative indexes\n",
    "\n",
    "If you use `-1` as the value inside of the brackets, something interesting happens:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fib = [1, 1, 2, 3, 5]\n",
    "fib[-1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The expression evaluates to the *last* item in the list. This is essentially the same thing as the following code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fib[len(fib) - 1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "... except easier to write. In fact, you can use any negative integer in the index brackets, and Python will count that many items from the end of the list, and evaluate the expression to that item."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fib[-3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the value in the brackets would \"go past\" the beginning of the list, Python will raise an error:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "ename": "IndexError",
     "evalue": "list index out of range",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mIndexError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-27-afdbe64c1786>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mfib\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;36m14\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mIndexError\u001b[0m: list index out of range"
     ]
    }
   ],
   "source": [
    "fib[-14]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generating lists with `range()`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The expression `list(range(n))` returns a list from 0 up to (but not including) `n`. This is helpful when you just want numbers in a sequence:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(range(10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can specify where the list should start and end by supplying two parameters to the call to `range`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[-10, -9, -8, -7, -6, -5, -4, -3, -2, -1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9]"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(range(-10, 10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## List slices\n",
    "\n",
    "The index bracket syntax explained above allows you to write an expression that evaluates to a particular item in a list, based on its position in the list. Python also has a powerful way for you to write expressions that return a *section* of a list, starting from a particular index and ending with another index. In Python parlance we'll call this section a *slice*.\n",
    "\n",
    "Writing an expression to get a slice of a list looks a lot like writing an expression to get a single value. The difference is that instead of putting one number between square brackets, we put *two* numbers, separated by a colon. The first number tells Python where to begin the slice, and the second number tells Python where to end it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[5, 6, 10]"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[4, 5, 6, 10, 12, 15][1:4]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the value after the colon specifies at which index the slice should end, but the slice does *not* include the value at that index. (You can tell how long the slice will be by subtracting the value before the colon from the value after it.)\n",
    "\n",
    "Also note that---as always!---any expression that evaluates to an integer can be used for either value in the brackets. For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[10, 12]"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = 3\n",
    "[4, 5, 6, 10, 12, 15][x:x+2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, note that the type of a slice is `list`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "list"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(my_numbers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "list"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(my_numbers[1:4])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Omitting slice values\n",
    "\n",
    "Because it's so common to use the slice syntax to get a list that is either a slice starting at the beginning of the list or a slice ending at the end of the list, Python has a special shortcut. Instead of writing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[5, 10, 15]"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_numbers[0:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can leave out the `0` and write this instead:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[5, 10, 15]"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_numbers[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Likewise, if you wanted a slice that starts at index 4 and goes to the end of the list, you might write:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[25, 30]"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_numbers[4:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Getting the last two items in `my_numbers`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[5, 10]"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_numbers[:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Negative index values in slices\n",
    "\n",
    "Now for some tricky stuff: You can use negative index values in slice brackets as well! For example, to get a slice of a list from the fourth-to-last element of the list up to (but not including) the second-to-last element of the list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[15, 20]"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_numbers[-4:-2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get the last three elements of the list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[5, 10, 15]"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_numbers[:-3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All items from `my_numbers` from the third item from the end of the list upto the end of the list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[20, 25, 30]"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_numbers[-3:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Strings and lists\n",
    "\n",
    "Strings and lists share a lot of similarities! The same square bracket slice and index syntax works on strings the same way it works on lists:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "message = \"importantly\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'m'"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "message[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'l'"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "message[-2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'ant'"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "message[-5:-2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Weirdly, `max()` and `min()` also work on strings... they just evaluate to the letter that comes latest and earliest in alphabetical order (respectively):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'y'"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "max(message)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'a'"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "min(message)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can turn a string into a list of its component characters by passing it to `list()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['i', 'm', 'p', 'o', 'r', 't', 'a', 'n', 't', 'l', 'y']"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(message)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['我', '爱', '猫', '！', '😻']"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(\"我爱猫！😻\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The letters in a string in alphabetical order:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['a', 'i', 'l', 'm', 'n', 'o', 'p', 'r', 't', 't', 'y']"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sorted(list(message))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## List comprehensions: Applying transformations to lists\n",
    "\n",
    "A very common task in both data analysis and computer programming is applying some operation to every item in a list (e.g., scaling the numbers in a list by a fixed factor), or to create a copy of a list with only those items that match a particular criterion (e.g., eliminating values that fall below a certain threshold). Python has a succinct syntax, called a *list comprehension*, which allows you to easily write expressions that transform and filter lists.\n",
    "\n",
    "A list comprehension has a few parts:\n",
    "\n",
    "- a *source list*, or the list whose values will be transformed or filtered;\n",
    "- a *predicate expression*, to be evaluated for every item in the list; \n",
    "- (optionally) a *membership expression* that determines whether or not an item in the source list will be included in the result of evaluating the list comprehension, based on whether the expression evaluates to `True` or `False`; and\n",
    "- a *temporary variable name* by which each value from the source list will be known in the predicate expression and membership expression.\n",
    "\n",
    "These parts are arranged like so:\n",
    "\n",
    "> `[` *predicate expression* `for` *temporary variable name* `in` *source list* `if` *membership expression* `]`\n",
    "\n",
    "The words `for`, `in`, and `if` are a part of the syntax of the expression. They don't mean anything in particular (and in fact, they do completely different things in other parts of the Python language). You just have to spell them right and put them in the right place in order for the list comprehension to work.\n",
    "\n",
    "Here's an example, returning the squares of integers zero up to ten. First, we'll create a list `to_ten` that contains the range of numbers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "to_ten = list(range(10))\n",
    "to_ten"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[x * x for x in to_ten]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the example above, `x*x` is the predicate expression; `x` is the temporary variable name; and `[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]` is the source list. There's no membership expression in this example, so we omit it (and the word `if`).\n",
    "\n",
    "There's nothing special about the variable `x`; it's just a name that we chose. We could easily choose any other temporary variable name, as long as we use it in the predicate expression as well. Below, I use the name of one of my cats as the temporary variable name, and the expression evaluates the same way it did with `x`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[shumai * shumai for shumai in to_ten]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that the type of the value that a list comprehension evaluates to is itself type `list`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "list"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type([x * x for x in to_ten])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `source` doesn't have to be a variable that contains a list. It can be *any expression that evaluates to a list*, or in fact any expression that evaluates to an [iterable](https://docs.python.org/3/glossary.html#term-iterable). For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 1, 1, 4, 4, 9, 9]"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[x * x for x in [0, -1, 1, -2, 2, -3, 3]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "... or:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[x * x for x in range(10)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've used the expression `x * x` as the predicate expression in the examples above, but you can use any expression you want. For example, to scale the values of a list by 0.5:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.0, 0.5, 1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5]"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[x * 0.5 for x in range(10)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In fact, the expression in the list comprehension can just be the temporary variable itself, in which case the list comprehension will simply evaluate to a copy of the original list: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[x for x in range(10)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You don't technically even need to use the temporary variable in the predicate expression:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[42, 42, 42, 42, 42, 42, 42, 42, 42, 42]"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[42 for x in range(10)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Bonus exercise: Write a list comprehension for the list `range(5)` that evaluates to a list where every value has been multiplied by two (i.e., the expression would evaluate to `[0, 2, 4, 6, 8]`). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The membership expression\n",
    "\n",
    "As indicated above, you can include an expression at the end of the list comprehension to determine whether or not the item in the source list will be evaluated and included in the resulting list. One way, for example, of including only those values from the source list that are greater than or equal to five:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[25, 36, 49, 64, 81]"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[x*x for x in range(10) if x >= 5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Splitting strings into lists\n",
    "\n",
    "The `split()` method is a funny thing you can do with a string to transform it into a list. If you have an expression that evaluates to a string, you can put `.split()` right after it, and Python will evaluate the whole expression to mean \"take this string, and 'split' it on white space, giving me a list of strings with the remaining parts.\" For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['this', 'is', 'a', 'test']"
      ]
     },
     "execution_count": 129,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"this is a test\".split()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notably, while the `type` of a string is `str`, the type of the result of `split()` is `list`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "list"
      ]
     },
     "execution_count": 130,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(\"this is a test\".split())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the string in question has some delimiter in it other than whitespace that we want to use to separate the fields in the resulting list, we can put a string with that delimiter inside the parentheses of the `split()` method. Maybe you can tell where I'm going with this at this point!\n",
    "\n",
    "### From string to list of numbers: an example\n",
    "\n",
    "For example, I happen to have here a string that represents the total points scored by LeBron James in each of his NBA games in the 2013-2014 regular season.\n",
    "\n",
    "> 17,25,26,25,35,18,25,33,39,30,13,21,22,35,28,27,26,23,21,21,24,17,25,30,24,18,38,19,33,26,26,15,30,32,32,36,25,21,34,30,29,27,18,34,30,24,31,13,37,36,42,33,31,20,61,22,19,17,23,19,21,24,43,15,25,32,38,17,13,32,17,34,38,29,37,36,27\n",
    "\n",
    "You can either cut-and-paste this string from the notes, or see a file on github with these values [here](https://gist.githubusercontent.com/aparrish/56ea528159c97b085a34/raw/8406bdd866101cf64347349558d9a806c82aceb7/scores.txt).\n",
    "\n",
    "Now if I just cut-and-pasted this string into a variable and tried to call list functions on it, I wouldn't get very helpful responses:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'9'"
      ]
     },
     "execution_count": 131,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "raw_str = \"17,25,26,25,35,18,25,33,39,30,13,21,22,35,28,27,26,23,21,21,24,17,25,30,24,18,38,19,33,26,26,15,30,32,32,36,25,21,34,30,29,27,18,34,30,24,31,13,37,36,42,33,31,20,61,22,19,17,23,19,21,24,43,15,25,32,38,17,13,32,17,34,38,29,37,36,27\"\n",
    "max(raw_str)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is wrong—we know that LeBron James scored more than nine points in his highest scoring game. The `max()` function clearly does strange things when we give it a string instead of a list. The reason for this is that all Python knows about a string is that it's a *series of characters*. It's easy for a human to look at this string and think, \"Hey, that's a list of numbers!\" But Python doesn't know that. We have to explicitly \"translate\" that string into the kind of data we want Python to treat it as.\n",
    "\n",
    "> Bonus advanced exercise: Take a guess as to why, specifically, Python evaluates `max(raw_str)` to `9`. Hint: what's the result of `type(max(raw_str))`?\n",
    "\n",
    "What we want to do, then, is find some way to convert this string that *represents* integer values into an actual Python list of integer values. We'll start by splitting this string into a list, using the `split()` method, passing `\",\"` as a parameter so it splits on commas instead of on whitespace:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['17',\n",
       " '25',\n",
       " '26',\n",
       " '25',\n",
       " '35',\n",
       " '18',\n",
       " '25',\n",
       " '33',\n",
       " '39',\n",
       " '30',\n",
       " '13',\n",
       " '21',\n",
       " '22',\n",
       " '35',\n",
       " '28',\n",
       " '27',\n",
       " '26',\n",
       " '23',\n",
       " '21',\n",
       " '21',\n",
       " '24',\n",
       " '17',\n",
       " '25',\n",
       " '30',\n",
       " '24',\n",
       " '18',\n",
       " '38',\n",
       " '19',\n",
       " '33',\n",
       " '26',\n",
       " '26',\n",
       " '15',\n",
       " '30',\n",
       " '32',\n",
       " '32',\n",
       " '36',\n",
       " '25',\n",
       " '21',\n",
       " '34',\n",
       " '30',\n",
       " '29',\n",
       " '27',\n",
       " '18',\n",
       " '34',\n",
       " '30',\n",
       " '24',\n",
       " '31',\n",
       " '13',\n",
       " '37',\n",
       " '36',\n",
       " '42',\n",
       " '33',\n",
       " '31',\n",
       " '20',\n",
       " '61',\n",
       " '22',\n",
       " '19',\n",
       " '17',\n",
       " '23',\n",
       " '19',\n",
       " '21',\n",
       " '24',\n",
       " '43',\n",
       " '15',\n",
       " '25',\n",
       " '32',\n",
       " '38',\n",
       " '17',\n",
       " '13',\n",
       " '32',\n",
       " '17',\n",
       " '34',\n",
       " '38',\n",
       " '29',\n",
       " '37',\n",
       " '36',\n",
       " '27']"
      ]
     },
     "execution_count": 132,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "str_list = raw_str.split(\",\")\n",
    "str_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks good so far. What does `max()` have to say about it?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'61'"
      ]
     },
     "execution_count": 133,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "max(str_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This.. works. (But only by accident—see below.) But what if we wanted to find the total number of points scored by LBJ? We should be able to do something like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "unsupported operand type(s) for +: 'int' and 'str'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-134-537ec6b789f1>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0msum\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstr_list\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m: unsupported operand type(s) for +: 'int' and 'str'"
     ]
    }
   ],
   "source": [
    "sum(str_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "... but we get an error. Why this error? The reason lies in what kind of data is in our list. We can check the data type of an element of the list with the `type()` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "str"
      ]
     },
     "execution_count": 135,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(str_list[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A-ha! The type is `str`. So the error message we got before (`unsupported operand type(s) for +: 'int' and 'str'`) is Python's way of telling us, \"You gave me a list of strings and then asked me to add them all together. I'm not sure what I can do for you.\"\n",
    "\n",
    "So there's one step left in our process of \"converting\" our \"raw\" string, consisting of comma-separated numbers, into a list of numbers. What we have is a list of strings; what we want is a list of numbers. Fortunately, we know how to write an expression to transform one list into another list, applying an expression to each member of the list along the way—it's called a list comprehension. Equally fortunately, we know how to write an expression that converts a string representing an integer into an actual integer (`int()`). Here's how to write that expression:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[17,\n",
       " 25,\n",
       " 26,\n",
       " 25,\n",
       " 35,\n",
       " 18,\n",
       " 25,\n",
       " 33,\n",
       " 39,\n",
       " 30,\n",
       " 13,\n",
       " 21,\n",
       " 22,\n",
       " 35,\n",
       " 28,\n",
       " 27,\n",
       " 26,\n",
       " 23,\n",
       " 21,\n",
       " 21,\n",
       " 24,\n",
       " 17,\n",
       " 25,\n",
       " 30,\n",
       " 24,\n",
       " 18,\n",
       " 38,\n",
       " 19,\n",
       " 33,\n",
       " 26,\n",
       " 26,\n",
       " 15,\n",
       " 30,\n",
       " 32,\n",
       " 32,\n",
       " 36,\n",
       " 25,\n",
       " 21,\n",
       " 34,\n",
       " 30,\n",
       " 29,\n",
       " 27,\n",
       " 18,\n",
       " 34,\n",
       " 30,\n",
       " 24,\n",
       " 31,\n",
       " 13,\n",
       " 37,\n",
       " 36,\n",
       " 42,\n",
       " 33,\n",
       " 31,\n",
       " 20,\n",
       " 61,\n",
       " 22,\n",
       " 19,\n",
       " 17,\n",
       " 23,\n",
       " 19,\n",
       " 21,\n",
       " 24,\n",
       " 43,\n",
       " 15,\n",
       " 25,\n",
       " 32,\n",
       " 38,\n",
       " 17,\n",
       " 13,\n",
       " 32,\n",
       " 17,\n",
       " 34,\n",
       " 38,\n",
       " 29,\n",
       " 37,\n",
       " 36,\n",
       " 27]"
      ]
     },
     "execution_count": 136,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[int(x) for x in str_list]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's double-check that the values in this list are, in fact, integers, by spot-checking the first item in the list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "int"
      ]
     },
     "execution_count": 137,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type([int(x) for x in str_list][0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hey, voila! Now we'll assign that list to a variable, for the sake of convenience, and then check to see if `sum()` works how we expect it to."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2089"
      ]
     },
     "execution_count": 139,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "int_list = [int(x) for x in str_list]\n",
    "sum(int_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wow! 2089 points in one season! Good work, King James."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Join: Making strings from lists\n",
    "\n",
    "Once we've created a list of words, it's a common task to want to take that\n",
    "list and \"glue\" it back together, so it's a single string again, instead of\n",
    "a list. So, for example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'hydrogen, and helium, and lithium, and beryllium, and boron'"
      ]
     },
     "execution_count": 105,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "element_list = [\"hydrogen\", \"helium\", \"lithium\", \"beryllium\", \"boron\"]\n",
    "glue = \", and \"\n",
    "glue.join(element_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `.join()` method needs a \"glue\" string to the left of it---this is the\n",
    "string that will be placed in between the list elements. In the parentheses\n",
    "to the right, you need to put an expression that evaluates to a list. Very\n",
    "frequently with `.join()`, programmers don't bother to assign the \"glue\"\n",
    "string to a variable first, so you end up with code that looks like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'this is a test'"
      ]
     },
     "execution_count": 106,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "words = [\"this\", \"is\", \"a\", \"test\"]\n",
    "\" \".join(words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we're working with `.split()` and `.join()`, our workflow usually looks\n",
    "something like this:\n",
    "\n",
    "1. Split a string to get a list of units (usually words).\n",
    "2. Use some of the list operations discussed above to modify or slice the list.\n",
    "3. Join that list back together into a string.\n",
    "4. Do something with that string (e.g., print it out).\n",
    "\n",
    "With this in mind, here's a program that splits a string into words, randomizes the order of the words, then prints out the results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'a it was night dark stormy and'"
      ]
     },
     "execution_count": 107,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text = \"it was a dark and stormy night\"\n",
    "words = text.split()\n",
    "random.shuffle(words)\n",
    "' '.join(words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lists and randomness\n",
    "\n",
    "Python's `random` library provides several helpful functions for performing\n",
    "chance operations on lists. The first is `shuffle`, which takes a list and\n",
    "randomly shuffles its contents:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['eggs', 'sugar', 'flour', 'milk']"
      ]
     },
     "execution_count": 140,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import random\n",
    "ingredients = [\"flour\", \"milk\", \"eggs\", \"sugar\"]\n",
    "random.shuffle(ingredients)\n",
    "ingredients"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The second is `choice`, which returns a single random element from list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'milk'"
      ]
     },
     "execution_count": 141,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import random\n",
    "ingredients = [\"flour\", \"milk\", \"eggs\", \"sugar\"]\n",
    "random.choice(ingredients)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, the `sample` function returns a list of values, selected at random,\n",
    "from a list. The `sample` function takes two parameters: the first is a list,\n",
    "and the second is how many items should be in the resulting list of randomly\n",
    "selected values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['flour', 'milk']"
      ]
     },
     "execution_count": 142,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import random\n",
    "ingredients = [\"flour\", \"milk\", \"eggs\", \"sugar\"]\n",
    "random.sample(ingredients, 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Text files and lists of lines\n",
    "\n",
    "The `open()` function allows you to read text from a file. When used as the source in a list comprehension, the predicate expression will be evaluated for *each line* of text in the file. For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Rose, harsh rose, \\n',\n",
       " 'marred and with stint of petals, \\n',\n",
       " 'meagre flower, thin, \\n',\n",
       " 'spare of leaf,\\n',\n",
       " '\\n',\n",
       " 'more precious \\n',\n",
       " 'than a wet rose \\n',\n",
       " 'single on a stem -- \\n',\n",
       " 'you are caught in the drift.\\n',\n",
       " '\\n',\n",
       " 'Stunted, with small leaf, \\n',\n",
       " 'you are flung on the sand, \\n',\n",
       " 'you are lifted \\n',\n",
       " 'in the crisp sand \\n',\n",
       " 'that drives in the wind.\\n',\n",
       " '\\n',\n",
       " 'Can the spice-rose \\n',\n",
       " 'drip such acrid fragrance \\n',\n",
       " 'hardened in a leaf?\\n']"
      ]
     },
     "execution_count": 143,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[line for line in open(\"sea_rose.txt\")]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What we have here is a *list of strings*. Each element in the list corresponds to a single line from the text file.\n",
    "\n",
    "Notice that you see the `\\n` character at the end of each string; this is Python letting you know that there's a character in the file that indicates where the linebreaks should occur. By default, Python doesn't strip this character out. To make our data a little bit cleaner, let's use the `.strip()` method in the predicate expression to get rid of the newline character (and any other accompanying whitespace characters):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 194,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Rose, harsh rose,',\n",
       " 'marred and with stint of petals,',\n",
       " 'meagre flower, thin,',\n",
       " 'spare of leaf,',\n",
       " '',\n",
       " 'more precious',\n",
       " 'than a wet rose',\n",
       " 'single on a stem --',\n",
       " 'you are caught in the drift.',\n",
       " '',\n",
       " 'Stunted, with small leaf,',\n",
       " 'you are flung on the sand,',\n",
       " 'you are lifted',\n",
       " 'in the crisp sand',\n",
       " 'that drives in the wind.',\n",
       " '',\n",
       " 'Can the spice-rose',\n",
       " 'drip such acrid fragrance',\n",
       " 'hardened in a leaf?']"
      ]
     },
     "execution_count": 194,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[line.strip() for line in open(\"sea_rose.txt\")]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Assigning this to a variable, we can do things like get the lines of the poem from index 3 up to index 9:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 195,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "poem = [line.strip() for line in open(\"sea_rose.txt\")]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 196,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['spare of leaf,',\n",
       " '',\n",
       " 'more precious',\n",
       " 'than a wet rose',\n",
       " 'single on a stem --',\n",
       " 'you are caught in the drift.']"
      ]
     },
     "execution_count": 196,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "poem[3:9]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or we can get a few lines at random:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 197,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['more precious', 'you are caught in the drift.', 'Can the spice-rose']"
      ]
     },
     "execution_count": 197,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "random.sample(poem, 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or sort the poem in alphabetical order:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 198,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['',\n",
       " '',\n",
       " '',\n",
       " 'Can the spice-rose',\n",
       " 'Rose, harsh rose,',\n",
       " 'Stunted, with small leaf,',\n",
       " 'drip such acrid fragrance',\n",
       " 'hardened in a leaf?',\n",
       " 'in the crisp sand',\n",
       " 'marred and with stint of petals,',\n",
       " 'meagre flower, thin,',\n",
       " 'more precious',\n",
       " 'single on a stem --',\n",
       " 'spare of leaf,',\n",
       " 'than a wet rose',\n",
       " 'that drives in the wind.',\n",
       " 'you are caught in the drift.',\n",
       " 'you are flung on the sand,',\n",
       " 'you are lifted']"
      ]
     },
     "execution_count": 198,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sorted(poem)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The resulting list variable can itself be used as the source expression in other list comprehensions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Rose,',\n",
       " 'marre',\n",
       " 'meagr',\n",
       " 'spare',\n",
       " '',\n",
       " 'more ',\n",
       " 'than ',\n",
       " 'singl',\n",
       " 'you a',\n",
       " '',\n",
       " 'Stunt',\n",
       " 'you a',\n",
       " 'you a',\n",
       " 'in th',\n",
       " 'that ',\n",
       " '',\n",
       " 'Can t',\n",
       " 'drip ',\n",
       " 'harde']"
      ]
     },
     "execution_count": 148,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[line[:5] for line in poem]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Transforming lines of text\n",
    "\n",
    "Wait, how did I do that thing with that poem, where the letters are all weird? It's like this weird pomo l=a=n=g=u=a=g=e poetry now. Completely unpublishable, I'll get kicked right out of the Iowa Writer's Workshop. Very cool.\n",
    "\n",
    "It turns out you can make changes to the predicate expression in order to make changes to the way the text looks in the output. We're modifying the text by transforming the strings. There are a handful of really easy things you can do to strings of characters in Python to make the text do weird and interesting things. I'm going to show you a few.\n",
    "\n",
    "First, I'm going to make a variable called `poem` and assign to it the result of reading in that Robert Frost poem. The road one where he decides to take a road, but not another road, and it is very momentous."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 201,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "poem = [line.strip() for line in open(\"frost.txt\")]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What we want to do now is write a list comprehension that transforms the lines in the poem somehow. The key to this is to change the predicate expression in a list comprehension. The simplest possible text transformation is nothing at all: just make a new list that looks like the old list in every way."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Two roads diverged in a yellow wood,',\n",
       " 'And sorry I could not travel both',\n",
       " 'And be one traveler, long I stood',\n",
       " 'And looked down one as far as I could',\n",
       " 'To where it bent in the undergrowth;',\n",
       " '',\n",
       " 'Then took the other, as just as fair,',\n",
       " 'And having perhaps the better claim,',\n",
       " 'Because it was grassy and wanted wear;',\n",
       " 'Though as for that the passing there',\n",
       " 'Had worn them really about the same,',\n",
       " '',\n",
       " 'And both that morning equally lay',\n",
       " 'In leaves no step had trodden black.',\n",
       " 'Oh, I kept the first for another day!',\n",
       " 'Yet knowing how way leads on to way,',\n",
       " 'I doubted if I should ever come back.',\n",
       " '',\n",
       " 'I shall be telling this with a sigh',\n",
       " 'Somewhere ages and ages hence:',\n",
       " 'Two roads diverged in a wood, and I—',\n",
       " 'I took the one less travelled by,',\n",
       " 'And that has made all the difference.']"
      ]
     },
     "execution_count": 155,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[line for line in poem]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That list comprehension basically translates to the following: \"Hey python, hey yes you, python! Take a look at the list of strings in the variable poem that I defined earlier. I want you to make a new list, and here's how that new list should look: for every item of that list—let's call the item line—put an item with whatever that line is into the new list.\" Python: \"So, uh, make a copy of the list?\" You: \"Yeah I guess basically.\"\n",
    "\n",
    "Another simple transformation is to not do anything with the data in the line at all, and have Python put another string altogether into the new list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[\"I'm Robert Frost, howdy howdy howdy\",\n",
       " \"I'm Robert Frost, howdy howdy howdy\",\n",
       " \"I'm Robert Frost, howdy howdy howdy\",\n",
       " \"I'm Robert Frost, howdy howdy howdy\",\n",
       " \"I'm Robert Frost, howdy howdy howdy\",\n",
       " \"I'm Robert Frost, howdy howdy howdy\",\n",
       " \"I'm Robert Frost, howdy howdy howdy\",\n",
       " \"I'm Robert Frost, howdy howdy howdy\",\n",
       " \"I'm Robert Frost, howdy howdy howdy\",\n",
       " \"I'm Robert Frost, howdy howdy howdy\",\n",
       " \"I'm Robert Frost, howdy howdy howdy\",\n",
       " \"I'm Robert Frost, howdy howdy howdy\",\n",
       " \"I'm Robert Frost, howdy howdy howdy\",\n",
       " \"I'm Robert Frost, howdy howdy howdy\",\n",
       " \"I'm Robert Frost, howdy howdy howdy\",\n",
       " \"I'm Robert Frost, howdy howdy howdy\",\n",
       " \"I'm Robert Frost, howdy howdy howdy\",\n",
       " \"I'm Robert Frost, howdy howdy howdy\",\n",
       " \"I'm Robert Frost, howdy howdy howdy\",\n",
       " \"I'm Robert Frost, howdy howdy howdy\",\n",
       " \"I'm Robert Frost, howdy howdy howdy\",\n",
       " \"I'm Robert Frost, howdy howdy howdy\",\n",
       " \"I'm Robert Frost, howdy howdy howdy\"]"
      ]
     },
     "execution_count": 156,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[\"I'm Robert Frost, howdy howdy howdy\" for line in poem]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Neither of these are interesting from anything other than a theoretical perspective, which means if you're a humanities scholar or something you can just stop here and start typing up your monograph on conceptions of identity and iteration in algorithmic media. But as for me and my students, we are ARTISTS and ENGINEERS and it's important that we CHANGE THE WORLD and SHOW RESULTS."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### String methods in the predicate expression\n",
    "\n",
    "Recall from the tutorial on strings that string expressions in Python have a number of methods that can be called on them that return a copy of the string with some transformation applied, like `.lower()` (converts the string to lower case) or `.replace()` (replaces matching substrings with some other string). We can use these *in the predicate expression* to effect that transformation on every item in the list. To make every string in the upper case, for example, call the .upper() method on the temporary variable line. This makes Frost look really mad:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 166,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['TWO ROADS DIVERGED IN A YELLOW WOOD,',\n",
       " 'AND SORRY I COULD NOT TRAVEL BOTH',\n",
       " 'AND BE ONE TRAVELER, LONG I STOOD',\n",
       " 'AND LOOKED DOWN ONE AS FAR AS I COULD',\n",
       " 'TO WHERE IT BENT IN THE UNDERGROWTH;',\n",
       " '',\n",
       " 'THEN TOOK THE OTHER, AS JUST AS FAIR,',\n",
       " 'AND HAVING PERHAPS THE BETTER CLAIM,',\n",
       " 'BECAUSE IT WAS GRASSY AND WANTED WEAR;',\n",
       " 'THOUGH AS FOR THAT THE PASSING THERE',\n",
       " 'HAD WORN THEM REALLY ABOUT THE SAME,',\n",
       " '',\n",
       " 'AND BOTH THAT MORNING EQUALLY LAY',\n",
       " 'IN LEAVES NO STEP HAD TRODDEN BLACK.',\n",
       " 'OH, I KEPT THE FIRST FOR ANOTHER DAY!',\n",
       " 'YET KNOWING HOW WAY LEADS ON TO WAY,',\n",
       " 'I DOUBTED IF I SHOULD EVER COME BACK.',\n",
       " '',\n",
       " 'I SHALL BE TELLING THIS WITH A SIGH',\n",
       " 'SOMEWHERE AGES AND AGES HENCE:',\n",
       " 'TWO ROADS DIVERGED IN A WOOD, AND I—',\n",
       " 'I TOOK THE ONE LESS TRAVELLED BY,',\n",
       " 'AND THAT HAS MADE ALL THE DIFFERENCE.']"
      ]
     },
     "execution_count": 166,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[line.upper() for line in poem]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And `.replace()` is a fun one. You need to put two comma-separated strings between the parentheses. Python will replace every occurrence of the first string with the second string. To make our poem more colloquial, for example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Two roads diverged in a yellow wood,',\n",
       " 'And sorry my dude could not travel both',\n",
       " 'And be one traveler, long my dude stood',\n",
       " 'And looked down one as far as my dude could',\n",
       " 'To where it bent in the undergrowth;',\n",
       " '',\n",
       " 'Then took the other, as just as fair,',\n",
       " 'And having perhaps the better claim,',\n",
       " 'Because it was grassy and wanted wear;',\n",
       " 'Though as for that the passing there',\n",
       " 'Had worn them really about the same,',\n",
       " '',\n",
       " 'And both that morning equally lay',\n",
       " 'my duden leaves no step had trodden black.',\n",
       " 'Oh, my dude kept the first for another day!',\n",
       " 'Yet knowing how way leads on to way,',\n",
       " 'my dude doubted if my dude should ever come back.',\n",
       " '',\n",
       " 'my dude shall be telling this with a sigh',\n",
       " 'Somewhere ages and ages hence:',\n",
       " 'Two roads diverged in a wood, and my dude—',\n",
       " 'my dude took the one less travelled by,',\n",
       " 'And that has made all the difference.']"
      ]
     },
     "execution_count": 161,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[line.replace(\"I\", \"my dude\") for line in poem]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you have ever wondered, \"What would this roady poem sound like if you tickled Robert Frost while he read it aloud,\" then you're in luck because Python has answered that question."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 170,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Two roaheHEEhaHAhads diverged in aheHEEhaHAha yellow wood,',\n",
       " 'And sorry I could not traheHEEhaHAhavel both',\n",
       " 'And be one traheHEEhaHAhaveler, long I stood',\n",
       " 'And looked down one aheHEEhaHAhas faheHEEhaHAhar aheHEEhaHAhas I could',\n",
       " 'To where it bent in the undergrowth;',\n",
       " '',\n",
       " 'Then took the other, aheHEEhaHAhas just aheHEEhaHAhas faheHEEhaHAhair,',\n",
       " 'And haheHEEhaHAhaving perhaheHEEhaHAhaps the better claheHEEhaHAhaim,',\n",
       " 'BecaheHEEhaHAhause it waheHEEhaHAhas graheHEEhaHAhassy aheHEEhaHAhand waheHEEhaHAhanted weaheHEEhaHAhar;',\n",
       " 'Though aheHEEhaHAhas for thaheHEEhaHAhat the paheHEEhaHAhassing there',\n",
       " 'HaheHEEhaHAhad worn them reaheHEEhaHAhally aheHEEhaHAhabout the saheHEEhaHAhame,',\n",
       " '',\n",
       " 'And both thaheHEEhaHAhat morning equaheHEEhaHAhally laheHEEhaHAhay',\n",
       " 'In leaheHEEhaHAhaves no step haheHEEhaHAhad trodden blaheHEEhaHAhack.',\n",
       " 'Oh, I kept the first for aheHEEhaHAhanother daheHEEhaHAhay!',\n",
       " 'Yet knowing how waheHEEhaHAhay leaheHEEhaHAhads on to waheHEEhaHAhay,',\n",
       " 'I doubted if I should ever come baheHEEhaHAhack.',\n",
       " '',\n",
       " 'I shaheHEEhaHAhall be telling this with aheHEEhaHAha sigh',\n",
       " 'Somewhere aheHEEhaHAhages aheHEEhaHAhand aheHEEhaHAhages hence:',\n",
       " 'Two roaheHEEhaHAhads diverged in aheHEEhaHAha wood, aheHEEhaHAhand I—',\n",
       " 'I took the one less traheHEEhaHAhavelled by,',\n",
       " 'And thaheHEEhaHAhat haheHEEhaHAhas maheHEEhaHAhade aheHEEhaHAhall the difference.']"
      ]
     },
     "execution_count": 170,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[line.replace(\"a\", \"aheHEEhaHAha\") for line in poem]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `.strip()` method is helpful! We used it above to strip off whitespace, but if you give it a string as a parameter (inside the parentheses), it will remove all of the characters inside that string from the beginning and end of every line. This is a convenient way to, e.g., remove punctuation from the ends of lines:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 172,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Two roads diverged in a yellow wood',\n",
       " 'And sorry I could not travel both',\n",
       " 'And be one traveler, long I stood',\n",
       " 'And looked down one as far as I could',\n",
       " 'To where it bent in the undergrowth',\n",
       " '',\n",
       " 'Then took the other, as just as fair',\n",
       " 'And having perhaps the better claim',\n",
       " 'Because it was grassy and wanted wear',\n",
       " 'Though as for that the passing there',\n",
       " 'Had worn them really about the same',\n",
       " '',\n",
       " 'And both that morning equally lay',\n",
       " 'In leaves no step had trodden black',\n",
       " 'Oh, I kept the first for another day',\n",
       " 'Yet knowing how way leads on to way',\n",
       " 'I doubted if I should ever come back',\n",
       " '',\n",
       " 'I shall be telling this with a sigh',\n",
       " 'Somewhere ages and ages hence',\n",
       " 'Two roads diverged in a wood, and I',\n",
       " 'I took the one less travelled by',\n",
       " 'And that has made all the difference']"
      ]
     },
     "execution_count": 172,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[line.strip(\",;.!:—\") for line in poem]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use the `+` operator to build up strings from each line as well:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 174,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['☛ Two roads diverged in a yellow wood, ☚',\n",
       " '☛ And sorry I could not travel both ☚',\n",
       " '☛ And be one traveler, long I stood ☚',\n",
       " '☛ And looked down one as far as I could ☚',\n",
       " '☛ To where it bent in the undergrowth; ☚',\n",
       " '☛  ☚',\n",
       " '☛ Then took the other, as just as fair, ☚',\n",
       " '☛ And having perhaps the better claim, ☚',\n",
       " '☛ Because it was grassy and wanted wear; ☚',\n",
       " '☛ Though as for that the passing there ☚',\n",
       " '☛ Had worn them really about the same, ☚',\n",
       " '☛  ☚',\n",
       " '☛ And both that morning equally lay ☚',\n",
       " '☛ In leaves no step had trodden black. ☚',\n",
       " '☛ Oh, I kept the first for another day! ☚',\n",
       " '☛ Yet knowing how way leads on to way, ☚',\n",
       " '☛ I doubted if I should ever come back. ☚',\n",
       " '☛  ☚',\n",
       " '☛ I shall be telling this with a sigh ☚',\n",
       " '☛ Somewhere ages and ages hence: ☚',\n",
       " '☛ Two roads diverged in a wood, and I— ☚',\n",
       " '☛ I took the one less travelled by, ☚',\n",
       " '☛ And that has made all the difference. ☚']"
      ]
     },
     "execution_count": 174,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[\"☛ \" + line + \" ☚\" for line in poem]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using string slices, we can create some abstract poetry from parts of each line. Here we smoosh the first five characters of each line up against the last five characters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 191,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Two rwood,',\n",
       " 'And s both',\n",
       " 'And bstood',\n",
       " 'And lcould',\n",
       " 'To whowth;',\n",
       " '',\n",
       " 'Then fair,',\n",
       " 'And hlaim,',\n",
       " 'Becauwear;',\n",
       " 'Thougthere',\n",
       " 'Had wsame,',\n",
       " '',\n",
       " 'And by lay',\n",
       " 'In lelack.',\n",
       " 'Oh, I day!',\n",
       " 'Yet k way,',\n",
       " 'I douback.',\n",
       " '',\n",
       " 'I sha sigh',\n",
       " 'Somewence:',\n",
       " 'Two rnd I—',\n",
       " 'I tood by,',\n",
       " 'And tence.']"
      ]
     },
     "execution_count": 191,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[line[:5] + line[-5:] for line in poem]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You may find discover a desire deep inside of you to use more than one of these transformations on the predicate expression. \"Impossible,\" says a nearby moustachioed man, monocle popping from his orbital socket. But it can be done! In two ways. First, you can perform the transformation by assigning the result of one list comprehension to a variable, and then using that result in a second list comprehension. For example, to turn this poem into a telegram, we'll first convert it to upper case:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 176,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "upper_frost = [line.upper() for line in poem]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And then we'll get rid of punctuation at the end of the line:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 177,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "upper_frost_no_punct = [line.strip(\",;.!:—\") for line in upper_frost]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And then append the string STOP to the end of each line:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 180,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['TWO ROADS DIVERGED IN A YELLOW WOOD STOP',\n",
       " 'AND SORRY I COULD NOT TRAVEL BOTH STOP',\n",
       " 'AND BE ONE TRAVELER, LONG I STOOD STOP',\n",
       " 'AND LOOKED DOWN ONE AS FAR AS I COULD STOP',\n",
       " 'TO WHERE IT BENT IN THE UNDERGROWTH STOP',\n",
       " ' STOP',\n",
       " 'THEN TOOK THE OTHER, AS JUST AS FAIR STOP',\n",
       " 'AND HAVING PERHAPS THE BETTER CLAIM STOP',\n",
       " 'BECAUSE IT WAS GRASSY AND WANTED WEAR STOP',\n",
       " 'THOUGH AS FOR THAT THE PASSING THERE STOP',\n",
       " 'HAD WORN THEM REALLY ABOUT THE SAME STOP',\n",
       " ' STOP',\n",
       " 'AND BOTH THAT MORNING EQUALLY LAY STOP',\n",
       " 'IN LEAVES NO STEP HAD TRODDEN BLACK STOP',\n",
       " 'OH, I KEPT THE FIRST FOR ANOTHER DAY STOP',\n",
       " 'YET KNOWING HOW WAY LEADS ON TO WAY STOP',\n",
       " 'I DOUBTED IF I SHOULD EVER COME BACK STOP',\n",
       " ' STOP',\n",
       " 'I SHALL BE TELLING THIS WITH A SIGH STOP',\n",
       " 'SOMEWHERE AGES AND AGES HENCE STOP',\n",
       " 'TWO ROADS DIVERGED IN A WOOD, AND I STOP',\n",
       " 'I TOOK THE ONE LESS TRAVELLED BY STOP',\n",
       " 'AND THAT HAS MADE ALL THE DIFFERENCE STOP']"
      ]
     },
     "execution_count": 180,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[line + \" STOP\" for line in upper_frost_no_punct]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not bad, but sort of inconvenient! You can actually write that whole thing using one expression. Any of those weird methods (`.lower()`, `.upper()`, etc.) mentioned above can be chained: you can attach them not just to line but to any other expression you made with line. Likewise, the `+` operator can be used with line but also any expression that results from performing a transformation on line. For example, you can rewrite the three list comprehensions above using one list comprehension with chained operators:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 182,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['TWO ROADS DIVERGED IN A YELLOW WOOD STOP',\n",
       " 'AND SORRY I COULD NOT TRAVEL BOTH STOP',\n",
       " 'AND BE ONE TRAVELER, LONG I STOOD STOP',\n",
       " 'AND LOOKED DOWN ONE AS FAR AS I COULD STOP',\n",
       " 'TO WHERE IT BENT IN THE UNDERGROWTH STOP',\n",
       " ' STOP',\n",
       " 'THEN TOOK THE OTHER, AS JUST AS FAIR STOP',\n",
       " 'AND HAVING PERHAPS THE BETTER CLAIM STOP',\n",
       " 'BECAUSE IT WAS GRASSY AND WANTED WEAR STOP',\n",
       " 'THOUGH AS FOR THAT THE PASSING THERE STOP',\n",
       " 'HAD WORN THEM REALLY ABOUT THE SAME STOP',\n",
       " ' STOP',\n",
       " 'AND BOTH THAT MORNING EQUALLY LAY STOP',\n",
       " 'IN LEAVES NO STEP HAD TRODDEN BLACK STOP',\n",
       " 'OH, I KEPT THE FIRST FOR ANOTHER DAY STOP',\n",
       " 'YET KNOWING HOW WAY LEADS ON TO WAY STOP',\n",
       " 'I DOUBTED IF I SHOULD EVER COME BACK STOP',\n",
       " ' STOP',\n",
       " 'I SHALL BE TELLING THIS WITH A SIGH STOP',\n",
       " 'SOMEWHERE AGES AND AGES HENCE STOP',\n",
       " 'TWO ROADS DIVERGED IN A WOOD, AND I STOP',\n",
       " 'I TOOK THE ONE LESS TRAVELLED BY STOP',\n",
       " 'AND THAT HAS MADE ALL THE DIFFERENCE STOP']"
      ]
     },
     "execution_count": 182,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[line.upper().strip(\",;.!:—\") + \" STOP\" for line in poem]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is especially useful for multiple replacements. Here's the Swedish Chef version:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 186,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Twö rööds dövurrgurd ön ö yurllöw wööd,',\n",
       " 'And sörry I cöuld nöt trövurl böth',\n",
       " 'And bur önur trövurlurr, löng I stööd',\n",
       " 'And löökurd döwn önur ös för ös I cöuld',\n",
       " 'Tö whurrur öt burnt ön thur undurrgröwth;',\n",
       " '',\n",
       " 'Thurn töök thur öthurr, ös just ös föör,',\n",
       " 'And hövöng purrhöps thur burtturr clööm,',\n",
       " 'Burcöusur öt wös grössy önd wönturd wurör;',\n",
       " 'Thöugh ös för thöt thur pössöng thurrur',\n",
       " 'Höd wörn thurm rurölly öböut thur sömur,',\n",
       " '',\n",
       " 'And böth thöt mörnöng urquölly löy',\n",
       " 'In lurövurs nö sturp höd tröddurn blöck.',\n",
       " 'Oh, I kurpt thur först för önöthurr döy!',\n",
       " 'Yurt knöwöng höw wöy luröds ön tö wöy,',\n",
       " 'I döubturd öf I shöuld urvurr cömur böck.',\n",
       " '',\n",
       " 'I shöll bur turllöng thös wöth ö sögh',\n",
       " 'Sömurwhurrur ögurs önd ögurs hurncur:',\n",
       " 'Twö rööds dövurrgurd ön ö wööd, önd I—',\n",
       " 'I töök thur önur lurss trövurllurd by,',\n",
       " 'And thöt hös mödur öll thur döffurrurncur.']"
      ]
     },
     "execution_count": 186,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[line.replace(\"i\", \"ö\").replace(\"o\", \"ö\").replace(\"a\", \"ö\").replace(\"e\", \"ur\") for line in poem]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Filtering lines\n",
    "\n",
    "Using the membership expression of a list comprehension, we can make a list of only the lines from the text file that match particular criteria. Any of the various expressions that answer questions about strings can be used in this spot. For example, to find all of the lines of a particular length:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 205,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['And sorry I could not travel both',\n",
       " 'And be one traveler, long I stood',\n",
       " 'And both that morning equally lay',\n",
       " 'I took the one less travelled by,']"
      ]
     },
     "execution_count": 205,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[line for line in poem if len(line) == 33]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lines that have the string `travel`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 206,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['And sorry I could not travel both',\n",
       " 'And be one traveler, long I stood',\n",
       " 'I took the one less travelled by,']"
      ]
     },
     "execution_count": 206,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[line for line in poem if \"travel\" in line]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lines that start with `And`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 213,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['And sorry I could not travel both',\n",
       " 'And be one traveler, long I stood',\n",
       " 'And looked down one as far as I could',\n",
       " 'And having perhaps the better claim,',\n",
       " 'And both that morning equally lay',\n",
       " 'And that has made all the difference.']"
      ]
     },
     "execution_count": 213,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[line for line in poem if line.startswith(\"And\")]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Text files and lists of words\n",
    "\n",
    "Lines are an interesting unit to work with, especially with poetic source texts, as they give us an easy handle on large (but not too large) syntactic units. A more traditional unit of analysis is the word. Fortunately for us, getting words from a text file is (relatively) easy.\n",
    "\n",
    "Calling `open(filename).read()` will read the file `filename` into a Python string. We can then use the `.split()` method to split that string into a list of words. Without any parameters, the `.split()` method just breaks the string up into units delimited by any kind of whitespace (whether that's a space character, a tab, a newline, etc.). So, for example, to get all of the words from our Frost poem:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 214,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "frost_txt = open(\"frost.txt\").read() # evaluates to a string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 220,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "words = frost_txt.split() # split evaluates to a list of strings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 218,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Two',\n",
       " 'roads',\n",
       " 'diverged',\n",
       " 'in',\n",
       " 'a',\n",
       " 'yellow',\n",
       " 'wood,',\n",
       " 'And',\n",
       " 'sorry',\n",
       " 'I',\n",
       " 'could',\n",
       " 'not',\n",
       " 'travel',\n",
       " 'both',\n",
       " 'And',\n",
       " 'be',\n",
       " 'one',\n",
       " 'traveler,',\n",
       " 'long',\n",
       " 'I',\n",
       " 'stood',\n",
       " 'And',\n",
       " 'looked',\n",
       " 'down',\n",
       " 'one',\n",
       " 'as',\n",
       " 'far',\n",
       " 'as',\n",
       " 'I',\n",
       " 'could',\n",
       " 'To',\n",
       " 'where',\n",
       " 'it',\n",
       " 'bent',\n",
       " 'in',\n",
       " 'the',\n",
       " 'undergrowth;',\n",
       " 'Then',\n",
       " 'took',\n",
       " 'the',\n",
       " 'other,',\n",
       " 'as',\n",
       " 'just',\n",
       " 'as',\n",
       " 'fair,',\n",
       " 'And',\n",
       " 'having',\n",
       " 'perhaps',\n",
       " 'the',\n",
       " 'better',\n",
       " 'claim,',\n",
       " 'Because',\n",
       " 'it',\n",
       " 'was',\n",
       " 'grassy',\n",
       " 'and',\n",
       " 'wanted',\n",
       " 'wear;',\n",
       " 'Though',\n",
       " 'as',\n",
       " 'for',\n",
       " 'that',\n",
       " 'the',\n",
       " 'passing',\n",
       " 'there',\n",
       " 'Had',\n",
       " 'worn',\n",
       " 'them',\n",
       " 'really',\n",
       " 'about',\n",
       " 'the',\n",
       " 'same,',\n",
       " 'And',\n",
       " 'both',\n",
       " 'that',\n",
       " 'morning',\n",
       " 'equally',\n",
       " 'lay',\n",
       " 'In',\n",
       " 'leaves',\n",
       " 'no',\n",
       " 'step',\n",
       " 'had',\n",
       " 'trodden',\n",
       " 'black.',\n",
       " 'Oh,',\n",
       " 'I',\n",
       " 'kept',\n",
       " 'the',\n",
       " 'first',\n",
       " 'for',\n",
       " 'another',\n",
       " 'day!',\n",
       " 'Yet',\n",
       " 'knowing',\n",
       " 'how',\n",
       " 'way',\n",
       " 'leads',\n",
       " 'on',\n",
       " 'to',\n",
       " 'way,',\n",
       " 'I',\n",
       " 'doubted',\n",
       " 'if',\n",
       " 'I',\n",
       " 'should',\n",
       " 'ever',\n",
       " 'come',\n",
       " 'back.',\n",
       " 'I',\n",
       " 'shall',\n",
       " 'be',\n",
       " 'telling',\n",
       " 'this',\n",
       " 'with',\n",
       " 'a',\n",
       " 'sigh',\n",
       " 'Somewhere',\n",
       " 'ages',\n",
       " 'and',\n",
       " 'ages',\n",
       " 'hence:',\n",
       " 'Two',\n",
       " 'roads',\n",
       " 'diverged',\n",
       " 'in',\n",
       " 'a',\n",
       " 'wood,',\n",
       " 'and',\n",
       " 'I—',\n",
       " 'I',\n",
       " 'took',\n",
       " 'the',\n",
       " 'one',\n",
       " 'less',\n",
       " 'travelled',\n",
       " 'by,',\n",
       " 'And',\n",
       " 'that',\n",
       " 'has',\n",
       " 'made',\n",
       " 'all',\n",
       " 'the',\n",
       " 'difference.']"
      ]
     },
     "execution_count": 218,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or, more succinctly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 221,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "words = open(\"frost.txt\").read().split()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can ask simple questions about this poem, like how many words does it have?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 223,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "144"
      ]
     },
     "execution_count": 223,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can create a new weird poem by randomly sampling words from the original:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 226,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['In',\n",
       " 'to',\n",
       " 'it',\n",
       " 'all',\n",
       " 'where',\n",
       " 'Yet',\n",
       " 'wood,',\n",
       " 'day!',\n",
       " 'I',\n",
       " 'morning',\n",
       " 'To',\n",
       " 'one',\n",
       " 'far',\n",
       " 'the',\n",
       " 'one',\n",
       " 'sigh',\n",
       " 'it',\n",
       " 'has',\n",
       " 'I',\n",
       " 'with']"
      ]
     },
     "execution_count": 226,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "random.sample(words, 20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or sort the words in alphabetical order:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 228,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['And',\n",
       " 'And',\n",
       " 'And',\n",
       " 'And',\n",
       " 'And',\n",
       " 'And',\n",
       " 'Because',\n",
       " 'Had',\n",
       " 'I',\n",
       " 'I',\n",
       " 'I',\n",
       " 'I',\n",
       " 'I',\n",
       " 'I',\n",
       " 'I',\n",
       " 'I',\n",
       " 'In',\n",
       " 'I—',\n",
       " 'Oh,',\n",
       " 'Somewhere',\n",
       " 'Then',\n",
       " 'Though',\n",
       " 'To',\n",
       " 'Two',\n",
       " 'Two',\n",
       " 'Yet',\n",
       " 'a',\n",
       " 'a',\n",
       " 'a',\n",
       " 'about',\n",
       " 'ages',\n",
       " 'ages',\n",
       " 'all',\n",
       " 'and',\n",
       " 'and',\n",
       " 'and',\n",
       " 'another',\n",
       " 'as',\n",
       " 'as',\n",
       " 'as',\n",
       " 'as',\n",
       " 'as',\n",
       " 'back.',\n",
       " 'be',\n",
       " 'be',\n",
       " 'bent',\n",
       " 'better',\n",
       " 'black.',\n",
       " 'both',\n",
       " 'both',\n",
       " 'by,',\n",
       " 'claim,',\n",
       " 'come',\n",
       " 'could',\n",
       " 'could',\n",
       " 'day!',\n",
       " 'difference.',\n",
       " 'diverged',\n",
       " 'diverged',\n",
       " 'doubted',\n",
       " 'down',\n",
       " 'equally',\n",
       " 'ever',\n",
       " 'fair,',\n",
       " 'far',\n",
       " 'first',\n",
       " 'for',\n",
       " 'for',\n",
       " 'grassy',\n",
       " 'had',\n",
       " 'has',\n",
       " 'having',\n",
       " 'hence:',\n",
       " 'how',\n",
       " 'if',\n",
       " 'in',\n",
       " 'in',\n",
       " 'in',\n",
       " 'it',\n",
       " 'it',\n",
       " 'just',\n",
       " 'kept',\n",
       " 'knowing',\n",
       " 'lay',\n",
       " 'leads',\n",
       " 'leaves',\n",
       " 'less',\n",
       " 'long',\n",
       " 'looked',\n",
       " 'made',\n",
       " 'morning',\n",
       " 'no',\n",
       " 'not',\n",
       " 'on',\n",
       " 'one',\n",
       " 'one',\n",
       " 'one',\n",
       " 'other,',\n",
       " 'passing',\n",
       " 'perhaps',\n",
       " 'really',\n",
       " 'roads',\n",
       " 'roads',\n",
       " 'same,',\n",
       " 'shall',\n",
       " 'should',\n",
       " 'sigh',\n",
       " 'sorry',\n",
       " 'step',\n",
       " 'stood',\n",
       " 'telling',\n",
       " 'that',\n",
       " 'that',\n",
       " 'that',\n",
       " 'the',\n",
       " 'the',\n",
       " 'the',\n",
       " 'the',\n",
       " 'the',\n",
       " 'the',\n",
       " 'the',\n",
       " 'the',\n",
       " 'them',\n",
       " 'there',\n",
       " 'this',\n",
       " 'to',\n",
       " 'took',\n",
       " 'took',\n",
       " 'travel',\n",
       " 'traveler,',\n",
       " 'travelled',\n",
       " 'trodden',\n",
       " 'undergrowth;',\n",
       " 'wanted',\n",
       " 'was',\n",
       " 'way',\n",
       " 'way,',\n",
       " 'wear;',\n",
       " 'where',\n",
       " 'with',\n",
       " 'wood,',\n",
       " 'wood,',\n",
       " 'worn',\n",
       " 'yellow']"
      ]
     },
     "execution_count": 228,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sorted(words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using a list comprehension, we can get all of the words that meet particular criteria, like the words that have more than seven characters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 231,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['diverged',\n",
       " 'traveler,',\n",
       " 'undergrowth;',\n",
       " 'Somewhere',\n",
       " 'diverged',\n",
       " 'travelled',\n",
       " 'difference.']"
      ]
     },
     "execution_count": 231,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[item for item in words if len(item) > 7]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or all of the words that start with the letter `a`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 239,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['a',\n",
       " 'as',\n",
       " 'as',\n",
       " 'as',\n",
       " 'as',\n",
       " 'and',\n",
       " 'as',\n",
       " 'about',\n",
       " 'another',\n",
       " 'a',\n",
       " 'ages',\n",
       " 'and',\n",
       " 'ages',\n",
       " 'a',\n",
       " 'and',\n",
       " 'all']"
      ]
     },
     "execution_count": 239,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[item for item in words if item.startswith(\"a\")]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Formatting lists\n",
    "\n",
    "You've doubtlessly noticed with the previous examples that whenever we evaluate a list, Jupyter Notebook displays *the actual syntax you'd need to reproduce the list*, i.e., with the square brackets, quotes, commas and everything. That's the default way Python shows values when you evaluate it—which is usually helpful, because it means you can just take the text in the cell output and paste it into another notebook and you've got exactly the same value, without needing to re-type it. However, when we're creating poetic output, the extra Python punctuation is undesirable.\n",
    "\n",
    "To make the output prettier, we need to do the following steps:\n",
    "\n",
    "* Create a string from the list\n",
    "* *Print* the string (don't just evaluate it)\n",
    "\n",
    "To create a string from the list, we use the `.join()` method, as outlined above! For example, here's Smooshed Frost from earlier in the notebook, assigned to a variable:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 240,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "smooshed = [line[:5] + line[-5:] for line in poem]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 241,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "list"
      ]
     },
     "execution_count": 241,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(smooshed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a *list of strings*. To create a version of this where all of the strings are joined together, we'll use `.join()`. In this case, let's say that we want each string in the list to be a single line of text in the output, so we'll use `\\n` (the newline character) as the \"glue.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 242,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Two rwood,\\nAnd s both\\nAnd bstood\\nAnd lcould\\nTo whowth;\\n\\nThen fair,\\nAnd hlaim,\\nBecauwear;\\nThougthere\\nHad wsame,\\n\\nAnd by lay\\nIn lelack.\\nOh, I day!\\nYet k way,\\nI douback.\\n\\nI sha sigh\\nSomewence:\\nTwo rnd I—\\nI tood by,\\nAnd tence.'"
      ]
     },
     "execution_count": 242,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"\\n\".join(smooshed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ugh, that's still not right though! It's one string now, but when we evaluate the expression Python is *showing* us the escape characters instead of *interpreting* them. To get Python to interpret them, we have to send the whole thing to the print function, like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 244,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Two rwood,\n",
      "And s both\n",
      "And bstood\n",
      "And lcould\n",
      "To whowth;\n",
      "\n",
      "Then fair,\n",
      "And hlaim,\n",
      "Becauwear;\n",
      "Thougthere\n",
      "Had wsame,\n",
      "\n",
      "And by lay\n",
      "In lelack.\n",
      "Oh, I day!\n",
      "Yet k way,\n",
      "I douback.\n",
      "\n",
      "I sha sigh\n",
      "Somewence:\n",
      "Two rnd I—\n",
      "I tood by,\n",
      "And tence.\n"
     ]
    }
   ],
   "source": [
    "print(\"\\n\".join(smooshed))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There we go! Ready to submit to our favorite poetry journal. Of course, you don't have to join with a newline character. Let's say you want to make a prose poem (i.e., no line breaks) by randomly sampling fifty words in the Frost poem. We'll use the space character as the glue:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 247,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "there knowing diverged another one And claim, roads where about black. undergrowth; In ever in both less ages Because as I wood, long ages for as leads and by, that has trodden sorry be took stood a just diverged And I day! I— hence: if how Two the I travel\n"
     ]
    }
   ],
   "source": [
    "print(\" \".join(random.sample(words, 50)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nice!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Making changes to lists\n",
    "\n",
    "Often we'll want to make changes to a list after we've created it---for\n",
    "example, we might want to append elements to the list, remove elements from\n",
    "the list, or change the order of elements in the list. Python has a number\n",
    "of methods for facilitating these operations.\n",
    "\n",
    "The first method we'll talk about is `.append()`, which adds an item on to\n",
    "the end of an existing list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 208,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['flour', 'milk', 'eggs', 'sugar']"
      ]
     },
     "execution_count": 208,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ingredients = [\"flour\", \"milk\", \"eggs\"]\n",
    "ingredients.append(\"sugar\")\n",
    "ingredients"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that invoking the `.append()` method doesn't itself evaluate to\n",
    "anything! (Technically, it evaluates to a special value of type `None`.)\n",
    "Unlike many of the methods and syntactic constructions we've looked at so far,\n",
    "the `.append()` method changes the underlying value---it doesn't return a\n",
    "new value that is a copy with changes applied.\n",
    "\n",
    "There are two methods to facilitate removing values from a list: `.pop()` and\n",
    "`.remove()`. The `.remove()` method removes from the list the first value that\n",
    "matches the value in the parentheses:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 209,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['milk', 'eggs', 'sugar']"
      ]
     },
     "execution_count": 209,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ingredients = [\"flour\", \"milk\", \"eggs\", \"sugar\"]\n",
    "ingredients.remove(\"flour\")\n",
    "ingredients"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(Note that `.remove()`, like `.append()` doesn't evaluate to anything---it\n",
    "changes the list itself.)\n",
    "\n",
    "The `.pop()` method works slightly differently: give it an expression that\n",
    "evaluates to an integer, and it evaluates to the expression at the index\n",
    "named by the integer. But it also has a side effect: it *removes* that item\n",
    "from the list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 210,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['flour', 'eggs', 'sugar']"
      ]
     },
     "execution_count": 210,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ingredients = [\"flour\", \"milk\", \"eggs\", \"sugar\"]\n",
    "ingredients.pop(1)\n",
    "ingredients"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> EXERCISE: What happens when you try to `.pop()` a value from a list at an index that doesn't exist in the list? What happens you try to `.remove()` an item from a list if that item isn't in that list to begin with?\n",
    "\n",
    "> ANOTHER EXERCISE: Write an expression that `.pop()`s the second-to-last item from a list. SPOILER: <span style=\"background: black;\">(Did you guess that you could use negative indexing with `.pop()`?</span>\n",
    "\n",
    "The `.sort()` and `.reverse()` methods do exactly the same thing as their\n",
    "function counterparts `sorted()` and `reversed()`, with the only difference\n",
    "being that the methods don't evaluate to anything, instead opting to change\n",
    "the list in-place."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 211,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['eggs', 'flour', 'milk', 'sugar']"
      ]
     },
     "execution_count": 211,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ingredients = [\"flour\", \"milk\", \"eggs\", \"sugar\"]\n",
    "ingredients.sort()\n",
    "ingredients"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 212,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['sugar', 'eggs', 'milk', 'flour']"
      ]
     },
     "execution_count": 212,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ingredients = [\"flour\", \"milk\", \"eggs\", \"sugar\"]\n",
    "ingredients.reverse()\n",
    "ingredients"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> EXERCISE: Write a Python command-line program that prints out the lines of a text file in random order."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Iterating over lists with `for`\n",
    "\n",
    "The list comprehension syntax discussed earlier is very powerful: it allows you to succinctly transform one list into another list by thinking in terms of filtering and modification. But sometimes your primary goal isn't to make a new list, but simply to perform a set of operations on an existing list.\n",
    "\n",
    "Let's say that you want to print every string in a list. Here's a short text:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "text = \"it was the best of times, it was the worst of times\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can make a list of all the words in the text by splitting on whitespace:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "words = text.split()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Of course, we can see what's in the list simply by evaluating the variable:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['it',\n",
       " 'was',\n",
       " 'the',\n",
       " 'best',\n",
       " 'of',\n",
       " 'times,',\n",
       " 'it',\n",
       " 'was',\n",
       " 'the',\n",
       " 'worst',\n",
       " 'of',\n",
       " 'times']"
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But let's say that we want to print out each word on a separate line, without any of Python's weird punctuation. In other words, I want the output to look like:\n",
    "\n",
    "    it\n",
    "    was\n",
    "    the\n",
    "    best\n",
    "    of\n",
    "    times,\n",
    "    it\n",
    "    was\n",
    "    the\n",
    "    worst\n",
    "    of\n",
    "    times\n",
    "    \n",
    "But how can this be accomplished? We know that the `print()` function can display an individual string in this manner:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hello\n"
     ]
    }
   ],
   "source": [
    "print(\"hello\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So what we need, clearly, is a way to call the `print()` function with every item of the list. We could do this by writing a series of `print()` statements, one for every item in the list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "it\n",
      "was\n",
      "the\n",
      "best\n",
      "of\n",
      "times,\n",
      "it\n",
      "was\n",
      "the\n",
      "worst\n",
      "of\n",
      "times\n"
     ]
    }
   ],
   "source": [
    "print(words[0])\n",
    "print(words[1])\n",
    "print(words[2])\n",
    "print(words[3])\n",
    "print(words[4])\n",
    "print(words[5])\n",
    "print(words[6])\n",
    "print(words[7])\n",
    "print(words[8])\n",
    "print(words[9])\n",
    "print(words[10])\n",
    "print(words[11])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nice, but there are some problems with this approach:\n",
    "\n",
    "1. It's kind of verbose---we're doing exactly the same thing multiple times, only with slightly different expressions. Surely there's an easier way to tell the computer to do this?\n",
    "2. It doesn't scale. What if we wrote a program that we want to produce hundreds or thousands of lines. Would we really need to write a `print` statement for each of those expressions?\n",
    "3. It requires us to know how many items are going to end up in the list to begin with."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Things are looking grim! But there's hope. Performing the same operation on all items of a list is an extremely common task in computer programming. So common,\n",
    "that Python has some built-in syntax to make the task easy: the `for` loop.\n",
    "\n",
    "Here's how a `for` loop looks:\n",
    "\n",
    "    for tempvar in sourcelist:\n",
    "        statements\n",
    "\n",
    "The words `for` and `in` just have to be there---that's how Python knows it's\n",
    "a `for` loop. Here's what each of those parts mean.\n",
    "\n",
    "* *tempvar*: A name for a variable. Inside of the for loop, this variable will contain the current item of the list.\n",
    "* *sourcelist*: This can be any Python expression that evaluates to a list---a variable that contains a list, or a list slice, or even a list literal that you just type right in!\n",
    "* *statements*: One or more Python statements. Everything tabbed over underneath the `for` will be executed once for each item in the list. The statements tabbed over underneath the `for` line are called the *body* of the loop.\n",
    "\n",
    "Here's what the `for` loop for printing out every item in a list might look like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "it\n",
      "was\n",
      "the\n",
      "best\n",
      "of\n",
      "times,\n",
      "it\n",
      "was\n",
      "the\n",
      "worst\n",
      "of\n",
      "times\n"
     ]
    }
   ],
   "source": [
    "for item in words:\n",
    "    print(item)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The variable name `item` is arbitrary. You can pick whatever variable name you like, as long as you're consistent about using the same variable name in the body of the loop. If you wrote out this loop in a long-hand fashion, it might look like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "it\n",
      "was\n",
      "the\n",
      "best\n"
     ]
    }
   ],
   "source": [
    "item = words[0]\n",
    "print(item)\n",
    "item = words[1]\n",
    "print(item)\n",
    "item = words[2]\n",
    "print(item)\n",
    "item = words[3]\n",
    "print(item)\n",
    "# etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Of course, the body of the loop can have more than one statement, and you can assign values to variables inside the loop:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "IT\n",
      "WAS\n",
      "THE\n",
      "BEST\n",
      "OF\n",
      "TIMES,\n",
      "IT\n",
      "WAS\n",
      "THE\n",
      "WORST\n",
      "OF\n",
      "TIMES\n"
     ]
    }
   ],
   "source": [
    "for item in words:\n",
    "    yelling = item.upper()\n",
    "    print(yelling)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also include other kinds of nested statements inside the `for` loop, like `if/else`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "IT\n",
      "   was\n",
      "   the\n",
      "best\n",
      "OF\n",
      "times,\n",
      "IT\n",
      "   was\n",
      "   the\n",
      "worst\n",
      "OF\n",
      "times\n"
     ]
    }
   ],
   "source": [
    "for item in words:\n",
    "    if len(item) == 2:\n",
    "        print(item.upper())\n",
    "    elif len(item) == 3:\n",
    "        print(\"   \" + item)\n",
    "    else:\n",
    "        print(item)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This structure is called a \"loop\" because when Python reaches the end of the statements in the body, it \"loops\" back to the beginning of the body, and executes the same statements again (this time with the next item in the list). \n",
    "\n",
    "Python programmers tend to use `for` loops most often when the problem would otherwise be too tricky or complicated to solve using a list comprehension. It's easy to paraphrase any list comprehension in `for` loop syntax. For example, this list comprehension, which evaluates to a list of the squares of even integers from 1 to 25:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[4, 16, 36, 64, 100, 144, 196, 256, 324, 400, 484, 576]"
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[x * x for x in range(1, 26) if x % 2 == 0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can rewrite this list comprehesion as a `for` loop by starting out with an empty list, then appending an item to the list inside the loop. The source list remains the same:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[4, 16, 36, 64, 100, 144, 196, 256, 324, 400, 484, 576]"
      ]
     },
     "execution_count": 104,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result = []\n",
    "for x in range(1, 26):\n",
    "    if x % 2 == 0:\n",
    "        result.append(x * x)\n",
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "We've put down the foundation today for you to become fluent in Python's very powerful and super-convenient syntax for lists. We've also done a bit of data parsing and analysis! Pretty good for day one.\n",
    "\n",
    "Further resources:\n",
    "\n",
    "* [Lists](http://openbookproject.net/thinkcs/python/english3e/lists.html), from [How To Think Like A Computer Scientist: Learning with Python](http://openbookproject.net/thinkcs/python/english3e/index.html)\n",
    "* [Lists](https://docs.python.org/3/tutorial/introduction.html#lists) and [More on Lists](https://docs.python.org/3/tutorial/datastructures.html#more-on-lists) from the [Official Tutorial](https://docs.python.org/3/tutorial/index.html)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}