Python hints and tricks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Python hints and tricks\n",
    "I’ve put together a collection of nice tricks and time savers that might help make your Python more Pythonic! \n",
    "\n",
    "In no particular order..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 1. Use list comprehensions\n",
    "These one line constructs make creating list objects trivially easy. e.g."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4\n"
     ]
    }
   ],
   "source": [
    "my_list = [ x**2 for x in range(100) ]\n",
    "\n",
    "print(my_list[2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "For the more adventurous it’s also possible to include logic statements and nested comprehensions, but don’t overdo it, I’ve seen 5 line comprehensions before and it’s not pretty!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "16\n"
     ]
    }
   ],
   "source": [
    "my_constrained_list = [ x**2 for x in range(100) if x%2 == 0]\n",
    "\n",
    "print(my_constrained_list[2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 2 Know when not to use list comprehensions - using generators instead\n",
    "Generators allow you to declare a function that behaves as an iterator. That is the resulting expression is not evaluated and stored in memory when it is declared (as for a list comprehension), rather it is evaluated each time the function is called."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "For cases where the expression is evaluated only once, or where the expression would be too large to store in memory, the benefits are obvious. It is easy to define functions which act as generators, but you can also use ‘generator comprehension’ which is almost identical to a list comprehension except using parenthesis, e.g."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "my_gen = ( x^2 for x in xrange(10**9) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Note however that you can't index them directly, only iterate over them (which makes sense if you think about it):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "50005002\n"
     ]
    }
   ],
   "source": [
    "total = 0\n",
    "for val in my_gen:\n",
    "    total += val\n",
    "    if val > 10**4: \n",
    "        break\n",
    "     \n",
    "print(total)\n",
    "# print my_gen[4] <-- This won't work"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 3 Dictionary comprehensions\n",
    "Dictionaries are a very useful construct in Python, and it is very easy to generate dictionaries using dictionary comprehensions to specify each key:value pair, e.g."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "my_dict = { 'Book' : 5, 'Monkey': 7, 'Paper': 23.4 }\n",
    "\n",
    "print(my_dict['Monkey'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Or make them using iterators:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "my_new_dict = { x: x**2 for x in range(100) }\n",
    "\n",
    "print(my_new_dict[5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 4 Dictionary values as functions / Classes\n",
    "It may not be immediately obvious to new Python programmers but because Classes and functions are first class objects it is trivially easy to store these in lists, or even dictionaries. (One great example of this is an implementation of the strategy pattern using dictionaries.)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Edam\n"
     ]
    }
   ],
   "source": [
    "def nuts():\n",
    "    return 'Peanuts'\n",
    "\n",
    "def cheese():\n",
    "    return 'Edam'\n",
    "\n",
    "feed = {'Monkey': nuts, 'Mouse': cheese}\n",
    "\n",
    "my_food = feed['Mouse']()\n",
    "\n",
    "print(my_food)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 5 The 'map' function\n",
    "This function makes it really easy to perform operations on any collection of objects. e.g."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "squares = map(lambda x: x**2, range(100))\n",
    "\n",
    "print(squares)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "It returns a list of results mapping the function given onto the list values (which may just be any form of iterable)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 6 Unpacking arguments\n",
    "It is possible to unpack a list into a function call as mandatory arguments. e.g."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def example_function(food_type, amount, colour=''):\n",
    "    print(\"Type: {}\".format(food_type))\n",
    "    print(\"Amount: {}\".format(amount))\n",
    "    print(\"Colour: {}\".format(colour))\n",
    "    \n",
    "arguments = ['Nuts', 50]\n",
    "example_function(*arguments) # unpacks my list into mandatory arguments"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "or, unpacking dictionaries for optional arguments:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "arg_dict = {'colour': 'Blue'}\n",
    "example_function('Cheese', 2, **arg_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "or, both:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "example_function(*arguments, **arg_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "You can even unpack numpy arrays! *Note* that the order matters for mandatory arguments, but not optional ones."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 7 Unpacking return values\n",
    "It’s also possible to unpack return values of a function directly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "def moments(x):\n",
    "    \"\"\"Return the first three momments of a (normal) distribution\"\"\"\n",
    "    return 1, np.mean(x), np.std(x)\n",
    "\n",
    "print(moments(np.arange(10)))\n",
    "\n",
    "first, second, third = moments(np.arange(10))\n",
    "\n",
    "print(third)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "A great example of this demonstrating this and the previous example is in-place value swapping - e.g:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5 10\n",
      "10 5\n"
     ]
    }
   ],
   "source": [
    "a, b = 5, 10\n",
    "\n",
    "print(a, b)\n",
    "\n",
    "a, b = b, a\n",
    "\n",
    "print(a, b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 8 For (almost) any numerical work use Numpy!\n",
    "Numpy is a numerical library with very fast linear algebra operations and a number of extremely useful constructs. See http://www.numpy.org/."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 9 Chained comparisons\n",
    "It is really easy to chain comparisons together in an intuitive way e.g."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def five():\n",
    "    print(\"5 being called\")\n",
    "    return 5\n",
    "\n",
    "def six():\n",
    "    print(\"6 being called\")\n",
    "    return 6\n",
    "\n",
    "if 1 < five() < six():\n",
    "    print(True)\n",
    "\n",
    "if 1 > five() > six():\n",
    "    print(True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Also, the function five() only gets evaluated once, and the second comparison still gets short circuited if the first fails."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 10 Advanced indexing\n",
    "There are a number of ways of indexing lists which you may not have been aware of:\n",
    " * You can count backwards, e.g. access the last element in a list using my_list[-1] \n",
    " * Reversing a list using my_list[::-1].\n",
    " * The above is just a special case of setting an increment e.g. my_list[::2] gives a step of 2. \n",
    " * All of the above work on strings!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 11 Using enumerate\n",
    "\n",
    "The function enumerate returns a counter as well as the item to be enumerated which can be very useful if you need the index of an item as well as the item itself. e.g."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "for i, x in enumerate(my_list):\n",
    "    print(i, my_list[i-2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 12 Default dictionary values\n",
    "In order to avoid having to catch KeyErrors every time you query a dictionary you can use the get method to provide a default value if the key is not present."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "val = 0\n",
    "try:\n",
    "    val = my_dict[101]\n",
    "except KeyError:\n",
    "    print(\"That key of the dictionary doesn't exist\")\n",
    "\n",
    "print(my_dict.get(101, 4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "Also - there is a defaultdict collection which gives keys default values, or use my_dict.setdefault to set a default on a standard dict. There are some subtle differences though about when the default is created, and some code might expect a KeyError, so take care with this one."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 13 Named formatting\n",
    "You may have noticed I've been using implicit formatting to fill in values. This is probably fine when there is only one value, and it works when there is more, but it's probably best to use named placeholders, e.g.:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "print(\"The {foo} is {bar}\".format(foo='answer', bar=42))\n",
    "\n",
    "# Note that you can also unpack a dict into format!\n",
    "\n",
    "words = {'foo': 'answer', 'bar': '7x6'}\n",
    "print(\"The {foo} is {bar}\".format(**words))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 14 Much more readable ternary operators\n",
    "Although generally frowned upon, the ternary operators in Python are actually fairly readable and intuitive when written as:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "test = 'Yes' if 1 < five() < six() else 'No'\n",
    "\n",
    "print('Did my test pass?: {}'.format(test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 15 Classes can be created at run-time\n",
    "This one is definitely not for the feint hearted. Because classes are first class objects in Python it is possible to define them at run-time, e.g. within if statements or even functions. Use with care!\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "x = 6\n",
    "\n",
    "if x < five():\n",
    "    class test(object):\n",
    "        def number(self):\n",
    "            return x\n",
    "else:\n",
    "    class test(object):\n",
    "        def number(self):\n",
    "            return 5\n",
    "        \n",
    "print(test().number())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from numpy.random import randn\n",
    "df = pd.DataFrame(randn(3, 4), columns=['obs_CCN', 'obs_N', 'model_N', 'model_CCN'], index=range(3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    ""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# 16 The with statement is your friend"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The with statement is a bit like a try, except block, but is intended for standard code flow, rather than exception handling. For example, a really common use is with file handling:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "with open('test', 'w') as f:\n",
    "    pass\n",
    "    # do something"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "The ‘with’ statement doesn’t take care of the fact that the file may not exist, or other IO errors, but it does ensure that if an exception occurs in the ‘do something’ block then the file gets closed regardless. Obviously, this is most useful for IO, or network connections where you have to ensure some finally block is executed, but should be extendable to more general scenarios."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "But it's also possible to create your own implementations. In order to be able to use a with statement in your own code you can create a context manager which implements both __enter()__ and __exit()__ methods (see PEP-343 for details), or more simply use the built-in contextlib. A good example is provided by StackOverflow (http://stackoverflow.com/questions/3012488/what-is-the-python-with-statement-designed-for):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from contextlib import contextmanager\n",
    "import os\n",
    "\n",
    "@contextmanager\n",
    "def working_directory(path):\n",
    "    current_dir = os.getcwd()\n",
    "    os.chdir(path)\n",
    "    try:\n",
    "        yield\n",
    "    finally:\n",
    "        os.chdir(current_dir)\n",
    "        \n",
    "with working_directory(\"/\"): \n",
    "    pass\n",
    "    # do something within data/stuff\n",
    "    \n",
    "# here I am back again in the original working directory"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2.0
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}