notebooks/overview.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "LwHphyYT2lo5"
   },
   "source": [
    "## Parallel Programming Models\n",
    "\n",
    "In this workshop we will explore how to make use of threading and multiple processes in Python via the threading and multiprocessing libraries. Much of this and other notebooks are adapated from the [Software Carpentry Foundation](https://software-carpentry.org/) set of teaching materials on intermediate Python and from the [High Performance Python](https://www.oreilly.com/library/view/high-performance-python/9781449361747/) O'Reilly book\n",
    "\n",
    "The Python code one ordinarily writes, aside from calls to C-extension libraries that take advantage of threading (think numpy or scipy), execute each line of code serially using one process on the system. In Python, single-CPU use is caused by the global interpreter lock (GIL), which allows only one thread to carry the Python interpreter at any given time. The GIL was implemented to handle a memory management issue, but as a result, Python is limited to using a single processor.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Serial Execution\n",
    "![](../imgs/serial.png)\n",
    "#### Parallel Execution\n",
    "![](../imgs/multiproc.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## So you wanna go fast? \n",
    "Probably you've gotten to the point where your programs cannot take advantage of performance increases offered by multi-core CPUs unless you modify your programs. Here we will discuss the differences between threads and processes and how one goes about exploiting them to decrease the overall run time of their programs.\n",
    "\n",
    "<p> <img src=\"https://i.makeagif.com/media/6-17-2015/10Jbv7.gif\"> </p>\n",
    "\n",
    "Using threads or multiple processes will not make any of the tasks to be executed any faster in python. It only allows us to do more tasks by distributing the tasks among the available cores, which acts to reduce the overall time to completion of the program. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CPU multi-processing / Distributed-memory parallelism\n",
    "\n",
    "Multi-processing is one way to execute tasks in parallel on a multi-core CPU, or across multiple computers in a computing cluster. In multi-processing, each task runs in its own process; each program running on your computer is represented by one or more processes. For example, if you are running a web browser, a text editor, and an e-mail client, you are running at least three processes (and likely many more background processes). On modern operating systems, each process gets its own portion of your computer’s memory, ensuring that no process can interfere with the execution of another (though tools like MPI and even Python’s multiprocessing library can be used to share data between processes running locally or in distributed computing environments).\n",
    "\n",
    "## CPU multi-threading (shared memory parallelism)\n",
    "\n",
    "Multi-processing is not to be confused with multi-threading, or shared-memory parallelism. In modern operating systems, each process contains one or more threads of execution. These threads share the same portion of memory assigned to their parent process; each thread can run in parallel if the computer has more than one CPU core. For certain algorithms, multi-threading can be more efficient than multi-processing (though multi-processing solutions such as MPI often scale better to larger problem sizes). However, multi-threading is more error-prone to program and is generally only done directly by expert systems programmers. Tools such as OpenMP should in general be used for multi-threading in scientific computing. Your use of shared memory parallelism will come from use of cPython C extension libraries that use threading in the C source code. For example, this is what is going on under the hood of many of the functions one uses in numpy or scipy for linear algebra. You still have the ability to control the number of threads used by setting external environment variables prior to importing the package library. More on this later. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Processes and threads\n",
    "\n",
    "- **Process** is an execution context: binary instructions, memory, resources.\n",
    "- **Thread** in a unit of execution, has virtualized processor. \n",
    "- Process has at least one thread, can have many.\n",
    "- Multiple threads share resources in a context of one process.\n",
    "- Processes run in a different address spaces.\n",
    "\n",
    "\n",
    "### Comparison of Threads and Processes in Python\n",
    "  \n",
    "\n",
    "\n",
    "| Processes                                                                | Threads                                                          |\n",
    "|------------------------------------------------------------------------|-----------------------------------------------------------------|\n",
    "|- can run in parallel in Python (Avoids GIL limitations)                                        |- can run only concurrently in Python (GIL) - no multicore       |\n",
    "|- separate memory space (easy handling, harder communications - IPC)    |- shared memory space (hard management, easy communication)      |\n",
    "|- larger memory footprint (usually used in tens - hundreds)             |- lightweight (can be used in hundreds - thousands)              |\n",
    "|- Child Processes can be interrupted/killed            |- Is not interruptible/killable              |\n",
    "|- Used for cPython CPU-bound processing             |- Great for I/O bound problems              |\n",
    "|- Separate Memory space for each process           |- cPython C extension modules use this (numpy & scipy)              |\n",
    "\n",
    "**Processes are your best option to utilize multiple cores in Python**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Multiprocessing allows you to create programs that can run concurrently (bypassing the GIL) and use the entirety of your CPU core. Though it is fundamentally different from the threading library, the syntax is quite similar. The multiprocessing library gives each process its own Python interpreter and each their own GIL.\n",
    "\n",
    "Because of this, the usual problems associated with threading (such as data corruption and deadlocks) are no longer an issue. Since the processes don't share memory, they can't modify the same memory concurrently."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Other Thoughts \n",
    "\n",
    "### GIL-less Python\n",
    "It’s worth mentioning that there are implementations of Python which do not include a GIL. In particular there are [Jython](http://www.jython.org/) and [IronPython.net](http://ironpython.net/).\n",
    "\n",
    "There's also ongoing effort to support Python with less reliance on the GIL through a technique and project of the same name called [Software Transactional Memory](http://doc.pypy.org/en/latest/stm.html).\n",
    "\n",
    "### Cluster Distribution / Task Management\n",
    "\n",
    "There are other python libraries to keep in mind that serve a role of managing distribution of tasks across resources (Cluster Distribution) and for automating and scheduling tasks in a workflow. Such examples include [Celery](http://www.celeryproject.org/) and [Parsl](http://parsl-project.org/)\n"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "default_view": {},
   "name": "multiprocessing.ipynb",
   "provenance": [],
   "version": "0.3.2",
   "views": {}
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}