notebooks/IO Python.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# A Functional Introduction To Python\n",
    "##  Section 1.  Introductory Concepts \n",
    "## Section 2.  Functions\n",
    "## Section 3.  Control Structures\n",
    "## Section 4.  Intermediate Topics\n",
    "## <span style=\"color:blue\">Section 5.  IO</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### I/O Operations in Python:\n",
    "\n",
    "* Writing a file\n",
    "* Reading a file\n",
    "* Using Subprocess Module\n",
    "* Reading YAML files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Writing to a file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "foo"
     ]
    }
   ],
   "source": [
    "f = open('workfile.txt', 'w')\n",
    "f.write(\"foo\")\n",
    "f.close()\n",
    "!cat workfile.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Writing to a file with 'context'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "bam"
     ]
    }
   ],
   "source": [
    "with open(\"workfile.txt\", \"w\") as workfile:\n",
    "    workfile.write(\"bam\")\n",
    "!cat workfile.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Reading a file in"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['bam']\n"
     ]
    }
   ],
   "source": [
    "f = open(\"workfile.txt\", \"r\")\n",
    "out = f.readlines()\n",
    "f.close()\n",
    "print(out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Write two things with complex sentence\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "number = 1.0\n",
    "my_string = \"My favorite number\"\n",
    "statement = f\"{my_string} {number}\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "My favorite number 1.0\n"
     ]
    }
   ],
   "source": [
    "print(statement)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "My favorite number 1.0"
     ]
    }
   ],
   "source": [
    "with open(\"workfile2.txt\", \"w\") as workfile:\n",
    "    workfile.write(statement)\n",
    "!cat workfile2.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### How do I create a DataFrame in Pandas\n",
    "* Load a csv file\n",
    "* Load a list or dictionary to create a dataframe\n",
    "* Create an empty dataframe and programtically insert values into it\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Worth Reading more about pandas I/O\n",
    "* https://chrisalbon.com/\n",
    "* https://github.com/jakevdp/PythonDataScienceHandbook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Using the subprocess command\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "import subprocess\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "res = subprocess.Popen(\"ls -l\", shell=True, stdout=subprocess.PIPE)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "out = res.stdout.readlines()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[b'total 368\\n', b'-rw-r--r--  1 noahgift  staff  20612 Nov 27 16:16 Functional_Introduction_To_Python_Section_1(Introductory_Concepts).ipynb\\n', b'-rw-r--r--  1 noahgift  staff  30980 Nov 27 11:16 Functional_Introduction_To_Python_Section_2(Functions).ipynb\\n', b'-rw-r--r--  1 noahgift  staff  11068 Nov 21 14:49 Functional_Introduction_To_Python_Section_3(Control_Structures).ipynb\\n', b'-rw-r--r--  1 noahgift  staff   9467 Nov 22 14:21 Functional_Introduction_To_Python_Section_4(Intermediate_Topics).ipynb\\n', b'-rw-r--r--  1 noahgift  staff  10792 Nov 29 10:26 IO Python.ipynb\\n', b'-rw-r--r--  1 noahgift  staff  64300 Nov 27 14:19 class1_scratchpad.ipynb\\n', b'-rw-r--r--  1 noahgift  staff   1059 Nov 28 08:40 class2_scratchpad.ipynb\\n', b'-rw-r--r--  1 noahgift  staff    664 Nov 29 08:24 class3_scratchpad.ipynb\\n', b'-rw-r--r--  1 noahgift  staff     20 Nov 29 10:15 data.json\\n', b'-rw-r--r--  1 noahgift  staff     14 Nov 29 10:21 data.yaml\\n', b'-rw-r--r--  1 noahgift  staff     32 Nov 29 10:05 mydictionary.pickle\\n', b'-rw-r--r--  1 noahgift  staff      3 Nov 29 10:39 workfile.txt\\n', b'-rw-r--r--  1 noahgift  staff     22 Nov 29 10:39 workfile2.txt\\n']\n"
     ]
    }
   ],
   "source": [
    "print(out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Multiprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1, 4, 9]\n"
     ]
    }
   ],
   "source": [
    "from multiprocessing import Pool\n",
    "\n",
    "def f(x):\n",
    "    return x*x\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    p = Pool(5)\n",
    "    print(p.map(f, [1, 2, 3]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[42, None, 'hello']\n"
     ]
    }
   ],
   "source": [
    "from multiprocessing import Process, Queue\n",
    "\n",
    "def f(q):\n",
    "    q.put([42, None, 'hello'])\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    q = Queue()\n",
    "    p = Process(target=f, args=(q,))\n",
    "    p.start()\n",
    "    print(q.get())    # prints \"[42, None, 'hello']\"\n",
    "    p.join()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Pyomo Article\n",
    "\n",
    "https://www.ibm.com/developerworks/cloud/library/cl-optimizepythoncloud1/\n",
    "\n",
    "![Worker Farm](https://www.ibm.com/developerworks/cloud/library/cl-optimizepythoncloud2/figure1.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Serialize a Python Dictionary to Pickle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "mydict = {\"one\":1, \"two\":2}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "import pickle\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "pickle.dump(mydict, open('mydictionary.pickle', 'wb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-r--r--  1 noahgift  staff  32 Nov 29 10:39 mydictionary.pickle\r\n"
     ]
    }
   ],
   "source": [
    "!ls -l mydictionary.pickle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "�\u0003}q\u0000(X\u0003\u0000\u0000\u0000oneq\u0001K\u0001X\u0003\u0000\u0000\u0000twoq\u0002K\u0002u."
     ]
    }
   ],
   "source": [
    "!cat mydictionary.pickle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "res = pickle.load(open('mydictionary.pickle', \"rb\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'one': 1, 'two': 2}\n"
     ]
    }
   ],
   "source": [
    "print(res)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Serialize a Python Dictionary to JSON\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "import json\n",
    "with open('data.json', 'w') as outfile:\n",
    "    json.dump(res, outfile)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"one\": 1, \"two\": 2}"
     ]
    }
   ],
   "source": [
    "!cat data.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "with open('data.json', 'rb') as outfile:\n",
    "    res2 = json.load(outfile)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'one': 1, 'two': 2}\n"
     ]
    }
   ],
   "source": [
    "print(res2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Save to Yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "import yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "with open(\"data.yaml\", \"w\") as yamlfile:                                               \n",
    "    yaml.safe_dump(res2, yamlfile, default_flow_style=False) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "one: 1\r\n",
      "two: 2\r\n"
     ]
    }
   ],
   "source": [
    "!cat data.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Load Yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "with open(\"data.yaml\", \"rb\") as yamlfile:                                               \n",
    "    res3 = yaml.safe_load(yamlfile) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'one': 1, 'two': 2}\n"
     ]
    }
   ],
   "source": [
    "print(res3)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}