011_lambda.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/rzl-ds/gu511/blob/master/011_lambda.ipynb\" target=\"_parent\">\n",
    "    <img src=\"https://colab.research.google.com/assets/colab-badge.svg\"/>\n",
    "</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# `aws lambda`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "think of the steps we needed to go through before we were able to execute our sad `alarm_clock.py` script in the previous lectures and homeworks:\n",
    "\n",
    "1. provision an `ec2` instance\n",
    "2. make an `ssh` connection to that server\n",
    "3. copy our `python` script to that server\n",
    "4. create an environment capable of executing that `python` script\n",
    "5. establish an `iam role` for `aws cli` and `boto3` process running on that server\n",
    "6. add a `crontab` entry to run that script with some frequency\n",
    "7. leave our `ec2` server up and running while the script runs once a day"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "that's a fair amount of work and infrastructure for doing something as simple as running a block of code. \n",
    "\n",
    "it's also relatively expensive: we didn't *need* our `ec2` service to be up and running constantly -- only for a few seconds each day. And we didn't utilize all of the resources available to our machine while we were executing that code, so even that time was somewhat wasted."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "`aws lambda` is a serverless code execution service which tries to provide a more streamlined and simple process for executing code in a way that is highly integrated with other `aws` services.\n",
    "\n",
    "it also generalizes from *scheduling* to *triggered* events -- meaning we can execute code with some temporal frequency, *or* any time some particular thing happens (*e.g.* a file is posted to a particular `s3` bucket). this makes it a sort of \"if this then that\" for `aws`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## how `lambda` functions are defined"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "a `lambda` function is comprised of three things:\n",
    "\n",
    "1. a runtime and an environment: a particular language and particular versions of that language, other environment setups\n",
    "2. a trigger: some event that causes a function to run\n",
    "3. a function: some code executed in that run-time after that trigger"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### run-time and execution environment\n",
    "\n",
    "this is basically the \"where\" of your code's execution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "the available runtimes are updated with some frequency, so it's usually best to look it up and check out [this page](https://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html). of course, the `lambda` function creation dialog will also make you choose one of them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "at the time of writing, though, the environment and runtimes were as follows:\n",
    "\n",
    "+ os: all executed on the Amazon Linux AMI, 64 bit\n",
    "+ runtimes:\n",
    "    + `node.js` v4.3.2 and 5.10.3\n",
    "    + `java` 8\n",
    "    + **`python` 3.6 and 2.7**\n",
    "    + `C# .NET Core` 1.0.1\n",
    "+ sdks\n",
    "    + `javascript`\n",
    "    + `boto3` for `python 2.7, python 3.6`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### triggers\n",
    "\n",
    "this is the \"when\" of your code's execution."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "there is a *huge* list of events and items that could trigger the execution of your `lambda` function. for example,\n",
    "\n",
    "1. any event in several `aws` services\n",
    "2. an open `api`\n",
    "3. an alexa\n",
    "4. nothing at all (manual)\n",
    "\n",
    "moral of the story: you can *probably* trigger it the way you want (especially via the `api` endpoint). It may not be obvious, but I bet you can do it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### functions\n",
    "\n",
    "finally, the \"what\" of your code's execution is the code itself."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "at their heart, every `aws lambda` function is a zip directory of code that can be executed in the given, fixed runtime from above, and a single functional endpoint called a \"handler\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "in the `python` world, that endpoint is specified as a python module (basically, a file name) and single function in that file which looks like\n",
    "\n",
    "```python\n",
    "# file: file_name.py\n",
    "def function_name(event, context):\n",
    "    # does something using the event json and the context\n",
    "```\n",
    "\n",
    "the handler here is `file_name.function_name`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### `event`\n",
    "\n",
    "the `event` item is a `json` object that is usually completely defined by the triggering event you chose above.\n",
    "\n",
    "given that we are usually writing `python` script here, you should know that the `json` library has already parsed this `event` into a `dict`\n",
    "\n",
    "```python\n",
    "event = {\n",
    "    'key1': 'val1',\n",
    "    'key2': ['list2elem1', 'list2elem2', 'list2elem3'],\n",
    "    'key3': {\n",
    "        'key3_1': 'val3_1',\n",
    "        'key3_2': 'val3_2',\n",
    "    }\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "you can either\n",
    "\n",
    "1. *use* that triggering event to change the way your code executes\n",
    "    1. *e.g.* `eventval1 = event['key1']`\n",
    "    2. this means that you need to know the structure of that event `json` object\n",
    "2. ignore the content of the event and simply resond to the fact that *some* event happened\n",
    "\n",
    "either way, though, it is expected that your `lambda` function code will take the event in as its first parameter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "`aws` has collected an example of every `event` type on the following page: http://docs.aws.amazon.com/lambda/latest/dg/eventsources.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### `context`\n",
    "\n",
    "in each of the runtimes, `aws` creates an object called a `context` that allows the user to interactively interact with the runtime and environment. the [`python` context object is discussed here](http://docs.aws.amazon.com/lambda/latest/dg/python-context-object.html).\n",
    "\n",
    "it will tell you (among many other things):\n",
    "\n",
    "+ how long till your function times out\n",
    "+ the memory limit of this `lambda` function\n",
    "+ the function name or version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "use of the `context` is pretty... context dependent. use of the `event` is much more common"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "let's create our first `lambda` function using the `hello-world-python3` blueprint\n",
    "\n",
    "<br><div align=\"center\">**walkthrough: create a hello world `lambda` function**</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "this is following [the walkthrough in the `lambda` documentation](https://docs.aws.amazon.com/lambda/latest/dg/getting-started-create-function.html)\n",
    "\n",
    "1. go to [the `lambda` service dashboard](https://console.aws.amazon.com/lambda/home?region=us-east-1#/home)\n",
    "2. create a function\n",
    "    1. blueprint\n",
    "        1. search for hello-world-python3 and select it\n",
    "    2. triggers\n",
    "        1. do nothing and click \"next\"\n",
    "    3. function\n",
    "        1. give it whatever name you want\n",
    "        2. the rest of the configurations are okay\n",
    "    4. review\n",
    "        1. just do it!\n",
    "3. testing it\n",
    "    1. be on the function page\n",
    "        1. you were dropped there after creating it\n",
    "        2. but if you left, you can get there be clicking \"functions\" in the left panel and selecting the name of your function\n",
    "    2. click the \"test\" button\n",
    "    3. use the default `json`\n",
    "        1. if there is no default, `{\"key3\": \"value, \"key2\": \"value2\",, \"key1\": \"value1\"}`\n",
    "4. monitoring\n",
    "    1. click the monitoring tab\n",
    "        1. check out the invocation and time used graph\n",
    "    2. click on the cloudwatch logs\n",
    "        1. look at the log stream "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## `trigger`s worth knowing about\n",
    "\n",
    "there are a lot of options for triggers, and depending on what you want to do with `aws` you will explore any number of them. that being said, there are a few that have immediate impact on services we've used already or will be likely to use while doing data science.\n",
    "\n",
    "generally you already know what the event you're responding to \"is\" -- you intentionally dropped an important file in an `s3` bucket, e.g."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "the main question to ask is:\n",
    "\n",
    "*what does the `json` object passed from the triggered event look like, and how can / should we use it in our function?*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### scheduled events\n",
    "\n",
    "the `lambda` world has its own version of `cron` in `CloudWatch Events`. the same way that we were able to create shell scripts that could be executed on some frequency, we can creaet `lambda` functions and execute them with some frequency."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "when we set the frequency of the scheduled event, we can use a modified `cron` syntax, or rate expressions -- docs are [here](https://docs.aws.amazon.com/lambda/latest/dg/tutorial-scheduled-events-schedule-expressions.html)\n",
    "\n",
    "```bash\n",
    "# rate example: daily\n",
    "rate(1 day)\n",
    "\n",
    "# cron example: work weekdays at 6 AM UTC\n",
    "cron(0 6 ? * MON-FRI *)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<div align=\"center\">**walkthrough: creating a `cloudwatch` scheduled event `lambda` function**</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "the walkthrough goes through the following steps\n",
    "\n",
    "+ create a new function\n",
    "+ on the trigger menu, select \"Cloudwatch Event\"\n",
    "+ enter a cron expression\n",
    "+ use the code below as the function defitinion\n",
    "+ submit\n",
    "\n",
    "```python\n",
    "def lambda_handler(event, context):\n",
    "    print(\"event = {}\".format(event))\n",
    "    return 'hello world'\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "whatever particular configuration we chose, an actual `cloudwatch` event `rule` was created. \n",
    "\n",
    "we also could have directly created the cloudwatch rule ahead of time via the `cloudwatch` service, or we could let `lambda` build one for us."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "let's check out the rule we created in our walkthrough by heading to [the `cloudwatch` service](https://console.aws.amazon.com/cloudwatch/home?region=us-east-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<div align=\"center\">**walkthrough: looking at `cloudwatch` scheduled event options**</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "the walkthrough goes through the following pages\n",
    "\n",
    "+ navigate to the `cloudwatch` service\n",
    "+ select the \"Events > Rule\" page from the left menu\n",
    "+ select any rule to go to the rule's summary page\n",
    "+ click edit\n",
    "    + source\n",
    "        + event pattern (trigger on *any* event across `aws` services)\n",
    "        + schedule (rate or cron expression\n",
    "        + dropdown of sample event indicates exactly what the event is\n",
    "    + target: `lambda` function\n",
    "        + this happens to point at a `lambda` function (because it was built for us via `lambda`)\n",
    "        + the `input` discussed here is the `input` `event` object for the `lambda` function\n",
    "        + we have options\n",
    "            + the exact object\n",
    "            + a single sub-element from the object\n",
    "            + a *constant* version\n",
    "            + a value constructure from the components of the original event object (based on key-value replacement)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "so, the basic `event` object is\n",
    "\n",
    "```json\n",
    "// cloudwatch scheduled event example\n",
    "{\n",
    "  \"account\": \"123456789012\",\n",
    "  \"region\": \"us-east-1\",\n",
    "  \"detail\": {},\n",
    "  \"detail-type\": \"Scheduled Event\",\n",
    "  \"source\": \"aws.events\",\n",
    "  \"time\": \"1970-01-01T00:00:00Z\",\n",
    "  \"id\": \"cdc73f9d-aea9-11e3-9d5a-835b769c0d9c\",\n",
    "  \"resources\": [\n",
    "    \"arn:aws:events:us-east-1:123456789012:rule/my-schedule\"\n",
    "  ]\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "we could take our `hello world` example and make a small modification to incorporate elements of this event:\n",
    "\n",
    "```python\n",
    "def lambda_handler(event, context):\n",
    "    print(\"event = {}\".format(event))\n",
    "    return 'hello world! it is {}'.format(event['time'])\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<div align=\"center\">**mini exercise: create a scheduled hello world `lambda` function**</div>\n",
    "\n",
    "1. create a new `lambda` function with whatever name you'd like\n",
    "2. use the definition below for the function\n",
    "3. trigger the event on a `cloudwatch` scheduled event\n",
    "4. schedule the even to trigger every M-F at 6:00 UTC\n",
    "5. test your lambda function\n",
    "\n",
    "```python\n",
    "def handler(event, context):\n",
    "    print(\"event = {}\".format(event))\n",
    "    return 'hello world! it is {}'.format(event['time'])\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### `s3`\n",
    "\n",
    "suppose you created a process which checks every day for an update to some important externally-managed datasource, and downloads any new items posted to that external source to your `s3` bucket -- good thinking!\n",
    "\n",
    "also suppose that this new data affects a model you already trained, but with the previous version of that data set. wouldn't it be awesome if you could have your training process kick off automatically whenever a new dataset arrives?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "it will surely come as a *great shock* that you can do this with `lambda` functions. can you believe it? I posed that arbitrary hypothetical in this lecture on `lambda` functions and here we are, possibly addressing that hypothetical.\n",
    "\n",
    "wild!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<div align=\"center\">**walkthrough: creating a useless `s3` event triggered `lambda` function**</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "the walkthrough does the following\n",
    "\n",
    "1. navigate to the `s3` service page\n",
    "    1. create an `s3` event-producing bucket\n",
    "    2. create a subfolder in that bucket\n",
    "2. navigate back to the `lambda` service page\n",
    "3. create a new `lambda` function\n",
    "    1. no blueprint\n",
    "    2. trigger type is `s3`\n",
    "    3. configure trigger\n",
    "        1. pick the bucket you create in step 1.1\n",
    "        2. event type is \"Object Created (All)\"\n",
    "        3. prefix is the subfolder you chose in step 1.2 (with a trailing \"/\" character, e.g. \"myfolder/\")\n",
    "        4. no suffix needed\n",
    "        5. enable trigger\n",
    "    4. configure function\n",
    "        1. use the `python 3.6` code below (does nothing but print our event)\n",
    "    5. save\n",
    "4. test new `lambda` function\n",
    "    1. click the \"actions\" dropdown to select the \"configure test event\" option\n",
    "    2. selet an `s3 put` event template\n",
    "    3. \"save and test\"\n",
    "    \n",
    "```python\n",
    "def lambda_handler(event, context):\n",
    "    print(\"event = {}\".format(event))\n",
    "    return 'hello world!'\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "toward the end of that walkthrough, we created a test `s3 PUT` event that looked like the following:\n",
    "\n",
    "```json\n",
    "{\n",
    "    \"Records\": [\n",
    "        {\n",
    "            \"eventVersion\": \"2.0\",\n",
    "            \"eventTime\": \"1970-01-01T00:00:00.000Z\",\n",
    "            \"requestParameters\": {\"sourceIPAddress\": \"127.0.0.1\"},\n",
    "            \"s3\": {\n",
    "                \"configurationId\": \"testConfigRule\",\n",
    "                \"object\": {\n",
    "                    \"eTag\": \"0123456789abcdef0123456789abcdef\",\n",
    "                    \"sequencer\": \"0A1B2C3D4E5F678901\",\n",
    "                    \"key\": \"HappyFace.jpg\",\n",
    "                    \"size\": 1024\n",
    "                },\n",
    "                \"bucket\": {\n",
    "                    \"arn\": \"arn:aws:s3:::mybucket\",\n",
    "                    \"name\": \"sourcebucket\",\n",
    "                    \"ownerIdentity\": {\"principalId\": \"EXAMPLE\"}\n",
    "                },\n",
    "                \"s3SchemaVersion\": \"1.0\"\n",
    "            },\n",
    "            \"responseElements\": {...},\n",
    "            \"awsRegion\": \"us-east-1\",\n",
    "            \"eventName\": \"ObjectCreated:Put\",\n",
    "            \"userIdentity\": {\"principalId\": \"EXAMPLE\"},\n",
    "            \"eventSource\": \"aws:s3\"\n",
    "        }\n",
    "    ]\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "let's update our hello world script to (pretend to) run our data science pipeline process\n",
    "\n",
    "```python\n",
    "def lambda_handler(event, context):\n",
    "    print(\"event = {}\".format(event))\n",
    "    s3info = event['Records'][0]['s3']\n",
    "    # return a str version of the command we could execute\n",
    "    cmd = 'dspipeline.update_file(bucket={}, key={})'.format(\n",
    "        s3info['bucket']['name'], \n",
    "        s3info['object']['key']\n",
    "    )\n",
    "    print(cmd)\n",
    "    return cmd\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "<div align=\"center\">**mini exercise: create an `s3`-triggered hello world `lambda` function**</div>\n",
    "\n",
    "1. create a new `s3` bucket and subfolder\n",
    "2. create a new `lambda` function triggered on any new file in that subfolder\n",
    "3. use the (`python 3.6` runtime) code below as the function defintion\n",
    "4. test it out by creating a new file in `s3` and looking at the `cloudwatch` logs\n",
    "    \n",
    "```python\n",
    "def lambda_handler(event, context):\n",
    "    print(\"event = {}\".format(event))\n",
    "    s3info = event['Records'][0]['s3']\n",
    "    # return a str version of the command we could execute\n",
    "    cmd = 'dspipeline.update_file(bucket={}, key={})'.format(\n",
    "        s3info['bucket']['name'], \n",
    "        s3info['object']['key']\n",
    "    )\n",
    "    print(cmd)\n",
    "    return cmd\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### api gateway\n",
    "\n",
    "this service allows you to create your own RESTful api endpoint -- a url to which you may send `GET, POST, PUT,` or `DELETE` requests to perform some action. that action that you perform can very easily be a `lambda` function!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "why might we want to do this?\n",
    "\n",
    "as we will see later, it is possible (even easy) to invoke a serverless `lambda` function from code which is written using one of the `aws sdk`s (e.g. `boto3`), and in the `aws` environment. but what if we don't have access to these tools or environments for parts of our project?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "RESTful apis are language agnostic and highly standardized ways of providing an integration point to the `lambda` function we wrote and external code. you can expose this `lambda` function as an api endpoint and anyone with knowledge and permissions can invoke it in any way they want"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<div align=\"center\">**walkthrough: creating an api gateway triggered `lambda` function**</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "the walkthrough does the following\n",
    "\n",
    "1. create a new `lambda` function\n",
    "    1. no blueprint\n",
    "    2. trigger type is `api gateway`\n",
    "    3. configure trigger\n",
    "        1. we can pick any name (e.g. `HelloWorldTest`)\n",
    "        2. we leave all other defaults the same\n",
    "    4. configure function\n",
    "        1. use the `python 3.6` code below (does nothing but print our event)\n",
    "    5. save\n",
    "2. test new `lambda` function\n",
    "    1. click the \"actions\" dropdown to select the \"configure test event\" option\n",
    "    2. enter the `json` test message below\n",
    "    3. \"save and test\"\n",
    "    \n",
    "```python\n",
    "import json\n",
    "\n",
    "def respond(err, res=None):\n",
    "    return {\n",
    "        'statusCode': '400' if err else '200',\n",
    "        'body': err.message if err else json.dumps(res),\n",
    "        'headers': {'Content-Type': 'application/json'},\n",
    "    }\n",
    "\n",
    "\n",
    "def lambda_handler(event, context):\n",
    "    print(\"event = {}\".format(event))\n",
    "\n",
    "    reqtype = event['httpMethod']\n",
    "    if reqtype == 'GET':\n",
    "        return respond(\n",
    "            err=None,\n",
    "            res='value 1 is {}'.format(\n",
    "                event['queryStringParameters'].get('key1')\n",
    "            )\n",
    "        )\n",
    "    else:\n",
    "        return respond(ValueError('Unsupported method \"{}\"'.format(reqtype)))\n",
    "```\n",
    "\n",
    "```json\n",
    "// simple json test GET request\n",
    "// change the method POST\" for different test case\n",
    "{\n",
    "  \"queryStringParameters\": {\n",
    "    \"foo\": \"bar\",\n",
    "    \"key1\": \"val1\"\n",
    "  },\n",
    "  \"httpMethod\": \"GET\"\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "one thing to note here: the `event` we get is pretty much a direct pass-through of the `http` request sent to the api gateway -- in this sense, we can define whatever structure we want, and impose it on the downstream consumer (so long as it fits within the [`http` message structure](https://developer.mozilla.org/en-US/docs/Web/HTTP/Messages)).\n",
    "\n",
    "let's look at this from the point of view of the api gateway we just created."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### `api gateway` object"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<div align=\"center\">**walkthrough: looking at `api gateway` event options**</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "the walkthrough goes through the following pages\n",
    "\n",
    "+ navigate to the `api gateway` service\n",
    "+ select the \"APIs > [YOUR API NAME] > Resources\" page from the left menu\n",
    "+ select the api endpoint you just created\n",
    "+ click on the ANY method (it could have been GET, POST, etc)\n",
    "    + test\n",
    "        + allows us to perform a http request (a la `curl`)\n",
    "        + can set arbitrary parameters and see what our response is\n",
    "        + try `GET` with nothing\n",
    "            + it fails with a cryptic message... maybe we should fix that... two steps later...\n",
    "        + try `GET` with paramaters `key1=val1&key2=val2`. SUCCESS!\n",
    "        + try `POST`. FAILURE, but more descriptive failure. that's progress.\n",
    "    + method request\n",
    "        + defines the interface of the api endpoint\n",
    "        + url query string parameters section\n",
    "            + let's update it to *require* parameter key1, and suggest key2 and key3\n",
    "            + let's update the docs for each of those (click the book on the same line)\n",
    "            + required error message says we should create a validator -- let's do it!\n",
    "        + request validator\n",
    "            + click the edit pencil\n",
    "            + select query string and header option\n",
    "    + back to test\n",
    "        + try `GET` with nothing now. cool!\n",
    "    + integration request\n",
    "        + this piece specifically configures the handoff to `lambda`\n",
    "    + lambda function is what we know from before\n",
    "    + integration response: NA\n",
    "    + method resposne\n",
    "        + allows us to provide specific documentation and structure for non-200 responses\n",
    "+ deploy!\n",
    "    + actions button dropdown > deploy api\n",
    "    + basic configs\n",
    "+ stage: prod\n",
    "    + check out export tab > export as swagger"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "let's put that api to work!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "url = 'https://qxmbtpq6lf.execute-api.us-east-1.amazonaws.com/prod/hello_world_api'\n",
    "\n",
    "resp = requests.get(url=url, params={'key1': 'val1', 'key2': 'val2'})\n",
    "resp.json()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## `cli` integration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### invoking a function\n",
    "\n",
    "the `aws lambda` function has an `invoke` subcommand. let's read up on it via the `aws lambda invoke help` command"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "let's go back to our first \"no trigger\" hello world lambda function. I called mine `helloworld_blueprint`, so I should be able to invoke it as follows:\n",
    "\n",
    "```bash\n",
    "aws lambda invoke  \\\n",
    "    --function-name helloworld_blueprint  \\\n",
    "    --payload '{\"key1\": \"val1\", \"key2\": \"val2\", \"key3\": \"val3\"}'  \\\n",
    "    /tmp/out.txt\n",
    "```\n",
    "\n",
    "*note*: if you experience an error, you will get a return with a status code of 200 and an error message. check `/tmp/out.txt` for details"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "`boto` also has tie-ins:\n",
    "\n",
    "```python\n",
    "import boto3\n",
    "import json\n",
    "\n",
    "session = boto3.session.Session()\n",
    "l = session.client('lambda')\n",
    "l.list_functions()\n",
    "\n",
    "payload = {\"key1\": \"val1\", \"key2\": \"val2\", \"key3\": \"val3\"}\n",
    "j = l.invoke(\n",
    "    FunctionName='helloworld_blueprint',\n",
    "    Payload=json.dumps(payload)\n",
    ")\n",
    "j['Payload'].read()\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "two important notes\n",
    "\n",
    "1. `lambda` is a reserved keyword in `python`, so if you get `SyntaxError`s popping up in your script make sure you're not playing fast and loose with the names\n",
    "2. there is not yet a `resource` for `lambda` (our preferred access approach), so we default to the lower-level `client`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### creating a function from local code files\n",
    "\n",
    "it is also possible to create `lambda` functions using the `cli` and not the web interface. this may be more convenient in the moment (*e.g.* the function code is collected and working on your `ec2` server, or you want to create 100s of functions with slight parameterization differences)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "in these instances it may be nice to be able to use the `aws lambda create-function` command. let's demo that now!\n",
    "\n",
    "this is a little more involved, but gives us a chance to stretch our command-line legs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<div align=\"center\">**mini-exercise: preparing a `lambda` function deployment package**</div>\n",
    "\n",
    "1. get the `alarm_clock.py` file\n",
    "    1. if you don't alread have it: `wget https://s3.amazonaws.com/shared.rzl.gu511.com/alarm_clock.py`\n",
    "2. create a zip archive with that `alarm_clock.py` file in it\n",
    "    1. install `zip`: `sudo apt install zip`\n",
    "    2. `zip -r9 /tmp/alarm_clock.zip alarm_clock.py` (must be in the directory containing `alarm_clock.py`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<div align=\"center\">**mini-exercise: deploying a `lambda` function with the `cli`**</div>\n",
    "\n",
    "1. look up the `iam role arn` you used for the first `helloworld` `lambda` function\n",
    "    1. https://console.aws.amazon.com/iam/home?region=us-east-1#/roles\n",
    "    2. *e.g.* `arn = arn:aws:iam::134461086921:role/service-role/helloworld_blueprint_role` \n",
    "2. execute the `aws cli` command below\n",
    "3. check it out at [the `lambda` console](https://console.aws.amazon.com/lambda/home?region=us-east-1#/functions)\n",
    "\n",
    "```bash\n",
    "# note: copy-paste may break dashes or quotes :(\n",
    "aws lambda create-function \\\n",
    "    --function-name helloworld_from_cli_test \\\n",
    "    --zip-file fileb:///tmp/alarm_clock.zip \\\n",
    "    --role \"arn:aws:iam::134461086921:role/service-role/helloworld_blueprint_role\" \\\n",
    "    --handler alarm_clock.lambda_handler \\\n",
    "    --runtime python3.6\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## free tier limits\n",
    "\n",
    "we've done a lot today, so now is perhaps not the *best* time to figure out whether or not we've blown through a stack of cash. as you might guess, what we've done so far is totally free (or I really messed up!)\n",
    "\n",
    "how much can we get away with for free? let's check out the pricing info:\n",
    "\n",
    "https://aws.amazon.com/lambda/pricing/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "***<div align=\"center\">not too baaaaaaaad</div>***\n",
    "<div align=\"center\"><img src=\"https://regmedia.co.uk/2014/11/13/lamb.jpg?x=1200&y=794\" width=\"500px\"></div>\n",
    "\n",
    "# END OF LECTURE\n",
    "\n",
    "next lecture: [AWS `databases`](012_dbs_1.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "things created in this lecture that you may want to clean up afterward:\n",
    "\n",
    "1. `lambda` functions\n",
    "    1. all are named `helloworld_{event type}_test`, and you can delete them all\n",
    "2. `iam role` items\n",
    "    1. all are named `helloworld_{event type}_test_role` and you can delete them all\n",
    "3. events\n",
    "    1. `s3` bucket, subdirectory, and attached event\n",
    "    2. `Cloudwatch` scheduled event rule\n",
    "    3. `api gateway` endpoint url"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}