log_workshop_instructions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Datadog Logo](images/dd_logo.png)\n",
    "\n",
    "# Logging with Datadog Log Management\n",
    "\n",
    "Welcome to Logging with Datadog Log Management workshop.\n",
    "This repository contains a \"dummy\" Water System microservices project, a single page web application with microservices, already instrumented and analyzed using Datadog's APM and infrastructure products.\n",
    "\n",
    "![Our Single Page App](images/dashboard.png)\n",
    "\n",
    "This workshop shows you how log management can reduce your mean time to resolution should you encounter an issue with a given application by giving you the best setup practice and a global tour of Datadog Log Management solution.\n",
    "\n",
    "You have a few requirements to use this workshop, refer to the setup instruction <a href=\"https://github.com/l0k0ms/log-workshop-2#before-the-workshop--prerequisites\" target=\"_blank\">in the main README.md file</a> to configure your environment.\n",
    "\n",
    "<a href=\"http://www.twitter.com/burningion\" target=\"_blank\">Kirk Kaiser</a> has worked hard to make this workshop as helpful possible, but if you see something that could be improved, please feel free to create a <a href=\"https://github.com/burningion/distributed-tracing-with-apm-workshop\" target=\"_blank\">Github issue</a> on the repository, or reach out via the <a href=\"https://chat.datadoghq.com/\" target=\"_blank\">Datadog public slack</a>."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting Started \n",
    "\n",
    "If you already followed the workshop setup instructions, you can directly jump to step 3.\n",
    "\n",
    "1. If not done already, the first thing to do is to create a <a href=\"https://www.datadoghq.com\" target=\"_blank\">Datadog account</a>.\n",
    "\n",
    "2. Then clone this repository on your local machine: ```git clone https://github.com/l0k0ms/log-workshop-2.git```\n",
    "\n",
    "3. Launch the application using the `<DD_API_KEY>` <a href=\"https://app.datadoghq.com/account/settings#api\" target=\"_blank\">from your trial account</a>.\n",
    "    Your command should look like the following on MacOS/Linux:\n",
    "    ```bash\n",
    "    POSTGRES_USER=postgres POSTGRES_PASSWORD=123456 DD_API_KEY=<DD_API_KEY> docker-compose up --build\n",
    "    ```\n",
    "    For Windows, the process of setting environment variables is a bit different:\n",
    "    \n",
    "    ```\n",
    "    PS C:\\dev> $env:POSTGRES_USER=postgres\n",
    "    PS C:\\dev> $env:POSTGRES_PASSWORD=123456\n",
    "    PS C:\\dev> $env:DD_API_KEY=<DD_API_KEY>\n",
    "    PS C:\\dev> docker-compose up --build\n",
    "    ```\n",
    "    \n",
    "    **If your application is already running, stop it first with `docker-compose stop && docker-compose rm`**\n",
    "    \n",
    "4. After running the above command and seeing the logs flowing in your terminal go to <a href=\"http://localhost:5000/\" target=\"_blank\">http://localhost:5000/</a> and see the single page web app.\n",
    "    Refresh the page, click around, add a pump, try adding a city. This begins to generate metrics, APM traces, and logs for your application.\n",
    "    \n",
    "5. Go to Datadog, to see your application corresponding data:\n",
    "    * In the <a href=\"https://app.datadoghq.com/process\" target=\"_blank\">live processes view</a>\n",
    "    * In the <a href=\"https://app.datadoghq.com/containers\" target=\"_blank\">containers view</a>\n",
    "    * In the <a href=\"https://app.datadoghq.com/apm/services\" target=\"_blank\">APM Services view</a>\n",
    "    * In the <a href=\"https://app.datadoghq.com/service/map?env=workshop\" target=\"_blank\">APM Services Map view</a>\n",
    "    * In the <a href=\"https://app.datadoghq.com/infrastructure/map?fillby=avg%3Aprocess.stat.container.io.wbps&sizeby=avg%3Anometric&groupby=short_image&nameby=name&nometrichosts=false&tvMode=false&nogrouphosts=true&palette=YlOrRd&paletteflip=false&node_type=container\" target=\"_blank\">container map view</a>\n",
    "\n",
    "Tab back over to your terminal, and look over the container logs, go to your <a href=\"Datadog log management page\" target=\"_blank\">https://app.datadoghq.com/logs</a> and notice that there is no log yet... \n",
    "\n",
    "Our first mission should you choose to accept it, configure the Datadog Agent to start forwarding your logs into your Datadog Application."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Gathering our first logs\n",
    "\n",
    "![Empty Log explorer](images/logs_workshop/empty_log_explorer.png)\n",
    "\n",
    "There is no log yet in your <a href=\"https://app.datadoghq.com/logs\" target=\"_blank\">Log Explorer page</a>, because the Datadog Agent is not configured to gather them, to change this let's follow those steps:\n",
    "\n",
    "1. Add the following configuration lines in your `docker-compose.yml` file at the root of the workshop directory:\n",
    "    ```\n",
    "    datadog:\n",
    "      environment:\n",
    "        (...)\n",
    "        - DD_LOGS_ENABLED=true\n",
    "        - DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL=true\n",
    "      volume:\n",
    "        (...)\n",
    "        - /opt/datadog-agent/run:/opt/datadog-agent/run:rw\n",
    "    ```\n",
    "    \n",
    "| Configuration                                      | type         | Explanations                                    |\n",
    "| :----                                              | :-----       | :-----                                          |\n",
    "| `DD_LOGS_ENABLED=true`                             | env variable | Enable log collection                           |\n",
    "| `DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL=true`        | env variable | Enable log collection for all containers        |\n",
    "| `/opt/datadog-agent/run:/opt/datadog-agent/run:rw` | volume       | Used to store pointers on container current log |\n",
    "<a href=\"https://docs.datadoghq.com/logs/log_collection/docker/\" target=\"_blank\">Refer to the Datadog Agent log collection documentation to learn more</a>. \n",
    "\n",
    "2. Then restart your application:\n",
    "    * Run `docker-compose stop && docker-compose rm`\n",
    "    * Run `POSTGRES_USER=postgres POSTGRES_PASSWORD=123456 DD_API_KEY=<DD_API_KEY> docker-compose up --build`\n",
    "    \n",
    "    *Note*: On some OS you might see this error popping:\n",
    "    ```\n",
    "    ERROR: for agent  Cannot start service agent: b'Mounts denied: \\r\\nThe path /opt/datadog-agent/run\\r\\nis not shared ...\n",
    "    ```\n",
    "    To fix it either give the mount permission to this folder on your machine, or remove `\n",
    "    /opt/datadog-agent/run:/opt/datadog-agent/run:rw` from the `docker-compose.yml` file.\n",
    "\n",
    "3. Finally, go to your Datadog application in <a href=\"https://app.datadoghq.com/logs/\" target=\"_blank\">`Log -> Explorer`</a> and check your logs flowing.\n",
    "\n",
    "<img src=\"images/logs_workshop/log_flow.png\" alt=\"Log Flow\" width=\"80%\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using Label to correctly tag logs\n",
    "\n",
    "As you can notice in the previous screenshot, all logs are currently showing up in the Datadog Log explorer view with the same service name `docker` -which is technically true- but in order to gain more visibility about which container emitted which logs and in order to bind your logs with the previously implemented APM and metrics, let's use Labels to specify the `source` and the `service` tags for each container logs. \n",
    "\n",
    "* **The `source` tag is key to enable the integration pipeline**\n",
    "\n",
    "    Datadog has a range of <a href=\"https://docs.datadoghq.com/integrations/#cat-log-collection\" target=\"_blank\">Log supported integrations</a>. In order to enable the <a href=\"https://docs.datadoghq.com/logs/processing/pipelines/\" target=\"_blank\">Log integration pipelines</a> in Datadog, pass the `source` name as a value for the source attribute with a docker label.\n",
    "\n",
    "* **The `service` tag is key for binding metrics traces and logs.**\n",
    "\n",
    "    The application is already instrumented for APM. Let's add the `service` tags to the `iot-frontend`, `noder`, `pumps`, `redis`, `sensor`, `db` and `adminer` containers in order to be able to bind their traces and their logs together.\n",
    "\n",
    "Update your `docker-compose.yml` file at the root directory of the workshop with the following labels:\n",
    "\n",
    "```\n",
    "version: '3'\n",
    "services:\n",
    "  agent:\n",
    "    (...)\n",
    "    labels:\n",
    "      com.datadoghq.ad.logs: '[{\"source\": \"docker\", \"service\": \"agent\"}]'\n",
    "\n",
    "  frontend:\n",
    "    (...)\n",
    "    labels:\n",
    "      com.datadoghq.ad.logs: '[{\"source\": \"iot-frontend\", \"service\": \"iot-frontend\"}]'\n",
    "\n",
    "  noder:\n",
    "    (...)\n",
    "    labels:\n",
    "      com.datadoghq.ad.logs: '[{\"source\": \"noder\", \"service\": \"noder\"}]'\n",
    "      \n",
    "  pumps:\n",
    "    (...)\n",
    "    labels:\n",
    "      com.datadoghq.ad.logs: '[{\"source\": \"pumps-service\", \"service\": \"pumps-service\"}]'\n",
    "\n",
    "  redis:\n",
    "    (...)\n",
    "    labels:\n",
    "      com.datadoghq.ad.logs: '[{\"source\": \"redis\", \"service\": \"redis\"}]'\n",
    "      \n",
    "  sensors:\n",
    "    (...)\n",
    "    labels:\n",
    "      com.datadoghq.ad.logs: '[{\"source\": \"sensors\", \"service\": \"sensors-api\"}]'\n",
    "\n",
    "  db:\n",
    "    (...)\n",
    "    labels:\n",
    "      com.datadoghq.ad.logs: '[{\"source\": \"postgresql\", \"service\": \"postgres\"}]'\n",
    "\n",
    "  adminer:\n",
    "    (...)\n",
    "    labels:\n",
    "      com.datadoghq.ad.logs: '[{\"source\": \"adminer\", \"service\": \"adminer\"}]'\n",
    "```\n",
    "\n",
    "Then restart your application:\n",
    "\n",
    "1. Run `docker-compose stop && docker-compose rm`\n",
    "2. Run `POSTGRES_USER=postgres POSTGRES_PASSWORD=123456 DD_API_KEY=<DD_API_KEY> docker-compose up`\n",
    "\n",
    "And go to <a href=\"http://localhost:5000/\" target=\"_blank\">http://localhost:5000/</a> to generate some actions. \n",
    "\n",
    "Finally, go to your Log explorer view to see the new `service` tags flowing in:\n",
    "\n",
    "<img src=\"images/logs_workshop/log_flow_with_service.png\" alt=\"Log flow with service\" width=\"80%\"/>\n",
    "\n",
    "### Switching between Traces and Logs\n",
    "\n",
    "The `service` tag now allows us to switch between our log explorer view and the corresponding APM service:\n",
    "\n",
    "<img src=\"images/logs_workshop/iot_frontend_switch.png\" alt=\"iot-frontend switch\" width=\"80%\"/>\n",
    "\n",
    "1. Open a log from `iot-frontend` by clicking on it.\n",
    "\n",
    "2. On top of the contextual panel click on the `iot-frontend` Service name.\n",
    "\n",
    "3. You should arrive on this page in Datadog APM:\n",
    "<img src=\"images/logs_workshop/iot-frontend_service_page.png\" alt=\"iot-frontend_service_page\" width=\"80%\"/>\n",
    "\n",
    "4. Open the `simulate_sensor` resource and then any given trace, when switching to the log tab you should see the corresponding logs:\n",
    "<img src=\"images/logs_workshop/logs_in_trace.png\" alt=\"logs_in_trace\" width=\"80%\"/>\n",
    "\n",
    "5. Click on the log to get back to the log explorer view.\n",
    "\n",
    "### Switching between Metrics and Logs\n",
    "\n",
    "<img src=\"images/logs_workshop/installed_integrations.png\" alt=\"installed_integrations\" width=\"50%\"/>\n",
    "\n",
    "Since our containers are correctly labeled, install the <a href=\"https://app.datadoghq.com/account/settings#integrations/docker\" target=\"_blank\">Datadog-Docker integration</a> and <a href=\"https://app.datadoghq.com/account/settings#integrations/redis\" target=\"_blank\">Datadog-Redis integration</a> to benefit from out of the box Dashboard:\n",
    "\n",
    "* <a href=\"https://app.datadoghq.com/screen/integration/52/docker---overview\" target=\"_blank\">Docker Dashboard</a>\n",
    "* <a href=\"https://app.datadoghq.com/screen/integration/15/redis---overview\" target=\"_blank\">Redis Dashboard</a>\n",
    "\n",
    "On a any given dashboard you can click on a displayed metric to switch to the corresponding logs:\n",
    "\n",
    "<img src=\"images/logs_workshop/metrics_switch_to_logs.png\" alt=\"metrics_switch_to_logs\" width=\"50%\"/>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Logging without limit\n",
    "\n",
    "Now that our logs are correctly labeled we are able to manipulate them during their processing in Datadog.\n",
    "\n",
    "Let's go to the <a href=\"https://app.datadoghq.com/logs/pipelines\" target=\"_blank\">Pipeline page</a> of Datadog and see what we have:\n",
    "\n",
    "The `source` tag already enabled the `Docker` and `Redis` integration pipeline\n",
    "\n",
    "<img src=\"images/logs_workshop/pipeline_page.png\" alt=\"Pipeline_page\" width=\"80%\"/>\n",
    "\n",
    "Which now automatically parse Docker Agent logs and Redis logs:\n",
    "\n",
    "<img src=\"images/logs_workshop/parsed_redis_log.png\" alt=\"Redis Logs\" width=\"50%\"/>\n",
    "\n",
    "\n",
    "### Exclusion filter \n",
    "\n",
    "Let's set up the following Index filters:\n",
    "\n",
    "![exclusion_filter](images/logs_workshop/exclusion_filter.png)\n",
    "\n",
    "#### Removing Agent log\n",
    "\n",
    "In order to clean our log explorer from logs that are not relevant for our use case let's implement an <a href=\"https://docs.datadoghq.com/logs/logging_without_limits/#exclusion-filters\" target=\"_blank\">index filter</a>:\n",
    "\n",
    "<img src=\"images/logs_workshop/index_filter_agent_log.png\" alt=\"index_filter_agent_log\" width=\"80%\"/>\n",
    "\n",
    "Learn more about <a href=\"https://docs.datadoghq.com/logs/logging_without_limits/\" target=\"_blank\">Logging without limits</a>.\n",
    "\n",
    "#### Removing Debug log\n",
    "\n",
    "As a general best practice, we also advise you to add an index filter on your Debug logs:\n",
    "\n",
    "<img src=\"images/logs_workshop/removing_debug_logs.png\" alt=\"removing_debug_logs\" width=\"80%\"/>\n",
    "\n",
    "Our log explorer view now only contains logs from our containers and no more from the Datadog Agent all logs matching the following query: `service:agent` are no longer reporting: \n",
    "\n",
    "<img src=\"images/logs_workshop/agent_filtered_out.png\" alt=\"agent_filtered_out\" width=\"80%\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Live tail \n",
    "\n",
    "Now that we filtered out our Agent logs and all our Debug logs, our explorer view is cleaner but we might still want to consult those logs. \n",
    "\n",
    "It's still possible with the of Datadog. <a href=\"https://app.datadoghq.com/logs/livetail\" target=\"_blank\">Live tail page</a>.\n",
    "\n",
    "The live tail page displays all logs after the Pipeline section but before the index filter one. If you enter the following query: `service:agent` you are able to see the parsed agent log even if they won't be indexed:\n",
    "\n",
    "![live_tail_agent](images/logs_workshop/live_tail_agent.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Parsing a full text log into JSON\n",
    "\n",
    "Most of the time all your logs won't be in JSON format, and if they are their attributes my differ between two log sources.\n",
    "\n",
    "Let's take the following log emitted by the `iot-frontend` service:\n",
    "\n",
    "<img src=\"images/logs_workshop/not_parsed_log.png\" alt=\"not_parsed_log\" width=\"80%\"/>\n",
    "\n",
    "```\n",
    "172.20.0.1 - - [12/Oct/2018 11:37:43] \"GET /simulate_sensors HTTP/1.1\" 200 -\n",
    "```\n",
    "\n",
    "And let's transform it to extract the IP adress, the date, the method, the URL, the scheme, and the status code: \n",
    "For this we are going to follow the <a href=\"https://docs.datadoghq.com/logs/processing/attributes_naming_convention/\" target=\"_blank\">Datadog Attribute Naming convention</a>.\n",
    "\n",
    "Let's start to go to the pipeline section again and create a new pipeline:\n",
    "\n",
    "<img src=\"images/logs_workshop/frontend_pipeline.png\" alt=\"frontend_pipeline\" width=\"80%\"/>\n",
    "\n",
    "**Note**: As a best practice it's recommended to set a filter for your pipeline in order to ensure that only logs matching a specific request will enter it.\n",
    "\n",
    "### Grok parser\n",
    "\n",
    "Create a *Grok parser* processor to parse your full text logs and transform it into a JSON.\n",
    "\n",
    "<img src=\"images/logs_workshop/grok_parser.png\" alt=\"grok_parser\" width=\"80%\"/>\n",
    "\n",
    "The full grok rule is:\n",
    "```\n",
    "rule %{ip:network.client_ip} - - \\[%{date(\"dd/MMM/yyyy HH:mm:ss\")}\\] \"%{word:http.method} %{notSpace:http.url} HTTP\\/%{number:http.version}\" %{number:http.status_code} %{notSpace:http.referer}\n",
    "```\n",
    "\n",
    "\n",
    "| Text                     | Pattern                               |\n",
    "| ---                      | ----                                  |\n",
    "| `172.20.0.1`             | `%{ip:network.client_ip}`             |\n",
    "| `[12/Oct/2018 11:44:58]` | `\\[%{date(\"dd/MMM/yyyy HH:mm:ss\")}\\]` |\n",
    "| `GET`                    | `%{word:http.method}`                 |\n",
    "| `/simulate_sensors`      | `%{notSpace:http.url}`                |\n",
    "| `HTTP/1.1`               | `HTTP\\/%{number:http.version}`        |\n",
    "| `200`                    | `%{number:http.status_code}`          |\n",
    "| `-`                      | `%{notSpace:http.referer}`            |\n",
    "\n",
    "### Category processor\n",
    "\n",
    "An access log by definition doesn't have any status attached, but there is a way to assign your log a status depending on the value of the `http.status_code` attribute. For this create a category processor:\n",
    "\n",
    "And add 4 categories to it:\n",
    "\n",
    "<img src=\"images/logs_workshop/create_a_category.png\" alt=\"create_a_category\" width=\"80%\"/>\n",
    "\n",
    "| All events that match:           | Appear under the value name: |\n",
    "| ----                             | ---                          |\n",
    "| `@http.status_code:[200 TO 299]` | ok                           |\n",
    "| `@http.status_code:[300 TO 399]` | notice                       |\n",
    "| `@http.status_code:[400 TO 499]` | warning                      |\n",
    "| `@http.status_code:[500 TO 599]` | error                        |\n",
    "\n",
    "\n",
    "### Status remapper\n",
    "\n",
    "Create a status remapper processor to take the category we just created and remap it as your official log status:\n",
    "\n",
    "<img src=\"images/logs_workshop/status_remapper.png\" alt=\"status_remapper\" width=\"80%\"/>\n",
    "\n",
    "### Url Parser\n",
    "\n",
    "Finally, create an url parser to extract all query parameters from your requested URL:\n",
    "\n",
    "<img src=\"images/logs_workshop/url_parser.png\" alt=\"url_parser\" width=\"80%\"/>\n",
    "\n",
    "### Final Log\n",
    "\n",
    "Now all your `iot-frontend` service logs are correctly parsed:\n",
    "\n",
    "<img src=\"images/logs_workshop/log_parsed.png\" alt=\"log_parsed\" width=\"80%\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Adding attribute as a Facet\n",
    "\n",
    "To add an attribute as a Facet and start using it in your log analytics, click on it:\n",
    "\n",
    "<img src=\"images/logs_workshop/create_facet.png\" alt=\"create_facet\" width=\"50%\"/>\n",
    "\n",
    "Don't forget to assign a group to your facet in order to avoid polluting your Log explorer view:\n",
    "\n",
    "<img src=\"images/logs_workshop/creating_facet_group.png\" alt=\"creating_facet_group\" width=\"50%\"/>\n",
    "\n",
    "You can then use this facet to filter your log explorer view:\n",
    "\n",
    "<img src=\"images/logs_workshop/using_facet_as_filter.png\" alt=\"using_facet_as_filter\" width=\"80%\"/>\n",
    "\n",
    "Or in your Log analytics:\n",
    "\n",
    "<img src=\"images/logs_workshop/log_analytics.png\" alt=\"log_analytics\" width=\"80%\"/>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Multi-line logs\n",
    "\n",
    "Let's kill a container and see what happens: \n",
    "\n",
    "1. Check the list of running container with `docker ps`\n",
    "2. Kill the container named `log-workshop-2_pumps` with `docker kill <CONTAINER_ID>`\n",
    "\n",
    "When trying to add a new pump in the application, nothing should happen and Traceback should appear in the log explorer, but they are not parsed well and the `\\n` inside of them is messing with the log wrapping: \n",
    "\n",
    "\n",
    "![traceback_not_wrapped](images/logs_workshop/traceback_not_wrapped.png)\n",
    "\n",
    "To compensate for this, two options are available:\n",
    "\n",
    "1. Log in JSON format in order to always have the Stacktrace properly wrapped (Recommended)\n",
    "2. Update the container label in order to specify to the Datadog Agent the pattern for a new log.\n",
    "\n",
    "Let's update the label with the following rule:\n",
    "\n",
    "```\n",
    "  frontend:\n",
    "    (...) \n",
    "    labels:\n",
    "      com.datadoghq.ad.logs: '[{\"source\": \"iot-frontend\", \"service\": \"iot-frontend\", \"log_processing_rules\": [{\"type\": \"multi_line\", \"name\": \"log_start_with_ip\", \"pattern\" : \"(172.20.0.1|Traceback)\"}]}]'\n",
    "```\n",
    "\n",
    "Then restart your application:\n",
    "\n",
    "1. Run `docker-compose stop && docker-compose rm`\n",
    "2. Run `POSTGRES_USER=postgres POSTGRES_PASSWORD=123456 DD_API_KEY=<DD_API_KEY> docker-compose up`\n",
    "\n",
    "Stacktraces from the `iot-frontend` service are now properly wrapped in the Log explorer view:\n",
    "\n",
    "![traceback_properly_wrapped](images/logs_workshop/traceback_properly_wrapped.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Monitoring your logs\n",
    "\n",
    "Let's build a monitor upon our logs that warns us if an error occurs and that send us the corresponding logs:\n",
    "\n",
    "\n",
    "1. Enter the search you want to monitor logs from in your Log explorer search bar:\n",
    "\n",
    "![monitor_query](images/logs_workshop/monitor_query.png)\n",
    "\n",
    "2. Click on the export to monitor button in the upper right corner of the Log explorer page:\n",
    "<img src=\"images/logs_workshop/export_to_monitor.png\" alt=\"export_to_monitor\" width=\"50%\"/>\n",
    "\n",
    "3. Set up a Warning and Alert threshold for your Log monitor\n",
    "\n",
    "4. Set the monitor title and template the notification sent.\n",
    "\n",
    "![monitor_configuration](images/logs_workshop/monitor_configuration.png)\n",
    "\n",
    "5. Save your monitor.\n",
    "\n",
    "6. Check that your monitor is correctly saved in your manage monitor page.\n",
    "\n",
    "![manage_monitor_page](images/logs_workshop/manage_monitor_page.png)\n",
    "\n",
    "\n",
    "### Monitor notification\n",
    "\n",
    "If you entered your email address in the notification, you should receive an email with a snippet of 10 logs matching your query:\n",
    "\n",
    "<img src=\"images/logs_workshop/mail_notification.png\" alt=\"mail_notification\" width=\"80%\"/>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}