Lectures/data_intensive_applications_the_sane_way/Testing Data Intensive Applications, the sane way-15min.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![](lecture_title.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# About Me 🙈🚀🚀🚀🚀🚀🚀🚀🚀"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Data architect at bigabid 👷.\n",
    "- Data architect consultant 🤓.\n",
    "- Big passion for python, big data, databases and machine learning 🐍🤖.\n",
    "- Online at [medium](https://medium.com/@Eyaltra) | [website](https://eyaltrabelsi.github.io/) 🌐"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Today\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<br>\n",
    "\n",
    "- Data-intensive applications in general.\n",
    "- Testing in general.\n",
    "- Tests to avoid dumb bugs.\n",
    "- Tests to check code correctness.\n",
    "- Tests to check artifacts' correctness."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Data Intensive applications\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "\n",
    "<br>\n",
    "\n",
    "- Complex by nature.\n",
    "- Consist of configurations, scripts/notebooks, queries, and pipelines.\n",
    "- Result in datasets, models, and dashboards.\n",
    "- Break in different ways than regular software."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<i><big>\"Testing is a contract between the present and the future.\"</big></i>📄"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "# Should we do automatic testing?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br>\n",
    "\n",
    "    + Reduces \"whoops\" moments 🦺.\n",
    "    + Enabling fearless development 🏗️.\n",
    "    + Documentation of the program expectations 📚."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "    - Knowing what can go wrong and how to test requires some experience 👴.\n",
    "    - They can fail incorrectly 🤡.\n",
    "    - May take a long time to run 🐢.\n",
    "    - Takes time to write and maintain them ⏰.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# We don't want dumb bugs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "outputs": [],
   "source": [
    "from IPython.display import display, HTML\n",
    "display(HTML(\"<style>.container { width:100% !important; }</style>\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<br>\n",
    "\n",
    "| What | Expectations | How | Do I Cover this? |\n",
    "| --- | --- | --- |--- |\n",
    "| Code | Has valid syntax | [precommit](https://pre-commit.com/) |🟩|\n",
    "| Code | Has valid syntax and types | [mypy](https://mypy.readthedocs.io/en/stable/getting_started.html) |🟥|\n",
    "| Query | Has valid syntax | [sqlfluff](https://github.com/sqlfluff/sqlfluff) |🟥 |\n",
    "| Pipeline | Dags are imported correctly | Importing dags |🟩|\n",
    "| Query | Has valid syntax and schema | Explain |🟩|\n",
    "| Configurations | Has valid syntax | [precommit](https://pre-commit.com/) |🟩|\n",
    "| Configurations | Has valid syntax and schema | [cerberus](https://docs.python-cerberus.org/en/stable/) | 🟥|\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## precommit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<br>\n",
    "\n",
    "- precommit identify simple issues using static code analysis.\n",
    "- A form of testing that looks at the code without running it.\n",
    "- precommit has many types of validations.\n",
    "- ```pip install pre-commit```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "\n",
    "``` yaml\n",
    "// .pre-commit-config.yaml\n",
    "repos:\n",
    "-   repo: https://github.com/pre-commit/pre-commit-hooks\n",
    "    rev: v2.3.0\n",
    "    hooks:\n",
    "    -   id: check-ast\n",
    "    -   id: check-yaml\n",
    "    -   id: check-json\n",
    "```      "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- **Difficulty**: Easy to test 👶.\n",
    "- **Execution time**: Fast 🐆."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- **Pro Tip**💃: start with the basic and add schema/types if needed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Queries have valid syntax and schema"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<br>\n",
    "\n",
    "- Prepend Explain to a query results in the query's execution plan.\n",
    "- If your query is invalid, the Explain query will fail.\n",
    "- Available in most SQL dialects.\n",
    "- Require database connectivity.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "``` python          \n",
    "def test_queries_synatx():\n",
    "    engine = create_engine('<COOL_CONNECTION_STRING>')\n",
    "    with engine.connect() as con:\n",
    "        queries_list = glob.glob('**/*.sql',recursive=True)\n",
    "        for query_file in queries_list:\n",
    "            with open(query_file) as f:\n",
    "                query = f.read()\n",
    "                test_query = f\"Explain {query}\"\n",
    "                con.execute(test_query)\n",
    "        \n",
    "                    \n",
    "    ```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- **Difficulty**: Easy* to test 👶.\n",
    "- **Execution time**: Fast* 🐆."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- **Pro Tip**💃: much \"stronger\" than static code analysis for queries."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Testing the code behavior"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<br>\n",
    "\n",
    "| What | Expectations | How | Do I Cover this? |\n",
    "| --- | --- | --- |--- |\n",
    "| Code | work in expected way on specific input | [pytest](https://docs.pytest.org/en/7.1.x/)/[doctest](https://docs.python.org/3/library/doctest.html) |🟩|\n",
    "| Code | is fast/scaleable enough | [pytest](https://docs.pytest.org/en/7.1.x/) |🟥|\n",
    "| Code | work in expected way on potential inputs | [pytest](https://docs.pytest.org/en/7.1.x/) + [hypothesis](https://hypothesis.readthedocs.io/) |🟩|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Work in expected way on specific input"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<br>\n",
    "\n",
    "- Given some example input(s), the output is correct.\n",
    "- Counter-examples should show up as incorrect."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "\n",
    "``` python\n",
    "def test_cool_code_on_specific_input():\n",
    "    text = 'WOW this is an awesome lecture'\n",
    "    text_bagged = bag_of_words(text)\n",
    "    assert len(text_bagged) == 6\n",
    "    assert ' ' not in text_bagged    \n",
    "```    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- **Difficulty**: Somewhat easy to test 👦.\n",
    "- **Execution time**: Tend to be fast 🐇."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- **Pro Tip**💃: isolate when possible.\n",
    "- **Pro Tip**💃: small is better*.\n",
    "- **Pro Tip**💃: cool functionality like @parametrize, @skipif, etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Work in expected way on potential inputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Given the input's specification, the output makes sense using properties.\n",
    "- Generating inputs using a specific strategy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Properties you could test:\n",
    "    - The code does not crash.\n",
    "    - Contract.Data invariants.\n",
    "    - Equivalent functions return the same results.\n",
    "    - Idempotent, cummutative, associative etc.\n",
    "    - Stateful model based tests."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "\n",
    "``` python\n",
    "\n",
    "from hypothesis import given\n",
    "from hypothesis import strategies as st\n",
    "\n",
    "@given(st.text())\n",
    "def test_cool_code_properties(s):\n",
    "    text_bagged = bag_of_words(s)\n",
    "    assert len(text_bagged) >= 0\n",
    "    assert ' ' not in text_bagged   \n",
    "    ```    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- **Difficulty**: Confusing, but somewhat easy to test 👦🤯.\n",
    "- **Execution time**: Tend to be slow 🐢."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- **Pro Tip**💃: when facing users' input, unit-tests are not enough.\n",
    "- **Pro Tip**💃: [PROPERTY TESTS + CONTRACTS = INTEGRATION TESTS](https://www.hillelwayne.com/post/pbt-contracts/).\n",
    "- **Pro Tip**💃: Many cool integrations if schema/type is known like [faker](https://github.com/joke2k/faker).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Testing the code artifacts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<br>\n",
    "\n",
    "| What | Expectations |  Do I Cover this? |\n",
    "| --- | --- | --- |\n",
    "| Dataset | generated datateset work in expected way |  🟩|\n",
    "| Model | generated model work in expected way|  🟥|\n",
    "| Model | is fast/scaleable enough|  🟥|\n",
    "| Dashboard | [generated dashboard work in expected way](https://www.youtube.com/watch?v=-MoWsngubI4&ab_channel=TableauTim)|  🟥|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Generated datateset  work in expected way"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Given the input's dataset, Run pipeline, Check output makes sense using data validations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Testing properties of the dataset:\n",
    "    - Has valid schema.\n",
    "    - Cardinality.\n",
    "    - Number of missing values.\n",
    "    - Underlying distributions.\n",
    "    - Check for leakage.\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "    \n",
    "\n",
    "``` python\n",
    "     \n",
    "def test_cool_dataset_properties():\n",
    "    df = load_data(\"<COOL_MOCK_DATA_PATH>\")\n",
    "    run_pipeline(df) # Persist to <COOL_DATA_PATH>\n",
    "    actual_df = load_data(\"<COOL_DATA_PATH>\")\n",
    "    assert(set(actual_df.columns) == set([\"X\", \"Y\", \"Z\"]))\n",
    "    assert(actual_df[\"X\"].isna().sum()==0)\n",
    "    ```        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- **Difficulty**: Confusing 🤯, and tend to be hard test. 👨.\n",
    "- **Execution time**: Tend to be slow 🐢."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- **Pro Tip**💃: Use tools like great-expectations.\n",
    "- **Pro Tip**💃: Useful as a data quality mechanism in production."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "- **Pro Tip**💃: Treat any shared data as immutable.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "## Generated model work in expected way\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br>\n",
    "\n",
    "- Does model work in expected way on potential inputs?\n",
    "- Testing properties of the model:\n",
    "    - Check the shape of your model output.\n",
    "    - Check the output ranges.\n",
    "    - Check the output distribution.\n",
    "    - Check idempotent mutations.\n",
    "    - Check directional mutations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "``` python\n",
    "def test_cool_model_properties():\n",
    "    df = load_mock_input(\"<COOL_DATA_PATH>\")\n",
    "    model = load_model(\"<COOL_MODEL_PATH>\")\n",
    "    predictions = model.predict(df)\n",
    "    assert all(0 =< prediction < 120 for prediction in predictions age >= 0)\n",
    "    ```    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "- Does the model perform worse than previous datasets?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "``` python\n",
    "def test_model_versions_on_mock():\n",
    "    df = load_mock_input(\"<COOL_DATA_PATH>\")\n",
    "    old_model = load_old_model(\"<COOL_MODEL_PATH>\")\n",
    "    new_model = load_new_model(\"<COOL_MODEL_PATH>\")\n",
    "    predictions = model.predict(df)\n",
    "    assert is_better_results(new_model.predict(df), old_model.predict(df))\n",
    "    ```    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "- **Difficulty**: somewhat easy to test 👦.\n",
    "- **Execution time**: Tend to be slow 🐢.\n",
    "\n",
    "- **Pro Tip**💃: metamorphic testing is awesome."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# What is enough testing?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<br>\n",
    "\n",
    "- [Testing is a spectrum](http://squarism.com/2018/11/08/testing-is-a-spectrum/) - there's no such thing as no testing (hopefully🙏).\n",
    "- Write tests whose estimated value is greater than their estimated cost 🤑. \n",
    "- Is test coverage a useful metric? 🗺️\n",
    "- Incremental by nature 🪜."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- **Pro Tip**💃: Test only what matter.\n",
    "- **Pro Tip**💃: Use other methodologies when beneficial.\n",
    "- **Pro Tip**💃: Cheat on your homework."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![](https://memegenerator.net/img/instances/63439861.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![](sponsors.png)\n"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}